Equipment control methods, devices, electronic equipment and storage media

Abstract

This application provides a device control method, apparatus, electronic device, and storage medium, relating to the field of Internet of Things (IoT) technology. The method includes: acquiring a target scene image to be detected; using a trained target detection model to detect target objects in the image to be detected, obtaining the detection result of the target objects; the target detection model is obtained by modifying model parameters and training, and the trained target detection model has the ability to detect target objects in the image to be detected; in an interactive interface used to display various smart devices in the target scene, binding the target objects to the smart devices in the target scene based on the detection results of the target objects; if a control command triggered in the interactive interface is detected, controlling the corresponding bound smart device to execute the action corresponding to the control command. This application solves the problem of poor device control effect caused by slow target detection speed and low accuracy in related technologies.

Description

Technical Field

[0001] This application relates to the field of Internet of Things (IoT) technology, and more specifically, to a device control method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of science and technology, artificial intelligence is increasingly being applied in people's daily lives. People are configuring various smart devices in the Internet of Things (IoT) and using object detection technology to better help users understand and use these smart devices.

[0003] Currently, object detection technology outputs several candidate boxes for each smart device, and makes predictions for each candidate box to determine whether a smart device exists in the candidate box. If it does, the detection result is output.

[0004] However, predicting for each candidate box involves a large amount of computation, resulting in slow target detection speed and low accuracy. Understandably, if users expect to control devices based on the detection results of smart devices, these defects will affect the effectiveness of device control. For example, if the target detection speed is slow and the smart device's detection results are delayed, then device control based on these detection results will also be delayed. Or, if the target detection results are erroneous and the smart device's detection results are incorrect, then device control of the corresponding smart device may not be possible based on these detection results. Summary of the Invention

[0005] This application provides a device control method, apparatus, electronic device, and storage medium, which can solve the problem of poor device control effect caused by slow target detection speed and low accuracy in related technologies. The technical solution is as follows: According to one aspect of this application, a device control method includes: acquiring a target scene image to be detected; using a trained target detection model to perform target detection on a target object in the target scene image to obtain a detection result of the target object; the target detection model is obtained by modifying the parameters related to candidate boxes in a machine learning model so that it no longer outputs multiple candidate boxes, but directly outputs a detection box for each category of target object, and training the machine learning model with modified parameters using training images, wherein the trained target detection model has the ability to detect target objects in the target scene image; in an interactive interface for displaying various smart devices in the target scene, determining the number of smart devices in the target scene that match the predicted category of the target object based on the detection result of the target object, and if the number is less than or equal to a set threshold, automatically binding the target object to the smart device in the target scene; if a control command triggered in the interactive interface is detected, controlling the corresponding bound smart device to perform an action corresponding to the control command.

[0006] According to one aspect of this application, a device control apparatus includes: an image acquisition module for acquiring a target scene image to be detected; a target detection module for using a trained target detection model to detect target objects in the target scene image and obtain the detection result of the target objects; the target detection model is obtained by modifying the parameters related to candidate boxes in a machine learning model so that it no longer outputs multiple candidate boxes, but directly outputs a detection box for each category of target object, and training the machine learning model with modified parameters using training images, wherein the trained target detection model has the ability to detect target objects in the target scene; a device binding module for determining, in an interactive interface for displaying various smart devices in the target scene, the number of smart devices in the target scene that match the predicted category of the target object based on the detection result of the target object, and automatically binding the target object to the smart devices in the target scene if the number is less than or equal to a set threshold; and a device control module for controlling the corresponding bound smart device to perform an action corresponding to the control command if a control command triggered in the interactive interface is detected.

[0007] In an exemplary embodiment, the device control apparatus further includes a model training module for acquiring training images of the target scene; the training images refer to images carrying labels; inputting the training images into a machine learning model with modified model parameters for target detection to obtain prediction results for each target object in the training images; calculating target loss values ​​based on the prediction results for each target object in the training images and the labels carried by the training images, and using the target loss values ​​to train the machine learning model until the target detection model that has been trained is obtained.

[0008] In an exemplary embodiment, the label is used to indicate the actual category of each target object in the training image and the actual position of each target object in the training image; the prediction result includes the prediction result score of the target object and the predicted position of the target object in the training image; the model training module is further used to calculate the category loss value of each target object in the training image based on the prediction result score of the target object and the actual category of the target object; calculate the position loss value of the target object based on the predicted position of the target object in the training image and the actual position of the target object in the training image; obtain the loss value of the target object based on the category loss value and the position loss value; and calculate the target loss value from the loss values ​​of each target object.

[0009] In an exemplary embodiment, the prediction result further includes the predicted category of the target object; the model training module is further configured to determine the expected value of the prediction result score based on whether the predicted category of the target object exists in the actual category of each target object; and calculate the category loss value of the target object based on the expected value of the prediction result score and the actual category of the target object.

[0010] In an exemplary embodiment, the model training module is further configured to calculate a first loss value and a second loss value based on the predicted position of the target object in the training image and the actual position of the target object in the training image; the first loss value reflects the difference between the predicted position and the actual position; the second loss value reflects the degree of overlap between the predicted position and the actual position; and the position loss value of the target object is calculated from the first loss value and the second loss value.

[0011] In an exemplary embodiment, the target detection module is further configured to input the image to be detected into the target detection model for feature extraction to obtain image features of the image to be detected; through the target detection model outputting a detection box for different categories of target objects, the position of each target object in the image to be detected, the predicted category of each target object, and the prediction result score of each target object are directly obtained based on the image features of the image to be detected; if the prediction result score of the target object meets the set conditions, the detection result of the target object is generated based on the position of the target object in the image to be detected and the predicted category of the target object.

[0012] In an exemplary embodiment, the detection result of the target object includes the predicted category and predicted location of the target object; the device binding module is further configured to display the interactive interface and search for the device location corresponding to the predicted location of the target object in the interactive interface; at the found device location, determine the number of smart devices in the target scene that match the predicted category of the target object; if the number is less than a set threshold, add corresponding control components to the interactive interface for the smart devices in the target scene according to the predicted category of the target object, so that the target object and the smart devices in the target scene automatically establish a binding relationship.

[0013] In an exemplary embodiment, the device binding module is further configured to detect a trigger operation related to a target smart device in the interactive interface if the number exceeds a set threshold; the target smart device refers to a smart device that matches the predicted category of the target object; if a trigger operation related to the target smart device is detected, a corresponding control component is added to the interactive interface for the target smart device according to the predicted category of the target object, so that the target object and the target smart device establish a binding relationship through the trigger operation.

[0014] In one exemplary embodiment, the device control module is further configured to generate a control command in response to a trigger operation on the control component in the interactive interface; send the control command to the smart device corresponding to the control component, so that the smart device responds to the control command and performs a corresponding action; the device binding module is further configured to update the component attributes of the corresponding control component according to the device status of the smart device after the action is performed, and update the display of the control component in the interactive interface.

[0015] According to one aspect of this application, an electronic device includes at least one processor and at least one memory, wherein program instructions or code are stored in the memory; the program instructions or code are loaded and executed by the processor, causing the electronic device to implement the device control method described above.

[0016] According to one aspect of this application, a storage medium has program instructions or code stored thereon, which are loaded and executed by a processor to implement a device control method as described above.

[0017] According to one aspect of this application, an application product includes program instructions or code stored in a storage medium. The processor of an electronic device reads the program instructions or code from the storage medium, loads and executes the program instructions or code, causing the electronic device to implement the device control method as described above.

[0018] The beneficial effects of the technical solution provided in this application are: In the above technical solution, a modified target detection model is used to detect target objects in the image to be detected. Based on the detection results, the target objects are bound to smart devices in the target scene, allowing users to control the smart devices through an interactive interface. Using a modified target detection model significantly reduces the amount of data computation during target detection, enabling rapid identification of target objects in the image with high accuracy. Therefore, on the one hand, rapid target detection enhances the real-time performance of device control; on the other hand, accurate detection results improve the precision of device control, thus solving the problem of poor device control performance caused by slow target detection speed and low accuracy in related technologies. Attached Figure Description

[0019] To more clearly illustrate the technical solutions provided in this application, the accompanying drawings used in the description of the various embodiments of this application will be briefly introduced below.

[0020] Figure 1 This is a schematic diagram of the implementation environment according to the embodiments of this application; Figure 2 This is a flowchart illustrating a device control method according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating a target detection model according to an exemplary embodiment; Figure 4 yes Figure 2 A flowchart of step 330 in one embodiment corresponds to the following example; Figure 5 yes Figure 2 A flowchart of the training machine learning model involved in the corresponding embodiment; Figure 6 yes Figure 5 A flowchart for calculating the target loss value as described in the corresponding embodiment; Figure 7yes Figure 2 The flowchart of the automatic binding of smart devices involved in the corresponding embodiment; Figure 8 yes Figure 2 The flowchart of the corresponding embodiment involving binding a smart device through a trigger operation; Figure 9 This is a schematic diagram illustrating an interactive interface according to an exemplary embodiment; Figure 10 yes Figure 2 A flowchart of the generation control instructions involved in the corresponding embodiment; Figures 11 to 12 This is a schematic diagram illustrating the specific implementation of a device control method in an application scenario; Figure 12a yes Figure 12 A schematic diagram of the interactive interface involved in the corresponding application scenario; Figure 12b yes Figure 12 A schematic diagram of the control component icons involved in the corresponding application scenarios; Figure 12c yes Figure 12 A schematic diagram of the personalized configuration control components involved in the corresponding application scenarios; Figure 13 This is a structural block diagram of a device control apparatus according to an exemplary embodiment; Figure 14 This is a hardware structure diagram of a server according to an exemplary embodiment; Figure 15 This is a hardware structure diagram of a terminal according to an exemplary embodiment; Figure 16 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0021] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0022] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0023] As mentioned earlier, the target detection model outputs a large amount of data, which generates a lot of computation.

[0024] A typical object detection model outputs detection results including m*(n+1+4), where m represents the number of candidate boxes, usually several thousand to tens of thousands, n represents the number of classes, 1 represents the background class, and 4 represents the positional deviation of the generated boxes.

[0025] Since prediction is required for each candidate box, it is necessary not only to determine whether the target object is contained, but also to calculate the localization difference between the candidate box containing the target object and the detection box. However, a large number of candidate boxes (m) will eventually result in only one detection box, which means that redundant candidate boxes need to be removed.

[0026] Currently, Non-Maximum Suppression (NMS) is generally used to remove redundant candidate boxes. However, NMS is only a post-processing operation on the detection results and does not affect the model structure. Using NMS cannot change the fact that object detection models are complex and post-processing is complex. Furthermore, object detection models have slow training and convergence speeds, resulting in poor speed and accuracy performance in object detection.

[0027] As can be seen from the above, there are still problems such as slow target detection speed and low accuracy.

[0028] Therefore, the device control method provided in this application can effectively improve the speed and accuracy of target detection, thereby reducing the delay of device control. Accordingly, the device control method is applicable to device control devices, which can be deployed on electronic devices configured with a von Neumann architecture, such as desktop computers, laptops, tablets, servers, etc.

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0030] Figure 1 This is a schematic diagram of an implementation environment involved in a device control method. The implementation environment includes a user terminal 110, a smart device 130, a gateway 150, a server 170, and a router 190.

[0031] Specifically, user terminal 110, which can also be considered as user terminal or terminal, can deploy (or install) the client associated with smart device 130. This user terminal 110 can be an electronic device such as a smartphone, tablet, laptop, desktop computer, smart control panel, or other device with display and control functions, without limitation here.

[0032] The client, associated with the smart device 130, is essentially where the user registers an account and configures the smart device 130. For example, the configuration includes adding a device identifier to the smart device 130, so that when the client runs on the user terminal 110, it can provide the user with functions such as device display and device control of the smart device 130. This client can be in the form of an application or a web page. Correspondingly, the interface for displaying the device on the client can be in the form of a program window or a web page, and there is no limitation here.

[0033] Smart device 130 is deployed in gateway 150 and communicates with gateway 150 through its own configured communication module, thereby being controlled by gateway 150. It should be understood that smart device 130 generally refers to one of multiple smart devices 130. This application embodiment only uses smart device 130 as an example; that is, this application embodiment does not limit the number or type of smart devices deployed in gateway 150. In one application scenario, smart device 130 is deployed in gateway 150 by accessing it through a local area network. The process of smart device 130 accessing gateway 150 through a local area network includes: gateway 150 first establishes a local area network, and smart device 130 joins the local area network established by gateway 150 by connecting to it. This local area network includes, but is not limited to, ZIGBEE or Bluetooth. Among them, the smart device 130 can be a smart printer, smart fax machine, smart camera, smart air conditioner, smart door lock, smart light, or electronic devices such as human body sensor, door and window sensor, temperature and humidity sensor, water immersion sensor, natural gas alarm, smoke alarm, wall switch, wall socket, wireless switch, wireless wall sticker switch, cube controller, curtain motor, etc., equipped with a communication module.

[0034] The interaction between user terminal 110 and smart device 130 can be achieved through a local area network (LAN) or a wide area network (WAN). In one application scenario, user terminal 110 establishes a wired or wireless communication connection with gateway 150 via router 190, such as Wi-Fi, allowing user terminal 110 and gateway 150 to be deployed on the same LAN, thus enabling user terminal 110 to interact with smart device 130 via the LAN path. In another application scenario, user terminal 110 establishes a wired or wireless communication connection with gateway 150 via server 170, such as 2G, 3G, 4G, 5G, or Wi-Fi, allowing user terminal 110 and gateway 150 to be deployed on the same WAN, thus enabling user terminal 110 to interact with smart device 130 via the WAN path.

[0035] The server-side 170 can also be considered as the cloud, cloud platform, platform side, server side, etc. This server-side 170 can be a single server, a server cluster consisting of multiple servers, or a cloud computing center consisting of multiple servers, in order to better provide backend services to a massive number of user terminals 110. For example, backend services include target detection services.

[0036] In one application scenario, the server 170 acquires the image to be detected in the target scene, performs target detection on the target object in the image to be detected using a target detection model, obtains the detection result of the target object, and then sends the detection result to the user terminal 110 so that the user can bind the target object to the smart device 130 in the target scene at the interactive interface provided by the user terminal 110 based on the detection result, thereby completing the binding of the target object and the smart device.

[0037] The user uses the user terminal 110 to view the interactive interface and control the smart device 130 in the interactive interface. The user terminal 110 sends the control command triggered in the interactive interface to the router 190. The router 190 sends the control command to the gateway 150. The gateway 150 then sends the control command to the smart device 130, so that the smart device 130 responds to the control command and performs an action.

[0038] In another application scenario, the user terminal 110 acquires the image to be detected in the target scene, performs target detection on the target object in the image to be detected through the target detection model, obtains the detection result of the target object, and displays the detection result to the user in the interactive interface so that the user can bind the target object to the smart device 130 in the target scene, thereby enabling the user to control the smart device 130 through the interactive interface.

[0039] Please see Figure 2 This application provides a device control method, which is applicable to electronic devices, specifically electronic devices that can be... Figure 1 The server-side 170 in the implementation environment shown can also be... Figure 1 User terminal 110 in the implementation environment shown.

[0040] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0041] like Figure 2 As shown, the method may include the following steps: Step 310: Obtain the image to be detected of the target scene.

[0042] The image to be detected of the target scene is obtained by capturing and acquiring the target scene using an image acquisition device. This image acquisition device can be a smart device with image acquisition capabilities in the Internet of Things (IoT), such as a camera or a smartphone with a camera. The image acquisition device can be deployed around the target scene, which can refer to an area within a smart home, such as the living room, bedroom, or kitchen; an area within a building, such as a conference room or elevator; or an area outside a building, such as a road. Correspondingly, the image acquisition device can be installed anywhere within the target scene to facilitate capturing the scene; for example, if the image acquisition device is a camera, it can be installed on the living room ceiling.

[0043] It is understood that shooting can be a single shot or continuous shooting. For the same target scene, continuous shooting results in a video, and the image can be any frame from that video. Multiple shots result in multiple photos, and the image can be any one of those photos. In other words, the image in this embodiment can come from moving images, such as multiple frames from a video or multiple photos, or from static images, such as any frame from a video or any one of multiple photos. Accordingly, target tracking in this embodiment can be performed frame by frame.

[0044] Regarding image acquisition, images can originate from real-time captures by an image acquisition device, or from images captured by the same device within a historical time period and pre-stored in the electronic device. Therefore, the electronic device can process images in real-time after capture, or it can pre-store them for later processing. For example, it can process images when the CPU is low, or it can process images according to instructions from staff. Thus, target tracking in this embodiment can target images acquired in real-time or images acquired within a historical time period; no specific limitation is made here.

[0045] Furthermore, the target scene includes several smart devices, and correspondingly, the image to be detected in the target scene also includes several smart devices.

[0046] Step 330: Using the trained target detection model, target detection is performed on the target object in the image to be detected, and the detection result of the target object is obtained.

[0047] As mentioned earlier, the target scene includes several smart devices. Therefore, using a target detection model to detect target objects in the image to be detected is essentially performing target detection on each smart device in the target scene. This target detection process includes determining the category of the target object and its location within the target scene to obtain the detection result. This detection result is used to indicate the location and category of each smart device in the target scene. Specifically, the target object can be used to indicate the smart devices in the target scene.

[0048] The object detection model is obtained by modifying its parameters and training it. A trained object detection model has the ability to detect target objects in an image. Specifically, the model parameters refer to the candidate bounding box parameters of the object detection model. By modifying these parameters, the model can remove candidate bounding boxes and output only a single detection bounding box when detecting a target object in the image, thus obtaining the detection result for that target object.

[0049] In one possible implementation Figure 3 The flowchart of the target detection model in this scheme is shown below, such as... Figure 3As shown, the target detection model in this scheme outputs predictions with two branches: the predicted category Cls: conv(m, n) and the predicted position Reg: conv(m, 4*n), with no candidate boxes n_anchor. It can be seen that this scheme modifies the parameters of the last layer's output to change the prediction results, thereby reducing the amount of data needed to train the machine learning model, lowering the training difficulty, and improving the accuracy of the machine learning model, ultimately enhancing the accuracy of target detection.

[0050] Regarding the detection of target objects in the image to be detected, one possible implementation is as follows: Figure 4 As shown, it includes the following steps: Step 331: Input the image to be detected into the target detection model for feature extraction to obtain the image features of the image to be detected.

[0051] Image features are an accurate description of the image to be detected. It should be understood that different images to be detected will have different image features, so that the image features can uniquely represent the image to be detected.

[0052] Of course, in other embodiments, image features can also be implemented by feature extraction algorithms, including but not limited to: histogram of oriented gradients feature extraction algorithm, LBP (Local Binary Pattern) feature extraction algorithm, Haar feature (Haar-like features) extraction algorithm, etc., which are not intended to constitute a specific limitation.

[0053] Step 333: The target detection model outputs a detection box for each category of target object. Based on the image features of the image to be detected, the position of each target object in the image to be detected, the predicted category of each target object, and the prediction score of each target object are obtained directly.

[0054] As mentioned earlier, when detecting target objects in an image, the object detection model outputs only one detection box. Furthermore, for different categories of target objects in the image, the object detection model outputs a separate detection box for each category. For example, if the image contains target objects categorized as lighting equipment and refrigeration equipment, the object detection model will output one detection box for the lighting equipment and another for the refrigeration equipment.

[0055] The category of the target object can be the equipment function type, such as lighting equipment, cooling equipment, air purification equipment, display equipment, etc.; or it can be the equipment model type, such as energy-saving lamps, table lamps, air conditioners, fans, fresh air systems, etc., without limitation.

[0056] It should be noted that the detection box is a rectangle. The detection box can be used to select target objects in the image to be detected. On one hand, by predicting the coordinates of the box in the image, the position of the selected target object can be obtained; on the other hand, based on the image features within the detection box, the predicted category and prediction score of the selected target object can be obtained. Of course, in other embodiments, the detection box can also be a circle, a polygon, etc. The shape and size of the detection box can be freely chosen to describe the detected target object with maximum accuracy; no limitation is imposed here.

[0057] The prediction score is calculated based on the image features within the detection box, determining the probability that the target object within the box belongs to different categories. In essence, target detection analyzes the probability that the target object belongs to which category; that is, during target detection, the probability of the target object belonging to different categories is obtained, and the category of the target object is determined based on the probability of belonging to each category.

[0058] Assuming the target object's category includes at least: energy-saving lamp, table lamp, and crystal lamp, then by predicting the target object's category, we can calculate the prediction scores P1, P2, and P3 for the target object belonging to the categories of energy-saving lamp, table lamp, and crystal lamp, respectively.

[0059] If P1 is the largest, the predicted category of the target object is identified as an energy-saving lamp. Similarly, if P2 is the largest, the predicted category of the target object is identified as a table lamp. If P3 is the largest, the predicted category of the target object is identified as a crystal lamp.

[0060] Step 335: If the prediction score of the target object meets the set conditions, the detection result of the target object is generated based on the position of the target object in the image to be detected and the prediction category of the target object.

[0061] First, it should be noted that the predicted category of a target object can be determined not only by directly determining the highest probability, but also after certain recognition conditions are met. These conditions can be score thresholds; when the predicted score of the target object is greater than the threshold, the predicted category of the target object is identified. For example, if the score threshold is 0.9, then the predicted category of the target object can only be determined if P1 / P2 / P3 are greater than 0.9.

[0062] Furthermore, the location of the target object in the image to be detected is determined, and the predicted category of the target object is determined. Then, based on the predicted category and location, the detection result of the target object is obtained.

[0063] Through the above process, the image features of the image to be detected are obtained by the object detection model, and a detection box is output based on the image features. This scheme does not require outputting candidate boxes for each target object. Based on the output detection box, the detection results of each target object can be obtained directly, which reduces the amount of computation of the object detection model when performing target detection and can greatly improve the speed of the object detection model in detecting target objects.

[0064] Step 350: In the interactive interface used to display the various smart devices in the target scene, the target object is bound to the smart devices in the target scene based on the detection results of the target object.

[0065] As mentioned earlier, the target scene includes several smart devices, and the detection results of the target object can indicate the location and category of each smart device. Therefore, based on the detection results, it is possible to bind the target object to the smart devices in the target scene.

[0066] The user interface serves as a medium for users to control various smart devices on their terminals. One user interface can correspond to one scene or multiple scenes; this is not limited here. The user interface displays the target scene and its various smart devices. After completing the above binding process, users can control the smart devices in the target scene through the user interface.

[0067] Step 370: If a control command triggered in the interactive interface is detected, the corresponding bound smart device is controlled to perform the action corresponding to the control command.

[0068] It should be noted that if a user wishes to control any smart device in the target scene, they will perform a corresponding operation on the interactive interface. Based on the operation, a control command triggered in the interactive interface will be detected, which is used to instruct the smart device to perform an action.

[0069] Through the above process, the modified target detection model performs target detection on the image to be detected. Based on the detection results, the target object is bound to a smart device in the target scene, allowing the user to control the smart device through an interactive interface. Using the modified target detection model significantly reduces the amount of data computation during target detection, enabling rapid identification of target objects in the image with high accuracy. Therefore, on the one hand, rapid target detection enhances the real-time performance of device control; on the other hand, accurate detection results improve the precision of device control, thus solving the problem of poor device control performance caused by slow target detection speed and low accuracy in related technologies.

[0070] Please see Figure 5 This application provides a possible implementation method. After step 370, the method may further include the following steps: Step 410: Obtain training images of the target scene.

[0071] Step 430: Input the training image into the machine learning model with modified model parameters for object detection to obtain the prediction results of each target object in the training image.

[0072] Step 450: Calculate the target loss value based on the prediction results of each target object in the training image and the label carried by the training image, and use the target loss value to train the machine learning model until the target detection model that has been trained is obtained.

[0073] Training images refer to labeled images. Training images can be obtained by web crawling of images of the target scene from the internet, or by capturing images of the target scene using image acquisition devices; no specific limitation is made here.

[0074] The labels are used to indicate the actual category of each target object in the training image and the actual location of each target object in the training image; the prediction results can include the prediction score of the target object and the predicted location of the target object in the training image.

[0075] It should be noted that labels can be obtained by manually annotating the training images. For example, when the training image contains a target object of the i-th category, the label of the target object of that category is z=[i,gt_x,gt_y,w,h]. z is added to y (y.append(z)), where y represents the label of the target object, i represents the category of the target object, (gt_x, gt_y) represents the coordinates of the center point of the target object, and (w,h) represents the length and width of the bounding rectangle of the object.

[0076] It should be noted that, since training images vary in size, larger images consume more resources. Therefore, before training the machine learning model, the training images can be scaled to a fixed size to reduce resource consumption and speed up data processing. For example, the training image can be scaled to 416x416 and then normalized to obtain the processed training image image1. Furthermore, a new label y1 is calculated based on the scaling ratio, and the processed training image image1 and its new label y1 are used for subsequent training of the machine learning model.

[0077] It is understandable that there may be discrepancies between the predicted categories and actual categories of target objects obtained by machine learning models, as well as between the predicted locations and actual locations. The purpose of training the machine learning model using training images is to reduce these discrepancies. When the discrepancy is reduced to a certain range, the machine learning model can be considered to have completed training, thus yielding an object detection model. For example, when the discrepancy is less than 0.5%, the machine learning model has completed training.

[0078] Furthermore, by utilizing the aforementioned bias, a target loss value can be calculated, which can then guide the next step of training the machine learning model in the correct direction.

[0079] In one possible implementation, such as Figure 6 As shown, the calculation of the target loss function includes the following steps: Step 451: For each target object in the training image, calculate the category loss value of the target object based on the predicted score of the target object and the actual category of the target object.

[0080] The prediction results also include the predicted category of the target object.

[0081] First, it should be noted that machine learning models perform object detection on training images and can output prediction results for each target object in the training images.

[0082] In one possible implementation, the prediction results for each target object are represented by a matrix. The i-th row of this matrix includes the prediction score for the i-th target object, as well as the center point position and width and height (i_pred_x, i_pred_y, i_pred_w, i_pred_h) of the detection box.

[0083] In another possible implementation, the detection bounding box (i_pred_x, i_pred_y, i_pred_w, i_pred_h) can represent the difference between itself and the fixed bounding box. Specifically, the center point of the fixed bounding box is at the center of the image to be detected (w / 2, h / 2), and its length and width are half of the image to be detected (w / 2, h / 2). (i_pred_x, i_pred_y) represents the difference between the center point of the target object and the center point (w / 2, h / 2) of the image to be detected. The center point of the target object is (w / 2 + i_pred_x, h / 2 + i_pred_y). (i_pred_w, i_pred_h) represents the difference between the length and width of the target object and (w / 2, h / 2). The length and width of the target object are (w / 2 + i_pred_w, h / 2 + i_pred_h).

[0084] Furthermore, since there may be a deviation between the predicted category and the actual category of each target object, the category loss value of the target object can be calculated based on the deviation between the predicted category and the actual category.

[0085] In one possible implementation, the category loss value of the target object is calculated based on the predicted score of the target object and the actual category of the target object, including: determining the expected value of the predicted score based on whether the predicted category of the target object exists in the actual category of each target object; and calculating the category loss value of the target object based on the expected value of the predicted score and the actual category of the target object.

[0086] First, it should be noted that when the prediction results for a target object have a large deviation, the predicted categories in the prediction results may not exist for the actual categories of each target object. In other words, the actual categories and predicted categories of all target objects in the machine learning model are different. Understandably, if this occurs, it indicates that the machine learning model has poor object detection capabilities. Based on this, different expected values ​​are assigned to the prediction result scores depending on whether the predicted category of the target object exists in the actual categories of each target object. For example, if the predicted category of the target object exists in the actual categories of each target object, the expected value of the prediction result score for that target object is 1; if it does not exist, the expected value is 0.

[0087] Furthermore, based on the expected value of the prediction score corresponding to each target object, the category loss value corresponding to each target object is calculated. In one possible implementation, the category loss value can be obtained by calculating the binary cross-entropy loss.

[0088] For example, if the predicted category of the target object is class i, the labels of each target object are read. If a label for class i exists, the expected value score_i of the target object is 1. The binary cross-entropy loss loss1 is calculated based on score_i and the label for class i. If no label exists, the expected value score_i of the target object is 0, and the binary cross-entropy loss loss1 is calculated.

[0089] Step 453: Calculate the position loss value of the target object based on the predicted position of the target object in the training image and the actual position of the target object in the training image.

[0090] Since there may be a deviation between the predicted position and the actual position of each target object, the position loss value of the target object can be calculated based on the deviation between the predicted position and the actual position.

[0091] In one possible implementation, the calculation of the location loss value includes: calculating a first loss value and a second loss value based on the predicted location of the target object in the training image and the actual location of the target object in the training image. The location loss value of the target object is then calculated from the first loss value and the second loss value.

[0092] The first loss value reflects the difference between the predicted and actual locations, while the second loss value reflects the degree of overlap between the predicted and actual locations.

[0093] In one possible implementation, the first loss value for the target object can be L1 loss, and the second loss value can be IOU loss. The predicted position of the target object can be (i_pred_x, i_pred_y, i_pred_w, i_pred_h), and the actual position is (gt_x, gt_y, w, h). The L1 loss and IOU loss are calculated based on the predicted position and the actual position. For example, the difference loss between the predicted position and the actual position can be calculated to obtain the L1 loss and IOU loss.

[0094] Step 455: Obtain the loss value of the target object based on the category loss value and location loss value of the target object.

[0095] Step 457: Calculate the target loss value from the loss values ​​of each target object.

[0096] For example, the category loss value of the target object is loss1, and the location loss values ​​are loss2 and loss3. loss_i = a1*loss1 + a2*loss2 + a3*loss3, where loss_i is the loss value of the i-th category of target object, and a1, a2 and a3 are weight parameters. The target loss value is obtained by adding the loss values ​​of the target objects of each category.

[0097] Regarding training machine learning models using the target loss value, one possible implementation is to use backpropagation based on the target loss value.

[0098] Under the above embodiments, the target loss value of each target object is obtained through the category loss value and position loss value of each target object. The machine learning model is trained using the target loss value to obtain a trained target detection model. This reduces the amount of data required for the machine learning model to learn, reduces the training difficulty, and greatly improves the convergence speed of the machine learning model. Furthermore, the target detection model trained using the target loss value not only has a fast target detection speed but also high accuracy, thereby enhancing the effect of equipment control and solving the problem of poor equipment control effect caused by slow target detection speed and low accuracy in related technologies.

[0099] Please see Figure 7 This application provides a possible implementation method. After step 370, the method may further include the following steps: Step 510: Display the interactive interface and find the device location corresponding to the predicted location of the target object in the interactive interface.

[0100] The interactive interface is used to control the various smart devices in the target scene. The interface displays a screen corresponding to the target scene, such as an image of the target scene showing the smart devices. This image can be uploaded by the user, captured by an image acquisition device of the actual target scene, or a two-dimensional or three-dimensional image obtained by processing the actual image of the target scene. For example, it can produce a virtual reality scene similar to the real-world target scene, or it can be the image to be detected mentioned above; no specific limitation is made here.

[0101] It should be noted that the detection result of the target object includes the predicted category and predicted location of the target object. Therefore, based on the predicted location in the detection result, the position of the target object on the interactive interface can be found. Furthermore, the device location of the corresponding smart device can be found, so as to bind the target object and the smart device. Here, device location refers to the position of the smart device within the interactive interface.

[0102] Step 530: At the found device location, add corresponding control components to the interactive interface for the smart devices in the target scene according to the predicted category of the target object, so that the target object and the smart devices in the target scene are bound together.

[0103] First, it should be noted that in the interactive interface, users can control corresponding smart devices using various control components. That is, beforehand, corresponding control components need to be added to the device location on the interactive page for each smart device. This can be done by finding the control component corresponding to the prediction category in the prediction results and adding that control component to the device location on the interactive interface, so that users can control the corresponding smart device using that control component.

[0104] Among them, the control component is the component used to control the smart device. Depending on the category, the control component corresponding to each smart device may be different.

[0105] In one possible implementation, as shown in Table 1, the relationship between each category and the control component is illustrated, and the component style and size of each control component are also shown. The component size refers to the size of the control component displayed in the interactive interface.

[0106] Table 1

[0107] For example, Table 1 provides control components categorized as curtains, ceiling lights, and table lamps. If the predicted category of the target object is ceiling light, then the control component categorized as ceiling light is selected, and the control component is added to the device location on the interactive interface. This establishes a binding relationship between the target object and the ceiling light, allowing the user to control the ceiling light through the control component.

[0108] Through the above process, based on the detection results of the target object, the corresponding device location is viewed in the interactive interface. A corresponding control component is then added at that location, completing the binding between the target object and the smart device. This allows users to control the corresponding smart device using the control components in the interactive interface, thereby improving the user's smart home experience. Furthermore, automatically locating the device in the interactive interface based on the target object's detection results reduces user operations and enhances the overall intelligence.

[0109] It should be noted that when there are multiple smart devices of the same category in the target scene, the target detection model only outputs one detection box for each target object of the same category. In other words, if there is more than one target object of the same category in the target scene, the target detection model will only output one detection result, which corresponds to only one smart device. Furthermore, based on this detection result, only one target object of the same category in the target scene can be bound to one smart device. Therefore, the remaining smart devices of the same category cannot be bound to the corresponding target object.

[0110] To solve the above problems, such as Figure 8As shown, binding multiple target objects of the same category includes the following steps: Step 610: Determine the number of target smart devices in the target scene that match the predicted category of the target object.

[0111] It's understandable that if there's more than one target object of the same category, there will also be more than one smart device of that category. Therefore, to avoid omissions during binding, it's necessary to first confirm the number of target smart devices in the target scene that match the predicted type of the target object. Here, a target smart device refers to a smart device in the target scene that matches the predicted type.

[0112] Step 630: If the number of target smart devices exceeds a set threshold, then detect trigger operations related to the target smart devices in the interactive interface.

[0113] As mentioned above, when the number of target smart devices exceeds one, binding the target object and target smart device solely based on the prediction result may result in missed bindings. Therefore, to prevent this, when the number of target smart devices exceeds a set threshold, a multi-object detection model can be used to detect targets in the target scene to obtain prediction results for each target object in the same category, and then binding the target object and target smart device can be performed based on each prediction result. Alternatively, a trigger operation for the target smart device can be provided in the interactive interface. This trigger operation is used to bind the target object and target smart device. If the user wishes to bind the target object and target smart device, they can achieve the binding through this trigger operation. It should be noted that the above solutions are only illustrative examples and do not constitute a limitation of the present invention.

[0114] The threshold value is related to the detection capability of the target detection model. If the target detection model has a strong ability to detect different target objects of the same category, the threshold can be set to be larger. If the target detection model has a weak ability to detect different target objects of the same category, the threshold can be set to be smaller. In one possible implementation, the threshold is set to "the number of target smart devices is 1". If the number of target smart devices is not 1, the threshold is exceeded.

[0115] Step 650: If a trigger operation related to the target smart device is detected, add the corresponding control component to the interactive interface for the target smart device according to the predicted category of the target object, so that the target object and the target smart device are bound together.

[0116] As mentioned earlier, the control components for each smart device can be different depending on the category. Therefore, if a trigger operation is detected for the target smart device, the corresponding control component is added to the interactive interface for the target smart device according to the predicted category of the target object.

[0117] For example, Figure 9 A schematic diagram of the interactive interface in one embodiment is shown, such as... Figure 9 As shown, the interactive interface displays the control component 901 for the smart curtains and the control component 903 for the smart ceiling light. Since the number of smart table lamps exceeds the set threshold, the interactive interface also displays the control components 905 and 907 for the smart table lamps.

[0118] Furthermore, if a user wishes to control a corresponding smart device using the various control components in the interactive interface, control can be achieved by triggering the control components. One possible implementation is as follows: Figure 10 As shown, controlling a smart device using a control component includes the following steps: Step 710: In response to a trigger operation on a control component in the interactive interface, generate a control command.

[0119] Step 730: Send the control command to the smart device corresponding to the control component, so that the smart device responds to the control command and performs the corresponding action.

[0120] Among them, the control command is the instruction used to instruct the smart device to perform an action. It can be understood that if the user expects the smart device to perform an action, the user will use the control component in the interactive interface corresponding to the smart device to control it. This will cause the electronic device to detect the trigger operation of the control component in the interactive interface, generate the corresponding control command, and send the control command to the smart device. After receiving the control command, the smart device will respond to the control command and perform the action expected by the user.

[0121] It is worth mentioning that the specific form of triggering operations for control components in the interactive interface can vary depending on the input components (mouse, touchscreen, keyboard) configured on the electronic device. For example, if the electronic device is a smartphone with a touchscreen, the triggering operation can be a gesture operation such as clicking or moving; if the electronic device is a desktop computer with a mouse, the triggering operation can be a mechanical operation such as clicking, double-clicking, or dragging. There is no limitation here.

[0122] Additionally, the control components corresponding to the smart device in the interactive interface can be updated according to the device status of the smart device. For example, when the smart device performs an action, the component style of the corresponding control component will change to facilitate users to view the real-time device status of the smart device in the interactive interface. In one possible implementation, the component properties of the corresponding control component are updated based on the device status of the smart device after the action is performed, and the control component is then updated and displayed in the interactive interface. Here, the device status indicates the state of the smart device, and the component properties indicate the state of the control component.

[0123] Furthermore, the user terminal provides control components for various smart devices. Each control component includes component attributes and component interactions. Component attributes include attributes such as component style. Component interactions refer to the interaction between the user's actions, the device status, and the component attributes when the user uses the control components to control the corresponding smart device in the interactive interface.

[0124] For example, users can control the on / off state of the ceiling light, as well as its color temperature, brightness, and other device statuses, through the "Ceiling Light" control component in the interactive interface. When the ceiling light is on, the control component's style is "On," and its color matches the ceiling light's color temperature. When the ceiling light is off, the control component's style is "Off." If a user wants to switch the ceiling light's on / off state, they can do so by clicking the control component. If a user wants to adjust the ceiling light's color temperature, they can long-press the control component to bring up the color temperature adjustment module, which allows them to adjust the ceiling light's color temperature.

[0125] Under the above embodiments, on the one hand, when the number of target smart devices exceeds a set threshold, each target object is bound to multiple corresponding target smart devices by triggering the target smart devices to prevent omissions; on the other hand, by viewing and controlling each smart device through the control components in the interactive interface, users can more directly and conveniently understand each smart device in the smart home network and control each smart device to perform actions; furthermore, based on the user's operation in the interactive interface, the interactive interface updates the corresponding control components in real time to realize the function of real-scene interaction and improve the user's smart home experience.

[0126] Figure 11 This is a schematic diagram illustrating a specific implementation of a device control method in an application scenario. This application scenario is a smart home scenario, suitable for... Figure 1 The implementation environment is shown.

[0127] like Figure 11 As shown, this application scenario includes the following steps: In step 810, the image to be detected is obtained by the camera capturing the target scene.

[0128] The camera can be set up in the target scene to capture the scene and obtain the image to be detected. For example, if the target scene is a living room, the camera can be set up on the wall of the living room and capture the scene to obtain the image to be detected.

[0129] In step 830, the image to be detected is input into the target detection model to obtain the detection result of the target object in the image.

[0130] It is understandable that a smart home scenario involves several smart devices. Users can control these devices through a user terminal. Furthermore, by performing target detection on the target object (smart device) in the image to be detected in the target scene, the detection results of each smart device are obtained. Based on the detection results, corresponding control components are added to the interactive interface in the user terminal, enabling users to control the smart devices using these control components.

[0131] Specifically, such as Figure 12 As shown, the target detection model detects target objects in the image to be detected, including the following steps: In step 831, the image to be detected is received.

[0132] In step 833, the image features of the image to be detected are extracted using the feature extractor in the object detection model.

[0133] Step 835 yields the predicted category of the target object.

[0134] Step 837 yields the predicted location of the target object.

[0135] In step 850, based on the detection results, the device location corresponding to the predicted location of the target object is found in the interactive interface, and a control component corresponding to the prediction type of the target object is added to the device location. The control component is then bound to the target object in the image to be detected.

[0136] For example, Figure 12a The diagram shows a schematic of the interactive interface in one embodiment, such as... Figure 12aAs shown, the interactive interface 12a1 includes a scene area 12a2, a control area 12a3, and a status area 12a4. The scene area 12a2 displays a target scene image, on which multiple control components are displayed. The control area 12a3 includes smart devices and control components corresponding to scenes under the smart home network. Each control component displays the name of its corresponding smart device or scene. Users can select and trigger the control component by viewing its name to control the smart device or scene corresponding to that control component. The status area 12a4 is used to display status information such as the environmental status and device status of the target scene.

[0137] The target detection model outputs the detection result of the target object, finds the device position corresponding to the predicted position of the target object in the target scene image within the scene area 12a2, and automatically adds a control component corresponding to the predicted category of the target object at the device position, thereby displaying the control component in the scene area 12a2 of the interactive interface.

[0138] In one possible implementation, the object detection model outputs a detection result for target object A after detecting the image to be detected. This detection result indicates that the predicted category of target object A is smart curtain, and the predicted location is (X, Y). Based on this detection result, the user terminal searches for the device location 12a5 corresponding to the predicted location of the target object in the target scene image within scene region 12a2, such as... Figure 12a As shown in scene area 12a2, a control component for the smart curtain has been added at device location 12a5.

[0139] Of course, users can also personalize the style of the control components through the user terminal so that the control components displayed on the interactive page are more in line with the user's preferences; Figure 12b A schematic diagram of control component icons is shown in one embodiment, such as... Figure 12b As shown, several categories of control component icons are displayed, including switch control components, sensor control components, alarm control components, etc. Users can access the corresponding personalization configuration page for each control component by triggering its personalization configuration entry point, thereby customizing the control component. To facilitate user configuration of control components, the personalization configuration entry point for each control component can be set in [the relevant location]. Figure 12a The control components of the inner scene area 12a2 and / or the control components of the control area 12a3 can also be set to... Figure 12b The control component icons shown are not limited here.

[0140] In one possible implementation, the user clicks the personalization configuration entry point of the switch control component to enter the personalization configuration page of the switch component. Figure 12cA schematic diagram of a personalized configuration control component in one embodiment is shown, such as... Figure 12c As shown, you can configure basic attributes such as component size, component position, transparency, and flip angle for the switch control component. You can also configure canvas attributes such as background color, rounded corners, border width, border color, border type, and border for the switch control component. Furthermore, you can configure control component icons for the on and off states.

[0141] In step 870, the user controls the smart device using the control components in the interactive interface, causing the smart device to perform corresponding actions.

[0142] Still with Figure 12a For example, users can click on control component 12a5 in scene area 12a2 to control the smart curtains, causing them to perform corresponding actions; they can also click on control components in control area 12a3 to control corresponding smart devices and / or scenes, causing them to perform corresponding actions. In this application scenario, using a modified target detection model to perform target detection on the image to be detected, outputting only the predicted category and predicted location of the target object, can greatly reduce the amount of data computation during target detection, achieving rapid identification of target objects in the image to be detected with high accuracy, thus solving the problems of slow target detection speed and low accuracy in related technologies.

[0143] It should be understood that although the steps in the flowcharts corresponding to the embodiments of this application are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0144] The following are embodiments of the apparatus described in this application, which can be used to execute the device control method involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the device control method involved in this application.

[0145] Please see Figure 13 This application provides a device control device 900, including but not limited to: an image acquisition module 910, a target detection module 930, a device binding module 950, and a device control module 970.

[0146] The image acquisition module 910 is used to acquire the image to be detected of the target scene.

[0147] The object detection module 930 is used to perform object detection on the target object in the image to be detected using a trained object detection model, and obtain the detection result of the target object. The object detection model is obtained by modifying the parameters related to the candidate boxes in the machine learning model so that it no longer outputs multiple candidate boxes, but directly outputs a detection box for each category of target object. The machine learning model with modified parameters is trained using training images. The trained object detection model has the ability to detect target objects in the image to be detected.

[0148] The device binding module 950 is used to determine the number of smart devices in the target scene that match the predicted category of the target object based on the detection result of the target object in the interactive interface for displaying each smart device in the target scene. If the number is less than or equal to a set threshold, the target object is automatically bound to the smart devices in the target scene.

[0149] The device control module 970 is used to control the corresponding bound smart device to perform the action corresponding to the control command if a control command triggered in the interactive interface is detected.

[0150] In one embodiment, the device control device 900 further includes a model training module for acquiring training images of the target scene; the training images refer to images carrying labels; the training images are input into a machine learning model with modified model parameters for target detection to obtain prediction results for each target object in the training images; based on the prediction results for each target object in the training images and the labels carried by the training images, the target loss value is calculated, and the machine learning model is trained using the target loss value until a target detection model that has been trained is obtained.

[0151] In one embodiment, the labels are used to indicate the actual category and actual position of each target object in the training image; the prediction results include the prediction score of the target object and the predicted position of the target object in the training image. The model training module is further configured to, for each target object in the training image, calculate the category loss value of the target object based on its prediction score and actual category; calculate the position loss value of the target object based on its predicted position and actual position in the training image; obtain the target object's loss value based on its category loss value and position loss value; and calculate the target loss value from the loss values ​​of each target object.

[0152] In one embodiment, the prediction result also includes the predicted category of the target object. The model training module is further configured to determine the expected value of the prediction result score based on whether the predicted category of the target object exists in the actual category of each target object; and to calculate the category loss value of the target object based on the expected value of the prediction result score and the actual category of the target object.

[0153] In one embodiment, the prediction result further includes the predicted category of the target object. The model training module is also used to calculate a first loss value and a second loss value based on the predicted position of the target object in the training image and the actual position of the target object in the training image; the first loss value reflects the difference between the predicted position and the actual position; the second loss value reflects the degree of overlap between the predicted position and the actual position; and the position loss value of the target object is calculated from the first loss value and the second loss value.

[0154] In one embodiment, the model training module is further configured to calculate a first loss value and a second loss value based on the predicted position of the target object in the training image and the actual position of the target object in the training image; the first loss value reflects the difference between the predicted position and the actual position; the second loss value reflects the degree of overlap between the predicted position and the actual position; and the position loss value of the target object is calculated from the first loss value and the second loss value.

[0155] In one embodiment, the target detection module 930 is further configured to input the image to be detected into the target detection model for feature extraction to obtain the image features of the image to be detected; through the target detection model, a detection box is output for different categories of target objects, and the position of each target object in the image to be detected, the predicted category of each target object, and the prediction result score of each target object are directly obtained based on the image features of the image to be detected; if the prediction result score of the target object meets the set conditions, the detection result of the target object is generated from the position of the target object in the image to be detected and the predicted category of the target object.

[0156] In one embodiment, the detection result of the target object includes the predicted category and predicted location of the target object. The device binding module 950 is further configured to display an interactive interface and locate the device location corresponding to the predicted location of the target object within the interactive interface; at the located device location, determine the number of smart devices in the target scene that match the predicted category of the target object; if the number is less than or equal to a set threshold, add corresponding control components to the interactive interface for the smart devices in the target scene according to the predicted category of the target object, so that the target object and the smart devices in the target scene are automatically bound together.

[0157] In one embodiment, the device binding module 950 is further configured to detect trigger operations related to the target smart device in the interactive interface if the number exceeds a set threshold; the target smart device refers to a smart device that matches the predicted category of the target object; if a trigger operation related to the target smart device is detected, a corresponding control component is added to the interactive interface for the target smart device according to the predicted category of the target object, so that the target object and the target smart device establish a binding relationship through the trigger operation.

[0158] In one embodiment, the device control module 970 is further configured to generate a control command in response to a trigger operation on the control component in the interactive interface; and send the control command to the smart device corresponding to the control component, causing the smart device to execute the corresponding action in response to the control command. The device binding module 950 is further configured to update the component attributes of the corresponding control component according to the device status of the smart device after the action is executed, and update the display of the control component in the interactive interface.

[0159] It should be noted that the device control device provided in the above embodiments is only illustrated by the division of the above functional modules when performing target detection. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the device control device will be divided into different functional modules to complete all or part of the functions described above.

[0160] Furthermore, the device control apparatus and device control method embodiments provided in the above embodiments belong to the same concept, and the specific way in which each module performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0161] Figure 14 A schematic diagram of the structure of a server is shown according to an exemplary embodiment. This server is suitable for... Figure 1 The server-side configuration of the implementation environment is shown as 170.

[0162] It should be noted that this server is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Nor should this server be interpreted as requiring or depending on any specific feature. Figure 14 One or more components of the exemplary server 2000 shown.

[0163] The hardware architecture of Server 2000 can vary significantly due to differences in configuration or performance, such as... Figure 14 As shown, the server 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0164] Specifically, power supply 210 is used to provide operating voltage for the various hardware devices on server 2000.

[0165] Interface 230 includes at least one wired or wireless network interface for interacting with external devices. For example, to perform... Figure 1 The diagram illustrates the interaction between user terminal 110 and server 170 in the implementation environment.

[0166] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 14 As shown, this does not constitute a specific limitation.

[0167] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0168] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 200, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0169] Application 253 is a program instruction or code based on operating system 251 that performs at least one specific task, and may include at least one module ( Figure 14 (Not shown), each module can contain program instructions or code for server 2000. For example, the device control device can be considered as application 253 deployed on server 2000.

[0170] Data 255 can be photos, pictures, etc. stored on a disk, or images to be detected, etc., and is stored in memory 250.

[0171] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read program instructions or code stored in the memory 250, thereby performing operations and processing on the massive amount of data 255 stored in the memory 250. For example, an information recommendation method may be implemented by the central processing unit 270 reading a series of program instructions or code stored in the memory 250.

[0172] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.

[0173] Please see Figure 15 , Figure 15 This is a schematic diagram illustrating the structure of a terminal according to an exemplary embodiment. The terminal is suitable for… Figure 1 The user terminal 110 in the implementation environment is shown.

[0174] It should be noted that this terminal is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Furthermore, this terminal should not be interpreted as requiring or depending on any specific feature. Figure 15 One or more components of the exemplary terminal 1100 shown.

[0175] like Figure 15 As shown, terminal 1100 includes memory 101, memory controller 103, and one or more ( Figure 15 (Only one is shown) Processor 105, peripheral interface 107, radio frequency module 109, positioning module 111, camera module 113, audio module 115, touch screen 117, and button module 119. These components communicate with each other through one or more communication buses / signal lines 121.

[0176] The memory 101 can be used to store computer programs and modules, such as the computer programs and modules corresponding to the device control method and apparatus in the exemplary embodiments of this application. The processor 105 executes various functions and data processing by running the computer programs stored in the memory 101, thereby completing the device control method.

[0177] The memory 101, as a carrier for resource storage, can be random access memory, such as high-speed random access memory, non-volatile memory, such as one or more magnetic storage devices, flash memory, or other solid-state memory. The storage method can be temporary storage or permanent storage.

[0178] The peripheral interface 107 may include at least one wired or wireless network interface, at least one serial-to-parallel conversion interface, at least one input / output interface, and at least one USB interface, etc., for coupling various external input / output devices to the memory 101 and the processor 105 to realize communication with various external input / output devices.

[0179] The radio frequency module 109 is used to transmit and receive electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, thereby enabling communication with other devices through a communication network. The communication network includes cellular telephone networks, wireless local area networks, or metropolitan area networks, and these communication networks can use various communication standards, protocols, and technologies.

[0180] The positioning module 111 is used to obtain the current geographical location of the terminal 1100. Examples of positioning modules 111 include, but are not limited to, Global Positioning System (GPS), positioning technologies based on wireless local area networks or mobile communication networks.

[0181] The camera module 113 is part of the camera and is used to capture pictures or videos. The captured pictures or videos can be stored in the memory 101 or transmitted to the host computer via the radio frequency module 109.

[0182] The audio module 115 provides an audio interface to the user, which may include one or more microphone jacks, one or more speaker jacks, and one or more headphone jacks. Audio data is exchanged with other devices through the audio interface. Audio data can be stored in the memory 101 and can also be transmitted via the radio frequency module 109.

[0183] The touchscreen 117 provides an input / output interface between the terminal 1100 and the user. Specifically, the user can perform input operations through the touchscreen 117, such as clicking, touching, and swiping gestures, so that the terminal 1100 can respond to the input operations. The terminal 1100 then displays the output content, which can be text, images, or videos in any form or combination thereof, to the user through the touchscreen 117.

[0184] The button module 119 includes at least one button, providing an interface for users to input information into the terminal 1100. Users can press different buttons to enable the terminal 1100 to perform different functions. For example, the volume adjustment button allows users to adjust the volume of the sound played by the terminal 1100.

[0185] Understandable. Figure 15 The structure shown is for illustrative purposes only; terminal 1100 may also include components that are more advanced than those shown. Figure 15 The more or fewer components shown, or having the same Figure 15 The different components are shown. Figure 15 The components shown can be implemented using hardware, software, or a combination thereof.

[0186] Please see Figure 16 This application provides an electronic device 4000, which may include a desktop computer, a laptop computer, and a server.

[0187] exist Figure 16 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0188] The data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 16 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0189] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0190] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0191] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 400, but not limited thereto.

[0192] The memory 4003 stores program instructions or code, and the processor 4001 can read the program instructions or code stored in the memory 4003 through the communication bus 4002.

[0193] When the program instructions or code are executed by the processor 4001, the device control methods in the above embodiments are implemented.

[0194] Furthermore, this application provides a storage medium storing program instructions or code, which are loaded and executed by a processor to implement the device control method as described above.

[0195] This application provides an application product, which includes program instructions or code stored in a storage medium. The processor of the electronic device reads the program instructions or code from the storage medium, loads and executes the program instructions or code, so that the electronic device implements the device control method described above.

[0196] Compared with related technologies, firstly, using a modified target detection model significantly reduces the amount of data computation during target detection, enabling rapid identification of target objects in the image and achieving high accuracy. Secondly, training the machine learning model using the target loss values ​​of each target object reduces the amount of data required for model learning, lowers training difficulty, and greatly improves the convergence speed of the machine learning model. Thirdly, rapid target detection enhances the real-time performance of device control, and accurate detection results improve the precision of device control, thereby enhancing the effectiveness of device control based on detection results. Fourthly, viewing and controlling various smart devices through the control components in the interactive interface allows users to more directly and conveniently understand the smart devices in the smart home network and control their actions. Furthermore, the interactive interface updates the corresponding control components in real time based on user actions, enabling real-scene interaction and enhancing the user's smart home experience.

[0197] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0198] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A device control method, characterized in that, The method includes: Acquire the image to be detected of the target scene; The target detection model is used to detect the target objects in the image to be detected, and the detection result of the target objects is obtained. The target detection model is obtained by modifying the parameters related to the candidate boxes in the machine learning model so that it no longer outputs multiple candidate boxes, but directly outputs a detection box for each category of target object. The machine learning model with modified parameters is trained using training images. The trained target detection model has the ability to detect target objects in the image to be detected. In the interactive interface used to display the smart devices in the target scene, the number of smart devices in the target scene that match the predicted category of the target object is determined according to the detection result of the target object. If the number is less than or equal to a set threshold, the target object is automatically bound to the smart devices in the target scene. If a control command triggered in the interactive interface is detected, the corresponding bound smart device is controlled to perform the action corresponding to the control command.

2. The method as described in claim 1, characterized in that, The training process of the target detection model includes: Acquire training images of the target scene; the training images refer to labeled images; The training images are input into a machine learning model with modified model parameters for target detection to obtain prediction results for each target object in the training images; Based on the prediction results of each target object in the training image and the labels carried by the training image, the target loss value is calculated, and the machine learning model is trained using the target loss value until the target detection model is obtained after training.

3. The method as described in claim 2, characterized in that, The labels are used to indicate the actual category of each target object in the training image and the actual position of each target object in the training image. The prediction result includes the prediction score of the target object and the predicted position of the target object in the training image; The step of calculating the target loss value based on the prediction results of each target object in the training image and the labels carried by the training image includes: For each target object in the training image, the category loss value of the target object is calculated based on the prediction score of the target object and the actual category of the target object; Calculate the position loss value of the target object based on the predicted position of the target object in the training image and the actual position of the target object in the training image; The loss value of the target object is obtained based on the category loss value and the location loss value of the target object; The target loss value is calculated from the loss value of each target object.

4. The method as described in claim 3, characterized in that, The prediction result also includes the predicted category of the target object; The step of calculating the category loss value of the target object based on the predicted score of the target object and the actual category of the target object includes: Based on whether the predicted category of the target object exists in the actual category of each target object, the expected value of the prediction result score is determined; The category loss value of the target object is calculated based on the expected value of the predicted score and the actual category of the target object.

5. The method as described in claim 3, characterized in that, The step of calculating the position loss value of the target object based on the predicted position of the target object in the training image and the actual position of the target object in the training image includes: Based on the predicted position of the target object in the training image and the actual position of the target object in the training image, a first loss value and a second loss value are calculated; the first loss value reflects the difference between the predicted position and the actual position; the second loss value reflects the degree of overlap between the predicted position and the actual position. The position loss value of the target object is calculated from the first loss value and the second loss value.

6. The method as described in claim 1, characterized in that, The step of using a trained target detection model to detect target objects in the image to be detected, and obtaining the detection result of the target object, includes: The image to be detected is input into the target detection model for feature extraction to obtain the image features of the image to be detected; The target detection model outputs a detection box for each category of target object, and the position of each target object in the image to be detected, the predicted category of each target object, and the prediction score of each target object are directly obtained based on the image features of the image to be detected. If the predicted score of the target object meets the set conditions, the detection result of the target object is generated based on the position of the target object in the image to be detected and the predicted category of the target object.

7. The method as described in claim 1, characterized in that, The detection result of the target object includes the predicted category and predicted location of the target object; In the interactive interface used to display various smart devices in the target scene, the number of smart devices in the target scene that match the predicted category of the target object is determined based on the detection result of the target object. If the number is less than or equal to a set threshold, the target object is automatically bound to the smart devices in the target scene, including: Display the interactive interface and search for the device location corresponding to the predicted location of the target object in the interactive interface; At the located device locations, determine the number of smart devices in the target scene that match the predicted category of the target object; If the number is less than or equal to a set threshold, then according to the predicted category of the target object, a corresponding control component is added to the interactive interface for the smart device in the target scene, so that the target object and the smart device in the target scene are automatically bound together.

8. The method as described in claim 1, characterized in that, After determining the number of smart devices in the target scene that match the predicted category of the target object based on the detection result of the target object in the interactive interface used to display each smart device in the target scene, the method further includes: If the number exceeds a set threshold, a trigger operation related to the target smart device is detected in the interactive interface; the target smart device refers to a smart device that matches the predicted category of the target object. If a trigger operation is detected concerning the target smart device, a corresponding control component is added to the interactive interface for the target smart device according to the predicted category of the target object, so that the target object and the target smart device establish a binding relationship through the trigger operation.

9. The method as described in claim 7 or 8, characterized in that, If a control command triggered in the interactive interface is detected, the corresponding bound smart device is controlled to perform the action corresponding to the control command, including: In response to a trigger operation on a control component in the interactive interface, a control command is generated; The control command is sent to the smart device corresponding to the control component, so that the smart device responds to the control command and performs the corresponding action. The method further includes: The component attributes of the corresponding control component are updated according to the device status of the smart device after the action is performed, and the control component is updated and displayed in the interactive interface.

10. A device control apparatus, characterized in that, The device includes: The image acquisition module is used to acquire the image to be detected of the target scene; The object detection module is used to perform object detection on the target object in the image to be detected using a trained object detection model, and obtain the detection result of the target object. The object detection model is obtained by modifying the parameters related to the candidate boxes in the machine learning model so that it no longer outputs multiple candidate boxes, but directly outputs a detection box for each category of target object. The machine learning model with modified parameters is trained using training images. The trained object detection model has the ability to detect target objects in the image to be detected. The device binding module is used to determine the number of smart devices in the target scene that match the predicted category of the target object based on the detection result of the target object in the interactive interface for displaying each smart device in the target scene. If the number is less than or equal to a set threshold, the target object is automatically bound to the smart devices in the target scene. The device control module is used to control the corresponding bound smart device to perform the action corresponding to the control command if a control command triggered in the interactive interface is detected.

11. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein, The memory stores program instructions or code; The program instructions or code are loaded and executed by the processor, causing the electronic device to implement the device control method as described in any one of claims 1 to 9.

12. A storage medium storing program instructions or code thereon, characterized in that, The program instructions or code are loaded and executed by the processor to implement the device control method as described in any one of claims 1 to 9.

Images