Device control method, device, electronic device and storage medium

Abstract

The present application relates to a device control method, apparatus, electronic device and storage medium, the method comprising: performing target recognition on an image collected under a target scene to obtain a target device in the image; obtaining a device under the space type corresponding to the target scene, and matching the target device with the device under the space type based on the device status of the target device; if there is a device matching the target device under the space type, then in the device control interface corresponding to the scene screen of the target scene, the target device is bound to the control component corresponding to the device matching under the space type; the control component is used to control the target device in the device control interface. In this way, since the binding process is based on the device status and the space type corresponding to the target scene, the degree of automation is higher, allowing users to bind devices and control components without or with only a small amount of operation, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of Internet of Things, and in particular to a device control method, apparatus, electronic device and storage medium. Background Art

[0002] With the development of Internet of Things technology, intelligent control of devices has been increasingly widely used in life. Taking smart home as an example, users can control smart devices through terminals to achieve all-round information interaction functions. In related technologies, by binding the devices identified in the target scene with the control components, the devices can be controlled more intuitively and conveniently. However, during the binding process, users are required to perform more operations, which affects the user experience. Summary of the invention

[0003] Based on this, it is necessary to provide a device control method, apparatus, electronic device and storage medium to address the above technical problems, which can enable users to bind devices and control components without or with only a small amount of operation, thereby improving user experience.

[0004] A device control method, the method comprising:

[0005] Performing target recognition on the image collected in the target scene to obtain the target device in the image;

[0006] Acquire a device of a space type corresponding to the target scene, and match the target device with the device of the space type based on a device state of the target device;

[0007] If there is a device matching the target device under the space type, then in the device control interface corresponding to the scene screen of the target scene, the target device is bound to the control component corresponding to the device matching the space type; the control component is used to control the target device in the device control interface.

[0008] A device control apparatus, comprising:

[0009] A target recognition module is used to perform target recognition on an image collected in a target scene to obtain a target device in the image;

[0010] A device matching module, used to obtain a device under the space type corresponding to the target scene, and match the target device with the device under the space type based on the device state of the target device;

[0011] A binding module is used to bind the target device to the control component corresponding to the device matching the target device under the space type in the device control interface corresponding to the scene screen of the target scene if there is a device matching the target device under the space type; the control component is used to control the target device in the device control interface.

[0012] In one embodiment, before acquiring the device of the space type corresponding to the target scene, the apparatus further includes a space type identification module, and the space type identification module is used to:

[0013] Performing feature extraction on the image to obtain spatial features in the image;

[0014] Based on the spatial features, recognition is performed to obtain the spatial type corresponding to the target scene.

[0015] In one embodiment, the device further includes a first model training module, and the space type corresponding to the target scene is identified by a trained space type deep learning model; the first model training module is used to:

[0016] Acquire sample images collected in a variety of real environments; the sample images include sample labels with marked spatial types;

[0017] Performing spatial recognition on the sample image through an initial spatial type deep learning model to obtain a spatial recognition result of the sample image; the spatial recognition result includes the spatial type of the sample image;

[0018] Based on the difference between the spatial recognition result and the sample label, the parameters of the initial spatial type deep learning model are adjusted and training is continued until the training condition is met, and the training is stopped to obtain a trained spatial type deep learning model.

[0019] In one embodiment, the target recognition module is used to:

[0020] Performing target recognition on the image collected in the target scene to obtain the device type and device status of the target device in the image;

[0021] The device matching module is used to:

[0022] Obtain the device of the space type corresponding to the target scene;

[0023] The target device is matched with the devices under the space type according to the device type and device status of the target device and the device type and device status of the devices under the space type.

[0024] In one embodiment, the device matching module is used to:

[0025] Matching a device type of the target device with a device type of the device under the space type;

[0026] If there are multiple devices of the same device type as the target device among the devices under the space type, matching the device state of the target device with the device states of the multiple devices;

[0027] If there is a device in the plurality of devices having the same device status as the target device, the device in the same device status is determined as a device matching the target device.

[0028] In one embodiment, the target recognition module is used to:

[0029] Performing target recognition on the image to obtain a position of a target device in the image;

[0030] According to the position of the target device in the image, intercepting a target area of ​​the image containing the target device;

[0031] The device status of the target area is identified to obtain the device status of the target device.

[0032] In one embodiment, the device further includes a second model training module, and the device state of the target device is identified by a trained device state deep learning model; the trained device state deep learning model is obtained by a model training step; and the second model training module is used to:

[0033] Acquire sample images of target devices collected from a variety of real environments; the sample images include sample labels of device states of the sample target devices;

[0034] Performing device state recognition on the sample image through an initial device state deep learning model to obtain a state recognition result of a sample target device in the sample image; the state recognition result includes the device state of the sample target device;

[0035] Based on the difference between the state recognition result and the sample label, the parameters of the initial device state deep learning model are adjusted and training is continued until the training condition is met, and the training is stopped to obtain a trained device state deep learning model.

[0036] In one embodiment, the device further comprises:

[0037] An interface display module, used to display the device control interface corresponding to the scene screen of the target scene;

[0038] A control component display module, used to display, in the device control interface, a control component corresponding to at least one target device in the target scene;

[0039] The instruction sending module is used to send a corresponding control instruction to a target device corresponding to the control component in response to a control operation triggered on the control component, so as to control the target device to execute the control instruction.

[0040] An electronic device comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the device control method as described above when executed by the processor.

[0041] A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the device control method described above are implemented.

[0042] A computer program product or a computer program, wherein the computer program product or the computer program comprises computer instructions, wherein the computer instructions are stored in a computer-readable storage medium; a processor of the computer device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, the steps in the device control method of each embodiment of the present application are implemented.

[0043] The above-mentioned device control method, apparatus, electronic device and storage medium perform target recognition on the image collected in the target scene to obtain the target device in the image; obtain the device under the space type corresponding to the target scene, and match the target device with the device under the space type based on the device status of the target device; if there is a device matching the target device under the space type, then in the device control interface corresponding to the scene screen of the target scene, bind the target device to the control component corresponding to the device matching under the space type; the control component is used to control the target device in the device control interface. In this way, since the binding process is based on the device status and the space type corresponding to the target scene, the degree of automation is higher, and the user can bind the device and the control component without or with only a small amount of operation, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 An application environment diagram of a device control method in an embodiment;

[0045] Figure 2 A schematic diagram of a flow chart of a device control method in an embodiment;

[0046] Figure 3 is a schematic diagram of a device control interface in one embodiment;

[0047] Figure 4A schematic diagram of the position of a target device in a scene image in an embodiment;

[0048] Figure 5 A schematic diagram of the principle of identifying the device status of a target device in an embodiment;

[0049] Figure 6 A schematic diagram of a matching process of a target device and a device of a space type corresponding to a target scene in an embodiment;

[0050] Figure 7 A schematic diagram of determining a device according to a space type corresponding to a target scene in an embodiment;

[0051] Figure 8 A schematic diagram of determining the device status of a device in a space type corresponding to a target scene in an embodiment;

[0052] Fig. 9 is a structural block diagram of a device control device in an embodiment;

[0053] Fig.10 is an internal structure diagram of an electronic device in one embodiment;

[0054] Fig.11 FIG. 4 is a diagram showing the internal structure of an electronic device in another embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The device control method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through the network, the server 104 communicates with the gateway 106 through the network, the Internet of Things device accesses the gateway 106 through the local area network, and the Internet of Things device includes the camera device 101, and may also include smart washing machines 103, smart air conditioners 105, curtain motors and other devices, which are not limited here.

[0057] The server 104, the terminal 102 or the gateway 106 can bind the target device to the control component based on the device status and the type of space where the device is located, thereby forming a control relationship between the control component and the target device in the device control interface. In this way, by operating the control component in the device control interface, the terminal 102 can send control instructions to the gateway 106 through the network and the server 104 to achieve remote control of the IoT device. In actual implementation, when the terminal 102 is connected to the local area network where the gateway 106 is located, it can also directly send control instructions to the gateway 106 through the local area network to achieve control of the IoT device.

[0058] The terminal 102 may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0059] like Figure 2 As shown, a device control method is provided, which can be applied to Figure 1 The terminal, server or gateway in the embodiment includes the following steps:

[0060] Step 202: Perform target recognition on the image captured in the target scene to obtain the target device in the image.

[0061] The target scene refers to the scene where the device needs to be controlled, which can be an environment where the target device is installed, such as a bedroom, living room, overall apartment, office, etc. The target device refers to the device in the target scene that can access the network for communication and control, which can be a smart device installed in the target scene, such as smart home devices, such as cameras, smart lights, smart curtains, smart air conditioners, smart TVs, door and window sensors, etc.

[0062] The image of the target scene is collected by an image acquisition device, which is, for example, a camera connected to the local area network where the gateway is located. The image acquisition device can be installed in the living room, bedroom, balcony, etc., and can collect images containing the target device. Taking the sending of images to the terminal as an example, the images collected by the image acquisition device can be sent to the terminal through the server. When the terminal is connected to the local area network where the gateway is located, the images collected by the image acquisition device can also be directly sent to the terminal through the gateway.

[0063] Since the target device is installed in the target scene, the image of the target scene captured by the image acquisition device will contain image features of the target device. By identifying the image features of the target device in the image, the target device contained in the image can be determined.

[0064] Step 204: Acquire the device of the space type corresponding to the target scene, and match the target device with the device of the space type based on the device status of the target device.

[0065] The space type corresponding to the target scene refers to the type of space corresponding to the target scene, and the space type can be divided based on the location where the device is installed, for example, including bedroom, living room, corridor, balcony, etc. The devices under the space type corresponding to the target scene are smart devices installed in the space corresponding to the target scene and connected to the network, for example, they can include smart home devices, such as cameras, smart lights, smart curtains, smart air conditioners, smart TVs, door and window sensors, and other devices.

[0066] Specifically, the device under the space type corresponding to the target scene can be the device being bound under the space type corresponding to the target scene, such as the device selected by the user to be bound, or all the devices that have been added under the space type corresponding to the target scene. Since the target device identified by the image is matched with the device under the space type corresponding to the target scene, the spatial range of the matched object is relatively consistent, which improves the efficiency and success rate of matching.

[0067] The device state of the target device refers to the current working state of the target device. For example, the device state of a smart light can be on, off, brightness, etc., and the device state of a curtain can be on, off, opening, etc. When the target device identified based on the image is matched with the device of the space type corresponding to the target scene based on the device state of the target device, the matching result is more accurate and the manual participation required in the matching process can be reduced.

[0068] Step 206, if there is a device matching the target device under the space type, then in the device control interface corresponding to the scene screen of the target scene, bind the target device to the control component corresponding to the device matching the space type; the control component is used to control the target device in the device control interface.

[0069] The scene picture of the target scene refers to a picture that can reflect the state of the target scene, and specifically can be a real picture or a virtual picture reflecting the target scene. The real picture can be obtained based on the image acquired by the image acquisition device, and the virtual picture can be obtained by modeling the target scene. The device control interface refers to an interface for controlling the target device in the target scene, and the device control interface can include a two-dimensional scene picture or a three-dimensional scene picture corresponding to the real target scene.

[0070] The devices and control components under the space type corresponding to the target scene are pre-bound. The control components are control components that have been connected to the local area network where the gateway is located. For example, a smart light that has been connected to the network is bound to a smart switch, and the smart light can be controlled by the smart switch. The control component can be a physical switch or a virtual switch. The virtual switch can be created in the home application.

[0071] Since the devices under the space type corresponding to the target scene are pre-bound to the control components, when the devices under the space type corresponding to the target scene have a device that matches the target device, that is, in the device control interface corresponding to the scene screen of the target scene, the target device is bound to the control component corresponding to the device matching the space type to form a control relationship, so that the control component can control the target device in the device control interface.

[0072] In the above device control method, since the binding process is based on the device status and the space type corresponding to the target scene, it is more automatic and the user can bind the device and the control component with no or only a small amount of operation, thereby improving the user experience.

[0073] In one embodiment, after binding the target device to the control component corresponding to the device matching the space type, the following steps are further included:

[0074] Display the device control interface corresponding to the scene screen of the target scene;

[0075] In the device control interface, a control component corresponding to at least one target device in the target scene is displayed;

[0076] In response to the control operation triggered on the control component, a corresponding control instruction is sent to a target device corresponding to the control component to control the target device to execute the control instruction.

[0077] Among them, after binding the target device with the control component corresponding to the device matching the space type, a binding relationship is formed between the control component and the corresponding target device, so that the corresponding target device can be controlled by the control component. For example, if a wireless switch controls a lamp, then the target device is the lamp and the control component is the wireless switch. After the control component in the device control interface receives the control operation, the terminal generates the corresponding control instruction, which can be sent to the gateway through the server to control the target device. When the terminal is connected to the local area network where the gateway is located, the control instruction can also be sent directly to the gateway to control the target device.

[0078] A schematic diagram of a device control interface displayed in a terminal is as follows Figure 3As shown in the figure, the corresponding target scene is the living room, and the scene screen is a two-dimensional scene screen. In the device control interface, there are multiple target devices such as smart curtains, smart air conditioners and smart lamps. The corresponding control components are displayed next to the target devices, and operations such as on / off can be performed. In this way, by displaying the device control interface corresponding to the scene screen of the target scene and displaying the control components bound to the target device in the interface, the smart devices in the scene screen can be controlled as seen, without the user having to select the control interface of the smart device multiple times. Multiple target devices can be controlled on one interface, which is more convenient and intuitive, greatly improving the user experience.

[0079] In one embodiment, in the device control interface, a control component corresponding to at least one target device in a target scene is displayed, including:

[0080] In the device control interface, a scene screen corresponding to the actual target scene is displayed;

[0081] In the scene picture, a control component corresponding to at least one target device in a real target scene is displayed.

[0082] Among them, the location of the target device in the scene screen is as follows Figure 4 As indicated by the box in the figure, the position where the control component is displayed in the scene picture is the position where the target device is located in the scene picture or the surrounding area of ​​the position where the control component is located. Therefore, the control component can be displayed superimposed on the corresponding target device, or displayed next to the corresponding target device, such as Figure 3 As shown, it is more convenient for users to quickly control the corresponding target device.

[0083] In one embodiment, the device control method further includes the following steps:

[0084] Display the device status entry corresponding to the target device in the control component;

[0085] In response to a trigger operation on a device status entry, current device status information of the corresponding target device is displayed.

[0086] The current device status information refers to the current status information of the target device. For example, for smart curtains, the current device status information may be fully open, closed, 60% open, etc. Figure 3 As shown, the device status entry corresponding to the target device is displayed in the control component. If the user clicks "Status", a trigger operation is generated for the device status entry, and then the current device status information of the corresponding target device is displayed, which is convenient for viewing the current status of the target device.

[0087] In one embodiment, the device control method further includes the following steps:

[0088] When the device status information of the target device is recognized to change based on the image collected in the target scene, the device status information in the device control interface is dynamically updated.

[0089] Among them, it is possible to identify whether the device status of the target device has changed based on the image collected in the target scene. For example, when the smart curtains change from fully open to 60% open, the state change of the smart curtains can be identified based on the image collected in the target scene, and then the device status information in the device control interface can be dynamically updated.

[0090] The following continues to introduce the binding process between the target device and the control component.

[0091] Please continue to refer to Figure 2 Before obtaining the device of the space type corresponding to the target scene in step 204, the following steps are also included:

[0092] Extract features from the image to obtain spatial features in the image;

[0093] Based on the spatial features, the spatial type corresponding to the target scene is obtained.

[0094] In one embodiment, the space type corresponding to the target scene is identified by a trained space type deep learning model. Among them, the trained space type deep learning model is used to identify the spatial features in the image, so as to determine the space type corresponding to the target scene based on the recognition result. Optionally, the spatial feature refers to a feature that can express the space type, including but not limited to the size of the space, the shape of the space, the light, the objects in the space, etc. For example, there are differences in the size, shape, light, and objects in the space of the bedroom, living room, corridor, and balcony. By identifying the spatial features, the corresponding space type can be determined. First, at least one frame of image is acquired, and then the image is preprocessed, such as normalization, scaling, etc., and then the preprocessed image is input into the trained space type deep learning model for space type recognition. The recognition result of the model includes the space type corresponding to the image. The trained space type deep learning model can adopt a model such as resnet, and a fully connected layer is connected after the feature extraction through the convolution layer.

[0095] In one embodiment, the trained spatial type deep learning model is obtained through a model training step; the model training step includes:

[0096] Obtain sample images collected in a variety of real environments; the sample images include sample labels with annotated spatial types;

[0097] Performing spatial recognition on the sample image through the initial spatial type deep learning model to obtain a spatial recognition result of the sample image; the spatial recognition result includes the spatial type of the sample image;

[0098] Based on the difference between the spatial recognition results and the sample labels, the parameters of the initial spatial type deep learning model are adjusted and training is continued until the training conditions are met, and the training is stopped to obtain a trained spatial type deep learning model.

[0099] Among them, when training the deep learning model of space type, multiple real environments refer to real environments under multiple acquisition conditions, and the acquisition conditions include at least one of the following: apartment type, room type, placement of acquisition equipment, and lighting. Room types include bedroom, living room, bathroom, kitchen, balcony, etc. Real environment images can be crawled from home purchase and rental websites or collected after actively arranging the home environment. By selecting real environment images collected under multiple acquisition conditions, different morphological rules of different space types under different apartment types, room types, placement of acquisition equipment, lighting, etc. can be obtained to improve the accuracy of recognition.

[0100] When training a spatial type deep learning model, first use the labelImg tool to annotate the training data with sample labels to give the spatial type of the sample image. Then, input the sample image into the initial spatial type deep learning model for forward propagation to obtain the recognition result, input the recognition result into the error function, compare it with the sample label to obtain the error, adjust the model parameters based on the error, and continue to iterate the model until the set number of iterations is completed or the error value no longer decreases. This can be considered to meet the training conditions and obtain a trained spatial type deep learning model.

[0101] In one embodiment, step 202, performing target recognition on an image captured in a target scene to obtain a target device in the image, includes:

[0102] Perform target recognition on the images collected in the target scene to obtain the device type and device status of the target device in the image.

[0103] The step of collecting images of the target scene may include: monitoring whether the image screen changes, collecting images if the image screen does not change, and continuing to collect images when the image screen is stationary if the image screen changes. When the position and angle of the image acquisition device remain unchanged, it can be considered that the image screen has not changed. Optionally, if the image screen has not changed, the frequency of image acquisition and the frequency of monitoring whether the image screen has changed can be reduced.

[0104] The device type refers to the type of the target device, such as a lamp, curtain, air conditioner, door, etc. In addition, by performing target recognition on the image, the position of the target device in the image can be obtained, and the position of the target device displayed in the scene image can be determined according to the position of the target device in the image, so that in the device control interface, the display position of the control component can be associated with the position of the target device displayed in the scene image.

[0105] In one embodiment, after the image acquisition device acquires the image, the target device in the image can be identified locally on the image acquisition device. Alternatively, after the image acquisition device acquires the image, the image can be sent to a server or terminal, and the server or terminal can identify the target device in the image.

[0106] In one embodiment, the target device in the image is identified by using a trained target deep learning model.

[0107] Among them, the trained target deep learning model is used to identify the target device in the image. First, at least one frame of image is acquired, and then the image is preprocessed, such as normalization, scaling, etc., and then the preprocessed image is input into the trained target deep learning model for target device identification. The recognition result of the model includes the device type of the target device, the display area in the image, the credibility and the bbox frame, etc. The device type of the target device and the display area in the image are finally selected for output based on the credibility.

[0108] In one embodiment, the trained target deep learning model is obtained through a model training step; the model training step includes:

[0109] Obtain sample images collected in a variety of real environments; the sample images include sample labels that have been marked with the location and category of the sample target device;

[0110] Performing target recognition on the sample image through the initial target deep learning model to obtain a target recognition result of the sample target device in the sample image; the target recognition result includes the location and category of the sample target device;

[0111] Based on the difference between the target recognition results and the sample labels, the parameters of the initial target deep learning model are adjusted and training is continued until the training conditions are met, and the training is stopped to obtain the trained target deep learning model.

[0112] Among them, when training the target deep learning model, multiple real environments refer to real environments under multiple acquisition conditions, and the acquisition conditions include at least one of the apartment type, room type, placement of the acquisition equipment, lighting, and the degree of occlusion of the target equipment. Room types include bedrooms, living rooms, bathrooms, kitchens, balconies, etc. Real environment images can be crawled from house buying and rental websites or collected after actively arranging the home environment.

[0113] When training the target deep learning model, first use the labelImg tool to annotate the sample labels of the training data to give the location and category of the device in the sample image. Then, input the sample image into the initial target deep learning model for forward propagation to obtain the recognition result, input the recognition result into the error function, compare it with the sample label to obtain the error, adjust the model parameters based on the error, and continue to iterate the model until the set number of iterations is completed or the error value no longer decreases. It can be considered that the training conditions are met and the trained target deep learning model is obtained.

[0114] By selecting real environment images collected under a variety of acquisition conditions, we can obtain the different morphological patterns of the target device under different apartment types, room types, placement of the acquisition device, lighting, degree of occlusion of the target device, etc., so that the identification object is not limited to specific product appearance and specific scenes, which can improve the accuracy of recognition and the range of identifiable devices. Even if the target device is replaced or added, it can ensure the smooth progress of device recognition and reduce the missed detection rate.

[0115] In one embodiment, obtaining the device status of the target device in the image includes:

[0116] Perform target recognition on the image to obtain the position of the target device in the image;

[0117] According to the position of the target device in the image, intercepting a target area of ​​the image containing the target device;

[0118] Perform device status identification on the target area to obtain the device status of the target device.

[0119] When the trained target deep learning model is used to identify the target in the image, the position of the target device in the image can be obtained. Please refer to Figure 5 , the box in the left figure shows the position of the target device in the image. Then, according to the position of the target device in the image, the target area of ​​the image containing the target device is intercepted to obtain Figure 5 The screenshot corresponding to the target area where the target device is located is shown on the right. After that, the device status of the target device can be obtained by performing device status recognition on the screenshot corresponding to the target area.

[0120] In one embodiment, the device state of the target device is identified by a trained device state deep learning model. First, the captured image of the target device is preprocessed, such as normalization, scaling, etc., and then the preprocessed image is input into the trained device state deep learning model for device state identification. The recognition result of the model includes the device state of the target device. The trained device state deep learning model can adopt models such as googleNet and resnet, which can be connected to a fully connected layer after feature extraction through a convolutional layer. The trained device state deep learning model can be a multi-classification model that can support the device state identification of multiple target devices. The trained device state deep learning model can also be multiple binary classification models, and each binary classification model corresponds to the identification of the device state of a target device.

[0121] In one embodiment, the trained device state deep learning model is obtained through a model training step; the model training step includes:

[0122] Acquire sample images of target devices collected from a variety of real environments; the sample images include sample labels of device states of the sample target devices annotated;

[0123] Performing device state recognition on the sample image through the initial device state deep learning model to obtain a state recognition result of the sample target device in the sample image; the state recognition result includes the device state of the sample target device;

[0124] Based on the difference between the state recognition results and the sample labels, the parameters of the initial device state deep learning model are adjusted and training is continued until the training conditions are met, and the training is stopped to obtain a trained device state deep learning model.

[0125] Among them, when training the device status deep learning model, multiple real environments refer to real environments under multiple acquisition conditions, and the acquisition conditions include at least one of the apartment type, room type, placement of the acquisition device, lighting, and the degree of occlusion of the target device. The room types include bedroom, living room, bathroom, kitchen, balcony, etc. The real environment image can be crawled from the house purchase and rental website or collected after actively arranging the home environment. By selecting the real environment images collected under multiple acquisition conditions, the different laws of the device status of the target device under different apartment types, room types, placement of the acquisition device, lighting, and the degree of occlusion of the target device can be obtained, thereby improving the accuracy of recognition. After obtaining the images collected under multiple real environments, the position of the target device in the image is first detected, and then a screenshot of the area where the target device is located is captured as a sample image of the target device, and the sample label is a device status label, such as curtains open, curtains closed, desk lamp on, desk lamp off, etc.

[0126] When training the device status deep learning model, first use the labelImg tool to annotate the training data with sample labels to give the device status of the device in the sample image. Then, input the sample image into the initial device status deep learning model for forward propagation to obtain the recognition result, input the recognition result into the error function, compare it with the sample label to obtain the error, adjust the model parameters based on the error, and continue to iterate the model until the set number of iterations is completed or the error value no longer decreases. It can be considered that the training conditions are met and a trained device status deep learning model is obtained.

[0127] In one embodiment, step 204, obtaining a device of a space type corresponding to a target scene, and matching the target device with the device of the space type based on the device state of the target device, includes:

[0128] Get the devices of the space type corresponding to the target scene;

[0129] The target device is matched with the device under the space type according to the device type and device status of the target device and the device type and device status of the device under the space type.

[0130] Please refer to Figure 6 After identifying the target device in the image, the device type of the target device and the position of the target device in the image are obtained. Then, according to the position of the target device in the image, the target area of ​​the image containing the target device is intercepted to identify the device status and obtain the device status of the target device. After determining the device type and device status of the target device, the device type and device status of the device under the space type corresponding to the target scene are obtained. According to the device type and device status of the target device and the device type and device status of the device under the space type corresponding to the target scene, the target device is matched with the device under the space type corresponding to the target scene, and the matching device is determined, and then the target device is bound to the control component corresponding to the matching device.

[0131] To obtain the device of the space type corresponding to the target scene, please refer to Figure 7 , the device in the space type corresponding to the target scene can be pre-set with the device location when joining the network, such as the default room. The device location can be matched with the space type corresponding to the target scene, so as to obtain the device in the space type corresponding to the target scene according to the device location. When obtaining the device status of the device in the space type corresponding to the target scene, please refer to Figure 8 , the device status of the device can be obtained from the control interface of the smart device, where Figure 8 (a) shows that the device status of the corridor wall light is off. Figure 8 (b) shows that the device status of the corridor wall light is on.

[0132] In one embodiment, matching the target device with the device under the space type according to the device type and device status of the target device and the device type and device status of the device under the space type includes:

[0133] Match the device type of the target device with the device type of the device under the space type;

[0134] If there are multiple devices of the same device type as the target device among the devices under the space type, the device state of the target device is matched with the device states of the multiple devices;

[0135] If there is a device with the same device status as the target device among the multiple devices, the device with the same device status is determined as a device that matches the target device.

[0136] Among them, first matching the device type of the target device with the device type of the device under the space type corresponding to the target scene can quickly narrow the matching range. If the device under the space type corresponding to the target scene has a unique device with the same device type as the target device, the target device can be directly bound to the control component of the corresponding device. For example, if it is recognized that there is a curtain in the target scene, and there is only one curtain in the device of the target scene, then it is a unique match, and the curtain can be controlled by controlling the action instructions of the curtain motor corresponding to the curtain.

[0137] If there are multiple devices of the same device type as the target device in the space type corresponding to the target scene, the device state of the target device is matched with the device states of the multiple devices. If there is a device with the same device state as the target device among the multiple devices, the device with the same device state is determined as the device that matches the target device. For example, if one curtain is identified, and there are two curtains in the target scene, the device state of each curtain can be further obtained to continue matching. Assuming that the identified curtain is closed, and one of the two curtains in the target scene is closed and the other is open, the curtain with the same device state as the identified curtain can be uniquely matched, and the binding is automatically completed. At this time, the user does not need to perform manual configuration, which is more convenient.

[0138] In actual implementation, if there is still more than one device with the same device status as the target device among multiple devices, the user can be allowed to participate in the selection and determination. At this time, a selection interface can be output, and the user can select and determine the matching device, and then the identified target device is bound to the control component of the device selected by the user. After the target device and the control component are bound, the devices that have been matched under the space type corresponding to the target scene can be deleted from the list and no longer participate in the matching of the remaining target devices, thereby improving the matching speed.

[0139] This application provides a home application scenario, taking the control of devices in the living room as an example. The devices in the living room may include curtains, air conditioners, air purifiers, etc. The home application scenario applies the above-mentioned device control method. Specifically, the application of the device control method in the home application scenario is as follows:

[0140] When the user wants to open the curtains in the living room, he can enter the home application in the terminal and then enter the device control interface. At this time, the image captured by the image acquisition device is sent to the server, the server recognizes the image, and completes the binding of the target device and the control component based on the space type of the current scene as the living room, the device type of the target device in the image, and the device status, and sends the binding result and the display position of the target device in the scene screen to the terminal. Then, the scene screen of the living room is displayed in the device control interface of the terminal, and the control component for controlling the target device is displayed at the display position of the target device, wherein the target device in the device control interface will include curtains and other devices, and the control button of the curtain motor is displayed next to the curtain, and the control components of each other are also displayed next to other devices, so that multiple target devices and corresponding control components are presented through the device control interface, giving the user a visible and controllable experience. At this time, the user can open the curtain by clicking the control button of the curtain motor in the device control interface. The binding process does not require user participation, and the operation process of the control device is more intuitive and easy to understand.

[0141] It should be understood that although Figure 2 , Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , Figure 6 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0142] Fig. 9 FIG. 1 is a structural block diagram of a device control device in an embodiment. Fig. 9 As shown, a device control device includes:

[0143] The target recognition module 902 is used to perform target recognition on the image collected in the target scene to obtain the target device in the image;

[0144] The device matching module 904 is used to obtain the device of the space type corresponding to the target scene, and match the target device with the device of the space type based on the device state of the target device;

[0145] Binding module 906 is used to bind the target device to the control component corresponding to the device matching the target device under the space type in the device control interface corresponding to the scene screen of the target scene if there is a device matching the target device under the space type; the control component is used to control the target device in the device control interface.

[0146] In one embodiment, before acquiring the device of the space type corresponding to the target scene, the apparatus further includes a space type identification module, and the space type identification module is used to:

[0147] Extract features from the image to obtain spatial features in the image;

[0148] Based on the spatial features, the spatial type corresponding to the target scene is obtained.

[0149] In one embodiment, the device further includes a first model training module, and the space type corresponding to the target scene is identified by a trained space type deep learning model; the first model training module is used to:

[0150] Obtain sample images collected in a variety of real environments; the sample images include sample labels with annotated spatial types;

[0151] Performing spatial recognition on the sample image through the initial spatial type deep learning model to obtain a spatial recognition result of the sample image; the spatial recognition result includes the spatial type of the sample image;

[0152] Based on the difference between the spatial recognition results and the sample labels, the parameters of the initial spatial type deep learning model are adjusted and training is continued until the training conditions are met, and the training is stopped to obtain a trained spatial type deep learning model.

[0153] In one embodiment, the target identification module 902 is used to:

[0154] Perform target recognition on the image collected in the target scene to obtain the device type and device status of the target device in the image;

[0155] The device matching module 904 is used to:

[0156] Get the devices of the space type corresponding to the target scene;

[0157] The target device is matched with the device under the space type according to the device type and device status of the target device and the device type and device status of the device under the space type.

[0158] In one embodiment, the device matching module 904 is used to:

[0159] Match the device type of the target device with the device type of the device under the space type;

[0160] If there are multiple devices of the same device type as the target device among the devices under the space type, the device state of the target device is matched with the device states of the multiple devices;

[0161] If there is a device with the same device status as the target device among the multiple devices, the device with the same device status is determined as a device that matches the target device.

[0162] In one embodiment, the target identification module 902 is used to:

[0163] Perform target recognition on the image to obtain the position of the target device in the image;

[0164] According to the position of the target device in the image, intercepting a target area of ​​the image containing the target device;

[0165] Perform device status identification on the target area to obtain the device status of the target device.

[0166] In one embodiment, the apparatus further includes a second model training module, and the device state of the target device is identified by a trained device state deep learning model; the trained device state deep learning model is obtained by a model training step; the second model training module is used to:

[0167] Acquire sample images of target devices collected from a variety of real environments; the sample images include sample labels of device states of the sample target devices annotated;

[0168] Performing device state recognition on the sample image through the initial device state deep learning model to obtain a state recognition result of the sample target device in the sample image; the state recognition result includes the device state of the sample target device;

[0169] Based on the difference between the state recognition results and the sample labels, the parameters of the initial device state deep learning model are adjusted and training is continued until the training conditions are met, and the training is stopped to obtain a trained device state deep learning model.

[0170] In one embodiment, the device further comprises:

[0171] An interface display module is used to display the device control interface corresponding to the scene screen of the target scene;

[0172] A control component display module, used to display a control component corresponding to at least one target device in a target scene in a device control interface;

[0173] The instruction sending module is used to send a corresponding control instruction to a target device corresponding to the control component in response to a control operation triggered on the control component, so as to control the target device to execute the control instruction.

[0174] For the specific definition of the device control device of this embodiment, please refer to the definition of the device control method in the above embodiments, which will not be repeated here. Each module in the above device control device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0175] In one embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The electronic device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store device control data. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a device control method is implemented.

[0176] Those skilled in the art will understand that Fig.10 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0177] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Fig.11As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a device control method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0178] Those skilled in the art will understand that Fig.11 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0179] In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0180] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0181] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented. For the specific definition of the steps, please refer to the definition of the device control method in the above-mentioned method embodiments, which will not be repeated here.

[0182] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0183] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A device control method, characterized in that: The method comprises: Performing target recognition on the image collected in the target scene to obtain the device type and device status of the target device in the image; Acquire a device under the space type corresponding to the target scene, and match the target device with the device under the space type based on the device type and device status of the target device and the device type and device status of the device under the space type; If there is a device matching the target device under the space type, then in the device control interface corresponding to the scene screen of the target scene, the target device is bound to the control component corresponding to the device matching the space type; the control component is used to control the target device in the device control interface.

2. The method according to claim 1, characterized in that The method further comprises: Performing feature extraction on the image to obtain spatial features in the image; Based on the spatial features, recognition is performed to obtain the spatial type corresponding to the target scene.

3. The method according to claim 1, characterized in that: The space type corresponding to the target scene is identified by a trained space type deep learning model; the trained space type deep learning model is obtained by a model training step; The model training step includes: Acquire sample images collected in a variety of real environments; the sample images include sample labels with marked spatial types; Performing spatial recognition on the sample image through an initial spatial type deep learning model to obtain a spatial recognition result of the sample image; the spatial recognition result includes the spatial type of the sample image; Based on the difference between the spatial recognition result and the sample label, the parameters of the initial spatial type deep learning model are adjusted and training is continued until the training condition is met, and the training is stopped to obtain a trained spatial type deep learning model.

4. The method according to claim 1, characterized in that The matching of the target device with the device under the space type according to the device type and device status of the target device and the device type and device status of the device under the space type includes: Matching a device type of the target device with a device type of the device under the space type; If there are multiple devices of the same device type as the target device among the devices under the space type, matching the device state of the target device with the device states of the multiple devices; If there is a device in the plurality of devices having the same device status as the target device, the device in the same device status is determined as a device matching the target device.

5. The method according to any one of claims 1 to 4, characterized in that Obtaining the device status of the target device in the image, including: Performing target recognition on the image to obtain a position of a target device in the image; According to the position of the target device in the image, intercepting a target area of ​​the image containing the target device; The device status of the target area is identified to obtain the device status of the target device.

6. The method according to claim 1, characterized in that The device state of the target device is identified by using a trained device state deep learning model; the trained device state deep learning model is obtained by a model training step; The model training step includes: Acquire sample images of target devices collected from a variety of real environments; the sample images include sample labels of device states of the sample target devices; Performing device state recognition on the sample image through an initial device state deep learning model to obtain a state recognition result of a sample target device in the sample image; the state recognition result includes the device state of the sample target device; Based on the difference between the state recognition result and the sample label, the parameters of the initial device state deep learning model are adjusted and training is continued until the training condition is met, and the training is stopped to obtain a trained device state deep learning model.

7. The method according to claim 1, characterized in that After the step of binding the target device to a control component corresponding to a device matching the space type, the method further includes: Displaying the device control interface corresponding to the scene screen of the target scene; In the device control interface, a control component corresponding to at least one target device in the target scene is displayed; In response to the control operation triggered on the control component, a corresponding control instruction is sent to a target device corresponding to the control component to control the target device to execute the control instruction.

8. A device control device, characterized in that: The device comprises: A target recognition module is used to perform target recognition on the image collected in the target scene to obtain the device type and device status of the target device in the image; A device matching module, used to obtain a device under the space type corresponding to the target scene, and match the target device with the device under the space type based on the device type and device status of the target device and the device type and device status of the device under the space type; A binding module is used to bind the target device to the control component corresponding to the device matching the target device under the space type in the device control interface corresponding to the scene screen of the target scene if there is a device matching the target device under the space type; the control component is used to control the target device in the device control interface.

9. An electronic device, characterized in that: The device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the device control method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the device control method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by one or more processors, the device control method according to any one of claims 1 to 7 is implemented.

Images