Equipment control method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an equipment control method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of intelligent device.The method comprises the steps that gesture data to be recognized are recognized through a gesture recognition model, a gesture recognition result is obtained, a target equipment instruction corresponding to the gesture recognition result is determined, and the target equipment instruction is sent to a server; and controlling the target equipment according to the target equipment instruction. The gesture recognition model is obtained by optimizing an initial recognition model based on a sample gesture recognition result and an expected gesture recognition result corresponding to the sample gesture recognition result, the sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the sample gesture recognition result. The accuracy of a model gesture recognition result is improved, and the experience of gesture-based equipment control is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of smart devices, and specifically to a device control method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] Currently, there are various ways to control smart devices. For example, gesture recognition can be used to control a device using gestures, and users can achieve different controls on the device by changing gestures.

[0003] However, the accuracy of the target gesture recognition results needs to be improved, resulting in a poor experience of gesture-based device control. Summary of the Invention

[0004] The embodiments of the present application provide a device control method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of gesture recognition results and enhance the experience of gesture-based device control.

[0005] In a first aspect, an embodiment of the present application provides a device control method, the method comprising: The gesture data to be recognized is recognized by the gesture recognition model to obtain a gesture recognition result; Determining a target device instruction corresponding to the gesture recognition result, and controlling the target device according to the target device instruction; The gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results.

[0006] In a second aspect, an embodiment of the present application further provides a device control apparatus, the apparatus comprising: A recognition module is used to recognize the gesture data to be recognized through a gesture recognition model to obtain a gesture recognition result; A control module, configured to determine a target device instruction corresponding to the gesture recognition result and control the target device according to the target device instruction; The gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results.

[0007] Optionally, in some embodiments of the present application, before the gesture data to be recognized is recognized by the gesture recognition model and a gesture recognition result is obtained, the method further includes: Recognizing the sample gesture data to be recognized by the initial recognition model to obtain a sample gesture recognition result, wherein the sample gesture recognition result is determined based on the confidence level of each sample recognition result of the sample gesture data; If the sample gesture recognition result indicates that no gesture is recognized, setting the sample recognition result corresponding to the highest confidence as the expected gesture recognition result; If the sample gesture recognition result indicates that a gesture is recognized, controlling the target device according to the sample device instruction corresponding to the sample gesture recognition result, and if a sample expected device instruction that meets a preset condition is further detected after controlling the target device, setting the sample recognition result corresponding to the sample expected device instruction as the expected gesture recognition result; The gesture recognition model is obtained by optimizing the initial recognition model according to the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result.

[0008] Optionally, in some embodiments of the present application, if the sample gesture recognition result indicates that the gesture is not recognized, before taking the sample recognition result corresponding to the highest confidence as the expected gesture recognition result, the method further includes: controlling a target device based on the sample gesture recognition result; If the target device does not respond after the first set time period has expired, the sample gesture recognition result is determined to be an unrecognized gesture.

[0009] Optionally, in some embodiments of the present application, if a sample expected device instruction that meets a preset condition is detected after controlling the target device, before using the sample recognition result corresponding to the sample expected device instruction as the expected gesture recognition result, the method further includes: If another device instruction different from the sample device instruction is detected within a second set time period after the target device is controlled according to the sample device instruction, the other device instruction is determined to be the sample expected device instruction that meets a preset condition.

[0010] Optionally, in some embodiments of the present application, optimizing the initial recognition model to obtain the gesture recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result includes: Counting the number of generated expected gesture recognition results within a first preset period; If the generated quantity reaches a quantity threshold, the initial recognition model is retrained according to each expected gesture recognition result and the corresponding sample gesture data and each sample gesture recognition result to obtain the gesture recognition model.

[0011] Optionally, in some embodiments of the present application, optimizing the initial recognition model to obtain the gesture recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result includes: According to a second preset period, the initial recognition model is fine-tuned based on each expected gesture recognition result within the second preset period and the corresponding sample gesture data and each sample gesture recognition result to obtain the gesture recognition model.

[0012] Optionally, in some embodiments of the present application, determining a target device instruction corresponding to the gesture recognition result includes: Determine the target device to be controlled; A target device instruction matching the target device is determined according to the gesture recognition result.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the above-mentioned device control method are implemented.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned device control method are implemented.

[0015] In a fifth aspect, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in the embodiments of the present application.

[0016] In summary, the embodiments of the present application use a gesture recognition model to identify gesture data to be recognized, obtain a gesture recognition result, determine a target device instruction corresponding to the gesture recognition result, and control a target device according to the target device instruction. The gesture recognition model is obtained by optimizing an initial recognition model based on sample gesture recognition results and an expected gesture recognition result corresponding to the sample gesture recognition result. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on response feedback to the sample gesture recognition result.

[0017] The gesture recognition model is used to recognize gesture data, and a target device instruction is determined based on the recognized gesture recognition result. The target device is then controlled based on the target device instruction, thereby realizing gesture-based device control.

[0018] Among them, by determining the expected gesture recognition results based on the sample gesture recognition results, and optimizing the model by combining the sample gesture recognition results and the expected gesture recognition results, the accuracy of the model gesture recognition results is improved, and the experience of gesture-based device control is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 Schematic diagram of a scenario in which a terminal device according to an embodiment of the present application executes the device control method; Figure 2 This is a flow chart of a device control method provided in an embodiment of the present application; Figure 3 It is a structural diagram of the device control device provided in an embodiment of the present application; Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application.

[0021] Description of Figure Numbers: 101 - terminal device; 301 - identification module; 302 - control module; 401 - processor; 402 - memory; 403 - power supply; 404 - input unit. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0023] The embodiments of the present application provide a device control method, apparatus, electronic device and computer-readable storage medium. Specifically, the embodiments of the present application provide a device control device suitable for an electronic device, wherein the electronic device includes a terminal device or a server, and the terminal device includes but is not limited to a mobile phone, a laptop computer, a desktop computer, a television or an extended reality device (such as virtual reality (VR), augmented reality (AR) and mixed reality (MR) devices). The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The server can be directly or indirectly connected via wired or wireless communication.

[0024] For example, see Figure 1 , Figure 1 : This is a schematic diagram of a scenario in which a terminal device according to an embodiment of the present application executes the device control method. Specifically, the specific execution process of the terminal device executing the device control method is as follows: The terminal device 101 recognizes the gesture data to be recognized through the gesture recognition model, obtains a gesture recognition result, determines a target device instruction corresponding to the gesture recognition result, and controls the target device according to the target device instruction.

[0025] Among them, the gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition result and the expected gesture recognition result corresponding to the sample gesture recognition result. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the response feedback to the sample gesture recognition result.

[0026] For example, in the case of AR glasses, where the terminal device is an extended reality device, the AR glasses initiate gesture recognition by actively identifying the scene environment or receiving user commands to control the opening of the gesture interaction channel. Furthermore, the AR glasses collect user gesture data, call the gesture recognition model to recognize the gesture data, and obtain a gesture recognition result. The target device command corresponding to the gesture recognition result is then determined, and the target device is controlled according to the target device command.

[0027] In summary, the embodiments of the present application recognize gesture data through a gesture recognition model, determine a target device instruction based on the recognized gesture recognition result, and control the target device based on the target device instruction, thereby realizing gesture-based device control.

[0028] Among them, by determining the expected gesture recognition results based on the sample gesture recognition results, and optimizing the model by combining the sample gesture recognition results and the expected gesture recognition results, the accuracy of the model gesture recognition results is improved, and the experience of gesture-based device control is improved.

[0029] It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0030] See also Figure 2 , Figure 2 This is a flow chart of a device control method provided by an embodiment of the present application. Although the flow chart shows a logical order, in some cases, the steps shown or described may be performed in a different order than that shown in the flow chart. Specifically, the specific process of the device control method is as follows: 201. Recognize the gesture data to be recognized using a gesture recognition model to obtain a gesture recognition result.

[0031] It is understood that the gesture recognition model is a deep learning model or a machine learning model, including but not limited to models trained based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), spatiotemporal graph convolutional networks (ST-GCNs), and other networks. In the embodiments of the present application, the gesture recognition model can be deployed in AR glasses, the cloud, and the like.

[0032] Among them, gesture data is data related to the hand, including finger sliding, wrist bending, etc., and the gesture data can be collected by sensors, for example, by an IMU sensor in the device. Among them, in the embodiment of the present application, gesture data can also be collected by other devices, for example, by using a sensor with a built-in IMU such as a watch, bracelet, ring, mobile phone, etc. to collect gesture data, and receiving the gesture data sent by the watch, bracelet, ring, mobile phone, etc. to obtain the gesture data to be recognized.

[0033] The gesture recognition result is a recognition result for the gesture data, and the gesture recognition result includes a gesture type. For example, the gesture recognition result includes but is not limited to gesture types such as click, lateral movement, and palm extension. If the gesture recognition result is empty, it indicates that the gesture was not recognized.

[0034] 202. Determine a target device instruction corresponding to the gesture recognition result, and control a target device according to the target device instruction.

[0035] For example, according to the correspondence between the gesture recognition result and the device command, the target device command corresponding to the current gesture recognition result is determined, for example, a confirmation command corresponding to a click, a screen drag command corresponding to a horizontal shift, and the like.

[0036] Accordingly, after determining the target device command, the target device is controlled according to the target device command. For example, based on the confirmation command, a control on the screen is clicked to confirm, and the screen is dragged using the drag command to change the interface display. Target devices include devices such as extended reality devices, smart TVs, or air conditioners.

[0037] In an embodiment of the present application, the gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition result and the expected gesture recognition result corresponding to the sample gesture recognition result. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the response feedback to the sample gesture recognition result.

[0038] Response feedback refers to feedback on the response to the sample gesture recognition result, or refers to the response result corresponding to the output response, such as feedback on the response collected, captured, or recognized by AR glasses. For example, response feedback includes no response to the sample gesture recognition result or a response that does not meet expectations. For example, if the user inputs a new device command after the target device responds, it indicates that the response to the sample gesture recognition result does not meet expectations, that is, the response to the gesture data does not meet expectations.

[0039] For example, the sample gesture data is recognized by the initial recognition model to obtain a sample gesture recognition result, and the expected gesture recognition result is determined based on the sample gesture recognition result. Then, the initial recognition model is optimized by combining the sample gesture recognition result and the expected gesture recognition result to obtain a gesture recognition model.

[0040] In summary, the embodiments of the present application recognize gesture data through a gesture recognition model, determine a target device instruction based on the recognized gesture recognition result, and control the target device based on the target device instruction, thereby realizing gesture-based device control.

[0041] Among them, by determining the expected gesture recognition results based on the sample gesture recognition results, and optimizing the model by combining the sample gesture recognition results and the expected gesture recognition results, the accuracy of the model gesture recognition results is improved, and the experience of gesture-based device control is improved.

[0042] In the embodiments of the present application, the corresponding expected gesture recognition results may be determined respectively depending on whether the initial recognition model recognizes the gesture. That is, optionally, in some embodiments of the present application, before the step of "recognizing the gesture data to be recognized by the gesture recognition model to obtain the gesture recognition result", the method further includes: Recognizing the sample gesture data to be recognized by the initial recognition model to obtain a sample gesture recognition result, wherein the sample gesture recognition result is determined based on the confidence level of each sample recognition result of the sample gesture data; If the sample gesture recognition result indicates that no gesture is recognized, setting the sample recognition result corresponding to the highest confidence as the expected gesture recognition result; If the sample gesture recognition result indicates that a gesture is recognized, controlling the target device according to the sample device instruction corresponding to the sample gesture recognition result, and if a sample expected device instruction that meets a preset condition is further detected after controlling the target device, setting the sample recognition result corresponding to the sample expected device instruction as the expected gesture recognition result; The gesture recognition model is obtained by optimizing the initial recognition model according to the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result.

[0043] It should be noted that the sample recognition results are preset recognition results, which include all possible results of gesture recognition. For example, three sample recognition results such as extending the palm, clicking, and horizontal sliding are pre-set. Gesture recognition calculates the matching degree between the sample gesture data and each sample recognition result, thereby filtering out the sample gesture recognition result from multiple sample recognition results.

[0044] It is understood that the initial recognition model determines the sample gesture recognition result based on the confidence level of each sample recognition result in the sample gesture data. For example, a sample recognition result with a confidence level greater than 90% is used as the sample gesture recognition result. Correspondingly, if there is no sample recognition result with a confidence level greater than 90%, a sample gesture recognition result indicating that the gesture was not recognized is obtained. In the embodiment of the present application, in the case where a gesture is not recognized, the sample recognition result with the highest confidence level is directly used as the expected gesture recognition result.

[0045] Furthermore, in the case where a gesture is recognized, the target device is controlled using the sample device command corresponding to the gesture, and a sample expected device command is determined based on whether a new device command corresponding to the target device's response is generated, thereby determining the expected gesture recognition result. For example, after controlling the target device based on the sample device command, the target device generates a response to the sample device command. If the user is dissatisfied with the response, they may re-enter a new device command, such as by voice or remote control, to enter an accurate device command. In this embodiment, the device command entered at this time is used as the sample expected device command, and thus, based on the correspondence between the device command and the gesture recognition result, the expected gesture recognition result corresponding to the sample expected device command is obtained.

[0046] Among them, the preset condition refers to a preset condition, for example, a time condition. For example, within a short period of time after controlling the target device based on the sample device instruction, if a new device instruction different from the sample device instruction is identified, the new device instruction is used as the sample expected device instruction.

[0047] Therefore, after obtaining the sample gesture recognition result and the expected gesture recognition result, the embodiment of the present application can optimize the initial recognition model in combination with the sample gesture data to obtain a gesture recognition model.

[0048] Among them, the solution of optimizing the initial recognition model based on sample gesture data is also applicable to the optimization of gesture recognition models in practical applications. For example, taking AR glasses with an extended reality device as an example, the AR glasses obtain gesture data to be recognized during use, recognize the gesture data through the gesture recognition model to obtain the actual gesture recognition result, and then determine the corresponding expected gesture recognition result based on whether the actual gesture recognition result recognizes the gesture difference, and automatically optimize the model based on the actual gesture recognition result and the expected gesture recognition result. For example, if the gesture recognition model is deployed in the AR glasses, the gesture recognition model deployed in the AR glasses is automatically optimized based on the actual gesture recognition result and the expected gesture recognition result.

[0049] In the embodiment of the present application, whether the actual gesture recognition result recognizes the gesture and how to determine the corresponding expected gesture recognition result can be set as an automated process, so that the user can realize the optimization of the gesture recognition model without feeling when using AR glasses.

[0050] In the embodiment of the present application, if it is detected that the target device does not respond for a long time, it is determined that the gesture is not recognized. That is, optionally, in some embodiments of the present application, before the step of "if the sample gesture recognition result indicates that the gesture is not recognized, then using the sample recognition result corresponding to the highest confidence as the expected gesture recognition result", the method further includes: controlling a target device based on the sample gesture recognition result; If the target device does not respond after the first set time period has expired, the sample gesture recognition result is determined to be an unrecognized gesture.

[0051] The first set duration refers to the duration for the target device to respond after the user makes a gesture, and the duration can be obtained by averaging the durations of multiple gesture recognition and responses.

[0052] Wherein, the lack of response from the target device indicates that the sample device instruction has not been obtained, and accordingly, it also indicates that the sample gesture recognition result has not been obtained.

[0053] In an embodiment of the present application, if other device instructions different from the sample device instructions are detected within a certain period of time after the target device is controlled based on the sample device instruction, the other device instructions are directly used as the sample expected device instructions. That is, optionally, in some embodiments of the present application, before the step of "if a sample expected device instruction that meets a preset condition is still detected after controlling the target device, the sample recognition result corresponding to the sample expected device instruction is used as the expected gesture recognition result", the method further includes: If another device instruction different from the sample device instruction is detected within a second set time period after the target device is controlled according to the sample device instruction, the other device instruction is determined to be the sample expected device instruction that meets a preset condition.

[0054] For example, after a user makes a gesture to control a device response, if the user re-enters another device command within a short period of time, it indicates that the user is more inclined to use the other device command to control the target device.

[0055] Among them, the second set duration can be determined based on the time interval between the generation of two sample device instructions. For example, through sampling statistics, it is obtained that the time interval between two user gesture interactions is approximately t, then the second set duration is set to t*y%, where y is greater than 0 and less than 100. y can be set according to actual needs and is not limited here.

[0056] It is understandable that when an expected gesture recognition result is generated, it indicates that the initial recognition model has deviated from the recognition of the sample gesture data and does not meet the user's expectations. The optimization of the model can be controlled based on the generation of the expected gesture recognition result. In an embodiment of the present application, the gesture recognition model can be obtained by retraining or fine-tuning the initial recognition model. For example, when the number of expected gesture recognition results generated within a first set period (the number of deviations in the corresponding gesture recognition) reaches or exceeds a threshold, the initial recognition model is retrained based on the sample gesture recognition results and the expected gesture recognition results to obtain the gesture recognition model. That is, optionally, in some embodiments of the present application, the step of "optimizing the initial recognition model to obtain the gesture recognition model based on the sample gesture data, the sample gesture recognition results and the expected gesture recognition results" includes: Counting the number of generated expected gesture recognition results within a first preset period; If the generated quantity reaches a quantity threshold, the initial recognition model is retrained according to each expected gesture recognition result and the corresponding sample gesture data and each sample gesture recognition result to obtain the gesture recognition model.

[0057] The quantity threshold can be preset based on actual needs.

[0058] Accordingly, in an embodiment of the present application, the initial recognition model may be fine-tuned according to a set period to obtain a gesture recognition model. That is, optionally, in some embodiments of the present application, the step of "optimizing the initial recognition model to obtain the gesture recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result" includes: According to a second preset period, the initial recognition model is fine-tuned based on each expected gesture recognition result within the second preset period and the corresponding sample gesture data and each sample gesture recognition result to obtain the gesture recognition model.

[0059] For example, the cross entropy loss is calculated by the difference between the sample gesture recognition result and the expected gesture recognition result, and the model is fine-tuned using the cross entropy loss to obtain the gesture recognition model.

[0060] Among them, the first preset period and the second preset period can also be set based on actual needs. The first preset period and the second preset period can be the same or different. For example, the first preset period and the second preset period can be set to 1 day, three days or a week.

[0061] In the embodiments of the present application, in order to ensure the accuracy of the generated target device instructions, the target device instructions for the device object may be generated based on the device object to be controlled (e.g., the target device). That is, optionally, in some embodiments of the present application, the step of "determining the target device instructions corresponding to the gesture recognition result" includes: Determine the target device to be controlled; A target device instruction matching the target device is determined according to the gesture recognition result.

[0062] For example, based on the gesture recognition results, different target device instructions are generated for different target devices. For example, based on the gesture recognition results for lateral movement, a target device instruction for screen sliding is generated for the user interface device, and for the air conditioner, a target device instruction for increasing the temperature and wind speed is generated. The generation of target device instructions based on the target device to be controlled improves the diversity and flexibility of gesture control for different devices, and also improves the accuracy of target device instructions, avoiding the generation of instructions that cannot control the target device.

[0063] In summary, the embodiments of the present application recognize gesture data through a gesture recognition model, determine a target device instruction based on the recognized gesture recognition result, and control the target device based on the target device instruction, thereby realizing gesture-based device control.

[0064] Among them, by determining the expected gesture recognition results based on the sample gesture recognition results, and optimizing the model by combining the sample gesture recognition results and the expected gesture recognition results, the accuracy of the model gesture recognition results is improved, and the experience of gesture-based device control is improved.

[0065] Among them, by detecting the instructions of another device when the target device does not respond and responds, the judgment of the flawed sample gesture recognition results is achieved, and the sample expected device instructions are obtained by means of confidence and the instructions of another device input by the user, thereby improving the accuracy of the device instructions.

[0066] The timing of model optimization is controlled by determining the generated quantity and the quantity threshold within the first preset period and performing fine-tuning within the second preset period.

[0067] To facilitate better implementation of the device control method of the present application, the present application also provides a device control apparatus based on the device control method. The meanings of the terms herein are the same as those in the device control method, and the specific implementation details can be referred to the description in the method embodiment.

[0068] See also Figure 3 , Figure 3: is a schematic diagram of the structure of the device control device provided in an embodiment of the present application, wherein the device control device can be specifically as follows: The recognition module 301 is used to recognize the gesture data to be recognized through the gesture recognition model to obtain a gesture recognition result; The control module 302 is used to determine the target device instruction corresponding to the gesture recognition result and control the target device according to the target device instruction; The gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results.

[0069] Optionally, in some embodiments of the present application, before the gesture data to be recognized is recognized by the gesture recognition model and a gesture recognition result is obtained, the method further includes: Recognizing the sample gesture data to be recognized by the initial recognition model to obtain a sample gesture recognition result, wherein the sample gesture recognition result is determined based on the confidence level of each sample recognition result of the sample gesture data; If the sample gesture recognition result indicates that no gesture is recognized, setting the sample recognition result corresponding to the highest confidence as the expected gesture recognition result; If the sample gesture recognition result indicates that a gesture is recognized, controlling the target device according to the sample device instruction corresponding to the sample gesture recognition result, and if a sample expected device instruction that meets a preset condition is further detected after controlling the target device, setting the sample recognition result corresponding to the sample expected device instruction as the expected gesture recognition result; The gesture recognition model is obtained by optimizing the initial recognition model according to the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result.

[0070] Optionally, in some embodiments of the present application, if the sample gesture recognition result indicates that the gesture is not recognized, before taking the sample recognition result corresponding to the highest confidence as the expected gesture recognition result, the method further includes: controlling a target device based on the sample gesture recognition result; If the target device does not respond after the first set time period has expired, the sample gesture recognition result is determined to be an unrecognized gesture.

[0071] Optionally, in some embodiments of the present application, if a sample expected device instruction that meets a preset condition is detected after controlling the target device, before using the sample recognition result corresponding to the sample expected device instruction as the expected gesture recognition result, the method further includes: If another device instruction different from the sample device instruction is detected within a second set time period after the target device is controlled according to the sample device instruction, the other device instruction is determined to be the sample expected device instruction that meets a preset condition.

[0072] Optionally, in some embodiments of the present application, optimizing the initial recognition model to obtain the gesture recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result includes: Counting the number of generated expected gesture recognition results within a first preset period; If the generated quantity reaches a quantity threshold, the initial recognition model is retrained according to each expected gesture recognition result and the corresponding sample gesture data and each sample gesture recognition result to obtain the gesture recognition model.

[0073] Optionally, in some embodiments of the present application, optimizing the initial recognition model to obtain the gesture recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result includes: According to a second preset period, the initial recognition model is fine-tuned based on each expected gesture recognition result within the second preset period and the corresponding sample gesture data and each sample gesture recognition result to obtain the gesture recognition model.

[0074] Optionally, in some embodiments of the present application, determining a target device instruction corresponding to the gesture recognition result includes: Determine the target device to be controlled; A target device instruction matching the target device is determined according to the gesture recognition result.

[0075] In the embodiment of the present application, the recognition module 301 first recognizes the gesture data to be recognized through the gesture recognition model to obtain a gesture recognition result. The control module 302 is used to determine the target device instruction corresponding to the gesture recognition result and control the target device according to the target device instruction. The gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition result and the expected gesture recognition result corresponding to the sample gesture recognition result. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the response feedback to the sample gesture recognition result.

[0076] In this embodiment of the present application, gesture data is recognized through a gesture recognition model, and a target device instruction is determined based on the recognized gesture recognition result, and the target device is controlled based on the target device instruction to realize gesture-based device control.

[0077] Among them, by determining the expected gesture recognition results based on the sample gesture recognition results, and optimizing the model by combining the sample gesture recognition results and the expected gesture recognition results, the accuracy of the model gesture recognition results is improved, and the experience of gesture-based device control is improved.

[0078] In addition, the present application also provides an electronic device, such as Figure 4 , which shows a schematic diagram of the structure of the electronic device provided in an embodiment of the present application, specifically: The electronic device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will understand that Figure 4 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. Processor 401 is the control center of the electronic device, connecting the various parts of the entire electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in memory 402 and accessing data stored in memory 402, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, processor 401 may include one or more processing cores; preferably, processor 401 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0079] Memory 402 can be used to store software programs and modules. Processor 401 executes various functional applications and data processing by running the software programs and modules stored in memory 402. Memory 402 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as sound playback or image playback); the data storage area may store data generated based on the use of the electronic device. Furthermore, memory 402 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 402 may also include a memory controller to provide processor 401 with access to memory 402.

[0080] The electronic device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power supply device debugging circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0081] The electronic device may further include an input unit 404, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0082] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby implementing the steps of any device control method provided in the embodiments of the present application.

[0083] In the embodiments of the present application, a gesture recognition model is used to identify gesture data to be recognized, obtain a gesture recognition result, determine a target device instruction corresponding to the gesture recognition result, and control a target device according to the target device instruction. The gesture recognition model is obtained by optimizing an initial recognition model based on sample gesture recognition results and an expected gesture recognition result corresponding to the sample gesture recognition result. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the sample gesture recognition result.

[0084] The gesture recognition model is used to recognize gesture data, and a target device instruction is determined based on the recognized gesture recognition result. The target device is then controlled based on the target device instruction, thereby realizing gesture-based device control.

[0085] Among them, by determining the expected gesture recognition results based on the sample gesture recognition results, and optimizing the model by combining the sample gesture recognition results and the expected gesture recognition results, the accuracy of the model gesture recognition results is improved, and the experience of gesture-based device control is improved.

[0086] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0087] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0088] To this end, the present application provides a computer-readable storage medium having a computer program stored thereon. The computer program can be loaded by a processor to execute the steps in any device control method provided in the present application.

[0089] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0090] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0091] Since the instructions stored in the computer-readable storage medium can execute the steps in any device control method provided in this application, the beneficial effects that can be achieved by any device control method provided in this application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0092] The above is a detailed introduction to a device control method, apparatus, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A device control method, characterized in that: The method comprises: The gesture data to be recognized is recognized by the gesture recognition model to obtain a gesture recognition result; Determining a target device instruction corresponding to the gesture recognition result, and controlling the target device according to the target device instruction; The gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results.

2. The device control method according to claim 1, wherein: Before the gesture recognition model is used to recognize the gesture data to be recognized and a gesture recognition result is obtained, the method further includes: Recognizing the sample gesture data to be recognized by the initial recognition model to obtain a sample gesture recognition result, wherein the sample gesture recognition result is determined based on the confidence level of each sample recognition result of the sample gesture data; If the sample gesture recognition result indicates that no gesture is recognized, setting the sample recognition result corresponding to the highest confidence as the expected gesture recognition result; If the sample gesture recognition result indicates that a gesture is recognized, controlling the target device according to the sample device instruction corresponding to the sample gesture recognition result, and if a sample expected device instruction that meets a preset condition is further detected after controlling the target device, setting the sample recognition result corresponding to the sample expected device instruction as the expected gesture recognition result; The gesture recognition model is obtained by optimizing the initial recognition model according to the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result.

3. The device control method according to claim 2, characterized in that: If the sample gesture recognition result indicates that no gesture is recognized, the method further includes: before taking the sample recognition result corresponding to the highest confidence as the expected gesture recognition result. controlling a target device based on the sample gesture recognition result; If the target device does not respond after the first set time period has expired, the sample gesture recognition result is determined to be an unrecognized gesture.

4. The device control method according to claim 2, wherein: If a sample expected device instruction that meets a preset condition is detected after controlling the target device, the method further includes: If another device instruction different from the sample device instruction is detected within a second set time period after the target device is controlled according to the sample device instruction, the other device instruction is determined to be the sample expected device instruction that meets a preset condition.

5. The device control method according to claim 2, wherein: The step of optimizing the initial recognition model to obtain the gesture recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result includes: Counting the number of generated expected gesture recognition results within a first preset period; If the generated quantity reaches a quantity threshold, the initial recognition model is retrained according to each expected gesture recognition result and the corresponding sample gesture data and each sample gesture recognition result to obtain the gesture recognition model.

6. The device control method according to claim 2, wherein: The step of optimizing the initial recognition model to obtain the gesture recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result includes: According to a second preset period, the initial recognition model is fine-tuned based on each expected gesture recognition result within the second preset period and the corresponding sample gesture data and each sample gesture recognition result to obtain the gesture recognition model.

7. The device control method according to claim 1, wherein: The determining of the target device instruction corresponding to the gesture recognition result includes: Determine the target device to be controlled; A target device instruction matching the target device is determined according to the gesture recognition result.

8. A device control device, characterized in that: The device comprises: A recognition module is used to recognize the gesture data to be recognized through a gesture recognition model to obtain a gesture recognition result; A control module, configured to determine a target device instruction corresponding to the gesture recognition result and control the target device according to the target device instruction; The gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results.

9. An electronic device, characterized in that: The device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the device control method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the device control method according to any one of claims 1 to 7 are implemented.

Images