Device control method and apparatus, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a device control method and device, electronic device and computer readable storage medium, and relate to the technical field of intelligent devices. The method comprises: identifying gesture data to be identified by a gesture recognition model to obtain a gesture recognition result, determining a target device instruction corresponding to the gesture recognition result, and controlling a target device according to the target device instruction. The gesture recognition model is obtained by optimizing an initial recognition model based on a sample gesture recognition result and an expected gesture recognition result corresponding to the sample gesture recognition result. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the sample gesture recognition result. The accuracy of the model gesture recognition result is improved, and the experience of gesture-based device control is improved.

Description

Technical Field

[0001] This application relates to the field of smart device technology, specifically to a device control method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, there are various ways to control smart devices. For example, by recognizing gestures to control devices, users can control the device in different ways by changing gestures.

[0003] However, the accuracy of the target's gesture recognition results still needs to be improved, resulting in a poor experience in gesture-based device control. Summary of the Invention

[0004] This application provides a device control method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of gesture recognition results and enhance the experience of gesture-based device control.

[0005] In a first aspect, embodiments of this application provide a device control method, the method comprising:

[0006] The gesture recognition model is used to identify the gesture data to be recognized, and the gesture recognition result is obtained.

[0007] Determine the target device instruction corresponding to the gesture recognition result, and control the target device according to the target device instruction;

[0008] The gesture recognition model is obtained by optimizing an initial recognition model based on sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results.

[0009] Secondly, embodiments of this application also provide a device control apparatus, the apparatus comprising:

[0010] The recognition module is used to recognize the gesture data to be recognized through the gesture recognition model and obtain the gesture recognition result;

[0011] The control module is used to determine the target device instruction corresponding to the gesture recognition result, and control the target device according to the target device instruction;

[0012] The gesture recognition model is obtained by optimizing an initial recognition model based on sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results.

[0013] Optionally, in some embodiments of this application, before the gesture recognition model is used to recognize the gesture data to be recognized and obtain the gesture recognition result, the method further includes:

[0014] The initial recognition model is used to identify the sample gesture data to be identified, and the sample gesture recognition result is obtained. The sample gesture recognition result is determined based on the confidence level of the sample gesture data for each sample recognition result.

[0015] If the sample gesture recognition result indicates that no gesture was recognized, then the sample recognition result with the highest confidence level is set as the expected gesture recognition result;

[0016] If the sample gesture recognition result indicates that a gesture has been recognized, then the target device is controlled according to the sample device instruction corresponding to the sample gesture recognition result. If a sample desired device instruction that meets the preset conditions is detected after controlling the target device, then the sample recognition result corresponding to the sample desired device instruction is set as the desired gesture recognition result.

[0017] Based on the sample gesture data, the sample gesture recognition results, and the expected gesture recognition results, the initial recognition model is optimized to obtain the gesture recognition model.

[0018] Optionally, in some embodiments of this application, before taking the sample recognition result with the highest confidence level as the expected gesture recognition result if the sample gesture recognition result indicates that no gesture has been recognized, the method further includes:

[0019] Control the target device based on the sample gesture recognition results;

[0020] If the target device does not respond after a first set time period, the sample gesture recognition result is determined to be that the gesture was not recognized.

[0021] Optionally, in some embodiments of this application, before the step of using the sample recognition result corresponding to the sample expectation device instruction as the expected gesture recognition result if a sample expectation device instruction that meets preset conditions is detected after controlling the target device, the method further includes:

[0022] If, within a second set time period after controlling the target device according to the sample device instruction, another device instruction different from the sample device instruction is detected, then the other device instruction is determined to be the sample expected device instruction that meets the preset conditions.

[0023] Optionally, in some embodiments of this application, optimizing the initial recognition model based on the sample gesture data, the sample gesture recognition result, and the desired gesture recognition result to obtain the gesture recognition model includes:

[0024] The number of desired gesture recognition results generated within the first preset period is counted.

[0025] If the number of generated gestures reaches a certain threshold, the initial recognition model is retrained based on the expected gesture recognition results and the corresponding sample gesture data and sample gesture recognition results to obtain the gesture recognition model.

[0026] Optionally, in some embodiments of this application, optimizing the initial recognition model based on the sample gesture data, the sample gesture recognition result, and the desired gesture recognition result to obtain the gesture recognition model includes:

[0027] According to the second preset period, the initial recognition model is fine-tuned based on the expected gesture recognition results and the corresponding sample gesture data and sample gesture recognition results within the second preset period to obtain the gesture recognition model.

[0028] Optionally, in some embodiments of this application, the step of determining the target device instruction corresponding to the gesture recognition result includes:

[0029] Identify the target equipment to be controlled;

[0030] The target device command matching the target device is determined based on the gesture recognition result.

[0031] Thirdly, embodiments of this application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the device control method described above.

[0032] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the device control method described above.

[0033] Fifthly, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in embodiments of this application.

[0034] In summary, this application embodiment uses a gesture recognition model to recognize the gesture data to be recognized, obtains the gesture recognition result, determines the target device command corresponding to the gesture recognition result, and controls the target device according to the target device command. Specifically, the gesture recognition model is obtained by optimizing an initial recognition model based on sample gesture recognition results and the expected gesture recognition result corresponding to those sample gesture recognition results. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the response feedback to the sample gesture recognition result.

[0035] Specifically, gesture recognition data is recognized through a gesture recognition model, and the target device command is determined based on the recognized gesture recognition result. The target device is then controlled based on the target device command, thereby realizing gesture-based device control.

[0036] Specifically, by determining the desired gesture recognition result based on the sample gesture recognition result, and then optimizing the model by combining the sample gesture recognition result and the desired gesture recognition result, the accuracy of the model's gesture recognition result is improved, thus enhancing the experience of gesture-based device control. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of a scenario in which a terminal device, according to an embodiment of this application, executes the device control method;

[0039] Figure 2 This is a schematic flowchart of the device control method provided in the embodiments of this application;

[0040] Figure 3 This is a schematic diagram of the structure of the device control apparatus provided in the embodiments of this application;

[0041] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0042] Explanation of icon numbers:

[0043] 101-Terminal device; 301-Identification module; 302-Control module; 401-Processor; 402-Memory; 403-Power supply; 404-Input unit. Detailed Implementation

[0044] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] This application provides a device control method, apparatus, electronic device, and computer-readable storage medium. Specifically, this application provides a device control apparatus suitable for electronic devices, which include terminal devices or servers. The terminal devices include, but are not limited to, mobile phones, laptops, desktop computers, televisions, or extended reality devices (such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) devices). The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server can be directly or indirectly connected via wired or wireless communication.

[0046] For example, please see Figure 1 , Figure 1 This is a schematic diagram illustrating a scenario where a terminal device, according to an embodiment of this application, executes the device control method. Specifically, the execution process of the terminal device executing the device control method is as follows:

[0047] Terminal device 101 recognizes the gesture data to be recognized through a gesture recognition model, obtains the gesture recognition result, determines the target device command corresponding to the gesture recognition result, and controls the target device according to the target device command.

[0048] The gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition result and the expected gesture recognition result corresponding to the sample gesture recognition result. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the response feedback to the sample gesture recognition result.

[0049] For example, taking AR glasses as an extended reality device as the terminal device, the AR glasses start performing gesture recognition tasks by actively recognizing the scene environment or receiving user commands to open the gesture interaction channel. Then, it collects the user's gesture data, calls the gesture recognition model to recognize the gesture data to obtain the gesture recognition result, determines the target device command corresponding to the gesture recognition result, and controls the target device according to the target device command.

[0050] In summary, the embodiments of this application recognize gesture data through a gesture recognition model, determine the target device command based on the recognized gesture recognition result, and control the target device based on the target device command, thereby realizing gesture-based device control.

[0051] Specifically, by determining the desired gesture recognition result based on the sample gesture recognition result, and then optimizing the model by combining the sample gesture recognition result and the desired gesture recognition result, the accuracy of the model's gesture recognition result is improved, thus enhancing the experience of gesture-based device control.

[0052] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0053] Please see Figure 2 , Figure 2 This is a flowchart illustrating a device control method provided in an embodiment of this application. Although the flowchart shows a logical sequence, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. Specifically, the specific flow of the device control method is as follows:

[0054] 201. The gesture recognition model is used to recognize the gesture data to be recognized, and the gesture recognition result is obtained.

[0055] It is understood that the gesture recognition model is a deep learning model or a machine learning model, which includes, but is not limited to, models trained based on networks such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Spatiotemporal Graph Convolutional Networks (ST-GCN). In this embodiment, the gesture recognition model can be deployed in AR glasses, the cloud, etc.

[0056] Gesture data refers to hand-related data, including finger swiping and wrist bending. This gesture data can be collected by sensors, such as an IMU sensor in a device. In this embodiment, other devices can also be used to collect gesture data, such as sensors with built-in IMUs in watches, bracelets, rings, or mobile phones. The gesture data received from these devices can then be used to obtain the gesture data to be recognized.

[0057] The gesture recognition result refers to the recognition outcome of gesture data. This result includes the gesture type, such as, but not limited to, gestures like clicking, lateral movement, and extending a palm. If the gesture recognition result is empty, it indicates that no gesture was recognized.

[0058] 202. Determine the target device instruction corresponding to the gesture recognition result, and control the target device according to the target device instruction.

[0059] For example, based on the correspondence between gesture recognition results and device commands, the target device command corresponding to the current gesture recognition result can be determined, such as the corresponding confirmation command for clicking, or the corresponding screen drag command for horizontal movement.

[0060] Accordingly, after determining the target device instruction, the target device is controlled according to that instruction. For example, a confirmation instruction can be used to click and confirm controls on the screen, or a drag instruction can be used to drag the screen and change the interface display. Target devices include extended reality devices, smart TVs, or air conditioners, among others.

[0061] In this embodiment of the application, the gesture recognition model is obtained by optimizing the initial recognition model based on the sample gesture recognition result and the expected gesture recognition result corresponding to the sample gesture recognition result. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the response feedback to the sample gesture recognition result.

[0062] In this context, "response feedback" refers to the feedback on the response to the sample gesture recognition result, or the response result corresponding to the output response. For example, this feedback can be collected, captured, or recognized by AR glasses. Response feedback may include no response or a response that does not meet expectations regarding the sample gesture recognition result. For instance, if the user inputs a new device command after the target device has responded, it indicates that the response to the sample gesture recognition result did not meet expectations, meaning the response to the gesture data did not meet expectations.

[0063] For example, the initial recognition model is used to identify sample gesture data to obtain sample gesture recognition results, and the expected gesture recognition results are determined based on the sample gesture recognition results. Then, the initial recognition model is optimized by combining the sample gesture recognition results and the expected gesture recognition results to obtain the gesture recognition model.

[0064] In summary, the embodiments of this application recognize gesture data through a gesture recognition model, determine the target device command based on the recognized gesture recognition result, and control the target device based on the target device command, thereby realizing gesture-based device control.

[0065] Specifically, by determining the desired gesture recognition result based on the sample gesture recognition result, and then optimizing the model by combining the sample gesture recognition result and the desired gesture recognition result, the accuracy of the model's gesture recognition result is improved, thus enhancing the experience of gesture-based device control.

[0066] In this embodiment of the application, depending on whether the initial recognition model recognizes the gesture, the corresponding expected gesture recognition result can be determined. That is, optionally, in some embodiments of this application, before the step "recognizing the gesture data to be recognized through the gesture recognition model to obtain the gesture recognition result", the method further includes:

[0067] The initial recognition model is used to identify the sample gesture data to be identified, and the sample gesture recognition result is obtained. The sample gesture recognition result is determined based on the confidence level of the sample gesture data for each sample recognition result.

[0068] If the sample gesture recognition result indicates that no gesture was recognized, then the sample recognition result with the highest confidence level is set as the expected gesture recognition result;

[0069] If the sample gesture recognition result indicates that a gesture has been recognized, then the target device is controlled according to the sample device instruction corresponding to the sample gesture recognition result. If a sample desired device instruction that meets the preset conditions is detected after controlling the target device, then the sample recognition result corresponding to the sample desired device instruction is set as the desired gesture recognition result.

[0070] Based on the sample gesture data, the sample gesture recognition results, and the expected gesture recognition results, the initial recognition model is optimized to obtain the gesture recognition model.

[0071] It should be noted that the sample recognition result is a preset recognition result, which includes all possible results of gesture recognition. For example, three sample recognition results are preset: extending the palm, clicking, and swiping horizontally. Gesture recognition, on the other hand, calculates the matching degree between the sample gesture data and each sample recognition result, thereby filtering out the sample gesture recognition result from multiple sample recognition results.

[0072] Understandably, the initial recognition model determines the sample gesture recognition result based on the confidence level of the recognition results for each sample gesture data. For example, the sample recognition result with a confidence level higher than 90% is taken as the sample gesture recognition result. Correspondingly, if there is no sample recognition result with a confidence level higher than 90%, the sample gesture recognition result for which the gesture was not recognized is obtained. However, in the embodiments of this application, for the case where the gesture is not recognized, the sample recognition result with the highest confidence level is directly taken as the expected gesture recognition result.

[0073] Furthermore, for cases where a gesture is recognized, the target device is controlled using the sample device command corresponding to that gesture. The desired sample device command is determined based on whether a new device command is generated in response to the target device, thereby determining the desired gesture recognition result. For example, after controlling the target device based on the sample device command, if the target device generates a response to that sample device command and the user is not satisfied with the response, they will re-enter a new device command, such as through voice or a remote control. In this embodiment, the device command entered at this time is taken as the desired sample device command, and the desired gesture recognition result corresponding to the desired sample device command is obtained based on the correspondence between the device command and the gesture recognition result.

[0074] Among them, preset conditions refer to preset conditions, such as time conditions. For example, if a new device instruction different from the sample device instruction is identified within a short period of time after the target device is controlled based on the sample device instruction, then the new device instruction is taken as the sample expected device instruction.

[0075] Therefore, in this embodiment of the application, after obtaining the sample gesture recognition result and the desired gesture recognition result, the initial recognition model can be optimized by combining the sample gesture data to obtain the gesture recognition model.

[0076] The scheme for optimizing the initial recognition model based on sample gesture data is also applicable to optimizing gesture recognition models in practical applications. For example, taking AR glasses as an extended reality device, the AR glasses acquire gesture data to be recognized during use, and the gesture recognition model recognizes the gesture data to obtain the actual gesture recognition result. Then, the expected gesture recognition result is determined based on whether the actual gesture recognition result recognizes the gesture. The model is then automatically optimized based on the actual gesture recognition result and the expected gesture recognition result. For example, if the gesture recognition model is deployed in AR glasses, the gesture recognition model deployed in AR glasses is automatically optimized based on the actual gesture recognition result and the expected gesture recognition result.

[0077] In this embodiment of the application, the process of whether the actual gesture recognition result recognizes the gesture and how to determine the corresponding expected gesture recognition result can be set as an automated process, so that the user can seamlessly optimize the gesture recognition model while using AR glasses.

[0078] In some embodiments of this application, if the target device is found to have not responded for an extended period, it is determined that the gesture has not been recognized. Specifically, in some embodiments of this application, before the step "if the sample gesture recognition result indicates that the gesture has not been recognized, then the sample recognition result with the highest confidence level is taken as the expected gesture recognition result," the method further includes:

[0079] Control the target device based on the sample gesture recognition results;

[0080] If the target device does not respond after a first set time period, the sample gesture recognition result is determined to be that the gesture was not recognized.

[0081] The first set duration refers to the duration for the target device to respond after the user makes a gesture. This duration can be obtained by averaging the durations of multiple gesture recognitions and responses.

[0082] The fact that the target device did not respond indicates that it did not receive instructions from the sample device, and correspondingly, it also indicates that it did not receive the sample gesture recognition result.

[0083] In this embodiment of the application, if, within a certain period of time after controlling the target device based on the sample device instruction, another device instruction different from the sample device instruction is detected, then the other device instruction is directly used as the expected sample device instruction. That is, optionally, in some embodiments of this application, before the step "if, after controlling the target device, another expected sample device instruction that meets the preset conditions is detected, then the sample recognition result corresponding to the expected sample device instruction is used as the expected gesture recognition result", the method further includes:

[0084] If, within a second set time period after controlling the target device according to the sample device instruction, another device instruction different from the sample device instruction is detected, then the other device instruction is determined to be the sample expected device instruction that meets the preset conditions.

[0085] For example, if a user makes a gesture to control the device and then re-enters another device command within a short period of time, it indicates that the user prefers to use that other device command to control the target device.

[0086] The second set duration can be determined based on the time interval between the generation of two sample device commands. For example, if the time interval between two user gesture interactions is approximately t, obtained through sampling statistics, the second set duration can be set to t*y%, where y is greater than 0 and less than 100. y can be set according to actual needs and is not limited here.

[0087] It is understandable that when the expected gesture recognition result is generated, it indicates that the initial recognition model has deviated from the recognition of the sample gesture data and does not meet the user's expectations. Therefore, the optimization of the model can be controlled based on the generation of the expected gesture recognition result. In this embodiment, the gesture recognition model can be obtained by retraining or fine-tuning the initial recognition model. For example, when the number of expected gesture recognition results generated within a first set period (corresponding to the number of times the gesture recognition deviated) reaches or exceeds a threshold, the initial recognition model is retrained based on the sample gesture recognition result and the expected gesture recognition result to obtain the gesture recognition model. That is, optionally, in some embodiments of this application, the step "optimizing the initial recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result to obtain the gesture recognition model" includes:

[0088] The number of desired gesture recognition results generated within the first preset period is counted.

[0089] If the number of generated gestures reaches a certain threshold, the initial recognition model is retrained based on the expected gesture recognition results and the corresponding sample gesture data and sample gesture recognition results to obtain the gesture recognition model.

[0090] The quantity threshold can be preset based on actual needs.

[0091] Accordingly, in the embodiments of this application, the initial recognition model can also be fine-tuned according to a set period to obtain the gesture recognition model. That is, optionally, in some embodiments of this application, the step "optimizing the initial recognition model according to the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result to obtain the gesture recognition model" includes:

[0092] According to the second preset period, the initial recognition model is fine-tuned based on the expected gesture recognition results and the corresponding sample gesture data and sample gesture recognition results within the second preset period to obtain the gesture recognition model.

[0093] For example, the cross-entropy loss is calculated by the difference between the sample gesture recognition result and the expected gesture recognition result, and the model is fine-tuned using this cross-entropy loss to obtain the gesture recognition model.

[0094] The first preset period and the second preset period can also be set according to actual needs. The first preset period and the second preset period can be the same or different. For example, the first preset period and the second preset period can be set to 1 day, 3 days or 1 week, etc.

[0095] In this embodiment of the application, to ensure the accuracy of the generated target device command, the target device command can also be generated based on the device object to be controlled (e.g., the target device). That is, optionally, in some embodiments of this application, the step "determining the target device command corresponding to the gesture recognition result" includes:

[0096] Identify the target equipment to be controlled;

[0097] The target device command matching the target device is determined based on the gesture recognition result.

[0098] For example, based on gesture recognition results, different target device commands are generated for different target devices. For instance, for a lateral movement gesture, a screen swipe command is generated for a user interface device, while a command to increase temperature or fan speed is generated for an air conditioner. This method of generating target device commands based on the target device enhances the diversity and flexibility of gesture control across different devices, improves the accuracy of target device commands, and avoids generating commands that cannot control the target device.

[0099] In summary, the embodiments of this application recognize gesture data through a gesture recognition model, determine the target device command based on the recognized gesture recognition result, and control the target device based on the target device command, thereby realizing gesture-based device control.

[0100] Specifically, by determining the desired gesture recognition result based on the sample gesture recognition result, and then optimizing the model by combining the sample gesture recognition result and the desired gesture recognition result, the accuracy of the model's gesture recognition result is improved, thus enhancing the experience of gesture-based device control.

[0101] Specifically, by detecting the command from another device when the target device does not respond and when it does respond, the system can judge the recognition results of flawed sample gestures. The system can also obtain the expected device command for the sample by using confidence level and the command from another device input by the user, thereby improving the accuracy of the device command.

[0102] The timing of model optimization is controlled by judging the number of units generated and the number threshold within the first preset period, and by fine-tuning the settings within the second preset period.

[0103] To facilitate better implementation of the equipment control method of this application, this application also provides an equipment control device based on the above-described equipment control method. The meanings of the terms used are the same as in the above-described equipment control method, and specific implementation details can be found in the descriptions of the method embodiments.

[0104] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of the device control apparatus provided in the embodiments of this application, wherein the device control apparatus may specifically be as follows:

[0105] The recognition module 301 is used to recognize the gesture data to be recognized through the gesture recognition model and obtain the gesture recognition result;

[0106] The control module 302 is used to determine the target device instruction corresponding to the gesture recognition result, and control the target device according to the target device instruction;

[0107] The gesture recognition model is obtained by optimizing an initial recognition model based on sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results.

[0108] Optionally, in some embodiments of this application, before the gesture recognition model is used to recognize the gesture data to be recognized and obtain the gesture recognition result, the method further includes:

[0109] The initial recognition model is used to identify the sample gesture data to be identified, and the sample gesture recognition result is obtained. The sample gesture recognition result is determined based on the confidence level of the sample gesture data for each sample recognition result.

[0110] If the sample gesture recognition result indicates that no gesture was recognized, then the sample recognition result with the highest confidence level is set as the expected gesture recognition result;

[0111] If the sample gesture recognition result indicates that a gesture has been recognized, then the target device is controlled according to the sample device instruction corresponding to the sample gesture recognition result. If a sample desired device instruction that meets the preset conditions is detected after controlling the target device, then the sample recognition result corresponding to the sample desired device instruction is set as the desired gesture recognition result.

[0112] Based on the sample gesture data, the sample gesture recognition results, and the expected gesture recognition results, the initial recognition model is optimized to obtain the gesture recognition model.

[0113] Optionally, in some embodiments of this application, before taking the sample recognition result with the highest confidence level as the expected gesture recognition result if the sample gesture recognition result indicates that no gesture has been recognized, the method further includes:

[0114] Control the target device based on the sample gesture recognition results;

[0115] If the target device does not respond after a first set time period, the sample gesture recognition result is determined to be that the gesture was not recognized.

[0116] Optionally, in some embodiments of this application, before the step of using the sample recognition result corresponding to the sample expectation device instruction as the expected gesture recognition result if a sample expectation device instruction that meets preset conditions is detected after controlling the target device, the method further includes:

[0117] If, within a second set time period after controlling the target device according to the sample device instruction, another device instruction different from the sample device instruction is detected, then the other device instruction is determined to be the sample expected device instruction that meets the preset conditions.

[0118] Optionally, in some embodiments of this application, optimizing the initial recognition model based on the sample gesture data, the sample gesture recognition result, and the desired gesture recognition result to obtain the gesture recognition model includes:

[0119] The number of desired gesture recognition results generated within the first preset period is counted.

[0120] If the number of generated gestures reaches a certain threshold, the initial recognition model is retrained based on the expected gesture recognition results and the corresponding sample gesture data and sample gesture recognition results to obtain the gesture recognition model.

[0121] Optionally, in some embodiments of this application, optimizing the initial recognition model based on the sample gesture data, the sample gesture recognition result, and the desired gesture recognition result to obtain the gesture recognition model includes:

[0122] According to the second preset period, the initial recognition model is fine-tuned based on the expected gesture recognition results and the corresponding sample gesture data and sample gesture recognition results within the second preset period to obtain the gesture recognition model.

[0123] Optionally, in some embodiments of this application, the step of determining the target device instruction corresponding to the gesture recognition result includes:

[0124] Identify the target equipment to be controlled;

[0125] The target device command matching the target device is determined based on the gesture recognition result.

[0126] In this embodiment, the recognition module 301 first recognizes the gesture data to be recognized using a gesture recognition model to obtain a gesture recognition result. The control module 302 is used to determine the target device command corresponding to the gesture recognition result and control the target device according to the target device command. The gesture recognition model is obtained by optimizing an initial recognition model based on a sample gesture recognition result and the expected gesture recognition result corresponding to the sample gesture recognition result. The sample gesture recognition result is generated based on the initial recognition model, and the expected gesture recognition result is determined based on the response feedback to the sample gesture recognition result.

[0127] In this embodiment of the application, gesture recognition model is used to recognize gesture data, and the target device command is determined based on the recognized gesture recognition result. The target device is then controlled based on the target device command to achieve gesture-based device control.

[0128] Specifically, by determining the desired gesture recognition result based on the sample gesture recognition result, and then optimizing the model by combining the sample gesture recognition result and the desired gesture recognition result, the accuracy of the model's gesture recognition result is improved, thus enhancing the experience of gesture-based device control.

[0129] In addition, this application also provides an electronic device, such as Figure 4 As shown, it illustrates a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically:

[0130] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0131] The processor 401 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0132] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0133] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power equipment debugging circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0134] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0135] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402, thereby implementing the steps in any of the device control methods provided in the embodiments of this application.

[0136] This application embodiment uses a gesture recognition model to recognize the gesture data to be recognized, obtains the gesture recognition result, determines the target device command corresponding to the gesture recognition result, and controls the target device according to the target device command. The gesture recognition model is obtained by optimizing an initial recognition model based on sample gesture recognition results and the expected gesture recognition result corresponding to those sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition result is determined based on the sample gesture recognition results.

[0137] Specifically, gesture recognition data is recognized through a gesture recognition model, and the target device command is determined based on the recognized gesture recognition result. The target device is then controlled based on the target device command, thereby realizing gesture-based device control.

[0138] Specifically, by determining the desired gesture recognition result based on the sample gesture recognition result, and then optimizing the model by combining the sample gesture recognition result and the desired gesture recognition result, the accuracy of the model's gesture recognition result is improved, thus enhancing the experience of gesture-based device control.

[0139] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0140] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0141] Therefore, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps of any of the device control methods provided in this application.

[0142] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0143] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0144] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the device control methods provided in this application, the beneficial effects that any of the device control methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0145] The above provides a detailed description of a device control method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A device control method, characterized in that, The method includes: The gesture recognition model is used to identify the gesture data to be recognized, and the gesture recognition result is obtained. Determine the target device instruction corresponding to the gesture recognition result, and control the target device according to the target device instruction; The gesture recognition model is obtained by optimizing an initial recognition model based on sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results. Before obtaining the gesture recognition result by recognizing the gesture data to be recognized through the gesture recognition model, the method further includes: The initial recognition model is used to identify the sample gesture data to be identified, and the sample gesture recognition result is obtained. The sample gesture recognition result is determined based on the confidence level of the sample gesture data for each sample recognition result. If the sample gesture recognition result indicates that no gesture was recognized, then the sample recognition result with the highest confidence level is set as the expected gesture recognition result; If the sample gesture recognition result indicates that a gesture has been recognized, then the target device is controlled according to the sample device instruction corresponding to the sample gesture recognition result. If a sample desired device instruction that meets the preset conditions is detected after controlling the target device, then the sample recognition result corresponding to the sample desired device instruction is set as the desired gesture recognition result. Based on the sample gesture data, the sample gesture recognition results, and the expected gesture recognition results, the initial recognition model is optimized to obtain the gesture recognition model.

2. The equipment control method according to claim 1, characterized in that, Before using the sample gesture recognition result as the expected gesture recognition result if the sample gesture recognition result indicates that no gesture was recognized, the method further includes: Control the target device based on the sample gesture recognition results; If the target device does not respond after a first set time period, the sample gesture recognition result is determined to be that the gesture was not recognized.

3. The equipment control method according to claim 1, characterized in that, If a sample desired device instruction that meets preset conditions is detected after controlling the target device, before using the sample recognition result corresponding to the sample desired device instruction as the desired gesture recognition result, the method further includes: If, within a second set time period after controlling the target device according to the sample device instruction, another device instruction different from the sample device instruction is detected, then the other device instruction is determined to be the sample expected device instruction that meets the preset conditions.

4. The equipment control method according to claim 1, characterized in that, The step of optimizing the initial recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result to obtain the gesture recognition model includes: The number of desired gesture recognition results generated within the first preset period is counted. If the number of generated gestures reaches a certain threshold, the initial recognition model is retrained based on the expected gesture recognition results and the corresponding sample gesture data and sample gesture recognition results to obtain the gesture recognition model.

5. The equipment control method according to claim 1, characterized in that, The step of optimizing the initial recognition model based on the sample gesture data, the sample gesture recognition result, and the expected gesture recognition result to obtain the gesture recognition model includes: According to the second preset period, the initial recognition model is fine-tuned based on the expected gesture recognition results and the corresponding sample gesture data and sample gesture recognition results within the second preset period to obtain the gesture recognition model.

6. The equipment control method according to claim 1, characterized in that, The instruction for determining the target device corresponding to the gesture recognition result includes: Identify the target equipment to be controlled; The target device command matching the target device is determined based on the gesture recognition result.

7. A device control apparatus, characterized in that, The device includes: The recognition module is used to recognize the gesture data to be recognized through the gesture recognition model and obtain the gesture recognition result; The control module is used to determine the target device instruction corresponding to the gesture recognition result, and control the target device according to the target device instruction; The gesture recognition model is obtained by optimizing an initial recognition model based on sample gesture recognition results and the expected gesture recognition results corresponding to the sample gesture recognition results. The sample gesture recognition results are generated based on the initial recognition model, and the expected gesture recognition results are determined based on the response feedback to the sample gesture recognition results. Before obtaining the gesture recognition result by recognizing the gesture data to be recognized through the gesture recognition model, the device further includes: The initial recognition model is used to identify the sample gesture data to be identified, and the sample gesture recognition result is obtained. The sample gesture recognition result is determined based on the confidence level of the sample gesture data for each sample recognition result. If the sample gesture recognition result indicates that no gesture was recognized, then the sample recognition result with the highest confidence level is set as the expected gesture recognition result; If the sample gesture recognition result indicates that a gesture has been recognized, then the target device is controlled according to the sample device instruction corresponding to the sample gesture recognition result. If a sample desired device instruction that meets the preset conditions is detected after controlling the target device, then the sample recognition result corresponding to the sample desired device instruction is set as the desired gesture recognition result. Based on the sample gesture data, the sample gesture recognition results, and the expected gesture recognition results, the initial recognition model is optimized to obtain the gesture recognition model.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the device control method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the device control method as described in any one of claims 1-6.

Images