Control method and electronic device

Abstract

This paper presents a method, related device, and system for controlling traditional games by replacing remote control buttons with motion sensing, thereby transforming traditional games into motion-sensing games. In this method, an electronic device learns specific motion sensing movements performed by the user and records specific remote control button selections. The electronic device then associates these movements with the buttons. When the user performs the aforementioned movements, the electronic device simulates the control signals generated by the buttons, thereby controlling the game operation. Implementing this method allows users to easily and quickly convert traditional games into motion-sensing games. This process requires no additional customization work from game developers, nor does it require users to use additional motion-sensing devices. Furthermore, users can exercise during gameplay. This method achieves the benefits of exercise and fitness while increasing the fun of exercise, thus enhancing the user experience.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to control methods and electronic devices. Background Technology

[0002] To address the sub-health issues arising from the fast-paced work lifestyles of today's society, many online TV channels have added smart fitness guidance for home workout programs. These programs offer professional fat loss and strength training to help users achieve fitness goals. However, these programs can be quite monotonous, making it difficult for users to stick to them. How to maintain the enjoyment of exercise while ensuring its effectiveness is a pressing issue that current home workout programs need to address. Summary of the Invention

[0003] This application provides a control method and an electronic device that, when implemented, can easily and quickly transform traditional games controlled by a remote control into motion-sensing games.

[0004] In a first aspect, embodiments of this application provide a control method applied to an electronic device. The method includes: the electronic device acquiring a first button signal, the first button signal being a control signal generated by a first button on a remote control; responding to the first button signal, an operating object performing a first action; the electronic device acquiring a first action template, the first action template being used to identify a first haptic action performed by a user; the electronic device associating the first button signal with the first action template; the electronic device acquiring an image sequence containing the user's actions; when the action indicated by the image sequence matches the first action template, the electronic device generating the first button signal; and responding to the first button signal, the operating object performing the first action.

[0005] By implementing the method provided in the first aspect, the electronic device can associate a user-designated remote control button with a specific haptic action performed by the user. Then, when the haptic action is detected, the electronic device can obtain the control signal of the button based on the haptic action, thereby achieving the same control effect as the button.

[0006] In conjunction with some embodiments of the first aspect, in some embodiments, after the electronic device associates the first button signal with the first action template, the method further includes: the electronic device displaying a first interface on which the association relationship between the first action template and the first button is displayed.

[0007] By implementing the method provided in the above embodiments, users can intuitively see which remote control button is associated with which motion sensing action by the electronic device. Furthermore, users can perform the above actions to replace the remote control button.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, before the electronic device acquires the first action template, the method further includes: the electronic device acquiring a first image containing user actions; the electronic device identifying skeletal nodes in the first image to obtain a first set of skeletal node data; after acquiring the first image, the electronic device acquiring a second image containing user actions; the electronic device identifying skeletal nodes in the second image to obtain a second set of skeletal node data; the electronic device calculating the difference between the first set of skeletal node data and the second set of skeletal node data to obtain a threshold range; and the electronic device generating the first action template, the first action template including: the first set of skeletal node data, and / or, the second set of skeletal node data, and the threshold range.

[0009] By implementing the method provided in the above embodiments, the electronic device can learn the specific posture of the user's body during the completion of a certain somatosensory action by acquiring images of the user completing the somatosensory action twice, thereby generating an action template. Using this template, the electronic device can identify whether any action performed by the user is a somatosensory action.

[0010] In conjunction with the above implementation method, in some embodiments, the method for an electronic device to acquire a first image containing user actions specifically includes: the electronic device acquiring a first set of image sequences of the user completing a haptic action; the electronic device determining the image frame in the first set of image sequences with the smallest change in the user's haptic action compared to the previous image frame as the first image; the method for an electronic device to acquire a second image containing user haptic actions specifically includes: the electronic device acquiring a second set of image sequences of the user completing a haptic action; the electronic device determining the image frame in the second set of image sequences with the smallest change in the user's haptic action compared to the previous image frame as the second image.

[0011] By implementing the method provided in the above embodiments, the electronic device can determine the image that best represents a certain somatosensory action from among numerous images of a user performing a certain somatosensory action, using it as learning data for learning that action. Through the above selection process, the electronic device can also greatly reduce computational complexity and save computing resources during the learning and recognition of user somatosensory actions, thereby quickly and efficiently recognizing user somatosensory actions.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, after the electronic device generates the first action template, the method further includes: the electronic device acquiring a third image containing user actions; the electronic device identifying skeletal nodes in the third image to obtain a third set of skeletal node data; the electronic device determining that the third set of skeletal node data matches the first action template; and the electronic device locking the first action template.

[0013] By implementing the method provided in the above embodiments, after learning a certain somatosensory movement and obtaining a movement template for that movement, the electronic device can also verify whether the movement template is correct. In this way, the electronic device can promptly detect whether there are any problems with its learning results.

[0014] In conjunction with the above implementation method, in some embodiments, the method by which the electronic device determines that the third set of skeletal node data matches the first action template specifically includes: the electronic device calculating the difference between the third set of skeletal node data and the first set of skeletal node data; when the difference is within the threshold range, the electronic device determines that the third set of skeletal node data matches the first action template.

[0015] By implementing the method provided in the above embodiments, when calculating whether a user's haptic action matches the action template, the electronic device can perform only one comparison, that is, select an image that best represents the user's haptic action and match it with the action template. This greatly reduces the computational complexity and saves computing resources, thereby enabling rapid and efficient recognition of the user's haptic actions.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: the electronic device storing the first action template as a preset action template.

[0017] By implementing the method provided in the above embodiments, after an electronic device successfully learns a certain motion-sensing action, it does not need to immediately pair it with remote control buttons, but can pair it only when needed. This allows users to separate the process of guiding the electronic device to learn motion-sensing actions from the process of pairing with remote control buttons, thereby increasing the flexibility of the learning and pairing process.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, the electronic device acquiring the first action template further includes: the electronic device displaying a second interface, the second interface displaying a plurality of preset action templates for selection; the electronic device selecting the first action template from the plurality of preset action templates.

[0019] By implementing the method provided in the above embodiments, users can know which motion-sensing actions the electronic device has learned through the displayed content. Therefore, users can directly use the existing motion templates of the electronic device to match the remote control buttons, thereby omitting the process of the electronic device learning motion-sensing actions and improving the user experience.

[0020] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: the electronic device learning multiple user actions of the user and storing them as multiple preset action templates, and / or the electronic device acquiring multiple shared preset action templates.

[0021] By implementing the methods provided in the above embodiments, the motion simulations acquired by the electronic device are not limited to local learning results. The electronic device can also acquire shared learning templates from other sources.

[0022] In conjunction with some embodiments of the first aspect, in some embodiments, the method for an electronic device to select the first action template from the plurality of preset action templates specifically includes: when it is recognized that an action performed by a user matches the first action template, the electronic device selects the first action template from the plurality of preset action templates.

[0023] By implementing the method provided in the above embodiments, users can select a template by performing the motion-sensing actions displayed in the preset templates. In this way, users can achieve the purpose of selection and also verify whether the electronic device can accurately and quickly recognize their actions.

[0024] In conjunction with some embodiments of the first aspect, in some embodiments, after the electronic device acquires the image sequence containing the user's actions, the method further includes: the electronic device acquiring a fourth image from the image sequence; the electronic device identifying skeletal nodes in the fourth image to obtain a fourth set of skeletal node data; the electronic device calculating the difference between the fourth set of skeletal node data and the first set of skeletal node data; and when the difference is within a threshold range, the electronic device identifying that the action indicated by the image sequence matches the first action template.

[0025] By implementing the method provided in the above embodiments, the electronic device can determine the image that best represents a certain haptic action from among numerous images of a user performing a haptic action, using it as learning data for learning that haptic action. Then, the image is matched with an action template to identify whether the user's action is a learned action that matches a remote control button press. In this way, the electronic device can significantly reduce computational complexity, save computing resources, and thus quickly and efficiently recognize the user's haptic actions.

[0026] In a second aspect, embodiments of this application provide an electronic device including one or more processors and one or more memories; wherein the one or more memories are coupled to one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by one or more processors, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0027] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0028] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0029] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0030] It is understood that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer program product provided in the fourth aspect, and the computer storage medium provided in the fifth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0031] Figure 1 This is a system diagram of an application scenario provided in an embodiment of this application;

[0032] Figure 2A This is a software framework diagram of an electronic device 100 provided in an embodiment of this application;

[0033] Figure 2B This is a hardware structure diagram of an electronic device 100 provided in an embodiment of this application;

[0034] Figures 3A-3I , Figures 4A-4D , Figures 5A-5C This is a set of user interfaces provided in the embodiments of this application;

[0035] Figure 6A This is a flowchart illustrating a motion-sensing motion matching remote control button control for a game, as provided in an embodiment of this application.

[0036] Figure 6B This is a flowchart illustrating how an electronic device 100 learns user motion sensing actions, as provided in an embodiment of this application.

[0037] Figures 6C-6E This is a schematic diagram of a set of skeletal nodes according to an embodiment of this application;

[0038] Figure 7 This is a flowchart of another motion-sensing motion matching remote control button control game provided in this application embodiment. Detailed Implementation

[0039] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0040] The term "user interface (UI)" used in the specification, claims, and drawings of this application refers to the medium through which an application or operating system interacts and exchanges information with the user. It converts the internal form of information into a form that the user can receive. The user interface of an application is source code written in a specific computer language such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content, such as images, text, buttons, and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined through tags or nodes, such as in XML. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a webpage, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (GTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a webpage is also defined through tags or nodes in the webpage source code; for example, GTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>To define the elements and attributes of a webpage.

[0041] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0042] Motion-sensing games can combine exercise and gaming. Users can achieve their goal of exercising at home through motion-sensing games, and they are highly entertaining. Therefore, exercise based on motion-sensing games can not only meet users' needs for exercising at home, but also enhance the fun of family exercise, enabling users to stick to it in the long term.

[0043] However, the number of existing motion-sensing games is relatively small, especially those readily available in the online TV market. Developing new motion-sensing games is extremely costly, often requiring custom customization by game developers. Some customized motion-sensing games can run directly on existing online TV hardware, while others require specialized motion-sensing devices. Therefore, this limits the widespread adoption of motion-sensing games.

[0044] To address the challenges of customizing motion-sensing games, lower the barrier to entry for motion-sensing games, enable users with internet TVs to experience motion-sensing games, and facilitate daily exercise through motion-sensing games, this invention provides a method, related device, and system for controlling traditional games by replacing remote control commands with motion-sensing actions, thereby transforming traditional games into motion-sensing games.

[0045] This method involves electronic devices (such as smart TVs) and remote controls. The electronic devices can capture images via a camera. By implementing this method, the electronic devices can recognize specific motion-sensing actions performed by the user and match these actions with specific remote control buttons selected by the user, thereby simulating the remote control button controls for game operations.

[0046] By implementing the above methods, traditional games (such as parkour games, Tetris, Sokoban, etc.) can be easily and quickly transformed into motion-sensing games. This process requires no additional customization work from game developers, nor does it require users to use additional motion-sensing devices. Furthermore, users can exercise while playing the game. Implementing these methods achieves the benefits of exercise and fitness while increasing the fun of exercise, enabling users to stick with it long-term.

[0047] The system 10 provided in the embodiments of this application will be introduced first below. Figure 1 The structure of system 10 is illustrated. As shown, system 10 includes: electronic device 100 and remote controller 200.

[0048] Electronic device 100 is a large-screen electronic device. Electronic device 100 includes, but is not limited to, internet TVs, home projectors, and customized interactive gaming screens. Exemplary embodiments of electronic device 100 include, but are not limited to, devices equipped with… A portable electronic device operating system such as Linux or other operating systems. The electronic device 100 has a camera 400. The camera 400 can be a camera fixed to the electronic device 100, or a camera that is connected to the electronic device 100 via wired or wireless connection, such as a smart home camera.

[0049] The camera 400 can capture real-time images and send them to the electronic device 100. When the camera 400 establishes a connection with the electronic device 100 via a wired or wireless connection, the camera 400 can send the captured real-time images to the electronic device 100 via a wired or wireless network. The wired connection is, for example, a data cable connecting the electronic device 100 and the camera 400. The wireless connection is, for example, a high-fidelity wireless communication (Wi-Fi) connection, a Bluetooth connection, an infrared connection, an NFC connection, a ZigBee connection, etc., and this application embodiment does not limit this.

[0050] Remote control 200 can be a physical remote control or a remote control application. A physical remote control refers to a physical remote control, such as the TV remote control shown in the figure. A remote control application refers to an application installed on a smart terminal that can control specific devices, such as various air conditioner remote control applications provided in the app store. This embodiment uses a physical remote control as an example to introduce a method of replacing remote control buttons with motion sensing.

[0051] Remote controller 200 can send control signals to electronic device 100. In response to a user operation on any remote controller button, remote controller 300 can generate a corresponding control signal for that button. Through a wireless transmitter, remote controller 200 can send these control signals to the target device (i.e., electronic device 100). Electronic device 100 has a remote controller command receiving module. This module can receive and recognize the control signals generated by remote controller 200. In response to these control signals, electronic device 100 can execute the operations indicated by the remote controller buttons, such as moving up, moving down, confirming selection, etc.

[0052] The remote control 200 can also be a remote control application installed on a smart terminal such as a mobile phone or tablet. In this case, the remote control 200 can wirelessly transmit the generated control signals to the target device. In some embodiments, the smart terminal with the remote control application installed also has an infrared transmission function, so the remote control 200 can transmit the generated control signals to the electronic device 100 via an infrared connection, which will not be elaborated further here.

[0053] In this system, electronic device 100 can acquire skeletal node data of user 300 performing motion-sensing movements via camera 400. Based on this data, electronic device 100 can recognize specific motion-sensing movements of the user.

[0054] Electronic device 100 can associate button operations of remote control 200 with specific motion-sensing actions of user 300. When a specific motion-sensing action of user 300 is detected, electronic device 100 can retrieve the remote control button associated with that action. Then, electronic device 100 can simulate the remote control button to control the game application running on electronic device 100.

[0055] The following is a software framework diagram of the electronic device 100 provided in the embodiments of this application. Figure 2A An exemplary software framework structure of electronic device 100 is shown, wherein camera 400 is a fixed component of electronic device 100. For example... Figure 2A As shown, the electronic device 100 may include a game module 201, a motion-sensing learning module 202, and a camera 400.

[0056] The game module 201 may include a signal receiving module 203 and a game response module 204.

[0057] The signal receiving module 203 can be used to receive control signals for controlling game operations. When motion sensing mode is not enabled, this module can receive control signals sent by the remote control 200. When motion sensing mode is enabled, this module can receive control signals sent by the electronic device 100 simulating the remote control 200. That is, when using the remote control to control game operations, the signal receiving module 203 can receive control signals sent by the remote control. After the electronic device 100 learns specific motion sensing actions for controlling game operations, the electronic device 100 can recognize these motion sensing actions and then simulate the remote control 200 sending control signals to the signal receiving module 203.

[0058] For example, in a parkour game, when a motion-sensing action corresponding to the "up button" on the remote control 200 is detected, the electronic device 100 can generate a control signal (analog signal) simulating the "up button". The signal receiving module 203 can receive the aforementioned analog signal.

[0059] In response to various control signals received by the signal receiving module 203, the game response module 204 can control the game subject to perform corresponding actions. For example, in response to the "up button" control signal received by the signal receiving module 203, the parkour player in the game can perform an upward jump. The aforementioned "up button" control signal can be generated by the remote control 200, or it can be generated by the electronic device 100 simulating the remote control 200, that is, generated by the electronic device 100 described in the foregoing embodiment after recognizing the motion-sensing action corresponding to the "up button" of the remote control 200.

[0060] The motion-sensing learning module 202 may include a remote control signal module 205, a data input module 206, a learning module 207, and a motion matching module 208. In addition, the electronic device 100 also includes a camera 400 and a skeletal node recognition module 209.

[0061] The remote control signal module 205 has two capabilities: first, it records the control signals of specific remote control buttons sent by the remote control 200; second, it simulates the aforementioned control signals.

[0062] Specifically, before or after the electronic device 100 acquires the user's motion data, the remote control signal module 205 can register the control signals of specific remote control buttons received from the remote control 200. After the electronic device 100 completes learning of a user's motion data, it can associate the registered control signals with the motion. That is, the electronic device 100 can record the control signals of specific remote control buttons selected by the user through this module.

[0063] After completing motion sensing learning and matching the control signals of the remote control buttons, the remote control signal module 205 can simulate the control signals generated by the remote control buttons to control the game. Specifically, after detecting a specific motion sensing action of the user, based on the matching relationship between the motion sensing action and the control signals of the remote control buttons, the electronic device 100 can query the control signals of the remote control buttons associated with the motion sensing action. Furthermore, the remote control signal module 205 can simulate the control signals to control the game operation.

[0064] For example, the remote control signal module 205 can record the control signal of the "up button" on the remote control 200. In response to a user operation that matches the "up button" with the motion-sensing action "jump," the electronic device 100 can associate the control signal of the "up button" with the motion-sensing action "jump." When the electronic device 100 detects a user performing a "jump" motion, it can retrieve the control signal of the "up button" on the remote control that matches the action. Then, the electronic device 100 can simulate the control signal (i.e., the analog signal) to issue control commands to the game. This analog signal achieves the same control effect as the control signal generated by the remote control's "up button," thereby achieving the purpose of controlling the game operation.

[0065] The data input module 206 can receive and store the user's skeletal node data. This data is obtained from the skeletal node recognition module 209. This skeletal node data includes data acquired in two periods: one during motion-sensing motion learning, and the other during gameplay. When the electronic device 100 is learning the user's motion-sensing motion, the data input module 206 can send the acquired data to the learning module 207. During gameplay, the data input module 206 can send the acquired data to the motion matching module 208.

[0066] The learning module 207 can learn from the received skeletal node data based on a motion-sensing learning algorithm and generate a motion template. This motion template is a set of data obtained by the electronic device 100 from learning the user's motion-sensing actions, including the user's skeletal nodes and threshold ranges when performing the action. This motion template can be used by the electronic device 100 to recognize the user's actions. For example, the electronic device 100 can learn the skeletal node data when the user performs a "jump" action to obtain a motion template for the motion-sensing action "jump." This template can then serve as a reference for the electronic device 100 to determine whether the user's action is a "jump" action.

[0067] The aforementioned threshold range can be used to measure whether the motion-sensing action being compared is similar to the action template. Specifically, when the difference between the skeletal node data of the action being compared and the data in the action template is within the range indicated by the threshold range, the electronic device 100 considers the action being compared to be the same action as indicated by the action template. This difference can be calculated by comparing the skeletal node data of the action being compared with the skeletal node data in the action template. Conversely, when the difference is outside the range indicated by the threshold range, the electronic device 100 considers the action performed by the user to be different from the action in the action template.

[0068] The motion matching module 208 can be used to compare whether a user's motion-sensing actions are similar to an action template. Specifically, the motion matching module 208 can obtain the user's skeletal node data during gameplay from the data input module 206. Then, the motion matching module 208 can calculate the distance between the skeletal node data and the aforementioned action template. When the distance is within a threshold range, the motion matching module 208 can determine that the motion-sensing action is similar to the action template. Furthermore, the motion matching module 208 can instruct the remote control signal module 205 to generate an analog signal, such as an analog remote control "up" button signal. Conversely, when the distance is not within the threshold range, the electronic device 100 cannot recognize the motion-sensing action performed by the user.

[0069] The camera 400 can capture images of the user performing a motion-sensing action and process these images into image data in a specific format that can be recognized by the skeletal node recognition module 209. The skeletal node recognition module 209 can extract the user's skeletal node data from the images sent by the camera 400. Furthermore, this module can send the data to the motion-sensing learning module 202 of the electronic device 100. Specifically, the data input module 206 in the motion-sensing learning module 202 receives and stores the skeletal node data.

[0070] In other embodiments, the skeletal node recognition module 209 may also be disposed in the camera 400, and this application does not limit this.

[0071] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer modules than illustrated, or combine some modules, or split some modules, or have different module arrangements.

[0072] In other embodiments, camera 400 may also be a camera that is connected to electronic device 100 via wired or wireless network. Therefore, Figure 2A The electronic device 100 and camera 400 shown can also be independent modules. The electronic device 100 and camera 400 can establish a connection and exchange information through a communication channel.

[0073] Figure 2B The hardware structure of electronic device 100 is illustrated as an example. For example... Figure 2B As shown, the electronic device 100 may include a processor 211, a memory 212, a wireless communication processing module 213, a power switch 214, a display screen 215, an audio module 216, a speaker 217, an infrared receiver 218, and a camera 219.

[0074] The processor 211 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0075] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0076] The processor 211 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 211 is a cache memory. This memory can store instructions or data that the processor 211 has just used or that are used repeatedly. If the processor 211 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 211, and thus improves the efficiency of the system.

[0077] Memory 212 is coupled to processor 211 and is used to store various software programs and / or multiple sets of instructions. Memory 212 can be used to store computer executable program code, which includes instructions. Processor 211 executes various functional applications and data processing of electronic device 100 by running the instructions stored in memory 212. Memory 212 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, image data to be displayed, etc.). In addition, memory 212 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0078] The wireless communication module 213 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0079] In some embodiments, the wireless communication processing module 213 may include a Bluetooth (BT) communication processing module 213A, a WLAN communication processing module 213B, and an infrared communication processing module 213C. One or more of the Bluetooth (BT) communication processing module 213A and the WLAN communication processing module 213B can listen to signals emitted by other devices, such as probe requests, scan signals, etc., and can send response signals, such as probe responses, scan responses, etc., enabling other devices to discover the electronic device 100 and establish wireless communication connections with other devices, communicating with other devices through one or more wireless communication technologies, including Bluetooth or WLAN. The Bluetooth (BT) communication processing module 213A can provide one or more Bluetooth communication solutions, including Classic Bluetooth (BR / EDR) or Bluetooth Low Energy (BLE). The WLAN communication processing module 213B may include one or more WLAN communication solutions, including Wi-Fi Direct, Wi-Fi LAN, or Wi-Fi SoftAP.

[0080] The power switch 214 can be used to control the power supply to the electronic device 100. In some embodiments, the power switch 214 can be used to control the power supply to the electronic device 100 from an external power source.

[0081] Display screen 215 can be used to display images, videos, etc. Display screen 215 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc.

[0082] Infrared receiver 218 can be used to receive infrared signals. For example, infrared receiver 218 can receive graphic infrared signals sent by a remote control device, such as circular infrared signals, ring infrared signals, cross-shaped infrared signals, etc.

[0083] The audio module 216 can be used to convert digital audio signals into analog audio signals for output, and can also be used to convert analog audio input into digital audio signals. The audio module 216 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 216 can be located in the processor 211, or some functional modules of the audio module 216 can be located in the processor 211. The audio module 216 can transmit audio signals to the wireless communication module 213 via a bus interface (e.g., a UART interface, etc.) to enable the playback of audio signals through a Bluetooth speaker.

[0084] Speaker 217 can be used to convert the audio signal sent by audio module 216 into a sound signal.

[0085] In some embodiments, the electronic device 100 may also include a microphone, also known as a "microphone" or "voice transducer," for converting sound signals into electrical signals. When a voice control command is given, the user can speak to input the sound signal into the microphone.

[0086] Camera 219 can be used to capture still images or videos. An object passes through a lens, generating an optical image that is projected onto a photosensitive element. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an image signal processor (ISP) to be converted into a digital image signal. The ISP outputs the digital image signal to a DSP for further processing. The DSP converts the digital image signal into standard RGB, YUV, or other image signal formats.

[0087] In some embodiments, the electronic device 100 may further include a wired local area network (LAN) communication processing module, a high-definition multimedia interface (HDMI) communication processing module, and a universal serial bus (USB) communication processing module. The wired LAN communication processing module can be used to communicate with other devices within a LAN via a wired LAN, and can also be used to connect to a WAN via a wired LAN to communicate with devices within the WAN. The HDMI communication processing module can be used to communicate with other devices via an HDMI interface. For example, the HDMI communication processing module can receive HDR video data sent by a set-top box via the HDMI interface, and so on. The USB communication processing module can be used to communicate with other devices via a USB interface.

[0088] In this embodiment, the infrared receiver 218 can receive infrared signals sent by the remote controller 200 (i.e., control signals from the remote controller 200 to control the operation of the electronic device 100). Further, the infrared communication processing module 213C processes the infrared signals received by the infrared receiver 218. The processor 211 can control the motion-sensing learning algorithm, motion-sensing game, and game animation processing process based on the control information included in the infrared signals. The aforementioned motion-sensing learning algorithm and motion-sensing game application can be stored in the memory 212.

[0089] During the process of receiving a specific remote control button selected by the user, the infrared receiver 218 and the infrared communication processing module 213C can record the control signal of the selected remote control button. In other remote control processes, in response to other operations of the user controlling the electronic device 100, the above two modules can control the electronic device 100 to display other user interfaces, such as displaying all the games installed on the electronic device 100, playing videos, etc.

[0090] In other embodiments provided in this application, when the remote controller 200 is a remote controller application installed on a smart terminal such as a mobile phone or tablet, the electronic device 100 can receive and process the control signals sent by the remote controller 200 through the Bluetooth (BT) communication processing module 213A and the WLAN communication processing module 213B.

[0091] Camera 219 can acquire images or videos of the user performing motion-sensing actions. The graphics processing unit (GPU) and / or neural network processor (NPU) of processor 211 can run a skeletal node recognition algorithm to obtain skeletal node data of the user's motion-sensing actions in the images or videos. In some embodiments, camera 219 may also be an external camera connected to electronic device 100 via a network.

[0092] During motion learning, one or more application processors (APs) of processor 211 can run motion learning algorithms to generate motion templates. During gameplay, one or more application processors (APs) of processor 211 can run motion learning algorithms to recognize the user's motion movements, and then match the control signals of the remote control buttons to simulate the aforementioned control signals, thereby controlling the game operation.

[0093] Based on the above method, users can control game operations by performing specific motion-sensing actions.

[0094] The following describes some user interfaces provided in the embodiments of this application. Figures 3A-3I An example is shown of a set of user interfaces for an electronic device 100 to learn user-defined motion-sensing movements. Figures 4A-4D An example is shown of an electronic device 100 that controls a game by recognizing user-defined motion-sensing movements.

[0095] Figure 3A An example is shown of an electronic device 100 displaying a user interface 31 of an installed game in game mode. The user interface 31 may include some option controls and motion-sensing settings controls. Wherein:

[0096] Options controls can be used to display the game's cover image. Users can learn about the game's content through the cover. For example... Figure 3A Option control 311 can indicate a parkour game. One option control corresponds to one game. Depending on the number of games installed, one or more option controls can be displayed in user interface 31. Taking the three games shown in the figure as an example, user interface 31 also includes option controls 312 and 313.

[0097] Users can browse all games installed on electronic device 100 using a remote control. In other embodiments, users can also search for desired games via voice input. Electronic device 100 can detect user actions on a particular option control, and in response to the action, electronic device 100 can display the game interface for that game.

[0098] In some embodiments, the user interface 31 may also include a control 315. The control 315 can be used by the electronic device 100 to access more games.

[0099] The motion-sensing control 314 can be used to display a motion-sensing settings page. The motion-sensing settings page can be used to display the motion-sensing actions and matching relationships learned by the electronic device 100. When the electronic device 100 detects a user operation applied to the motion-sensing control 314, in response to that operation, the electronic device 100 can display, as shown below... Figure 3B The motion-sensing settings page shown.

[0100] It is understood that the user interface 31 is an exemplary game page of an electronic device 100 and should not constitute a limitation on the embodiments of this application.

[0101] Figure 3B An exemplary embodiment is shown of a user interface 32 of an electronic device 100 for displaying learned motion sensing actions and matching relationships. The user interface 32 may include a switch control 321, a region 322, an add control 323, and a return control 325.

[0102] The switch control 321 can be used to indicate whether the electronic device 100 has enabled the motion control mode. As shown in the figure, the "ON" symbol displayed by the switch control 321 indicates that the electronic device 100 has enabled the motion control mode. In motion control mode, when entering any game, the electronic device 100 can recognize the user's current body movements according to the matching relationship indicated in area 322, thereby controlling the game. The electronic device 100 can detect user operations applied to the switch control 321. In response to this operation, the switch control 321 can display the "OFF" symbol, thereby exiting the motion control mode. After exiting the motion control mode, the electronic device 100 will control the game operation by receiving control signals sent by the remote control 200.

[0103] Area 322 can be used to display the motion-sensing actions learned by electronic device 100 and the corresponding remote control operations for each motion-sensing action. As shown in the figure, the motion-sensing actions learned by electronic device 100 include "jump up," "crouch down," and "right punch," where "jump up," "crouch down," and "right punch" correspond to the "up," "down," and "left" buttons on the remote control, respectively. For example, in a parkour game, when electronic device 100 detects that the user has completed the "jump up" action, electronic device 100 can simulate the "up" button on the remote control to control the parkour player in the game to jump upwards.

[0104] When the switch control indicates that motion control mode is enabled, the matching relationship between the remote control buttons indicated in area 322 and the motion sensing actions can be applied to any game displayed in interface 31. After entering the game, the electronic device 100 can activate the camera 400 to acquire the user's motion sensing actions, and then control the game operation.

[0105] In other embodiments, the above-described motion-sensing setting process can also be completed after entering a game. That is, when the user enters a game, the electronic device 100 can display... Figure 3B The user interface 32 described above allows users to configure the matching relationship between remote control buttons and motion sensing actions in the game.

[0106] In other embodiments, the electronic device 100 may display simultaneously before and after entering the game. Figure 3B The user interface described above allows users to configure the matching relationship between remote control buttons and motion sensing actions before entering the game. After entering the game, users can also adjust the aforementioned matching relationships through this interface. This application does not impose any limitations on this.

[0107] The return control 325 can be used to close the user interface 32. In response to a user action applied to the return control 325, the electronic device 100 can display a user interface 31, etc. The add control 323 can be used by the electronic device 100 to learn new motion-sensing actions. When a user action is detected applied to the add control 323, in response to that action, the electronic device 100 can display, etc. Figure 3C The user interface shown is 33.

[0108] In some embodiments, the user interface 32 may further include a playback control 324. The playback control 324 can be used to display various motion-sensing movements learned by the electronic device 100. Users can learn about the specific body shapes of each motion-sensing movement through the aforementioned videos.

[0109] The user interface 33 may include a dialog box 331. The dialog box 331 can be used to record the names of the remote control buttons selected by the user and the motion-sensing actions that the electronic device 100 is about to learn. The names can be used to label the motion-sensing actions. For example, the user can enter the name "left fist" in window 332 in the dialog box 331, and then select to match the motion-sensing action indicated by the name "left fist" to the "right button" of the remote control buttons.

[0110] In some embodiments, the user can use a scrolling control to select the remote control button to match the aforementioned motion-sensing action, see reference. Figure 3C The scrolling control 333 is included. In other embodiments, the electronic device 100 may also directly record the user's next remote control button operation after the user completes the name input, and use the button used in that button operation as the remote control button for the aforementioned motion-sensing action matching. This application does not limit this.

[0111] In some embodiments, the electronic device 100 may first learn motion sensing movements and then receive remote control button selections from the user. This application does not limit this approach.

[0112] Dialog box 331 also includes a cancel control 334 and a confirm control 335. Upon detecting a user action on the cancel control 334, the electronic device 100 may display, in response to that action, the following: Figure 3B The user interface 32 shown is the close dialog box 331. Upon detecting a user action on the confirmation control 335, the electronic device 100 can display, in response to that action, the following... Figure 3D The user interface shown is 34.

[0113] Figure 3D An exemplary user interface 34 is shown for an electronic device 100 to acquire a first image of a user-defined motion-sensing action. As shown, the user interface 34 may include a window 341, a prompt window 342, a window 343, and a prompt window 345.

[0114] Window 341 can be used to display the real-time image captured by camera 300. Window 343 can be used to display the first image of a custom motion-sensing action locked by electronic device 100. The aforementioned first image is an image of the user when relatively still while performing a certain motion-sensing action. For example, when the user performs... Figure 3D During the left fist movement, the user generates a series of limb postures, and finally, the movement is frozen in the posture shown in window 341 for a relatively long time. At this time, the image containing the above-mentioned limb postures can be referred to as the first image. Then, the electronic device 100 can display the above-mentioned first image in window 343. In other embodiments, the electronic device 100 can also display the first image containing the above-mentioned skeletal nodes in window 343 after recognizing the skeletal nodes in the first image. Alternatively, in another optional manner, the electronic device 100 can also directly display the skeletal nodes obtained by recognizing the above-mentioned first image in window 343. This application does not limit this.

[0115] The determination of the first image mentioned above can be accomplished through stillness detection. The stillness detection process will be described in detail in subsequent embodiments, and will not be repeated here. In some embodiments, the first image displayed in window 343 may also simultaneously display the user's skeletal nodes.

[0116] Specifically, when the electronic device 100 enters the motion-sensing learning state, i.e., when the user interface 34 is displayed, the electronic device 100 can activate the camera 400 to capture real-time images. The electronic device 100 can then display the real-time images captured by the camera 400 in window 341.

[0117] Simultaneously, the electronic device 100 can display a prompt window 345 to prompt the user to complete a motion-sensing action in front of the electronic device 100. Then, the electronic device 100 can detect whether the motion-sensing action in window 341 is in a static state. When a static state is detected, the electronic device 100 can lock the image displayed in window 341 in window 343. The motion-sensing action indicated in the locked image (first image) is the specific motion-sensing action recognized by the electronic device 100. The skeletal nodes in this motion-sensing action are the first set of skeletal node data (first set of data) acquired by the electronic device 100.

[0118] After locking the first image, the electronic device 100 may also display a symbol 344 to indicate to the user that it has been recognized. In some embodiments, the symbol 344 may also be text, such as "recognized" or "passed". This application does not limit this.

[0119] While displaying the indicator 344, the electronic device 100 may also display a prompt window 342. The prompt window 342 can be used to notify the user that the first image acquisition of the aforementioned motion-sensing action has been completed. After displaying the prompt window 342 for a period of time, the electronic device 100 may display... Figure 3E The user interface 35 is shown. The aforementioned time period can be a time interval such as 3 seconds or 7 seconds. In some embodiments, a dedicated control, such as a confirmation control, can also be set in the prompt window 342. In response to a user operation applied to the aforementioned control, the electronic device 100 can display... Figure 3E The user interface shown is 35.

[0120] Figure 3E An exemplary user interface 35 is shown for an electronic device 100 to acquire a second image of a user-defined motion-sensing action. The user interface 35 may include a window 351, a prompt window 352, a window 353, and a prompt window 354.

[0121] After the electronic device 100 completes the acquisition of the first image, the electronic device 100 can display a prompt window 354. The prompt window 354 can prompt the user to rotate their body at a certain angle so that they are facing the electronic device 100 from the side, and then complete the aforementioned haptic action again.

[0122] Similarly, the electronic device 100 can detect whether the motion-sensing action in window 351 is in a static state. When a static state is detected, the electronic device 100 can lock the image displayed in window 351 in window 353. At this time, the locked image is the second image of the aforementioned motion-sensing action acquired by the electronic device 100. The skeletal nodes of the user completing the motion-sensing action obtained by recognizing the aforementioned second image are the second set of skeletal node data (second set of data) acquired by the electronic device 100.

[0123] During the acquisition of the second image, the electronic device 100 can verify the angle of the user's body rotation. That is, the user does not need to manually control the angle of rotation. The electronic device 100 can determine whether the user's body rotation angle is too large based on the skeletal node data in the second image after the user's rotation. Specifically, after acquiring the second image, the electronic device 100 can first identify the key limbs indicated in the second image. When the key limbs change, the electronic device 100 considers the second image incorrect. Furthermore, the electronic device 100 can prompt the user that the second image acquisition was unsuccessful and instruct the user to rotate their body again and complete the aforementioned actions to reacquire the second image.

[0124] For example, when the user rotates their body at a 90° angle, the recognition effect of the same action is completely different. In this case, the electronic device 100 recognizes that the user's left and right legs, and the left and right buttons, are basically overlapping. Therefore, the electronic device 100 can conclude that the user's body rotation range is too large. The aforementioned key limbs refer to the limbs that play a major distinguishing role in recognizing a certain action. Subsequent embodiments will describe the key limbs in detail, and will not be repeated here.

[0125] After the second image is locked, the electronic device 100 may display a checkmark symbol and a prompt window 352. The prompt window 352 may indicate to the user that the second image has been recorded.

[0126] Similarly, after displaying prompt window 352, electronic device 100 can display... Figure 3F The user interface 36 shown may include window 363, window 362, prompt window 363, and prompt window 364.

[0127] The prompt window 363 can prompt the user to face the electronic device 100 and complete the aforementioned motion-sensing action again. During the process of the user completing the above action again, window 361 can display the image captured by the camera 400 in real time. Similarly, when the motion-sensing action in window 361 is detected to be in a static state, the electronic device 100 can lock the image displayed in window 361 in window 362 at this time.

[0128] At this time, the image locked in window 362 is the third image of the motion-sensing action acquired by electronic device 100. The skeletal nodes in this motion-sensing action are the third set of skeletal node data (third set of data) acquired by electronic device 100.

[0129] When the electronic device 100 determines that the motion-sensing action in the third image is similar to the action in the first image, it can display a prompt window 364 to indicate to the user that the learning of the motion-sensing action has been completed. Then, the electronic device 100 can display... Figure 3H The user interface 38 is shown. Conversely, when it is determined that the motion-sensing action in the third image is not similar to the action in the first image, the electronic device 100 can display... Figure 3G The user interface 37 is shown. The user interface 37 can indicate to the user that the third action performed is significantly different from the previous two learned actions, and the electronic device 100 cannot recognize the third action performed by the user.

[0130] In some embodiments, the electronic device 100 may also directly instruct the user to face the screen and complete three motion-sensing actions. From these three completions of the motion-sensing actions, the electronic device 100 can acquire three sets of skeletal node data, thereby learning the aforementioned actions.

[0131] Figure 3G The device may include a window 371 and a prompt window 372. The prompt window 372 may indicate to the user that the action is incorrect and prompt the user to re-enter a second image, referring to the user interface 35. In some embodiments, the prompt window 372 may also prompt the user to re-enter a third image. When the re-entered image is similar to the first image, the electronic device 100 also prompts the user that the learning of the motion-sensing action has been completed.

[0132] like Figure 3H As shown, the user interface 38 may include controls 381 and 382. Control 381 can display... Figures 3D-3F In the middle, the electronic device 100 learns the name of the motion-sensing action. Control 382 can display the name of the remote control button that replaces the above-mentioned motion-sensing action.

[0133] When electronic device 100 passes Figures 3D-3F During the process of learning a new motion-sensing action, the user interface 38 can display the name of the motion-sensing action learned by the electronic device 100, and display the name of the remote control button that matches the motion-sensing action name after it, thereby indicating the substitution relationship between the two to the user.

[0134] In some embodiments, the electronic device 100 may also support adjusting the matching relationship between learned motion-sensing actions and remote control buttons. For example, the electronic device 100 may set a selection button after the motion-sensing action name, such as control 383. When an operation is detected on the user control 383, the electronic device 100 may display all learned motion-sensing actions, such as... Figure 3I The window 391 is shown in the diagram. The user can then select any remote control button from the learned motion-matching control 382. For example, the user can select the "squat" motion-matching remote control right button.

[0135] Window 391 may also include control 392. Control 392 can be used by electronic device 100 to learn new motion-sensing actions and match those actions to remote control buttons represented by control 382. For example, in response to a user action on control 392, electronic device 100 may display... Figures 3D-3F The user interface shown above. By implementing the method shown in the user interface above, the electronic device 100 can learn new motion-sensing actions. After the learning process is completed, the electronic device 100 can add a new action to control 392 and match the action to the remote control button represented by control 382. Specifically, the electronic device 100 associates the learning template obtained from learning the action with the control signal of the aforementioned remote control button.

[0136] Similarly, electronic device 100 can also set a selection button after the remote control button name, such as control 384. Thus, users can also choose to change the remote control buttons that match motion sensing.

[0137] Generally, the correspondence between motion sensing actions and remote control buttons is one-to-one; that is, one motion sensing action corresponds to only one remote control button, and one remote control button is replaced by only one motion sensing action. In some embodiments, the electronic device 100 may also support multiple motion sensing actions matching one remote control button. However, one motion sensing action cannot replace multiple remote control buttons.

[0138] Electronic device 100 can detect user actions performed on return control 385, and in response to such actions, electronic device 100 can display user interface 31. Then, when a user action performed on a game option is detected, electronic device 100 can display the game interface for that game. For example, when a user action performed on control 314 is detected, electronic device 100 can display... Figure 4A The user interface 41 shown.

[0139] Figure 4A An example is shown of a user interface 41 in a parkour game before the game starts. User interface 41 may include a start control 411. Upon detecting a user action on the start control 411, the electronic device 100 may display... Figure 4B The user interface shown is shown. The user interface 41 may also include some game setting controls, such as controls 412 for selecting the game map, controls 412 for purchasing game items, controls 414 for viewing the game character's equipment, etc., which will not be described in detail here.

[0140] Figures 4B-4C An example is shown of a set of user interfaces in a parkour game where an electronic device 100 recognizes the user's motion-sensing actions to control the game operation.

[0141] like Figure 4B As shown, the user interface 42 may include runway 1, runway 2, runway 3, operation object 4, and window 422. Window 422 may display the user's body shape as captured by the camera 400 by the current electronic device 100.

[0142] At a later point, the electronic device 100 can acquire a specific motion-sensing action from the user. Then, the electronic device 100 can locate the remote control button associated with the aforementioned motion-sensing action. Furthermore, the electronic device 100 can simulate the button press to control the game's target to produce a corresponding action. As shown in window 432 of the user interface 43, the electronic device 100 can acquire the user's "left fist" motion. At this time, the electronic device 100 can recognize this motion as a learned "left fist" motion. Based on the association between the "left fist" motion and the right button on the remote control, the electronic device 100 can simulate the right button to control the game's target 4 to move to the right, that is, from track 2 in the user interface 42 to track 3 in the user interface 43.

[0143] In some embodiments, the electronic device 100 may also include motion-sensing settings controls on the game start page, for example... Figure 4D The control 441 is shown. In response to a user action on the control 441, the electronic device 100 can display a motion-sensing settings page, such as... Figure 3B Then, the user can view the motion-sensing actions and matching relationships learned by the electronic device 100 in the motion-sensing settings, and instruct the electronic device 100 to learn new motion-sensing actions, etc. For details, please refer to [link / reference needed]. Figures 3B-3I .

[0144] This application embodiment also provides another method for controlling game operations using motion sensing instead of remote control buttons. In this method, the electronic device 100 can first learn the user's custom motion sensing actions and generate action templates. Then, when the user needs to establish a matching relationship between motion sensing actions and remote control buttons, the electronic device 100 can display the aforementioned action templates for the user to select. The learned action templates that have not yet established a matching relationship with a specific remote control button are preset templates. During the selection process, the user can select a preset template by performing a motion sensing action in it, and then establish a matching relationship between that motion sensing action and a specific remote control button. In the game, when the electronic device 100 detects that the user has performed an action in the aforementioned preset template, it can match the aforementioned remote control button, thereby controlling the game.

[0145] Figures 5A-5C This is a set of user interfaces provided in this application embodiment for implementing the above method. Specifically, when a user operation on the added control 323 of the user interface 32 is detected, the electronic device 100 can also display, in response to the operation, a user interface 32. Figure 5A The user interface 51 shown.

[0146] User interface 51 may include dialog box 511. Dialog box 511 may record the remote control button selected by the user to be replaced. In response to user operation on selection control 512, electronic device 100 may display scroll control 513. Scroll control 513 may sequentially display the buttons of the remote control. The user may select one of the buttons as the remote control button to be replaced. Similarly, electronic device 100 may also support other methods of selecting a specific remote control button, see reference. Figure 3C The details of the introduction will not be repeated here.

[0147] After a user selects a remote control button, such as the left button, the user can confirm the option using the confirmation control 514. The electronic device 100 can detect the user's selected remote control button and the confirmation operation; in response to this operation, the electronic device 100 can display... Figure 5B The user interface shown is 52.

[0148] Figure 5B An exemplary user interface of an electronic device 100 is shown, in which the user selects a preset template for a motion sensing action. As shown, the user interface 52 may include preset templates 521, 522, and 523, a window 524, and a prompt window 525.

[0149] Preset templates 521, 522, and 523 are all pre-set motion-sensing templates for electronic device 100. These preset templates can display motion-sensing movements that electronic device 100 has already learned. That is, electronic device 100 uses them as references. Figures 3A-3I The user's motion-sensing movements learned during the process can also be saved as motion templates. When it is necessary to match remote control buttons, the electronic device 100 can call upon the aforementioned templates. It is understood that in specific practices, the user interface 52 may display more or fewer preset templates, and this embodiment does not impose any limitations on this. Window 524 may display images captured by camera 300. Prompt window 525 may be used to prompt the user on how to complete the operation of selecting a preset template.

[0150] First, the user can learn which motion-sensing actions are included in the preset templates 521, 522, and 523. Then, the user can complete any one of the many templates according to the prompts in the prompt window 525. The camera 400 can capture images of the user completing the motion-sensing action in real time and send these images to the electronic device 100. The electronic device 100 can display these images in window 524. At the same time, the electronic device 100 can identify which motion-sensing action in which template the user has completed by calculating the difference between the motion-sensing action indicated in the image and the action in the preset template. For example, the electronic device 100 identifies that the user has completed the "stepping" action in preset template 522. When the identification is successful, the electronic device 100 can display mark 526 in window 524 and mark 527 in preset template 522.

[0151] At this point, the electronic device 100 completes the process of receiving the user's selected motion-sensing preset template. Then, the electronic device 100 can display... Figure 5C The user interface shown is 53.

[0152] In the user interface 53, area 531 can display the name of the preset template selected by the user, and area 532 can display the name of the remote control button selected by the user. The "equal to" symbol 533 indicates the correspondence between the two.

[0153] The user can then exit the motion-sensing settings page via the return control 534. In response to the user's action on the return control 534, the electronic device 100 can display... Figure 3A The user interface 31 is shown. The user can then select any game. In this game, the electronic device 100 can recognize the user's motion-sensing movements, match these movements to preset simulations, and further match these movements to replaced remote control buttons through the matching relationships in the motion-sensing settings, thereby controlling the game operation.

[0154] Similarly, in other embodiments, the electronic device 100 may first receive a preset template of the user-selected motion sensing action, and then the electronic device 100 may receive the user-selected remote control button. This application embodiment does not limit this.

[0155] In other embodiments, the preset templates displayed by the electronic device 100 can also be downloaded from the Internet, meaning the electronic device 100 can also acquire motion-sensing templates shared by other electronic devices. Specifically, after the electronic device 100 learns and generates a motion-sensing template, the user can also send the motion-sensing action to the Internet to share with other users. Therefore, the user of the electronic device 100 can download motion templates uploaded by other users from the Internet as preset templates. Then, the electronic device 100 can associate the above templates with specific remote control buttons. During gameplay, the electronic device 100 can recognize the actions in the above templates, query the remote control buttons corresponding to the action templates, and then simulate the above buttons to control the objects in the game to produce corresponding actions.

[0156] In some embodiments, while recognizing user gestures and controlling game operations, the electronic device 100 may also simultaneously support remote control 300 to control the game. This application does not impose any limitations on this.

[0157] Figures 3A-3I , Figures 4A-4D , Figures 5A-5C This paper introduces a user interface system where an electronic device 100 recognizes user motion sensing actions and uses these actions to control games, replacing remote control buttons. By implementing this method, users can select specific motion sensing actions to replace remote control button operations according to their preferences. These actions can be either arbitrarily performed by the user or preset actions by the electronic device 100. Users can then control game operations through these motion sensing actions. This control process transforms traditional games controlled by remote control buttons into motion-sensing games without the need for specific motion sensing peripherals.

[0158] By implementing this method, users can exercise while playing games, thereby achieving fitness and maintaining health. Furthermore, the types and number of motion-sensing games modified by the method provided by this invention are very rich, allowing users to frequently switch games, thus satisfying their need for novelty and encouraging them to stick to exercise long-term.

[0159] Below, this application embodiment will describe the process by which the electronic device 100 learns user motion sensing actions and replaces remote control operation, in conjunction with the above-described user interface. Figure 6A An example is shown of the process by which an electronic device 100 learns motion sensing actions and replaces remote control operations.

[0160] S101 electronic device 100 enters learning mode.

[0161] Referring to user interface 32, when electronic device 100 detects a user operation that adds control 323, it displays user interface 33 in response to the operation. At this time, electronic device 100 enters a mode for learning user motion sensing. In this mode, electronic device 100 can receive control signals from remote control buttons selected by the user, recognize and learn the user's motion sensing, and match the control signals from the remote control buttons with the motion sensing to establish a correspondence between the two.

[0162] S102 electronic device 100 acquires the control signal generated by the remote control button selected by the user.

[0163] Upon entering learning mode, the electronic device 100 first receives control signals from a specific remote control button selected by the user. This remote control button is the one subsequently replaced by the learned motion-sensing action. Referring to the user interface 33, the electronic device 100 can detect the user's operation of selecting a remote control button through the scroll control 333. In response to this operation, the electronic device 100 can record the control signal of the specific remote control button selected by the user. For example, in response to the user clicking the "right button," the remote control 200 sends a "right button" control signal to the electronic device 100.

[0164] The S103 electronic device 100 learns user-defined motion-sensing movements and generates motion templates.

[0165] After recording the control signals of the remote control buttons selected by the user, the electronic device 100 can access the camera 400. Through the camera 400, the electronic device 100 can acquire motion data including the user's skeletal nodes. First, the electronic device 100 can obtain a set of skeletal node data (first set of data) of a certain motion action performed by the user facing the screen. Then, the electronic device 100 can prompt the user to change the angle facing the screen and repeat the above motion action. Thus, the electronic device 100 can obtain another set of skeletal node data (second set of data) of the above motion action performed by the user at a different angle.

[0166] Using the two sets of data mentioned above, the electronic device 100 can generate a motion template. This motion template may include the skeletal node features and threshold ranges of the aforementioned somatosensory movements. It is understood that the motion template may also include other parameters or indicators to make the motion template generated by the electronic device 100 more accurate.

[0167] The aforementioned skeletal node features include the coordinates of the skeletal nodes and the limb vector formed by the skeletal nodes. The limb vector includes its magnitude and the angle between the vector and the Y-axis. With the ground as the horizontal plane, the positive direction of the Y-axis is perpendicular to the horizontal plane and upwards, i.e., the direction from the feet perpendicular to the horizontal plane towards the head is the positive direction of the Y-axis. The threshold interval indicates the receiving range of the electronic device 100. When the distance between any set of skeletal node data and the first set of data is within the receiving range, the electronic device 100 considers the two sets of data to indicate similar somatosensory actions, which are the same action; otherwise, the two sets of actions are dissimilar and are different actions. Specifically, how the electronic device 100 learns somatosensory actions and calculates the threshold interval will be described in detail in the subsequent somatosensory learning algorithm section of this application embodiment, and will not be repeated here.

[0168] Taking the example of the electronic device 100 learning the motion-sensing action "left fist" shown in user interfaces 34 and 35, the electronic device 100 obtains a first set of skeletal node data for "left fist" through a first recorded image. Then, the electronic device 100 obtains a second set of skeletal node data for "left fist" through a second recorded image. The electronic device 100 can then calculate the distance between the second set of data and the first set of data. This distance indicates the threshold range for the electronic device 100 to recognize "left fist". The two sets of skeletal node data and the threshold range together form a "left fist" template. This template can guide the electronic device 100 to recognize whether the user's motion-sensing action in the game is a "left fist" action.

[0169] S104 Electronic Device 100 Lock Action Template.

[0170] After generating the motion template using the above method, the electronic device 100 can also ask the user to perform the above-mentioned motion-sensing action again in front of the camera 400 of the electronic device 100, so that the electronic device 100 can obtain the third set of skeletal node data (third set of data) of the above-mentioned action. The third set of data can be used to confirm and lock the motion template.

[0171] Specifically, after the electronic device 100 obtains the third set of data, it can calculate the distance between the third set of data and the first set of data. Then, the electronic device 100 can compare this distance with a threshold range in the aforementioned motion template. When the distance is within the threshold range, the electronic device 100 considers the motion-sensing action indicated by the third set of data to be consistent with the action indicated by the aforementioned motion template, i.e., the two are the same action. Simultaneously, the electronic device 100 considers the user to be able to perform this motion-sensing action normally. Therefore, the electronic device 100 can lock the aforementioned motion template, that is, lock the skeletal node features and threshold range of the motion-sensing action indicated by the motion template.

[0172] If the distance between the third set of data and the first set of data is not within the threshold range of the aforementioned action template, the electronic device 100 will not be able to recognize the motion-sensing action indicated by the third set of data. Furthermore, the electronic device 100 may also assume that the user cannot complete the action normally. Therefore, the electronic device 100 cannot lock the action template.

[0173] Therefore, the electronic device 100 can relearn the action, for example, by reacquiring the second set of data and then recalculating the threshold range. In other embodiments, the electronic device 100 may also require the user to reconfirm the action to ensure that the user can complete the action normally, ultimately enabling the electronic device 100 to recognize the motion-sensing action indicated by the third set of data based on the action template, thereby locking the action template.

[0174] Referring to user interface 36, electronic device 100 can prompt the user to perform the motion-sensing action "left fist" again while facing the screen via prompt window 363. Electronic device 100 can then obtain the third set of skeletal node data for "left fist". Next, electronic device 100 can calculate the distance between the third set of data and the first set of data obtained from user interface 34. When the distance is within the threshold range of the "left fist" template, electronic device 100 can recognize the motion indicated by the third set of data as "left fist" and display prompt window 364 to prompt the user that the "left fist" template is locked. When the distance is not within the threshold range of the "left fist" template, referring to user interface 37, electronic device 100 can display prompt window 372, prompting the user that electronic device 100 needs to relearn the "left fist" action.

[0175] S105 electronic device 100 determines the matching relationship between control signals and action templates.

[0176] Once the motion template is locked, the electronic device 100 can associate the motion template with the control signal of the remote control button selected by the user. As shown in the user interface 38, after locking the "left fist" template, that is, after learning the "left fist" motion, the electronic device 100 can display control 381 in the "Learned Motion Sensing Motions" area. Control 381 can indicate the motion sensing motion "left fist" learned by the electronic device 100. At the same time, the electronic device 100 can display control 382 after control 381. Control 382 can instruct the user to select the remote control button "right". Thus, the user knows that the motion sensing motion "left fist" can replace the remote control button "right".

[0177] S106 enters the game, and electronic device 100 acquires images containing the user's body movements.

[0178] After the electronic device 100 completes the learning of motion sensing actions and the matching of control signals with the remote control buttons, the electronic device 100 can detect the user's action of selecting to start a certain game. In response to this action, the electronic device 100 can display the game interface. Simultaneously, the electronic device 100 can activate the camera 400. The camera 400 can acquire images containing the user's motion sensing actions. The electronic device 100 can display these images.

[0179] In some embodiments, the electronic device 100 may also display the skeletal nodes identified by the skeletal node recognition module while displaying the above-described image.

[0180] The S107 electronic device 100 matches the user's motion-sensing movements with learned motion templates.

[0181] The camera 400 can send the aforementioned images to the electronic device 100. The electronic device 100 can identify the user's skeletal nodes through the aforementioned images. Furthermore, the electronic device 100 can compare the aforementioned skeletal nodes with the motion templates of all learned somatosensory movements, thereby identifying the somatosensory movements indicated by the aforementioned skeletal node data.

[0182] Combination Figure 2A The software structure diagram of the electronic device 100 shown illustrates that after acquiring an image containing the user's body movements, the camera 400 can send the image to the skeletal node recognition module 209. The skeletal node recognition module 209 can calculate the skeletal node data of the user's body movements in the image. If the image is a two-dimensional image, the module can obtain the two-dimensional data of the user's skeletal nodes from the image. If the image is a three-dimensional image with depth data, the module can obtain the three-dimensional data of the user's skeletal nodes from the image.

[0183] Then, the skeletal node recognition module 209 can send the recognized skeletal node data to the data input module 206. At this time, the data input module 206 will first determine the source scenario of the above data. When the data input module 206 determines that the above data comes from the camera 400 during the game, the data input module 206 can send the above data to the motion matching module 208.

[0184] The motion matching module 208 can extract skeletal node features from the above data. Furthermore, the motion matching module 208 can calculate the distance between these features and the motion template of the learned somatosensory movements. When this distance is within a threshold range of a certain motion template, the motion matching module 208 can recognize the somatosensory movement indicated by the above data. This movement is the somatosensory movement indicated by the aforementioned motion template.

[0185] Referring to user interface 43, window 432 can display an image captured by camera 400 containing the user's body movements. This image may also include skeletal nodes identified by the skeletal node recognition module. Electronic device 100 can recognize the body movements in the image as a "left fist" motion.

[0186] The S108 electronic device 100 simulates remote control buttons that match the action template to control the game.

[0187] Combination Figure 2A The software structure diagram of the electronic device 100 shown shows that after the motion matching module 208 recognizes the user's motion, the remote control signal module 205 can obtain the replacement information of the above-mentioned motion based on the matching relationship between the motion and the control signal of the remote control button. That is, the electronic device 100 can know which remote control button's control signal is replaced by the above-mentioned motion.

[0188] The remote control signal module 205 can simulate the aforementioned control signals to control the game. Specifically, the remote control signal module 205 can send analog signals to the game module 201. In response to the analog signals, the signals can control the characters in the game to perform corresponding actions.

[0189] Referring to user interface 43, when the electronic device 100 detects that the user has made a "left fist" gesture, the electronic device 100 can, based on the matching relationship between the "left fist" gesture and the right button on the remote control, such as... Figure 3H As shown, control the game character 4 to move to the right. In response to the right-moving control, the game character 4 moves from track 2 to track 3.

[0190] Then, the electronic device 100 can recognize the user's second motion-sensing action and simulate the control signal of the remote control button in place of the second motion-sensing action to control the game. This process is repeated until the game ends.

[0191] The process of learning user-defined motion-sensing movements by the electronic device 100 described in S103 will be explained in detail below. Figure 6B The specific calculation steps of the above process are illustrated by example.

[0192] first, Figure 6C An exemplary set of skeletal node data in a standard standing posture is shown, including 15 skeletal nodes and 14 limb vectors, where each limb vector can be calculated from the coordinate positions of the aforementioned 15 skeletal nodes. Table 1 exemplarily shows the 14 limbs in the first set of data.

[0193] Table 1

[0194] < / canvas> < / video> serial number name serial number name A neck H trunk B right shoulder I Right hip C Right upper arm J Right thigh D Right forearm K Right calf E left shoulder L Left hip F left upper arm M Left thigh G left forearm N left calf

[0195] The aforementioned skeletal nodes are derived from two-dimensional images captured by camera 400.

[0196] In some embodiments, the camera 400 may also have the capability to acquire depth images, thus enabling the camera 400 to obtain depth images of the user performing a motion-sensing action. Furthermore, the electronic device 100 can obtain three-dimensional skeletal node data of this motion-sensing action. The process of obtaining three-dimensional data by referring to two-dimensional data is not elaborated further in this embodiment.

[0197] It is understood that the skeletal node recognition module 209 can also recognize more skeletal nodes, and this application embodiment does not limit this.

[0198] Skeletal node recognition algorithms include similarity metrics. Similarity metrics are used by electronic devices to determine whether a recognized somatosensory action is similar to a learned action. The similarity metric includes a threshold, or minimum similarity. When the calculated similarity between two sets of skeletal node data is lower than the minimum similarity, the somatosensory actions indicated by the two sets of data are considered different actions. Conversely, the somatosensory actions indicated by the two sets of data are considered the same action.

[0199] Similarity can be calculated using limb weights and the distance between the limb being calculated and the limbs in the action template. The calculation process is as follows:

[0200] Similarity = Weight of limb 1 * Distance 1 + Weight of limb 2 * Distance 2 + ... + Weight of limb 15 * Distance 15.

[0201] It is understood that the above calculation process is one possible example, and the embodiments of this application do not limit it.

[0202] S201 electronic device 100 acquires the first set of skeletal node data for custom motion sensing actions.

[0203] When entering learning mode, the electronic device 100 can instruct the user to perform a customized motion-sensing action while facing the screen. The camera 400 can acquire a series of image frames showing the user completing the action. The electronic device 100 can detect when the user has completed the motion-sensing action through stillness detection, and then use the image in the still state as the first image. The skeletal node data in the first image is the first set of skeletal node data (i.e., the first set of data). Figure 6D The first set of data is shown as an example.

[0204] The aforementioned stillness detection can be achieved by comparing several consecutive frames of images, where the user's skeletal node data is essentially consistent across those frames. At this point, the electronic device 100 can determine that the user has completed the motion-sensing action.

[0205] In other embodiments, the electronic device 100 may also acquire a series of image frames of the user completing a motion-sensing action as the first set of skeletal node data, and this application does not limit this.

[0206] S202 electronic devices 100 screen key limbs and increase the weight of similarity calculation for key limbs.

[0207] Electronic device 100 can calculate the similarity between the limb vectors of the first set of data and the skeletal node data in the standard standing posture. Based on the similarity, electronic device 100 can filter out key limbs in the first set of data.

[0208] For example, comparison Figure 6C , Figure 6D The leg, torso, and right arm movements in the image are highly similar to their corresponding limbs in the standard standing posture. Therefore, these limbs are not considered key limbs. However, the left upper arm and left forearm differ significantly from their corresponding limbs in the standard standing posture; therefore, the left upper arm and left forearm can be considered key limbs. Figure 6D The first set of data shown represents the key limbs, while the rest represent non-key limbs.

[0209] After identifying key limbs, the electronic device 100 can correspondingly increase the weight of the key limbs while decreasing the weight of non-key limbs. For example, when the similarity between non-key limbs is 90%, the weight of the non-key limbs can be reduced by 10%, i.e., the new weight = 1 - similarity; when the similarity between key limbs is -70%, the weight of the key limbs can be increased to 170%. It is understood that the above adjustment strategy is only an exemplary demonstration, and the electronic device 100 may also adopt other methods to adjust the weight of key limbs. Therefore, the above adjustment strategy should not constitute a limitation on the embodiments of this application.

[0210] The S203 electronic device 100 acquires the second skeletal node data of the custom motion sensing action.

[0211] After acquiring the user's first set of data, the electronic device 100 can instruct the user to change the angle facing the screen and then perform the aforementioned motion-sensing action again. After passing the stillness detection, the electronic device 100 can obtain a second image of the aforementioned action. The skeletal node data in the second image is the second set of skeletal node data (i.e., the second set of data).

[0212] After acquiring the second set of data, the electronic device 100 can extract the key limbs from the second set of data, referring to the method in S202. If the key limbs in the second set of data are inconsistent with the key limbs in the first set of data, the electronic device 100 can consider the second set of data to be incorrect. Therefore, the electronic device 100 can prompt the user to readjust the angle facing the camera 400 and complete the above action. Furthermore, the electronic device 100 can reacquire the second image and the second set of data.

[0213] When the key limb in the second set of data matches the key limb in the first set of data, the electronic device 100 can perform the following step: calculate the threshold range of the above-mentioned somatosensory action using the first set of data and the second set of data.

[0214] Figure 6E The example illustrates the second set of data acquired by camera 400 after the user rotates their body. The user's body rotation causes a significant difference between the second set of data acquired by electronic device 100 and the first set of data. Therefore, when electronic device 100 learns the threshold range for this motion-sensing action, the threshold range it obtains will be larger. This reduces the requirements for electronic device 100 to recognize the user's actions during the game, thus providing the user with a better gaming experience.

[0215] Optionally, in other embodiments, the second image acquired by the electronic device 100 may also be completed by the user facing the screen, that is, the user faces the screen and repeats a certain motion-sensing action three times.

[0216] S204 Electronic Device 100 Calculation Threshold Range.

[0217] Based on the limb vectors in the first set of data, the electronic device 100 can calculate the angle between each limb vector and the Y-axis. Similarly, the electronic device 100 can calculate the angle between each limb vector in the second set of data and the Y-axis. For any limb vector, the combination of angles from the two sets of data constitutes the threshold range for that vector.

[0218] For example, with Figure 6D and Figure 6E Taking the left forearm G as an example, the angle between the left forearm G and the Y-axis calculated using the first set of data is 75°. The angle between the left forearm G and the Y-axis calculated using the second set of data is 60°. Therefore, the threshold range for the angle between the left forearm G and the Y-axis is (60°, 75°).

[0219] The S205 electronic device 100 acquires the third set of skeletal node data for custom motion sensing movements.

[0220] After calculating the threshold range, the electronic device 100 can ask the user to perform the aforementioned motion-sensing action again, thereby obtaining a third image of the action. The third image may include a third set of skeletal node data (the third set of data).

[0221] The S206 electronic device 100 identifies the third set of data and locks the action template.

[0222] Based on the skeletal node data in the third set of data, the electronic device 100 can calculate each limb vector, and further obtain the angle between each limb vector and the Y-axis. Then, the angle is compared with the threshold range of each limb vector calculated in S204.

[0223] If the angle between each limb vector in the third set of data and the Y-axis is within the threshold range of that limb vector, then the similarity M between the third set of data and the first set of data is further calculated, referring to the aforementioned similarity calculation process. When the similarity M is higher than the minimum similarity, the electronic device 100 can determine that the somatosensory action indicated by the third set of data is similar to the somatosensory action indicated by the first set of data.

[0224] Therefore, the electronic device 100 can consider the learning results of the aforementioned two sets of data to be correct, that is, the threshold range determined by the first and second sets of data is appropriate. Furthermore, the electronic device 100 can confirm that the user can normally complete the somatosensory action, rather than an accidental action that is difficult to repeat. Thus, the electronic device 100 can determine the threshold range of the somatosensory action learned in the aforementioned process, i.e., lock the learning template.

[0225] Conversely, the electronic device 100 may determine that the motion-sensing action indicated by the third set of data is dissimilar to the motion-sensing action indicated by the first set of data. Therefore, the electronic device 100 can reacquire the second set of data to adjust the threshold range in the action template and improve its recognition ability. In other embodiments, the electronic device 100 may also reacquire the third set of data to correct the user's actions and improve the readiness of the user's actions.

[0226] Once the motion template is locked, the electronic device 100 completes the motion learning process. During gameplay, the electronic device 100 can use the locked template as a basis to recognize the user's motion movements, and then match them with remote control buttons to control the game.

[0227] In an optional implementation, the electronic device 100 may first save the learned motion-sensing templates as preset templates. Then, when needed, the user can select remote control buttons and preset templates, and associate the templates with the control signals of the remote control buttons. The process by which the electronic device 100 learns the user's motion-sensing movements and replaces the remote control buttons in the game when using the preset motion templates can be found in [reference needed]. Figure 7 .

[0228] like Figure 7 As shown, after the electronic device 100 enters learning mode, it can learn the user's customized motion-sensing movements and generate motion templates. For details, please refer to [reference needed]. Figure 6A The descriptions of S103 and S104 shown will not be repeated here.

[0229] Then, before the user opens a game, or after opening a game but before starting the game, the electronic device 100 can display a motion-sensing configuration page, see reference. Figure 5A At this time, the electronic device 100 can receive control signals from remote control buttons selected by the user, such as the "right button" control signal shown in the user interface 51.

[0230] Then, the electronic device 100 can display the learned action templates, i.e., preset templates. Examples include preset templates 521, 522, 523, etc., in the user interface 52. In response to a user operation applied to a preset template, the electronic device 100 can establish a matching relationship between the motion-sensing action indicated by that template and the control signal of the previously selected remote control button. Referring to the user interface 52, the user's operation of selecting a motion-sensing action can be to complete an action from a preset template.

[0231] The electronic device 100 can detect when a user completes a motion-sensing action in a preset template. At this time, the electronic device 100 can acquire image frames detected by stillness detection when the user completes the motion-sensing action. Furthermore, the electronic device 100 can obtain skeletal node data of the user during the completion of the action. By calculating the difference between the above data and the skeletal node data in the preset template, the electronic device 100 can identify which preset template's motion-sensing action the user has completed based on the relationship between the above difference and the threshold range of each preset template. Then, the electronic device 100 can associate the preset template with the control signal of the previously selected remote control button to determine the matching relationship between the two. For example, when it detects that the user has completed the action indicated by the preset template "step," the electronic device 100 can associate the preset template "step" with the control signal of the remote control "right button" selected in the aforementioned process.

[0232] By implementing the above method for selecting preset templates, the electronic device 100 can not only determine what preset template the user wants to select, but also detect whether the user's motion movements are standard during the selection process. For example, when the user wants to select the stepping motion indicated by preset template 522, the electronic device 100 can detect the skeletal node data of the user completing the stepping motion. If the electronic device 100 cannot recognize the motion, it means that the user's execution of the motion is not standard. Therefore, the electronic device 100 can prompt the user to complete the motion more correctly. This is more conducive to the user achieving the goal of exercising and fitness while playing the game.

[0233] In other embodiments, the electronic device 100 can also select a preset template by means of a remote control, which will not be described in detail in this application embodiment.

[0234] Then, after matching the motion-sensing action indicated by the preset template with the control signals of the remote control buttons, the user can control the game operation through motion-sensing actions. During the game, when the electronic device 100 recognizes the motion-sensing action indicated by the preset template, it can query the specific remote control button control signal that the action represents based on the matching relationship between the motion-sensing action and the control signals of the remote control buttons. Then, the electronic device 100 can simulate the control signals to control the objects in the game to produce corresponding actions.

[0235] Specifically, the electronic device 100 can acquire the user's skeletal node data during the game. This data comes from images of the user's body shape captured by the camera 400. Based on the aforementioned skeletal node data, the electronic device 100 can identify which preset template's motion-sensing action the user performed. Then, the electronic device 100 can query the control signal of the remote control button matching that motion-sensing action. Furthermore, the electronic device 100 can simulate the aforementioned control signal to control the game operation.

[0236] In the embodiments of this application:

[0237] In method S102 described above, the control signal generated by the electronic device receiving a specific remote control button selected by the user can be referred to as a first button signal. This specific remote control button is the first button, for example... Figure 3C The "right-click" option is shown.

[0238] In the above method S107, the character in the game can be referred to as the manipulated object, for example... Figure 4B The operation object 4 in the game. The change that the character undergoes can be called the first action, such as the action of operation object 4 jumping from track 2 to track 3.

[0239] In the above method S103, the specific action performed by the user in front of the electronic device can be referred to as the first somatosensory action. The action template generated by the electronic device learning the user's first somatosensory action can be referred to as the first action template, such as the "left fist" action template obtained by learning the user's "left fist" action.

[0240] The remote control buttons and motion-sensing gestures displayed on electronic devices can be referred to as the first interface, for example... Figure 3H The user interface shown is 38.

[0241] In method S103 described above, the sequence of image frames acquired by the electronic device after the user completes a haptic action can be referred to as the first set of image sequences. The image that passes stillness detection and is determined by the electronic device from the first set of image sequences can be referred to as the first image, such as the image displayed in window 343 of the user interface 34. The skeletal node data obtained by the electronic device from recognizing the user's skeletal nodes in the first image can be referred to as the first set of skeletal node data.

[0242] The sequence of image frames acquired by the electronic device after the user completes a haptic action can be referred to as the second set of image frame sequences. The image that passes stillness detection and is determined by the electronic device from the second set of image frame sequences can be referred to as the second image, such as the image displayed in window 353 in user interface 35. The skeletal node data obtained by the electronic device from recognizing the user's skeletal nodes in the second image can be referred to as the second set of skeletal node data.

[0243] As mentioned above, the image that passes stillness detection in the image frame sequence obtained by the electronic device during the third acquisition of the user's haptic movements can be referred to as the third image. For example, the image displayed in window 363 in user interface 36. The skeletal node data obtained by the electronic device from recognizing the user's skeletal nodes in the third image can be referred to as the third set of skeletal node data.

[0244] Pre-stored motion templates that have already been learned by electronic devices can be called preset motion templates, such as the multiple motion templates displayed in the user interface 52.

[0245] In the above method S107, the image collected by the electronic device during the game, including the user's completion of a specific somatosensory action, can be called the fourth image, and the skeletal node data obtained by the electronic device from recognizing the user's skeletal nodes in the fourth image can be called the fourth set of skeletal node data.

[0246] Implementation Figure 7 The illustrated electronic device 100 learns user motion-sensing movements and replaces remote control buttons in a game. The electronic device 100 can separate the learning process from the process of establishing a matching relationship with the control signals of the remote control buttons. The electronic device 100 can learn multiple users' motion-sensing movements in advance and store the corresponding movement templates as preset templates. Therefore, the user does not need to determine the corresponding remote control button when learning the motion-sensing movement.

[0247] Before starting a game, users can associate the learned motion-sensing actions with remote control buttons, allowing them to select motion-sensing actions according to the needs of different games.

[0248] During the motion-sensing learning process, the electronic device 100 can learn custom movements through the motion-sensing learning module. Therefore, users can choose specific motion-sensing movements to replace remote control button operations according to their preferences. The motion-sensing template generation method used in this learning module improves the robustness of motion-sensing matching by imitating and transforming templates from multiple angles and calculating threshold ranges for each limb, thus enabling the machine to accurately recognize the user's motion-sensing movements.

[0249] Furthermore, by implementing the above method during gameplay, users can easily and quickly convert traditional games controlled by remote control buttons into motion-sensing games controlled by motion sensors. This conversion process requires no specific motion-sensing peripherals and no custom game development. By converting traditional games into motion-sensing games, users can exercise while playing, achieving fitness and maintaining health. In addition, the types and number of motion-sensing games modified by the method provided by this invention are very rich, allowing users to frequently change games, thus satisfying their need for novelty and encouraging long-term exercise adherence.

[0250] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if..." or "after..." or "in response to determining..." or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is detected" can be interpreted as meaning "if determining..." or "in response to determining..." or "when (the stated condition or event) is detected" or "in response to detecting (the stated condition or event)".

[0251] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0252] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.< / videoview> < / imgview> < / textview>

Claims

1. A control method applied to electronic devices, characterized in that, The method includes: After displaying the first dialog box for setting the matching relationship, the first key signal is obtained, which is the control signal generated by the first key on the remote control; Turn on the camera and acquire a first image and a second image. Determine a first somatosensory action based on the first image and the second image. The first image and the second image are images of the user performing the first somatosensory action from different angles captured by the camera. Associate the first button with the first motion-sensing action; After the game starts, turn on the camera; When the first haptic motion is detected from the image captured by the camera, the control object is controlled to perform the first action corresponding to the first button.

2. The method according to claim 1, characterized in that, After associating the first button with the first haptic gesture, the method further includes: The first interface is displayed, showing the association between the first button and the first motion-sensing action.

3. The method according to claim 1 or 2, characterized in that, Determining the first somatosensory action based on the first image and the second image includes: Identify the skeletal nodes in the first image to obtain the first set of skeletal node data; Identify the skeletal nodes in the second image to obtain a second set of skeletal node data; The threshold range is obtained by calculating the difference between the first set of skeletal node data and the second set of skeletal node data. Generate a first action template, the first action template including: a first set of skeletal node data, and / or, a second set of skeletal node data, and the threshold range.

4. The method according to claim 3, characterized in that, The acquisition of the first image and the second image includes: Collect the first set of image sequences of the user completing the motion-sensing actions; The image frame with the smallest change in the user's perceived motion compared to the previous image frame in the first image sequence is identified as the first image. Collect the second set of image sequences of the user completing the motion-sensing actions; The image frame with the smallest change in the user's perceived motion compared to the previous image frame in the second image sequence is identified as the second image.

5. The method according to claim 3, characterized in that, The method further includes: Get the third image containing the user's actions; Identify the skeletal nodes in the third image to obtain a third set of skeletal node data; The third set of skeletal node data is determined to match the first motion template; Lock the first action template.

6. The method according to claim 5, characterized in that, Determining that the third set of skeletal node data matches the first motion template specifically includes: Calculate the difference between the third set of skeletal node data and the first set of skeletal node data; When the difference is within the threshold range, it is determined that the third set of skeletal node data matches the first action template.

7. The method according to claim 3, characterized in that, The method further includes: The first action template is stored as a preset action template.

8. The method according to claim 7, characterized in that, The method further includes: The second interface is displayed, which shows multiple preset action templates for selection; Select the first action template from the plurality of preset action templates.

9. The method according to claim 8, characterized in that, The method further includes: Learn multiple user actions and store them as multiple preset action templates; And / or, obtain multiple shared preset action templates.

10. The method according to claim 8 or 9, characterized in that, Selecting the first action template from the plurality of preset action templates specifically includes: when it is recognized that the user's action matches the first action template, selecting the first action template from the plurality of preset action templates.

11. The method according to claim 3, characterized in that, The process of recognizing the first somatosensory action from the image captured by the camera includes: The fourth set of skeletal node data is obtained by recognizing the first somatosensory action from the image captured by the camera; Calculate the difference between the fourth set of skeletal node data and the first set of skeletal node data; When the difference is within the threshold range, it is determined that the fourth set of skeletal node data matches the first action template.

12. An electronic device, characterized in that, The electronic device includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-11.

13. A chip system applied to an electronic device, the chip system comprising one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-11.

14. A computer program product comprising instructions that, when executed on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-11.

15. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-11.

Images