Object control method and electronic device based on human posture recognition

Abstract

The present application discloses an object control method based on human posture recognition, which is applied to electronic devices and includes the following steps: obtaining user posture information; in a first scenario, determining the user's skeletal node information based on the posture information, and determining object posture information based on the skeletal node information; in a second scenario, if the user's posture is determined to be a posture in a preset posture library that matches the second scenario based on the posture information, obtaining preset object posture information corresponding to the user's posture from the posture library; otherwise, determining the user's skeletal node information based on the posture information, and determining object posture information based on the skeletal node information; the object posture information is used to control the target object to present a posture corresponding to the object posture information. That is, in different scenarios, different object control methods are adopted, which can optimize posture determination while ensuring the user's immersive experience, thereby effectively improving the user experience. The present application also discloses an electronic device.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an object control method and electronic device based on human posture recognition. Background Art

[0002] With the development of science and technology, it is now possible to obtain posture-related data of the actual human body (i.e., real people) through devices such as cameras and posture sensors. Then, the posture-related data of the actual human body can be used to control (or also be called drive), for example, three-dimensional virtual model objects in virtual reality (VR) scenes to imitate the actual human body movements, or to control physical model objects such as robots to imitate the actual human body movements. This provides technology for applications in virtual reality, games, movie digital special effects, human-computer interaction, health monitoring, rehabilitation training, dance training, and sports analysis.

[0003] For example, currently, in the process of controlling the movement of a three-dimensional virtual model through data related to the posture of an actual human body, the three-dimensional virtual model is usually controlled to present a corresponding posture based on the data related to the posture of the actual human body, or the user posture is directly identified based on the data related to the posture of the actual human body, and a preset corresponding model posture picture is played. The solution of controlling the three-dimensional virtual model to present a corresponding posture based on the data related to the posture of the actual human body has problems such as the inconsistency between the proportions of the model and the actual human body, resulting in the three-dimensional virtual model being unable to accurately restore the posture of the actual human body. In some scenes where the model needs to accurately restore the posture of the actual human body, there are problems that affect the model control effect and the user experience. The solution of directly playing a preset model posture picture based on the data related to the posture of the actual human body has the problem that the posture of the model is not completely consistent with the posture of the actual human body, that is, the three-dimensional virtual model cannot move along with the movement of the actual human body, resulting in a lack of immersive experience for the user and also affecting the user experience. Summary of the Invention

[0004] The present application provides an object control method and electronic device based on human posture recognition, which can solve the problem in the prior art that objects such as three-dimensional virtual models cannot accurately restore the posture of the actual human body, affecting the object control effect and user experience. That is, it can enable objects such as three-dimensional virtual models to more accurately restore the posture of the actual human body, improve the object control effect and improve the user experience.

[0005] To solve the above technical problems, in the first aspect, an embodiment of the present application provides an object control method based on human posture recognition, which is applied to an electronic device, and the method includes: obtaining the posture information of the user; in a first scenario, determining the user's skeleton node information according to the posture information, and determining the object posture information according to the skeleton node information; in a second scenario, if the user's posture is determined to be a posture in a preset posture library that matches the second scenario according to the posture information, then obtaining the preset object posture information corresponding to the user's posture from the posture library, otherwise, determining the user's skeleton node information according to the posture information, and determining the object posture information according to the skeleton node information; wherein the object posture information is used to control the target object to present a posture corresponding to the object posture information.

[0006] In the first scenario, the object posture information is determined based on the user's actual posture to control the movement of the target object, so that the posture of the target object is consistent with the user's actual posture. This allows the target object to move along with the user's movement, ensuring an immersive user experience and improving the user experience.

[0007] In the second scenario, if the user's posture is determined to match a preset posture library matching the second scenario based on the posture information, the target object's motion is controlled based on the preset model object posture information corresponding to the user's posture, obtained from the posture library. Compared to the object posture information corresponding to the user's actual posture, the object posture information in the posture library better meets the posture requirements of the second scenario. This allows for greater accuracy in the target object's posture, effectively ensuring effective object control, and enhancing the user's gaming experience.

[0008] In the second scenario, if the user's posture, as determined based on the user's posture information, is not a posture matching the second scenario in the preset posture library, the target object's movement is controlled based on the posture information corresponding to the user's actual posture, ensuring that the target object's posture matches the user's actual posture. This allows the target object to follow the user's movements, ensuring an immersive user experience and improving the user experience.

[0009] In this way, depending on the scene type of the target object, the target object's motion is alternately controlled based on the object posture information corresponding to the user's actual posture, or based on the object posture information pre-set in the posture library. In other words, using different object control methods in different scenarios can ensure an immersive user experience while optimizing posture determination in scenes with high posture requirements, effectively improving the user experience.

[0010] In a possible implementation of the first aspect, the second scenario has a higher requirement on the accuracy of the posture of the target object than the first scenario.

[0011] That is, the first scenario can be one that requires lower accuracy for the target object's posture, while the second scenario can be one that requires higher accuracy. This allows different object control methods to be used for scenarios with different posture accuracy requirements. This ensures an immersive user experience while optimizing posture determination in scenarios with high posture requirements, effectively improving the user experience.

[0012] In a possible implementation of the first aspect above, the method further includes determining the type of the scene where the target object is located in the following manner: if the scene where the target object is located is determined to be a preset key scene, then determining the scene where the target object is located as the second scene; otherwise, determining the scene where the target object is located as the first scene.

[0013] Key scenes may be scenes that require high accuracy of the target object's posture. Through key scenes, it is convenient to determine the scene where the target object is located, so as to select different control methods, which can effectively improve the user experience.

[0014] In a possible implementation of the first aspect above, the method also includes determining that the scene where the target object is located is a key scene in the following manner: if, based on the status information between the target object and the preset posture guidance object, the state between the target object and the posture guidance object is determined to be a preset key state, then the scene where the target object is located is determined to be a key scene, wherein the posture guidance object is an object in the scene where the target object is located, which is used to guide the user to present a posture.

[0015] In a possible implementation of the first aspect above, the status information is the distance information between the target object and the posture guidance object, and the critical state is that the distance between the target object and the posture guidance object is less than or equal to a preset distance threshold; or the status information is the time information required for the target object and the posture guidance object to approach each other, and the critical state is that the time required for the target object and the posture guidance object to approach each other is less than or equal to a preset time threshold.

[0016] The key state may be a state that requires a high degree of accuracy in the posture of the target object. The key state can be used to easily determine whether the scene where the target object is located is a key scene, thereby selecting different control methods, which can effectively improve the user experience.

[0017] In a possible implementation of the first aspect above, the method also includes determining that the scene where the target object is located is a key scene in the following manner: if, based on the type information of the target object, the type of the target object is determined to be a preset key object type, then the scene where the target object is located is determined to be a key scene; if, based on the type information of the user, the type of the user is determined to be a preset key user type, then the scene where the target object is located is determined to be a key scene.

[0018] The setting and judgment of key scenes can be set according to specific application scenarios to facilitate the selection of different object control methods.

[0019] In a possible implementation of the first aspect above, determining the user's posture as a posture in a preset posture library that matches the second scene based on the posture information includes: if a preset posture label is obtained by posture recognition based on the posture information, then determining the user's posture as a posture in the posture library that matches the second scene; obtaining object posture information corresponding to the user's posture from the posture library includes: obtaining object posture information corresponding to the posture tag from the posture library based on the posture label.

[0020] By performing gesture recognition according to the gesture information to determine whether a gesture label is obtained, it is possible to conveniently determine whether the user's gesture is a gesture matching the second scene in a preset gesture library.

[0021] In one possible implementation of the first aspect, performing gesture recognition based on the gesture information to obtain a gesture label includes: performing gesture recognition based on the gesture information; and obtaining a gesture label corresponding to the gesture information if a degree of match between the recognized user gesture and a gesture corresponding to a second scene in a gesture library is greater than or equal to a preset matching threshold. Gesture recognition may be achieved through machine learning or other methods.

[0022] In a possible implementation of the first aspect above, the user's skeleton node information is determined based on the posture information, and the object posture information is determined based on the skeleton node information, including: performing skeleton node detection and identification based on the posture information to obtain skeleton node information; determining the user posture data based on the skeleton node information; and performing motion redirection based on the user posture data to obtain object posture information.

[0023] In a possible implementation of the first aspect above, the skeletal node information includes coordinate information of key skeletal nodes of the user, and the user posture data includes coordinate information and rotation information of the key skeletal nodes.

[0024] In a possible implementation of the first aspect above, the object posture information is an object posture data frame; controlling the target object to present a posture corresponding to the object posture information according to the object posture information includes: controlling the target object to move from a posture corresponding to the current object posture data frame to a posture corresponding to the object posture data frame according to the object posture data frame.

[0025] In a possible implementation of the first aspect above, controlling the target object to move from a posture corresponding to the current object posture data frame to a posture corresponding to the object posture data frame according to the object posture data frame includes: determining, according to the current object posture data frame and the object posture data frame, a time delay for the target object to move from a posture corresponding to the current object posture data frame to a posture corresponding to the object posture data frame, and determining an intermediate object posture data frame; and forming and displaying an object motion animation including the intermediate object posture data frame and the object posture data frame by interpolation according to the time delay, the intermediate object posture data frame and the object posture data frame.

[0026] In a possible implementation of the first aspect above, obtaining the user's posture information includes: obtaining the user's body image data frame as the posture information through a camera; or obtaining the user's posture sensing data as the posture information through a posture sensor worn by the user.

[0027] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing a computer program, the computer program including program instructions; a controller for executing the program instructions so that the electronic device executes the object control method based on human posture recognition provided by the above-mentioned first aspect and / or any possible implementation of the first aspect.

[0028] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions. The program instructions are executed by a computer to enable the computer to execute the object control method based on human posture recognition provided by the first aspect and / or any possible implementation of the first aspect.

[0029] It can be understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings used in the description of the implementation methods.

[0031] Figure 1 1 is a schematic diagram showing a model control scenario according to some embodiments of the present application;

[0032] Figure 2 1 is a schematic structural diagram of a large-screen device 100 according to some embodiments of the present application;

[0033] Figure 3 is a schematic diagram showing an application scenario of an object control method based on human posture recognition provided by the present application according to some embodiments of the present application;

[0034] Figure 4A is a flow chart illustrating a method for controlling an object based on human posture recognition according to some embodiments of the present application;

[0035] Figure 4B According to some embodiments of the present application, a Figure 4A Schematic diagram of the implementation process of step S102 in the object control method based on human posture recognition shown;

[0036] Figure 4C 1 is a schematic diagram showing key skeletal nodes of a user 200 according to some embodiments of the present application;

[0037] Figure 5 is a flowchart illustrating another object control method based on human posture recognition provided by the present application according to some embodiments of the present application;

[0038] Figure 6 is a flow chart illustrating another object control method based on human posture recognition provided by the present application, according to some embodiments of the present application;

[0039] Figure 7 is a schematic diagram showing the structure of another large-screen device 100 and a flowchart of another object control method based on human posture recognition provided by the present application according to some embodiments of the present application;

[0040] Figure 8 According to some embodiments of the present application, Figure 7 Schematic diagram of the real-time process of some modules in the middle;

[0041] Figure 9 is a schematic diagram showing another application scenario of the object control method based on human posture recognition provided by the present application according to some embodiments of the present application;

[0042] Figure 10 1 is a schematic diagram showing the structure of an electronic device according to some implementations of the present application;

[0043] Figure 11 The figure shows a structural diagram of a system on chip (SoC) according to some implementations of the present application. DETAILED DESCRIPTION

[0044] The technical solution of this application will be described in further detail below with reference to the accompanying drawings.

[0045] Take the game scene as an example, see Figure 1 , the game scene includes, for example, a large-screen device 100 and a user 200, and the large-screen device 100 can be a device such as a television. The large-screen device 100 displays a game screen in which the game is running, and the game screen includes a game character model 110. The game character model 110 is a three-dimensional virtual model based on a three-dimensional (3D) human skeleton model, as an example of a target object to be controlled. The large-screen device 100 can obtain the posture information (also referred to as posture data, action data, motion data, etc.) of the user 200 located in front of the screen of the large-screen device 100 through a camera, or obtain the posture information of the user 200 through a posture sensor worn by the user 200 (such as a bracelet) or a handheld posture sensor (such as a handle). Then, the large-screen device 100 can control the game character model 110 to imitate the movement of the user 200 (i.e., the actual human body) according to the posture information of the user 200.

[0046] The game scene can be, for example, a game similar to "Ring Fit Adventure". In this game, the large-screen device 100 can control the game character model 110 to present a corresponding posture based on the somatosensory data (as an example of posture information) of the user 200 detected by the fitness ring (as an example of a posture sensor) held by the user 200, or directly play the preset posture picture of the game character model 110 according to the somatosensory data.

[0047] In this game scenario, if the large-screen device 100 controls the game character model 110 to present a corresponding posture based on the somatosensory data of the user 200 detected by the fitness ring. Due to the problem of inconsistency between the actual body proportions of the game character model 110 and the user 200, the posture of the game character model 110 is not completely consistent with the actual posture of the user 200, that is, the game character model 110 cannot move along with the actions of the user 200, which will result in the game character model 110 being unable to accurately restore the actual posture of the user 200. In addition, the quality of the posture required by the user 200 to complete the game varies from person to person, which will result in the game character model 110 accurately restoring the actual posture of the user 200 to a certain extent, but still failing to meet the requirements of the game, thereby causing the user to fail continuously and reducing the user experience. In addition, if the large-screen device 100 directly matches and plays the preset posture picture of the game character model 110 (i.e., the game action picture) based on the somatosensory data of the user 200, on the one hand, the game difficulty will become too low, and the user will lack the challenge, which will also affect the user experience. On the other hand, the posture of the preset game character model 110 is not completely consistent with the actual posture of the user 200, which may cause the user to lack immersive experience, thereby affecting the user experience.

[0048] The present application provides an object control method based on human posture recognition, which can be applied to a game scene, which includes a large-screen device 100 and a user 200. The object control method based on human posture recognition provided by the present application includes, in a first scenario (for example, some general scenarios in which the game character model 110 is required to present a relatively simple posture such as stretching an arm), the large-screen device 100 determines the skeleton node information of the user 200 according to the posture information of the user 200, and determines the model posture information corresponding to the actual posture of the user 200 according to the skeleton node information (as an example of object posture information), and then controls the game character model 110 to present a posture corresponding to the model posture information according to the model posture information, that is, to present a posture consistent with the posture of the user 200, so as to accurately restore the actual posture of the user 200. In this way, the large-screen device 100 can control the movement of the game character model 110 according to the actual posture of the user 200, so that the posture of the game character model 110 is consistent with the actual posture of the user 200. The game character model 110 can be made to move along with the movement of the user 200, accurately restoring the actual posture of the user 200, ensuring the user's immersive experience and improving the user experience.

[0049] In the second scenario (for example, some key scenarios that require the game character model 110 to present more complex postures such as squatting), the large-screen device 100 determines whether the posture of the user 200 is a posture in the preset posture library that matches the second scenario based on the posture information of the user 200. If so, the large-screen device 100 obtains the preset model posture information corresponding to the posture of the user 200 from the posture library, and controls the game character model 110 to present the posture corresponding to the model posture information based on the model posture information, that is, to present a posture that matches the posture of the user 200.

[0050] The second scenario typically requires higher accuracy for the pose of the game character model 110 than the first scenario. Furthermore, the model pose information corresponding to the user's 200 pose, as pre-set in the pose library, is more consistent with the pose requirements of the game character model 110 in the second scenario, a key scenario, than the model pose information corresponding to the user's 200 actual pose. Therefore, by optimizing pose determination, the pose accuracy of the game character model 110 can be increased, effectively ensuring effective model control and improving the user's gaming and user experience.

[0051] In the second scenario, if the large-screen device 100 determines, based on the posture information of the user 200, that the posture of the user 200 is not a posture that matches the second scenario in the preset posture library, the large-screen device 100 determines the skeletal node information of the user 200 based on the posture information of the user 200, and determines the model posture information corresponding to the actual posture of the user 200 based on the skeletal node information. Then, based on the model posture information, the large-screen device 100 controls the game character model 110 to present a posture corresponding to the model posture information, that is, to present a posture consistent with the posture of the user 200. In this way, the large-screen device 100 can control the movement of the game character model 110 based on the actual posture of the user 200, so that the posture of the game character model 110 is consistent with the actual posture of the user 200. The game character model 110 can move in accordance with the user's movement, accurately restoring the actual posture of the user 200, ensuring the user's immersive experience, and improving the user experience.

[0052] In this implementation, the large-screen device 100 can, according to the scene type in which the game character model 110 is located, alternately choose to control the movement of the game character model 110 according to the model posture information corresponding to the actual posture of the user 200, or control the movement of the game character model 110 according to the pre-set model posture information. On the one hand, it can ensure the user's immersive experience, and on the other hand, it can optimize the posture judgment in some scenes with high posture requirements. Controlling the movement of the game character model 110 according to the pre-set model posture information can effectively improve the playability of the game and effectively improve the user experience.

[0053] See Figure 2 , Figure 2 A structural diagram of a large-screen device 100 provided in this application as an example of an electronic device is shown.

[0054] The large-screen device 100 may include a processor 101 , a display screen 102 , a camera 103 , a memory 104 , an audio module 105 , a communication module 106 and a power module 107 .

[0055] The processor 101 may include one or more processing units. For example, the processor 101 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0056] The processor 101 can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and execution.

[0057] A memory may also be provided in the processor 101 for storing instructions and data. In some embodiments, the memory in the processor 101 is a cache memory. The memory may store instructions or data that have just been used or are recycled by the processor 101. If the processor 101 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 101, and thus improves the efficiency of the system. The processor 101 is used to execute relevant functional applications and data processing of the large-screen device 100.

[0058] Large-screen device 100 implements display functionality through display screen 102, a GPU, and an application processor. The GPU is a microprocessor for image processing that connects display screen 102 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 101 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0059] The display screen 102 is used to display game images, videos, etc., such as the aforementioned game character model 110. The display screen 102 includes a display panel. In some embodiments, the large-screen device 100 may include one or N display screens 102, where N is a positive integer greater than one.

[0060] The large-screen device 100 can implement a shooting function through an ISP, a camera 103 , a video codec, a GPU, a display screen 102 , and an application processor, so as to obtain the aforementioned posture information of the user 200 .

[0061] The camera 103 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the large-screen device 100 may include 1 or N cameras 103, where N is a positive integer greater than 1.

[0062] The NPU is a neural network (NN) computing processor that rapidly processes input information and continuously self-learns by drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain. The NPU can enable intelligent cognitive applications such as image recognition, face recognition, speech recognition, and text comprehension in the large-screen device 100.

[0063] The internal memory 104 can be used to store computer executable program code, which includes instructions. The internal memory 104 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the large-screen device 100 (such as audio data, etc.), etc. In addition, the internal memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 101 executes various functional applications and data processing of the large-screen device 100 by running instructions stored in the internal memory 104 and / or instructions stored in a memory provided in the processor, such as executing the object control method based on human posture recognition provided in this application.

[0064] The large-screen device 100 can implement audio functions through the audio module 105 and the application processor. For example, music playback, etc. The audio module 105 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 105 can also be used to encode and decode audio signals. In some embodiments, the audio module 105 can be set in the processor 101, or some functional modules of the audio module 105 can be set in the processor 101.

[0065] The large-screen device 100 can communicate with other electronic devices such as mobile phones and servers through the communication module 106.

[0066] The large-screen device 100 can be powered by receiving charging input through the power module 107.

[0067] It is understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the large-screen device 100. In other embodiments of the present application, the large-screen device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0068] The object control method based on human posture recognition provided by this application will be described in more detail below.

[0069] See Figure 3 , the object control method based on human posture recognition provided by this application can be applied to the "wall-walking game" scene (as an example of an object control scene). Figure 3 As shown, the game scene includes a large-screen device 100 and a user 200, wherein a game screen is displayed on the display screen 102 of the large-screen device 100, and the game screen includes a game character model 110 (as an example of a target object) and at least one wall model 120 (as an example of a posture guidance object), and the wall model 120 includes a posture example 121 for instructing the user 200 to present a corresponding posture (which can also be understood as being used to instruct the game character model 110 to present a corresponding posture).

[0070] In the "wall-walking game" scenario, the large-screen device 100 collects a human body image data frame (as an example of posture information, it can also be called human body image data or human body image frame) of the user 200 through the camera 103, and controls the game character model 110 to present a posture corresponding to the posture of the user 200 according to the human body image data frame. For example, Figure 3As shown, the game character model 110 presents a posture consistent with the posture of the user 200. Furthermore, if the user 200 strikes a posture identical to the posture example 121 according to the posture example 121 on the wall model 120, the large-screen device 100 can control the game character model 110 to present a posture consistent with the posture example 121 (i.e., the posture of the user 200) based on the collected human body image data frame of the user 200.

[0071] In addition, in the game scene, the game character model 110 is usually located at a fixed position A, and the wall model 120 is along Figure 3 In the direction B shown, the player approaches the game character model 110 at a certain speed. If the game character model 110 adopts the same posture as the posture example 121 on the wall model 120, the game character model 110 can pass through the posture example 121 on the wall model 120, and the game character model 110 successfully passes through the wall. If the game character model 110 does not adopt the same posture as the posture example 121 on the wall model 120, the game character model 110 cannot pass through the posture example 121 on the wall model 120, and the game character model 110 fails to pass through the wall.

[0072] See Figure 4A In one implementation of the object control method based on human posture recognition provided in the present application, the process of controlling the movement of the game character model 110 by the large-screen device 100 according to the human body image data frame of the user 200 includes the following steps:

[0073] S101, after the game starts, the large screen device 100 displays the game screen, which includes Figure 3 The game character model 110 and at least one wall model (in Figure 3 Only one wall model 120 is shown as an example, and the wall models are arranged one by one along the Figure 3 The direction B shown is gradually approaching the game character model 110 at a certain speed.

[0074] At this time, the user 200 can see the gesture examples 121 included in the wall model 120 through the display screen 102 of the large-screen device 100 , and perform a gesture that is the same as or similar to the gesture example 121 according to the gesture example 121 .

[0075] At the same time, the large-screen device 100 captures images of the user 200 in real time via the camera 103 to obtain a human body image data frame of the user 200. Alternatively, the large-screen device 100 periodically captures images of the user 200 via the camera 103 according to a preset capture period to obtain a human body image data frame of the user 200. The capture period can be, for example, 0.5s, 1s, 2s, etc., and its value range can be 0.5s to 2s. Of course, the capture period can also be specifically set according to actual needs.

[0076] In this implementation, the large-screen device 100 needs to determine the scene in which the game character model 110 is located based on the status information between the wall model 120 and the game character model 110, and determine the control method for the corresponding action of the game character model 110. The distance information between the wall model 120 and the game character model 110 can be used as an example of the status information between the wall model 120 and the game character model 110. Through this distance information, it can be determined whether the state between the wall model 120 and the game character model 110 is a critical state, thereby determining whether the control scene is a critical scene. For example, if the distance between the wall model 120 and the game character model 110 is greater than a preset distance threshold, the large-screen device 100 considers that the state between the wall model 120 and the game character model 110 is a non-critical state, that is, the current game scene is a general scene (i.e., a non-critical scene). If the distance between the wall model 120 and the game character model 110 is equal to or less than a preset distance threshold, the large-screen device 100 considers that the state between the wall model 120 and the game character model 110 is a critical state, and the current game scene is a critical scene.

[0077] The distance threshold value can be set according to the game scene and the coordinate system of the game scene, for example, it can be 1m, 1.5m, 2m, etc. in the game scene coordinate system, that is, its value range can be 1m to 2m in the game scene coordinate system. Of course, it can also be set to other values ​​as needed.

[0078] After the game starts, the first wall model 120 gradually approaches the game character model 110. At this time, since the distance between the first wall model 120 and the game character model 110 is relatively far, it is generally believed that the distance between the first wall model 120 and the game character model 110 is greater than the preset distance threshold. Then the large-screen device 100 considers that the current game scene is a general scene, and the large-screen device 100 executes S102.

[0079] S102, the large-screen device 100 performs skeletal node detection and identification based on the collected human body image data frame of the user 200 to obtain the skeletal node information of the user 200. The skeletal node information can be, for example, the coordinate information of the key skeletal nodes on the body of the user 200. The key skeletal nodes can be, for example, the head, arms, waist, legs and other parts.

[0080] Then, the large-screen device 100 determines a user posture data frame (also referred to as user posture data) based on the skeletal node information. The user posture data frame generally includes coordinate information and rotation information of key skeletal nodes. In addition, the user posture data frame is data used to characterize the motion information of the user 200, and can also be referred to as human motion data (or human motion data frame).

[0081] Next, the large-screen device 100 performs motion redirection according to the user posture data frame to obtain a model posture data frame (as an example of model posture information, or may also be referred to as model posture data, object posture data frame) F1.

[0082] Then, the large-screen device 100 controls the game character model 110 to present a posture corresponding to the model posture data frame F1 according to the model posture data frame F1 , that is, to present a posture consistent with the posture of the user 200 .

[0083] See Figure 4B For example, in this implementation, in step S102, the large-screen device 100 obtains the model posture data frame F1 based on the collected human body image data frame of the user 200, which may specifically include the following steps:

[0084] S1021 , the large-screen device 100 determines the current human body image data frame of the user 200 captured by the camera 103 .

[0085] S1022, the large-screen device 100 uses a feature recognition algorithm or a machine learning algorithm to detect and identify skeleton nodes of the human body image data frame.

[0086] S1023, the large-screen device 100 identifies and obtains the coordinate information of key skeletal nodes on the body of the user 200.

[0087] Key skeletal nodes may include, for example, Figure 4C The user 200 shown has a waist node 1, a neck node 2, a head node 3, a right shoulder node 4, a right elbow node 5, a right wrist node 6, a left shoulder node 7, a left elbow node 8, a left wrist node 9, a right thigh node 10, a right knee node 11, a right ankle node 12, a left thigh node 13, a left knee node 14 and a left ankle node 15.

[0088] The coordinate information of the key bone nodes can be (X i , Y i , Z i ), where i is the aforementioned node number, i∈(1, 2, ..., 15).

[0089] S1024, the large screen device 100 calculates the quaternion (x) corresponding to the coordinates of each key bone node according to the coordinate information of the key bone node through a model driven algorithm (which can be understood as a quaternion calculation related algorithm). i ,y i , z i , w i ), and obtain the rotation information of the key bone nodes. In this implementation, the coordinate information and rotation information of the key bone nodes are used as examples of the user posture data frame.

[0090] S1025, the large screen device 100 uses the motion redirection algorithm to first calculate the coordinate information (X i , Y i , Z i ), and the quaternion (x i ,y i , z i , w i ), we get {(X1, Y1, Z1), (x i ,y i , z i , w i )|i∈P}, that is, the coordinates of the root node (i.e. waist node 1) and the quaternions of all other key bone nodes are obtained, and P is the set of key bone nodes.

[0091] S1026, the large-screen device 100 uses the coordinates of the root node to set the root node position of the game character model 110, and sets the rotation degree of each corresponding key bone node from the root node (i.e., the root) to the leaf node according to the topological structure of the bone node to obtain the model posture data frame F1.

[0092] Furthermore, in step S102 of the present implementation, the large-screen device 100 controls the game character model 110 to present a posture corresponding to the model posture data frame F1 according to the model posture data frame F1, including: the large-screen device 100 determines the time delay ΔT of the game character model 110 from the posture corresponding to the model posture data frame FS to the posture corresponding to the model posture data frame F1 according to the current model posture data frame FS of the game character model 110 and the model posture data frame F1. Since in most scenarios (i.e., general scenarios), the large-screen device 100 selects the model posture data frame F1 to control the game character model 110, the jump of the model posture data frame is small, and ΔT can be 0, that is, no subsequent interpolation or interpolation processing is required. That is, the large-screen device 100 directly controls the movement of the game character model 110 according to the model posture data frame F1, so that the game character model 110 presents the posture corresponding to the model posture data frame F1, that is, presents a posture consistent with (i.e., the same as) the posture of the user 200.

[0093] In this implementation, the large-screen device 100 can control the movement of the game character model 110 according to the model posture data frame F1 corresponding to the actual posture of the user 200, so that the posture of the game character model 110 is consistent with the actual posture of the user 200, so as to accurately restore the actual posture of the user 200, that is, the game character model 110 can move with the user's movement, effectively ensuring the user's immersive experience and improving the user experience.

[0094] Please continue to see Figure 4A , along the first wall model 120 Figure 3 In the process of gradually approaching the game character model 110 at a certain speed in the direction B shown, the large-screen device 100 further performs the following steps:

[0095] S103, the large-screen device 100 determines the distance between the first wall model 120 and the game character model 110 in real time, and determines whether the distance between the wall model 120 and the game character model 110 is less than or equal to the aforementioned preset distance threshold.

[0096] If the large-screen device 100 determines that the distance between the first wall model 120 and the game character model 110 is greater than the aforementioned preset distance threshold, the large-screen device 100 determines that the current control scene is a general scene, and the large-screen device 100 continues to execute S102, that is, performing skeleton node detection and identification based on the human body image data frame of the user 200 collected in real time to obtain the skeleton node information of the user 200, and obtaining a new model posture data frame F1 based on the skeleton node information, and controlling the movement of the game character model 110 so that the game character model 110 presents a posture consistent with that of the user 200.

[0097] If the large-screen device 100 determines that the distance between the first wall model 120 and the game character model 110 is equal to or less than a preset distance threshold, the large-screen device 100 determines that the state between the wall model 120 and the game character model 110 is a preset critical state, the large-screen device 100 determines that the current control scene is a critical scene, and the large-screen device 100 executes S104.

[0098] S104 , the large-screen device 100 performs posture recognition based on the collected human body image data frame of the user 200 , and then executes S105 .

[0099] In this implementation, the large-screen device 100 performs posture recognition based on the collected human body image data frame of the user 200. For example, posture recognition can be performed through a pre-trained posture recognition neural network model. The pre-trained posture neural network model can be a model for realizing posture recognition function based on some common neural network model settings. That is, the posture neural network model can obtain the actual posture of the user 200 based on the human body image data frame, and obtain the model posture corresponding to each model posture data frame based on the model posture data frame pre-set in the posture library. And the posture neural network model can determine whether the actual posture of the user 200 matches the model posture corresponding to the model posture data frame corresponding to the current key scene. If it matches, the neural network model outputs the corresponding posture label. If it does not match, the neural network model does not output the posture label. The specific model structure of the neural network model can be specifically set as needed, and this application does not provide detailed descriptions of this.

[0100] In this implementation, the large-screen device 100 performs posture recognition based on the collected human body image data frame of the user 200. Alternatively, the large-screen device 100 obtains a corresponding model posture data frame based on the human body image data frame, and then calculates the degree of match between the model posture data frame and the model posture data frame corresponding to the current control scene pre-set in the posture library, that is, determines whether the degree of match between the user 200's posture and the posture corresponding to the current control scene in the posture library is greater than or equal to a preset matching threshold. If the matching degree is greater than or equal to the preset matching threshold, the large-screen device 100 obtains the corresponding posture label. If the matching degree is less than the preset matching threshold, the large-screen device 100 cannot obtain the posture label.

[0101] The value range of the matching threshold may be 95% to 100%, for example, 95%, 97%, 98.5%, 100%, etc. Of course, the value of the matching threshold may also be specifically set according to the control scenario and needs.

[0102] S105 , the large-screen device 100 determines whether a gesture tag is obtained.

[0103] If the large-screen device 100 does not obtain a posture label after performing posture recognition based on the human image data frame, it means that the posture of the user 200 is not a posture that matches the key scene in the preset posture library. The large-screen device 100 then executes the aforementioned S102, i.e., performs skeleton node detection and recognition based on the human image data frame of the user 200 collected in real time to obtain the skeleton node information of the user 200, and obtains a new model posture data frame F1 based on the skeleton node information, and controls the movement of the game character model 110 so that the game character model 110 presents a posture consistent with that of the user 200.

[0104] If the large-screen device 100 can obtain the posture tag, it means that the posture of the user 200 is a posture that matches the key scene in the preset posture library, and the large-screen device 100 executes S106.

[0105] It should be noted that in this implementation, the large-screen device 100 can pre-set multiple posture tags and corresponding model posture data frames in the posture library according to different game scenarios. The model posture data frame contains posture data used to control the movement of the game character model 110. Compared with the model posture data frame corresponding to the actual posture of the user 200, the model posture data frame corresponding to the posture tag preset in the posture library has a posture that better matches the posture example 121, which can improve the accuracy of the posture of the game character model 110, that is, it can ensure that the game character model 110 can pass through the posture example 121 with accuracy, thereby improving the playability of the game and enhancing the user's gaming experience.

[0106] In this implementation, key scenes are usually scenes that require strict model postures. In addition, the model posture data frames preset in the posture library are key posture data sets for the game character model 110. Key postures can be complex postures such as squatting and jumping.

[0107] S106, the large-screen device 100 obtains a preset model posture data frame F2 corresponding to the posture label (ie, the posture of the user 200) from the posture library, and controls the game character model 110 to present the posture corresponding to the model posture data frame F2 according to the model posture data frame F2.

[0108] In this implementation, if the large-screen device 100 determines that the distance between the wall model 120 and the game character model 110 is equal to or less than a preset distance threshold, and the large-screen device 100 obtains a posture label after performing posture recognition based on the human body image data frame, the large-screen device 100 controls the movement of the game character model based on the model posture data frame F2 corresponding to the posture of the user 200 (i.e., the current control scene) obtained from the posture library. Compared with the model posture data frame F1 corresponding to the actual posture of the user 200, the model posture data frame F2 preset in the posture library usually has a model posture corresponding to the model posture data frame F2 that is more closely matched with the posture example 121. This can improve the accuracy of the posture of the game character model 110, making it easier for the game character model 110 to match the posture example 121 of the hollowed-out portion of the wall model 120, that is, it can better meet the posture requirements of the game character model 110 in key scenes, thereby improving the user's overall gaming experience, that is, improving the user experience.

[0109] In this implementation, the large-screen device 100 controls the game character model 110 to present a posture corresponding to the model posture data frame F2 according to the model posture data frame F2, including that the large-screen device 100 determines the time delay △T' for the game character model 110 to move from the posture corresponding to the model posture data frame FS to the posture corresponding to the model posture data frame F1 according to the current model posture data frame FS and the model posture data frame F2 of the game character model 110, and determines the intermediate model posture data frame F0. Then, the large-screen device 100 generates a model action animation of a duration of △T' from the model posture data frame FS to the intermediate model posture data frame F0, and then to the model posture data frame F2 from the model posture data frame FS by interpolation or interpolation. Then, the large-screen device 100 displays the model action animation, that is, the game character model 110 presents a posture corresponding to the model posture data frame F2.

[0110] In this implementation, while the large-screen device 100 is executing the aforementioned steps S101 to S106, that is, while the first wall model 120 reaches position A where the game character model 110 is located, and while the game character model 110 passes through the first wall model 120, the second wall model 120 is also gradually approaching the game character model 110 at a certain speed. At this time, the large-screen device 100 is also looping through the aforementioned steps S101 to S106, that is, while the second wall model 120 is approaching the game character model 110, the control method of the game character model 110 is determined. And as the third, fourth, and so on wall models 120 subsequently approach the game character model 110, the large-screen device 100 also executes the aforementioned steps S101 to S106 to determine the control method of the game character model 110.

[0111] The object control method based on human posture recognition provided by this implementation is that when the distance between the wall model 120 and the game character model 110 is greater than a preset distance threshold, that is, when the wall model 120 is far away from the game character model 110, the large-screen device 100 considers the current control scene to be a general scene. The large-screen device 100 performs skeletal node detection and recognition based on the human body image data frame of the user 200 to determine the skeletal node information, and performs motion redirection based on the skeletal node information to obtain the model posture data frame F1 corresponding to the user's actual posture, and controls the movement of the game character model 110 based on the model posture data frame F1. In this way, the large-screen device 100 can control the movement of the game character model 110 in real time according to the actual posture of the user 200, so that the posture of the game character model 110 is consistent with the actual posture of the user 200. That is, the game character model 100 can follow the movement of the user 200, ensuring the user's immersive experience and improving the user experience.

[0112] When the distance between the wall model 120 and the game character model 110 is equal to or less than a preset distance threshold, that is, when the wall model 120 is close to the game character model 110, the current control scene is considered to be a key scene. The large-screen device 100 performs posture recognition based on the human body image data frame of the user 200. If the posture label corresponding to the human body image data frame can be identified from the posture library, the corresponding model posture data frame F2 is obtained from the posture library, and the model movement is controlled according to the model posture data frame F2 to ensure the accuracy of the model posture, which can effectively improve the user's gaming experience. If the corresponding label cannot be identified, the movement of the game character model 110 is controlled according to the aforementioned model posture data frame F1 to ensure the user's immersive experience. In this way, the playability of the game can be better guaranteed and the user experience can be improved.

[0113] See Figure 5In another implementation of the present application, the object control method based on human posture recognition provided by the present application includes the following steps:

[0114] S201 , the game starts, the large-screen device 100 captures an image of the user 200 through the camera 103 , obtains a human body image data frame of the user 200 , and obtains distance information between the game character model 110 and the first wall model 120 .

[0115] Compared with the aforementioned Figure 4A The object control method based on human posture recognition shown in the figure, in this implementation, after the game starts, the large-screen device 100 can simultaneously obtain the human body image data frame of the user 200, and obtain the distance information between the game character model 110 and the first wall model 120, for use in the decision-making of the control of the game character model 110.

[0116] S202: The large-screen device 100 determines whether the distance between the game character model 110 and the first wall model 120 is less than or equal to the aforementioned preset distance threshold.

[0117] If the large screen device determines that the distance between the game character model 110 and the wall model 120 is greater than the preset distance threshold, the large screen device executes S203. If the large screen device determines that the distance between the game character model 110 and the wall model 120 is less than or equal to the preset distance threshold, the large screen device executes S204.

[0118] S203, the large-screen device 100 obtains a user posture data frame through skeleton node detection and identification based on the human body image data frame, and then the large-screen device 100 performs motion redirection based on the user posture data frame to obtain a model posture data frame F1, and then the large-screen device 100 controls the game character model 110 to present a posture corresponding to the model posture data frame F1 based on the model posture data frame F1, that is, presents a posture corresponding to the actual posture of the user 100.

[0119] S204, the large-screen device 100 performs posture recognition based on the human body image data frame, and then executes S205.

[0120] S205: The large-screen device 100 determines whether a gesture tag is obtained.

[0121] If the large screen device 100 performs posture recognition based on the human body image data frame and obtains a posture label, the large screen device 100 executes S206; if the large screen device 100 does not recognize a posture label based on the human body image data frame, the large screen device 100 executes S203.

[0122] S206 , the large-screen device 100 obtains the corresponding model posture data frame F2 from the preset posture library according to the posture tag, and controls the game character model 110 to present the posture corresponding to the model posture data frame F2 according to the model posture data frame F2 .

[0123] In this implementation, the specific implementation process of each step is the same as the above Figures 4A-4C The corresponding processes of the corresponding embodiments are the same or similar and will not be described again here.

[0124] In this implementation, after the game begins, the large-screen device 100 obtains a human image data frame of the user 200, as well as the distance information between the game character model 110 and the wall model 120. Based on this distance information, the device determines whether the control scene is a general scene or a key scene, and executes the corresponding model control method based on the different scene types and gesture recognition results. This ensures an immersive user experience while also ensuring the accuracy of the model gestures, improving game playability and user experience.

[0125] See Figure 6 In another implementation of the present application, the object control method based on human posture recognition provided by the present application includes the following steps:

[0126] S301, the game starts, the large-screen device 100 obtains the human body image data frame of the user 200 through the camera 103, and executes S302 and S303 respectively.

[0127] In step S302, the large-screen device 100 obtains a user posture data frame by detecting and identifying skeleton nodes based on the human body image data frame.

[0128] S303: The large-screen device 100 performs posture recognition based on the human body image data frame. If the large-screen device 100 recognizes a posture tag, the large-screen device 100 executes S305; otherwise, the large-screen device 100 continues to execute S302.

[0129] S304 , the large-screen device 100 performs motion redirection according to the user posture data frame to obtain a model posture data frame F1 .

[0130] S305 , the large-screen device 100 extracts a preset model posture data frame F2 from a preset posture library according to the posture label.

[0131] S306, the large-screen device 100 determines the distance between the wall model 120 and the game character model 110 to decide which model posture data frame F1 and the model posture data frame F2 to use as the target key frame (also called model posture key frame, or model action key frame) for controlling the game character model 110.

[0132] If the large-screen device 100 determines that the distance between the wall model 120 and the game character model 110 is greater than the aforementioned preset distance threshold, the large-screen device 100 decides to use the model pose data frame F1 as the target key frame for controlling the model. If the large-screen device 100 determines that the distance between the wall model 120 and the game character model 110 is equal to or less than the aforementioned preset distance threshold, the large-screen device 100 decides to use the model pose data frame F2 as the target key frame for controlling the game character model 110.

[0133] S307, the large screen device 100 sets the target key frame according to the decision, and performs frame insertion processing from the current posture data frame to complete the model action animation. The method of generating the model action animation is as described above and will not be repeated here.

[0134] In this implementation, the delay from step S301 to steps S302 and S305 corresponds to the traditional delay, while steps S306 and S307 correspond to the additional delay incurred during the process of determining which of model pose data frame F1 and model pose data frame F2 to use as the target keyframe for controlling game character model 110. Typically, both delays are relatively small. This means that the object control method based on human pose recognition provided by this implementation provides a relatively low delay throughout the entire model control process, effectively enhancing the user's gaming experience.

[0135] Please continue to see Figure 6 In this implementation, the wall model 120 is Figure 3 In the process of gradually approaching the game character model 110 at a certain speed in the direction B shown, if the large-screen device 100 determines that the distance between the wall model 120 and the game character model 110 is greater than the aforementioned distance threshold, the large-screen device 100 can continue to execute S3081 and 3082, that is, directly determine the model posture data frame F1 based on the user posture data frame to control the movement of the game character model 110.

[0136] In the implementation method, during the execution of the aforementioned S303, if the matching time of the large-screen device 100 exceeds the preset maximum delay during the process of posture recognition to match the posture label, the large-screen device 100 executes S309, that is, abandons the scheme of controlling the game character model 110 according to the model posture data frame F2, and can execute S302, that is, directly determine the model posture data frame F1 according to the user posture data frame to control the movement of the game character model 110.

[0137] The maximum delay may be in the range of 1s to 2s, such as 1s, 1.5s, 2s, etc. Of course, it may also be set to other values ​​as needed.

[0138] In the implementation method, during the execution of the aforementioned S303, if the large-screen device 100 fails to match the posture tag during posture recognition to match the posture tag, the large-screen device 100 executes S310, that is, abandons the scheme of controlling the game character model 110 according to the model posture data frame F2, and can execute S302, that is, directly determine the model posture data frame F1 according to the user posture data frame to control the movement of the game character model 110.

[0139] The object control method based on human posture recognition provided by this implementation is an object control method based on human posture recognition based on posture recognition (or also known as action recognition, or posture tag recognition) plus skeleton node detection and recognition. The large-screen device 100 controls the game character model 110 by dynamically combining posture recognition and skeleton node detection and recognition. On the one hand, in general scenarios, the large-screen device 100 performs skeleton node detection and recognition based on the human body image data frame of the user 200 to determine the skeleton node information, and obtains the model posture data frame F1 based on the skeleton node information to control the movement of the game character model 110. It can accurately restore the actual posture of the user 200, so that the game character model 110 can move with the user's actions, which can enhance the user's immersive experience. On the other hand, in a specified scenario (i.e., a key scenario), only when the actual posture of the user 200 matches the posture example 121 in the current control scenario, the large-screen device 100 will choose to extract the corresponding model posture data frame F2 from the preset posture library to control the movement of the game character model 110. That is, the posture of the game character model 110 can be corrected by combining posture recognition to control the movement of the game character model 110, thereby meeting certain posture requirements of the game character model 110, ensuring the accuracy of the game character model 110's movements, and improving the playability of the game, thereby effectively improving the user experience. If the actual posture of the user 200 does not match the posture example 121 in the current control scenario, the large-screen device 100 performs skeletal node detection and recognition based on the human body image data frame of the user 200 to determine the skeletal node information, and obtains the model posture data frame F1 based on the skeletal node information to control the movement of the game character model 110. This allows the game character model 110 to move along with the user's actions, which can enhance the user's immersive experience.

[0140] See Figure 7 In another implementation of the present application, the present application also provides a model control system, which includes an image acquisition module, a posture recognition module, a skeleton node detection module, a model driving module, a three-dimensional model module and a posture library.

[0141] The image acquisition module is responsible for acquiring human body images and outputting human body image data frames for subsequent data frame analysis.

[0142] The posture recognition module is responsible for identifying the human body image data frame as a specified posture type and outputting the corresponding posture label. Of course, there may be cases where posture label recognition is unsuccessful. For example, after analyzing the actual posture of the user 200 and the matching degree between the model posture corresponding to each model posture data frame, if the matching degree is less than a preset matching degree threshold, the recognition may be considered unsuccessful.

[0143] The skeleton node detection module is responsible for identifying key skeleton nodes and other data from the human body image data frame and outputting the user posture data frame. The user posture data frame contains basic motion data, such as the coordinate information of key skeleton nodes, rotation information, etc.

[0144] The model driver module, based on the posture tags, retrieves model motion data frame F1 from a pre-set posture library. It also converts the user's posture data frame into model motion data frame F2 through motion redirection. It then determines which model posture data frame F1 or F2 to use as the next keyframe in the model action animation based on factors such as the control scenario and the aforementioned posture recognition accuracy. It then controls the game character model 110 to move to that keyframe. Specifically, interpolation or frame insertion can be used to achieve smooth transitions between model motion data frames.

[0145] The posture library includes a set of key model posture data frames of pre-set models. Key postures can be, for example, complex movements such as squatting.

[0146] The 3D model module is used to build virtual 3D models in scenes such as games.

[0147] Please continue to see Figure 7 This implementation provides an object control method based on human posture recognition based on the model control system, comprising the following steps:

[0148] In step S401, the image acquisition module acquires an image of the user 200, obtains a human body image data frame of the user 200, and outputs the human body image data frame to the posture recognition module and the skeleton node detection module, respectively. The posture recognition module and the skeleton node detection module then execute corresponding steps S402 and S403, respectively.

[0149] S402, the posture recognition module performs posture recognition based on the human body image data frame, and if a posture label is obtained through recognition, the posture label is output and sent to the model driving module.

[0150] S403, the skeleton node detection module obtains a user posture data frame through skeleton node detection and recognition according to the human body image data frame, and sends the user posture data frame to the model driving module.

[0151] S404 , the model driving module obtains a preset model posture data frame F2 corresponding to the posture label from the posture library according to the posture label sent by the posture recognition module.

[0152] In step S405, the model driving module obtains and determines the distance between the wall model 120 and the game character model 110, and obtains a model pose data frame F1 based on the user pose data frame sent by the skeletal node detection module. The model driving module then determines which of the model pose data frame F2 and the model pose data frame F1 obtained from the pose library will be used as the model pose data frame for controlling the game character model 110, and outputs the model pose data frame to the 3D model module.

[0153] S406, after the three-dimensional model module receives the model posture data frame sent by the model driving module, it determines the delay based on the current model posture data frame of the game character model 110 and the model posture data frame, and then generates a model action animation with a delay time from the current model posture data frame to the model posture data frame by interpolating the model posture data frame of the current model, and outputs the model action animation and displays it to achieve the effect of controlling the movement of the game character model 110.

[0154] Further, see Figure 8 In this implementation, the skeleton node detection module obtains the user posture data frame through skeleton node detection and recognition based on the human body image data frame, such as Figure 8 As shown, the skeleton node detection module may perform the same processing as steps S1021-S1024 to obtain the coordinate information (X i , Y i , Z i ), and rotation information (x i ,y i , z i , w i ), as an example of a user posture data frame. In addition, the process of the model driving module obtaining the model posture data frame F1 is as follows: Figure 8 As shown, the model driving module can specifically perform the same processing as steps S1025 and S1026 to obtain the model posture data frame F1. The specific process is as shown above. Figure 4B Some of the contents have been described above and will not be repeated here.

[0155] In another implementation of the present application, the time information of the wall model 120 approaching the game character model 110 can also be used as an example of the status information between the wall model 120 and the game character model 110. Through this time information, it can be determined whether the state between the wall model 120 and the game character model 110 is a critical state, thereby determining whether the control scene is a critical scene. For example, if the time required for the wall model 120 to approach the game character model 110 is greater than a preset time threshold, the large-screen device 100 considers that the state between the wall model 120 and the game character model 110 is a non-critical state, and the current game scene is a general scene. If the time required for the wall model 120 to approach the game character model 110 is equal to or less than the preset time threshold, the large-screen device 100 considers that the state between the wall model 120 and the game character model 110 is a critical state, and the current game scene is a critical scene.

[0156] The time threshold may be in the range of 0.3s to 1s, for example, 0.3s, 0.5s, 1s, etc. Currently, the enterprise may set it to other values ​​as needed.

[0157] In another implementation of the present application, the large-screen device 100 may also determine whether the control scene is a key scene based on the type information of the control scene. For example, if the large-screen device 100 determines that the type of the control scene is a preset key scene type based on the type information of the control scene, the control scene is determined to be a key scene. The key scene type may be, for example, a scene of "in the wall-penetrating game, posture example 121 is a squatting posture", or a scene of "in the wall-penetrating game, posture example 121 is a jumping posture", etc. Of course, the key scene type may also be other scene types, which can be specifically set according to the control scene and needs. In this way, the different model control needs of users can be met, and the user experience can be effectively improved.

[0158] In another implementation of the present application, the large-screen device 100 can also determine whether the control scene is a key scene based on the type information of the target object. For example, if the large-screen device 100 determines that the type of the target object is a preset key object type based on the type information of the target object, the control scene is determined to be a key scene. The key object type can be, for example, a specific game character in a game scene, which can be specifically set according to the control scene and needs. In this way, the different model control needs of users can be met, effectively improving the user experience.

[0159] In another implementation of the present application, the large-screen device 100 may also determine whether the control scene is a key scene based on the user's type information. For example, if the large-screen device 100 determines that the user type is a preset key user type based on the user's type information, the control scene is determined to be a key scene. Key user types can be, for example, specific users such as children and the elderly, which can be specifically set according to the control scene and needs. In this way, the different model control needs of different users can be met, effectively improving the user experience.

[0160] In some other implementations of the present application, the aforementioned first scene and second scene can be selected and set according to the specific application scenario, wherein the second scene has higher requirements for the posture accuracy of the target object to be controlled than the first scene has for the posture accuracy of the target object.

[0161] In other implementations of the present application, Figure 9 As shown, the user 200 can wear a posture sensing device (also called a motion sensing device), for example, worn on key bone nodes such as the arms, legs, waist, and neck of the user 200. During the movement of the user 200, the posture sensing device collects the posture sensing data (also called motion data) of the user 200. And the posture sensing device sends the posture sensing data to the large-screen device 100 via wireless communication methods such as Bluetooth and WiFi, or wired methods. The large-screen device 100 performs bone node data detection and recognition based on the posture sensing data to obtain the aforementioned model posture data frame F1, or performs posture recognition based on the posture sensing data to determine whether a posture label can be obtained, thereby obtaining the model posture data frame F2.

[0162] In this implementation, the posture sensing device can be a device including a three-axis gyroscope or a three-axis accelerometer for detecting and identifying the user's movement, or other devices based on other principles for detecting and identifying the user's movement, which can be selected as needed.

[0163] In other implementations of the present application, the object control method based on human posture recognition provided by the present application can also be used for controlling virtual game models in other game scenes. For example, in a football game scene, the player model in the game (as another example of the target object) can be controlled to kick the football (as another example of the corresponding posture guidance model). In addition, if the football is far away from the player model, the large-screen device 100 can determine the skeleton node information of the user 200 based on the posture information of the user 200, and determine the model posture information based on the skeleton node information, and then control the player model to present a posture corresponding to the model posture information based on the model posture information. In this way, the large-screen device 100 can control the movement of the player model according to the actual posture of the user 200, so that the posture of the player model is consistent with the actual posture of the user 200, that is, the player model can be made to move following the movement of the user 200, thereby ensuring the user's immersive experience and improving the user's experience. If the football is close to the player model, it is considered a key scene. The large-screen device 100 controls the movement of the player model based on the preset model posture information obtained from the posture library, which can make the player model's posture more accurate, that is, it can better meet the posture requirements in key scenes, ensure the user's gaming experience, and enhance the user experience.

[0164] Of course, the object control method based on human posture recognition provided in this application can also be used in other ball game scenarios, such as playing basketball, or in other scenarios that require model control. The aforementioned key scenes, object posture information, etc. can be selected and set according to different control scenarios. In addition, the aforementioned game character model 110 can also be an anthropomorphic animal or other object that needs to be controlled.

[0165] In other implementations of the present application, the object control method based on human posture recognition provided by the present application can also be used for the control of other virtual models. For example, in movie digital special effects, the virtual model can be controlled to present the corresponding posture by the collected actor's posture information. In this scenario, if the virtual model needs to present more complex postures such as leaping, the current control scene can be considered as a key control scene, and it can be determined based on the user's posture information whether the corresponding model posture information can be obtained from a preset posture library so that the virtual model moves according to the model posture information. If the virtual model presents some simple postures, it is determined that the current control scene is not a preset key scene, i.e., a general scene, and the virtual model can determine the user's skeletal node information based on the posture information, and determine the model posture information based on the skeletal node information, and then control the virtual model to present a posture that matches the model posture information based on the model posture information.

[0166] In other implementations of the present application, the object control method based on human posture recognition provided by the present application can also be used for controlling physical model objects such as robots. For example, the robot controller obtains the user's posture information through a camera, and determines the control method based on the posture information and the current control scene. For example, if the current control scene is that the robot needs to perform some complex jumping and other postures, the robot controller determines that the current control scene is a preset key scene, and determines whether the corresponding model posture information can be obtained from the preset posture library based on the user's posture information, so that the robot moves according to the model posture information. If the current control scene is that the robot walks normally, the robot controller determines that the current control scene is not a preset key scene, and the robot controller can determine the user's skeletal node information based on the posture information, and determine the model posture information based on the skeletal node information, and then control the robot to present a posture that matches the model posture information based on the model posture information.

[0167] In some implementations of the present application, the camera and the robot controller may be integrated with the robot body.

[0168] In some other implementations of the present application, the camera and the robot controller may also be provided separately from the robot body. The camera sends the collected user's posture information to the robot controller, and the robot controller controls the movement of the robot.

[0169] In other implementations of the present application, the robot controller may also obtain the user's posture information through the aforementioned posture sensor.

[0170] In other implementations of the present application, a separately set camera or posture sensor can send the acquired user posture information to a control device such as the aforementioned large-screen device, robot, server, mobile terminal, etc., and then the control device determines the posture of the target object based on the posture information and controls the target object to move.

[0171] In summary, the object control method based on human posture recognition provided by this implementation is an object control method based on human posture recognition based on posture tag recognition plus skeleton node detection and recognition, which can be applied to model control scenarios in the fields of games, virtual reality, movie digital special effects, human-computer interaction, etc. The object control method based on human posture recognition realizes model control by dynamically combining posture recognition and skeleton node detection and recognition. On the one hand, in general scenarios, the movement of the target object can be controlled according to the object posture information corresponding to the user's actual posture, so that the target object can accurately restore the user's posture, that is, the target object moves with the user's actions, which can enhance the user's immersive experience. On the other hand, in a specified scenario (i.e., a key scenario), the action of the target object is corrected by combining posture recognition, that is, the movement of the target object can be controlled according to the object posture information in the preset posture library, thereby meeting certain object control needs, ensuring the object control effect, and enhancing the user experience.

[0172] This means that the control method for the target object can be determined based on the specific control scenario, and the target object can be controlled in real time or on demand. Compared to methods that control the model to present corresponding postures based solely on actual human posture data, or methods that directly identify the user's posture based on actual human posture data and play preset corresponding model posture images, this method can meet the model control requirements in different control scenarios and effectively improve the user experience.

[0173] In addition, compared to obtaining user posture-related data based on cameras, posture sensors and other devices, dynamically constructing and rendering human body areas (such as Xbox, after obtaining image data through Kinect, it constructs and renders a human-shaped shadow area), and completing some two-dimensional model collision-type games (such as cutting fruit) through this area, the object control method based on human posture recognition provided by this application can provide more game scenes and avoid the problem of single game scenes; and can provide a controlled model for interacting with the user, avoiding the problem of being unable to control a specified model; it can also effectively improve the user experience.

[0174] In other implementations of the present application, the object control method based on human posture recognition provided by the present application can be applied to mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MIDs), wearable devices (for example, including: smart watches, smart bracelets, pedometers, etc.), personal digital assistants, portable media players, navigation devices, video game devices, set-top boxes, virtual reality and / or augmented reality devices, Internet of Things devices, industrial control equipment, streaming media client devices, robots and other electronic devices.

[0175] See Figure 10 , Figure 10 The figure shows a schematic diagram of the structure of an electronic device 900 provided according to one embodiment of the present application. The electronic device 900 may include one or more processors 901 coupled to a controller hub 904. For at least one embodiment, the controller hub 904 communicates with the processor 901 via a multi-drop bus such as a front-side bus (FSB), a point-to-point interface such as a QuickPath Interconnect (QPI), or a similar connection. The processor 901 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 904 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown in the figure) and an input / output hub (IOH) (which may be on a separate chip) (not shown in the figure), wherein the GMCH includes a memory and a graphics controller and is coupled to the IOH.

[0176] The electronic device 900 may also include a coprocessor 906 and a memory 902 coupled to the controller hub 904. Alternatively, one or both of the memory 902 and the GMCH may be integrated within the processor 901 (as described herein), with the memory 902 and the coprocessor 906 coupled directly to the processor 901 and the controller hub 904, with the controller hub 904 being in a single chip with the IOH.

[0177] The memory 902 may be, for example, a dynamic random access memory (DRAM), a phase change memory (PCM), or a combination of the two.

[0178] In one embodiment, the coprocessor 906 is a special-purpose processor, such as, for example, a high-throughput Many Integrated Core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose graphics processing unit (GPGPU), or an embedded processor. The optional nature of the coprocessor 906 is indicated by a dashed line in FIG. Figure 10 middle.

[0179] In one embodiment, the electronic device 900 may further include a network interface card (NIC) 903. The network interface 903 may include a transceiver for providing a radio interface for the electronic device 900, thereby communicating with any other suitable device (such as a front-end module, antenna, etc.). In various embodiments, the network interface 903 may be integrated with other components of the electronic device 900. The network interface 903 may implement the functions of the communication unit in the above-mentioned embodiments.

[0180] The electronic device 900 may further include input / output (I / O) devices 905. The I / O devices 905 may include: a user interface designed to enable a user to interact with the electronic device 900; a peripheral component interface designed to enable peripheral components to interact with the electronic device 900; and / or sensors designed to determine environmental conditions and / or location information related to the electronic device 900.

[0181] It is worth noting that Figure 10 This is for illustrative purposes only. Figure 10 It is shown that the electronic device 900 includes multiple devices such as a processor 901, a controller hub 904, a memory 902, etc. However, in actual applications, the devices using the methods of the present application may only include a part of the devices of the electronic device 900, for example, it may only include the processor 901 and the NIC 903. Figure 10 The properties of the optional devices are shown with dotted lines.

[0182] One or more tangible, non-transitory computer-readable media for storing data and / or instructions may be included in the memory of the electronic device 900. The computer-readable storage medium stores instructions, and more specifically, stores temporary and permanent copies of the instructions.

[0183] In the present application, the electronic device 900 may be a terminal device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), or a desktop computer. The instructions stored in the memory of the electronic device may include instructions that, when executed by at least one unit in the processor, cause the electronic device to implement the object control method based on human gesture recognition as mentioned above.

[0184] See Figure 11 , Figure 11 FIG2 is a schematic diagram of the structure of a SoC (System on Chip) 1000 provided according to an embodiment of the present application. Figure 111000. Similar components have the same reference numerals. Furthermore, dashed boxes indicate optional features of a more advanced SoC 1000. The SoC 1000 can be used in any electronic device according to the present application, and can implement corresponding functions depending on the device in which it is located and the instructions stored therein.

[0185] exist Figure 11 In the embodiment, SoC 1000 includes: an interconnect unit 1002 coupled to a processor 1001; a system agent unit 1006; a bus controller unit 1005; an integrated memory controller unit 1003; a set of one or more coprocessors 1007, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1008; and a direct memory access (DMA) unit 1004. In one embodiment, the coprocessor 1007 includes a special-purpose processor, such as a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0186] The SRAM unit 1008 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of the instructions. The instructions may include instructions that, when executed by at least one unit in the processor 1001, cause the electronic device to implement the object control method based on human gesture recognition as described above.

[0187] It should be noted that the terms "first", "second", etc. are only used for distinction and description, and cannot be understood as indicating or implying relative importance.

[0188] It should be noted that in the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0189] Although the present application has been illustrated and described with reference to certain preferred embodiments of the present application, those skilled in the art will appreciate that the above descriptions are provided as further details of the present application in conjunction with specific embodiments, and that the specific implementation of the present application should not be limited to these descriptions. Those skilled in the art may make various changes in form and detail, including simple deductions or substitutions, without departing from the spirit and scope of the present application.

Claims

1. An object control method based on human posture recognition, applied to electronic equipment, characterized in that: The method comprises: Get user's posture information; In the first scenario, determining the user's skeleton node information according to the posture information, and determining the object's posture information according to the skeleton node information; In the second scenario, if it is determined according to the posture information that the posture of the user is a posture in a preset posture library that matches the second scenario, then obtaining preset object posture information corresponding to the user's posture from the posture library; otherwise, determining the user's skeleton node information according to the posture information, and determining the object posture information according to the skeleton node information; The object posture information is used to control the target object to present a posture corresponding to the object posture information, and the second scene has a higher requirement on the posture accuracy of the target object than the first scene.

2. The object control method based on human posture recognition according to claim 1, characterized in that: The method further includes determining the type of the scene where the target object is located by: If it is determined that the scene where the target object is located is a preset key scene, then the scene where the target object is located is determined to be the second scene; otherwise, the scene where the target object is located is determined to be the first scene.

3. The object control method based on human posture recognition according to claim 2, characterized in that: The method further includes determining the scene where the target object is located as the key scene by: If, based on the status information between the target object and the preset posture guidance object, it is determined that the status between the target object and the preset posture guidance object is a preset key status, then the scene where the target object is located is determined to be the key scene, wherein the posture guidance object is an object in the scene where the target object is located, which is used to guide the posture presented by the user.

4. The object control method based on human posture recognition according to claim 3, characterized in that: The state information is the distance information between the target object and the posture guidance object, and the key state is that the distance between the target object and the posture guidance object is less than or equal to a preset distance threshold; or The state information is time information required for the target object and the posture guidance object to approach each other, and the key state is that the time required for the target object and the posture guidance object to approach each other is less than or equal to a preset time threshold.

5. The object control method based on human posture recognition according to claim 2, characterized in that: The method further includes determining the scene where the target object is located as the key scene by: If it is determined according to the type information of the target object that the type of the target object is a preset key object type, then determining the scene where the target object is located as the key scene; If it is determined, based on the user type information, that the user type is a preset key user type, then the scene where the target object is located is determined to be the key scene.

6. The object control method based on human posture recognition according to any one of claims 1 to 5, characterized in that: Determining, based on the posture information, that the user's posture is a posture in a preset posture library that matches the second scene, comprising: if posture recognition is performed based on the posture information to obtain a preset posture label, determining that the user's posture is a posture in the posture library that matches the second scene; Acquiring object posture information corresponding to the user's posture from the posture library includes: acquiring object posture information corresponding to the posture tag from the posture library according to the posture tag.

7. The object control method based on human posture recognition according to claim 6, characterized in that: Performing posture recognition according to the posture information to obtain a posture label includes: performing posture recognition according to the posture information; If the matching degree between the recognized user posture and the posture corresponding to the second scene in the posture library is greater than or equal to a preset matching degree threshold, the posture label corresponding to the posture information is obtained.

8. The object control method based on human posture recognition according to claim 1, characterized in that: Determining the user's skeleton node information according to the posture information, and determining the object's posture information according to the skeleton node information, including: Performing skeleton node detection and recognition according to the posture information to obtain the skeleton node information; Determine user posture data according to the skeleton node information; The object posture information is obtained by performing motion redirection according to the user posture data.

9. The object control method based on human posture recognition according to claim 8, characterized in that: The skeleton node information includes coordinate information of the key skeleton nodes of the user, and the user posture data includes the coordinate information and rotation information of the key skeleton nodes.

10. The object control method based on human posture recognition according to claim 1, characterized in that: The object posture information is an object posture data frame; Controlling the target object to present a posture corresponding to the object posture information according to the object posture information includes: The target object is controlled to move from a posture corresponding to a current object posture data frame to a posture corresponding to the object posture data frame according to the object posture data frame.

11. The object control method based on human posture recognition according to claim 10, characterized in that: Controlling the target object to move from a posture corresponding to the current object posture data frame to a posture corresponding to the object posture data frame according to the object posture data frame includes: determining, based on the current object posture data frame and the object posture data frame, a time delay for the target object to move from a posture corresponding to the current object posture data frame to a posture corresponding to the object posture data frame, and determining an intermediate object posture data frame; According to the time delay, the intermediate object posture data frame and the object posture data frame, an object motion animation including the intermediate object posture data frame and the object posture data frame is formed and displayed by interpolation.

12. The object control method based on human posture recognition according to claim 1, characterized in that: Acquiring the posture information of the user includes: Acquiring a human body image data frame of the user as the posture information through a camera; or The posture sensing data of the user is obtained through a posture sensor worn by the user as the posture information.

13. An electronic device, characterized in that: include: a memory for storing a computer program, wherein the computer program includes program instructions; A processor is used to execute the program instructions so that the electronic device performs the object control method based on human posture recognition according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. The program instructions are executed by an electronic device to enable the electronic device to execute the object control method based on human posture recognition according to any one of claims 1 to 12.

Images