Method and related device for realizing body posture comparison

Abstract

The present disclosure provides a method and related devices for realizing body posture comparison. The method includes: determining the identity information of a user; obtaining a template body model corresponding to the user according to the identity information of the user; obtaining first image data including a full-body image of the user; processing the first image data to obtain joint position information of the user; adjusting the template body model according to the joint position information; and comparing and displaying the first image data and the adjusted template body model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a method and related devices for realizing body posture comparison. Background Art

[0002] In daily scenarios such as buying clothes, losing weight, and fitness, people need to compare with clothing, standard body shapes, etc. In addition, in some gyms and dance studios, comparison and reference of body movements are generally required. Summary of the Invention

[0003] The present disclosure provides a method and related devices for realizing body posture comparison.

[0004] In a first aspect of the present disclosure, there is provided a method for realizing body posture comparison, including:

[0005] Determine the identity information of the user;

[0006] Obtain the template body shape model corresponding to the user according to the identity information of the user;

[0007] Obtain first image data including a full-body image of the user;

[0008] Process the first image data to obtain the joint position information of the user;

[0009] Adjust the template body shape model according to the joint position information; and

[0010] Compare and display the first image data and the adjusted template body shape model.

[0011] In a second aspect of the present disclosure, there is provided an electronic device, characterized by including one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect.

[0012] In a third aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to execute the method according to the first aspect.

[0013] In a fourth aspect of the present disclosure, there is provided a computer program product, including computer program instructions, which, when run on a computer, cause the computer to execute the method according to the first aspect.

[0014] The method and related devices for realizing body posture comparison provided by the present disclosure adjust the template body model by using the joint position information of the user, so that a body posture comparison picture more fitting the current action state of the user can be provided during body posture comparison, thereby better body posture comparison effect can be given and the user experience is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following description are only the embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1A FIG. shows a schematic diagram of the hardware structure of the exemplary electronic device provided in this embodiment.

[0017] Figure 1B FIG. shows a schematic diagram of the exemplary memory according to an embodiment of the present disclosure.

[0018] Figure 1C FIG. shows a schematic diagram of the exemplary user information according to an embodiment of the present disclosure.

[0019] Figure 1D FIG. shows a schematic diagram of using a depth camera to collect image data of a user according to an embodiment of the present disclosure.

[0020] Figure 2A FIG. shows a schematic diagram of obtaining joint position information according to an embodiment of the present disclosure.

[0021] Figure 2B FIG. shows a schematic diagram of the exemplary image according to an embodiment of the present disclosure.

[0022] Figure 2C FIG. shows a schematic diagram of the adjusted template body model according to an embodiment of the present disclosure.

[0023] Figure 2D FIG. shows a schematic diagram of the exemplary interface according to an embodiment of the present disclosure.

[0024] Figure 3 FIG. shows a schematic flowchart of the exemplary method provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the following will further describe the present disclosure in detail with reference to specific embodiments and the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the ordinary meanings understood by those of ordinary skill in the art to which the present disclosure belongs. The "first", "second" and similar terms used in the embodiments of the present disclosure do not denote any order, quantity or importance, but are only used to distinguish different components. Words such as "including" or "comprising" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Upper", "lower", "left", "right", etc. are only used to represent relative position relationships, and when the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0027] The embodiments of the present disclosure provide an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, a method for realizing posture comparison is implemented, so as to realize automatic posture comparison of human body images.

[0028] Figure 1A FIG. shows a schematic hardware structure diagram of an exemplary electronic device 100 provided in this embodiment. The device may include: a processor 102, a memory 104, an input / output interface 106, a communication interface 108, and a bus 110. Among them, the processor 102, the memory 104, the input / output interface 106, and the communication interface 108 are communicatively connected to each other inside the device through the bus 110.

[0029] The processor 102 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. The processor 102 may be used to execute functions related to the technologies described in the present disclosure. In some embodiments, the processor 102 may further include multiple processors integrated as a single logic component.

[0030] The memory 104 may be configured to store data (e.g., instructions, computer code, etc.). As Figure 1AAs shown, the data stored in the memory may include program instructions (e.g., program instructions for implementing the method for posture comparison according to the present disclosure) and data to be processed (e.g., the memory may store documents of logical system design, etc.). The processor 102 can also access the program instructions and data stored in the memory and execute the program instructions to operate on the data to be processed. The memory 104 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 104 may include a random access memory (RAM), a read-only memory (ROM), an optical disc, a magnetic disk, a hard disk, a solid state drive (SSD), a flash memory, a memory stick, etc.

[0031] The input / output interface 106 is used to connect to the input / output module to connect the device 100 to one or more peripheral devices, thereby realizing information input and output. The input / output module may be configured as a component in the device 100 (not shown in the figure) or externally connected to the device 100 to provide corresponding functions. The peripheral devices may include input devices such as a keyboard, a mouse, a touch screen, a microphone, various sensors, etc. In some embodiments, as Figure 1A shown, the input device may further include a camera 112 for collecting images of the user. The peripheral devices may further include output devices such as a display screen (or display panel / display) 114, a speaker, a vibrator, an indicator light, etc. In some embodiments, the display screen 114 may provide a display screen for displaying the user's image and the template image.

[0032] The communication interface 108 is used to connect to a communication module (not shown in the figure) to realize the communication interaction between the device 100 and other devices. The communication module may communicate through a wired manner (e.g., USB, network cable, etc.) or through a wireless manner (e.g., mobile network, WIFI, Bluetooth, etc.). It can be understood that the communication manner is not limited to the above specific examples. In some embodiments, the communication interface 108 may include any combination of any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc.

[0033] The bus 110 includes a path for transmitting information between various components of the device (e.g., the processor 102, the memory 104, the input / output interface 106, and the communication interface 108). The bus 110 may include an internal bus (e.g., the processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0034] Note that although the above device only shows the processor 102, the memory 104, the input / output interface 106, the communication interface 108, and the bus 110, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0035] In some embodiments, the electronic device 100 may be an all-in-one machine with a display function and may have a large-size display screen (for example, 75 inches, 80 inches, 82 inches, 85 inches, 98 inches, 120 inches). The electronic device 100 can be applied in various occasions. For example, it can help users compare their body shapes in a gym. Another example is that in a dance studio or a gym, it can compare the user's movements with the template movements.

[0036] To implement body posture comparison, in some embodiments, the electronic device 100 may first input the relevant information of the user. For each user, their information can be input accordingly. Figure 1B Shows a schematic diagram of an exemplary memory 104 according to an embodiment of the present disclosure. As Figure 1B shown, the memory 104 can store the user information of multiple users. For example, user information 1042, 1044, 1046, 1048. Each user information corresponds to a user respectively. After the electronic device 100 inputs the information of the user, it stores the information in the memory 104. In some embodiments, the memory 104 for storing user information may be an independent database and may be implemented by an independent external device, and data is transmitted through interaction with the electronic device 100.

[0037] The user information input by the electronic device 100 can be various types of information. Figure 1C Shows a schematic diagram of an exemplary user information 1042 according to an embodiment of the present disclosure.

[0038] As Figure 1C shown, the user information 1042 may include basic information 10422. The basic information 10422 may further include the name, age, gender, and height of the user. The user can directly input the basic information 10422 into the electronic device 100 to provide it to the electronic device for storage.

[0039] In some embodiments, as Figure 1CAs shown, in order to implement user identity recognition, the user information 1042 may further include the user's face information. In this way, the electronic device 100 can subsequently identify the user's identity through face recognition. When entering the face information 10424, the user can first send a face information collection instruction to the electronic device 100 (for example, send this instruction to the electronic device 100 through a mouse, keyboard, or voice), and then the electronic device 100 can turn on the camera 112. When a face is detected, the electronic device 100 can start collecting the user's face image. In some embodiments, in order to ensure the accuracy of the face information 10424, the electronic device 100 can collect 360-degree face images. For example, at the initial angle, the user's face image is collected. After the collection at this angle is completed, the user is prompted by voice to rotate a certain angle and perform the face image collection at the next angle until the 360-degree face image collection is completed. After the collection is completed, the electronic device 100 can correspondingly enter the face information 10424 into the user's user information 1042.

[0040] In some embodiments, the electronic device 100 can also enter the user's body model for subsequent body posture comparison. As Figure 1C shown, the user information 1042 may include the user's body model 10426.

[0041] In some embodiments, the electronic device 100 can use the images collected by the depth camera to create the body model 10426. The depth camera can be a component of the electronic device 100 or an external device of the electronic device 100. In this embodiment, the case where the depth camera is provided in the electronic device 100 as one of its components is taken as an example for description. For example, the depth camera can be provided above the display screen of the electronic device 100.

[0042] Before using the depth camera to capture the user's image, the electronic device 100 can first capture the user's face image, and then use the aforementioned face information 10424 for face recognition to ensure that the current user is the user corresponding to the user information 1042. After the user's identity recognition is successful, the electronic device 100 can enter the body information collection mode, turn on the depth camera, and at the same time can display the interface for using the depth camera to capture the user's image.

[0043] Figure 1D Shows a schematic diagram of using the depth camera 120 to collect the image data of the user 130 according to an embodiment of the present disclosure. As Figure 1D shown, the electronic device 100 can use the depth camera 120 to collect image data including the full-body image of the user 130 (for example, the third image data).

[0044] When collecting image data including a full-body image of the user 130, a 360-degree full-body image of the user 130 can be collected. For the convenience of collection, the user can use a voice recognition interaction method to assist in completing the collection during the collection process. For example, the user being scanned stands at a position about 2m in front of the display screen 114 and issues a voice command of "start collection" to enable the electronic device 100 to control the depth camera 120 to collect an image of the user. After collecting an image at one angle, the user adjusts the position (for example, rotates about 45 degrees in place), and after the adjustment is completed, issues a voice command of "continue collection" to enable the electronic device 100 to control the depth camera 120 to collect an image of the user again. And so on, until the user rotates 360 degrees in place, that is, 8 times of data are collected, and the depth camera 120 completes the data collection.

[0045] After the depth camera 120 completes the image collection process of the user 130, the electronic device 100 can obtain a set of image collections including multiple depth images 142 and multiple color images (for example, RGB images) 144. Among them, the RGB image 144 provides the x and y coordinates in the pixel coordinate system, while the depth image 142 provides the z coordinate in the camera coordinate system, that is, the distance between the camera and the point.

[0046] Therefore, the electronic device 100 can generate the point cloud data of the user 130 based on the multiple depth images 142 and the multiple color images (for example, RGB images) 144.

[0047] Specifically, according to the information of the RGB image 144 and the internal parameters of the depth camera 120, the coordinates of any pixel point in the camera coordinate system can be calculated. Then, according to the information of the RGB image 144 and the internal and external parameters of the depth camera 120, the coordinates of any pixel point in the world coordinate system can be calculated. Within the camera's field of view, the coordinates of the obstacle points in the camera coordinate system are the point cloud sensor data, that is, the point cloud data in the camera coordinate system. The point cloud sensor data can be calculated based on the coordinates provided by the RGB image 144 and the internal parameters of the depth camera 120. Then, the coordinates of all the obstacle points in the world coordinate system are the point cloud map data, that is, the point cloud data in the world coordinate system. The point cloud map data can be calculated based on the coordinates provided by the RGB image 144 and the internal and external parameters of the depth camera 120. In this way, the point cloud data of the user 130 is obtained.

[0048] After obtaining the point cloud data, the electronic device 100 can also preprocess the point cloud data. For example, the background of the point cloud data can be removed and smoothed and denoised through an image processing algorithm, and then the preprocessed point cloud data is obtained. Next, the joint bilateral filtering method is used to repair the depth data in the preprocessed point cloud data to obtain the repaired point cloud data. Then, the repaired point cloud data is registered to obtain the registered point cloud data. Finally, the Poisson surface reconstruction method is used to complete the surface reconstruction of the registered point cloud data and map the texture to obtain the three-dimensional (3D) body model of the user. Among them, registering the repaired point cloud data may include coarse registration based on the fast point feature histogram (FPFH), fine registration using the iterative closest point (ICP) algorithm for searching the closest points of adjacent two frames of point clouds, and non-rigid registration based on the deformation map.

[0049] In some embodiments, as Figure 1C shown, the user can also enter the template body information 10428 corresponding to their ideal body type into their user information 1042. In some embodiments, the memory 104 of the electronic device 100 (for example, it can include a database of peripherals) can pre-store a three-dimensional model library of multiple body types. In this three-dimensional database, each height and weight corresponds to 6 three-dimensional models of different body types, including 3 male models and 3 female models. The user can input their ideal weight in the electronic device 100, and then the electronic device 100 selects 3 standard body types that match the user's basic information 10422 (such as height, gender) and their ideal weight from this three-dimensional model library. The user can select a template body for body posture comparison as their template body model 10428.

[0050] After completing the input of the user information, the electronic device 100 can use this user information to achieve body posture comparison.

[0051] In the initial state, the user can send a body posture comparison instruction (for example, issue this instruction through a mouse, keyboard, or voice) to the electronic device 100 to enter the body posture comparison mode.

[0052] Before performing the body posture comparison, the electronic device 100 can first determine the user's identity information. For example, after entering the body posture comparison mode, the electronic device 100 can first turn on the camera 112, then collect the user's face image through the camera 112 and perform face recognition, and then determine the user's identity information based on the face recognition result.

[0053] Then, the electronic device 100 can obtain the template body model of the user according to the identity information of the user. For example, if the aforementioned face recognition result matches the face information 10424, the corresponding user information 1042 can be determined, and then the template body model 10428 of the user can be found based on it.

[0054] Next, the electronic device 100 can further obtain the joint position information of the user.

[0055] In some embodiments, the electronic device 100 can further use the following method to obtain the joint position information of the user. Figure 2A The schematic diagram of obtaining joint position information according to an embodiment of the present disclosure is shown.

[0056] First, the electronic device 100 can collect the image data 202 (e.g., the first image data) including the whole body image of the user again. The image data 202 can be a set of images. For example, the set can be the images collected by a camera during the user's movement, obtaining 20 frames per second, and then generating a frame sequence. Then, according to the image data 202, the human body image 204 of the user is processed. Next, based on the human body image 204, the human body skeleton model 206 of the user is constructed. Finally, the spatio-temporal features of the joint vectors are extracted from the human body skeleton model 206 as the joint position information 208 of the user. Among them, the spatio-temporal features of the joint vectors can include direction cosine features, included angle cosine features, and angle change rate features.

[0057] In some embodiments, the electronic device 100 can further use the following method to process and obtain the human body image 204 of the user.

[0058] First, the first image data is converted from a color image to a grayscale image. Then, after processing the grayscale image using the Sobel edge detection operator, the local binary pattern method (LPB) is used to detect the human body contour. Next, the linear support vector machine (SVM) classifier excludes a predetermined proportion (e.g., 90%) of the regions in the human body contour, and then the nonlinear support vector machine classifier filters the remaining partial regions of the human body contour to obtain the human body image of the user. In some embodiments, if the image data 202 can be a set of images, the steps of processing to obtain the human body image 204 of the user can be to process each image separately.

[0059] In some embodiments, the electronic device 100 can further use the following method to construct the human body skeleton model 206 of the user. First, use OpenPose (an open-source human pose recognition project on Github) to obtain the two-dimensional coordinate information of the joint points in the human body image of the user. As Figure 2AAs shown, it can be seen that when constructing the human skeleton, the number of joint points is 24. However, since some joint points of the user's body in the scene may be occluded, and due to the limitations of the OpenPose algorithm model, data loss of some joint points may occur, making it difficult to obtain the information of 24 joint points when processing a frame of image. Therefore, a forward and backward frame correlation algorithm can be used for linear regression to handle missing values, and then the two-dimensional coordinate information of the joint points obtained by OpenPose can be complemented. For example, by combining the image frames before and after the current frame, the two-dimensional coordinate information of the respective joint points is identified, and then the data of the current frame is complemented using the data obtained from the forward and backward frames. Then, the electronic device 100 can construct the human skeleton model 206 based on the complemented two-dimensional coordinate information of the joint points.

[0060] Thus, the electronic device 100 can obtain the joint position information 208 of the user.

[0061] Since the user's template body model 10428 is modeled based on a static human body. Therefore, for a dynamic user, the template body model 10428 can be adjusted according to the joint position information during their movement.

[0062] Therefore, after obtaining the joint position information 208, the electronic device 100 can adjust the template body model 10428 according to the joint position information 208. For example, by changing the joint change angles of the template body model 10428 based on the direction cosine feature, included angle cosine feature, and angle change rate feature in the joint position information 208, etc., so that the template body model 10428 matches the current movement state of the user.

[0063] After adjusting the template body model 10428, the electronic device 100 can compare and display the current image of the user (for example, the current frame in the first image data used to obtain the joint position information 208) and the adjusted template body model on the display screen 114. Figure 2B and Figure 2C respectively show schematic diagrams of the current frame image 210 and the adjusted template body model 212 according to an embodiment of the present disclosure.

[0064] Figure 2D shows a schematic diagram of an exemplary interface 214 according to an embodiment of the present disclosure. As Figure 2DAs shown, when the electronic device 100 compares and displays the current frame image 210 and the adjusted template body model 212 in the interface 214, it can align the contour image of the user in the current frame image with the central axis of the template body model 212, so as to display the template body model 212 in the interface 214 according to the obtained user posture. Among them, for the purpose of distinguishing and comparing, the template body model 212 can be gray with a transparency of 50%. In this way, a 3D body posture comparison diagram is formed.

[0065] In some embodiments, the electronic device 100 can also collect image data including the full-body image of the user in real time, and based on the above method, adjust the template body model 212 in real time for comparison and display, so as to realize real-time 3D body posture comparison from multiple angles.

[0066] In some embodiments, during the above comparison and display process, the electronic device 100 can also receive a first voice command from the user to adjust the display screen, and according to this first voice command, adjust the display screen in real time. Adjusting the screen through voice commands enables the user not to need to operate hardware devices such as a mouse or keyboard during the body posture comparison process, and the body posture comparison process will not be interrupted, making it more convenient to use.

[0067] In some embodiments, the first voice command can include commands such as zoom in, zoom out, turn left, turn right, and restore. When these commands need to be issued, the user only needs to say "zoom in", "zoom out", "turn left", "turn right", "restore".

[0068] Among them, the "zoom in" command can zoom in the current screen to 110%, and the portrait is centered. When the "zoom in" command is issued again, it can be zoomed in to 120%, 130%, 140% until 200% in sequence. When it is zoomed in to 200%, after receiving the "zoom in" command again, it will not be zoomed in anymore.

[0069] The "zoom out" command can zoom out the screen when it is zoomed in, each time reducing by 10% until it is reduced to the original screen ratio. After receiving the "zoom out" command again, it will not be zoomed out anymore.

[0070] The "turn left" command can turn the contour image of the user in the screen and the template body model 212 around the central axis to the left by 15 degrees.

[0071] The "turn right" command can turn the contour image of the user in the screen and the template body model 212 around the central axis to the right by 15 degrees.

[0072] The "restore" command can restore to the original screen ratio and direction after zooming in, zooming out, or rotating the screen. When the current is the original screen ratio and direction, after receiving the "restore" command, no adjustment is made.

[0073] In some scenarios, such as dance teaching, fitness, etc., it may be necessary to provide standard actions for users to refer to, or to compare the user's actions with the standard actions.

[0074] Therefore, in some embodiments, the electronic device 100 can first identify the user's identity (e.g., face recognition), and then, the electronic device 100 can receive a motion template comparison instruction from the user (e.g., the user selects a certain motion template); then, according to the motion template comparison instruction, the corresponding motion template is determined. The motion template can be a template for dance or fitness motions, and the joint motion information of these motions is stored in the database of the electronic device 100.

[0075] Next, the electronic device 100 can accordingly obtain the joint motion information corresponding to the motion template, and adjust the user's body model according to the joint motion information corresponding to the motion template (e.g., Figure 1C the body model 10426) and display it. That is, the joint motion information of the motion template is combined with the user's body model 10426 and then displayed on the display screen 114. In this way, since the motion template and the user have the same body shape, it is convenient for the user to compare and learn.

[0076] In scenarios such as dance teaching, the template library may not cover all actions. Therefore, a way is needed to enable users to create custom motion templates. Therefore, in some embodiments, the electronic device 100 can receive the user's template creation instruction (e.g., the user clicks the custom template creation button, or, says "create custom template" by voice). Then, based on the template creation instruction, the electronic device 100 can determine whether it has received a second voice instruction from the user to capture the action image (e.g., the user says "start"). If the electronic device 100 receives the second voice instruction, it can start capturing the user's action image using the camera 114, and then make the captured user's action image into an action template.

[0077] In some embodiments, the user controls the start of recording through a voice instruction, and then can perform a series of consecutive actions. During the action process, these actions can be segmented through voice instructions and made into multiple action templates. During the action process, the camera captures the user's image, and then processes it to obtain the joint position information and saves it as an action template.

[0078] During the process of creating an action template, the user can also issue other voice commands. For example, when recording the nth action, the "save" command can save the current (nth) action to create an action template and simultaneously start recording the next (n + 1)th action; the "delete" command can delete the current (nth) action and restart recording the next action (the nth); the "end" command can save the current (nth) action to create an action template and end the recording.

[0079] In some other embodiments, an automatic following mode can also be adopted for body posture comparison. For example, the electronic device 100 can receive the user's automatic following command. For example, the user selects the automatic following mode. Then, the electronic device 100 can, according to this automatic following command, continuously collect second image data including the full-body images of the user within a preset time period (for example, within 20 s). Since this second image data is a set of images, the electronic device 100 can obtain the joint position information of the user based on each frame of the image in this second image data, and then connect them together to obtain the joint motion information of the user.

[0080] Then, the electronic device 100 can input the joint motion information of this user into the K-Nearest Neighbor (KNN) classifier for classification. It can be understood that this KNN classifier can be trained using a known action template database. Then, this KNN classifier can output a predicted template action. This predicted template action can be the template action with the highest similarity to the joint motion information of this user. Then, the electronic device 100 can, according to this predicted template action, combine the body model of this user (for example, Figure 1C the body model 10426), generate an action template consistent with the three-dimensional model of the user's body and display it on the display screen 114.

[0081] In some embodiments, during the user's movement, the electronic device 100 can real-time detect the joint motion information of the user, and then input this joint motion information into the aforementioned KNN classifier for classification to calculate the similarity. When the similarity between the joint motion information of the user and the predicted template action is lower than a preset threshold (for example, 70%), an error prompt message is output (for example, the color of the action template turns red).

[0082] In some embodiments, to facilitate the user to adjust the position, voice control commands can be further added. For example, when the user shouts "pause", the playback of the action template is paused (for example, stopping further generation of new action templates), and the screen stays at the current action posture. At the same time, the joint position information of the user is continuously detected and compared with the current action template. The user can adjust their actions according to the actions and colors of the current action template. After the adjustment is completed, through the "continue" voice command, the template action can be continued for action guidance.

[0083] The electronic device 100 provided by the embodiments of the present disclosure can help users perform 3D body posture comparison, and can also use customized templates for motion correction and custom input templates. At the same time, it also enriches the application scenarios of large-size terminals.

[0084] The electronic device 100 provided by the embodiments of the present disclosure extracts the joint motion information of the user to achieve real-time 3D body posture comparison, comparative analysis of joint motion information, and automatic matching of joint actions. At the same time, it can also achieve motion correction and automatic following based on a customized model and the production of custom action templates. In addition, it can also achieve voice control for adjusting the size and direction of the 3D portrait.

[0085] Based on a similar inventive concept, the embodiments of the present disclosure also provide a method for realizing body posture comparison. Figure 3 The flowchart of the exemplary method 300 provided by the embodiments of the present disclosure is shown. This method 300 can be implemented by the electronic device 100, and, as Figure 3 shown, this method 300 may include the following steps.

[0086] In step 302, the electronic device 100 may determine the identity information of the user (for example, Figure 1C basic information 10422).

[0087] In step 304, the electronic device 100 may obtain the template body model corresponding to the user according to the identity information of the user (for example, Figure 1C template body model 10428).

[0088] In step 306, the electronic device 100 may obtain first image data including a full-body image of the user.

[0089] In step 308, the electronic device 100 may process the first image data to obtain the joint position information of the user.

[0090] In some embodiments, processing the first image data to obtain the joint position information of the user may further include: processing the first image data (for example, Figure 2A image 202) to obtain the human body image of the user (for example, Figure 2A image 204); constructing the human body skeleton model of the user according to the human body image of the user (for example, Figure 2A human body skeleton model 206); and extracting the spatio-temporal features of the joint vector from the human body skeleton model as the joint position information of the user (for example, Figure 2Aof the joint position information 208). In some embodiments, the spatio-temporal features of the joint vector include direction cosine features, included angle cosine features, and angular rate of change features.

[0091] In some embodiments, according to the first image data, processing to obtain the user's body image may further include: converting the first image data into a grayscale image; processing the grayscale image using a Sobel edge detection operator to obtain an edge detection image; detecting the human body contour in the edge detection image using a local binary pattern method; excluding a predetermined proportion of areas in the human body contour using a linear support vector machine classifier to obtain a partial area of the human body contour; and screening the partial area of the human body contour using a non-linear support vector machine classifier to obtain the user's body image.

[0092] In some embodiments, according to the user's body image, constructing the user's body skeleton model may further include: using OpenPose to obtain the two-dimensional coordinate information of the joint points in the user's body image; using a front and back frame association algorithm to complete the two-dimensional coordinate information of the joint points to obtain the completed two-dimensional coordinate information of the joint points; and constructing the body skeleton model according to the completed two-dimensional coordinate information of the joint points.

[0093] In step 310, the electronic device 100 may adjust the template body model according to the joint position information.

[0094] In step 312, the electronic device 100 may compare and display the first image data and the adjusted template body model. For example, as Figure 2D shown.

[0095] In some embodiments, the method 300 may further include: receiving the user's action template comparison instruction; determining the corresponding action template according to the action template comparison instruction; obtaining the joint motion information corresponding to the action template; and adjusting the user's body model (for example, Figure 1C the body model 10426) according to the joint motion information corresponding to the action template and displaying it.

[0096] In some embodiments, the method 300 may further include: receiving the user's first voice instruction for adjusting the display screen; and adjusting the display screen according to the first voice instruction.

[0097] In some embodiments, the method 300 may further include: receiving an automatic following instruction of the user; continuously collecting second image data including a full-body image of the user within a preset time period according to the automatic following instruction; detecting joint movement information of the user based on the second image data; inputting the joint movement information of the user into a nearest neighbor node algorithm classifier for classification, and outputting a predicted template action; and generating and displaying an action template according to the predicted template action in combination with the body model of the user.

[0098] In some embodiments, outputting the predicted template action further includes: outputting an error prompt message in response to a similarity between the joint movement information of the user and the predicted template action being lower than a preset threshold.

[0099] In some embodiments, the method 300 may further include: using a depth camera (e.g., Figure 1D camera 120) to collect third image data including a full-body image of the user (e.g., Figure 1D images 142 and 144), where the third image data includes a depth image and a color image; generating point cloud data based on the depth image and the color image; performing background removal processing and smoothing denoising processing on the point cloud data to obtain preprocessed point cloud data; using a joint bilateral filtering method to repair depth data in the preprocessed point cloud data to obtain repaired point cloud data; registering the repaired point cloud data to obtain registered point cloud data; and using a Poisson surface reconstruction method to complete surface reconstruction and texture mapping of the registered point cloud data to obtain the body model of the user.

[0100] In some embodiments, registering the repaired point cloud data includes: coarse registration based on a point feature histogram, fine registration using a nearest point search algorithm for adjacent two frames of point cloud, and non-rigid registration based on a deformation graph.

[0101] In some embodiments, the method 300 may further include: receiving a template making instruction of the user; determining whether a second voice instruction for the user to collect an action image is received based on the template making instruction; collecting an action image of the user in response to receiving the second voice instruction; and making the collected action image of the user into an action template.

[0102] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied to a distributed scenario, and completed by the cooperation of multiple devices. In such a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present disclosure, and these multiple devices will interact with each other to complete the described method.

[0103] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the above embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0104] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method 300 as described in any of the above embodiments.

[0105] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0106] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the method 300 as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0107] Based on the same inventive concept, corresponding to the method 300 in any of the above embodiments, the present disclosure also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to execute the method 300. Corresponding to the execution subjects of the respective steps in the embodiments of the method 300, the processors that execute the corresponding steps may belong to the corresponding execution subjects.

[0108] The computer program product of the above embodiments is used to cause a processor to execute the method 300 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0109] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; under the concept of the present disclosure, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of brevity.

[0110] In addition, for simplicity of explanation and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. In addition, the devices may be shown in block diagram form to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (i.e., these details should be completely within the understanding of those skilled in the art). In the case where specific details (such as circuits) are set forth to describe the exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0111] Although the present disclosure has been described in connection with specific embodiments of the present disclosure, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) can be used in the embodiments discussed.

[0112] The embodiments of the present disclosure are intended to cover all such alternatives, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for realizing body posture comparison, comprising: Determining the identity information of a user; Obtaining the template body model corresponding to the user according to the identity information of the user; Obtaining first image data including a full-body image of the user; Processing the first image data to obtain the joint position information of the user; Adjusting the template body model according to the joint position information; And Comparatively displaying the first image data and the adjusted template body model; The method further includes: Receiving an automatic following instruction of the user; According to the automatic following instruction, continuously collecting second image data including a full-body image of the user within a preset time period; Based on the second image data, detecting the joint movement information of the user; Inputting the joint movement information of the user into a nearest neighbor node algorithm classifier for classification, and outputting a predicted template action; and Generating and displaying an action template according to the predicted template action in combination with the body model of the user.

2. The method according to claim 1, wherein Processing the first image data to obtain the joint position information of the user includes: Processing the first image data to obtain a human body image of the user; Constructing a human body skeleton model of the user according to the human body image of the user; and Extracting the spatio-temporal features of the joint vector from the human body skeleton model as the joint position information of the user.

3. The method according to claim 2, wherein, Processing the first image data to obtain a human body image of the user includes: Converting the first image data into a grayscale image; Processing the grayscale image using a Sobel edge detection operator to obtain an edge detection image; Detecting the human body contour in the edge detection image using a local binary pattern method; Using a linear support vector machine classifier to exclude a predetermined proportion of regions in the human body contour to obtain a partial region of the human body contour; and Using a non-linear support vector machine classifier to screen the partial region of the human body contour to obtain the human body image of the user.

4. The method according to claim 2, wherein Constructing a human body skeleton model of the user according to the human body image of the user includes: Using OpenPose to obtain the two-dimensional coordinate information of the joint points in the human body image of the user; Completing the two-dimensional coordinate information of the joint points using a front and back frame correlation algorithm to obtain the completed two-dimensional coordinate information of the joint points; and Constructing the human body skeleton model according to the completed two-dimensional coordinate information of the joint points.

5. The method according to claim 2, wherein, The spatio-temporal features of the joint vector include direction cosine features, included angle cosine features, and angle change rate features.

6. The method according to claim 1, further comprising: Receiving an action template comparison instruction of the user; Determining a corresponding action template according to the action template comparison instruction; Obtaining the joint movement information corresponding to the action template; And Adjusting and displaying the body model of the user according to the joint movement information corresponding to the action template.

7. The method according to claim 1 or 6, further comprising: Receiving a first voice instruction for the user to adjust the display screen; And Adjusting the display screen according to the first voice instruction.

8. The method according to claim 1, wherein, Outputting the predicted template action further includes: Output an error prompt message in response to the similarity between the joint motion information of the user and the predicted template action being lower than a preset threshold.

9. The method according to claim 1, 6 or 8, further comprising: Collect third image data including a full-body image of the user by using a depth camera, where the third image data includes a depth image and a color image; Generate point cloud data based on the depth image and the color image; Perform background removal processing and smoothing and denoising processing on the point cloud data to obtain preprocessed point cloud data; Use a joint bilateral filtering method to repair the depth data in the preprocessed point cloud data to obtain repaired point cloud data; Perform registration on the repaired point cloud data to obtain registered point cloud data; And Use a Poisson surface reconstruction method to complete surface reconstruction and texture mapping on the registered point cloud data to obtain the body model of the user.

10. The method according to claim 9, wherein, Performing registration on the repaired point cloud data includes: coarse registration based on the fast point feature histogram, fine registration using the nearest point search algorithm for adjacent two frames of point cloud, and non-rigid registration based on the deformation map.

11. The method according to claim 1, further comprising: Receive a template making instruction of the user; Based on the template making instruction, determine whether a second voice instruction for the user to collect an action image is received; In response to receiving the second voice instruction, collect the action image of the user; And Make the collected action image of the user into an action template.

12. An electronic device, characterized in that, Comprising one or more processors, a memory; and one or more programs, where the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1-11.

13. A non-volatile computer-readable storage medium containing a computer program, when the computer program is executed by one or more processors, enabling the processors to execute the method according to any one of claims 1-11.

14. A computer program product, including computer program instructions, when the computer program instructions run on a computer, enabling the computer to execute the method according to any one of claims 1-11.

Images