A human action analysis processing method and device and an intelligent fitness system

Abstract

The present application provides a human action analysis processing method, device and intelligent fitness system, the method comprises the following steps: establishing a synchronization relationship among a cloud device, an intelligent fitness mirror and a plurality of intelligent terminals; during a warm-up activity of a target user, a three-dimensional human body model of the target user is established according to image information of the target user collected by the intelligent fitness mirror and the plurality of intelligent terminals; during the movement of the target user, the network processing capacity of each device is calculated to determine a master device according to the network processing capacity; the master device allocates a movement video processing time sequence of the target user, so that each device processes the movement video of the target user according to the processing time sequence; and the master device arranges and merges the movement video of the target user to obtain a movement data set. The present application combines cloud and local MEC to realize fast video action analysis processing and ensure the accuracy of model data as much as possible.

Description

Technical Field

[0001] This article belongs to the field of computer technology, specifically relating to a human motion analysis and processing method, device, and intelligent fitness system. Background Technology

[0002] As the COVID-19 pandemic continues, people's awareness of home fitness has gradually increased, leading to the rise of the smart fitness mirror market and the emergence of various related products. As a product that extends the reach of offline gyms to users, smart fitness mirrors are increasingly developing towards intelligent terminals, supporting not only voice recognition input but also motion sensing input and artificial intelligence (AI) video motion assistance analysis.

[0003] Current smart fitness mirrors primarily capture video data via integrated wide-angle cameras and analyze it using cloud-based GPU server clusters or onboard AI processing chips. However, this current approach presents several problems:

[0004] 1. While wide-angle lenses can capture the maximum angle of movement within a limited distance, the inherent lens distortion results in distorted video of human motion, making it impossible to accurately predict the magnitude of actual limb movements. Furthermore, because cameras on mirror surfaces primarily capture frontal shots, they often fail to capture movements of the cervical and lumbar spine.

[0005] 2. If a cloud-based GPU server is used to analyze and process the captured video, the analysis and processing process will be significantly delayed due to network bandwidth and latency jitter, and the processing results cannot be obtained in real time.

[0006] 3. If the AI ​​processing chip integrated into the local device is used for processing, the processing capability is far from meeting the requirements of real-time accurate motion prediction and calculation due to limitations in manufacturing process, chip cost, and chip power consumption.

[0007] Because of the aforementioned issues, manufacturers can only simplify video analysis and processing algorithms to achieve low-precision, simple qualitative analysis. As a result, users generally feel that the so-called AI in these smart fitness mirrors is not as intelligent as they imagined, which greatly reduces the user experience. Summary of the Invention

[0008] In view of the above-mentioned problems in the prior art, the purpose of this paper is to provide a human motion analysis and processing method, device and intelligent fitness system, which can improve the efficiency and reliability of human motion analysis by fitness mirrors.

[0009] To solve the above-mentioned technical problems, the specific technical solution presented in this paper is as follows:

[0010] On the one hand, this paper provides a method for human motion analysis and processing, which is applied to an intelligent fitness system. The system includes a cloud device, an intelligent fitness mirror, and multiple intelligent terminals. The intelligent fitness mirror and multiple intelligent terminals form a full-view video acquisition range for the target user. The method includes:

[0011] Establish a synchronization relationship between the cloud device, the smart fitness mirror, and multiple smart terminals;

[0012] During the target user's warm-up activities, a three-dimensional human body model of the target user is established based on the image information of the target user collected by the smart fitness mirror and the multiple smart terminals.

[0013] During the target user's exercise, the network processing capability of each device is calculated based on the network parameters of the cloud device, the smart fitness mirror, and multiple smart terminals, so as to determine the master control device based on the network processing capability.

[0014] According to the preset processing strategy and combined with the network processing capabilities of each device, the main control device allocates the motion video processing sequence of the target user so that each device processes the motion video of the target user in accordance with the processing sequence.

[0015] The main control device organizes and merges the target user's motion video into a motion data set based on the target user's three-dimensional human body model and the processing results of each device.

[0016] Furthermore, establishing a synchronization relationship between the cloud device, the smart fitness mirror, and multiple smart terminals includes:

[0017] Each of the aforementioned smart terminals establishes a binding relationship with the smart fitness mirror through the unique identifier of the smart fitness mirror;

[0018] Each of the smart terminals sends the binding relationship to the cloud device so that the cloud device, the smart fitness mirror, and the multiple smart terminals can establish a clock synchronization relationship.

[0019] Furthermore, during the target user's warm-up activities, a three-dimensional human body model of the target user is established based on the image information of the target user collected by the smart fitness mirror and the multiple smart terminals, including:

[0020] During the target user's warm-up activities, the distance between the smart fitness mirror and the target user is determined based on the camera hardware parameters of the smart fitness mirror and the multiple smart terminals and the image information of the target user collected.

[0021] Based on the image information collected by the smart fitness mirror, the information of each joint point of the target user's body is identified by a convolutional neural network, and an initial three-dimensional model of the target user is established based on the joint point information.

[0022] Based on the initial 3D model and the distances between the smart fitness mirror and the multiple smart terminals and the target user, the actual distances between the target user's body joints are determined.

[0023] A 3D human body model based on spherical coordinates is established based on the actual distance between the joints of the target user's body.

[0024] Furthermore, the method also includes:

[0025] Based on the target user's movement type, determine the required number and location distribution of smart terminals.

[0026] Furthermore, during the target user's exercise, the network processing capabilities of each device are calculated based on the network parameters of the cloud device, the smart fitness mirror, and multiple smart terminals, and the master control device is determined based on the network processing capabilities, including:

[0027] Real-time acquisition of network hardware parameters and bandwidth information of the smart fitness mirror, the cloud device, and multiple smart terminals;

[0028] Based on the network hardware parameters and bandwidth information of the smart fitness mirror, the cloud device, and multiple smart terminals, the network processing capability of each device is calculated in real time.

[0029] Determine the processing capacity weight of each device based on the processing capacity of each network;

[0030] The device with the highest processing capacity weight is designated as the master control device.

[0031] Furthermore, determining the processing capacity weight of each device based on the processing capacity of each network further includes:

[0032] Based on the target user's movement type, obtain the target user's preset temporal movement characteristics, which include at least the target user's standard movement posture during the movement process;

[0033] Based on the target user's preset temporal motion characteristics, determine the calculation weights of different devices at different times;

[0034] The processing capacity weight of each device is calculated based on the real-time network processing capacity of each device and the calculation weight of different devices at different times.

[0035] Furthermore, the preset processing strategy includes one or more of the following strategies: sequential selection strategy, round-robin selection strategy, least idle device priority strategy, busiest device priority strategy, and balanced task strategy.

[0036] Further, the step of having each device process the motion video of the target user according to the processing sequence includes:

[0037] Based on the target user's movement type, determine the frequency of the target user's body movements in each preset time period;

[0038] Based on the body movement frequency, a target acquisition device and a video extraction rule corresponding to the body movement frequency are selected from the smart fitness mirror and multiple smart terminals;

[0039] According to the video extraction rules, the target acquisition device extracts target video information from the acquired motion video and processes it according to the processing sequence.

[0040] On the other hand, this paper also provides a human motion analysis and processing device, which is applied to an intelligent fitness system. The system includes a cloud device, an intelligent fitness mirror, and multiple intelligent terminals. The intelligent fitness mirror and multiple intelligent terminals form a full-view video acquisition range for the target user. The device includes:

[0041] A synchronization relationship establishment module is used to establish synchronization relationships between the cloud device, the smart fitness mirror, and multiple smart terminals.

[0042] The 3D model building module is used to build a 3D human body model of the target user based on the image information of the target user collected by the smart fitness mirror and the multiple smart terminals during the target user's warm-up activities.

[0043] The master control device determination module is used to calculate the network processing capability of each device based on the network parameters of the cloud device, the smart fitness mirror and multiple smart terminals during the target user's exercise, so as to determine the master control device based on the network processing capability.

[0044] The processing module is used to allocate the motion video processing sequence of the target user by the main control device according to the preset processing strategy and the network processing capabilities of each device, so that each device processes the motion video of the target user according to the processing sequence.

[0045] The integration module is used by the main control device to organize and merge the motion video of the target user into a motion data set based on the target user's three-dimensional human body model and the processing results of each device.

[0046] Finally, this paper also provides an intelligent fitness system, which includes cloud devices, an intelligent fitness mirror and multiple intelligent terminals, and the intelligent fitness mirror and multiple intelligent terminals form a full-view video capture range for the target user;

[0047] The system applies the human motion analysis and processing method described above.

[0048] Using the above technical solution, this paper describes a human motion analysis and processing method, device, and intelligent fitness system. The method is applied to an intelligent fitness system, which includes a cloud device, an intelligent fitness mirror, and multiple intelligent terminals. The intelligent fitness mirror and multiple intelligent terminals form a full-view video acquisition range for the target user. The method includes: establishing a synchronization relationship between the cloud device, the intelligent fitness mirror, and the multiple intelligent terminals; during the target user's warm-up activities, establishing a three-dimensional human body model of the target user based on the image information of the target user acquired by the intelligent fitness mirror and the multiple intelligent terminals; during the target user's exercise, establishing a three-dimensional human body model based on the image information of the target user acquired by the cloud device and the intelligent fitness mirror and the multiple intelligent terminals; and during the target user's exercise, establishing a three-dimensional human body model based on the image information of the target user acquired by the cloud device and the intelligent fitness mirror and the multiple intelligent terminals. The network parameters of the fitness mirror and multiple smart terminals are used to calculate the network processing capabilities of each device, and the master control device is determined based on the network processing capabilities. According to a preset processing strategy and combined with the network processing capabilities of each device, the master control device allocates the motion video processing sequence for the target user, so that each device processes the target user's motion video according to the processing sequence. The master control device organizes and merges the target user's motion video into a motion data set based on the target user's 3D human body model and the processing results of each device. This paper adopts a combination of cloud and local MEC to achieve fast video motion analysis and processing, and to ensure the accuracy of the model data as much as possible.

[0049] To make the above and other objects, features and advantages of this document more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 The network topology diagram of the intelligent fitness system provided in the embodiments of this article is shown;

[0052] Figure 2 This document illustrates a step-by-step diagram of a human motion analysis and processing method provided in an embodiment.

[0053] Figure 3 This diagram illustrates the relationship between object distance u, phase distance v, and focal length f in the embodiments described herein.

[0054] Figure 4 The geometric relationship between the spherical coordinate system and the rectangular coordinate system in the embodiments of this paper is shown;

[0055] Figure 5 A schematic diagram of the human body structure model established in the embodiments of this article is shown;

[0056] Figure 6 This document shows a schematic diagram of the structure of a human motion analysis and processing device provided in an embodiment of the invention.

[0057] Figure 7 A schematic diagram of the structure of the computer device provided in the embodiments of this article is shown.

[0058] Explanation of symbols in the attached drawings:

[0059] 100. Synchronization Relationship Establishment Module;

[0060] 200. 3D Model Creation Module;

[0061] 300. Main control equipment determination module;

[0062] 400. Processing module;

[0063] 500. Integration module;

[0064] 702. Computer equipment;

[0065] 704, Processor;

[0066] 706. Memory;

[0067] 708. Drive mechanism;

[0068] 710. Input / Output Module;

[0069] 712. Input devices;

[0070] 714. Output devices;

[0071] 716. Presentation equipment;

[0072] 718. Graphical User Interface;

[0073] 720. Network interface;

[0074] 722. Communication link;

[0075] 724. Communication bus. Detailed Implementation

[0076] The technical solutions in the embodiments described below will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments described herein, and not all of the embodiments. Based on the embodiments described herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0077] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0078] With rapid societal development, users' fitness needs are increasing, leading to the rise of the smart fitness mirror market and the emergence of various related products. As a product that extends the reach of offline gyms to users, smart fitness mirrors are gradually developing towards terminal intelligence, supporting not only voice recognition input but also motion sensing input and AI-assisted video motion analysis. However, due to the limitations of the mirror's shooting angle and the high demands on its processing power, it is difficult to meet users' high needs, resulting in a significantly reduced user experience.

[0079] To address the aforementioned issues, this embodiment provides a human motion analysis and processing method that can improve the efficiency of human motion analysis and significantly enhance the user's fitness experience.

[0080] Based on the methods provided above, this specification also provides an intelligent fitness system. The system includes a cloud device, an intelligent fitness mirror, and multiple intelligent terminals. The intelligent fitness mirror and multiple intelligent terminals form a full-view video capture range for the target user. It can be understood that by adding multiple intelligent terminals in conjunction with the intelligent fitness mirror, 360° shooting of the user's human body movements can be achieved, thereby obtaining more comprehensive and realistic user fitness movements. Then, through the coordinated cooperation of the cloud device, intelligent fitness mirror, and intelligent terminals, the rapid analysis and processing of user exercise data is improved, thereby improving the user's fitness efficiency and effectiveness.

[0081] like Figure 1The diagram shown is a framework schematic of an intelligent fitness system in an embodiment of this specification. The intelligent fitness mirror, smartphone 1, and smartphone 2 form a video acquisition system for the exerciser. The exerciser moves within a preset range, and the intelligent fitness mirror, smartphone 1, and smartphone 2 can capture video of the entire exercise process from a fixed position. By combining the acquired video data with cloud-based collaborative processing, the processing efficiency of the intelligent fitness system is improved, thereby enhancing the user experience of using the intelligent fitness mirror.

[0082] like Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the steps of a human motion analysis and processing method provided in the embodiments of this document. This specification provides the operational steps of the method described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or accompanying drawings can be executed sequentially or in parallel.

[0083] The method is applied to an intelligent fitness system, which includes a cloud device, a smart fitness mirror, and multiple smart terminals. The smart fitness mirror and multiple smart terminals form a full-view video capture range for the target user, specifically as follows: Figure 2 As shown, the method may include:

[0084] S101: Establish a synchronization relationship between the cloud device, the smart fitness mirror, and multiple smart terminals;

[0085] S102: During the target user's warm-up activity, a three-dimensional human body model of the target user is established based on the image information of the target user collected by the smart fitness mirror and the multiple smart terminals.

[0086] S103: During the target user's exercise, the network processing capability of each device is calculated based on the network parameters of the cloud device, the smart fitness mirror, and multiple smart terminals, so as to determine the master control device based on the network processing capability;

[0087] S104: According to the preset processing strategy and combined with the network processing capabilities of each device, the main control device allocates the motion video processing sequence of the target user so that each device processes the motion video of the target user according to the processing sequence.

[0088] S105: The main control device organizes and merges the motion video of the target user based on the target user's three-dimensional human body model and the processing results of each device to obtain a motion data set.

[0089] This can be understood as follows: by setting up multiple smart terminals around the user in conjunction with a smart fitness mirror, comprehensive video data of the user can be collected without blind spots. That is, the video collection angles of the multiple smart terminals and the smart fitness mirror are different, and they can form a 360° video collection range of the user without blind spots or most of these angles (i.e., effective angles). This can obtain more comprehensive and direct video data, and improve the accuracy and reliability of the user's human body work analysis and processing.

[0090] The effective angle can be one that accurately and reliably processes and analyzes user movements. For example, a single smartphone or multiple smartphones can form multiple camera positions with the fitness mirror, creating a 90° angle with the person exercising in the center. Figure 1 As shown, this ensures that athletes can be photographed without obstruction.

[0091] Furthermore, the embodiments in this specification combine cloud devices, smart fitness mirrors, and smart terminals to work together, improving the timeliness and reliability of user exercise video processing. This enhances the user experience while ensuring the user's video analysis and processing capabilities, thereby increasing the market competitiveness of the smart fitness mirror.

[0092] In the embodiments of this specification, the smart fitness mirror can be a device with a display screen and a camera that can guide users in exercise and intelligently analyze the user's actual movements; the smart terminal can be a physical device such as a smartphone, desktop computer, tablet computer, laptop computer, digital assistant, or smart wearable device; among which, smart wearable devices can include smart bracelets, smartwatches, smart glasses, smart helmets, etc. Of course, the smart terminal is not limited to the above-mentioned physical electronic devices; it can also be software running on the above-mentioned electronic devices. For example, the smart terminal can be a webpage or application provided to the user by a service provider; the cloud device can be a cloud server, such as a cluster server or a distributed server.

[0093] In some embodiments described herein, establishing a synchronization relationship between the cloud device, the smart fitness mirror, and multiple smart terminals includes:

[0094] Each of the aforementioned smart terminals establishes a binding relationship with the smart fitness mirror through the unique identifier of the smart fitness mirror;

[0095] Each of the smart terminals sends the binding relationship to the cloud device so that the cloud device, the smart fitness mirror, and the multiple smart terminals can establish a clock synchronization relationship.

[0096] This can be understood as follows: Since user exercise data processing involves the collaborative work of cloud devices, smart fitness mirrors, and smart terminals, and rapid and timely data processing is crucial during user exercise, a synchronization relationship needs to be established among these three components to ensure the reliability and accuracy of data processing. Specifically, the smart fitness mirror is equipped with a unique, identifiable identifier, such as a dynamic and unique device QR code. Each smart terminal can then bind to the smart fitness mirror by scanning the code with an installed fitness app, and the cloud devices, smart fitness mirrors, and smart terminals can then synchronize their clocks via the network.

[0097] In some embodiments of this specification, during the target user's warm-up activities, a three-dimensional human body model of the target user is established based on the image information of the target user collected by the smart fitness mirror and the plurality of smart terminals, including:

[0098] During the target user's warm-up activities, the distance between the smart fitness mirror and the target user is determined based on the camera hardware parameters of the smart fitness mirror and the multiple smart terminals and the image information of the target user collected.

[0099] Based on the image information collected by the smart fitness mirror, the information of each joint point of the target user's body is identified by a convolutional neural network, and an initial three-dimensional model of the target user is established based on the joint point information.

[0100] Based on the initial 3D model and the distances between the smart fitness mirror and the multiple smart terminals and the target user, the actual distances between the target user's body joints are determined.

[0101] A 3D human body model based on spherical coordinates is established based on the actual distance between the joints of the target user's body.

[0102] This specification can be understood as follows: the embodiments establish a three-dimensional model of the user during warm-up stretching exercises before the target user actually exercises, thereby facilitating subsequent analysis of the user's movement state based on this three-dimensional model. Specifically, this can involve: using the optical relationship between image distance and focal length (i.e., the camera lens hardware parameters of the smart fitness mirror and the multiple smart terminals) to perform initial measurements on the exerciser's limbs, including:

[0103] 1) The distance u between the athlete and each camera position is measured by the camera's autofocus. i ;

[0104] For example: Figure 3 The figure shows the relationship between object distance u, phase distance v, and focal length f. Based on this relationship... Using the camera's autofocus function, at a focal length f i and distance v i Given the information, the corresponding distance can be calculated. like Figure 3 As shown.

[0105] a) The distance between the smart fitness mirror and the exerciser is u0;

[0106] b) The distance between the first smartphone and the athlete is u1;

[0107] c) The distance between the second smartphone and the athlete is u2;

[0108] d) Similarly, by determining the focal length and phase distance of each camera when capturing the user's image, the distance between the user (i.e., the mover) and each device can be calculated.

[0109] It should be noted that the distance between the device and the user can be the distance between a certain feature point of the user and the device, such as the center point of the user's head, the center point of the waist, etc.

[0110] Furthermore, a convolutional neural network is used to identify various joints of the athlete's limbs, including the left ear, left eye, tip of the nose, right eye, right ear, left shoulder, left elbow, left wrist, left fingertip, right shoulder, right elbow, right wrist, right fingertip, left hip, left knee, left ankle, left toe tip, right hip, right knee, right ankle, and right toe tip (see specific point numbering). Figure 4 And build a three-dimensional model, combining it with the previous ranging data. i The actual distances between relevant human joints are calculated by combining the size data of other reference objects (the standard height H of a fitness mirror hardware device of a known model);

[0111] Taking the measurement of an athlete's height (h) as an example, the relevant correspondence is as follows (refer to the relevant information). Figure 3 :

[0112] a) When the distance between the fitness mirror and the camera is u 11 The height is H; the distance between the athlete and the camera is u. 12 The height is h;

[0113] Based on the relationship between object distance u, phase distance v, and focal length f Image height of gym mirrors

[0114] b) When the fitness mirror moves to a distance of u from the camera 12 The image height of the distance is

[0115] c) Based on the relationship between the height H of the fitness mirror and the height h of the exerciser, and the proportional relationship between the mirror height and the exerciser's image height, that is:

[0116] d) Therefore, the athlete's height

[0117] Similarly, the actual distances between different joints can be obtained.

[0118] Furthermore, based on the actual distances between the target user's human body joints, and combined with the initial 3D model, a 3D human body model based on spherical coordinates can be established. Optionally, the relative positions (such as relative angles and relative positional relationships) between human body joints can be determined through the initial 3D model. Then, by combining the updated actual distances between human body joints, the initial 3D model can be optimized to establish a 3D human body model based on spherical coordinates.

[0119] For example, see Figure 5 Using the center points of the two hip joints as the origin 0 of the first spherical coordinate system; the center points of the two shoulder joints as the origin 1 of the second spherical coordinate system; and the tip of the nose as the origin 2 of the second spherical coordinate system; then using the shoulder joint, elbow joint, wrist joint, hip joint, knee joint, and ankle joint as the origins 7-12 and 15-20 of the spherical coordinate systems respectively, calculate the radial distance, zenith angle, and azimuth angle originating from these origins respectively. For the geometric relationship between spherical coordinates and rectangular coordinates, see [link to relevant documentation]. Figure 4 .

[0120] In the embodiments described in this specification, the method further includes:

[0121] Based on the target user's movement type, determine the required number and location distribution of smart terminals.

[0122] This can be understood as adapting different processing scenarios based on the type of sport chosen by the athlete, determining the required number of camera positions, and further determining the position of each smart terminal, thereby ensuring the authenticity and reliability of video capture.

[0123] In some other embodiments, the method may further include:

[0124] The target user's exercise area is determined based on the position of the smart fitness mirror and the orientation of its display screen;

[0125] The candidate location region of the smart terminal is determined based on the motion region;

[0126] Based on the target user's movement type, determine the main movement posture of the movement type and the direction corresponding to the main movement posture;

[0127] Based on the direction corresponding to the main motion posture, determine the required number and direction of smart terminals;

[0128] Based on the candidate location area of ​​the smart terminal, the required number and orientation of the smart terminals, the location of the smart terminal is determined.

[0129] For example, the display screen of the smart fitness mirror faces forward, and a circle with a radius of 1 meter centered 5 meters in front can be considered the target user's exercise area. Further, an outer circle extending 4 meters outward from this exercise area can be used as a candidate location area. Based on the frequency of the movement, the exercise type can be divided into static exercise and dynamic exercise. Dynamic exercise is further divided into aerobic dynamic exercise and anaerobic dynamic exercise, etc. Of course, the exercise method can also be used as a classification basis, such as fitness exercises, combat aerobics, yoga, etc. The specific classification method is not limited in this embodiment of the specification. Since each type of exercise has its own characteristics, a representative exercise posture can be determined from the exercise type as the main exercise posture. The smart terminal is set in the direction of the main exercise posture, which can accurately and reliably obtain the target user's mastery of the exercise type, thereby improving the reliability of the user's fitness data analysis and improving the user experience.

[0130] In this embodiment of the specification, the step of calculating the network processing capabilities of each device based on the network parameters of the cloud device, the smart fitness mirror, and multiple smart terminals during the target user's exercise, and determining the master control device based on the network processing capabilities, includes:

[0131] Real-time acquisition of network hardware parameters and bandwidth information of the smart fitness mirror, the cloud device, and multiple smart terminals;

[0132] Based on the network hardware parameters and bandwidth information of the smart fitness mirror, the cloud device, and multiple smart terminals, the network processing capability of each device is calculated in real time.

[0133] Determine the processing capacity weight of each device based on the processing capacity of each network;

[0134] The device with the highest processing capacity weight is designated as the master control device.

[0135] It can be understood that the network hardware parameters can be CPU / GPU / NPU / APU / TPU / memory capacity / flash memory capacity and read / write speed, and the bandwidth information can be the uplink and downlink bandwidth of the network line. Both the network hardware parameters and the bandwidth information can display the data processing performance of the device. The network processing capability can be determined by a preset calculation formula. The specific calculation formula is not limited in the embodiments of this specification.

[0136] For example, based on parameters such as the CPU / GPU / NPU / APU / TPU / memory capacity / flash memory capacity and read / write speed of the fitness mirror and the paired mobile phone, the uplink and downlink bandwidth of the network line, and the CPU / GPU / memory capacity / hard disk capacity and read / write speed of the cloud GPU server, the corresponding AI processing capability weight 'a' is calculated comprehensively. i ;

[0137] a) Based on a comprehensive calculation of the fitness mirror's CPU / GPU / NPU / APU / TPU / memory capacity / flash memory capacity and read / write speed, network line uplink and downlink bandwidth, its processing capacity is W0;

[0138] b) Based on a comprehensive calculation of the CPU / GPU / NPU / APU / TPU / memory capacity / flash memory capacity and read / write speed, network line uplink and downlink bandwidth, etc. of the first smartphone bound to it, its processing capability is W1;

[0139] c) Based on a comprehensive calculation of the CPU / GPU / NPU / APU / TPU / memory capacity / flash memory capacity and read / write speed, network line uplink and downlink bandwidth, etc. of the bound second smartphone, its processing capability is W2;

[0140] d) And so on;

[0141] e) Based on a comprehensive calculation of parameters such as the cloud GPU server's CPU / GPU / memory capacity / hard disk capacity, read / write speed, and network uplink / downlink bandwidth, its processing capacity is estimated to be W. n-1 .

[0142] So the processing capacity weights of each device Furthermore, the smart terminal with the highest weight is selected as the master control device, which ensures that the best-performing device can participate in the timely master control processing of the target user's motion data at any time, thus guaranteeing processing efficiency.

[0143] In some other embodiments of this specification, determining the processing capacity weight of each device based on the processing capacity of each network further includes:

[0144] Based on the target user's movement type, obtain the target user's preset temporal movement characteristics, which include at least the target user's standard movement posture during the movement process;

[0145] Based on the target user's preset temporal motion characteristics, determine the calculation weights of different devices at different times;

[0146] The processing capacity weight of each device is calculated based on the real-time network processing capacity of each device and the calculation weight of different devices at different times.

[0147] This can be understood as follows: since the target user's movement is a set of continuous changes in human body movements, as discussed above, different types of movement have different characteristics and correspondingly different representative movements. At the same time, different devices acquire video image information differently due to different acquisition directions. For example, the same movement will be acquired differently in different directions. Therefore, the calculation weight of different devices at different times can be determined based on this. For example, the calculation weight of representative movements is greater than that of non-representative movements. In representative movements, the calculation weight of the device corresponding to the face orientation is greater than that of the device corresponding to the non-face orientation, and so on. The method of determining the calculation weight is not limited in the embodiments of this specification.

[0148] Based on determining the calculation weights of different devices at different times, the processing capacity weights can be calculated using the following formula:

[0149]

[0150] Among them, a i W represents the processing capacity weight of the i-th device. i P represents the network processing capacity of the i-th device. i Let be the calculated weight of the i-th device.

[0151] The above methods can improve the ability to handle and adapt to different types of motion, thereby enabling targeted selection of motion types and improving the utilization efficiency of the equipment.

[0152] In some embodiments of this specification, the preset processing strategy includes one or more of the following strategies: sequential selection strategy, round-robin selection strategy, least idle device priority strategy, busiest device priority strategy, and balanced task strategy.

[0153] This can be understood as follows: for the aforementioned various sports scenarios, the main control device groups the various intelligent devices and assigns a weight 'a' to each device based on its processing power. i The processing time slots are allocated according to the corresponding allocation strategy to ensure that each device is used effectively and to take into account a certain level of processing efficiency. Multiple devices in the cloud and on the local machine perform parallel analysis and processing to achieve real-time analysis results and minimize data transmission and processing latency.

[0154] The allocation strategy can be automatically adjusted by AI, and one or more allocation strategies can be used simultaneously. The allocation strategies that can be adjusted include, but are not limited to, the following:

[0155] Sequential selection strategy: Processing time is allocated sequentially based on the order in which devices are bound. First, the device bound to the device is assigned a time slot; if the first device is busy, the device bound to the second device is assigned a time slot, and so on.

[0156] Polling selection strategy: Processing time is allocated in turn according to the order of device binding time;

[0157] Least idle device priority strategy: The processing time slot is always allocated to the device with the longest idle time.

[0158] Busiest device priority strategy: Always allocate processing time slots to the device that was most recently allocated them.

[0159] Task balancing strategy: Always allocate processing time slots to the device with the shortest total processing time.

[0160] In some embodiments of this specification, the step of having each device process the motion video of the target user according to the processing timing includes:

[0161] Based on the target user's movement type, determine the frequency of the target user's body movements in each preset time period;

[0162] Based on the body movement frequency, a target acquisition device and a video extraction rule corresponding to the body movement frequency are selected from the smart fitness mirror and multiple smart terminals;

[0163] According to the video extraction rules, the target acquisition device extracts target video information from the acquired motion video and processes it according to the processing sequence.

[0164] This can be understood as follows: during the actual acquisition and processing of video data, at least some of the smart terminals can be selected to work based on the frequency of the user's body movements, thus avoiding the ineffective consumption of resources and improving the utilization efficiency of smart devices.

[0165] For example, for movements with low body movement frequency, such as static movements, the analysis of human movements is easier due to the lower frequency of movement, requiring less complex video data. Therefore, fewer smart terminals can be used for video capture during this time period, and consequently, fewer video frames can be analyzed, thus reducing performance consumption while maintaining video analysis quality. Conversely, for movements with high body movement frequency, such as movements with large amplitude, the analysis of human movements is more difficult due to the higher frequency of movement, requiring a large amount of complex video data. Therefore, more smart terminals can be used for video capture during this time period, and consequently, more video frames can be analyzed, thus ensuring video analysis quality.

[0166] Furthermore, in the embodiments of this specification, each device can automatically adapt and select the corresponding key frames during the video data acquisition process. For example, for static motion, a single-camera timed shooting mode is used for analysis and processing; for slow-moving motion, low-frame-rate images are extracted from the video for analysis and processing; for motion with large amplitude, as many cameras as possible are used for shooting and processing; for fast-moving motion, high-frame-rate video is used for processing, etc.

[0167] Furthermore, after each device analyzes and processes the images in a specific time slot, it sends the corresponding data model results to the main control device via the network. The main control device then merges the data results according to the order of the processing time slots to form a continuous set of motion data.

[0168] Based on the human motion analysis and processing method provided above, this specification also provides a human motion analysis and processing device. This device is applied to an intelligent fitness system, which includes a cloud device, an intelligent fitness mirror, and multiple intelligent terminals. The intelligent fitness mirror and the multiple intelligent terminals form a full-view video acquisition range for the target user, such as... Figure 6 As shown, the device includes:

[0169] Synchronization relationship establishment module 100 is used to establish synchronization relationships between the cloud device, the smart fitness mirror and multiple smart terminals;

[0170] The 3D model building module 200 is used to build a 3D human body model of the target user based on the image information of the target user collected by the smart fitness mirror and the multiple smart terminals during the target user's warm-up activities.

[0171] The master control device determination module 300 is used to calculate the network processing capability of each device based on the network parameters of the cloud device, the smart fitness mirror and multiple smart terminals during the target user's exercise, so as to determine the master control device based on the network processing capability.

[0172] The processing module 400 is used to allocate the motion video processing sequence of the target user by the main control device according to the preset processing strategy and in combination with the network processing capabilities of each device, so that each device processes the motion video of the target user according to the processing sequence.

[0173] The integration module 500 is used by the main control device to organize and merge the motion video of the target user into a motion data set based on the target user's three-dimensional human body model and the processing results of each device.

[0174] The beneficial effects achieved by the above-described device are the same as those achieved by the above-described method, and will not be described in detail in the embodiments of this specification.

[0175] like Figure 7 As shown, a computer device provided in this embodiment is described. The apparatus described herein can be the computer device in this embodiment, performing the methods described above. The computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 702 may also include any memory 706 for storing information of any kind, such as code, settings, data, etc. Without limitation, for example, memory 706 may include any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 702. In one case, when processor 704 executes associated instructions stored in any memory or combination of memories, the computer device 702 can perform any operation of the associated instructions. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0176] Computer device 702 may also include an input / output module 710 (I / O) for receiving various inputs (via input device 712) and providing various outputs (via output device 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface (GUI) 718. In other embodiments, the input / output module 710 (I / O), input device 712, and output device 714 may be omitted, and the device may function solely as a computer device within a network. Computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0177] Communication link 722 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0178] Corresponding to Figure 2 In addition to the methods described above, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described methods.

[0179] This embodiment also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the following: Figure 2 The method shown.

[0180] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0181] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0182] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0184] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0186] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.

Claims

1. A method for analyzing and processing human motion, characterized in that, The method is applied to an intelligent fitness system, which includes a cloud device, an intelligent fitness mirror, and multiple intelligent terminals. The intelligent fitness mirror and the multiple intelligent terminals form a full-view video capture range for the target user. The method includes: Establish a synchronization relationship between the cloud device, the smart fitness mirror, and multiple smart terminals; During the target user's warm-up activities, the distance between the smart fitness mirror and the target user is determined based on the camera hardware parameters of the smart fitness mirror and the multiple smart terminals and the image information of the target user collected. Based on the image information collected by the smart fitness mirror, the information of each joint point of the target user's body is identified by a convolutional neural network, and an initial three-dimensional model of the target user is established based on the joint point information. Based on the initial 3D model and the distances between the smart fitness mirror and the multiple smart terminals and the target user, the actual distances between the target user's body joints are determined. Based on the actual distance between the target user's human body joints, a three-dimensional human body model based on spherical coordinates is established. During the target user's exercise, the network processing capability of each device is calculated based on the network parameters of the cloud device, the smart fitness mirror, and multiple smart terminals, so as to determine the master control device based on the network processing capability. According to the preset processing strategy and combined with the network processing capabilities of each device, the main control device allocates the motion video processing sequence of the target user so that each device processes the motion video of the target user in accordance with the processing sequence. The main control device organizes and merges the target user's motion video based on the target user's three-dimensional human body model and the processing results of each device to obtain a motion data set; Determining the distance between the smart fitness mirror and the target user based on the camera hardware parameters of the smart fitness mirror and the multiple smart terminals and the image information of the target user collected includes: Given focal length f i and distance v i Calculate the corresponding object distance ; The step of determining the actual distance between the target user's body joints based on the initial 3D model and the distances between the smart fitness mirror, the multiple smart terminals, and the target user includes: Establish a 3D model, combined with object distance u i The actual distance between relevant human joints is calculated by combining the size data of other reference objects, including the standard height H of a known model of fitness mirror hardware.

2. The method according to claim 1, characterized in that, Establishing a synchronization relationship between the cloud device, the smart fitness mirror, and multiple smart terminals includes: Each of the aforementioned smart terminals establishes a binding relationship with the smart fitness mirror through the unique identifier of the smart fitness mirror; Each of the smart terminals sends the binding relationship to the cloud device so that the cloud device, the smart fitness mirror, and the multiple smart terminals can establish a clock synchronization relationship.

3. The method according to claim 1, characterized in that, The method further includes: Based on the target user's movement type, determine the required number and location distribution of smart terminals.

4. The method according to claim 1, characterized in that, During the target user's exercise, the network processing capabilities of each device are calculated based on the network parameters of the cloud device, the smart fitness mirror, and multiple smart terminals. The main control device is then determined based on these network processing capabilities. This includes: Real-time acquisition of network hardware parameters and bandwidth information of the smart fitness mirror, the cloud device, and multiple smart terminals; Based on the network hardware parameters and bandwidth information of the smart fitness mirror, the cloud device, and multiple smart terminals, the network processing capability of each device is calculated in real time. Determine the processing capacity weight of each device based on the processing capacity of each network; The device with the highest processing capacity weight is designated as the master control device.

5. The method according to claim 4, characterized in that, The step of determining the processing capacity weight of each device based on the processing capacity of each network further includes: Based on the target user's movement type, obtain the target user's preset temporal movement characteristics, which include at least the target user's standard movement posture during the movement process; Based on the target user's preset temporal motion characteristics, determine the calculation weights of different devices at different times; The processing capacity weight of each device is calculated based on the real-time network processing capacity of each device and the calculation weight of different devices at different times.

6. The method according to claim 1, characterized in that, The preset processing strategy includes one or more of the following strategies: sequential selection strategy, polling selection strategy, least idle device priority strategy, busiest device priority strategy, and balanced task strategy.

7. The method according to claim 1, characterized in that, The step of having each device process the target user's motion video according to the processing sequence includes: Based on the target user's movement type, determine the frequency of the target user's body movements in each preset time period; Based on the body movement frequency, a target acquisition device and a video extraction rule corresponding to the body movement frequency are selected from the smart fitness mirror and multiple smart terminals; According to the video extraction rules, the target acquisition device extracts target video information from the acquired motion video and processes it according to the processing sequence.

8. A human motion analysis and processing device, characterized in that, The device is applied to an intelligent fitness system, which includes a cloud device, an intelligent fitness mirror, and multiple intelligent terminals. The intelligent fitness mirror and multiple intelligent terminals form a full-view video capture range for the target user. The device includes: A synchronization relationship establishment module is used to establish synchronization relationships between the cloud device, the smart fitness mirror, and multiple smart terminals. The 3D model building module is used to determine the distance between the smart fitness mirror and the multiple smart terminals and the target user during the target user's warm-up activities, based on the camera hardware parameters of the smart fitness mirror and the multiple smart terminals and the image information of the target user collected. Based on the image information collected by the smart fitness mirror, the information of each joint point of the target user's body is identified by a convolutional neural network, and an initial three-dimensional model of the target user is established based on the joint point information. Based on the initial 3D model and the distances between the smart fitness mirror and the multiple smart terminals and the target user, the actual distances between the target user's body joints are determined. Based on the actual distance between the target user's human body joints, a three-dimensional human body model based on spherical coordinates is established. The master control device determination module is used to calculate the network processing capability of each device based on the network parameters of the cloud device, the smart fitness mirror and multiple smart terminals during the target user's exercise, so as to determine the master control device based on the network processing capability. The processing module is used to allocate the motion video processing sequence of the target user by the main control device according to the preset processing strategy and the network processing capabilities of each device, so that each device processes the motion video of the target user according to the processing sequence. The integration module is used by the main control device to organize and merge the motion video of the target user into a motion data set based on the target user's three-dimensional human body model and the processing results of each device. Determining the distance between the smart fitness mirror and the target user based on the camera hardware parameters of the smart fitness mirror and the multiple smart terminals and the image information of the target user collected includes: Given focal length f i and distance v i Calculate the corresponding object distance ; The step of determining the actual distance between the target user's body joints based on the initial 3D model and the distances between the smart fitness mirror, the multiple smart terminals, and the target user includes: Establish a 3D model, combined with object distance u i The actual distance between relevant human joints is calculated by combining the size data of other reference objects, including the standard height H of a known model of fitness mirror hardware.

9. An intelligent fitness system, characterized in that, The system includes cloud devices, a smart fitness mirror, and multiple smart terminals, which together form a full-view video capture range for the target user. The system applies the method described in any one of claims 1 to 7.

Images