Synchronous analysis method based on multi-person postures and dynamic time warping

Abstract

The invention discloses a synchronous analysis method, device and equipment based on multi-person postures and dynamic time warping and a computer readable storage medium, and the method comprises the steps: carrying out the processing of a video stream containing a plurality of individuals, so as to detect and output a face bounding box of each individual in a scene; for each face bounding box, generating an independent skeleton point sequence uniquely corresponding to each individual; based on each independent skeleton point sequence, constructing an independent action sequence of a plurality of individuals; calculating and outputting an individual similarity score of each individual according to each independent action sequence and the corresponding standard action template; performing statistical variance calculation on the set of the similarity scores of all the individuals to obtain a group synchronism index representing group action synchronism; and generating an analysis report containing the evaluation result of the individual performance and the group synchronism. The method has the advantages that the key points of the individuals in the multi-target scene are accurately associated, and the collaboration of group actions is objectively analyzed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-person pose analysis, and in particular to a multi-person pose and dynamic time warping based synchronous analysis method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] In the field of computer vision, analyzing actions in a multi-person scene is an important and challenging technical direction, which has significant application value in group dance teaching, sports cooperative training, and security monitoring scenes. The core requirement of these application scenarios is to independently track the actions of multiple individuals in the scene and evaluate the consistency or synchronization between their actions. The technical basis for this process is a multi-person pose estimation model, such as OpenPose, which can detect the key points of all human skeletons from video images.

[0003] However, the existing technology mainly faces two major bottlenecks in solving the above requirements. First, in individual tracking, although the pose estimation model can detect all key points in the scene, it cannot automatically and accurately attribute these key points to different individuals. This problem is called "multi-person key point confusion", which will cause cross interference of subsequent analysis data. One existing solution is to first segment the region through the human detection box, and then extract the key points independently for each region. However, this method has poor adaptability and insufficient precision in scenes where the crowd is dense or there is overlap between individuals. Second, in the synchronization evaluation aspect, traditional dynamic time warping (DTW) algorithms are mainly used for single action sequence comparison and cannot be directly used for multi-person action synchronization analysis. Some methods attempt to treat multi-person actions as a whole sequence for similarity calculation, but this ignores the performance differences between individuals, resulting in biased evaluation results.

[0004] In summary, the existing technology in handling multi-person action analysis tasks generally has two core defects: one is the inability to accurately separate and independently track the skeletal key points of each individual in complex scenes; two is the lack of effective means to quantify the temporal consistency between multiple individual actions. These defects result in an inability to objectively and accurately evaluate the coordination of group actions. Therefore, there is an urgent need in the field for a new technical solution that first solves the problem of accurately associating individual key points in a multi-target scene, and then enables quantitative analysis and evaluation of the synchronization of group actions. SUMMARY

[0005] The embodiment of the present application provides a synchronous analysis method based on multi-person posture and dynamic time warping, which can realize accurate association of individual key points in a multi-target scene, and further objectively and quantitatively analyze the coordination of group actions.

[0006] To achieve the above object, the embodiment of the present application provides a synchronous analysis method based on multi-person posture and dynamic time warping, which comprises:

[0007] processing a video stream containing multiple individuals to detect and output a face bounding box of each individual in the scene;

[0008] For each of the face bounding boxes, performing posture estimation within a constraint range to generate an independent bone point sequence corresponding to each individual uniquely;

[0009] Based on each of the independent bone point sequences, extracting time series data containing at least one of joint node coordinates, motion speed or joint angle to construct independent action sequences of multiple individuals;

[0010] Comparing each of the independent action sequences with a corresponding standard action template by performing a dynamic time warping algorithm to calculate and output an individual similarity score of each individual;

[0011] Performing statistical variance calculation on a set of all the individual similarity scores to obtain a group synchronization index representing group action synchronization;

[0012] Based on the individual similarity scores and the group synchronization index, generating an analysis report containing individual performance and group synchronization evaluation results.

[0013] In an embodiment, processing a video stream containing multiple individuals to detect and output a face bounding box of each individual in the scene comprises:

[0014] Applying a face detection model to each frame image of the video stream to identify each face present in the frame image and output coordinate data defining the position of each face as the face bounding box.

[0015] In an embodiment, for each of the face bounding boxes, performing posture estimation within a constraint range to generate an independent bone point sequence corresponding to each individual uniquely comprises:

[0016] Extracting a face feature vector from image data within each of the face bounding boxes;

[0017] Applying a multi-person posture estimation model within each of the face bounding boxes to extract bone key points;

[0018] The skeleton key points are associated with unique individual IDs based on spatial proximity of the face feature vectors and the skeleton key points by an ID assignment algorithm to generate the independent skeleton point sequences.

[0019] In an embodiment, based on each of the independent skeleton point sequences, time series data containing at least one of information of joint coordinates, motion speed or joint angles is extracted to construct independent action sequences of multiple individuals, including:

[0020] The three-dimensional coordinates of the joints in the independent skeleton point sequences are taken as first type of time series data;

[0021] Based on the changes of the three-dimensional coordinates at consecutive time stamps, second type of time series data representing motion speed is calculated;

[0022] Based on the three-dimensional coordinates, connection vectors between joints are calculated and the included angles between the connection vectors are calculated to obtain third type of time series data representing joint angles;

[0023] The first type, second type and third type of time series data are combined to construct the independent action sequences.

[0024] In an embodiment, each of the independent action sequences is compared with a corresponding standard action template by performing a dynamic time warping algorithm to calculate and output individual similarity scores of each individual, including:

[0025] The joint position difference values, motion speed difference values and joint angle difference values of each pair of frames in the independent action sequences and the standard action template are calculated;

[0026] According to a preset action type of the standard action template, the joint position difference values, motion speed difference values and joint angle difference values are weighted and summed to obtain kinematic local cost;

[0027] The kinematic local cost is taken as matrix elements to construct a cost matrix;

[0028] Based on the cost matrix, a dynamic programming recursive formula is applied to construct a cumulative cost matrix and determine a minimum cumulative total cost located at the end point of the cumulative cost matrix;

[0029] The minimum cumulative total cost is normalized to obtain the individual similarity score.

[0030] In an embodiment, statistical variance calculation is performed on a set of all the individual similarity scores to obtain a group synchronization index representing group action synchronization, including:

[0031] The calculated statistical variance is compared with a preset synchronization threshold.

[0032] According to the comparison result, a synchronization level of the group is determined as the synchronism indicator.

[0033] In an embodiment, based on the individual similarity scores and the group synchronism indicator, an analysis report containing individual performance and group synchronism evaluation results is generated, including:

[0034] Comparing each of the individual similarity scores with the average value of all scores to identify target individuals whose scores significantly deviate from the average value;

[0035] On a user interaction interface, highlighting or color marking is applied to the virtual image or data label corresponding to the identified target individuals;

[0036] The group synchronism indicator is visually displayed in numerical or level form.

[0037] To achieve the above object, the embodiment of the present application further proposes a synchronization analysis device based on multi-person poses and dynamic time warping, comprising:

[0038] A face detection module is configured to process a video stream containing multiple individuals to detect and output a face bounding box of each individual in the scene;

[0039] A pose association module is configured to perform pose estimation within the constraint range of each face bounding box to generate an independent skeletal point sequence corresponding to each individual;

[0040] A sequence generation module is configured to extract time series data containing at least one of joint coordinates, motion speed or joint angles based on each independent skeletal point sequence to construct independent action sequences of multiple individuals;

[0041] An individual comparison module is configured to compare each independent action sequence with a corresponding standard action template by performing a dynamic time warping algorithm to calculate and output an individual similarity score of each individual;

[0042] A synchronism analysis module is configured to perform statistical variance calculation on the set of all individual similarity scores to obtain a group synchronism indicator representing group action synchronism;

[0043] A report generation module is configured to generate an analysis report containing individual performance and group synchronism evaluation results based on the individual similarity scores and the group synchronism indicator.

[0044] To achieve the above object, the embodiment of the present application further provides a synchronization analysis device based on multi-person posture and dynamic time warping, which comprises a memory, a processor and a synchronization analysis program based on multi-person posture and dynamic time warping stored in the memory and capable of running on the processor, and the processor implements the synchronization analysis method based on multi-person posture and dynamic time warping when executing the synchronization analysis program based on multi-person posture and dynamic time warping.

[0045] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, which stores a synchronization analysis program based on multi-person posture and dynamic time warping, and the synchronization analysis program based on multi-person posture and dynamic time warping implements the synchronization analysis method based on multi-person posture and dynamic time warping when executed by a processor.

[0046] The synchronization analysis method based on multi-person posture and dynamic time warping has the following beneficial effects:

[0047] 1. High-precision multi-target individual tracking and separation is achieved

[0048] The present application successfully solves the problem that the traditional method is difficult to accurately attribute the skeleton key points in the multi-person dense or overlapping scene through the technical path of "locating the face first, recognizing the posture in the frame, and finally ID association". The joint modeling of the face and the posture makes the key point association accuracy extremely high, and provides a reliable and non-interference data basis for all subsequent individual analysis.

[0049] 2. Objective and quantitative group synchronization evaluation is provided

[0050] The present application performs statistical variance calculation on the similarity score set of all individuals, thereby converting the fuzzy and subjective problem of evaluating team synergy into an objective index that can be quantified and compared.

[0051] 3. High-efficiency real-time processing capability is provided

[0052] The grouping DTW algorithm in the present application independently compares the actions of multiple individuals, and each comparison task is independent in calculation, so that parallel computing can be supported. This architecture design significantly improves the processing efficiency of the system. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in these drawings without any creative effort.

[0054] Figure 1 A module structure diagram of an embodiment of the present application based on multi-person posture and dynamic time warping synchronous analysis equipment;

[0055] Figure 2 A flowchart of an embodiment of the present application based on multi-person posture and dynamic time warping synchronous analysis method.

[0056] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0058] In order to better understand the above technical solutions, the exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of other elements or steps than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, the several means can be embodied by one and the same item of hardware. The use of the terms "first", "second", and "third", etc. does not imply any order but rather are used as names, and are also used for the purpose of nomenclature.

[0060] As Figure 1 shown, Figure 1 is a structural schematic diagram of a server 1 (also called a multi-person posture and dynamic time warping synchronous analysis equipment) of a hardware running environment involved in an embodiment of the present application.

[0061] The server of the embodiment of the present application is, for example, an "Internet of Things device", a smart air conditioner with networking function, a smart electric lamp, a smart power supply, an AR / VR device with networking function, a smart sound box, an automatic driving vehicle, a PC, a smart phone, a tablet computer, an electronic book reader, a portable computer, and the like.

[0062] As shown in Figure 1 The server 1 includes a memory 11, a processor 12, and a network interface 13.

[0063] The memory 11 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 11 can be an internal storage unit of the server 1, such as a hard disk of the server 1. In other embodiments, the memory 11 can also be an external storage device of the server 1, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like.

[0064] Further, the memory 11 can include both an internal storage unit and an external storage device of the server 1. The memory 11 can be used not only to store application software and various data installed in the server 1, such as the code of the multi-person pose and dynamic time warping based synchronization analysis program 10, but also to temporarily store data that has been output or will be output.

[0065] The processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, which is used to run the program code or process data stored in the memory 11, such as the multi-person pose and dynamic time warping based synchronization analysis program 10.

[0066] The network interface 13 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and the like, which is generally used to establish a communication connection between the server 1 and other electronic devices.

[0067] The network may be the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). Various devices in the network environment may be configured to connect to the communication network according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and / or Bluetooth communication protocol, or a combination thereof.

[0068] Optionally, the server may further include a user interface, which may include a display and an input unit such as a keyboard. The optional user interface may also include a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display, which may also be referred to as a display screen or display unit, is used to display information processed in the server 1 and to display a visual user interface.

[0069] Figure 1 Only the server 1 having components 11-13 and the synchronization analysis program 10 based on multi-person gestures and dynamic time warping is shown. It can be understood by those skilled in the art that Figure 1 The structure shown does not constitute a limitation on the server 1 , and the server 1 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0070] In this embodiment, the processor 12 may be configured to call a synchronization analysis program based on multi-person gestures and dynamic time warping stored in the memory 11 and perform the following operations:

[0071] Processing a video stream containing multiple individuals to detect and output the face bounding box of each individual in the scene;

[0072] For each of the face bounding boxes, performing pose estimation within a constrained range to generate an independent skeleton point sequence uniquely corresponding to each individual;

[0073] extracting time-series data containing at least one of joint coordinate, motion velocity or joint angle information from each of the independent skeletal point sequences to construct independent motion sequences of the individuals;

[0074] comparing each of the independent motion sequences with a corresponding standard motion template by performing a dynamic time warping algorithm to calculate and output individual similarity scores of each individual;

[0075] performing statistical variance calculation on a set of the individual similarity scores to obtain a group synchronism indicator representing group motion synchronism;

[0076] generating an analysis report containing individual performance and group synchronism evaluation results based on the individual similarity scores and the group synchronism indicator.

[0077] Based on the hardware architecture of the above-mentioned multi-person pose and dynamic time warping based synchronism analysis device, embodiments of the multi-person pose and dynamic time warping based synchronism analysis method are proposed. The multi-person pose and dynamic time warping based synchronism analysis method aims to provide a technical solution that can first realize accurate association of individual key points in a multi-target scene, and then objectively and quantitatively analyze the coordination of group motion.

[0078] Reference Figure 2 , Figure 2 For an embodiment of the multi-person pose and dynamic time warping based synchronism analysis method, the method includes the following steps:

[0079] S10, process the video stream containing multiple individuals to detect and output the face bounding box of each individual in the scene. Among them, the face bounding box refers to a set of coordinate data that can define the position and range of a certain face target in the image, usually defined by the coordinates of the upper left corner and the lower right corner of the rectangular box. The multiple individuals are the objects to be analyzed for synchronism, and the video stream can be obtained by real-time acquisition through a visual sensor, such as one or more cameras.

[0080] Specifically, the purpose of this step is to accurately locate and "capture" each independent user target in a complex visual scene. The output of this step, a set of face bounding boxes containing the position information of all individuals in the scene, provides a crucial spatial anchor point and processing range for subsequent independent pose estimation and motion analysis of each individual.

[0081] In one specific embodiment, the aforementioned process of processing the video stream to detect the face bounding boxes is implemented by performing frame-by-frame analysis on the video stream. Specifically, the system fetches image frames from the video stream one by one. For each image frame, the system applies a pre-trained face detection model to scan and analyze. The model can be a lightweight face detection model, such as the MTCNN (Multi-Task Cascaded Convolutional Networks) model, to ensure real-time processing. The model is capable of identifying all the faces present in the image and outputs a set of coordinate data defining the precise location and size of each identified face in the image. This set of coordinate data constitutes the face bounding box of the corresponding individual.

[0082] For example, suppose a frame of the input video stream contains both individual A and individual B. When the frame is fed into the face detection model, the model identifies two valid face targets. Then, the model outputs a set of coordinate data for each face. For example, for individual A's face, the model outputs a set of coordinates [(x1_A, y1_A), (x2_A, y2_A)], which defines a rectangular region that completely encloses individual A's face; similarly, for individual B's face, the model outputs another set of coordinates [(x1_B, y1_B), (x2_B, y2_B)]. These two sets of coordinate data are the final output of this step, which will be used as input for the subsequent steps to process the poses of individual A and individual B, respectively.

[0083] It can be understood that by applying the face detection model to process the video stream frame by frame, a preliminary and independent spatial positioning anchor point (i.e., the face bounding box) can be automatically established for each human target appearing in the complex visual scene.

[0084] S20, for each face bounding box, performing pose estimation within the constraint range to generate an independent skeletal point sequence corresponding to each individual. Here, the independent skeletal point sequence refers to a time series data associated with a unique user identity ID, containing continuous multi-frame pose information, where each frame of data contains the spatial coordinates of all body joints of the user and the corresponding timestamp.

[0085] Specifically, step S20 is to perform pose estimation in the constrained image range defined by each specific face bounding box output by the previous step, and further apply an association algorithm for processing, finally generating an independent bone point sequence for each individual in the scene, which is uniquely corresponding to the individual's identity. By constraining the range of pose estimation near the face bounding box associated with a single individual, it can fundamentally avoid mistakenly attributing the limb key points of other individuals in the distance to the current individual, thereby providing clean and accurate data input for subsequent independent analysis.

[0086] In a specific embodiment, the above process of generating an independent bone point sequence for each individual can be implemented by the following steps S21 to S23:

[0087] First, in step S21, the system will extract a high-dimensional face feature vector representing the identity characteristics of the face from the image data within each face bounding box determined by S10. Then, in step S22, the system applies a multi-person pose estimation model (such as the AlphaPose model) within the same face bounding box to detect and extract all visible human skeleton key points in the region. Finally, in step S23, the system matches and associates the skeleton key points extracted in S22 with the face feature vector extracted in S21 through an ID assignment algorithm. This association process takes into account the similarity of face features and the spatial proximity between skeleton key points and face bounding boxes, ultimately assigning a unique ID to each individual and combining all skeleton key points belonging to the ID to form an independent bone point sequence.

[0088] Specifically, for each detected face in the scene, the system will extract two types of information in parallel: one is the "face feature vector" that can uniquely identify the individual obtained through face recognition technology (step S21); the other is the "skeleton key point set" detected in the region around the face without identity attribution (step S22). The ID assignment step (step S23) can use optimal assignment strategies such as the Hungarian algorithm. This algorithm will find the globally optimal match based on a cost matrix, and the cost values in the cost matrix can consider the spatial distance between the centroid of a "skeleton key point set" and the center of a "face bounding box". In this way, the algorithm can match the most adjacent body and face with the highest probability, thereby assigning a unique ID to each detected complete body skeleton, which is derived from its face and can be continuously tracked in the video sequence.

[0089] For example, after the system obtains the face bounding boxes of individual A and individual B, in step S21, the system extracts the face feature vector VA of individual A and the face feature vector VB of individual B from the two bounding boxes respectively. Meanwhile, in step S22, the pose estimation model may detect two complete sets of skeletal key points in the entire scene, S_body1 and S_body2. At this time, the system does not know whether S_body1 belongs to A or B. In step S23, the ID assignment algorithm starts to work. It calculates the distance between the center of mass of S_body1 and the bounding boxes of individual A and B, and the distance between the center of mass of S_body2 and the bounding boxes of individual A and B. Suppose S_body1 is closer to the bounding box of individual A, and S_body2 is closer to the bounding box of individual B. Based on this spatial proximity information, the algorithm makes the optimal assignment: binds the ID of individual A (maintained through its face feature vector VA) to the skeletal point set S_body1, and binds the ID of individual B to S_body2. In this way, the system successfully generates two independent skeletal point sequences with clear identity attribution.

[0090] It can be understood that, by positioning the face first and then performing pose estimation within the constraint range of the face box, the search space for key point matching can be greatly reduced, effectively avoiding the problem of incorrectly associating limb key points belonging to different individuals together, and significantly improving the accuracy of pose estimation in a multi-target scene. In addition, through the ID assignment algorithm, the skeletal data is bound to the face feature vector with unique identity, thereby ensuring stable and continuous tracking of the same individual in consecutive video frames, and laying a solid data foundation for subsequent independent action analysis of each individual.

[0091] S30, based on each of the independent skeletal point sequences, extracting time series data containing at least one of the information of joint coordinates, motion speed or joint angle, to construct independent action sequences of multiple individuals. Among them, the independent action sequence is time series data further processed based on the independent skeletal point sequence and richer in information. In addition to containing the original joint coordinates, it also contains one or more derived kinematic information such as motion speed and joint angle, so as to more comprehensively represent the action of an individual.

[0092] Specifically, the purpose of step S30 is to further extract features from the independent skeletal point sequence corresponding to each individual to construct an independent action sequence that can more comprehensively and robustly describe the action of an individual, to improve the accuracy of subsequent similarity comparison.

[0093] In a specific embodiment, the above process of constructing an independent action sequence can be implemented by the following steps S31 to S34:

[0094] Firstly, in step S31, the system directly takes the original 3D coordinates contained in the independent skeletal point sequence as the first type of basic time series data. Secondly, in step S32, the system calculates the second type of time series data representing the instantaneous motion speed of the joint by performing a difference operation on the 3D coordinates of the same joint between consecutive frames and dividing by the corresponding timestamp difference. Thirdly, in step S33, the system selects adjacent joints (e.g., shoulder, elbow, wrist) according to the human anatomy structure to construct skeletal link vectors, and then calculates the included angles between these vectors to obtain the third type of time series data representing the body posture, i.e., joint angles, which are not sensitive to body size. Finally, in step S34, the system combines or concatenates the three types of time series data of position, speed, and angle extracted or calculated above to form a multi-dimensional feature vector sequence, which is the final constructed independent action sequence for subsequent comparison.

[0095] Specifically, the core of the above embodiment is to construct a multi-dimensional and more informative action representation. This process first retains the most basic position information (step S31). In order to capture the dynamic characteristics of the action, it further calculates the "first derivative" of the position data with respect to time, i.e., the motion speed (step S32), which is crucial for distinguishing between fast and slow actions. Further, in order to eliminate the influence of height, arm length, and other physical differences between different individuals, making the comparison more fair, the embodiment also obtains relative posture information by calculating the included angles between joints, i.e., joint angles (step S33). Finally, in step S34, for each frame in the time series, the system combines the position, speed, angle, and other feature data of this frame into a high-dimensional feature vector. The entire action is represented as a time series composed of a series of high-dimensional feature vectors, i.e., an independent action sequence with more information.

[0096] For example, for the posture of a certain individual at a certain frame, its independent skeletal point sequence may only contain the coordinates (x, y, z) of its wrist joint. In step S31, this coordinate (x, y, z) is directly adopted. In step S32, the system finds the coordinates (x_prev, y_prev, z_prev) and timestamp of the wrist joint in the previous frame, and calculates the velocity components (vx, vy, vz) in the three axial directions. In step S33, the system also finds the elbow and shoulder coordinates of the individual, constructs the shoulder-elbow vector and elbow-wrist vector, and calculates the included angle between these two vectors to obtain the elbow joint angle θ_elbow. In step S34, these information is combined into a multi-dimensional feature vector, for example: [x, y, z, vx, vy, vz, θ_elbow,...]. Arranging all the multi-dimensional feature vectors of all frames in chronological order forms the independent action sequence of the individual with more information.

[0097] It can be understood that, since not only the original joint coordinates are adopted, but also the motion speed representing the dynamic characteristics of the action and the joint angle representing the relative relationship of the posture are additionally calculated and introduced, the independent action sequence constructed has richer feature dimensions and stronger description ability compared with the sequence containing only position information. In addition, since the introduction of the joint angle feature is invariant to the size of the individual, the subsequent action comparison based on the sequence can more fairly and accurately evaluate the action similarity between individuals of different heights and body types, thereby improving the robustness and generalization ability of the algorithm.

[0098] S40, compare each independent action sequence with the corresponding standard action template by performing a dynamic time warping algorithm to calculate and output an individual similarity score for each individual.

[0099] Specifically, the purpose of step S40 is to compare the independent action sequence constructed for each individual in the previous step with a corresponding standard action template respectively, to calculate and output an individual similarity score that can quantify the action completion degree of the individual. By applying the dynamic time warping algorithm, the system can handle the speed and rhythm changes that may occur when the individual performs the action, which are inconsistent with the standard template, so as to obtain an objective and quantitative score. This individual similarity score is the basic data for subsequent group synchronization analysis.

[0100] In a specific embodiment, the process of comparing by the dynamic time warping algorithm to calculate the score can be realized by the following steps S41 to S45:

[0101] First, in step S41, when comparing the independent action sequence of an individual with any pair of corresponding frames in the standard template, the system will calculate the sub-difference values of the two in three different kinematic dimensions in parallel: the joint position difference value, the motion speed difference value and the joint angle difference value. The technical details of the calculation are as follows:

[0102] Position difference value (d_pos): the system will calculate the Euclidean distance between the coordinate vectors of all K corresponding joints in the two frames, and sum or average them. Its formula can be expressed as:

[0103]

[0104] Speed difference value (d_vel): the system will first calculate the speed vectors of each joint in the player and the standard template by first-order difference (i.e. (pos_t-pos_{t-1}) / Δt) of the position coordinates between consecutive frames. Then, the sum of the Euclidean distances of the corresponding joint speed vectors between the two frames is calculated.

[0105] Angle difference value (d ang): for any frame of pose data, the system will automatically identify the key joints that need to be analyzed according to the pre-set human body skeletal model, such as elbow joint, knee joint, etc. For each joint, the system will determine the two connected body parts that constitute the joint (for example, for the elbow joint, it is the upper arm and the lower arm), and calculate the spatial angle between the two parts. The system will calculate such an angle value for all pre-defined key joints, thereby obtaining a "joint angle set" that can fully describe the body posture of the frame. Then, the angle difference between two frames is calculated. The system will obtain the "joint angle set" of the user frame and the standard template frame respectively. By comparing the angle values of all corresponding joints in the two sets (for example, comparing the elbow joint angle of the user frame with the elbow joint angle of the standard frame), the system will calculate the angle difference of each corresponding joint. Finally, the system will accumulate or integrate all the joint angle differences to obtain a single quantitative value that can represent the degree of dissimilarity between the two frames in the overall relative structure, which is the "angle difference value". The formula for calculating the angle difference value d ang is:

[0106]

[0107] where J is the total number of key joints being compared, θ_{user,j} and θ_{std,j} are the angle values of the jth joint in the user frame and the standard template frame respectively.

[0108] Further, the formula for calculating the joint angle value θ is:

[0109]

[0110] where V_1 and V_2 are two joint vectors, V_1·V_2 is the dot product of the two vectors, |V_1| and |V_2| are the modules (i.e. lengths) of the two vectors respectively.

[0111] Next, in step S42, the system will dynamically assign a set of different weight coefficients to the three dimension sub-difference values calculated in the previous step according to the "action type" preset in the current standard action template, and perform weighted summation to obtain a comprehensive kinematic local cost. The formula for calculating the kinematic local cost is:

[0112] d_local = a · d_pos + β · d_vel + γ · d_ang, where a, β, γ are pre-set weight coefficients.

[0113] After obtaining this local cost that can fully reflect the comprehensive difference between the two frames, the system will perform a standard dynamic time warping calculation process.

[0114] In step S43, the system constructs a cost matrix D with the kinematic local cost as the matrix elements. The size of the matrix D is N x M, where N and M are the lengths of the user sequence and the template sequence respectively, and the element D(i,j) in the matrix at the i-th row and j-th column is the local cost between the i-th frame and j-th frame in the two sequences.

[0115] In step S44, the system constructs a cumulative cost matrix γ based on the cost matrix D by using the classical dynamic programming recursion formula, and determines the minimum cumulative total cost between the two sequences from the matrix. The element γ(i,j) in the cumulative cost matrix γ is filled according to the formula:

[0116] γ(i,j) = D(i,j) + min{γ(i-1,j), γ(i-1,j-1), γ(i,j-1)},

[0117] where the value γ(N,M) at the end point (N,M) of the matrix is the minimum cumulative total cost.

[0118] Finally, in step S45, the system processes the minimum cumulative total cost by a pre-set normalization function to obtain the final individual similarity score. For example, the normalization function can be:

[0119]

[0120] where C_{max} is a pre-set maximum possible cost according to the sequence length and data range.

[0121] For example, assume that the system is evaluating a group gymnastics action, which contains two types of sub-actions: one is a "balance pose" requiring static holding, and the other is a "synchronous leg kick" requiring consistent rhythm. When evaluating the "balance pose", the action type is pre-set as "pose-dominant". In step S42, the system assigns a set of weights for this, for example, assigning higher weights (e.g. 0.45 each) to the difference values of "position" and "angle", and a lower weight (e.g. 0.1) to the difference value of "speed". This is because for the pose action, the accuracy of the pose is the focus of the evaluation. When evaluating the "synchronous leg kick", the action type is pre-set as "rhythm-dominant". At this time, the system assigns another set of weights, for example, assigning a high weight (e.g. 0.8) to the difference value of "motion speed", and lower weights (e.g. 0.1 each) to the positions and angles.

[0122] It can be understood that by fusing the information of multiple kinematic dimensions such as position, velocity, angle, etc. when calculating the local cost of DTW, and dynamically assigning different weights to these dimensions according to the differences of action types, the action completion quality of each individual can be more comprehensively, accurately and targetedly quantified. Further, this high-quality individual evaluation provides extremely reliable and high-differentiation data input for the subsequent step S50 of analyzing the overall group synchrony by statistical methods (such as calculating variance), which is a key prerequisite to ensure the accuracy of the final synchrony index. S50, performing statistical variance calculation on the set of all individual similarity scores to obtain a group synchrony index representing the group action synchrony.

[0123] Specifically, this step uses the statistical tool of variance to quantify the consistency of the performance of team members. In theory, if the actions of all team members are very standard and consistent, their respective scores will be very close, and the calculated variance value will be small; on the contrary, if the action levels of the members are uneven, the score difference is huge, and the calculated variance value will be large. Therefore, this "group synchrony index" can very objectively reflect the overall coordination level of the team.

[0124] In a specific embodiment, the above process of calculating the group synchrony index can be realized by the following steps S51 to S52:

[0125] First, in step S51, the system compares the calculated statistical variance value with one or more pre-set synchrony threshold values in terms of numerical size. Then, in step S52, the system determines a discrete and easy-to-understand synchrony level for the current group performance according to the comparison result of S51, as the group synchrony index. For example, the synchrony level can be divided into multiple gears such as "high synchrony", "medium synchrony" or "asynchronous", which can be used to display in the final report.

[0126] For example, assuming that there are three members (A, B, C) in a dance group, after performing an action, their respective individual similarity scores obtained by the system in step S40 are 85, 83 and 88 respectively. In step S50, the system first calculates the average value of this set of scores: (85+83+88) / 3=85.33. Then, calculate the sum of squares of the difference between each score and the average value: (85-85.33) 2 +(83-85.33) 2 +(88-85.33) 2≈0.11 + 5.43 + 7.13 = 12.67. Finally, the sum of squares is divided by the number of members to obtain the statistical variance, which is approximately 12.67 / 3 = 4.22. In the subsequent step, the system assumes that the preset threshold of "high synchronization" is 5. Since the calculated variance 4.22 is less than the threshold 5, the system determines the performance of the current group as "high synchronization" level in step S52.

[0127] It can be understood that by calculating the statistical variance of the similarity score set of all individuals, a single synchronization index that can macroscopically represent the consistency level of the whole group is quantitatively and objectively converted from the performances of multiple discrete individuals. In addition, by comparing the calculated variance value with the preset threshold to determine an intuitive synchronization level, the system can provide the user or manager with an explicit conclusion about team coordination that is easy to understand, rather than just an abstract statistical value.

[0128] S60, based on the individual similarity scores and the group synchronization index, generating an analysis report containing individual performance and group synchronization evaluation results.

[0129] Specifically, the core role of this step is to convert the relatively abstract numerical data (i.e., the similarity score of each individual and the variance value of the team) produced in the previous steps into a final output that is rich in information, clear in primary and secondary, and highly readable, thereby realizing the final application value of the method.

[0130] In a specific embodiment, the above-mentioned process of generating an analysis report can be realized by the following steps S61 to S63:

[0131] First, in step S61, in order to find out the members whose performance is not coordinated in the group, the system compares the similarity score of each individual with the arithmetic mean of all member scores. By setting a deviation threshold (e.g., the difference from the average value exceeds a certain percentage), the system can automatically identify those target individuals whose scores are too high or too low, i.e., whose performance is significantly different from the overall level of the team. Then, in step S62, the system visually marks the target individuals identified in S61 on the user interaction interface. This marking can be highlighting the virtual character model corresponding to the target individual, such as displaying it in red; or adding a special warning icon next to the data tag displaying its score. Finally, in step S63, the system displays the previously calculated group synchronization index (i.e., the variance value) and its corresponding synchronization level (e.g., "high synchronization") in a prominent position on the interface. These three parts (individual anomaly identification, visual highlighting, group synchronization display) together constitute this analysis report that is rich in content, clear in primary and secondary, and highly readable.

[0132] For example, the system obtains scores of three members A, B, C as 85, 83, 88 respectively, an average score of 85.33, and a group synchronism index (variance) of 4.22, corresponding to a synchronism level of “high synchronism”. In step S61, the system compares the score of each person with the average score 85.33, but no individual needs special attention is identified because the differences are not significant. Therefore, in step S62, the virtual image of all members on the interface is displayed in normal color. In step S63, the system displays on the summary panel of the screen: “Team synchronism: 4.22 | Synchronism level: high synchronism”. As another example, if the scores of the three members are 95, 92 and 60 respectively, the average score is 82.33, and the variance becomes very large. The system easily identifies in S61 that the member C with a score of 60 is an individual with a significant deviation. Therefore, in S62, the virtual image of member C is marked in red, and in S63, the summary panel may display: “Team synchronism: 284.2 | Synchronism level: unsynchronized”, thereby clearly conveying to the user the overall performance of the team and the problem of the individual.

[0133] It can be understood that by comparing the score of each individual with the average score, individuals with abnormal performance or out of sync can be automatically screened from the team, and accurate focus on individual problems is achieved. In addition, since the analysis results are presented through various visualization means such as highlighting, numerical values and levels, the finally generated analysis report is rich in information, clear in primary and secondary, intuitive and easy to understand, greatly improving the practicality and guiding value of the evaluation results.

[0134] In summary, the synchronism analysis method based on multi-person pose and dynamic time warping has at least the following beneficial effects:

[0135] 1. High-precision multi-target individual tracking and separation is achieved

[0136] The present application successfully solves the problem that traditional methods cannot accurately attribute the skeleton key points in a multi-person dense or overlapping scene by the technical path of “locating the face first, recognizing the pose in the frame, and finally ID association”. This joint modeling of face and pose makes the key point association accuracy very high, providing a reliable and interference-free data basis for all subsequent individualized analysis.

[0137] 2. An objective and quantitative group synchronism evaluation is provided

[0138] The present application performs statistical variance calculation on the similarity score set of all individuals, thereby converting the fuzzy and subjective problem of evaluating team collaboration into a quantifiable and comparable objective index.

[0139] 3. High efficient real-time processing capability

[0140] The grouping DTW algorithm in the application compares the actions of multiple individuals independently, and each comparison task is independent in calculation, so that parallel computing can be supported. The architecture design significantly improves the processing efficiency of the system.

[0141] The embodiment of the application also provides a synchronous analysis device based on multiple-person posture and dynamic time warping, which comprises:

[0142] A face detection module is configured to process a video stream containing multiple individuals to detect and output a face bounding box of each individual in the scene.

[0143] A posture correlation module is configured to perform posture estimation within the constraint range of each face bounding box to generate an independent bone point sequence corresponding to each individual.

[0144] A sequence generation module is configured to extract time series data containing at least one of joint coordinate, motion speed or joint angle based on each independent bone point sequence to construct independent action sequences of multiple individuals.

[0145] An individual comparison module is configured to compare each independent action sequence with a corresponding standard action template by performing a dynamic time warping algorithm to calculate and output an individual similarity score of each individual.

[0146] A synchronism analysis module is configured to perform statistical variance calculation on a set of all individual similarity scores to obtain a group synchronism index representing group action synchronism.

[0147] A report generation module is configured to generate an analysis report containing individual performance and group synchronism evaluation results based on the individual similarity scores and the group synchronism index.

[0148] The steps implemented by each functional module of the synchronous analysis device based on multiple-person posture and dynamic time warping can refer to each embodiment of the synchronous analysis method based on multiple-person posture and dynamic time warping, and will not be described here.

[0149] In addition, the embodiment of the present application also provides a computer readable storage medium, which can be any one or any combination of a hard disk, a multimedia card, an SD card, a flash card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a USB memory, etc. The computer readable storage medium includes the synchronization analysis program 10 based on multi-person poses and dynamic time warping. The specific implementation of the computer readable storage medium of the present application is substantially the same as the specific implementation of the synchronization analysis method based on multi-person poses and dynamic time warping and the server 1, and will not be described here.

[0150] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0151] The present application is described in reference to the flowcharts and / or block diagrams of the method, apparatus (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as a combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flow Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0152] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices that implement the flow Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0153] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the flowFigure 1 one or more processes and / or functions specified in one or more blocks Figure 1 one or more blocks or steps of the functions specified in the one or more blocks.

[0154] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the preferred embodiments by those of skill in the art once they have the benefit of the present disclosure. Therefore, the appended claims are intended to encompass within their scope all possible variations and modifications of the preferred embodiments. 1

[0155] It is apparent that a person skilled in the art can make a variety of changes and modifications to the application without departing from the spirit and scope thereof. Thus, if these modifications and changes fall within the scope of the claims and their equivalents, it is intended to include them in the application.

Claims

1. A synchronous analysis method based on multi-person posture and dynamic time warping, characterized in that: include: Processing a video stream containing multiple individuals to detect and output the face bounding box of each individual in the scene; For each of the face bounding boxes, performing pose estimation within a constrained range to generate an independent skeleton point sequence uniquely corresponding to each individual; Extracting time series data containing at least one of joint point coordinates, movement speed, or joint angle information based on each of the independent skeleton point sequences to construct independent action sequences for multiple individuals; Comparing each of the independent action sequences with the corresponding standard action template by executing a dynamic time warping algorithm to calculate and output an individual similarity score for each individual; performing statistical variance calculation on a set of all the individual similarity scores to obtain a group synchronization index characterizing group movement synchronization; Based on the individual similarity scores and the group synchronization index, an analysis report including individual performance and group synchronization evaluation results is generated.

2. The synchronous analysis method based on multi-person posture and dynamic time warping according to claim 1, characterized in that: Process a video stream containing multiple individuals to detect and output the face bounding box of each individual in the scene, including: A face detection model is applied to each frame image of the video stream to identify each face present in the frame image, and coordinate data defining the position of each face is output as the face bounding box.

3. The synchronous analysis method based on multi-person posture and dynamic time warping according to claim 1, characterized in that: For each of the face bounding boxes, pose estimation is performed within the constraints to generate an independent skeleton point sequence uniquely corresponding to each individual, including: Extracting a facial feature vector from the image data within each face bounding box; Applying a multi-person pose estimation model within each face bounding box to extract skeletal key points; The skeleton key points are associated with unique individual IDs based on the spatial proximity between the facial feature vector and the skeleton key points through an ID allocation algorithm to generate the independent skeleton point sequence.

4. The synchronous analysis method based on multi-person posture and dynamic time warping according to claim 1, characterized in that: Based on each of the independent skeleton point sequences, extracting time series data containing at least one of the information of joint point coordinates, motion speed, or joint angle to construct multiple individual independent action sequences, including: The three-dimensional coordinates of the joint points in the independent skeleton point sequence are used as the first type of time series data; Based on the changes of the three-dimensional coordinates at the continuous time stamps, a second type of time series data representing the movement speed is calculated; Calculating line vectors between joints based on the three-dimensional coordinates, and calculating angles between the line vectors to obtain third-type time series data representing joint angles; The first, second and third types of time series data are combined to construct the independent action sequence.

5. The synchronous analysis method based on multi-person posture and dynamic time warping according to claim 1, characterized in that: Each of the independent action sequences is compared with the corresponding standard action template by executing a dynamic time warping algorithm to calculate and output an individual similarity score for each individual, including: Calculating the joint point position difference, motion speed difference, and joint angle difference between each pair of frames in the independent action sequence and the standard action template; According to the preset action type of the standard action template, weighted summation is performed on the joint point position difference value, the movement speed difference value and the joint angle difference value to obtain the kinematic local cost; Constructing a cost matrix using the kinematic local costs as matrix elements; Based on the cost matrix, a dynamic programming recursive formula is applied to construct a cumulative cost matrix, and a minimum cumulative total cost at an end point of the cumulative cost matrix is ​​determined; The minimum cumulative total cost is normalized to obtain the individual similarity score.

6. The synchronous analysis method based on multi-person posture and dynamic time warping according to claim 1, characterized in that: Statistical variance calculation is performed on the set of all the individual similarity scores to obtain a group synchronization index characterizing the group movement synchronization, including: Comparing the calculated statistical variance with a preset synchronicity threshold; According to the comparison result, the synchronization level of the group is determined as the synchronization index.

7. The synchronous analysis method based on multi-person posture and dynamic time warping according to claim 1, characterized in that: Based on the individual similarity scores and the group synchronization index, an analysis report containing individual performance and group synchronization evaluation results is generated, including: Comparing each of the individual similarity scores with the average of all scores to identify target individuals whose scores deviate significantly from the average; On the user interaction interface, highlighting or color-marking the virtual image or data label corresponding to the identified target individual; The group synchronization index is visually displayed in the form of a numerical value or a level.

8. A synchronous analysis device based on multi-person posture and dynamic time warping, characterized in that: include: A face detection module is used to process a video stream containing multiple individuals to detect and output the face bounding box of each individual in the scene; a pose association module, configured to perform pose estimation on each face bounding box within its constraints to generate a sequence of independent skeleton points uniquely corresponding to each individual; A sequence generation module is used to extract time series data containing at least one of the following information: joint point coordinates, movement speed or joint angle based on each of the independent skeleton point sequences, so as to construct independent action sequences of multiple individuals; An individual comparison module is used to compare each of the independent action sequences with the corresponding standard action template by executing a dynamic time warping algorithm to calculate and output an individual similarity score for each individual; a synchronization analysis module, configured to perform statistical variance calculation on the set of all the individual similarity scores to obtain a group synchronization index characterizing the synchronization of group movements; A report generation module is used to generate an analysis report including individual performance and group synchronization evaluation results based on the individual similarity score and the group synchronization index.

9. A synchronous analysis device based on multi-person posture and dynamic time warping, characterized in that: It includes a memory, a processor, and a synchronous analysis program based on multi-person posture and dynamic time regularization stored in the memory and runnable on the processor. When the processor executes the synchronous analysis program based on multi-person posture and dynamic time regularization, it implements the synchronous analysis method based on multi-person posture and dynamic time regularization as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a synchronization analysis program based on multi-person postures and dynamic time regularization. When the synchronization analysis program based on multi-person postures and dynamic time regularization is executed by the processor, it implements the synchronization analysis method based on multi-person postures and dynamic time regularization as described in any one of claims 1-7.

Images