A multi-view human key point sequence data recognition method, device and medium

Abstract

The application discloses a multi-view human body key point sequence data recognition method and device and a medium, relates to the multi-view human body key point sequence data recognition technical field, and comprises the following steps: acquiring two-dimensional key point sequence data of a to-be-recognized human body under multiple synchronous views; based on the two-dimensional key point sequence data, constructing global key point coordinate representation data and skeleton hierarchical offset representation data; respectively performing spatial dimension, view dimension and time dimension feature recognition on the global key point coordinate representation data and the skeleton hierarchical offset representation data, obtaining global key point coordinate representation features and skeleton hierarchical offset representation features, and generating a three-dimensional coordinate data set.

Description

Technical Field

[0001] This invention relates to the field of multi-view human key point sequence data recognition technology, and more specifically, to a method, device and medium for multi-view human key point sequence data recognition. Background Technology

[0002] Multi-view human keypoint sequence data processing typically determines the three-dimensional spatial coordinates of human keypoints based on two-dimensional keypoint data from multiple simultaneous viewpoints. While this approach leverages complementary observation information from different viewpoints, existing methods often process human keypoint sequences within a single coordinate representation space, making it difficult to simultaneously consider both the overall structural relationships of the human body and the local offset relationships between parent and child keypoints. Furthermore, when occlusion, viewpoint shifts, or local keypoint detection errors occur, the single-path data recognition method can easily lead to the accumulation of errors during the three-dimensional coordinate determination process, affecting the stability of the final coordinate data.

[0003] Furthermore, existing multi-view keypoint data fusion methods typically employ fixed-weight superposition, average fusion, or feature stitching, making it difficult to adaptively adjust the information contribution ratio based on the reliability differences of different time frames, different human keypoints, and different representation channels. After outputting 3D coordinates, they also lack a lightweight and stable coordinate compensation and calibration mechanism, making it difficult to further eliminate minor deviations in the fusion results. Therefore, there is still room for improvement in the accuracy and robustness of the conversion from multi-view human keypoint sequences to 3D coordinate data.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, and medium for identifying multi-view human key point sequence data, which at least solves the technical problem that a single representation space in multi-view human key point data processing methods cannot simultaneously take into account global structure and local offset.

[0006] According to one aspect of the present invention, in order to achieve the above-mentioned objective, a method for recognizing human keypoint sequence data from multiple perspectives is provided, comprising: acquiring two-dimensional keypoint sequence data of the human body to be identified under multiple simultaneous perspectives; constructing global keypoint coordinate representation data and skeleton-level offset representation data based on the two-dimensional keypoint sequence data; performing feature recognition on the global keypoint coordinate representation data and skeleton-level offset representation data in spatial dimension, perspective dimension and time dimension respectively, to obtain global keypoint coordinate representation features and skeleton-level offset representation features, and generating a three-dimensional coordinate data set.

[0007] Furthermore, the process of acquiring two-dimensional key point sequence data of the human body to be identified from multiple synchronous perspectives includes: acquiring synchronous image frames acquired by multiple acquisition devices deployed at different spatial locations and kept in time synchronization during the acquisition process; determining the two-dimensional coordinates of each key point of the human body based on the continuous image frames from each perspective, and assembling two-dimensional key point sequence data in chronological order.

[0008] Furthermore, global keypoint coordinate representation data and skeleton-level offset representation data are constructed, including: performing missing completion preprocessing on the two-dimensional keypoint sequence data to determine the processed two-dimensional keypoint sequence data; and constructing global keypoint coordinate representation data and skeleton-level offset representation data based on the processed two-dimensional keypoint sequence data.

[0009] Furthermore, the two-dimensional keypoint sequence data undergoes a missing completion preprocessing step to determine the processed two-dimensional keypoint sequence data. This includes: when there are missing two-dimensional keypoint data from at least one viewpoint, the missing data is filled in by interpolation, zero-value filling, or estimation based on keypoint data from other synchronous viewpoints to determine the filled two-dimensional keypoint sequence data; based on the filled two-dimensional keypoint sequence data, the input data is kept at a preset fixed dimension to determine the processed two-dimensional keypoint sequence data.

[0010] Furthermore, based on the processed two-dimensional keypoint sequence data, skeleton-level offset representation data is constructed, including: dividing human keypoints into root keypoints and non-root keypoints according to the parent-child topological relationship of the human skeleton; for root keypoints, retaining their two-dimensional coordinates in the processed two-dimensional keypoint sequence data as the root node reference coordinates in the skeleton-level offset representation data; for non-root keypoints, calculating the difference between the two-dimensional coordinates of the keypoint and the two-dimensional coordinates of its parent keypoint as the relative two-dimensional coordinates of the non-root keypoint in the skeleton-level offset representation data.

[0011] Furthermore, the feature recognition of spatial dimension, view dimension, and temporal dimension includes: identifying the spatial structural relationship between different human body key points within the same time frame through the spatial dimension recognition layer; identifying the complementary observation relationship between multiple synchronous viewpoints through the view dimension recognition layer; identifying the key point coordinate change relationship between consecutive time frames through the temporal dimension recognition layer; the spatial dimension recognition layer, view dimension recognition layer, and temporal dimension recognition layer constitute a feature recognition block in the order of spatial dimension, view dimension, and temporal dimension.

[0012] Furthermore, a 3D coordinate data set is generated based on the identified global keypoint coordinate representation features and skeleton-level offset representation features. This includes: normalizing and linearly mapping the global keypoint coordinate representation features to determine candidate results for global keypoint 3D coordinates; normalizing and linearly mapping the skeleton-level offset representation features to determine candidate results for skeleton-level 3D offsets, and recursively restoring the skeleton-level 3D offset candidate results from the root keypoint down according to the parent-child topological relationship of the human skeleton to determine the skeleton-level restored coordinate results; determining frame-by-frame, joint-by-joint adaptive weights based on the global keypoint coordinate representation features and skeleton-level offset representation features, performing coordinate-level fusion on the global keypoint 3D coordinate candidate results and skeleton-level restored coordinate results based on the frame-by-frame, joint-by-joint adaptive weights to determine the fused 3D coordinate results, and performing feature-level fusion on the global keypoint coordinate representation features and skeleton-level offset representation features to determine the feature-level fusion result; inputting the feature-level fusion result into the 3D coordinate residual compensation calibration module to generate a 3D coordinate residual compensation amount, and superimposing the fused 3D coordinate result and the 3D coordinate residual compensation amount according to a preset residual scaling factor to determine the 3D coordinate data set.

[0013] Furthermore, after determining the three-dimensional coordinate data set, the following steps are also included: during the training phase, optimization is performed using a joint error constraint function, which includes the final coordinate error, bone segment length consistency error, global coordinate auxiliary error, and hierarchical offset auxiliary error; among which, for the hierarchical offset auxiliary error, the root keypoints in the skeleton hierarchical offset representation data use the real three-dimensional coordinates as the supervision target, and the non-root keypoints in the skeleton hierarchical offset representation data use the offset of their real three-dimensional coordinates relative to the real three-dimensional coordinates of their parent keypoints as the supervision target.

[0014] According to one embodiment of the present invention, a multi-view human key point sequence data recognition device is also provided, comprising: an acquisition module for acquiring two-dimensional key point sequence data of the human body to be identified under multiple synchronous viewpoints; a construction module for constructing global key point coordinate representation data and skeleton level offset representation data based on the two-dimensional key point sequence data; and a recognition module for performing spatial dimension, viewpoint dimension, and time dimension feature recognition on the global key point coordinate representation data and skeleton level offset representation data respectively, to obtain global key point coordinate representation features and skeleton level offset representation features, and to generate a three-dimensional coordinate data set.

[0015] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0016] In this embodiment of the invention, a skeleton topology dual-representation collaborative recognition mechanism is used to collaboratively process global keypoint coordinate representation and skeleton hierarchical offset representation, thereby achieving separate expression of the overall human body structural relationship and the relative offset relationship between parent and child keypoints. Simultaneously, it utilizes keypoint spatial relationships, synchronous viewpoint complementary relationships, and continuous frame coordinate change relationships. Through a frame-by-frame, joint-by-joint adaptive weight fusion mechanism, the information contribution ratio of the two types of results is dynamically allocated according to the representation characteristics of different frames and different keypoints. Through a three-dimensional coordinate residual compensation calibration module, minor coordinate deviations are compensated while maintaining the stability of the fusion result, improving the accuracy, stability, and robustness under complex motion conditions of converting multi-view human keypoint sequence data into three-dimensional coordinate data. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 This is a flowchart of a method for recognizing multi-view human key point sequence data according to one embodiment of the present invention;

[0019] Figure 2 This is an overall structural diagram of a multi-view human key point sequence data recognition method according to one embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram illustrating the conversion between global keypoint coordinate representation data and skeleton hierarchical offset representation data in a multi-view human keypoint sequence data recognition method according to one embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the spatial-visual-temporal key point relationship recognition unit structure in a multi-view human key point sequence data recognition method according to one embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of the frame-by-frame, joint-by-joint adaptive weight fusion mechanism in a multi-view human key point sequence data recognition method according to one embodiment of the present invention.

[0023] Figure 6 This is a schematic diagram of the three-dimensional coordinate residual compensation calibration module in a multi-view human key point sequence data recognition method according to one embodiment of the present invention;

[0024] Figure 7 This is a structural block diagram of a multi-view human key point sequence data recognition device according to one embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] According to an embodiment of the present invention, a method for recognizing human key point sequence data from multiple perspectives is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] This method embodiment can be executed in an electronic device or similar computing device that includes memory and a processor. Taking operation on a terminal as an example, the terminal may include one or more processors (processors may include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), digital signal processing (DSP) chips, microcontroller units (MCUs), field-programmable gate arrays (FPGAs), neural network processors (NPUs), tensor processors (TPUs), artificial intelligence (AI) type processors, etc.) and memory for storing data. Optionally, the terminal may also include transmission devices, input / output devices, and display devices for communication functions. Those skilled in the art will understand that the above structural description is merely illustrative and does not limit the structure of the terminal. For example, the terminal may include more or fewer components than described above, or have a different configuration than described above.

[0029] The memory can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multi-view human key point sequence data recognition method in this embodiment of the invention. The processor executes various functional applications and data processing by running the computer program stored in the memory, thereby realizing the aforementioned multi-view human key point sequence data recognition method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0030] The transmission device is used to receive or send data via a network. Specific examples of the network mentioned above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0031] Display devices can be, for example, touchscreen liquid crystal displays (LCDs) and touch displays (also referred to as "touchscreens" or "touch displays"). The LCD allows users to interact with the user interface of the mobile terminal. In some embodiments, the mobile terminal has a graphical user interface (GUI), which allows users to interact with the GUI through finger contact and / or gestures on a touch-sensitive surface. Optional human-computer interaction functions include: creating web pages, drawing, word processing, creating electronic documents, playing games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital video, playing digital music, and / or web browsing, etc. Executable instructions for performing the above human-computer interaction functions are configured / stored in one or more processor-executable computer program products or readable storage media.

[0032] Figure 1 This is a flowchart of a multi-view human key point sequence data recognition method according to one embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0033] Step S110: Obtain the two-dimensional keypoint sequence data of the human body to be identified from multiple simultaneous viewpoints. The specific content is as follows:

[0034] In step S110, multiple acquisition devices are deployed at different spatial locations and maintain time synchronization during the acquisition process to obtain multi-view observation data of the same human body at the same time. The multiple acquisition devices observe the human body to be identified from different spatial locations, thereby obtaining observation data of the human body to be identified from multiple synchronized perspectives.

[0035] Two-dimensional keypoint sequence data can be extracted from synchronized image frames using a two-dimensional keypoint detection program, or it can be read from a dataset where keypoints have already been labeled or detected. For consecutive image frames from each viewpoint, the two-dimensional coordinates of each keypoint of the human body are obtained, and then arranged in chronological order to form two-dimensional keypoint sequence data.

[0036] In this embodiment, the two-dimensional keypoint sequence data can be represented as data consisting of five dimensions: batch size, number of consecutive frames, number of synchronized viewpoints, number of human body keypoints, and two-dimensional coordinate components. Here, batch size represents the number of data batches input at one time, number of consecutive frames represents the number of consecutive image frames, number of synchronized viewpoints represents the number of multiple synchronized viewpoints, number of human body keypoints represents the number of human body keypoints, and two-dimensional coordinate components represent the two-dimensional x-coordinate and y-coordinate components of each keypoint. In a specific embodiment, the number of synchronized viewpoints is 4, the number of consecutive frames is 27, and the number of human body keypoints is 17. The same number of synchronized viewpoints are used for input during the training and inference phases to ensure consistency in the dimensions of the input data.

[0037] Optionally, when there are missing 2D keypoint data from at least one viewpoint, they can be filled in before input by interpolation, zero-value filling, or estimation based on keypoint data from other synchronous viewpoints, so that the input data maintains a preset fixed dimension. Through the above processing, standardized, multi-view, and time-aligned 2D keypoint sequence data is obtained, providing an input basis for subsequent skeleton topology dual-representation collaborative recognition.

[0038] Step S120: Based on the two-dimensional keypoint sequence data, construct global keypoint coordinate representation data and skeleton-level offset representation data, as detailed below:

[0039] In step S120, global keypoint coordinate representation data and skeleton-level offset representation data are constructed based on the two-dimensional keypoint sequence data. Specifically, the two-dimensional keypoint sequence data is divided into two types of representation data: the first type of global keypoint coordinate representation data and the second type of skeleton-level offset representation data.

[0040] In one implementation, the global keypoint coordinate representation data is denoted as Xo, which directly uses the original two-dimensional keypoint sequence data. The global keypoint coordinate representation data is used to preserve the overall distribution relationship of each human body keypoint in the original two-dimensional coordinate system. This data reflects the overall structure of the human body, the positional constraints between different limbs, and the spatial layout of keypoints throughout the body.

[0041] In one implementation, the skeleton hierarchy offset representation data is denoted as Xh, which is converted from the original two-dimensional keypoint sequence data based on the parent-child topological relationship of the human skeleton. The parent-child topological relationship of the human skeleton is used to describe the connection hierarchy between keypoints, such as the parent-child relationship between root keypoints, torso keypoints, limb keypoints, and distal keypoints. Let p(j) be the parent keypoint index of the j-th keypoint. When p(j) = -1, the j-th keypoint is the root keypoint; when p(j) ≠ -1, the j-th keypoint is a non-root keypoint.

[0042] For root keypoints, since they have no parent keypoint, their two-dimensional coordinates in the original two-dimensional keypoint sequence data are retained as the root node reference coordinates in the skeleton-level offset representation. For non-root keypoints, the difference between the two-dimensional coordinates of the keypoint and the two-dimensional coordinates of its parent keypoint is calculated, and this difference is used as the relative two-dimensional coordinates of the non-root keypoint in the skeleton-level offset representation.

[0043] In one implementation, the transformation relationship can be expressed as follows: when p(j) = -1, Xh(j) = Xo(j); when p(j) ≠ -1, Xh(j) = Xo(j) - Xo(p(j)). Here, Xo(j) represents the two-dimensional coordinates of the j-th keypoint in the global keypoint coordinate representation data, and Xh(j) represents the two-dimensional coordinates of the j-th keypoint in the skeleton-level offset representation data.

[0044] It should be noted that the root keypoint is not a zero vector in the skeleton-level offset representation, but rather retains its original two-dimensional coordinates. This setting allows the skeleton-level offset representation to express both the local relative offsets between parent and child keypoints and to preserve the root node reference information required for subsequent recursive reconstruction.

[0045] Step S140 involves performing spatial, viewpoint, and temporal feature recognition on the global keypoint coordinate representation data and the skeleton hierarchical offset representation data, respectively, to obtain the global keypoint coordinate representation features and the skeleton hierarchical offset representation features, and generating a three-dimensional coordinate data set. The specific content is as follows:

[0046] In step S140, the global keypoint coordinate representation data and the skeleton hierarchical offset representation data are respectively input into two isomorphic representation channels. The two representation channels process data of different representation forms to obtain global keypoint coordinate representation features and skeleton hierarchical offset representation features. Specifically, the global keypoint coordinate representation data is used to preserve the overall distribution relationship of each human body keypoint in the original two-dimensional coordinate system, and the skeleton hierarchical offset representation data is used to express the local relative offset between keypoints according to the parent-child topological relationship of the human skeleton.

[0047] In one implementation, spatial, viewpoint, and temporal features are identified in the global keypoint coordinate representation data and the skeleton-level offset representation data, respectively. Spatial feature identification is used to identify the spatial structural relationships between different human keypoints within the same time frame; viewpoint feature identification is used to identify complementary observation relationships between multiple synchronous viewpoints; and temporal feature identification is used to identify keypoint coordinate changes between consecutive time frames. The spatial, viewpoint, and temporal feature identifications are arranged in the order of spatial, viewpoint, and temporal dimensions to form a feature identification block. These feature identification blocks can be stacked N times, where N is an integer greater than or equal to 2. In one specific embodiment, N=4.

[0048] In one optional implementation, for the input data of any representation channel, the batch dimension and the view dimension are first merged, and the time frame dimension and the two-dimensional coordinate channel dimension are merged to form keypoint features for spatial dimension recognition. Then, the features identified in the spatial dimension are rearranged into view features expanded in batches of time frames for view dimension recognition. Finally, the features identified in the view dimension are rearranged into time features expanded in batches of view views for time dimension recognition. Through the above rearrangement process, the same set of keypoint sequence data can be used to identify relationships in the three dimensions of keypoint space, synchronous view, and continuous time.

[0049] In one optional implementation, feature recognition in the spatial dimension, view dimension, and temporal dimension can be achieved using attention computation, normalization processing, and feedforward mapping. For the input features, a query matrix, key matrix, and value matrix are obtained through linear mapping. The attention computation results are then normalized, residual connected, and feedforward mapped to output the corresponding representation features. Thus, the global keypoint coordinate representation channel outputs the global keypoint coordinate representation features, and the skeleton level offset representation channel outputs the skeleton level offset representation features.

[0050] In one implementation, candidate 3D coordinates are determined based on global keypoint coordinate representation features and skeleton-level offset representation features. The global keypoint coordinate representation features are normalized and linearly mapped to obtain candidate 3D coordinates for global keypoints. These candidate 3D coordinates represent the candidate spatial coordinates of each human body keypoint in a unified 3D coordinate system. The skeleton-level offset representation features are normalized and linearly mapped to obtain candidate 3D offsets for skeleton levels. In these candidate 3D offsets, root keypoints are represented in 3D coordinate form, and non-root keypoints are represented in 3D offset form relative to their parent keypoints.

[0051] To enable the fusion of skeleton-level 3D offset candidate results with global keypoint 3D coordinate candidate results in the same coordinate system, the skeleton-level 3D offset candidate results are recursively restored to skeleton-level restored coordinate results. Specifically, for root keypoints, the 3D coordinates of the root keypoint in the skeleton-level 3D offset candidate results are directly used as the restored root keypoint 3D coordinates; for non-root keypoints, the 3D offset of the non-root keypoint is added to the restored 3D coordinates of its parent keypoint to obtain the restored 3D coordinates of the non-root keypoint, and this process is recursively performed from the root keypoint downwards according to the parent-child topology of the human skeleton. Through the above processing, the 3D offset output by the skeleton-level offset representation channel is transformed into a unified 3D coordinate system for fusion with the global keypoint 3D coordinate candidate results.

[0052] In one implementation, frame-by-frame, joint-by-joint adaptive weights are determined based on global keypoint coordinate representation features and skeleton-level offset representation features, and coordinate-level fusion and feature-level fusion are performed based on these frame-by-frame, joint-by-joint adaptive weights. First, average pooling is performed on the global keypoint coordinate representation features and skeleton-level offset representation features along the synchronous viewpoint dimension to obtain frame-by-frame keypoint feature representations. Average pooling is used to aggregate features of the same keypoint from multiple viewpoints into a unified representation, thereby obtaining features corresponding to time frames and human keypoints. Then, the frame-by-frame keypoint feature representations corresponding to the global keypoint coordinate representation channel and the skeleton-level offset representation channel are concatenated along the feature dimension and input into a multilayer perceptron. The multilayer perceptron includes a linear mapping layer, an activation function, and a sigmoid activation function, outputting frame-by-frame, joint-by-joint adaptive weights with values ​​ranging from 0 to 1.

[0053] In one alternative implementation, the frame-by-frame, joint-by-joint adaptive weight can be represented as g(t,j), where t represents the frame index and j represents the keypoint index. In coordinate-level fusion, the frame-by-frame, joint-by-joint adaptive weight is shared by the three coordinate components of the same keypoint; in feature-level fusion, the frame-by-frame, joint-by-joint adaptive weight is broadcast along the synchronous viewpoint dimension and the feature channel dimension, and then multiplied element-wise with the corresponding representation feature.

[0054] Furthermore, in coordinate-level fusion, based on frame-by-frame and joint-by-joint adaptive weights, the candidate 3D coordinates of global keypoints and the reconstructed coordinates of the skeleton level are weighted and fused to obtain the fused 3D coordinate result. In feature-level fusion, based on frame-by-frame and joint-by-joint adaptive weights, the representation features of global keypoint coordinates and the representation features of skeleton level offsets are fused to obtain the feature-level fusion result. Through the coordinated processing of coordinate-level fusion and feature-level fusion, the information contribution ratio of the two types of representation features can be dynamically adjusted according to the state of different time frames and different keypoints, reducing the information bias caused by the fixed-weight fusion method.

[0055] In one implementation, the feature-level fusion result is input into a 3D coordinate residual compensation calibration module to generate a 3D coordinate residual compensation amount. The 3D coordinate residual compensation calibration module may include a normalization layer and a linear mapping layer. The weights and biases of the linear mapping layer are initialized to zero, ensuring that the 3D coordinate residual compensation amount is close to zero in the early stages of training, and the model primarily outputs coordinate data based on the fused 3D coordinate result. As parameters are updated during training, the 3D coordinate residual compensation calibration module gradually learns the subtle coordinate deviations in the fused 3D coordinate result and generates corresponding residual compensation amounts. To avoid excessive perturbation of the fused 3D coordinate result by the residual compensation amount, it is suppressed according to a preset residual scaling factor before being superimposed on the fused 3D coordinate result. The final 3D coordinate data set can be represented as Y(t,j)=Yc(t,j)+λR(t,j), where λ represents the preset residual scaling factor, and R(t,j) represents the 3D coordinate residual compensation amount of the j-th keypoint in frame t. In a specific embodiment, λ=0.1. The final set of three-dimensional coordinate data is a set of three-dimensional spatial coordinates of key points of each human body in the same three-dimensional coordinate system.

[0056] Through the above processing, a skeleton topology dual-representation collaborative recognition mechanism is used to model and collaboratively process the global keypoint coordinate representation and the skeleton hierarchical offset representation respectively; spatial, viewpoint, and temporal features are used to identify the spatial relationships of keypoints, the complementary relationships of synchronous viewpoints, and the coordinate change relationships of continuous frames; a frame-by-frame, joint-by-joint adaptive weight fusion mechanism is used to dynamically integrate the two types of 3D coordinate results and two types of intermediate features; and a 3D coordinate residual compensation calibration module is used to compensate for the fused 3D coordinate results. Therefore, the accuracy, stability, and robustness under complex motion conditions of converting multi-view human keypoint sequence data into 3D coordinate data can be improved.

[0057] Based on steps S110 to S140 above, in this embodiment of the invention, a skeleton topology dual-representation collaborative recognition mechanism is used to collaboratively process the global key point coordinate representation and the skeleton hierarchical offset representation, thereby achieving the separate expression of the overall human body structural relationship and the relative offset relationship between parent and child key points. Simultaneously, the spatial relationship of key points, the complementary relationship of synchronous viewpoints, and the coordinate change relationship of continuous frames are utilized. Through a frame-by-frame, joint-by-joint adaptive weight fusion mechanism, the information contribution ratio of the two types of results is dynamically allocated according to the representation characteristics of different frames and different key points. Through a three-dimensional coordinate residual compensation calibration module, small coordinate deviations are compensated while maintaining the stability of the fusion result, thereby improving the accuracy, stability, and robustness under complex motion conditions of converting multi-view human key point sequence data into three-dimensional coordinate data.

[0058] The multi-view human key point sequence data recognition method of the present invention acquires two-dimensional key point sequence data of the human body to be identified under multiple synchronous viewpoints, including: acquiring synchronous image frames acquired by multiple acquisition devices respectively deployed at different spatial locations and kept in time synchronization during the acquisition process; determining the two-dimensional coordinates of each key point of the human body based on the continuous image frames of each viewpoint, and assembling two-dimensional key point sequence data in chronological order, thereby providing a time-aligned and viewpoint-complementary input basis for subsequent multi-view human key point three-dimensional coordinate recognition.

[0059] Furthermore, the construction of global keypoint coordinate representation data and skeleton hierarchical offset representation data includes: performing missing data completion preprocessing on the two-dimensional keypoint sequence data to determine the processed two-dimensional keypoint sequence data; and constructing global keypoint coordinate representation data and skeleton hierarchical offset representation data based on the processed two-dimensional keypoint sequence data. This ensures that the input data maintains a preset fixed dimension, thus providing a complete and dimensionally consistent input foundation for the subsequent construction of global keypoint coordinate representation data and skeleton hierarchical offset representation data.

[0060] Furthermore, the two-dimensional keypoint sequence data undergoes a missing-filling preprocessing step to determine the processed two-dimensional keypoint sequence data. This includes: when at least one viewpoint has missing two-dimensional keypoint data, it is filled in by interpolation, zero-value filling, or estimation based on keypoint data from other synchronous views to determine the filled two-dimensional keypoint sequence data; based on the filled two-dimensional keypoint sequence data, the input data is kept at a preset fixed dimension to determine the processed two-dimensional keypoint sequence data, thereby ensuring the integrity and dimensional consistency of the multi-view input data and providing a reliable foundation for the subsequent construction of global keypoint coordinate representation data and skeleton hierarchical offset representation data.

[0061] Furthermore, based on the processed two-dimensional keypoint sequence data, skeleton-level offset representation data is constructed, including: dividing human keypoints into root keypoints and non-root keypoints according to the parent-child topological relationship of the human skeleton; for root keypoints, retaining their two-dimensional coordinates in the processed two-dimensional keypoint sequence data as root node reference coordinates in the skeleton-level offset representation data; for non-root keypoints, calculating the difference between the two-dimensional coordinates of the keypoint and the two-dimensional coordinates of its parent keypoint as the relative two-dimensional coordinates of the non-root keypoint in the skeleton-level offset representation data, thereby expressing the local relative offset of parent-child keypoints while retaining the root node reference information required for subsequent recursive reconstruction.

[0062] Furthermore, the feature recognition in the spatial, viewpoint, and temporal dimensions includes: identifying the spatial structural relationships between different human keypoints within the same time frame through a spatial dimension recognition layer; identifying complementary observation relationships between multiple synchronous viewpoints through a viewpoint recognition layer; and identifying keypoint coordinate changes between consecutive time frames through a temporal dimension recognition layer. The spatial, viewpoint, and temporal dimension recognition layers, arranged in the order of spatial, viewpoint, and temporal dimensions, constitute a feature recognition block. This enables sequential relationship recognition across the three dimensions of keypoint space, synchronous viewpoint, and consecutive time, improving the accuracy and robustness of multi-view human keypoint sequence feature recognition.

[0063] Furthermore, a 3D coordinate data set is generated based on the identified global keypoint coordinate representation features and skeleton-level offset representation features. This includes: normalizing and linearly mapping the global keypoint coordinate representation features to determine candidate results for global keypoint 3D coordinates; normalizing and linearly mapping the skeleton-level offset representation features to determine candidate results for skeleton-level 3D offsets, and recursively restoring the skeleton-level 3D offset candidate results from the root keypoint downwards according to the parent-child topological relationship of the human skeleton to determine the skeleton-level restored coordinate results; determining frame-by-frame, joint-by-joint adaptive weights based on the global keypoint coordinate representation features and skeleton-level offset representation features, and then using frame-by-frame, joint-by-joint adaptive weights... The system performs coordinate-level fusion of candidate 3D coordinates of global keypoints and skeleton-level reconstructed coordinates based on weights to determine the fused 3D coordinate result. It also performs feature-level fusion of global keypoint coordinate representation features and skeleton-level offset representation features to determine the feature-level fusion result. This feature-level fusion result is input into the 3D coordinate residual compensation calibration module to generate a 3D coordinate residual compensation amount. The fused 3D coordinate result and the 3D coordinate residual compensation amount are then superimposed according to a preset residual scaling factor to determine the 3D coordinate data set. This dynamically adjusts the information contribution ratio of the two types of representations and suppresses residual disturbances, improving the accuracy, stability, and robustness of the 3D coordinate data set under complex action conditions.

[0064] Furthermore, after determining the 3D coordinate dataset, the training phase includes optimization using a joint error constraint function. This function includes final coordinate error, bone segment length consistency error, global coordinate auxiliary error, and hierarchical offset auxiliary error. Specifically, for the hierarchical offset auxiliary error, the root keypoints in the skeleton hierarchical offset representation data use their true 3D coordinates as the supervision target, while the non-root keypoints in the skeleton hierarchical offset representation data use the offset of their true 3D coordinates relative to the true 3D coordinates of their parent keypoints as the supervision target. This improves the accuracy of the 3D coordinate dataset, the consistency of the skeleton structure, and robustness under complex motion conditions.

[0065] Another method for recognizing multi-view human key point sequence data according to one embodiment of the present invention, such as Figures 2-6 As shown, the method includes the following steps:

[0066] Step 201: Obtain two-dimensional key point sequence data of the human body to be identified from multiple synchronous perspectives.

[0067] Multiple acquisition devices are deployed at different spatial locations and maintain time synchronization during the acquisition process to obtain multi-view observation data of the same human body at the same time. The two-dimensional keypoint sequence data can be extracted from synchronized image frames through a two-dimensional keypoint detection program, or it can be read from a dataset where keypoints have been labeled or detected. For consecutive image frames from each viewpoint, the two-dimensional coordinates of each keypoint of the human body are obtained and assembled into a two-dimensional keypoint sequence data in chronological order.

[0068] In this embodiment, the two-dimensional keypoint sequence data can be represented as follows: Where B represents the batch size, F represents the number of consecutive frames, V represents the number of synchronized viewpoints, J represents the number of human keypoints, and 2 represents the two-dimensional x-coordinate and y-coordinate components of each keypoint. In a specific embodiment, V=4, F=27, and J=17. The same number of synchronized viewpoint inputs are used in both the training and inference phases to ensure consistency in the dimensionality of the input data.

[0069] Optionally, when the two-dimensional keypoint data from a certain perspective is missing, it can be filled in before input by interpolation, zero-value filling, or estimation based on keypoint data from other synchronous perspectives, so that the input data maintains a preset fixed dimension. Through the above processing, standardized, multi-view, and time-aligned two-dimensional keypoint sequence data is obtained, providing an input basis for subsequent skeleton topology dual-representation collaborative recognition.

[0070] Step S202: Construct global keypoint coordinate representation data and skeleton level offset representation data based on the two-dimensional keypoint sequence data.

[0071] Two-dimensional keypoint sequence data is divided into two categories of representation data. The first category is global keypoint coordinate representation data, denoted as... It directly uses the original two-dimensional keypoint sequence data to preserve the overall distribution relationship of each human body keypoint in the original two-dimensional coordinate system. This representation can reflect the overall structure of the human body, the positional constraints between different limbs, and the spatial layout of keypoints throughout the body.

[0072] The second category is skeleton-level offset representation data, denoted as... It is derived from the original two-dimensional keypoint sequence data based on the parent-child topological relationships of the human skeleton. The parent-child topological relationships of the human skeleton are used to describe the connection hierarchy between keypoints, such as the parent-child relationships between root keypoints, torso keypoints, limb keypoints, and distal keypoints. Let the parent keypoint index of the j-th keypoint be . ,when When the j-th keypoint is the root keypoint; when When the j-th key point is not a root key point.

[0073] For root keypoints, since they have no parent keypoints, their two-dimensional coordinates in the original two-dimensional keypoint sequence data are retained as the root node reference coordinates in the skeleton-level offset representation. For non-root keypoints, the difference between the two-dimensional coordinates of the keypoint and its parent keypoint is calculated, and this difference is used as the relative two-dimensional coordinates of the non-root keypoint in the skeleton-level offset representation. The transformation relationship can be expressed as:

[0074]

[0075]

[0076] in, This represents the two-dimensional coordinates of the j-th keypoint in the global keypoint coordinate representation data. This represents the two-dimensional coordinates of the j-th keypoint in the skeleton-level offset representation data. It's important to note that the root keypoint is not a zero vector in the skeleton-level offset representation; instead, its original two-dimensional coordinates are retained. This setting allows the skeleton-level offset representation to express both the local relative offsets between parent and child keypoints and to preserve the root node reference information needed for subsequent recursive reconstruction.

[0077] Step S203: Use the spatial-visual-temporal key point relationship recognition unit to perform feature recognition on the global key point coordinate representation data and the skeleton level offset representation data respectively.

[0078] Representing global keypoint coordinates data and skeleton level offset characterization data Two isomorphic representation channels are input separately. Both channels include a spatial-view-temporal keypoint relationship recognition unit, but process data with different representation formats to obtain global keypoint coordinate representation features. and skeleton level offset characterization features .

[0079] like Figure 4As shown, the spatial-visual-temporal keypoint relationship recognition unit includes a spatial dimension recognition layer, a viewpoint dimension recognition layer, and a temporal dimension recognition layer. The spatial dimension recognition layer is used to identify the spatial structural relationships between different human body keypoints within the same time frame; the viewpoint dimension recognition layer is used to identify the complementary observation relationships between multiple synchronous viewpoints; and the temporal dimension recognition layer is used to identify the keypoint coordinate change relationships between consecutive time frames. The spatial dimension recognition layer, viewpoint dimension recognition layer, and temporal dimension recognition layer constitute a spatial-visual-temporal keypoint relationship recognition block in the order of spatial dimension recognition, viewpoint dimension recognition, and temporal dimension recognition. This recognition block can be stacked N times, where N is an integer greater than or equal to 2. In a specific embodiment, N=4.

[0080] In one optional implementation, for the input data of any representation channel, the batch dimension and the view dimension are first merged, and the time frame dimension and the two-dimensional coordinate channel dimension are merged to form keypoint features for spatial dimension recognition. Then, the features identified in the spatial dimension are rearranged into view features expanded in batches of time frames for view dimension recognition. Finally, the features identified in the view dimension are rearranged into time features expanded in batches of view views for time dimension recognition. Through the above rearrangement process, the same set of keypoint sequence data can be used to identify relationships in the three dimensions of keypoint space, synchronous view, and continuous time.

[0081] Specifically, let the input feature of any representation channel be... Where C represents the number of keypoint coordinate channels. First, the batch dimension and view dimension are merged, and the time frame dimension and coordinate channel dimension are merged to obtain the spatial dimension input features. ,in, Then, through linear mapping... Projection is a feature of high-dimensional space And superimposed spatial location coding The spatial embedding result is obtained:

[0082] ,in, ;

[0083] .

[0084] in, Represents the spatial dimension recognition layer. Indicates spatial location encoding, This represents the embedded features after spatial dimension relationship recognition.

[0085] After obtaining the spatial embedding results, Rearranged into viewpoint-dimensional input features expanded in batches of time frames:

[0086] ,in, .

[0087] Then overlay the view position code Furthermore, by using a perspective dimension recognition layer to identify complementary relationships across multiple perspectives, the following results were obtained:

[0088] .

[0089] in, Represents the viewpoint dimension recognition layer. Indicates the viewpoint position encoding. This represents the embedded features after the recognition of the viewpoint-dimensional relationship.

[0090] After obtaining the view embedding results, Rearranged into time-dimensional input features expanded in batches based on viewpoints:

[0091] .

[0092] Then overlay time and location codes Furthermore, by identifying the coordinate change relationships between consecutive frames through a time-dimensional recognition layer, the following results were obtained:

[0093] .

[0094] In one alternative implementation, the spatial-visual-temporal key point relationship recognition unit can be implemented using attention computation, normalization processing, and feedforward mapping. For input features... The query matrix Q, key matrix K, and value matrix are obtained through linear mapping. Attention calculation can be expressed as:

[0095] .

[0096] Where d represents the dimension of the key vector. Represents a value matrix, This is distinguished from V, which represents the number of synchronized viewpoints. The attention calculation results are normalized, residually connected, and fed-forward mapped before outputting the corresponding representation features. Thus, the global keypoint coordinate representation channel is output. Skeleton level offset characterizes channel output .

[0097] make This represents an attention mapping module that performs a linear mapping on the input X to obtain a query matrix, a key matrix, and a value matrix, and obtains the output according to the attention calculation method described above.

[0098] Let the input features of any recognition layer be... The intermediate feature is Then the residual connection and feedforward mapping process can be expressed as:

[0099] ;

[0100] ;

[0101] .

[0102] in, Presentation layer normalization operation, and Let GeLU denote the linear projection matrix in the feedforward mapping, and let GeLU denote the activation function. This represents the output feature of the corresponding recognition layer.

[0103] Step S204: Determine the candidate results of three-dimensional coordinates based on the global key point coordinate representation features and the skeleton level offset representation features.

[0104] Global keypoint coordinate representation features Normalization and linear mapping are performed to obtain candidate results for the 3D coordinates of global key points. This result is used to represent the candidate spatial coordinates of key points on each human body in a unified three-dimensional coordinate system.

[0105] Characteristic representation of skeleton hierarchical offset Normalization and linear mapping are performed to obtain candidate results for 3D offset at the skeleton level. In this result, root keypoints are represented in 3D coordinates, and non-root keypoints are represented in 3D offsets relative to their parent keypoints. To enable the skeleton-level 3D offset candidate results to be fused with the global keypoint 3D coordinate candidate results in the same coordinate system, the skeleton-level 3D offset candidate results need to be recursively restored to the skeleton-level restored coordinate results. .

[0106] Specifically, for root keypoints, the 3D coordinates of the root keypoints in the candidate 3D offset results of the skeleton hierarchy are directly used as the restored 3D coordinates of the root keypoints; for non-root keypoints, the 3D offset of the non-root keypoint is added to the restored 3D coordinates of its parent keypoint, and this is used as the restored 3D coordinates of the non-root keypoint. The process is then recursively executed from the root keypoint downwards according to the parent-child topology of the human skeleton. The above restoration process can be represented as:

[0107] when hour, ;

[0108] when hour, .

[0109] in, This represents the skeleton-level reconstructed coordinate result of the j-th keypoint. Through the above processing, the 3D offset output by the skeleton-level offset characterization channel is transformed into a unified 3D coordinate system for fusion with the global keypoint 3D coordinate candidate results.

[0110] Step S205: Determine frame-by-frame, joint-by-joint adaptive weights based on global keypoint coordinate representation features and skeleton-level offset representation features, and perform coordinate-level fusion and feature-level fusion based on the frame-by-frame, joint-by-joint adaptive weights.

[0111] like Figure 5 As shown, the features of global keypoint coordinates are first characterized. and skeleton level offset characterization features Average pooling is performed along the synchronous viewpoint dimension to obtain frame-by-frame keypoint feature representations. The average pooling is used to aggregate the features of the same keypoint from multiple viewpoints into a unified representation, thereby obtaining features corresponding to time frames and human keypoints.

[0112] Then, the frame-by-frame keypoint feature representations corresponding to the global keypoint coordinate representation channel and the frame-by-frame keypoint feature representations corresponding to the skeleton level offset representation channel are concatenated along the feature dimension and input into a multilayer perceptron. The multilayer perceptron includes a linear mapping layer, an activation function, and a sigmoid activation function, and outputs frame-by-frame joint-by-joint adaptive weights with values ​​ranging from 0 to 1.

[0113] In one alternative implementation, the frame-by-frame, joint-by-joint adaptive weights can be expressed as:

[0114] .

[0115] in, This indicates that average pooling is performed along the synchronous viewpoint dimension, where t represents the frame index and j represents the keypoint index. This represents the adaptive weight corresponding to the j-th keypoint in frame t. In coordinate-level fusion, this adaptive weight is shared by the three coordinate components of the same keypoint; in feature-level fusion, the frame-by-frame, joint-by-joint adaptive weight is broadcast in the synchronous viewpoint dimension and feature channel dimension, and then multiplied element-wise with the corresponding representation feature.

[0116] Furthermore, in coordinate-level fusion, based on the frame-by-frame, joint-by-joint adaptive weights, the candidate results of the global keypoint 3D coordinates are... and the coordinate results of the skeleton hierarchy reconstruction Weighted fusion is performed to obtain the fused 3D coordinate result. It can be represented as:

[0117] .

[0118] In feature-level fusion, the global keypoint coordinates are characterized by the frame-by-frame, joint-by-joint adaptive weights. and the skeleton level offset characterization features The fusion is performed to obtain the feature-level fusion result. It can be represented as:

[0119] .

[0120] Through the coordinated processing of coordinate-level fusion and feature-level fusion, the information contribution ratio of the two types of representation features can be dynamically adjusted according to the state of different time frames and different key points, thereby reducing the information bias caused by the fixed-weight fusion method.

[0121] Step S206: Input the feature-level fusion result into the three-dimensional coordinate residual compensation calibration module to generate the three-dimensional coordinate residual compensation amount, and superimpose the fused three-dimensional coordinate result with the three-dimensional coordinate residual compensation amount according to the preset residual scaling factor to obtain the final three-dimensional coordinate data set.

[0122] like Figure 6 As shown, the feature-level fusion result The input is a 3D coordinate residual compensation calibration module, which generates a 3D coordinate residual compensation amount R. This module may include a normalization layer and a linear mapping layer. The weights and biases of the linear mapping layer are initialized to zero, ensuring that the 3D coordinate residual compensation amount is close to zero in the early stages of training. The model primarily outputs coordinate data based on the fused 3D coordinate results.

[0123] As parameters are updated during training, the 3D coordinate residual compensation and calibration module gradually learns the subtle coordinate deviations in the fused 3D coordinate results and generates corresponding residual compensation values. To avoid excessive disturbance to the fused 3D coordinate results by the residual compensation values, they are suppressed according to a preset residual scaling factor before being superimposed on the fused 3D coordinate results. The final 3D coordinate dataset can be represented as follows:

[0124] .

[0125] in, This indicates the preset residual scaling factor. This represents the 3D coordinate residual compensation amount for the j-th keypoint in the t-th frame. In one specific embodiment... The final set of 3D coordinate data is the set of 3D spatial coordinates of each key point on the human body in the same 3D coordinate system, which can be represented as:

[0126] .

[0127] During the training phase, this embodiment can employ a joint error constraint function for optimization. This joint error constraint function includes the final coordinate error, bone segment length consistency error, global coordinate auxiliary error, and hierarchical offset auxiliary error. The joint error constraint function can be expressed as:

[0128] .

[0129] in, Indicates the final coordinate error. This indicates the error in the consistency of bone segment length. The coordinates of global key points represent the channel auxiliary error. The skeleton-level offset characterizes the channel auxiliary error; α, β, and γ represent the weight coefficients of the corresponding error terms. For the hierarchical offset auxiliary error, the root keypoint uses its true 3D coordinates as the supervision target, while non-root keypoints use the offset of their true 3D coordinates relative to the true 3D coordinates of their parent keypoint as the supervision target. In a specific embodiment, , , .

[0130] Through steps S201 to S206 above, this embodiment utilizes a skeleton topology dual-representation collaborative recognition mechanism to model and collaboratively process global keypoint coordinate representation and skeleton hierarchical offset representation respectively; it uses a space-view-time keypoint relationship recognition unit to identify keypoint spatial relationships, synchronous viewpoint complementary relationships, and continuous frame coordinate change relationships; it uses a frame-by-frame, joint-by-joint adaptive weight fusion mechanism to dynamically integrate two types of 3D coordinate results and two types of intermediate features; and it uses a 3D coordinate residual compensation calibration module to compensate for the fused 3D coordinate results. Therefore, the accuracy, stability, and robustness under complex motion conditions of converting multi-view human keypoint sequence data into 3D coordinate data can be improved.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0132] This invention also provides a multi-view human key point sequence data recognition device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0133] Figure 7 According to one embodiment of the present invention, a multi-view human key point sequence data recognition device includes:

[0134] The acquisition module 301 is used to acquire two-dimensional key point sequence data of the human body to be identified under multiple synchronous viewpoints;

[0135] Module 302 is used to construct global keypoint coordinate representation data and skeleton level offset representation data based on two-dimensional keypoint sequence data;

[0136] The recognition module 303 is used to perform spatial dimension, view dimension and time dimension feature recognition on the global key point coordinate representation data and the skeleton level offset representation data respectively, to obtain the global key point coordinate representation features and the skeleton level offset representation features, and generate a three-dimensional coordinate data set.

[0137] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0138] According to one embodiment of the present invention, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described multi-view human key point sequence data recognition method during runtime.

[0139] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0140] Step S1: Obtain two-dimensional key point sequence data of the human body to be identified from multiple simultaneous viewpoints;

[0141] Step S2: Based on the two-dimensional keypoint sequence data, construct global keypoint coordinate representation data and skeleton hierarchical offset representation data;

[0142] Step S3: Perform spatial dimension, view dimension and time dimension feature recognition on the global key point coordinate representation data and skeleton level offset representation data respectively to obtain the global key point coordinate representation features and skeleton level offset representation features, and generate a three-dimensional coordinate data set.

[0143] According to one embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the storage medium is located to execute the above-described multi-view human key point sequence data recognition method.

[0144] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0145] Step S1: Obtain two-dimensional key point sequence data of the human body to be identified from multiple simultaneous viewpoints;

[0146] Step S2: Based on the two-dimensional keypoint sequence data, construct global keypoint coordinate representation data and skeleton hierarchical offset representation data;

[0147] Step S3: Perform spatial dimension, view dimension and time dimension feature recognition on the global key point coordinate representation data and skeleton level offset representation data respectively to obtain the global key point coordinate representation features and skeleton level offset representation features, and generate a three-dimensional coordinate data set.

[0148] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0149] According to one embodiment of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described multi-view human key point sequence data recognition method.

[0150] Optionally, in this embodiment, the above-mentioned computer program product can be configured as a computer program that performs the following steps:

[0151] Step S1: Obtain two-dimensional key point sequence data of the human body to be identified from multiple simultaneous viewpoints;

[0152] Step S2: Based on the two-dimensional keypoint sequence data, construct global keypoint coordinate representation data and skeleton hierarchical offset representation data;

[0153] Step S3: Perform spatial dimension, view dimension and time dimension feature recognition on the global key point coordinate representation data and skeleton level offset representation data respectively to obtain the global key point coordinate representation features and skeleton level offset representation features, and generate a three-dimensional coordinate data set.

[0154] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0155] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0160] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for recognizing human key point sequence data from multiple perspectives, characterized in that, include: Acquire two-dimensional keypoint sequence data of the human body to be identified from multiple simultaneous viewpoints; Based on the two-dimensional keypoint sequence data, global keypoint coordinate representation data and skeleton hierarchical offset representation data are constructed. The global keypoint coordinate representation data and the skeleton hierarchical offset representation data are subjected to feature recognition in spatial dimension, view dimension and time dimension respectively, to obtain global keypoint coordinate representation features and skeleton hierarchical offset representation features, and generate a three-dimensional coordinate data set.

2. The method for recognizing multi-view human key point sequence data according to claim 1, characterized in that, Acquire the two-dimensional keypoint sequence data of the human body to be identified from multiple simultaneous viewpoints, including: Acquire synchronized image frames from multiple acquisition devices deployed in different spatial locations, maintaining time synchronization during the acquisition process; Based on continuous image frames from various perspectives, the two-dimensional coordinates of each key point of the human body are determined, and the two-dimensional key point sequence data is composed in chronological order.

3. The method for recognizing multi-view human key point sequence data according to claim 2, characterized in that, Constructing the global keypoint coordinate representation data and the skeleton hierarchical offset representation data includes: The two-dimensional keypoint sequence data is preprocessed with missing data completion to determine the processed two-dimensional keypoint sequence data; Based on the processed two-dimensional keypoint sequence data, the global keypoint coordinate representation data and the skeleton level offset representation data are constructed.

4. The method for recognizing multi-view human key point sequence data according to claim 3, characterized in that, The two-dimensional keypoint sequence data is preprocessed with missing data completion to determine the processed two-dimensional keypoint sequence data, including: When there are missing 2D keypoint data from at least one viewpoint, the missing data is filled by interpolation, zero-value filling, or estimation based on keypoint data from other synchronous views, and the filled 2D keypoint sequence data is determined. Based on the completed two-dimensional keypoint sequence data, the input data is kept at a preset fixed dimension to determine the processed two-dimensional keypoint sequence data.

5. The method for recognizing multi-view human key point sequence data according to claim 3, characterized in that, Based on the processed two-dimensional keypoint sequence data, the skeleton-level offset representation data is constructed, including: Based on the parent-child topological relationship of the human skeleton, the key points of the human body are divided into root key points and non-root key points. For the root keypoint, its two-dimensional coordinates in the processed two-dimensional keypoint sequence data are retained as the root node reference coordinates in the skeleton hierarchical offset characterization data; For the non-root keypoint, the difference between the two-dimensional coordinates of the keypoint and the two-dimensional coordinates of its parent keypoint is calculated, and used as the relative two-dimensional coordinates of the non-root keypoint in the skeleton-level offset representation data.

6. The method for recognizing multi-view human key point sequence data according to claim 1, characterized in that, The feature recognition of the spatial dimension, view dimension, and temporal dimension includes: The spatial structural relationship between different human body key points within the same time frame is identified by the spatial dimension recognition layer. The complementary observation relationship between multiple synchronous viewpoints is identified through the viewpoint dimension recognition layer; The temporal dimension recognition layer identifies the coordinate changes of key points between consecutive time frames. The spatial dimension recognition layer, the view dimension recognition layer, and the time dimension recognition layer constitute a feature recognition block in the order of spatial dimension, view dimension, and time dimension.

7. The method for recognizing multi-view human key point sequence data according to claim 1, characterized in that, The three-dimensional coordinate data set is generated based on the identified global keypoint coordinate representation features and the skeleton hierarchical offset representation features, including: The global keypoint coordinate representation features are normalized and linearly mapped to determine candidate results of global keypoint 3D coordinates; the skeleton level offset representation features are normalized and linearly mapped to determine candidate results of skeleton level 3D offset; and the skeleton level 3D offset candidate results are recursively restored from the root keypoint downwards according to the parent-child topological relationship of the human skeleton to determine the skeleton level restored coordinate results. Based on the global keypoint coordinate representation features and the skeleton level offset representation features, frame-by-frame and joint-by-joint adaptive weights are determined. Based on the frame-by-frame and joint-by-joint adaptive weights, coordinate-level fusion is performed on the global keypoint 3D coordinate candidate results and the skeleton level restored coordinate results to determine the fused 3D coordinate results. Feature-level fusion is performed on the global keypoint coordinate representation features and the skeleton level offset representation features to determine the feature-level fusion results. The feature-level fusion result is input into the three-dimensional coordinate residual compensation calibration module to generate a three-dimensional coordinate residual compensation amount. The fused three-dimensional coordinate result and the three-dimensional coordinate residual compensation amount are then superimposed according to a preset residual scaling factor to determine the three-dimensional coordinate data set.

8. The method for recognizing multi-view human key point sequence data according to claim 7, characterized in that, After determining the three-dimensional coordinate data set, the method further includes: During the training phase, a joint error constraint function is used for optimization. The joint error constraint function includes the final coordinate error, bone segment length consistency error, global coordinate auxiliary error, and hierarchical offset auxiliary error. Specifically, for the hierarchical offset auxiliary error, the root keypoints in the skeleton hierarchical offset representation data use the real 3D coordinates as the supervision target, and the non-root keypoints in the skeleton hierarchical offset representation data use the offset of their real 3D coordinates relative to the real 3D coordinates of their parent keypoints as the supervision target.

9. A multi-view human key point sequence data recognition device, characterized in that, include: The acquisition module is used to acquire two-dimensional key point sequence data of the human body to be identified from multiple synchronous perspectives; The construction module is used to construct global keypoint coordinate representation data and skeleton level offset representation data based on the two-dimensional keypoint sequence data; The identification module is used to perform spatial dimension, view dimension and time dimension feature identification on the global key point coordinate representation data and the skeleton level offset representation data respectively, to obtain the global key point coordinate representation features and the skeleton level offset representation features, and generate a three-dimensional coordinate data set.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the multi-view human key point sequence data recognition method of any one of claims 1 to 8.

Images