A multi-view based real-time martial arts three-dimensional human posture estimation method and system

Abstract

The application provides a multi-view-based real-time martial arts three-dimensional human body posture estimation method and system. The method synchronously collects RGB image data of martial arts postures through multiple cameras; a martial arts picture data set suitable for three-dimensional measurement is constructed to improve the accuracy of triangulation for solving three-dimensional joint nodes; an LRANet backbone network is designed to enhance the distinguishing ability of a two-dimensional human body posture estimation algorithm for different human body joint nodes, and the LRANet backbone network is further trained and deployed to batch calculate multiple-view two-dimensional human body joint nodes; the internal parameters and external parameters of the cameras are solved through a camera calibration method; the two-dimensional coordinates of the human body joint nodes under multiple views and the confidence weight information of each point detected by a pre-trained posture estimation algorithm are used to calculate martial arts three-dimensional human body joint nodes through a learnable triangulation method. The application can accurately detect martial arts three-dimensional human body joint nodes in real time in a relatively complex environment, and the whole system is easy to deploy.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional human posture estimation, specifically relating to a real-time martial arts three-dimensional human posture estimation method and system based on multiple perspectives. Background Technology

[0002] Human pose estimation is an important research area in computer vision, referring to the detection and identification of the position and pose angle information of a human body from images or videos. It involves knowledge from multiple fields such as computer vision, machine learning, and image processing. With the development of deep learning and the improvement of hardware computing power, the accuracy and speed of human pose estimation have been greatly improved, enabling this technology to be widely used in many fields, such as human-computer interaction, virtual reality, game and film production, and sports.

[0003] In the field of sports, every movement of an athlete is a crucial reference standard for referees' scoring. Traditional sports scoring methods mainly rely on human analysis and judgment. Training each referee requires a long period of time, and in sports competitions, the same movement often needs to be judged by multiple referees or coaches. This incurs high human resource costs and is also susceptible to errors and omissions due to differing subjective judgments. Human posture estimation technology addresses this by monitoring athletes' movements and postures, assisting coaches in analyzing athletes' techniques, and providing objective scores for each movement. This allows athletes to promptly summarize experiences, improve shortcomings, and enhance training effectiveness.

[0004] Although single-view 2D / 3D human pose estimation has achieved excellent results in many simple sports scenarios (Real-time 3D human body shape and pose reconstruction method and device under multi-view camera (CN110599540A)), many problems still exist in martial arts action scoring tasks that require more accurate 3D human pose information.

[0005] (1) 3D labeled datasets are lacking and difficult to produce: Most existing 3D human pose datasets are collected by motion capture systems in indoor environments, and it is difficult and expensive to obtain accurate 3D labeled information.

[0006] (2) Uncertainty in 2D to 3D conversion: Most of the data collected so far is based on RGB format and lacks depth information. In two-stage 3D human pose estimation methods, the 2D pose is first estimated from a single RGB image, and then the 2D pose is upsampled to 3D. These methods all have an uncertainty problem because multiple 3D points can be mapped to the same 2D point.

[0007] (3) Occlusion problem: The actual human body being detected may be occluded by other objects or exceed the field of view of the shooting device. There is also the problem of self-occlusion when the shooting angle is on the side, which makes the 2D / 3D human pose estimation under single view may become more inaccurate.

[0008] (4) Problem of complex postures: Martial arts movements are more complex than the posture data in common datasets, which brings adjustments to the correct acquisition of three-dimensional information.

[0009] (5) Application problems: On the one hand, many models achieve good results on public datasets, but their generalization in real-world environments is not good enough to meet actual needs; on the other hand, the accuracy, speed and hardware requirements of human pose estimation models often cannot achieve a good balance. Pursuing higher accuracy requires better computing power and other resources, but this will sacrifice the real-time performance of the model. Conversely, pursuing the real-time performance of the model will reduce its performance. Summary of the Invention

[0010] To address the aforementioned issues, this invention provides a multi-view real-time martial arts 3D human pose estimation method and system based on large-kernel attention. It avoids the problems of scarce and difficult-to-create 3D datasets by using a 2D-to-3D upscaling approach; it improves the uncertainty and occlusion issues in 2D-to-3D conversion by employing multi-view information fusion; a custom martial arts dataset makes the model more suitable for complex pose changes and more consistent with 3D computation; and the proposed LRANet-SimCC model achieves real-time detection speed while maintaining high accuracy. The entire system solution features low computational cost, strong scalability, real-time operation, low cost, and easy deployment.

[0011] The objective of this invention is achieved by at least one of the following technical solutions.

[0012] A real-time martial arts 3D human pose estimation method based on multiple perspectives includes the following steps:

[0013] S1. Set up multiple network cameras around the indoor venue scene, and calibrate the internal and external parameters of each camera using the camera calibration method;

[0014] S2. Create a 2D dataset of martial arts human postures;

[0015] S3. Multiple calibrated network cameras are used to simultaneously capture human martial arts movements to obtain image data. The image data is preprocessed and encoded into the required RGB image. Based on this, the RTMDet target detection model is used to detect human target boxes from the RGB image. The trained LRANet-SimCC model is used to infer human movements in the target boxes in batches, and the two-dimensional coordinates and confidence weight information of human joints under each view are extracted.

[0016] S4. Triangulate the extracted two-dimensional coordinates and confidence weight information to obtain the three-dimensional joint coordinate information of the human body.

[0017] Furthermore, in step S1, the intrinsic and extrinsic parameters of each camera are calibrated using a camera calibration method, as follows:

[0018] Select one of the network cameras as the reference coordinate system, and calibrate the intrinsic and extrinsic parameters of each network camera relative to the reference coordinate system using the Zhang's checkerboard calibration method.

[0019] Further, in step S2, the creation of the martial arts human two-dimensional posture dataset includes:

[0020] Annotation tools were used to annotate RGB images of martial arts human poses, including those with self-occlusion. In cases of occlusion, it was necessary to ensure that the annotation points fell on the actual positions of the human body.

[0021] Furthermore, in step S3, the network camera that performs synchronous shooting supports the PTP protocol. The synchronous shooting captures Bayer format image data, which is transmitted to the PCIe network card via network cable and then converted into RGB three-channel format.

[0022] Furthermore, step S3 includes the following steps:

[0023] S3.1 Construct the LRANet-SimCC model and pre-train it on the COCO2017 dataset. Then train the pre-trained model on the martial arts dataset to obtain the trained LRANet-SimCC model, which is more suitable for calculating the two-dimensional information of the three-dimensional joints in martial arts.

[0024] S3.2 Deploy the trained LRANet-SimCC model to the GPU using TensorRT, converting the model into batch inference form to improve the inference speed of the model;

[0025] S3.3. Use the RTMDet target detection model to detect human target boxes from RGB images, and use the detected result images as input to the LRANet-SimCC model;

[0026] S3.4. Using the LRANet-SimCC model, the two-dimensional coordinates and confidence weights of human joints from each viewpoint are obtained through image inference based on the target detection results.

[0027] Further, in step S3.1, the LRANet-SimCC model is constructed, as follows:

[0028] The LRANet-SimCC model consists of two parts: the backbone network LRANet and the SimCC classification head. The backbone network LRANet first obtains the feature maps X of the joints of the input image through three ConvModules. Then, it enhances the ability to extract target feature information through multiple LRA layers composed of ConvModules, depthwise separable convolutions, and large kernel residual attention. Next, the feature output layer composed of ConvModules, LRAModules, and SimSPPF extracts the final features. Finally, a feature encoding Neck layer encodes the feature information and feeds it into the SimCC detection head to obtain the final output result.

[0029] The ConvModule consists of a 3×3 convolution, a BN layer, and a SiLU activation function. The SiLU activation function is calculated as follows:

[0030]

[0031] Where x is the feature map output after the image passes through a 3×3 convolution and a BN layer, and silu(x) is the output of the SiLU activation function;

[0032] The input image size of the backbone network LRANet is H×W. The Neck layer encodes the features extracted by the backbone network LRANet into a one-dimensional vector. Finally, the linear projection operation in the SimCC detection head encodes each one-dimensional vector into two independent one-dimensional vectors representing (n,σH) and (n,σW), where σ is the scaling factor in SimCC and n is the number of joints to be detected.

[0033] Furthermore, in the LRA layer composed of the large kernel residual attention, the feature maps X output by the three ConvModules are passed through a 7×7 large convolutional kernel to obtain a feature map U with a larger receptive field. While preserving contextual information, the feature map U strengthens the weight information of adjacent key points. Then, U is subjected to average pooling and max pooling operations to obtain U0. avg and U max This will preserve both overall and edge information separately;

[0034] The two features are aggregated and then processed through a 7×7 convolution followed by sigmoid activation to obtain the result. This reduces the weight of incorrect points and increases the weight of correct points. The calculation formula is shown below:

[0035]

[0036] in, It's U avg and U maxThe feature map obtained after 7×7 convolution, where σ represents the sigmoid function. The feature maps are stacked, with 2 stacked channels and a value range of (0,1).

[0037] Finally, the different weights are multiplied by feature map X and feature map U respectively, and then weighted to obtain the final output result M. The calculation formula is as follows:

[0038]

[0039] in, express The first passage, express The second passage, This indicates element-wise multiplication of matrices. This indicates that corresponding elements are added together.

[0040] Furthermore, the Neck layer aggregates all features through a 7×7 convolution, compresses them into a one-dimensional vector V through an MLP layer, and finally connects to a GAU self-attention module to refine the keypoint representation to obtain the output result O. The calculation formula of the GAU self-attention module is shown in the following formula:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] Where s is the scaling factor, Q(V) and K(Z) represent the linear transformation of V and Z respectively, and rule 2 (*) represents the square of the ReLU activation function. is the activation function, W is the trainable weight matrix, and ⊙ represents element-wise dot product;

[0047] The final SimCC classifier introduces two linear classifiers with different dimensions to classify two independent one-dimensional vector representations (n,σH) and (n,σW) respectively. The maximum value of each class is used as the prediction result pre_x and pre_y. The weight of the larger of the two is taken as the confidence weight information θ of the joint. The [pre_x,pre_y] is combined and then divided by the SimCC scaling factor σ to obtain the final joint detection result.

[0048] Further, in step S4, the extracted two-dimensional coordinates and confidence weight information are triangulated to obtain the three-dimensional joint coordinate information of the human body, the mathematical representation of which is as follows:

[0049]

[0050] Where, p n Let P represent the two-dimensional coordinates [u,v] of the joint in the nth view, and let M represent the three-dimensional coordinates obtained from the multi-view solution. n [I 0] is a 3*3 matrix, representing the intrinsic parameter matrix of the nth view; [I 0] is a 3*4 matrix, representing the extrinsic parameter matrix of the reference coordinate system; and [RT] represents the extrinsic parameter matrix of the nth view relative to the reference coordinate system.

[0051] Combining the confidence weight θ with the above formula, we can obtain the following overdetermined homogeneous linear equation system:

[0052]

[0053] Where, m 1 ,m 2 ,m 3 It is M n The first, second, and third rows of the matrix, θ n This represents the confidence weight of the same 3D point in the nth view.

[0054] The least squares solution P = [x, y, z] of the equation is obtained by using singular value decomposition, which is the final result of the three-dimensional human joint points.

[0055] A real-time martial arts 3D human pose estimation system based on multiple perspectives, used to implement a real-time martial arts 3D human pose estimation method based on multiple perspectives, including:

[0056] Hardware building modules are used to construct multi-view real-time shooting environments;

[0057] The image acquisition module is used to acquire martial arts image data;

[0058] The two-dimensional pose recognition module is used to extract human joint points and confidence weight information from human image data using the LRANet-SimCC algorithm;

[0059] The 3D pose recognition module is used to solve for the 3D human joint information by solving for the human joints and confidence weight information;

[0060] The data visualization module is used to visualize the solved 3D human joint information and 3D human skeleton on the desktop.

[0061] Compared with the prior art, the present invention can achieve at least the following beneficial effects:

[0062] By using a 2D-to-3D upscaling approach, the problem of insufficient and difficult-to-create 3D datasets is avoided. Multi-view information fusion improves the uncertainty and occlusion issues in the 2D-to-3D conversion. A custom martial arts dataset makes the model more suitable for complex pose changes and more compatible with 3D computation. The proposed LRANet-SimCC model achieves real-time detection speed while maintaining high accuracy. The entire system solution features low computational cost, strong scalability, real-time operation, low cost, and easy deployment. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the overall process of a real-time martial arts 3D human posture estimation method based on multiple perspectives, according to an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the LRANet-SimCC model according to an embodiment of the present invention.

[0065] Figure 3 This is a schematic diagram of the large kernel attention module in an embodiment of the present invention.

[0066] Figure 4 This is a schematic diagram of the structure of the real-time detection system according to an embodiment of the present invention.

[0067] Figure 5 This is a schematic diagram of the detection operation process of the real-time detection system according to an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example:

[0070] A real-time martial arts 3D human pose estimation method based on multiple perspectives, such as... Figure 1 As shown, it includes the following steps:

[0071] S1. Set up multiple network cameras around the indoor venue scene, and calibrate the internal and external parameters of each camera using the camera calibration method;

[0072] In one embodiment of the present invention, a Hikvision industrial camera with PTP time frame synchronization is used, which is connected to a computer via a 4-port PCIe network card; a checkerboard pattern is printed, and the camera's intrinsic and extrinsic parameters are calculated by using Eazymocap's calibration tool to calibrate the multi-view camera system.

[0073] S2. Create a 2D dataset of martial arts human postures;

[0074] The creation of the martial arts human two-dimensional posture dataset includes:

[0075] Annotation tools were used to annotate RGB images of martial arts human poses, including those with self-occlusion. In cases of occlusion, it was necessary to ensure that the annotation points fell on the actual positions of the human body.

[0076] In one embodiment of the present invention, the extraction is specifically performed by capturing martial arts videos with a calibrated camera. Using the Labelme annotation tool, the images are labeled with 17 human body joints in COCO format, where the meaning of each joint is as follows:

[0077] Table 1. Correspondence between COCO-17 joint serial numbers

[0078] Serial Number serial number Location Serial Number serial number Location 0 Nose nose 9 LWrist left wrist 1 LEye Left eye 10 RWrist right wrist 2 Reye right eye 11 LHip Left buttock 3 Lear left ear 12 RHip Right buttock 4 Rear right ear 13 LKnee Left knee 5 LShoulder left shoulder 14 RKnee Right knee 6 RShoulder right shoulder 15 LAnke left ankle 7 LElbow left elbow 16 RAnke right ankle 8 RElbow right elbow

[0079] The self-occluding and mutual-occluding joints need to be labeled on the actual human body position to improve the shortcomings of the COCO dataset and to meet the requirements of multi-view 3D measurement.

[0080] S3. Multiple calibrated network cameras are used to simultaneously capture human martial arts movements to obtain image data. The image data is preprocessed and encoded into the required RGB image. Based on this, the RTMDet target detection model is used to detect human target boxes from the RGB image. The trained LRANet-SimCC model is used to infer human movements in the target boxes in batches, and the two-dimensional coordinates and confidence weight information of human joints under each view are extracted.

[0081] The network camera used for synchronous shooting supports the PTP protocol. The synchronous shooting captures Bayer format image data, which is then transmitted to the PCIe network card via network cable and converted into RGB three-channel format.

[0082] Step S3 includes the following steps:

[0083] S3.1 Build the LRANet-SimCC model and pre-train it on the COCO2017 dataset. Then train the pre-trained model on the martial arts dataset to obtain the trained LRANet-SimCC model.

[0084] Constructing the LRANet-SimCC model, such as Figure 2 As shown, the details are as follows:

[0085] The LRANet-SimCC model consists of two parts: the backbone network LRANet and the SimCC classification head. The backbone network LRANet first obtains the feature maps X of the joints of the input image through three ConvModules. Then, it enhances the ability to extract target feature information through multiple LRA layers composed of ConvModules, depthwise separable convolutions, and large kernel residual attention. Next, the feature output layer composed of ConvModules, LRAModules, and SimSPPF extracts the final features. Finally, a feature encoding Neck layer encodes the feature information and feeds it into the SimCC detection head to obtain the final output result.

[0086] The ConvModule consists of a 3×3 convolution, a BN layer, and a SiLU activation function. The SiLU activation function is calculated as follows:

[0087]

[0088] Where x is the feature map output after the image passes through a 3×3 convolution and a BN layer, and silu(x) is the output of the SiLU activation function;

[0089] The input image size of the backbone network LRANet is H×W. The Neck layer encodes the features extracted by the backbone network LRANet into a one-dimensional vector. Finally, the linear projection operation in the SimCC detection head encodes each one-dimensional vector into two independent one-dimensional vectors representing (n,σH) and (n,σW), where σ is the scaling factor in SimCC, which is set to 2 in this scheme, and n is the number of joints to be detected, which is set to 17 in this scheme.

[0090] like Figure 3 As shown, in the LRA layer composed of the large kernel residual attention, the feature maps X output by the three ConvModules are passed through a 7×7 large convolutional kernel to obtain a feature map U with a larger receptive field. The feature map U strengthens the weight information of adjacent key points while preserving context information. Then, U is subjected to average pooling and max pooling operations to obtain U0. avg and U max This will preserve both overall and edge information separately;

[0091] The two features are aggregated and then processed through a 7×7 convolution followed by sigmoid activation to obtain the result. This reduces the weight of incorrect points and increases the weight of correct points. The calculation formula is shown below:

[0092]

[0093] in, It's U avg and U max The feature map obtained after 7×7 convolution, where σ represents the sigmoid function. The feature maps are stacked, with 2 stacked channels and a value range of (0,1).

[0094] Finally, the different weights are multiplied by feature map X and feature map U respectively, and then weighted to obtain the final output result M. The calculation formula is as follows:

[0095]

[0096] in, express The first passage, express The second passage, This indicates element-wise multiplication of matrices. This indicates that corresponding elements are added together.

[0097] Furthermore, the Neck layer aggregates all features through a 7×7 convolution, compresses them into a one-dimensional vector V through an MLP layer, and finally connects to a GAU self-attention module to refine the keypoint representation to obtain the output result O. The calculation formula of the GAU self-attention module is shown in the following formula:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] Where s is the scaling factor, which is set to 128 in this scheme; Q(V) and K(Z) represent the linear transformation of V and Z, respectively; and rule 2 (*) represents the square of the ReLU activation function. is the activation function, W is the trainable weight matrix, and ⊙ represents element-wise dot product;

[0104] The final SimCC classifier introduces two linear classifiers with different dimensions to classify the two independent one-dimensional vector representations (n,σH) and (n,σW) respectively. The maximum value of each class is used as the prediction result pre_x and pre_y. The weight of the larger of the two is taken as the confidence weight information θ of the joint. The [pre_x,pre_y] is combined and then divided by the SimCC scaling factor σ = 2 to obtain the final joint detection result.

[0105] S3.2 In one embodiment, the trained LRANet-SimCC model is simplified by int8 quantization using TensorRT and then deployed to the GPU. The model is converted into a batch inference form to improve the inference speed of the model.

[0106] In one embodiment, image frames captured by an industrial camera are preprocessed by transcoding, compressing, and packaging the image data into an n*640x480 RGB image, which is then fed into the inference pipeline for real-time inference. Post-processing uses Kalman filtering for smoothing, where n is the number of viewpoints (n>=2).

[0107] S3.3. Use the RTMDet target detection model to detect human target boxes from RGB images, and use the detected result images as input to the LRANet-SimCC model;

[0108] S3.4. Using the LRANet-SimCC model, the two-dimensional coordinates and confidence weights of human joints from each viewpoint are obtained through image inference based on the target detection results.

[0109] S4. Triangulate the extracted two-dimensional coordinates and confidence weight information to obtain the three-dimensional joint coordinate information of the human body;

[0110] The extracted two-dimensional coordinates and confidence weight information are triangulated to obtain the three-dimensional joint coordinate information of the human body, and its mathematical representation is as follows:

[0111]

[0112] Where, p n Let P represent the two-dimensional coordinates [u,v] of the joint in the nth view, and let M represent the three-dimensional coordinates obtained from the multi-view solution. n [I 0] is a 3*3 matrix, representing the intrinsic parameter matrix of the nth view; [I 0] is a 3*4 matrix, representing the extrinsic parameter matrix of the reference coordinate system; and [RT] represents the extrinsic parameter matrix of the nth view relative to the reference coordinate system.

[0113] Combining the confidence weight θ with the above formula, we can obtain the following overdetermined homogeneous linear equation system:

[0114]

[0115] Where, m 1 ,m 2 ,m 3 It is M n The first, second, and third rows of the matrix, θ n This represents the confidence weight of the same 3D point in the nth view.

[0116] The least squares solution P = [x, y, z] of the equation is obtained by using singular value decomposition, which is the final result of the three-dimensional human joint points.

[0117] This invention provides a multi-view real-time martial arts 3D human joint point detection system, such as... Figure 4 and Figure 5 As shown, it includes:

[0118] The hardware setup module is used for selecting cameras and lenses, as well as setting up the shooting environment;

[0119] The image acquisition module is used to acquire and annotate martial arts human image data and perform real-time detection.

[0120] The 2D pose detection module is used to extract 2D joint data of the human body from human image data from multiple perspectives using the lightweight LRANet-SimCC algorithm, and to smooth the 2D joint data of the human body to obtain 2D results.

[0121] The 3D pose detection module is used to triangulate and solve 2D results from multiple perspectives to obtain 3D results.

[0122] The data visualization module is used to visualize 3D results by combining them with a human skeleton model. The visualization uses TCP communication, which can send local 3D point data to a remote machine for visualization.

[0123] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A real-time martial arts 3D human pose estimation method based on multiple perspectives, characterized in that, Includes the following steps: S1. Set up multiple network port cameras and calibrate the internal and external parameters of each camera using camera calibration methods; S2. Create a 2D dataset of martial arts human postures; S3. Multiple calibrated network cameras are used to simultaneously capture human martial arts movements to obtain image data. The image data is preprocessed and encoded into the required RGB images. Based on this, the RTMDet target detection model is used to detect human target boxes from the RGB images. The trained LRANet-SimCC model is used to infer human movements in the target boxes in batches, and the two-dimensional coordinates and confidence weights of human joints from each viewpoint are extracted. The LRANet-SimCC model consists of two parts: the backbone network LRANet and the SimCC classification head. The backbone network LRANet first obtains the feature map X of the joint points through three ConvModules from the input image. Then, it enhances the ability to extract target feature information through multiple LRA layers composed of ConvModules, depthwise separable convolutions, and large kernel residual attention. Next, the final features are extracted through the feature output layer composed of ConvModules, LRAModules, and SimSPPF. Then, a feature encoding Neck layer encodes the feature information and feeds it into the SimCC detection head to obtain the final output result. The ConvModule consists of a 3×3 convolution, a BN layer, and a SiLU activation function. The SiLU activation function is calculated as follows: Where x is the feature map output after the image passes through a 3×3 convolution and a BN layer, and silu(x) is the output of the SiLU activation function; The input image size of the backbone network LRANet is H×W. The Neck layer will encode the features extracted by the backbone network LRANet into a one-dimensional vector. Finally, the linear projection operation in the SimCC detection head will encode each one-dimensional vector into two independent one-dimensional vectors representing (n,σH) and (n,σW), where σ is the scaling factor in SimCC and n is the number of joints to be detected. S4. Triangulate the extracted two-dimensional coordinates and confidence weight information to obtain the three-dimensional joint coordinate information of the human body.

2. The method for real-time martial arts 3D human posture estimation based on multiple perspectives according to claim 1, characterized in that, In step S1, the intrinsic and extrinsic parameters of each camera are calibrated using a camera calibration method, as follows: Select one of the network cameras as the reference coordinate system, and calibrate the intrinsic and extrinsic parameters of each network camera relative to the reference coordinate system using the Zhang's checkerboard calibration method.

3. The method for real-time martial arts 3D human posture estimation based on multiple perspectives according to claim 1, characterized in that, In step S2, the creation of the martial arts human two-dimensional posture dataset includes: Annotation tools were used to annotate RGB images of martial arts human poses, including those with self-occlusion. In cases of occlusion, it was necessary to ensure that the annotation points fell on the actual positions of the human body.

4. The method for real-time martial arts 3D human posture estimation based on multiple perspectives according to claim 1, characterized in that, In step S3, the network camera that performs synchronous shooting supports the PTP protocol. The synchronous shooting captures Bayer format image data, which is transmitted to the PCIe network card via network cable and then converted into RGB three-channel format.

5. The method for real-time martial arts 3D human posture estimation based on multiple perspectives according to claim 1, characterized in that, Step S3 includes the following steps: S3.1 Build the LRANet-SimCC model and pre-train it on the COCO2017 dataset. Then train the pre-trained model on the martial arts dataset to obtain the trained LRANet-SimCC model. S3.2 Deploy the trained LRANet-SimCC model to the GPU using TensorRT, and convert the model into batch inference format; S3.

3. Use the RTMDet target detection model to detect human target boxes from RGB images, and use the detected result images as input to the LRANet-SimCC model; S3.

4. Using the LRANet-SimCC model, the two-dimensional coordinates and confidence weights of human joints from each viewpoint are obtained through image inference based on the target detection results.

6. The method for real-time martial arts 3D human posture estimation based on multiple perspectives according to claim 1, characterized in that, In the LRA layer composed of the large kernel residual attention, the feature maps X output by the three ConvModules are passed through a 7×7 large convolutional kernel to obtain a feature map U with a larger receptive field. Then, U is subjected to average pooling and max pooling operations to obtain U0. avg and U max ; The two features are aggregated and then processed through a 7×7 convolution followed by sigmoid activation to obtain the result. The calculation formula is shown below: in, It's U avg and U max The feature map obtained after 7×7 convolution, where σ represents the sigmoid function. The feature maps are stacked, with 2 stacked channels and a value range of (0,1). Finally, the different weights are multiplied by feature map X and feature map U respectively, and then weighted to obtain the final output result M. The calculation formula is as follows: in, express The first passage, express The second passage, This indicates element-wise multiplication of matrices. This indicates that corresponding elements are added together.

7. A real-time martial arts 3D human posture estimation system based on multiple perspectives, characterized in that, A method for implementing a real-time martial arts 3D human posture estimation method based on multiple perspectives as described in any one of claims 1 to 6, comprising: Hardware building modules are used to construct multi-view real-time shooting environments; The image acquisition module is used to acquire martial arts image data; The two-dimensional pose recognition module is used to extract human joint points and confidence weight information from human image data using the LRANet-SimCC algorithm; The 3D pose recognition module is used to solve for the 3D human joint information by solving for the human joints and confidence weight information; The data visualization module is used to visualize the solved 3D human joint information and 3D human skeleton on the desktop.

Images