Multi-person multi-view human body posture prediction method

Abstract

The invention discloses a multi-person multi-view human body posture prediction method, and belongs to the technical field of human body posture prediction.The multi-person multi-view human body posture prediction method comprises the steps that a target image set and a 2D heat map of each key point are input into a pre-trained Pose-NeRF prediction network, and voxel feature weights of the key points are obtained; 3D coordinates of all key points are determined according to voxel feature weights of the key points and projection relations of different camera visual angles of a target image set, a Pose-NeRF prediction network can fully acquire spatial information, the position of a central point of each voxel and the opacity of spatial features are used as weights, the prediction accuracy is greatly improved, and the prediction efficiency is improved. The problems that a current prediction method cannot bring the advantage of multi-view consistency to attitude estimation and the prediction result is inaccurate are solved.

Description

Technical Field

[0001] The invention relates to a method for predicting human postures of multiple people and multiple eye views, and belongs to the technical field of human posture prediction. Background Art

[0002] Multi-person Multi-view 3D Human Pose Estimation is a topic that estimates the 3D human poses of multiple people from multi-view RGB images or videos. It is a basic problem in computer vision. It can help a series of downstream tasks such as human-computer interaction and action recognition, and can also realize applications such as sports events and virtual animation.

[0003] One aspect of this is multi-person and multi-view 3D human pose estimation from RGB images. In multi-person scenes, multi-view views complement the information missing from the monocular view due to occlusion, alleviating the occlusion and blur problems. Because 2D human pose estimation has been developed and mature, most research results are based on 2D human poses, focusing on the process of upgrading from 2D poses to 3D poses. Among them, the volume representation-based method projects the 2D features or heat maps in each view into the 3D voxel space, and then fuses the multi-view features. The resulting voxel features are more robust to the occlusion of a single camera. However, the voxel features obtained from the fusion of each view alone are not enough to fully utilize the semantic information. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a multi-person and multi-view human body posture prediction method that can fully obtain spatial information and use the opacity of the center point position of each voxel and the spatial feature as a weight, thereby greatly improving the prediction accuracy.

[0005] In order to achieve the above purpose / solve the above technical problems, the present invention is implemented by adopting the following technical solutions: A multi-person multi-view human body posture prediction method, comprising: Get a 2D heat map of each key point on the target atlas, where the target atlas includes multi-camera perspective images of the same scene, and the key points include other key points and root key points; Input the target atlas and the 2D heat map of each key point into the pre-trained Pose-NeRF prediction network to obtain the voxel feature weights of the key points; The pre-trained Pose-NeRF prediction network includes the NeRF branch network and the Pose branch network; Predict the 3D coordinates of all key points based on the voxel feature weights of the key points and the projection relationship of different camera perspectives of the target atlas; According to the 3D coordinates of all key points, the human body posture in the target atlas is obtained.

[0006] Further, the NeRF branch network includes two MLP layers connected in sequence, wherein one of the MLP layers is a G-MLP layer, and the other MLP layer is a C-MLP layer; The Pose branch network includes a CPN network and a PRN network, and the G-MLP layer is connected to the CPN network and the PRN network respectively.

[0007] Furthermore, the training of the pre-trained Pose-NeRF prediction network includes: Establishing a training image set, wherein the training image set includes images of multiple camera viewpoints of the same scene; Randomly select one image from the training image set as the supervision view and the remaining images as the training views; The training image set is input into the target Pose-NeRF prediction network to supervise the color of the view sampling points, so that the target Pose-NeRF prediction network can accurately predict the density value of any position in the 3D space formed by the supervision view. Input the key point heat map corresponding to the pre-acquired training image set into the target Pose-NeRF prediction network to obtain the voxel features of the root key point; Supervised training is performed based on the distance from each voxel point to the root key point Ground-Truth until the target Pose-NeRF prediction network can generate the corresponding position view that meets the requirements; Get the trained Pose-NeRF prediction network.

[0008] Furthermore, the training image set is input into the target Pose-NeRF prediction network to supervise the color supervision training of the view sampling points, so that the target Pose-NeRF prediction network can accurately predict the density value of any position in the 3D space formed by the supervision view, including: Get the variance and mean of RGB features at the same point in 3D space from different views in the training image set; According to the variance and average of RGB features of different views at the same point in 3D space in the training image set, the enhanced features between views are obtained; Perform ray sampling on the supervision view and determine the 3D position of the sampling point on each ray; The density of the sampling points is calculated based on the 3D positions of the sampling points and the enhanced features between the views; Predict the color value of the sampling point based on the density of the sampling point; The correct density value of the sampling point is determined by predicting the color value of the sampling point and the loss function of Ground-Truth.

[0009] Furthermore, the method of obtaining enhanced features between views according to the variance and average of RGB features of different views at the same point in 3D space in the training image set includes: ; in: Represents sampling points of different views in 3D space p Enhanced features of Represents the sampling points in 3D space of different views in the training image p The average value of the RGB features, ; Represents the sampling points in 3D space of different views in the training image p The variance of the RGB features, ; represents the number of valid 2D projections, Indicates i RGB enhancement features of each view.

[0010] Furthermore, the density of the sampling points is calculated based on the 3D positions of the sampling points and the enhanced features between the views, and the color values ​​of the sampling points are predicted according to the density of the sampling points, including: The density of the sampling points is calculated using the following formula: ;in: Indicates sampling point p The density of Represents sampling points of different views in 3D space p The enhanced features of Indicates sampling point p The position code, represents the estimated geometry; Based on the density of the sampling points, the potential features of the sampling points are calculated, and the potential features of the sampling points are calculated using the following formula: ; in: Indicates sampling point p potential characteristics of Based on the potential features of the sampling points, calculate the color value of the sampling points: ; in: Indicates sampling point p The color value of Indicates the sampling pointp The observation direction, represents the encoded viewing direction, Indicates estimated color.

[0011] Furthermore, the target atlas and the 2D heat map of each key point are input into the pre-trained Pose-NeRF prediction network to obtain the voxel feature weights of the key points, including: Divide the 3D space of the target atlas into voxel grids and obtain the voxel features of key points; Obtain the opacity value of the voxel feature in the target atlas view through the NeRF branch network; Use the opacity value as the voxel feature weight.

[0012] Furthermore, obtaining the opacity value of the voxel feature in the target atlas view through the NeRF branch network includes: Get the density value of key points in the target atlas view; The density value of the key point is converted into the opacity value corresponding to the voxel feature of the key point.

[0013] Furthermore, the 3D coordinates of all key points are predicted based on the voxel feature weights of the key points and the projection relationship of different camera perspectives of the target atlas, including: Divide the 3D space of the target atlas into a first voxel grid and obtain the first voxel feature of each root key point; The opacity value of each first voxel feature is used as the first voxel feature weight; Generate a 3D heat map of the root keypoints based on the first voxel feature weight of each root keypoint; Based on the 3D heat map of the root key point, the possible center position and confidence of the human body in the target atlas are obtained through non-maximum suppression; The correct human body center position is determined according to the confidence level, and the 3D space in the surrounding fixed area is re-divided into a second voxel grid with the correct human body center position as the center, so as to obtain the second voxel features of all key points; The opacity values ​​of the second voxel features of all key points are used as the second voxel feature weights; Generate a 3D heat map of each key point based on the second voxel feature weight of each key point; Based on the 3D heatmap of each keypoint, the 3D coordinates of all keypoints are predicted.

[0014] Furthermore, taking the opacity value of each first voxel feature as the first voxel feature weight includes: The density value corresponding to each root key point is calculated by the following formula: ; Indicates sampling point p Density value, sampling point p It is the root key point; Represents enhanced features, Indicates positional encoding; represents the estimated geometry; The density value corresponding to each root key point is converted into an opacity value by the following formula:

[0015] Indicates sampling point p The opacity value of the sampling point p It is the root key point; Represented as real number e An exponential function with base ; The obtained opacity value is used as the first voxel feature weight.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention inputs the target atlas and the 2D heat map of each key point into the pre-trained Pose-NeRF prediction network to obtain the voxel feature weights of the key points; according to the voxel feature weights of the key points and the projection relationship of different camera angles of the target atlas, the 3D coordinates of all key points are determined. The Pose-NeRF prediction network can fully obtain spatial information, and use the center point position of each voxel and the opacity of the spatial feature as weights, which greatly improves the prediction accuracy and solves the problem that the current prediction method cannot bring multi-view to posture estimation. Figure 1 The advantage of consistency and the problem of inaccurate prediction results.

[0017] 2. The present invention uses the opacity value of the first voxel feature of each root key point as the first voxel feature weight, generates a 3D heat map of the root key point, and obtains the possible human body center position and confidence in the target atlas through non-maximum suppression; determines the correct human body center position according to the confidence, and re-divides the 3D space in the surrounding fixed area into a voxel grid with the correct human body center position as the center to obtain the second voxel features of all key points; uses the opacity value of the second voxel feature of all key points as the second voxel feature weight and generates a 3D heat map of each key point, and finally determines the 3D coordinates of all key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a multi-person multi-view human body posture prediction method provided by an embodiment of the present invention; Figure 2It is a Pose-NeRF prediction network training flow chart in the multi-person multi-view human posture prediction method provided by an embodiment of the present invention; Figure 3 Schematic diagram of the structure of the Pose branch network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0020] like Figure 1 FIG. 1 is a flowchart of a method for predicting human posture in multiple multi-view images, including: Get the 2D heat map of each key point on the target atlas, where the target atlas includes multi-camera view images of the same scene; the data format of the multi-camera view images is , Indicates the number of channels of RGB images, 3 indicates the number of channels of RGB images, Indicates the height of the image. Indicates the width of the image. The key points include other key points and root key points. Input the target atlas and the 2D heat map of each key point into the pre-trained Pose-NeRF prediction network to obtain the voxel feature weights of the key points; The pre-trained Pose-NeRF prediction network includes a NeRF branch network and a Pose branch network, and the NeRF branch network includes two MLP layers connected in sequence, wherein one MLP layer is a G-MLP layer and the other MLP layer is a C-MLP layer; like Figure 3 As shown, the Pose branch network includes a CPN network and a PRN network. The G-MLP layer is connected to the CPN network and the PRN network respectively. The structure of the CPN network includes: ;

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] H represents the 3D heat map features generated from the projection of each view, Represents the final output 3D heat map, Represents the convolution kernel as 3D convolution, The convolution kernel is 3D convolution, The convolution kernel is 3D convolution, The convolution kernel is 3D convolution, Indicates two The concatenated residual blocks, represents pooling, represents deconvolution; The structure of the PRN network is the same as that of the CPN network. The CPN network is used to regress the center position of the character in the scene, and the PRN network is used to regress all key points.

[0027] The 3D space of the target atlas is divided into a first voxel grid, where the size of the first voxel grid is , Respectively represent the length, width, and height of the 3D space; Get the first voxel feature of each root key point, including:

[0028] Indicates sampling point p The first voxel feature, sampling point p For the root key point, represents the number of valid 2D projections, Indicates i RGB enhancement features of each view; Connect the mean and variance of the root key points in each view as enhanced features; According to the enhanced features, the first voxel feature weight is determined, including: The density value corresponding to each 3D point is calculated by the following formula: ; Indicates sampling point p The density value of Represents enhanced features, Indicates positional encoding; The density value corresponding to each 3D point is converted into an opacity value using the following formula: Indicates sampling point p The opacity value of Represented as real number e An exponential function with base ; The obtained opacity value is used as the first voxel feature weight; According to the first voxel feature weight of each root key point, a 3D heat map of the root key point is generated. Specifically: The first voxel feature weighted by the first voxel feature weight is calculated by the following formula: represents the weighted first voxel feature; Based on the weighted first voxel features, a 3D heat map of the root keypoint is predicted.

[0029] Generate a 3D heat map of the root key points based on the prediction, and obtain the possible human center position and confidence in the target atlas through non-maximum suppression; The correct human body center position is determined according to the confidence level. The 3D space in the surrounding fixed area is re-divided into a second voxel grid with the correct human body center position as the center, and the second voxel features of all key points are obtained, where the size of the second voxel grid is , The opacity values ​​of the second voxel features of all key points are used as the second voxel feature weights; Generate a 3D heat map of each key point based on the second voxel feature weight of each key point; Based on the 3D heat map of each key point, determine the 3D coordinates of all key points; According to the 3D coordinates of all key points, the human body posture in the target atlas is obtained.

[0030] like Figure 2 As shown in the figure, the training of the pre-trained Pose-NeRF prediction network is as follows: Establishing a training image set, wherein the training image set includes images of multiple camera viewpoints of the same scene; Randomly select one image from the training image set as the supervision view and the remaining images as the training views; The training image set is input into the target Pose-NeRF prediction network to supervise the color supervision training of the view sampling points, so that the target Pose-NeRF prediction network can accurately predict the density value of any position in the 3D space formed by the supervision view, including: The variance and mean of the RGB features of the same point in 3D space in different views of the training image are calculated by the following formula:

[0031]

[0032] Represents the sampling points in 3D space of different views in the training imagep The average value of the RGB features, Represents the sampling points in 3D space of different views in the training image p The variance of the RGB features, represents the number of valid 2D projections, Indicates i RGB enhancement features of each view; According to the variance and average of RGB features of different views at the same point in 3D space in the training set, the enhanced features between views are obtained, including: ; Represents sampling points of different views in 3D space p Enhanced features.

[0033] Perform ray sampling on the supervisory view and determine the 3D position of the sampling point on each ray, including: Determine the Ground-Truth value of the supervisory view light, as well as the origin and direction of the light, and randomly select some of the light as sampling light; The 3D position of the sampling point is uniformly obtained according to the range of the sampling light r and the number of sampling points:

[0034] represents the first i sampling points, represents the number of sampling points, Indicates the starting point of the sampling range, Indicates the end point of the sampling range.

[0035] The density of the sampling points is calculated based on the 3D positions of the sampling points and the enhanced features between the views, and the color values ​​of the sampling points are predicted according to the density of the sampling points, including: The density of the sampling points is calculated by the following formula:

[0036] Indicates the position code of the sampling point, represents the potential features of the sampling points, Indicates the use of G-MLP layer to estimate the geometric shape; The color value of the sampling point is calculated by the following formula:

[0037] Indicates the viewing direction, represents the encoded viewing direction, Indicates that the color is estimated using the C-MLP layer, Indicates the color value; In order to enhance the performance of NeRF renderings in representing high-frequency changes in color and geometry, the position encoding calculation formula for spatial position and viewing direction is as follows:

[0038] t Represents the coordinate value and observation direction of the sampling point, Acting on the three coordinate values ​​of the sampling point and the two viewing directions, L Represents a hyperparameter. In this example, for the sampling point p 3D Position Settings L =10 is encoded as , for the observation direction d set up L =4 is encoded as ; Volume rendering is used to generate the final pixel color and depth, including:

[0039]

[0040] Represents the pixel color of the sampling point, Indicates the depth of the sampling point; Represents the distance between the sampling point and the camera, Indicates i The color predicted by the sampling point; Indicates the time from the first sampling point to the i The cumulative transmittance of the sampling points is , Indicates j The density predicted by the sampling points, ; Indicates i The opacity of the sampling points, .

[0041] By predicting the color value of the sampling point and the loss function of Ground-Truth, the correct density value of the sampling point is determined, including: Since RGB images are used as supervision, only the color values ​​of the sampling points need to be used as supervision, and the loss function is: Represents the color value predicted by the NeRF branch network and the loss function of Ground-Truth, represents the number of randomly selected sampling rays for the supervision view, Indicates i The Ground-Truth value of the color of the sampling ray. Indicates i The predicted value of the sampled ray; The NeRF branch network is trained using the color value of the sampling point as supervision, so that it can learn the density value at any position from the 3D voxel features; Input the key point heat map corresponding to the pre-acquired training image set into the target Pose-NeRF prediction network to obtain the voxel features of the root key point; Based on the distance from each voxel point to the root key point Ground-Truth, supervised training is performed until the target Pose-NeRF prediction network can generate the corresponding position view that meets the requirements. Specifically: Calculate a 3D heat map based on the distance from each voxel point to the character's root key point Ground-Truth; For multi-person scenes, a voxel point has multiple scores, and the maximum score of the voxel point is taken as the actual score; The predicted 3D heat map of the corresponding person is obtained through the CPN network prediction of the target Pose-NeRF prediction network; Based on the 3D heat map, the predicted 3D heat map corresponding to the 3D heat map, and the voxel point score, the CPN loss is used for training until the target Pose-NeRF prediction network can generate the corresponding position view that meets the requirements. The loss function of the predicted character root key point heat map and the heat map generated by Ground-Truth is:

[0042] Represents the loss function of the predicted character root key point heat map and the heat map generated by Ground-Truth, Represents the 3D heat map corresponding to the predicted position generated by the distance from the voxel point to the root key point Ground-Truth. Represents a 3D heat map generated based on the distance from the voxel point to the root key point Ground-Truth; Get the voxel features of all key points; Based on the Ground-Truth distance from each voxel point to all key points, supervised training is performed until the target Pose-NeRF prediction network can generate 3D coordinates of the corresponding position that meet the requirements. Specifically: The PRN network of the target Pose-NeRF prediction network generates 3D coordinates based on the Ground-Truth value, and predicts the predicted 3D coordinates of the corresponding character; Based on 3D coordinates, predict 3D coordinates to train the PRN network, including:

[0043] Represents the loss function of the predicted key point position and the key point position generated based on the Ground-Truth value, represents the number of key points, Indicates k predicted 3D coordinates, Indicates k A 3D coordinate generated based on the Ground-Truth value;

[0044] Represents the prediction generated k A three-dimensional heat map, Indicates k 3D coordinates of a 3D heat map; Get the trained Pose-NeRF prediction network.

[0045] In this embodiment, the training image set is selected from the Panoptic dataset, which uses dozens of high-definition cameras to record people's daily lives, in which 17 key points are set for each person; As shown in Table 1, some sequences of images and cameras with corresponding numbers are selected as the data of the research scene and grouped into training sets and test sets; Table 1. Statistics of the sequences required by the Panoptic dataset

[0046] Use the backbone network of ResNet-50 to train 2D human pose estimation so that the image input to the network can obtain the heat map corresponding to each key point; The network model is implemented in the PyTorch framework, where the PyTorch version is 1.4.0, the CUDA version is 10.0, the training batch size is 1, and the confidence threshold of the character center position is 0.3; The pre-trained 2D pose estimation network was trained for 20 epochs with an initial learning rate of 1e to 4, which was reduced to 1e to 5 and 1e to 6 at the 10th and 15th times respectively; the entire network including CPN, PRN and NeRF was trained for 10 epochs with a learning rate of 1e to 4; the number of random rays of NeRF was 2048, and the number of points used on each ray was 64; when dividing the entire scene into voxels, the center coordinates of the entire 3D space were set to (0.0, -500.0, 800.0), the space size was 8000mm*8000mm*2000mm, and the number of divided voxels was 48*48*12; the size of the 3D bounding box centered on the character was fixed to 2000mm*2000mm*2000mm, and the number of divided voxels was 32*32*32.

[0047] As shown in Table 2, six indicators are used for performance evaluation, namely AP25, AP50, AP100, AP150, Recall, and MPJPE; Table 2. Comparison results of various methods

[0048] In the table, ↓ indicates that the smaller the value of this indicator, the better, * indicates that the network structure is deformed, and the best value of each indicator in the figure is bolded, where: AP25, AP50, AP100, and AP150 represent average precision, which refers to the thresholds of 25, 50, 100, and 150. When MPJPE is less than the threshold, it is considered to be a correct prediction, otherwise it is a wrong prediction, indicating the average precision under different thresholds; MPJPE refers to the mean per-joint position error, which represents the Euclidean distance between the predicted joint point and the Ground-Truth joint point; Recall represents the proportion of all real targets correctly detected by the model when 500 mm is the MPJPE threshold; For each method, the average of three rounds of results was taken as the final result. At the same time, in order to verify the effectiveness of the model (Pose-NeRF) proposed in this invention, the baseline model VoxelPose using volume representation and the Faster-Voxel model based on it were compared.

[0049] In addition, the model proposed in the present invention is deformed, and only the 3D voxel features of each key point are weighted, which is expressed as Pose-NeRF*.

[0050] Compared with the VoxePose and Faster-Voxel methods, the method proposed in the present invention has better performance in AP25, AP50, AP100, AP150, and MPJPE indicators, while Pose-NeRF* has better performance in the Recall indicator.

[0051] Specifically, the performance of the proposed method is better than that of the method using only voxel features. It utilizes the powerful geometric structure representation capability brought by NeRF, can better perform multi-person and multi-view 3D human pose estimation, and has a positive effect on the prediction of key points. In addition, the method proposed in the present invention is better than Pose-NeRF* in most indicators, indicating that the collective structure representation capability of NeRF plays a positive role in character positioning.

[0052] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0053] The present application is described with reference to the flowcharts of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 The steps of a specified function in a process or multiple processes.

[0056] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A method for predicting human posture in multi-person and multi-view environments, characterized in that: include: Get a 2D heat map of each key point on the target atlas, where the target atlas includes multi-camera perspective images of the same scene, and the key points include other key points and root key points; Input the target atlas and the 2D heat map of each key point into the pre-trained Pose-NeRF prediction network to obtain the voxel feature weights of the key points; The pre-trained Pose-NeRF prediction network includes the NeRF branch network and the Pose branch network; Predict the 3D coordinates of all key points based on the voxel feature weights of the key points and the projection relationship of different camera perspectives of the target atlas; According to the 3D coordinates of all key points, the human body posture in the target atlas is obtained.

2. The method for predicting human posture in multiple multi-view images according to claim 1, characterized in that: The NeRF branch network includes two MLP layers connected in sequence, wherein one of the MLP layers is a G-MLP layer and the other MLP layer is a C-MLP layer; The Pose branch network includes a CPN network and a PRN network, and the G-MLP layer is connected to the CPN network and the PRN network respectively.

3. The method for predicting human posture in multiple multi-view images according to claim 1, characterized in that: The training of the pre-trained Pose-NeRF prediction network includes: Establishing a training image set, wherein the training image set includes images of multiple camera viewpoints of the same scene; Randomly select one image from the training image set as the supervision view and the remaining images as the training views; The training image set is input into the target Pose-NeRF prediction network to supervise the color of the view sampling points, so that the target Pose-NeRF prediction network can accurately predict the density value of any position in the 3D space formed by the supervision view. Input the key point heat map corresponding to the pre-acquired training image set into the target Pose-NeRF prediction network to obtain the voxel features of the root key point; Based on the distance from each voxel point to the root key point Ground-Truth, supervised training is performed until the target Pose-NeRF prediction network can generate the corresponding position view that meets the requirements; Get the trained Pose-NeRF prediction network.

4. The method for predicting human posture in multiple multi-view images according to claim 3, characterized in that: The training image set is input into the target Pose-NeRF prediction network to supervise the color supervision training of the view sampling points, so that the target Pose-NeRF prediction network can accurately predict the density value of any position in the 3D space formed by the supervision view, including: Get the variance and mean of RGB features at the same point in 3D space from different views in the training image set; According to the variance and average of RGB features of different views at the same point in 3D space in the training image set, the enhanced features between views are obtained; Perform ray sampling on the supervision view and determine the 3D position of the sampling point on each ray; The density of the sampling points is calculated based on the 3D positions of the sampling points and the enhanced features between the views; Predict the color value of the sampling point based on the density of the sampling point; The correct density value of the sampling point is determined by predicting the color value of the sampling point and the loss function of Ground-Truth.

5. The method for predicting human posture in multiple multi-view images according to claim 4, characterized in that: The method of obtaining enhanced features between views according to the variance and average of RGB features of the same point in 3D space in different views in the training image set includes: ； in: Represents sampling points of different views in 3D space p Enhanced features of Represents the sampling points in 3D space of different views in the training image p The average value of the RGB features, ; Represents the sampling points in 3D space of different views in the training image p The variance of the RGB features, ; represents the number of valid 2D projections, Indicates i RGB enhancement features of each view.

6. The method for predicting human posture in multiple multi-view images according to claim 4, characterized in that: The step of calculating the density of the sampling points based on the 3D positions of the sampling points and the enhanced features between the views, and predicting the color values ​​of the sampling points according to the density of the sampling points, includes: The density of the sampling points is calculated using the following formula: ； in: Indicates sampling point p The density of Represents sampling points of different views in 3D space p The enhanced features of Indicates sampling point p The position code, represents the estimated geometry; Based on the density of the sampling points, the potential features of the sampling points are calculated, and the potential features of the sampling points are calculated using the following formula: ； in: Indicates sampling point p potential characteristics of Based on the potential features of the sampling points, calculate the color value of the sampling points: ； in: Indicates sampling point p The color value of Indicates the sampling point p The observation direction, represents the encoded viewing direction, Indicates estimated color.

7. The method for predicting human posture in multiple multi-view images according to claim 1, characterized in that: The target atlas and the 2D heat map of each key point are input into the pre-trained Pose-NeRF prediction network to obtain the voxel feature weights of the key points, including: Divide the 3D space of the target atlas into voxel grids and obtain the voxel features of key points; Obtain the opacity value of the voxel feature in the target atlas view through the NeRF branch network; Use the opacity value as the voxel feature weight.

8. The method for predicting human posture in multiple multi-view images according to claim 7, characterized in that: The method of obtaining the opacity value of the voxel feature in the target atlas view through the NeRF branch network includes: Get the density value of key points in the target atlas view; The density value of the key point is converted into the opacity value corresponding to the voxel feature of the key point.

9. The method for predicting human posture in multiple multi-view images according to claim 7, characterized in that: The 3D coordinates of all key points are predicted based on the voxel feature weights of the key points and the projection relationship of different camera perspectives of the target atlas, including: Divide the 3D space of the target atlas into a first voxel grid and obtain the first voxel feature of each root key point; The opacity value of each first voxel feature is used as the first voxel feature weight; Generate a 3D heat map of the root keypoints based on the first voxel feature weight of each root keypoint; Based on the 3D heat map of the root key point, the possible center position and confidence of the human body in the target atlas are obtained through non-maximum suppression; The correct human body center position is determined according to the confidence level, and the 3D space in the surrounding fixed area is re-divided into a second voxel grid with the correct human body center position as the center, so as to obtain the second voxel features of all key points; The opacity values ​​of the second voxel features of all key points are used as the second voxel feature weights; Generate a 3D heat map of each key point based on the second voxel feature weight of each key point; Based on the 3D heatmap of each keypoint, the 3D coordinates of all keypoints are predicted.

10. The method for predicting human posture in multiple multi-view images according to claim 9, characterized in that: The method of using the opacity value of each first voxel feature as the first voxel feature weight includes: The density value corresponding to each root key point is calculated by the following formula: ； Indicates sampling point p Density value, sampling point p It is the root key point; Represents enhanced features, Indicates positional encoding; represents the estimated geometry; The density value corresponding to each root key point is converted into an opacity value by the following formula: Indicates sampling point p The opacity value of the sampling point p It is the root key point; Represented as real number e An exponential function with base ; The obtained opacity value is used as the first voxel feature weight.

Images