Joint data labeling method and device based on multiple view angles, equipment and medium

Abstract

The invention relates to the field of image processing, in particular to a joint data labeling method and device based on multiple view angles, equipment and a medium. The method comprises the following steps: receiving a plurality of mark point images and a plurality of human body images, fully utilizing the synergistic effect of a motion capture camera and a reference camera, identifying the 2D coordinates of the mark points in each mark point image, and constructing a 3D human body model according to the 2D coordinates, so that accurate conversion from two dimensions to three dimensions is realized; the human body 3D skeleton data corresponding to the current moment is further generated and mapped to the corresponding human body image, so that the 2D skeleton data is obtained, and the 2D joint position of the current human body can be accurately determined by synthesizing the 2D skeleton data corresponding to the multiple mark point images. According to the step, complementarity of multi-view data is fully utilized, and accuracy and robustness of joint position marking are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device, equipment and medium for labeling joint data based on multiple perspectives. Background Art

[0002] Joint data annotation technology is widely used in motion capture, motion analysis, medical rehabilitation and other fields. By accurately annotating the positions of human joints, high-precision reconstruction and real-time monitoring of human posture can be achieved. In recent years, with the development of computer vision technology and deep learning algorithms, various joint data annotation methods have emerged, greatly promoting the progress and development of related fields.

[0003] In order to realize joint data labeling, the existing technology usually uses a single camera to capture human body images, and then uses a feature point detection algorithm to find the same key nodes in the image and calculate the spatial coordinates of these nodes to obtain the joint point positions in three-dimensional space.

[0004] However, existing joint data annotation methods have some shortcomings. During the acquisition of single-view images, due to factors such as ambient lighting changes and occlusion, some key nodes may not be clearly visible, resulting in a significant reduction in the accuracy of joint data annotation. Therefore, how to improve the accuracy of joint data annotation is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to improve the accuracy of joint data labeling, the present application provides a joint data labeling method, device, equipment and medium based on multi-perspective.

[0006] In the first aspect, the present application provides a joint data labeling method based on multi-view, which adopts the following technical solution:

[0007] A joint data labeling method based on multiple perspectives, comprising:

[0008] Receive multiple marker point images and multiple human body images corresponding to the current moment, wherein the marker point images are images of marker points on the human body acquired by a motion capture camera, and the human body images are images of the current human body acquired by a reference camera, wherein one camera corresponds to one human body image at any moment, and the human body image includes complete current human body image information, and the current human body corresponds to multiple marker points;

[0009] Identify the 2D coordinates of each marker point in each of the marker point images, and construct a 3D human body model of the current human body based on the 2D coordinates of each marker point;

[0010] Constructing human 3D skeleton data corresponding to the current moment based on the 3D human body model, and mapping the human 3D skeleton data to the corresponding human body image to obtain 2D skeleton data, wherein the human 3D skeleton data is used to characterize the three-dimensional spatial position of each marker point;

[0011] Based on the 2D skeleton data corresponding to the multiple marker point images, the 2D joint positions of the current human body are determined.

[0012] By adopting the above technical solution, multiple marker point images and multiple human body images are received, and the synergy of the motion capture camera and the reference camera is fully utilized. The motion capture camera focuses on accurately capturing the marker points on the human body, while the reference camera provides complete current human body image information, ensuring the comprehensiveness and accuracy of the data. This step lays a solid foundation for subsequent processing. By identifying the 2D coordinates of the marker points in each marker point image and constructing a 3D human body model based on this, the method realizes the accurate conversion from two-dimensional to three-dimensional. This step not only increases the dimension of the data, but also provides a reliable basis for the subsequent construction of three-dimensional skeleton data. Then, based on the constructed 3D human body model, the 3D skeleton data of the human body corresponding to the current moment is further generated and mapped to the corresponding human body image to obtain 2D skeleton data. By integrating the 2D skeleton data corresponding to multiple marker point images, the 2D joint position of the current human body can be accurately determined. This step makes full use of the complementarity of multi-view data and improves the accuracy and robustness of joint position annotation.

[0013] In a possible implementation, constructing a 3D human body model of the current human body based on the 2D coordinates of each marker point includes:

[0014] From the plurality of marker point images, determine the coincident marker points in each of the marker point images, and determine the marker points in each of the marker point images except the coincident marker points as unique marker points, wherein the coincident marker points are coincident points among the marker points corresponding to at least two marker point images;

[0015] A 3D human body model of the current human body is constructed based on the 2D coordinates of the overlapping marking points and the 2D coordinates of the unique marking point in each of the marking point images.

[0016] By adopting the above technical solution, the overlapping marker points in each image can be accurately identified and determined from multiple marker point images. These overlapping marker points are marker points that appear in at least two marker point images, and they provide key spatial reference points for constructing a 3D model. At the same time, the marker points other than the overlapping marker points are determined as unique marker points. These unique marker points reflect the unique characteristics of the human body under different viewing angles, and based on the 2D coordinates of these overlapping marker points and unique marker points, a 3D human body model is constructed. By utilizing the spatial consistency of the overlapping marker points, the positions of these marker points in three-dimensional space can be accurately calculated, and then the basic skeletal structure of the human body can be constructed. At the same time, combined with the information of the unique marker points, the 3D model can be further improved and optimized to make it closer to the real human body shape.

[0017] In a possible implementation, constructing a 3D human body model of the current human body based on the 2D coordinates of the coincident marking points and the 2D coordinates of the unique marking point in each of the marking point images includes:

[0018] For each of the coincident marking points, based on the 2D coordinates of each of the coincident marking points, calculating the image gradient of the coincident marking point in the corresponding marking point image to obtain a plurality of image gradients of the coincident marking point;

[0019] Based on the multiple image gradients corresponding to each of the coincident mark points, calculating the image gradient mean corresponding to each of the coincident mark points;

[0020] Obtaining an initial confidence level corresponding to each of the coincident marking points;

[0021] Determine the confidence interval corresponding to each of the coincident marking points based on the image gradient average value and the initial confidence level corresponding to each of the coincident marking points;

[0022] Based on the confidence interval corresponding to each of the coincident marking points and the image gradient value corresponding to each of the coincident marking points, determining the image to be eliminated corresponding to each of the coincident marking points;

[0023] Eliminate the image to be eliminated corresponding to each of the coincident marking points from each of the marking point images to obtain a target marking point image corresponding to each of the coincident marking points;

[0024] A 3D human body model corresponding to the current human body is constructed based on the 2D coordinates of the respective overlapping marking points in the respective target marking point images and the 2D coordinates of the respective unique marking points in the respective marking point images.

[0025] By adopting the above technical solution, the initial confidence level of each overlapping marking point is obtained by calculating the image gradient in different marking point images and taking the average value. In combination with the image gradient average value, the confidence interval is determined for each overlapping marking point. Based on the confidence interval and image gradient value of the overlapping marking point, the images whose image gradient values ​​deviate greatly from the confidence interval are identified and eliminated, that is, the images need to be eliminated. The errors caused by factors such as image noise and perspective changes are effectively reduced, and the accuracy of subsequent 3D model construction is improved. After eliminating the error image, based on the 2D coordinates of the overlapping marking points in these target marking point images and the 2D coordinates of the unique marking points, the solution successfully constructs the 3D human body model of the current human body.

[0026] In a possible implementation, constructing a 3D human body model corresponding to the current human body based on the 2D coordinates of the respective overlapping marking points in the respective target marking point images and the 2D coordinates of the respective unique marking points in the respective marking point images includes:

[0027] Determine the camera coordinate system corresponding to each motion capture camera, and establish the conversion relationship between each camera coordinate system and the world coordinate system;

[0028] Calculate the matching degree of the same coincident marking point in each two target marking point images, and take the marking point with a matching degree greater than the matching degree threshold as the target marking point;

[0029] Based on the target marking point and the unique marking point, determining basic feature data corresponding to the current human body;

[0030] Based on the basic feature data and the transformation relationship between each camera coordinate system and the world coordinate system, the three-dimensional coordinates corresponding to each marking point are calculated;

[0031] Based on the three-dimensional coordinates corresponding to each marking point, a 3D human body model corresponding to the current human body is generated.

[0032] By adopting the above technical solution, the camera coordinate system corresponding to each motion capture camera is clarified, and the conversion relationship between these camera coordinate systems and the world coordinate system is established. By calculating the matching degree of the same coincident marker point in each two target marker point images, the marker points with a matching degree higher than the threshold are selected as target marker points. Based on the target marker points and the unique marker points, the basic feature data of the current human body is determined. With the basic feature data, the three-dimensional coordinate table corresponding to each marker point is calculated using the conversion relationship between each camera coordinate system and the world coordinate system. Based on the three-dimensional coordinates of each marker point, the 3D human body model of the current human body is successfully generated.

[0033] In a possible implementation, based on the basic feature data and the transformation relationship between each camera coordinate system and the world coordinate system, the three-dimensional coordinates corresponding to each marker point are calculated. The calculation process of the three-dimensional coordinates corresponding to each marker point includes:

[0034] Based on the basic feature data, determining the corresponding relationship between the marker points in the corresponding marker point image, and determining the intrinsic parameter matrix and the extrinsic parameter matrix of the reference camera corresponding to the marker points;

[0035] Obtaining the image coordinates of the marking point in the corresponding second human body image;

[0036] Based on the conversion relationship of the reference camera device corresponding to the target human feature and the internal parameter matrix, the image coordinates of the marked point in the corresponding second human image are converted into the target ray equation corresponding to the marked point;

[0037] Based on the respective transformation relationships between the camera coordinate systems and the world coordinate system, the extrinsic parameter matrix and the target ray equation, the three-dimensional coordinates corresponding to the marking point are calculated.

[0038] By adopting the above technical solution, based on the basic feature data, the corresponding relationship of the marker point in the corresponding marker point image is accurately determined, and the intrinsic parameter matrix and extrinsic parameter matrix of the reference camera corresponding to the marker point are obtained. The image coordinates of the marker point in the corresponding second human image are obtained, and the image coordinates of the marker point in the second human image are converted into the target ray equation corresponding to the marker point by using the conversion relationship of the reference camera and the intrinsic parameter matrix. Based on the conversion relationship between each camera coordinate system and the world coordinate system, the extrinsic parameter matrix and the target ray equation, the three-dimensional coordinates corresponding to the marker point are calculated. This step is the core of the three-dimensional coordinate calculation. It uses ray information from multiple perspectives and obtains the precise position of the marker point in three-dimensional space by solving the ray intersection.

[0039] In a possible implementation, mapping the human 3D skeleton data onto a corresponding human image to obtain 2D skeleton data includes:

[0040] Based on the human 3D skeleton data, the vector angle between adjacent marking points is calculated to obtain the marking point angle information;

[0041] The Zhang Zhengyou calibration method is used to calculate the intrinsic parameter matrix of each motion capture camera, and based on the intrinsic parameter matrix of each motion capture camera, the focal length value corresponding to each motion capture camera is determined;

[0042] Calculate the marker distance between each marker point and the corresponding motion capture camera;

[0043] Based on the focal length value and marker distance corresponding to each marker point, the 3D skeleton data of the human body is converted into 2D skeleton data using the perspective projection formula;

[0044] A scaling factor of the 2D skeleton data is determined, and based on the scaling factor, the 2D skeleton data is mapped onto a corresponding human body image.

[0045] By adopting the above technical solution, based on the 3D skeleton data, the vector angle between adjacent marker points is calculated, thereby obtaining the angle information of the marker points, and the Zhang Zhengyou calibration method is used to accurately calculate the intrinsic parameter matrix of each motion capture camera, and the focal length value of each camera is determined accordingly, and the marker distance between each marker point and the corresponding motion capture camera is calculated. The perspective projection formula is used to accurately convert the 3D skeleton data into 2D skeleton data, determine the scaling factor of the 2D skeleton data, and based on the scaling factor, accurately map the 2D skeleton data to the corresponding human body image. The introduction of the scaling factor is to ensure the consistency of the size of the 2D skeleton data and the human body image, avoiding errors caused by size mismatch.

[0046] In a possible implementation, determining a scaling factor of the 2D skeleton data includes:

[0047] Obtain a human body image corresponding to each 2D skeleton data, and obtain a resolution corresponding to each of the human body images;

[0048] Determine the coordinate range corresponding to each 2D skeleton data, and calculate the offset corresponding to each 2D skeleton data based on the coordinate range corresponding to each 2D skeleton data;

[0049] Each offset is optimized based on the least square method to obtain an optimized offset;

[0050] Based on the optimized offset, a scaling factor of the 2D skeleton data is determined.

[0051] By adopting the above technical solution, the human body image corresponding to each 2D bone data is obtained, and the resolution information of each image is extracted. The coordinate range of each 2D bone data in the image is determined, and the offset of each bone data is calculated accordingly. The scheme uses the least squares method to optimize each offset. The least squares method is a mathematical optimization technique that finds the best function match for the data by minimizing the sum of squares of errors. Based on the optimized offset, the scheme determines the scaling factor of the 2D bone data. The scaling factor is a key parameter that connects 3D bone data with 2D images. It ensures that the bone data can maintain the correct proportion and size when mapped to the image.

[0052] In the second aspect, the present application provides a joint data labeling device based on multi-view, which adopts the following technical solution:

[0053] A joint data labeling device based on multiple perspectives, comprising:

[0054] A receiving module, used to receive a plurality of marked point images and a plurality of human body images corresponding to the current moment, wherein the marked point images are images of marked points on a human body acquired by a motion capture camera, and the human body images are images of the current human body acquired by a reference camera, wherein one camera corresponds to one human body image at any moment, and the human body image includes complete current human body image information, and the current human body corresponds to a plurality of marked points;

[0055] A recognition module, used for recognizing the 2D coordinates of each marker point in each of the marker point images, and constructing a 3D human body model of the current human body based on the 2D coordinates of each marker point;

[0056] A mapping module, used to construct human 3D skeleton data corresponding to the current moment based on the 3D human body model, and map the human 3D skeleton data to the corresponding human body image to obtain 2D skeleton data, wherein the human 3D skeleton data is used to characterize the three-dimensional spatial position of each marker point;

[0057] The determination module is used to determine the 2D joint positions of the current human body based on the 2D skeleton data corresponding to the multiple marker point images.

[0058] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0059] An electronic device, comprising:

[0060] at least one processor;

[0061] Memory;

[0062] At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the multi-view based joint data labeling method described in the first aspect above.

[0063] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0064] A computer-readable storage medium, comprising: storing a computer program that can be loaded by a processor and execute the multi-view joint data labeling method described in the first aspect.

[0065] In summary, this application includes the following beneficial technical effects:

[0066] By receiving multiple marker point images and multiple human body images, the synergy of the motion capture camera and the reference camera is fully utilized. The motion capture camera focuses on accurately capturing the marker points on the human body, while the reference camera provides complete current human body image information, ensuring the comprehensiveness and accuracy of the data. This step lays a solid foundation for subsequent processing. By identifying the 2D coordinates of the marker points in each marker point image and constructing a 3D human body model based on this, the method achieves accurate conversion from two-dimensional to three-dimensional. This step not only increases the dimension of the data, but also provides a reliable basis for the subsequent construction of three-dimensional skeleton data. Then, based on the constructed 3D human body model, the 3D skeleton data of the human body corresponding to the current moment is further generated and mapped to the corresponding human body image to obtain 2D skeleton data. By integrating the 2D skeleton data corresponding to multiple marker point images, the 2D joint position of the current human body can be accurately determined. This step makes full use of the complementarity of multi-view data and improves the accuracy and robustness of joint position annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a scene interaction diagram of an image acquisition device and a human body provided in an embodiment of the present application;

[0068] Figure 2 It is a flowchart of a joint data labeling method based on multiple perspectives provided in an embodiment of the present application;

[0069] Figure 3 It is a block diagram of a joint data labeling device based on multiple perspectives provided in an embodiment of the present application;

[0070] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following is combined with Figure 1 -Attached Figure 4 This application is described in further detail.

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] The existing technology usually uses a single camera to capture human body images, and then uses a feature point detection algorithm to find the same key nodes in the image and calculate the spatial coordinates of these nodes to obtain the joint point positions in three-dimensional space. However, the existing joint data annotation methods have some shortcomings. During the acquisition process of single-view images, due to the influence of factors such as ambient lighting changes and occlusion, some key nodes may not be clearly visible, resulting in a significant reduction in the accuracy of joint data annotation. Therefore, how to improve the accuracy of joint data annotation is a technical problem that needs to be solved urgently.

[0074] In view of this, the present application embodiment provides a joint data annotation method based on multi-view. Figure 1 There are multiple motion capture cameras 110 and reference cameras 120 around the human body, which are used to capture images of the human body. Each motion capture camera 110 and each reference camera 120 has a fixed position and a fixed posture. Each motion capture camera 110 and each reference camera 120 captures an image containing the human body at every moment. The electronic device establishes a communication connection with the multiple motion capture cameras 110 and the reference cameras 120, and the connection can be achieved through a wired network (such as Ethernet) or a wireless network (such as Wi-Fi). At the same time, the electronic device can obtain the images containing the human body captured by each motion capture camera 110 and each reference camera 120.

[0075] The present application embodiment provides a joint data annotation method based on multiple perspectives, such as Figure 2 As shown, the method provided in the embodiment of the present application is executed by an electronic device, which can be a server or a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. The method includes steps S201 to S204, wherein:

[0076] Step S201: receiving a plurality of marker point images and a plurality of human body images corresponding to the current moment.

[0077] Among them, the marker point image is the image of the marker points on the human body captured by the motion capture camera, and the human body image is the image of the current human body captured by the reference camera. Among them, one camera corresponds to a human body image at any time, and the human body image includes complete current human body image information. There are multiple marker points corresponding to the current human body.

[0078] Specifically, the electronic device establishes a connection with the motion capture camera and the reference camera through a specific communication interface (such as an Ethernet interface, a high-speed USB interface, etc.). The motion capture camera and the reference camera respectively perform image acquisition at the same time according to a pre-set synchronization mechanism (such as using a high-precision synchronization clock module).

[0079] The motion capture camera shoots the markers on the human body at a high frame rate (e.g. 200 frames / second) and a suitable resolution (e.g. 1080p). These markers are usually highly reflective, specially made objects that are easy for the motion capture camera to capture accurately. The captured marker images will be accompanied by information such as timestamps and camera numbers, and transmitted to electronic devices via network protocols (e.g. TCP / IP protocols).

[0080] The reference camera shoots the current human body from different angles (such as front, back, left, right, top, bottom, etc.) to obtain images containing complete human image information, including details such as the human appearance, posture, and clothing. The shooting frame rate (such as 60 frames per second) and resolution (such as 4K) of the reference camera are also set according to actual needs. The captured human body image also carries identification information such as timestamp and camera number, which is transmitted to the electronic device through the network.

[0081] After receiving these images, the electronic device classifies and stores the images according to the timestamp and camera number, and stores the mark point images and human body images at the same time in different folders or data areas for subsequent processing.

[0082] Step S202: Identify the 2D coordinates of each marker point in each marker point image, and construct a 3D human body model of the current human body based on the 2D coordinates of each marker point.

[0083] Specifically, a deep learning-based target detection algorithm (such as a specially trained YOLO algorithm) can be used to process each marker point image. The target detection algorithm has been trained with a large amount of image data containing marker points, and can accurately locate each marker point in the image and output its two-dimensional coordinates in the image plane (in pixels, with the upper left corner of the image as the origin, the horizontal right as the positive x-axis, and the vertical downward as the positive y-axis).

[0084] Furthermore, after obtaining the 2D coordinates of each marker point in each marker point image, a 3D human body model of the current human body may be constructed based on the 2D coordinates of each marker point.

[0085] In this embodiment, a 3D human body model of the current human body is constructed based on the 2D coordinates of each marker point, including: determining the overlapping marker points in each marker point image from multiple marker point images, and determining the marker points in each marker point image except the overlapping marker points as unique marker points, and the overlapping marker points are the overlapping points of the marker points corresponding to at least two marker point images; based on the 2D coordinates of the overlapping marker points in each marker point image and the 2D coordinates of the unique marker points, a 3D human body model of the current human body is constructed.

[0086] Read multiple marker point images in sequence, and use existing marker point recognition algorithms (such as target detection algorithms based on deep learning, such as the improved YOLO algorithm, which is specially trained for marker point features) to accurately identify the 2D coordinates of all marker points in each marker point image. Then, compare and analyze the 2D coordinates of the marker points in all marker point images. For any two marker point images, set a coordinate difference threshold (for example, in pixels, if the horizontal and vertical coordinate differences of the two marker points are less than the threshold, they are considered to be likely to overlap) to determine whether the marker points overlap. Specifically, for a marker point in marker point image A, and a marker point in the marker image B ,if ≤ threshold and ≤threshold, P1 and P2 are considered to be overlapping marking points.

[0087] After obtaining the 2D coordinates of the coincident marking points and the 2D coordinates of the unique marking point in each marking point image, a 3D human body model of the current human body can be constructed.

[0088] Specifically, in the present embodiment, based on the 2D coordinates of the coincident marking points in each marking point image and the 2D coordinates of the unique marking point, a 3D human body model of the current human body is constructed, including: for each coincident marking point, based on the 2D coordinates of each coincident marking point, calculating the image gradient of the coincident marking point in the corresponding marking point image to obtain multiple image gradients of the coincident marking point; based on the multiple image gradients corresponding to each coincident marking point, calculating the image gradient mean corresponding to each coincident marking point; obtaining the initial confidence level corresponding to each coincident marking point; based on the image gradient mean value corresponding to each coincident marking point and the initial confidence level, determining the confidence interval corresponding to each coincident marking point; based on the confidence interval corresponding to each coincident marking point and the image gradient value corresponding to each coincident marking point, determining the image to be removed corresponding to each coincident marking point; removing the image to be removed corresponding to each coincident marking point from each marking point image to obtain the target marking point image corresponding to each coincident marking point; and constructing the 3D human body model corresponding to the current human body based on the 2D coordinates of the respective coincident marking points in each target marking point image and the 2D coordinates of the respective unique marking points in each marking point image.

[0089] Specifically, for each coincident marker point, operations are performed in each marker point image where it appears. With the 2D coordinates of the coincident marker point as the center, a small neighborhood range is defined (for example, a 3×3 or 5×5 pixel area centered on the marker point coordinates). In this neighborhood, image gradient calculation operators (such as Sobel operator, Prewitt operator, etc.) can be used to calculate the gradient components of the image in the horizontal direction (x direction) and the vertical direction (y direction). For each direction, the gradient value is obtained by weighted difference operation of the grayscale values ​​of the pixels in the neighborhood, and then the gradient components in the horizontal and vertical directions are combined (such as using the Euclidean norm to calculate the composite gradient) to obtain the image gradient of the coincident marker point in the current marker point image. Such calculations are performed for each coincident marker point in all marker point images where it appears, thereby obtaining multiple image gradients of the coincident marker point.

[0090] Furthermore, for each coincident marker point, the electronic device collects multiple image gradients calculated in different marker point images. Assume that the image gradients of a coincident marker point in n marker point images are respectively: These image gradients are summarized, and the horizontal and vertical gradient components are summed up respectively, and then divided by n respectively to obtain the average gradient components in the horizontal and vertical directions, and then the image gradient mean corresponding to the coincident mark point is synthesized. In this way, the average image gradient feature of each coincident mark point in multiple images is obtained.

[0091] Furthermore, the initial confidence level corresponding to each coincident marker point is obtained from the database, and then the confidence interval corresponding to each coincident marker point can be determined by a statistical and weight-based method. Generally speaking, the image gradient average reflects the changes in the image features around the marker point, and the initial confidence level reflects the degree of trust in the marker point itself. Therefore, a mathematical model can be constructed to weight the variance of the image gradient average (indicating the degree of fluctuation of the image gradient) and the initial confidence level. For example, for coincident marker points with a small variance of the image gradient average (indicating that the image features around the marker point are stable) and a high initial confidence level, the confidence interval is relatively narrow; conversely, for coincident marker points with a large variance or a low initial confidence level, the confidence interval is relatively wide. Through such calculations, a reasonable confidence interval is determined for each coincident marker point to measure the feature consistency and reliability range of the marker point in different images.

[0092] For each overlapping marker point, its image gradient value in each marker point image can be compared with the corresponding confidence interval. If the image gradient value of the overlapping marker point in a marker point image exceeds its corresponding confidence interval, it means that the feature of the overlapping marker point in the image is greatly different from the average feature in other images, and there may be inaccurate data or interference, and such a marker point image is determined as an image to be eliminated. By comparing the image gradient values ​​of each overlapping marker point in all marker point images one by one, a list of images to be eliminated corresponding to each overlapping marker point is determined. Then, according to the list of images to be eliminated, all marker point images are screened, and the remaining marker point images are the target marker point images corresponding to each overlapping marker point. The overlapping marker points in these target marker point images have more reliable image features and can provide more accurate data support for the subsequent construction of 3D human body models.

[0093] Furthermore, after obtaining the 2D coordinates of the respective overlapping marking points in each target marking point image and the 2D coordinates of the respective unique marking points in each marking point image, a 3D human body model corresponding to the current human body can be constructed based on the 2D coordinates of the respective overlapping marking points in each target marking point image and the 2D coordinates of the respective unique marking points in each marking point image.

[0094] Specifically, in the present embodiment, a 3D human body model corresponding to the current human body is constructed based on the 2D coordinates of the respective overlapping marking points in each target marking point image and the 2D coordinates of the respective unique marking points in each marking point image, including: determining the camera coordinate system corresponding to each motion capture camera, and establishing the respective transformation relationship between each camera coordinate system and the world coordinate system; calculating the matching degree of the same overlapping marking point in each two target marking point images, and taking the marking point with a matching degree greater than a matching degree threshold as the target marking point; determining the basic feature data corresponding to the current human body based on the target marking point and the unique marking point; calculating the three-dimensional coordinates corresponding to each marking point based on the basic feature data and the respective transformation relationship between each camera coordinate system and the world coordinate system; and generating a 3D human body model corresponding to the current human body based on the three-dimensional coordinates corresponding to each marking point.

[0095] Specifically, for each image captured by a motion capture camera, a corner detection algorithm (such as the Harris corner detection algorithm based on sub-pixel accuracy) can be used to identify the two-dimensional coordinates of the corners on the calibration object in the image. Since the three-dimensional coordinates of the corners of the calibration object in the world coordinate system are known (determined according to the design and manufacturing accuracy of the calibration object), these known world coordinates and the detected two-dimensional image coordinates are used to calculate the intrinsic parameter matrix (including focal length, principal point coordinates and other parameters reflecting the internal optical characteristics of the camera) and extrinsic parameter matrix (including rotation matrix and translation vector, describing the position and posture of the camera in the world coordinate system) of each motion capture camera through mature camera calibration algorithms such as Zhang Zhengyou calibration method.

[0096] Through the intrinsic parameter matrix and the extrinsic parameter matrix, the conversion relationship between the camera coordinate system and the world coordinate system of each motion capture camera is established. In this way, in subsequent processing, the coordinates in the camera coordinate system can be converted to the world coordinate system, or vice versa. Among them, the intrinsic parameter matrix is ​​a matrix that describes the internal optical properties of the motion capture camera, which projects points in three-dimensional space onto the two-dimensional image plane. The intrinsic parameter matrix contains parameters such as the focal length and principal point coordinates of the camera. Different cameras have different intrinsic parameter matrices. The extrinsic parameter matrix consists of a rotation matrix and a translation vector, which is used to describe the position and posture of the motion capture camera in the world coordinate system. The extrinsic parameter matrix can convert coordinates in the camera coordinate system to the world coordinate system, or convert coordinates in the world coordinate system to the camera coordinate system.

[0097] Furthermore, each two target marker images are taken out in turn for processing. For the same coincident marker in each pair of target marker images, the image features around the marker can be extracted (for example, using feature extraction algorithms such as SIFT, SURF or ORB to extract local feature descriptors around the marker). Then, the degree of matching between the two markers is determined by calculating the similarity between these feature descriptors (such as using measurement methods such as Euclidean distance and cosine similarity). Specifically, if the Euclidean distance is used, the smaller the distance, the more similar the features of the two markers are, and the higher the degree of matching; if the cosine similarity is used, the closer the similarity value is to 1, the higher the degree of matching. A matching degree threshold is obtained (for example, the Euclidean distance is less than a certain value, or the cosine similarity is greater than a certain value), and the markers with a matching degree greater than the threshold are identified as target markers. By performing such processing on all target marker images, target markers that meet the requirements are screened out.

[0098] For target markers, since they are reliable markers obtained through screening, their two-dimensional coordinate information has a high degree of credibility. For unique markers, they are unique markers in each marker image, and their two-dimensional coordinates are also recorded. The two-dimensional coordinate information of these markers and their related features in the image (such as the image grayscale value and texture features around the markers) are sorted and stored to form a data set. This data set contains the location and feature information of the target markers and unique markers as the basic feature data corresponding to the current human body.

[0099] Furthermore, for each marker point in the basic feature data (including target marker points and unique marker points), its three-dimensional coordinates can be calculated using the triangulation principle based on its two-dimensional coordinates in multiple target marker point images and the corresponding intrinsic parameter matrix and extrinsic parameter matrix of the motion capture camera (i.e., the conversion relationship between the camera coordinate system and the world coordinate system). Specifically, for each marker point, its two-dimensional coordinates in different camera images can be converted into a ray equation in the camera coordinate system in combination with the camera's intrinsic parameter matrix. Then, these ray equations are converted into the world coordinate system using the camera's extrinsic parameter matrix. By combining the ray equations of multiple cameras (because each camera provides a ray equation about the marker point), the intersection of these equations is solved to obtain the three-dimensional coordinates of the marker point in the world coordinate system. Such calculations are performed for all marker points in the basic feature data to obtain the three-dimensional coordinates corresponding to each marker point, forming a data set containing the three-dimensional coordinates of all marker points.

[0100] Furthermore, the 3D coordinates of each marker point can be used to generate a 3D human body model corresponding to the current human body using a 3D reconstruction algorithm (such as a Poisson reconstruction algorithm, a grid-based reconstruction algorithm, etc.). Specifically, taking the Poisson reconstruction algorithm as an example, the 3D coordinates of all marker points can be used as input data. The algorithm will generate a continuous 3D surface model by constructing an implicit surface based on the spatial distribution and density information of these points. During the generation process, the algorithm will consider the distance and direction relationship between the marker points to ensure that the generated model can accurately reflect the shape and structure of the human body. For the grid-based reconstruction algorithm, an initial grid structure is first constructed based on the 3D coordinates of the marker points, and then the grid is adjusted and refined through an optimization algorithm (such as a Laplace smoothing algorithm, etc.) to make the grid surface smoother and accurately fit the shape of the human body. After these steps, the 3D human body model corresponding to the current human body is finally generated.

[0101] In this embodiment, the three-dimensional coordinates corresponding to each marker point are calculated based on basic feature data and the respective transformation relationships between each camera coordinate system and the world coordinate system. The calculation process of the three-dimensional coordinates corresponding to each marker point includes: based on the basic feature data, determining the corresponding relationship between the marker point in the corresponding marker point image, and determining the intrinsic parameter matrix and the extrinsic parameter matrix of the reference camera corresponding to the marker point; obtaining the image coordinates of the marker point in the corresponding second human image; based on the transformation relationship of the reference camera corresponding to the target human feature and the intrinsic parameter matrix, converting the image coordinates of the marker point in the corresponding second human image into the target ray equation corresponding to the marker point; based on the respective transformation relationships between each camera coordinate system and the world coordinate system, the extrinsic parameter matrix and the target ray equation, calculating the three-dimensional coordinates corresponding to the marker point.

[0102] Specifically, relevant information of the marker point is extracted from the basic feature data, including the two-dimensional coordinates of the marker point and the identifier of the marker point image where it is located. For each marker point, the electronic device determines the correspondence between the marker point in different marker point images by comparing the marker point information in different marker point images and using a feature matching algorithm (such as a matching algorithm based on feature descriptors, such as SIFT, SURF, etc.). For example, for a marker point, in marker point image A and image B, the similarity of the feature descriptors around it is calculated to determine whether it is the same marker point. After determining the correspondence, the identifier of the reference camera that took the image can be found based on the image information where the marker point is located. Then, the intrinsic parameter matrix (including parameters such as focal length and principal point coordinates) and extrinsic parameter matrix (including rotation matrix and translation vector, describing the position and posture of the camera in the world coordinate system) corresponding to the reference camera are retrieved from the pre-stored camera parameter database.

[0103] According to the image information of the marker point where the marker point is located, find the corresponding second human image (the second human image is an image taken by the reference camera containing complete human body information). Then, use image recognition and positioning technology (such as a template matching-based method or using the position information of the previous marker point in the image for association) to accurately find the position of the marker point in the second human image and obtain its two-dimensional coordinates in the second human image (in pixels, usually with the upper left corner of the image as the coordinate origin, horizontal to the right as the positive direction of the x-axis, and vertically downward as the positive direction of the y-axis).

[0104] Furthermore, for the image coordinates (u, v) of the marker point in the second human body image, the intrinsic parameter matrix is ​​used to convert it into the normalized coordinates (x, y) in the camera coordinate system. The conversion formula is: ,(in is the horizontal coordinate of the principal point in the intrinsic parameter matrix, is the horizontal focal length); ,(in is the ordinate of the principal point, is the vertical focal length). Furthermore, after obtaining the normalized coordinates, the relationship between the camera coordinate system and the world coordinate system can be determined by combining the external parameter matrix (rotation matrix and translation vector) of the reference camera. Specifically, in the camera coordinate system, with the camera optical center as the origin, a ray is determined by the normalized coordinates, and the direction vector of the ray is (x, y, 1) (in the camera coordinate system), thereby obtaining the target ray equation corresponding to the marked point ,in is the coordinate of the camera's optical center in the camera coordinate system (usually the origin (0, 0, 0), is a parameter, = (x, y, 1) is the direction vector of the ray.

[0105] Furthermore, these ray equations can be converted to the world coordinate system, and the extrinsic matrix of the camera can be used to convert the ray equations in the camera coordinate system into the equations in the world coordinate system. Then, by combining the ray equations of multiple cameras in the world coordinate system (because these rays should pass through the actual position of the marker point in the world coordinate system), the intersection of these equations is solved. The coordinates of this intersection are the three-dimensional coordinates (X, Y, Z) of the marker point in the world coordinate system, and the three-dimensional coordinate values ​​are obtained by solving the system of equations (such as solving it using linear algebra methods).

[0106] Step S203: construct the human 3D skeleton data corresponding to the current moment based on the 3D human body model, and map the human 3D skeleton data to the corresponding human body image to obtain 2D skeleton data.

[0107] Among them, the human body 3D skeleton data is used to represent the three-dimensional spatial position of each marker point.

[0108] The preset key nodes of the human body are obtained from the database, and the key joints (such as shoulder joints, elbow joints, knee joints, etc.) are identified in the 3D human body model based on the preset key nodes of the human body. The three-dimensional coordinates of each joint point are determined by analyzing the distribution and connection relationship of the marker points in the 3D human body model. At the same time, according to the bone connection relationship between the joint points, the 3D bone data of the human body is constructed, including the three-dimensional coordinates of the joint points and the connection information between the joint points (such as the length and angle of the bones, etc.), so as to obtain the 3D bone data of the human body.

[0109] In this embodiment, the 3D skeleton data of the human body is mapped to the corresponding human body image to obtain 2D skeleton data, including: based on the 3D skeleton data of the human body, calculating the vector angle between adjacent marking points to obtain the angle information of the marking points; using the Zhang Zhengyou calibration method to calculate the intrinsic parameter matrix of each motion capture camera, and based on the intrinsic parameter matrix of each motion capture camera, determining the focal length value corresponding to each motion capture camera; calculating the marking distance between each marking point and the corresponding motion capture camera; based on the focal length value corresponding to each marking point and the marking distance, and using the perspective projection formula to convert the 3D skeleton data of the human body into 2D skeleton data; determining the scaling factor of the 2D skeleton data, and based on the scaling factor, mapping the 2D skeleton data to the corresponding human body image.

[0110] Specifically, the three-dimensional coordinate information of each marker point is extracted from the acquired human 3D skeleton data. For each group of adjacent marker points (which marker points are adjacent is determined based on the structure and connection relationship of the human skeleton, for example, the marker points corresponding to the shoulder joint and the elbow joint are adjacent), their three-dimensional coordinates can be obtained respectively. Let two adjacent marker points be and More specifically, we calculate the vector from one marked point to another marked point, such as the vector

[0111] Then, this vector calculation is performed for all pairs of adjacent marker points. Further, the dot product formula of the vector can be used (in and are two vectors, is the angle between them), calculate the angle between adjacent vectors. By inverse trigonometric functions (Assume that A, B, and C are three adjacent marking points to calculate and The angle between adjacent marking points is obtained by combining these angle information to form the angle information of the marking point.

[0112] For each image captured by the motion capture camera, the electronic device uses a corner detection algorithm (such as the Harris corner detection algorithm) to accurately detect the two-dimensional coordinates of the corners on the chessboard in the image. Since the three-dimensional coordinates of the corners of the chessboard in the world coordinate system are known (determined according to the design and manufacturing accuracy of the chessboard), these known three-dimensional coordinates and the detected two-dimensional image coordinates can be used to calculate using the Zhang Zhengyou calibration method. Specifically, the Zhang Zhengyou calibration method solves the camera's intrinsic parameter matrix through a series of mathematical operations. The general form of the intrinsic parameter matrix is ,in and are the focal lengths of the camera in the horizontal and vertical directions, and is the coordinate of the principal point in the image plane. The calculated internal parameter matrix extracts the sum of the focal lengths of each motion capture camera in the horizontal and vertical directions for subsequent coordinate transformation calculations.

[0113] Furthermore, the three-dimensional coordinates of each marker point (derived from the 3D human skeleton data) and the position coordinates of each motion capture camera in the world coordinate system are known (the external parameter matrix obtained by the previous camera calibration can determine the position of the camera in the world coordinate system). For each marker point and the corresponding motion capture camera, the spatial distance formula can be used to calculate the spatial distance between them. This distance is the marker distance between the marker point and the corresponding motion capture camera.

[0114] Furthermore, for each marker point in the human 3D skeleton data, the focal length value of the motion capture camera corresponding to the marker point calculated previously (including the horizontal focal length and vertical focal length ) and the marker distance d between the marker point and the motion capture camera. Specifically, the perspective projection formula can be used for coordinate transformation. For a marker point P (X, Y, Z) in three-dimensional space, the projection coordinates on the two-dimensional image plane are calculated as follows: , (Here, it can be roughly regarded as the distance between the marker point and the motion capture camera.) Such perspective projection calculation is performed on all marker points to convert the three-dimensional coordinates of the marker points in the human body 3D skeleton data into two-dimensional coordinates. The set of these two-dimensional coordinates is the 2D skeleton data.

[0115] First, obtain the size information of the corresponding human image, including the width and height of the image. At the same time, analyze the range of the coordinates of the marker points in the 2D skeleton data to determine its maximum and minimum coordinate values ​​in the horizontal and vertical directions. Then calculate the scaling factors of the 2D skeleton data in the horizontal and vertical directions, and multiply the two-dimensional coordinates of each marker point in the 2D skeleton data by the corresponding scaling factor to obtain the new coordinates. Map the converted coordinates to the corresponding human image to complete the mapping process of 2D skeleton data to human image.

[0116] In this embodiment, determining the scaling factor of 2D bone data includes: obtaining a human body image corresponding to each 2D bone data, and obtaining a resolution corresponding to each human body image; determining a coordinate range corresponding to each 2D bone data, and based on the coordinate range corresponding to each 2D bone data, calculating an offset corresponding to each 2D bone data; optimizing each offset based on the least squares method to obtain an optimized offset; and determining the scaling factor of the 2D bone data based on the optimized offset.

[0117] Specifically, the correspondence between 2D skeleton data and human body images is obtained from the database, and based on this association table, the human body image file corresponding to each 2D skeleton data is searched and read from the storage device (such as a hard disk, solid-state drive, etc.). For the read human body image, the relevant functions of the image library can be called to parse the metadata of the image file. Usually, the metadata of the image file contains the resolution information of the image, and the electronic device extracts the width (in pixels) and height (in pixels) of the image from it. These two values ​​constitute the resolution of the human body image.

[0118] Furthermore, all the joint point coordinates in each 2D skeleton data are traversed. For each joint point, its x coordinate and y coordinate are recorded. By comparing the x coordinates of all joint points, the minimum value is found. and maximum value ; Similarly, find the minimum value of the y coordinates of all nodes and maximum value .so, and This constitutes the coordinate range of the 2D bone data in the x-axis and y-axis directions.

[0119] The offset is calculated based on the resolution of the human image and the coordinate range of the 2D skeleton data. Assuming that the width of the human image is W and the height is H, the middle values ​​of the 2D skeleton data in the x-axis and y-axis directions are calculated, which are and Then, the middle position of the human body image in the x-axis and y-axis directions is calculated, which are and Finally, the offset in the x-axis direction , the offset in the y-axis direction .

[0120] Furthermore, an objective function is defined to measure the error between the 2D skeleton data and the expected position in the human body image after offset adjustment. Assume that the expected joint point position is , after the current offset The adjusted joint point position is (u, v). For all related nodes i=1,2,…n, the objective function S can be expressed as

[0121] The expected position here can be predetermined based on the characteristics of the human body image or prior knowledge. Furthermore, the least square method can be used for iterative optimization. In each iteration, the objective function is calculated with respect to the offset The partial derivative of ,and According to the direction and size of the partial derivative, adjust the value of the offset so that the value of the objective function gradually decreases. The specific adjustment method can be the gradient descent method, that is, , ,in is the learning rate, which is used to control the step size of each iteration. The iterative process is repeated until the objective function converges to a minimum value, at which point the offset obtained is the optimized offset.

[0122] The coordinates of the 2D skeleton data are adjusted according to the optimized offset. For each joint point coordinate in the 2D skeleton data, the adjusted coordinates are

[0123] . Further, the coordinate range of the adjusted 2D bone data is re-determined to obtain a new , , and Finally, the scaling factor of the 2D bone data is calculated. The horizontal scaling factor , the vertical scaling factor , where W and H are the width and height of the human image, respectively.

[0124] Step S204: Determine the 2D joint positions of the current human body based on the 2D skeleton data corresponding to the multiple marker point images.

[0125] The 2D skeleton data corresponding to multiple marker point images are integrated. For each joint point, its 2D coordinate information in different marker point images is collected. Then, by using the principle of multi-view geometry, the corresponding relationship between the 2D coordinates of the joint points in different images is analyzed by comparing them. Specifically, a feature matching algorithm (such as a matching algorithm based on feature descriptors, such as SIFT or SURF algorithm) can be used to determine the corresponding coordinates of the same joint points in different images. Then, based on these corresponding coordinates, combined with the parameters of the reference camera (such as the intrinsic parameter matrix, the extrinsic parameter matrix) and the relative position relationship between the images, the 2D coordinates of the joint points are optimized and corrected through the triangulation method, so as to determine the 2D joint position of the current human body on the human body image, making the determination of the joint position more accurate and reliable.

[0126] The embodiment of the present application provides a method for labeling joint data based on multiple perspectives, which makes full use of the synergy of the motion capture camera and the reference camera by receiving multiple marker point images and multiple human body images. The motion capture camera focuses on accurately capturing the marker points on the human body, while the reference camera provides complete current human body image information, ensuring the comprehensiveness and accuracy of the data. This step lays a solid foundation for subsequent processing. By identifying the 2D coordinates of the marker points in each marker point image and building a 3D human body model accordingly, the method achieves an accurate conversion from two dimensions to three dimensions. This step not only improves the dimension of the data, but also provides a reliable basis for the subsequent construction of three-dimensional skeleton data. Then, based on the constructed 3D human body model, the human body 3D skeleton data corresponding to the current moment is further generated, and it is mapped to the corresponding human body image, thereby obtaining 2D skeleton data, and by integrating the 2D skeleton data corresponding to multiple marker point images, the 2D joint position of the current human body can be accurately determined. This step makes full use of the complementarity of multi-perspective data and improves the accuracy and robustness of joint position labeling.

[0127] See also Figure 3 The multi-view joint data labeling device 30 may specifically include: a receiving module 301, a recognition module 302, a mapping module 303 and a determination module 304, wherein:

[0128] A joint data labeling device 30 based on multiple perspectives, comprising:

[0129] The receiving module 301 is used to receive a plurality of marked point images and a plurality of human body images corresponding to the current moment, wherein the marked point images are images of marked points on the human body acquired by the motion capture camera, and the human body images are images of the current human body acquired by the reference camera, wherein one camera corresponds to one human body image at any moment, and the human body image includes complete current human body image information, and there are a plurality of marked points on the current human body;

[0130] The recognition module 302 is used to recognize the 2D coordinates of each marker point in each marker point image, and construct a 3D human body model of the current human body based on the 2D coordinates of each marker point;

[0131] A mapping module 303 is used to construct the human 3D skeleton data corresponding to the current moment based on the 3D human body model, and map the human 3D skeleton data to the corresponding human body image to obtain 2D skeleton data, where the human 3D skeleton data is used to characterize the three-dimensional spatial position of each marker point;

[0132] The determination module 304 is used to determine the 2D joint positions of the current human body based on the 2D skeleton data corresponding to the multiple marker point images.

[0133] In a possible implementation of the embodiment of the present application, when the recognition module 302 constructs the 3D human body model of the current human body based on the 2D coordinates of each marker point, it can be specifically used to:

[0134] From the plurality of marker point images, determining the coincident marker points in each marker point image, and determining the marker points in each marker point image except the coincident marker points as unique marker points, wherein the coincident marker points are coincident points among the marker points corresponding to at least two marker point images;

[0135] A 3D human body model of the current human body is constructed based on the 2D coordinates of the coincident markers in each marker image and the 2D coordinates of the unique marker.

[0136] In a possible implementation of the embodiment of the present application, when the recognition module 302 constructs the 3D human body model of the current human body based on the 2D coordinates of the coincident marking points and the 2D coordinates of the unique marking points in each marking point image, it can be specifically used to:

[0137] For each coincident marking point, based on the 2D coordinates of each coincident marking point, the image gradient of the coincident marking point in the corresponding marking point image is calculated to obtain multiple image gradients of the coincident marking point;

[0138] Based on the multiple image gradients corresponding to each coincident marking point, a mean value of the image gradient corresponding to each coincident marking point is calculated;

[0139] Get the initial confidence level corresponding to each coincident mark point;

[0140] Determine the confidence interval corresponding to each coincident marking point based on the image gradient average value corresponding to each coincident marking point and the initial confidence level;

[0141] Based on the confidence interval corresponding to each coincident marking point and the image gradient value corresponding to each coincident marking point, determine the image to be removed corresponding to each coincident marking point;

[0142] The image to be removed corresponding to each coincident marking point is removed from each marking point image to obtain the target marking point image corresponding to each coincident marking point;

[0143] Based on the 2D coordinates of the respective overlapping marking points in each target marking point image and the 2D coordinates of the respective unique marking points in each marking point image, a 3D human body model corresponding to the current human body is constructed.

[0144] In a possible implementation of the embodiment of the present application, when the recognition module 302 constructs a 3D human body model corresponding to the current human body based on the 2D coordinates of the respective overlapping marking points in each target marking point image and the 2D coordinates of the respective unique marking points in each marking point image, it can be specifically used to:

[0145] Determine the camera coordinate system corresponding to each motion capture camera, and establish the conversion relationship between each camera coordinate system and the world coordinate system;

[0146] Calculate the matching degree of the same coincident marking point in each two target marking point images, and take the marking point with a matching degree greater than the matching degree threshold as the target marking point;

[0147] Based on the target marking point and the unique marking point, determine the basic feature data corresponding to the current human body;

[0148] Based on the basic feature data and the transformation relationship between each camera coordinate system and the world coordinate system, the three-dimensional coordinates corresponding to each marker point are calculated;

[0149] Based on the three-dimensional coordinates corresponding to each marker point, a 3D human body model corresponding to the current human body is generated.

[0150] In a possible implementation of the embodiment of the present application, the recognition module 302 calculates the three-dimensional coordinates corresponding to each marking point based on the basic feature data and the conversion relationship between each camera coordinate system and the world coordinate system. The calculation process of the three-dimensional coordinates corresponding to each marking point can be specifically used for:

[0151] Based on the basic feature data, determine the corresponding relationship between the marker points in the corresponding marker point image, and determine the intrinsic parameter matrix and extrinsic parameter matrix of the reference camera corresponding to the marker points;

[0152] Obtaining the image coordinates of the marking point in the corresponding second human body image;

[0153] Based on the conversion relationship of the reference camera device corresponding to the target human feature and the internal parameter matrix, the image coordinates of the marker point in the corresponding second human image are converted into the target ray equation corresponding to the marker point;

[0154] Based on the transformation relationship between each camera coordinate system and the world coordinate system, the external parameter matrix and the target ray equation, the three-dimensional coordinates corresponding to the marked point are calculated.

[0155] In a possible implementation of the embodiment of the present application, when the mapping module 303 maps the 3D skeleton data of the human body to the corresponding human body image to obtain the 2D skeleton data, it can be specifically used to:

[0156] Based on the 3D skeleton data of the human body, the vector angle between adjacent marker points is calculated to obtain the angle information of the marker points;

[0157] The Zhang Zhengyou calibration method is used to calculate the intrinsic parameter matrix of each motion capture camera, and based on the intrinsic parameter matrix of each motion capture camera, the focal length value corresponding to each motion capture camera is determined;

[0158] Calculate the marker distance between each marker point and the corresponding motion capture camera;

[0159] Based on the focal length value and marker distance corresponding to each marker point, the 3D skeleton data of the human body is converted into 2D skeleton data using the perspective projection formula;

[0160] A scaling factor of the 2D skeleton data is determined, and based on the scaling factor, the 2D skeleton data is mapped onto a corresponding human body image.

[0161] In a possible implementation of the embodiment of the present application, when determining the scaling factor of the 2D skeleton data, the mapping module 303 may be specifically used to:

[0162] Obtain a human body image corresponding to each 2D skeleton data, and obtain a resolution corresponding to each human body image;

[0163] Determine the coordinate range corresponding to each 2D skeleton data, and calculate the offset corresponding to each 2D skeleton data based on the coordinate range corresponding to each 2D skeleton data;

[0164] Each offset is optimized based on the least square method to obtain an optimized offset;

[0165] Based on the optimized offset, determine the scaling factor of the 2D bone data.

[0166] See also Figure 4 , the embodiment of the present application also introduces an electronic device from the perspective of a physical device, such as Figure 4 As shown, Figure 4The electronic device 400 shown includes: a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, such as through a bus 402. Optionally, the electronic device 400 may also include a transceiver 404. It should be noted that in actual applications, the transceiver 404 is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application.

[0167] Processor 401 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0168] The bus 402 may include a path to transmit information between the above components. The bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0169] The memory 403 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0170] The memory 403 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 403 to implement the contents shown in the above method embodiment.

[0171] The electronic devices include but are not limited to: mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc., and can also be servers, etc. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0172] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0173] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0174] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A joint data labeling method based on multi-view, characterized in that: include: Receive multiple marker point images and multiple human body images corresponding to the current moment, wherein the marker point images are images of marker points on the human body acquired by a motion capture camera, and the human body images are images of the current human body acquired by a reference camera, wherein one camera corresponds to one human body image at any moment, and the human body image includes complete current human body image information, and the current human body corresponds to multiple marker points; Identify the 2D coordinates of each marker point in each of the marker point images, and construct a 3D human body model of the current human body based on the 2D coordinates of each marker point; Constructing human 3D skeleton data corresponding to the current moment based on the 3D human body model, and mapping the human 3D skeleton data to the corresponding human body image to obtain 2D skeleton data, wherein the human 3D skeleton data is used to characterize the three-dimensional spatial position of each marker point; Based on the 2D skeleton data corresponding to the multiple marker point images, the 2D joint positions of the current human body are determined.

2. The method for labeling joint data based on multiple perspectives according to claim 1, characterized in that: The step of constructing a 3D human body model of the current human body based on the 2D coordinates of each marker point comprises: From the plurality of marker point images, determine the coincident marker points in each of the marker point images, and determine the marker points in each of the marker point images except the coincident marker points as unique marker points, wherein the coincident marker points are coincident points among the marker points corresponding to at least two marker point images; A 3D human body model of the current human body is constructed based on the 2D coordinates of the overlapping marking points and the 2D coordinates of the unique marking point in each of the marking point images.

3. The multi-view joint data labeling method according to claim 2, characterized in that: The step of constructing a 3D human body model of the current human body based on the 2D coordinates of the coincident marking points and the 2D coordinates of the unique marking point in each of the marking point images comprises: For each of the coincident marking points, based on the 2D coordinates of each of the coincident marking points, calculating the image gradient of the coincident marking point in the corresponding marking point image to obtain a plurality of image gradients of the coincident marking point; Based on the multiple image gradients corresponding to each of the coincident mark points, calculating the image gradient mean corresponding to each of the coincident mark points; Obtaining an initial confidence level corresponding to each of the coincident marking points; Determine the confidence interval corresponding to each of the coincident marking points based on the image gradient average value and the initial confidence level corresponding to each of the coincident marking points; Based on the confidence interval corresponding to each of the coincident marking points and the image gradient value corresponding to each of the coincident marking points, determining the image to be eliminated corresponding to each of the coincident marking points; Eliminate the image to be eliminated corresponding to each of the coincident marking points from each of the marking point images to obtain a target marking point image corresponding to each of the coincident marking points; A 3D human body model corresponding to the current human body is constructed based on the 2D coordinates of the respective overlapping marking points in the respective target marking point images and the 2D coordinates of the respective unique marking points in the respective marking point images.

4. The multi-view joint data labeling method according to claim 3, characterized in that: The constructing a 3D human body model corresponding to the current human body based on the 2D coordinates of the respective overlapping marking points in the respective target marking point images and the 2D coordinates of the respective unique marking points in the respective marking point images comprises: Determine the camera coordinate system corresponding to each motion capture camera, and establish the conversion relationship between each camera coordinate system and the world coordinate system; Calculate the matching degree of the same coincident marking point in each two target marking point images, and take the marking point with a matching degree greater than the matching degree threshold as the target marking point; Based on the target marking point and the unique marking point, determining basic feature data corresponding to the current human body; Based on the basic feature data and the transformation relationship between each camera coordinate system and the world coordinate system, the three-dimensional coordinates corresponding to each marking point are calculated; Based on the three-dimensional coordinates corresponding to each marking point, a 3D human body model corresponding to the current human body is generated.

5. The method for labeling joint data based on multiple perspectives according to claim 4, characterized in that: The three-dimensional coordinates corresponding to each marking point are calculated based on the basic feature data and the conversion relationship between each camera coordinate system and the world coordinate system. The calculation process of the three-dimensional coordinates corresponding to each marking point includes: Based on the basic feature data, determining the corresponding relationship between the marker points in the corresponding marker point image, and determining the intrinsic parameter matrix and the extrinsic parameter matrix of the reference camera corresponding to the marker points; Obtaining the image coordinates of the marking point in the corresponding second human body image; Based on the conversion relationship of the reference camera device corresponding to the target human feature and the internal parameter matrix, the image coordinates of the marked point in the corresponding second human image are converted into the target ray equation corresponding to the marked point; Based on the respective transformation relationships between the camera coordinate systems and the world coordinate system, the extrinsic parameter matrix and the target ray equation, the three-dimensional coordinates corresponding to the marking point are calculated.

6. The method for labeling joint data based on multiple perspectives according to any one of claims 1 to 5, characterized in that: Mapping the human 3D skeleton data onto a corresponding human image to obtain 2D skeleton data includes: Based on the human 3D skeleton data, the vector angle between adjacent marking points is calculated to obtain the marking point angle information; The Zhang Zhengyou calibration method is used to calculate the intrinsic parameter matrix of each motion capture camera, and based on the intrinsic parameter matrix of each motion capture camera, the focal length value corresponding to each motion capture camera is determined; Calculate the marker distance between each marker point and the corresponding motion capture camera; Based on the focal length value and marker distance corresponding to each marker point, the 3D skeleton data of the human body is converted into 2D skeleton data using the perspective projection formula; A scaling factor of the 2D skeleton data is determined, and based on the scaling factor, the 2D skeleton data is mapped onto a corresponding human body image.

7. The method for labeling joint data based on multiple perspectives according to claim 6, characterized in that: The step of determining the scaling factor of the 2D skeleton data comprises: Obtain a human body image corresponding to each 2D skeleton data, and obtain a resolution corresponding to each of the human body images; Determine the coordinate range corresponding to each 2D skeleton data, and calculate the offset corresponding to each 2D skeleton data based on the coordinate range corresponding to each 2D skeleton data; Each offset is optimized based on the least square method to obtain an optimized offset; Based on the optimized offset, a scaling factor of the 2D skeleton data is determined.

8. A joint data labeling device based on multi-view, characterized in that: include: A receiving module, used to receive a plurality of marked point images and a plurality of human body images corresponding to the current moment, wherein the marked point images are images of marked points on a human body acquired by a motion capture camera, and the human body images are images of the current human body acquired by a reference camera, wherein one camera corresponds to one human body image at any moment, and the human body image includes complete current human body image information, and the current human body corresponds to a plurality of marked points; A recognition module, used for recognizing the 2D coordinates of each marker point in each of the marker point images, and constructing a 3D human body model of the current human body based on the 2D coordinates of each marker point; A mapping module, used to construct human 3D skeleton data corresponding to the current moment based on the 3D human body model, and map the human 3D skeleton data to the corresponding human body image to obtain 2D skeleton data, wherein the human 3D skeleton data is used to characterize the three-dimensional spatial position of each marker point; The determination module is used to determine the 2D joint positions of the current human body based on the 2D skeleton data corresponding to the multiple marker point images.

9. An electronic device, characterized in that: The electronic device includes: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the multi-view based joint data labeling method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the multi-view joint data labeling method according to any one of claims 1 to 7.

Images