AI human shape recognition method and system based on multi-view vision fusion

Abstract

The invention provides an AI human shape recognition method and system based on multi-view vision fusion, and the method comprises the steps: collecting a multi-view image sequence containing a target human body, and recognizing an initial human body key point in a three-dimensional space corresponding to each single-view image; performing non-rigid alignment on a preset parameterized human body model and an initial human body surface point cloud generated by the multi-view image sequence by taking the initial human body key point as space guidance to obtain an initialized three-dimensional human body model; projecting the initialized three-dimensional human body model to the imaging plane of each visual angle, generating a projection contour map of the target human body under each visual angle, and determining the multi-visual-angle contour difference of the target human body according to the projection contour map under each visual angle and the corresponding actual human body contour; and taking the multi-view contour difference as an optimization target, driving the parameterized human body model to carry out iterative optimization, and further outputting an effective three-dimensional model of a target human body. According to the technical scheme provided by the invention, the real body contour of the target human body can be accurately restored under multi-view stereoscopic vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human recognition, more particularly, the present application relates to an AI human recognition method and system based on multi-view vision fusion. BACKGROUND

[0002] Human recognition is a core task in the field of computer vision, aiming to locate and identify human bodies from images or videos. Its technical development has evolved from traditional methods relying on hand-designed features to today's dominance of deep learning. Modern human recognition is mainly based on models such as convolutional neural networks, which are trained on large amounts of data to enable machines to accurately detect human positions and further analyze poses, actions, and even identities. Human recognition is a key foundation for many applications such as intelligent video surveillance, human-computer interaction, autonomous driving, and virtual reality. By enabling machines to recognize human behavior, it has greatly promoted the practical process of artificial intelligence.

[0003] In existing human recognition, human recognition is centered on deep learning. First, the input image or video frame is preprocessed to reduce environmental interference. Second, convolutional neural networks are used to automatically learn from low-level edges and contours to high-level semantic features. Then, based on semantic features, an anchor mechanism is used to generate human candidate regions, and a classifier is used to determine whether the human candidate region is a human. If so, the human recognition result is output. However, in three-dimensional human reconstruction based on multi-view stereo vision, the initial point cloud data generated by stereo matching is sparse, noisy, and has holes. Existing three-dimensional reconstruction methods rely solely on flawed point cloud data for model-driven reconstruction, which can cause the reconstruction process to collapse to the average shape of the model prior in areas where point cloud data is missing, making it impossible to accurately restore the true shape of the target human body. Therefore, how to accurately restore the true shape of the target human body under multi-view stereo vision has become a difficult problem in the industry. SUMMARY

[0004] The present application provides an AI human recognition method and system based on multi-view vision fusion, which can accurately restore the true shape of the target human body under multi-view stereo vision.

[0005] In a first aspect, the present application provides an AI human recognition method based on multi-view vision fusion, comprising the following steps: Collecting a multi-view image sequence containing a target human body, and determining a human region mask corresponding to each single-view image in the multi-view image sequence; Identifying human pose key points corresponding to each single-view image based on all human region masks, and converting the human pose key points into initial human key points in three-dimensional space; stereo matching is performed on each single-view image in the multi-view image sequence to generate an initial human body surface point cloud, a preset parametric human body model is non-rigidly aligned with the initial human body surface point cloud with the initial human body key points as spatial guidance, and an initialized three-dimensional human body model is obtained; The initialized three-dimensional human body model is projected to an imaging plane of each view to generate a projection contour map of the target human body under each view, the projection contour map under each view is compared and analyzed with an actual human body contour extracted from a corresponding single-view image, and a multi-view contour difference of the target human body is obtained. The posture parameters and shape parameters of the parametric human body model are iteratively optimized with the multi-view contour difference as an optimization target until the iterative optimization converges, and an effective three-dimensional model of the target human body is output.

[0006] In some embodiments, determining the human body region mask corresponding to each single-view image in the multi-view image sequence specifically includes: The single-view images in the multi-view image sequence are preprocessed to obtain preprocessed single-view images; Human body detection is performed based on the preprocessed single-view images to obtain human body candidate regions; Semantic segmentation is performed on the human body candidate regions to obtain the human body region mask corresponding to each single-view image.

[0007] In some embodiments, identifying the human body posture key points corresponding to each single-view image based on all human body region masks and converting the human body posture key points into initial human body key points in a three-dimensional space specifically includes: Based on all human body region masks, a human body region of interest is extracted from each single-view image; A human body posture detection model is used to process the human body region of interest of each single-view image to obtain two-dimensional human body posture key points corresponding to each single-view image; The two-dimensional human body posture key points corresponding to each single-view image are triangulated according to internal and external parameters of the multi-view camera to obtain initial human body key points in a three-dimensional space.

[0008] In some embodiments, stereo matching is performed on each single-view image in the multi-view image sequence to generate an initial human body surface point cloud specifically includes: Based on the calibration parameters of the multi-view camera, epipolar correction is performed on each single-view image in the multi-view image sequence to obtain corrected single-view images; Stereo matching calculation is performed on the corrected single-view images to obtain a parallax map corresponding to each view; Based on the intrinsic parameters of the multi-view camera and each disparity map, the depth map corresponding to each viewpoint is calculated. Based on the depth maps corresponding to each viewpoint and the extrinsic parameters of the multi-view camera, an initial 3D point cloud is generated. The initial 3D point cloud is obtained by filtering the human body region mask corresponding to each single-view image.

[0009] In some embodiments, using the initial human body key points as spatial guides, a preset parametric human body model is non-rigidly aligned with the initial human body surface point cloud to obtain an initialized 3D human body model. This specifically includes: Based on the initial human body key points, the preset parametric human body model is coarsely aligned in pose to obtain a coarsely aligned parametric human body model. Calculate the spatial distance between the surface points of the coarsely aligned parameterized human body model and the initial human body surface point cloud to obtain the distance error field; Using the distance error field as a constraint, the coarsely aligned parameterized human body model is subjected to non-rigid deformation optimization to obtain an initialized three-dimensional human body model.

[0010] In some embodiments, projecting the initialized 3D human body model onto the imaging plane of each viewpoint to generate a projected contour map of the target human body from each viewpoint specifically includes: Obtain the camera intrinsic and extrinsic parameters corresponding to each viewpoint to obtain the imaging parameters of each viewpoint; The surface mesh of the initialized three-dimensional human body model is extracted to obtain the model surface mesh; Based on the imaging parameters of each viewpoint, the model surface mesh is projected onto the imaging plane of the corresponding viewpoint to obtain the two-dimensional projection mesh under each viewpoint. Edge contours are extracted from the two-dimensional projection meshes from various viewpoints to generate projection contour maps of the target human body from each viewpoint.

[0011] In some embodiments, a multi-view image sequence containing the target human body is acquired synchronously using a multi-view camera.

[0012] Secondly, this application provides an AI human recognition system based on multi-view vision fusion, comprising: The acquisition module is used to acquire a multi-view image sequence containing a target human body and determine the human body region mask corresponding to each single-view image in the multi-view image sequence. The processing module is used to identify the human pose key points corresponding to each single-view image based on all human body region masks, and convert the human pose key points into initial human body key points in three-dimensional space. The processing module is also used to perform stereo matching on each single-view image in the multi-view image sequence to generate an initial human body surface point cloud. Using the initial human body key points as spatial guides, the preset parameterized human body model is non-rigidly aligned with the initial human body surface point cloud to obtain an initialized three-dimensional human body model. The processing module is also used to project the initialized three-dimensional human body model onto the imaging plane of each viewpoint, generate the projected contour map of the target human body under each viewpoint, compare and analyze the projected contour map under each viewpoint with the actual human body contour extracted from the corresponding single-view image, and then obtain the multi-view contour difference of the target human body. The execution module is used to drive the pose parameters and shape parameters of the parameterized human body model to perform iterative optimization with the multi-view contour difference as the optimization target until the iterative optimization converges, and then outputs an effective three-dimensional model of the target human body.

[0013] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory storing code, the processor being configured to acquire the code and execute the above-described AI human figure recognition method based on multi-view vision fusion.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned AI human figure recognition method based on multi-view vision fusion.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The AI ​​human figure recognition method and system based on multi-view vision fusion provided in this application first acquires a multi-view image sequence containing the target human body and determines the human body region mask corresponding to each single-view image in the multi-view image sequence. Second, based on all human body region masks, the human body posture key points corresponding to each single-view image are identified, and the human body posture key points are converted into initial human body key points in three-dimensional space. Further, stereo matching is performed on each single-view image in the multi-view image sequence to generate an initial human body surface point cloud. Using the initial human body key points as spatial guidance, a preset parameterized human body model is non-rigidly aligned with the initial human body surface point cloud to obtain an initialized three-dimensional human body model. Then, the initialized three-dimensional human body model is projected onto the imaging plane of each viewpoint to generate a projected contour map of the target human body under each viewpoint. The projected contour map under each viewpoint is compared and analyzed with the actual human body contour extracted from the corresponding single-view image to obtain the multi-view contour difference of the target human body. Finally, using the multi-view contour difference as the optimization target, the posture parameters and shape parameters of the parameterized human body model are driven to iteratively optimize until the iterative optimization converges, and an effective three-dimensional model of the target human body is output.

[0016] Therefore, this application can accurately reconstruct the true shape and contour of the target human body under multi-view stereo vision. First, by acquiring multi-view image sequences and determining the human body region mask, the integrity of the observation data and multi-angle support can be ensured through multi-view coverage, so as to accurately isolate the human target from the complex background and reduce the interference of environmental noise on subsequent target human body recognition and 3D reconstruction. Second, the key points of human body posture are identified and converted into 3D initial human body key points, which can provide spatial anchor points that conform to human body structure for the subsequent alignment of parametric models. Furthermore, by generating the initial human body surface point cloud through stereo matching and completing the non-rigid alignment of the parametric model, both discrete geometric data representing the human body surface morphology are obtained, and the initial 3D human body model can quickly fit the overall posture and surface contour of the target human body, effectively avoiding getting trapped in a local optimum due to excessive initial deviation during the subsequent optimization process. Then, for each single view in the multi-view image sequence The image undergoes stereo matching to generate an initial human body surface point cloud. Using the initial human body key points as spatial guides, a pre-set parametric human body model is non-rigidly aligned with the initial human body surface point cloud to obtain an initialized 3D human body model. This method can comprehensively capture contour deviation details from different perspectives, avoiding the recognition deviation of the target human body's true shape contour caused by sparse initial point cloud data, severe noise interference, and the presence of holes during stereo matching. Finally, the parametric human body model is iteratively optimized based on multi-view contour differences, and an effective 3D model of the target human body is output. This ensures that the posture and shape of the parametric human body model continuously conform to the real human body features. The initial point cloud data generated by stereo matching is sparse, subject to severe noise interference, and contains holes, ultimately resulting in a high-precision 3D human body model with natural posture and accurate shape. In summary, the technical solution provided in this application can accurately restore the true shape contour of the target human body under multi-view stereo vision. Attached Figure Description

[0017] Figure 1 This is an exemplary flowchart of an AI human figure recognition method based on multi-view vision fusion, according to some embodiments of this application; Figure 2 This is an exemplary flowchart illustrating the determination of initial human body key points according to some embodiments of this application; Figure 3 This is a schematic diagram of the structure of an AI human recognition system based on multi-view vision fusion, according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of a computer device implementing an AI human figure recognition method based on multi-view vision fusion, according to some embodiments of this application. Detailed Implementation

[0018] To better understand the technical solution of this application, the technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] refer to Figure 1 This figure is an exemplary flowchart of an AI human recognition method based on multi-view vision fusion according to some embodiments of this application. The figure mainly includes the following steps: In step S101, a multi-view image sequence containing the target human body is acquired, and the human body region mask corresponding to each single-view image in the multi-view image sequence is determined.

[0020] In practice, a multi-view image sequence containing the target human body is simultaneously acquired by a multi-view camera. The multi-view camera is a variable-focus multi-view camera. Specifically, the multi-view camera exposes and captures the target human body in the scene at the same moment to ensure that the acquired images are single-view images of the same instant of movement from multiple perspectives. Then, the single-view images acquired by the camera from each perspective are aligned on the timeline to form a multi-view image sequence that is strictly corresponding in both time and space.

[0021] It should be noted that, in this application, a multi-view image sequence refers to a collection of single-view images from multiple perspectives. The single-view image refers to an image containing a target human body captured by a multi-view camera from different perspectives. By determining the multi-view image sequence, a data foundation can be provided for subsequent stereoscopic vision 3D reconstruction and multi-view information fusion processing.

[0022] In some embodiments, determining the human body region mask corresponding to each single-view image in the multi-view image sequence is achieved by the following steps: The single-view images in the multi-view image sequence are preprocessed to obtain preprocessed single-view images. Human detection is performed based on the preprocessed single-view image to obtain human candidate regions; Semantic segmentation is performed on the candidate human body regions to obtain the human body region mask corresponding to each single-view image.

[0023] In specific implementation, firstly, Gaussian filtering is applied to the single-view images in the multi-view image sequence, followed by size normalization at a fixed resolution to obtain preprocessed single-view images. These preprocessed single-view images refer to those that have undergone denoising and size normalization. Next, the histogram of directional gradients (HCG) features of the preprocessed single-view images are extracted and input into a pre-trained support vector machine (SVM) classifier. The SVM classifier discriminates the HCG features to locate the rectangular boundary containing the human body, obtaining candidate human body regions. These candidate regions are rectangular spatial regions that can completely enclose the target human body. Finally, a fully convolutional network is used to perform pixel-level semantic segmentation on the candidate human body regions. The encoder module of the fully convolutional network extracts the deep feature maps of the candidate human body regions, and the decoder module restores the spatial resolution of the deep feature maps. Each pixel is then classified as either human or background, and a binary image retaining only human pixels is output, resulting in the human body region mask for each single-view image.

[0024] It should be noted that the human body region mask in this application refers to a binary image used to separate the human body from the background. Specifically, the human body region mask is an image obtained by setting the human body region pixels as foreground values ​​and the background region pixels as background values ​​after pixel-level classification processing. By determining the human body region mask, the target human body can be accurately separated from the complex background, eliminating interference noise from non-human body regions such as irrelevant objects and lighting changes in the environment. This provides a focused and effective human body region for subsequent human pose key point recognition, avoids misjudging the background region by the detection model, and narrows the key point search range to improve detection efficiency and positioning accuracy.

[0025] In step S102, the human pose key points corresponding to each single-view image are identified based on all human body region masks, and the human pose key points are converted into initial human body key points in three-dimensional space.

[0026] In some embodiments, reference Figure 2 As shown in the figure, this is an exemplary flowchart of determining initial human body key points according to some embodiments of this application. In this embodiment, the human body pose key points corresponding to each single-view image are identified based on all human body region masks, and the human body pose key points are converted into initial human body key points in three-dimensional space. This can be achieved by the following steps: In step S1021, the human region of interest is extracted from each single-view image based on all human region masks; In step S1022, a human pose detection model is used to process the human region of interest in each single-view image to obtain the two-dimensional human pose key points corresponding to each single-view image. In step S1023, triangulation is performed on the two-dimensional human pose key points corresponding to each single-view image based on the intrinsic and extrinsic parameters of the multi-view camera to obtain the initial human key points in three-dimensional space.

[0027] In specific implementation, firstly, based on all human body region masks, each single-view image is fused with its corresponding human body region mask through pixel-by-pixel logical AND operation, retaining only the pixel information marked as human body regions in the human body region mask. Then, the image region that minimizes the human body is cropped according to the boundary coordinates of the human body region mask, obtaining the human body region of interest (ROI). The ROI refers to the image region that contains only the target human body and removes background interference. Then, the hourglass network in the human pose detection model is used to process the ROI of each single-view image. The multi-scale encoder module of the hourglass network progressively extracts the deep spatial feature map of the human body within the ROI. The decoder module performs scale restoration and detail enhancement on the deep spatial feature map, outputting the response heatmap corresponding to each joint position. The pixel coordinates with the highest response value in the response heatmap are used as the positions of the corresponding joints, and the positions of the corresponding joints are used as two-dimensional human posture key points. This yields the two-dimensional human posture key points corresponding to each single-view image. The two-dimensional human posture key points refer to the set of two-dimensional coordinates on the single-view image plane that can represent the positions of key human joints. The two-dimensional human posture key points include, but are not limited to, the human head, shoulders, elbows, hips, and knees. Finally, based on the intrinsic and extrinsic parameters obtained from the multi-view camera calibration, the direct linear transformation method is used to establish the multi-view coordinate correspondence of the two-dimensional human posture key points corresponding to each single-view image. The overdetermined equation system is solved by the least squares method to calculate the spatial coordinates of each two-dimensional human posture key point in the three-dimensional world coordinate system, thereby obtaining the initial human key points in three-dimensional space.

[0028] It should be noted that, in this application, the initial human body key points in three-dimensional space refer to coordinates that can initially characterize the three-dimensional position distribution of human body joints. The determination of the initial human body key points in three-dimensional space can provide a core spatial structural benchmark for subsequent three-dimensional human body reconstruction. It characterizes the three-dimensional position distribution of key human body joints and provides clear spatial anchor points for the coarse alignment of the pose of the preset parametric human body model, ensuring that the model can quickly fit the overall pose and spatial position of the human body and avoid model offset during the initialization stage. At the same time, the initial human body key points, as human body structural constraints, can assist in screening effective point clouds that match the human body structure in the initial human body surface point cloud processing, eliminate noisy point clouds that deviate from the spatial distribution of human body joints, and improve the effectiveness of point cloud data.

[0029] In step S103, stereo matching is performed on each single-view image in the multi-view image sequence to generate an initial human body surface point cloud. Using the initial human body key points as spatial guides, the preset parameterized human body model is non-rigidly aligned with the initial human body surface point cloud to obtain an initialized three-dimensional human body model.

[0030] In some embodiments, stereo matching is performed on each single-view image in the multi-view image sequence to generate an initial human body surface point cloud, which is achieved by the following steps: Based on the calibration parameters of the multi-view camera, epipolar correction is performed on each single-view image in the multi-view image sequence to obtain the corrected single-view image. Stereo matching calculations are performed on the corrected single-view image to obtain disparity maps corresponding to each viewpoint. Based on the intrinsic parameters of the multi-view camera and each disparity map, the depth map corresponding to each viewpoint is calculated. Based on the depth maps corresponding to each viewpoint and the extrinsic parameters of the multi-view camera, an initial 3D point cloud is generated. The initial 3D point cloud is obtained by filtering the human body region mask corresponding to each single-view image.

[0031] In specific implementation, firstly, based on the intrinsic and extrinsic parameters obtained from the pre-calibrated multi-view cameras, the existing epipolar geometry principle is used to construct the spatial mapping relationship between different viewpoints (i.e., firstly, the essential matrix representing the spatial positional relationship between the two cameras is calculated using the extrinsic parameters, and then the essential matrix is ​​converted into a fundamental matrix describing the epipolar constraints on the image plane by combining the intrinsic parameters of the two cameras; based on this fundamental matrix, the epipolar line corresponding to any pixel in the first-view image can be determined in the second-view image, where the epipolar line is the straight line distributed by all pixels that may form corresponding points with the pixel in the second viewpoint; conversely, the epipolar line corresponding to a pixel in the second-view image in the first-view image can be determined in the reverse direction; through this bidirectional epipolar constraint, different... The one-to-one correspondence between "pixels and epipolar lines" in the viewpoint images forms a mapping relationship that can characterize the spatial association between the two viewpoint images. Horizontal and vertical deviations between viewpoints are eliminated through image rotation and translation transformations, ensuring that corresponding points in the human body region fall on the same horizontal scan line, resulting in a corrected single-viewpoint image. This corrected single-viewpoint image refers to a single-viewpoint image that has undergone geometric transformation processing to ensure that the epipolar lines of corresponding points are parallel to the horizontal axis of the image. Next, a semi-global block matching algorithm is used to perform stereo matching calculations on the corrected single-viewpoint image. This involves traversing each pixel of the reference viewpoint image (i.e., the base image selected from the multi-viewpoint corrected single-viewpoint images) using a rectangular window of fixed size, and calculating... The grayscale similarity between pixels within a rectangular window and corresponding windows of the same size on the scan line of the target view image (i.e., another corrected single-view image paired with the reference view image) is used as the matching cost. A dynamic programming algorithm is used to accumulate the matching cost along multiple scan directions to obtain the global cost. The pixel correspondence relationship that minimizes the global cost is then solved to obtain the disparity map for each viewpoint. The disparity map represents the offset of pixel coordinates between corresponding points in the reference and target viewpoints. Further, based on the intrinsic parameters of the multi-view camera (including focal length and principal point coordinates) and each disparity map, the geometric relationship between disparity and depth derived from the pinhole camera imaging model is used to convert the disparity value of each pixel into the corresponding depth value. The three-dimensional spatial depth information is used to obtain depth maps corresponding to each viewpoint. The depth map refers to an image in which each pixel value represents the vertical distance from the corresponding scene point to the camera's imaging plane. Then, based on the depth maps corresponding to each viewpoint and the extrinsic parameters (including rotation matrix and translation vector) of the multi-view camera, the two-dimensional coordinates of each pixel in the depth map are converted into three-dimensional coordinate points in the three-dimensional world coordinate system through matrix operations of pixel coordinates, depth values ​​and extrinsic parameters. The three-dimensional coordinate points of all viewpoints are then collected to form a three-dimensional point set containing the human body and background in the scene, resulting in an initial three-dimensional point cloud. The initial three-dimensional point cloud refers to a discrete point set containing the three-dimensional coordinates of all spatial points in the scene, converted from the multi-view depth information.Finally, the pixel range of the target human body in the image is determined by the human body region mask corresponding to each single-view image. Combined with camera intrinsic parameters, this pixel range is mapped to a constrained region in three-dimensional space. Points located within this constrained region in the initial three-dimensional point cloud are selected, and redundant point clouds corresponding to the background region are removed to obtain the initial human body surface point cloud.

[0032] It should be noted that the initial human body surface point cloud in this application refers to a discrete set of points containing only the three-dimensional coordinates of the target human body surface and its neighboring regions. The determination of the initial human body surface point cloud provides direct geometric data support for the three-dimensional human body reconstruction. By filtering out redundant background point clouds, it retains discrete three-dimensional spatial points that correspond only to the human body region. This not only eliminates environmental interference to ensure data purity, but also fully represents the spatial morphology and contour details of the human body surface. It provides a precise geometric reference for the non-rigid alignment of the pre-parametric human body model, enabling the model to quickly fit the actual surface shape of the human body. Ultimately, it lays a reliable geometric data foundation for the generation of a high-precision and effective three-dimensional model.

[0033] In some embodiments, the initial human body key points are used as spatial guides to non-rigidly align a preset parametric human body model with the initial human body surface point cloud to obtain an initialized three-dimensional human body model. This is achieved through the following steps: Based on the initial human body key points, the preset parametric human body model is coarsely aligned in pose to obtain a coarsely aligned parametric human body model. Calculate the spatial distance between the surface points of the coarsely aligned parameterized human body model and the initial human body surface point cloud to obtain the distance error field; Using the distance error field as a constraint, the coarsely aligned parameterized human body model is subjected to non-rigid deformation optimization to obtain an initialized three-dimensional human body model.

[0034] In specific implementation, firstly, based on the initial human body key points, a preset parametric human body model (e.g., a skinned multi-person linear model based on skeleton binding) is selected. The rigid body transformation matrix is ​​solved using the least squares method (specifically determined by minimizing the sum of squared Euclidean distances between two sets of corresponding points after rotation and translation transformations using the least squares method, which will not be elaborated here). Then, the translation and rotation parameters of the parametric human body model are adjusted using the rigid body transformation matrix so that the standard joint points of the parametric human body model initially coincide with the positions of the initial human body key points, resulting in a coarsely aligned parametric human body model. The coarsely aligned parametric human body model refers to the parametric human body model whose standard joint points are roughly matched with the initial human body key points after rigid body transformation adjustment. Then, the point-to-surface Euclidean distance calculation method is used to traverse the initial human body surface. For each point in the point cloud, the nearest point on the coarsely aligned parameterized human body model surface is found, and the spatial distance between the two points is calculated. All the obtained spatial distances are used to form a continuous error distribution field according to the spatial distribution of the point cloud, thus obtaining the distance error field. The distance error field refers to the set of deviation values ​​that characterize the spatial distribution between the initial human body surface point cloud and the coarsely aligned parameterized human body model surface. Finally, using the distance error field as a constraint, a model deformation function is constructed using a thin plate spline interpolation algorithm (i.e., Gaussian radial basis function is selected as the model deformation function). Deformation weights are allocated according to the deviation ratio of each position in the distance error field, and the mesh vertices of the coarsely aligned parameterized human body model are adjusted point by point through the model deformation function to minimize the deviation at each position, thus obtaining the initialized three-dimensional human body model.

[0035] It should be noted that the initial three-dimensional human body model in this application refers to a three-dimensional mesh model whose posture and surface morphology initially fit the target human body. The determination of the initial three-dimensional human body model provides an accurate and physiologically consistent basic template for subsequent multi-view contour difference calculation and model iterative optimization. It initially restores the overall posture and surface morphology of the target human body by aligning and fitting with the initial human body key points and the initial human body surface point cloud in the early stage, avoiding convergence difficulties or getting trapped in local optima due to excessive deviation of the initial model in the subsequent optimization process. This provides a clear direction for subsequent parameter optimization to generate a high-precision effective three-dimensional model of the target human body.

[0036] In step S104, the initialized three-dimensional human body model is projected onto the imaging plane of each viewpoint to generate the projected contour map of the target human body under each viewpoint. The projected contour map under each viewpoint is compared and analyzed with the actual human body contour extracted from the corresponding single-view image to obtain the multi-view contour difference of the target human body.

[0037] In some embodiments, the initial three-dimensional human body model is projected onto the imaging plane of each viewpoint to generate the projected contour map of the target human body under each viewpoint, which is achieved by the following steps: Obtain the camera intrinsic and extrinsic parameters corresponding to each viewpoint to obtain the imaging parameters of each viewpoint; The surface mesh of the initialized three-dimensional human body model is extracted to obtain the model surface mesh; Based on the imaging parameters of each viewpoint, the model surface mesh is projected onto the imaging plane of the corresponding viewpoint to obtain the two-dimensional projection mesh under each viewpoint. Edge contours are extracted from the two-dimensional projection meshes from various viewpoints to generate projection contour maps of the target human body from each viewpoint.

[0038] In specific implementation, firstly, the camera intrinsic parameters (including focal length and principal point coordinates) and extrinsic parameters (including the camera's rotation matrix and translation vector relative to the world coordinate system) for each viewpoint are obtained. These parameters are pre-obtained and stored through camera calibration experiments (e.g., calibration using a checkerboard calibration method, which will not be elaborated here). The camera intrinsic and extrinsic parameters are then combined to form the imaging parameters for the corresponding viewpoint. These imaging parameters refer to the complete set of parameters characterizing the camera's imaging geometry and spatial pose for 3D point-to-2D image projection calculations. Secondly, all vertex coordinates and triangular facet connections between vertices of the initialized 3D human model's surface morphology are extracted. The mesh topology of the model surface is preserved to obtain the model surface mesh. This mesh, composed of discrete 3D vertices and triangular facets, characterizes the geometry of the initialized 3D human model's surface. The model surface mesh is first constructed using a grid structure. Then, based on the imaging parameters of each viewpoint, the known pinhole camera imaging principle is used to transform the coordinates of each three-dimensional vertex of the model surface mesh from the world coordinate system to the camera coordinate system using extrinsic parameters. Then, the three-dimensional coordinates in the camera coordinate system are projected onto the two-dimensional imaging plane using intrinsic parameters. This allows the calculation of the two-dimensional image coordinates corresponding to each vertex in the model surface mesh, while preserving the connection relationships of the triangular facets in the model surface mesh. This yields the two-dimensional projection mesh for each viewpoint. The two-dimensional projection mesh refers to the two-dimensional vertex and triangular facet combination structure formed on the imaging plane of the corresponding viewpoint after the model surface mesh is transformed by camera projection. Finally, the Canny edge detection operator in image processing is used to extract the edges of the two-dimensional projection mesh for each viewpoint, and the extracted edge points are connected to form a continuous closed contour, generating the projection contour map of the target human body for each viewpoint.

[0039] It should be noted that, in this application, the projected contour map refers to a two-dimensional image representing the visible contour of the initialized 3D human body model from a corresponding viewpoint. Determining the projected contour map allows the establishment of a correlation between the 3D human body model and the actual two-dimensional multi-view images. The 3D model cannot be directly compared with the 2D image in terms of morphology, but the projected contour map can map the initialized 3D human body model into a 2D contour according to the camera parameters of each viewpoint, making it the same dimension as the actual human body contour in the single-view image. This provides a comparable basis for subsequent morphological consistency verification. By comparing with the actual human body contour, the morphological differences between the model and the real human body can be accurately located, avoiding distorted morphology that contradicts actual observation. Ultimately, this ensures that the appearance of the optimized 3D model is consistent with the real human body from each viewpoint, improving the reconstruction accuracy and reliability.

[0040] In some embodiments, the following steps are used to compare and analyze the projected contour maps from each viewpoint with the actual human body contours extracted from the corresponding single-view images to obtain the multi-view contour differences of the target human body: The actual human body contour is extracted based on the human body region mask of the corresponding single-view image, and the actual human body contour under each view is obtained. The projected contour map under each viewpoint is spatially aligned with the actual human body contour under the corresponding viewpoint to obtain the aligned projected contour and the actual contour. Calculate the pixel-level difference between the aligned projected contour and the actual contour to obtain the single-view contour difference under each viewpoint. The multi-view contour differences of the target human body are determined based on the single-view contour differences under various perspectives.

[0041] In specific implementation, firstly, based on the human body region mask of the corresponding single-view image, a gray-level gradient operator is used to locate the boundary pixels between the human body region and the background in the human body region mask image. Then, the discrete boundary pixels are integrated into continuous closed lines to obtain the actual human body contour under each view. The actual human body contour refers to the continuous closed lines representing the edge shape of the target human body extracted from the single-view image. Secondly, for the projected contour map under each view and the actual human body contour of the corresponding view, feature points (such as contour inflection points and curvature extrema points) on the contour are extracted respectively. The Euclidean distance between the feature points is calculated to match paired feature points, and the optimal translation and rotation parameters solved by the least squares method are used to adjust the spatial deviation of the two sets of feature points, so as to minimize the spatial deviation of the two sets of feature points, thereby obtaining the aligned projection. The aligned projected contour and the actual contour refer to two sets of contours whose feature points are matched after coordinate adjustment and can be directly compared at the pixel level. Then, the aligned projected contour and the actual contour are converted into binary images of the same resolution (i.e., the contour lines are set as foreground pixels and the remaining areas are set as background pixels). By traversing and comparing the overlap of foreground pixels pixel by pixel, the number of pixels in the non-overlapping areas is counted as the single-view contour difference under the corresponding viewpoint, and thus the single-view contour difference under each viewpoint is obtained. The single-view contour difference refers to the index that characterizes the deviation between the projected contour and the actual human body contour shape under a single viewpoint. Finally, the multi-view contour difference of the target human body is determined based on the single-view contour difference under each viewpoint, that is, the single-view contour differences under each viewpoint are combined into the multi-view contour difference of the target human body.

[0042] It should be noted that, in this application, multi-view contour difference refers to the data set characterizing the deviation between the projected shape of the 3D model and the shape of the real human body from various viewpoints. The determination of multi-view contour difference provides a comprehensive and accurate core basis for the iterative optimization of the 3D human body model. It integrates the quantitative deviation between the projected contour and the actual human body contour under different observation angles, which not only avoids the one-sidedness caused by occlusion and imaging angle limitations of a single viewpoint, but also fully captures the shape deviation of the target human body in various directions (such as the deviation of the torso width in the front and the deviation of the limb thickness in the side). This provides a data basis for the posture adjustment and shape correction of the parametric human body model. At the same time, it constrains the model optimization process to always conform to the multi-view shape characteristics of the real human body, avoiding the situation of local optimization or contradiction with some viewpoint observations. Ultimately, it ensures that the iterated 3D model can highly match the contour of the real human body in all viewpoints.

[0043] In step S105, the pose and shape parameters of the parameterized human body model are iteratively optimized using the multi-view contour difference as the optimization target until the iterative optimization converges, at which point an effective three-dimensional model of the target human body is output.

[0044] In some embodiments, the pose and shape parameters of the parameterized human body model are iteratively optimized using the multi-view contour differences as the optimization objective until the iterative optimization converges. The output of an effective 3D model of the target human body is achieved through the following steps: Contour difference loss function is constructed based on the multi-view contour differences; Using the contour difference loss function as a constraint, the pose parameters and shape parameters of the parameterized human body model are updated by gradient descent to obtain the updated parameterized human body model. The difference between the updated contour difference loss function value and the loss function value of the previous iteration is calculated to obtain the amount of loss change. Verify whether the change in loss meets the preset convergence threshold. If it does not, return to the parameter update step. If it does, output the updated parameterized human body model as the effective 3D model of the target human body.

[0045] In specific implementation, firstly, based on the single-view contour differences in the multi-view contour differences, a contour difference loss function is constructed using a weighted summation method. The weight value corresponding to each single-view contour difference can be set between 0 and 1 depending on the occlusion level of the viewpoint; less occlusion results in a larger weight value, which is not limited here. The contour difference loss function refers to a mathematical expression aimed at minimizing the multi-view contour deviation. Secondly, using the contour difference loss function as a constraint, the stochastic gradient descent algorithm is used to calculate the partial derivatives of the contour difference loss function with respect to the posture parameters (such as joint rotation angles) and shape parameters (such as body proportion coefficients) of the parameterized human model, obtaining the gradient direction of each parameter. The parameter values ​​are then adjusted along the negative gradient direction by a preset step size to reduce the loss function value, thereby obtaining the updated parameterized human model. The updated parametric human model refers to the parametric human model that has undergone one round of parameter adjustment. Then, the absolute difference between the contour difference loss function value corresponding to the updated parametric human model and the loss function value of the previous iteration is calculated to obtain the loss change. The loss change refers to the absolute value of the difference between the loss function values ​​of two adjacent iterations, which is used to measure the effect of parameter update. Finally, a convergence threshold is preset according to actual needs (i.e., the degree of improvement in the representation deviation has met the accuracy requirements), and it is checked whether the loss change is less than the convergence threshold. If it is not satisfied, the parameter update step is returned to continue iterating. If it is satisfied, it indicates that the model parameters have become stable and the multi-view contour deviation has reached an acceptable range. The updated parametric human model at this time is then used as the effective 3D model of the target human body and output.

[0046] It should be noted that the effective three-dimensional model of the target human body in this application refers to a high-precision three-dimensional human body model that fits the real human body.

[0047] Furthermore, in another aspect of this application, in some embodiments, this application provides an AI human recognition system based on multi-view vision fusion, referencing... Figure 3 The figure is a schematic diagram of the structure of an AI human recognition system based on multi-view vision fusion according to some embodiments of this application. The AI ​​human recognition system based on multi-view vision fusion includes: a data acquisition module 201, a processing module 202, and an execution module 203, which are described below: Acquisition module 201, in this application, is mainly used to acquire a multi-view image sequence containing a target human body, and to determine the human body region mask corresponding to each single-view image in the multi-view image sequence. Processing module 202, in this application, is mainly used to identify the human pose key points corresponding to each single-view image based on all human body region masks, and convert the human pose key points into initial human body key points in three-dimensional space. The processing module 202 is also used to perform stereo matching on each single-view image in the multi-view image sequence to generate an initial human body surface point cloud. Using the initial human body key points as spatial guides, the preset parameterized human body model is non-rigidly aligned with the initial human body surface point cloud to obtain an initialized three-dimensional human body model. In addition, the processing module 202 is also used to project the initialized three-dimensional human body model onto the imaging plane of each viewpoint, generate the projected contour map of the target human body under each viewpoint, compare and analyze the projected contour map under each viewpoint with the actual human body contour extracted from the corresponding single-view image, and then obtain the multi-view contour difference of the target human body. The execution module 203 in this application is mainly used to drive the posture parameters and shape parameters of the parameterized human body model to perform iterative optimization with the multi-view contour difference as the optimization target until the iterative optimization converges, and output the effective three-dimensional model of the target human body.

[0048] In addition, this application also provides a computer device, the computer device including a memory and a processor, the memory storing code, the processor being configured to acquire the code and execute the above-described AI human figure recognition method based on multi-view vision fusion.

[0049] In some embodiments, reference Figure 4 The figure is a schematic diagram of the structure of a computer device implementing an AI human figure recognition method based on multi-view vision fusion, according to some embodiments of this application. The AI ​​human figure recognition method based on multi-view vision fusion in the above embodiments can... Figure 4 The computer device shown is used to implement this, and the computer device includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.

[0050] The processor 301 can be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more devices used to control the execution of the AI ​​human recognition method based on multi-view vision fusion in this application.

[0051] The communication bus 302 can be used to transmit information between the aforementioned components.

[0052] The memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via the communication bus 302. The memory 303 may also be integrated with the processor 301.

[0053] The memory 303 stores program code for executing the scheme of this application, and its execution is controlled by the processor 301. The processor 301 executes the program code stored in the memory 303. The program code may include one or more software modules. In the above embodiments, the determination of the AI ​​human recognition method based on multi-view vision fusion can be achieved by the processor 301 and one or more software modules in the program code in the memory 303.

[0054] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0055] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0056] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0057] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described AI human figure recognition method based on multi-view vision fusion.

[0058] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0059] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An AI human shape recognition method based on multi-view vision fusion, characterized in that, The method comprises the following steps: Collecting a multi-view image sequence containing a target human body, and determining a human body region mask corresponding to each single-view image in the multi-view image sequence; Identifying human posture key points corresponding to each single-view image based on all human body region masks, and converting the human posture key points into initial human key points in a three-dimensional space; Performing stereo matching on each single-view image in the multi-view image sequence to generate initial human surface point clouds, performing non-rigid alignment between a preset parameterized human model and the initial human surface point clouds with the initial human key points as spatial guides, and then obtaining an initialized three-dimensional human model; Projecting the initialized three-dimensional human model to an imaging plane of each view to generate a projection contour map of the target human body under each view, comparing and analyzing the projection contour map under each view with an actual human contour extracted from a corresponding single-view image, and then obtaining multi-view contour differences of the target human body; Taking the multi-view contour differences as an optimization target, driving iterative optimization of posture parameters and shape parameters of the parameterized human model until the iterative optimization converges, and then outputting an effective three-dimensional model of the target human body.

2. The method of claim 1, wherein, Determining a human body region mask corresponding to each single-view image in the multi-view image sequence specifically comprises: Preprocessing single-view images in the multi-view image sequence to obtain preprocessed single-view images; Performing human body detection based on the preprocessed single-view images to obtain human candidate regions; Performing semantic segmentation on the human candidate regions to obtain a human body region mask corresponding to each single-view image.

3. The method of claim 1, wherein, Identifying human posture key points corresponding to each single-view image based on all human body region masks, and converting the human posture key points into initial human key points in a three-dimensional space specifically comprises: Extracting a human region of interest from each single-view image based on all human body region masks; Processing the human region of interest of each single-view image using a human posture detection model to obtain two-dimensional human posture key points corresponding to each single-view image; Triangulating the two-dimensional human posture key points corresponding to each single-view image according to internal and external parameters of a multi-view camera to obtain initial human key points in a three-dimensional space.

4. The method of claim 1, wherein, Performing stereo matching on each single-view image in the multi-view image sequence to generate initial human surface point clouds specifically comprises: Performing epipolar correction on each single-view image in the multi-view image sequence based on calibration parameters of a multi-view camera to obtain corrected single-view images; Performing stereo matching calculation on the corrected single-view images to obtain a disparity map corresponding to each view; Calculating a depth map corresponding to each view according to internal parameters of the multi-view camera and the disparity maps; Generating initial three-dimensional point clouds based on the depth map corresponding to each view and external parameters of the multi-view camera; Performing regional filtering on the initial three-dimensional point clouds through human body region masks corresponding to each single-view image to obtain initial human surface point clouds.

5. The method of claim 1, wherein, The initial human key points are used as spatial guidance to perform non-rigid alignment between the preset parametric human model and the initial human surface point cloud, and an initialized three-dimensional human model is obtained, specifically including: Based on the initial human key points, posture coarse alignment is performed on the preset parametric human model to obtain a coarsely aligned parametric human model; The spatial distance between the surface points of the coarsely aligned parametric human model and the initial human surface point cloud is calculated to obtain a distance error field; The coarsely aligned parametric human model is deformed and optimized non-rigidly under the constraint of the distance error field to obtain the initialized three-dimensional human model.

6. The method of claim 1, wherein, The initialized three-dimensional human model is projected onto the imaging plane of each view to generate a projection contour map of the target human under each view, specifically including: Camera intrinsic and extrinsic parameters corresponding to each view are obtained to obtain imaging parameters of each view; Surface mesh extraction is performed on the initialized three-dimensional human model to obtain a model surface mesh; The model surface mesh is projected onto the imaging plane of the corresponding view according to the imaging parameters of each view to obtain a two-dimensional projection mesh under each view; Edge contour extraction is performed on the two-dimensional projection mesh under each view to generate a projection contour map of the target human under each view.

7. The method of claim 1, wherein, A multi-view camera is used to synchronously capture a multi-view image sequence containing a target human.

8. An AI human shape recognition system based on multi-view vision fusion, configured to perform the AI human shape recognition method based on multi-view vision fusion according to any one of claims 1 to 7. The system includes: An acquisition module is configured to acquire a multi-view image sequence containing a target human and determine a human region mask corresponding to each single-view image in the multi-view image sequence; A processing module is configured to identify human posture key points corresponding to each single-view image based on all human region masks and convert the human posture key points into initial human key points in a three-dimensional space; The processing module is further configured to perform stereo matching on each single-view image in the multi-view image sequence to generate an initial human surface point cloud, perform non-rigid alignment between a preset parametric human model and the initial human surface point cloud using the initial human key points as spatial guidance, and obtain an initialized three-dimensional human model; The processing module is further configured to project the initialized three-dimensional human model onto the imaging plane of each view to generate a projection contour map of the target human under each view, compare and analyze the projection contour map under each view with an actual human contour extracted from the corresponding single-view image, and obtain a multi-view contour difference of the target human; An execution module is configured to drive iterative optimization of posture parameters and shape parameters of the parametric human model using the multi-view contour difference as an optimization target until the iterative optimization converges, and output an effective three-dimensional model of the target human.

9. A computer device, comprising: The computer device includes a memory and a processor, the memory stores code, and the processor is configured to acquire the code and execute the AI human shape recognition method based on multi-view visual fusion according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the AI human shape recognition method based on multi-view visual fusion according to any one of claims 1 to 7.

Images