A large batch multi-view target image segmentation system based on a segmentation large model
Patent Information
- Application Number
- CN202410661696.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-27
- Publication Date
- 2026-08-18
- Estimated Expiration
- 2044-05-27
AI Technical Summary
[0002]Alexander Kirillov等人通过为基础大模型Segment Anything Model(SAM)构造合适的prompt，能够计算多种不同类别目标对象的掩码(mask)，并具备对新样本计算掩码的zero-shot能力，但是缺少定向分割指定目标的能力，尤其是在人体分割方面，无法实现大批量分割人体图片的任务
[0023]1、本发明提供一种基于分割大模型的大批量多视角目标图像分割系统，通过提示生成模块根据人体关键点信息得到一组新的提示点坐标信息或者其他提示形式的信息，用处理得到的提示信息作为基于分割大模型的图像分割模块的输入，获取最终分割掩码结果；也就是说，本发明生成了额外的提示信息，用于提示指引分割大模型去分割用户指定的目标，自动化程度高，不需要额外提供绿幕或者空背景等预设信息、不需要固定照相机、不需要单一视角，即可进行高效的、精准的、鲁棒性的目标分割掩码计算，即使在动态背景下、动态目标的复杂情况下也能够实现大批量的目标图像分割掩码。
Smart Images
Figure CN118447253B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence, segmentation of large models and 3D reconstruction of models, and particularly relates to a large-batch multi-view target image segmentation system based on large segmentation models. Background Technology
[0002] Alexander Kirillov et al., by constructing a suitable prompt for the basic large-scale model Segment Anything Model (SAM), were able to compute masks for various object categories and possess zero-shot mask computation capabilities for new samples. However, they lacked the ability to segment specific targets, especially in human segmentation, failing to achieve large-scale segmentation of human images. Matthew Loper et al., by combining the large-scale segmentation model Segment Anything with the object detection algorithm YOLO v8, achieved segmentation of objects with specified labels; however, due to different training methods, there was a significant gap between the segmentation results of SAM and the results of the object detection algorithm, failing to provide accurate results. Furthermore, some researchers improved the quality and temporal consistency of human video matting by utilizing temporal information in videos and enhanced the robustness of the model through new training strategies; however, this method is only suitable for matting videos with fixed backgrounds and performs poorly for human portrait matting with changing backgrounds. Other researchers achieved good results in human portrait matting tasks in natural scenes by introducing background images and additional information, realizing a simpler and more efficient matting process; however, this method requires empty background frames and is unsuitable for matting with moving shots and multiple viewpoints.
[0003] It is evident that large segmentation models trained with massive amounts of data segment all targets in the input image data, including not only human bodies but also various other objects, or image regions of similar objects with obvious or indistinct boundaries. They lack the ability to segment specific targets and cannot provide accurate results. Therefore, there is an urgent need for a new target image segmentation method that uses additional input to guide the large segmentation model to segment specific targets. Summary of the Invention
[0004] To address the aforementioned issues, this invention provides a large-scale multi-view target image segmentation system based on a large segmentation model. Based on a pre-trained large segmentation model and a target pose estimation algorithm, it achieves efficient, accurate, and robust target image segmentation calculations, suitable for 3D and 4D reconstruction of targets.
[0005] A large-scale multi-view target image segmentation system based on a large segmentation model includes an image acquisition module, a key point acquisition module, a prompt generation module, an image segmentation module, and a 3D reconstruction module.
[0006] The image acquisition module is used to acquire target images of the target to be segmented from different perspectives, at any time, and in different poses.
[0007] The key point acquisition module is used to extract key point information from each target image to distinguish different parts of the target to be segmented.
[0008] The prompt generation module is used to generate prompt information corresponding to each target image based on the key point information corresponding to each target image. The prompt information includes the coordinates of positive prompt points, the coordinates of negative prompt points, the bounding box information that completely encloses the target to be segmented, and / or the text description information of the target to be segmented. Positive prompt points are pixels with a probability of belonging to the target to be segmented greater than a set value, and negative prompt points are pixels with a probability of belonging to the target to be segmented not greater than a set value.
[0009] The image segmentation module is used to output the segmentation mask result of the target to be segmented in each target image according to the prompt information corresponding to each target image;
[0010] The 3D reconstruction module is used to perform 3D or 4D modeling of the target to be segmented based on the segmentation mask results corresponding to each target image.
[0011] Furthermore, it also includes a 3D pose extraction module and an adaptive correction module;
[0012] The three-dimensional pose extraction module is used to predict the pose of the target to be segmented in each target image based on the key point information of the target to be segmented in each target image, and to reconstruct a three-dimensional deformable mesh of the target based on each pose.
[0013] The adaptive correction module adaptively corrects each segmentation mask result output by the image segmentation module based on the target three-dimensional deformable mesh with set posture information. Then, the corrected segmentation mask result is input into the three-dimensional reconstruction module for three-dimensional or four-dimensional modeling. The adaptive correction includes target composition integrity correction, target contour boundary correction, and target part subordination relationship correction.
[0014] Furthermore, the key point acquisition module also includes a key point joint extraction module and a key point mapping module;
[0015] The key point joint extraction module uses a neural network based on deep learning and supervised training to predict the initial key points of the target to be segmented in each target image;
[0016] The key point mapping module clusters the initial key points of the target to be segmented in each target image, and retains only the key points in each category that are closest to the cluster center, thus obtaining the final key point information in each target image.
[0017] Furthermore, the bounding box that completely encloses the target to be segmented is a hollow rectangular box without rotation angle;
[0018] The bounding box information includes the x-coordinates and y-coordinates of the top-left corner of the rectangle, the width of the rectangle, the height of the rectangle, or the x-coordinates and y-coordinates of the top-left corner and the bottom-right corner of the rectangle.
[0019] Furthermore, the textual description information of the target to be segmented includes the target's category, appearance features, and location information.
[0020] Furthermore, the target to be segmented is a human body, and the key point information used to distinguish different parts of the human body includes: left eye coordinates, right eye coordinates, left ear coordinates, right ear coordinates, nose coordinates, left shoulder coordinates, right shoulder coordinates, left elbow coordinates, right elbow coordinates, left wrist coordinates, right wrist coordinates, left palm coordinates, right palm coordinates, left hip coordinates, right hip coordinates, left knee coordinates, right knee coordinates, left ankle coordinates, right ankle coordinates, left foot coordinates, or right foot coordinates.
[0021] Furthermore, the image segmentation module processes the prompt information corresponding to each target image based on the basic large model Segment Anything Model, and then outputs the segmentation mask result of the target to be segmented in each target image.
[0022] Beneficial effects:
[0023] 1. This invention provides a large-scale, multi-view target image segmentation system based on a large segmentation model. A prompt generation module obtains a new set of prompt point coordinates or other prompt forms based on human keypoint information. This processed prompt information is used as input to an image segmentation module based on a large segmentation model to obtain the final segmentation mask result. In other words, this invention generates additional prompt information to guide the large segmentation model in segmenting user-specified targets. It boasts a high degree of automation, requiring no additional green screen or empty background presets, no fixed camera, and no single viewpoint. It enables efficient, accurate, and robust target segmentation mask calculation, even under complex conditions with dynamic backgrounds and dynamic targets.
[0024] 2. This invention provides a large-scale multi-view target image segmentation system based on a large segmentation model. The system uses text descriptions of the category, appearance features, and location information of the target to be segmented in a single image as prompts for the large segmentation model. This helps the large segmentation model identify the objects that need to be segmented and improves the accuracy of target segmentation.
[0025] 3. This invention provides a large-scale multi-view target image segmentation system based on a large segmentation model. The image segmentation module is a segmentation model pre-trained with a large amount of data. It has strong generalization ability and is highly adaptable to human bodies with different facial features and different clothing. The output results are stable and reliable, which can improve the model reconstruction quality of multi-view human body 3D reconstruction. Attached Figure Description
[0026] Figure 1 This invention provides a structural diagram of a large-batch, multi-view target image segmentation system based on a large segmentation model.
[0027] Figure 2 A schematic diagram of the multi-view human image acquisition module provided by the present invention;
[0028] Figure 3 A schematic diagram illustrating the key point acquisition module of the present invention acquiring key points of a human target;
[0029] Figure 4 A schematic diagram illustrating the key point simplification method provided by this invention;
[0030] Figure 5 A simplified diagram illustrating the key points provided by this invention;
[0031] Figure 6 A flowchart of a large-batch, multi-view target image segmentation system based on a large segmentation model provided by the present invention;
[0032] Figure 7 These are schematic diagrams of the collected images and intermediate results provided by this invention.
[0033] Figure 8 This invention provides a device diagram for a large-scale, multi-view target image segmentation system based on a large segmentation model. Detailed Implementation
[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0035] like Figure 1 As shown, a large-scale multi-view target image segmentation system based on a large segmentation model includes an image acquisition module, a key point acquisition module, a prompt generation module, an image segmentation module, a 3D pose extraction module, an adaptive correction module, and a 3D reconstruction module.
[0036] The image acquisition module is used to acquire target images of the target to be segmented from different perspectives, at any time, and in different poses.
[0037] It should be noted that, as Figure 2 As shown, the image acquisition module includes multiple camera hardware devices, fixed frames in different positions, a controllable camera shutter system, an illumination system, and a data transmission and storage system.
[0038] The key point acquisition module is used to extract key point information from each target image to distinguish different parts of the target to be segmented; wherein, the key point acquisition module further includes a key point joint extraction module and a key point mapping module; the key point joint extraction module uses a neural network based on deep learning and supervised training to predict the initial key points of the target to be segmented in each target image; the key point mapping module clusters the initial key points of the target to be segmented in each target image, and retains only the key points in each category that are closest to the cluster center, to obtain the final key point information in each target image.
[0039] It should be noted that, as Figure 3 As shown, when the target to be segmented is a human body, the key point information used to distinguish different parts of the human body includes, but is not limited to: left eye coordinates, right eye coordinates, left ear coordinates, right ear coordinates, nose coordinates, left shoulder coordinates, right shoulder coordinates, left elbow coordinates, right elbow coordinates, left wrist coordinates, right wrist coordinates, left palm coordinates, right palm coordinates, left hip coordinates, right hip coordinates, left knee coordinates, right knee coordinates, left ankle coordinates, right ankle coordinates, left foot coordinates, and right foot coordinates. The coordinates can be two-dimensional (X, Y) or three-dimensional (X, Y, Z).
[0040] Simultaneously, the cameras obtain their own position and orientation information, including but not limited to parameters such as optical center and focal length, through automatic or manual calibration procedures, thereby achieving the sharing of the same world coordinate system. Human body key point information in all views can be corrected through global optimization, avoiding information loss or misidentification of key point positions due to factors such as part occlusion, perspective errors, or insufficient texture.
[0041] In other words, the keypoint acquisition module of this invention is responsible for mapping the original target keypoint group coordinates to a new keypoint group coordinate system, adapting it to the input requirements of the large-scale image segmentation module. Since the target keypoint coordinates can be used directly or indirectly as cue information for the large-scale image segmentation module, this invention typically uses a mapping algorithm to map the original 2D keypoint group A to keypoint group B, which serves as the cue point input in the final large-scale image segmentation input type. For example... Figure 5As shown, when using keypoints indirectly, the keypoint coordinates can be simplified through methods such as keypoint clustering, keypoint position correction, and adjustment by combining pose information. These intermediate processing and transformations yield the final cue information suitable for segmenting large models. Mapping algorithms include, but are not limited to, K-Means clustering, K-Means++ clustering, Mean-shift clustering, DBSCAN clustering (density-based noise application space clustering), CFSFDP clustering (density-based clustering), Gaussian Mixture Model (GMM) clustering (probability-based clustering, using Gaussian Mixture Model Expectation-Maximization clustering), agglomerative hierarchical clustering, SOM (model-based clustering), spectral clustering, and other clustering algorithms.
[0042] The prompt generation module is used to generate prompt information for each target image based on the key point information corresponding to each target image. The prompt information includes the coordinates of positive prompt points, the coordinates of negative prompt points, the bounding box information completely enclosing the target to be segmented, and / or the text description information of the target to be segmented. A positive prompt point is a pixel with a probability greater than a set value that belongs to the target to be segmented. A negative prompt point is a pixel with a probability greater than a set value that belongs to the target to be segmented. A pixel (point) is a pixel whose probability of belonging to the target to be segmented is no greater than a set value. Typically, the prediction results of the large-scale segmentation model's image segmentation module will include positive cue points as much as possible while avoiding negative cue points as much as possible. The textual description information of the target to be segmented includes the target's category, appearance features, and location information to help the large-scale segmentation model identify the object to be segmented. The bounding box that completely encloses the target to be segmented is a hollow rectangle without rotation. The bounding box information can be described in two ways: the first is the x-coordinates and y-coordinates of the top-left corner of the rectangle, the width of the rectangle, and the height of the rectangle; the second is the x-coordinates and y-coordinates of the top-left corner and the bottom-right corner of the rectangle. Alternatively, in a combination of these two forms, the data can be replaced with normalized values relative to the image width / height. The bounding box is input into the image segmentation module of the large-scale segmentation model, and the image segmentation module will output the segmentation mask of the target within the bounding box.
[0043] It should be noted that there are several different types of prompts, which can be used independently or in combination to achieve a mutual correction effect.
[0044] The three-dimensional pose extraction module is used to process the key point information of the target to be segmented in each target image based on the linear parameter model of human pose, predict the pose of the target to be segmented in each target image, and restore a three-dimensional deformable mesh of the target based on each pose, which is used to assist in the correction of the final human body segmentation mask result and the integrity verification of the human body.
[0045] The image segmentation module processes the prompt information corresponding to each target image based on the basic large model Segment Anything Model, and then outputs the segmentation mask result of the target to be segmented in each target image.
[0046] The adaptive correction module adaptively corrects each segmentation mask result output by the image segmentation module based on the target three-dimensional deformable mesh with set posture information. The adaptive correction includes target composition integrity correction, target contour boundary correction, and target part subordination relationship correction.
[0047] It should be noted that some of the segmentation mask results output by the image segmentation module may include parts of the background that are similar to human skin color / shape, while others may over-segment the target, with the segmentation mask only containing a local part of the target. Therefore, it is necessary to correct the segmentation mask results output by the image segmentation module to finally obtain the correct target segmentation mask information.
[0048] The 3D reconstruction module is used to perform 3D or 4D modeling of the target to be segmented based on the corrected segmentation mask results corresponding to each target image. Specifically, this invention performs 3D or 4D modeling of the target to be segmented based on traditional stereo matching modeling, neural radiation field modeling, and 3D Gaussian modeling.
[0049] Therefore, it is evident that the input to the image segmentation module based on the large-scale segmentation model must include image data. Since the large-scale segmentation model trained on massive amounts of data will segment all targets in the input image data, including not only human bodies but also various other objects, or image regions of similar objects with obvious or indistinct boundaries, this invention requires additional input to guide the large-scale segmentation model in performing its computational tasks. This additional input is a prompt message for the target to be segmented, used to guide the large-scale segmentation model to segment the user-specified target.
[0050] The output of the large segmentation model is a segmentation mask result of the user-specified target contained in the input information. This segmentation mask result is usually in single-channel 8-bit data format. It can be a binary hard segmentation (the entire data map only has two values representing True and False, without other intermediate values, and there is a clear and steep boundary between the segmented target and the background, with no transition region) or a grayscale soft segmentation (the boundary between the target and the background has a smooth transition region). In this invention, the user-specified target is the human body region in the image. If there are more than two human body instances in the image, the segmentation can be single-person segmentation or multi-person segmentation. The user-specified information is a crucial basis for identifying which human body is the target to be segmented.
[0051] In summary, the workflow of the large-batch, multi-view target image segmentation system based on a large segmentation model provided by this invention can be summarized as follows:
[0052] like Figure 6 As shown, firstly, a multi-camera system is used to acquire images and videos of the human body from multiple perspectives; then, the image and video data are stored in a database for later use; a pose estimation model is used to extract key points of each human target in each frame of the image (if there are multiple human targets in the image, pose estimation is performed on all human targets to calculate the coordinate information of the human pose key points); at the same time, a linear 3D parametric model is used to simulate the human body's pose and shape, and the images are superimposed and rendered onto the original images with the same scale and camera position parameters; then, through the human key point mapping module, the coordinates of the original 2D human key point group are mapped into a new coordinate point group suitable for segmentation of large model inputs. Based on the mapped keypoint group information of each human target, the cue generation module generates effective positive and negative cue point coordinates and other types of cue for that target. Using a pre-trained segmentation model, based on the positive and negative cue point information as input (optionally, other types of cue information such as target bounding boxes, text descriptions, etc.), a segmentation mask for each human target in each frame is generated. Based on the predicted human 3D deformable mesh with specific pose information, the human segmentation mask results are adaptively corrected, including corrections to human body composition integrity, body contour boundaries, and the hierarchical relationship of human body parts. Finally, the correct segmentation mask results for all images are obtained. These results are stored in the database with the same file structure corresponding to the original data file numbers. For multi-view 3D reconstruction applications, using the acquired original images and the processed human target segmentation mask results, a 3D model of the human body can be generated based on algorithms such as stereo matching, neural radiation fields, or 3D Gaussian sputtering.
[0053] Figure 7This display shows the original images and key intermediate results acquired in a multi-view human image acquisition system. It is divided into four rows. The first row shows the original acquisition results, the output of the multi-view human image acquisition module. The right side contains images captured by 16 cameras with different viewpoints, evenly distributed around the human body as the center. In a real image acquisition system, the number of cameras can be arbitrary, including single cameras. The distribution of cameras can be uniform or non-uniform. Cameras can be arranged in circles / spheres, hemispheres, etc. The second row shows the human pose estimation results from the parametric model, the result of the human 3D pose estimation module. The third row shows the extracted human keypoints as input prompts for the image segmentation module based on a large segmentation model, the output of the prompt generation module. The fourth row shows the final segmentation mask result, the final output of the system. All segmentation results correspond one-to-one with the camera viewpoint and human pose of the input image.
[0054] Furthermore, the equipment involved in this invention, such as Figure 8 As shown, it mainly includes multi-view image acquisition equipment, human-computer interaction equipment, storage equipment, computing equipment, and display equipment.
[0055] In summary, this invention proposes a large-batch, multi-view target image segmentation system based on a large segmentation model. Based on a pre-trained large segmentation model and a human pose estimation algorithm, it achieves efficient, accurate, and robust human image segmentation calculation, and is particularly suitable for three-dimensional and four-dimensional reconstruction systems of the human body.
[0056] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.
Claims
1. A large-batch, multi-view target image segmentation system based on a large segmentation model, characterized in that, It includes an image acquisition module, a key point acquisition module, a prompt generation module, an image segmentation module, a 3D reconstruction module, a 3D pose extraction module, and an adaptive correction module; The image acquisition module is used to acquire target images of the target to be segmented from different perspectives, at any time, and in different poses. The key point acquisition module is used to extract key point information from each target image to distinguish different parts of the target to be segmented. The prompt generation module is used to generate prompt information corresponding to each target image based on the key point information corresponding to each target image. The prompt information includes the coordinates of positive prompt points, the coordinates of negative prompt points, the bounding box information that completely encloses the target to be segmented, and / or the text description information of the target to be segmented. Positive prompt points are pixels with a probability of belonging to the target to be segmented greater than a set value, and negative prompt points are pixels with a probability of belonging to the target to be segmented not greater than a set value. The image segmentation module is used to output the segmentation mask result of the target to be segmented in each target image according to the prompt information corresponding to each target image; The 3D reconstruction module is used to perform 3D or 4D modeling of the target to be segmented based on the segmentation mask results corresponding to each target image; The three-dimensional pose extraction module is used to predict the pose of the target to be segmented in each target image based on the key point information of the target to be segmented in each target image, and to reconstruct a three-dimensional deformable mesh of the target based on each pose. The adaptive correction module adaptively corrects each segmentation mask result output by the image segmentation module based on the target three-dimensional deformable mesh with set posture information. Then, the corrected segmentation mask result is input into the three-dimensional reconstruction module for three-dimensional or four-dimensional modeling. The adaptive correction includes target composition integrity correction, target contour boundary correction, and target part subordination relationship correction.
2. The large-batch, multi-view target image segmentation system based on a large segmentation model as described in claim 1, characterized in that, The key point acquisition module also includes a key point joint extraction module and a key point mapping module; The key point joint extraction module uses a neural network based on deep learning and supervised training to predict the initial key points of the target to be segmented in each target image; The key point mapping module clusters the initial key points of the target to be segmented in each target image, and retains only the key points in each category that are closest to the cluster center, thus obtaining the final key point information in each target image.
3. The large-batch, multi-view target image segmentation system based on a large segmentation model as described in claim 1, characterized in that, The bounding box that completely encloses the target to be segmented is a hollow rectangle without any rotation angle; The bounding box information includes the x-coordinates and y-coordinates of the top-left corner of the rectangle, the width of the rectangle, the height of the rectangle, or the x-coordinates and y-coordinates of the top-left corner and the bottom-right corner of the rectangle.
4. The large-batch multi-view target image segmentation system based on a large segmentation model as described in claim 1, characterized in that, The textual description information of the target to be segmented includes the target's category, appearance features, and location information.
5. The large-batch, multi-view target image segmentation system based on a large segmentation model as described in claim 1, characterized in that, The target to be segmented is the human body, and the key point information used to distinguish different parts of the human body includes: left eye coordinates, right eye coordinates, left ear coordinates, right ear coordinates, nose coordinates, left shoulder coordinates, right shoulder coordinates, left elbow coordinates, right elbow coordinates, left wrist coordinates, right wrist coordinates, left palm coordinates, right palm coordinates, left hip coordinates, right hip coordinates, left knee coordinates, right knee coordinates, left ankle coordinates, right ankle coordinates, left foot coordinates, or right foot coordinates.
6. A large-scale multi-view target image segmentation system based on a large segmentation model as described in any one of claims 1 to 5, characterized in that, The image segmentation module processes the prompt information corresponding to each target image based on the basic large model Segment Anything Model, and then outputs the segmentation mask result of the target to be segmented in each target image.