Three-dimensional target positioning method, system and related equipment

Abstract

The invention discloses a three-dimensional target positioning method and system based on two-dimensional segmentation and three-dimensional reconstruction and related equipment, and belongs to the technical field of three-dimensional localization, and the method comprises the steps: carrying out the key target segmentation of a visual perception video based on a two-dimensional image segmentation model, obtaining a video frame, and carrying out the RGB label labeling of the video; inputting the marked video frame into the three-dimensional reconstruction model to obtain a three-dimensional point cloud model with a mask; screening point clouds on the basis of label numerical values; analyzing the screened point clouds by using a clustering algorithm to generate a clustering point set; and extracting three-dimensional space center points of different clustering point sets as three-dimensional positions of the segmented target, and obtaining a spatial position of the target relative to the shooting point. According to the method, the robot can be better supported to complete identification, three-dimensional segmentation and positioning of a key target by using the instantly acquired two-dimensional optical image information, and the robot is supported to complete tasks such as navigation and operation by using related information.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional positioning technology, and in particular to a three-dimensional target positioning method, system and related equipment. Background Art

[0002] Interactive 3D segmentation has long been a topic of considerable interest, with potential applications in a variety of fields, including scene manipulation, automatic annotation, and VR. For example, in the field of robotic scene manipulation, describing the spatial position and control methods of operable / movable components can help robots plan reliable operational behaviors. While direct segmentation of 3D point clouds can also identify and locate 3D targets, building a model library for complex and diverse operating components is undoubtedly a challenging task. Furthermore, the application of 3D point cloud segmentation models requires a pre-built 3D scene model, which may impose certain limitations on the practical application of related technologies. Summary of the Invention

[0003] In order to solve the above problems, the present invention proposes a three-dimensional target positioning method, system and related equipment. The method proposed in the present invention is based on three-dimensional target positioning of two-dimensional segmentation and three-dimensional reconstruction, which can better support the robot to use the two-dimensional optical image information collected in real time to complete the identification, three-dimensional segmentation and positioning of key targets, and support the robot to use relevant information to complete navigation, operation and other tasks.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: In a first aspect, the present invention provides a three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction, comprising: Acquire a visual perception video, segment the visual perception video into key objects based on a two-dimensional image segmentation model to obtain video frames, and annotate the video with RGB labels; Input the annotated video frames into the 3D reconstruction model to obtain a 3D point cloud model with a mask; Based on the label value, the point cloud related to the target object in the masked 3D point cloud model is filtered; Use clustering algorithm to analyze the screened point cloud and generate a set of clustered points; The three-dimensional spatial center points of different cluster point sets are extracted as the three-dimensional position of the segmented target. The three-dimensional position coordinates are transformed according to the set line of sight starting point, line of sight yaw angle, and pitch angle to obtain the spatial position of the target relative to the shooting point.

[0005] As a further improvement of the present invention, the step of obtaining the visually perceived video at least includes: Obtain the robot's visual perception video of the operation site and operation target.

[0006] As a further improvement of the application, the robot at least includes a live-line working robot or an inspection robot.

[0007] As a further improvement of the application, the key target segmentation of the visual perception video based on the two-dimensional image segmentation model comprises: The visual perception video is decomposed into video frames, and the first frame is sent to the Grounding DINO model, and the key target is framed according to the artificially provided semantic vocabulary; The framing result is sent to the two-dimensional image segmentation model for processing, and the key target of all video frames is segmented through time sequence propagation, and the segmented video frame is obtained.

[0008] As a further improvement of the application, the RGB label annotation of the video comprises: The annotation is completed by covering the RGB label on the segmented video frame output by the two-dimensional image segmentation model.

[0009] As a further improvement of the application, the (R, G, B) values of the RGB label are selected as (255, 0, 0), (0, 255, 0) and (0, 0, 255), and the RGB label colors of similar targets are unified, and the RGB label colors of different types of targets are distinguished.

[0010] As a further improvement of the application, the video frame with the label is input into a three-dimensional reconstruction model to obtain a three-dimensional point cloud model with a mask, comprising: The video frame with the label is sent to a neural radiation field or a Gaussian splashing three-dimensional reconstruction model, and the three-dimensional reconstruction model restores the three-dimensional shape of the labeled target object according to the shape of the RGB label and the information of the image around the label, and obtains a three-dimensional point cloud model with a mask.

[0011] As a further improvement of the application, the three-dimensional point cloud model with the mask is filtered according to the label value, and the point cloud related to the target object is filtered, comprising: The (R, G, B) values of the RGB label are used as a basis to filter the point cloud related to the target object.

[0012] As a further improvement of the application, the clustering algorithm comprises k-means or DBSCAN.

[0013] In a second aspect, the application provides a three-dimensional target positioning system based on two-dimensional segmentation and three-dimensional reconstruction, comprising: A video segmentation module is configured to obtain a visual perception video, segment key targets of the visual perception video based on a two-dimensional image segmentation model to obtain video frames, and annotate the video with an RGB label. a three-dimensional reconstruction module, configured to input the labeled video frame into a three-dimensional reconstruction model to obtain a three-dimensional point cloud model with a mask; a key point screening module, configured to screen the point cloud related to the target object in the three-dimensional point cloud model with the mask according to the label value; a clustering identification module, configured to analyze the screened point cloud by using a clustering algorithm to generate a clustering point set; a three-dimensional positioning module, configured to extract a three-dimensional space center point of different clustering point sets as a three-dimensional position of a segmentation target, and perform coordinate conversion on the three-dimensional position according to a set visual line origin and a visual line yaw angle and a pitch angle to obtain a spatial position of the target relative to a shooting point.

[0014] In the video segmentation module, the visual perception video is at least obtained by: obtaining a visual perception video of a robot for an operation site and an operation target.

[0015] As a further improvement of the present application, the robot at least includes a live-line working robot or a patrol robot.

[0016] The video segmentation module is specifically configured to: divide the visual perception video into a video frame combination, and send a first frame into a Grounding DINO model to frame the key target according to a human-provided semantic vocabulary; send the framing result into a two-dimensional image segmentation model for processing, and segment the key target of all video frames through time sequence propagation to obtain segmented video frames.

[0017] The video segmentation module is further configured to: complete the labeling by covering the RGB label on the segmented video frames output by the two-dimensional image segmentation model.

[0018] The (R, G, B) values of the RGB label are selected as (255, 0, 0), (0, 255, 0), and (0, 0, 255), and the RGB label colors of the same type of target are unified, and the RGB label colors of different types of targets are distinguished.

[0019] The three-dimensional reconstruction module is specifically configured to: send the labeled video frame into a neural radiance field or a Gaussian splatting type three-dimensional reconstruction model, and the three-dimensional reconstruction model restores the three-dimensional morphology of the labeled target object according to the shape of the RGB label and the information of the image around the label to obtain a three-dimensional point cloud model with a mask.

[0020] The key point screening module is specifically configured to: screen the point cloud related to the target object according to the (R, G, B) values of the RGB label.

[0021] The clustering algorithm comprises k-means or DBSCAN.

[0022] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction.

[0023] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction.

[0024] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises computer instructions, and the computer instructions instruct a computer to execute the three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction.

[0025] The present application has the following beneficial effects compared with the prior art: The method can realize three-dimensional segmentation and three-dimensional positioning through three-dimensional reconstruction of the two-dimensional segmentation result by using the segmentation capability of the Segment Anything model and the Gaussian sputtering three-dimensional reconstruction technology. Compared with a direct three-dimensional segmentation model, the algorithm proposed in the method can complete fine tuning only by using two-dimensional image data, and can directly derive the required three-dimensional segmentation result from the two-dimensional optical image obtained on site, thereby directly meeting the actual application requirements, saving sensor cost and three-dimensional model construction time cost. The method proposed in the present application can quickly realize three-dimensional reconstruction of the work site, identification of key targets, three-dimensional segmentation and three-dimensional positioning by using optical perception information of a robot. Meanwhile, the method has the advantages of simple deployment and application, convenient updating and fine tuning, and good scalability required in actual application of related technologies. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0027] Figure 1 Flowchart of the three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction; Figure 2A three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction is provided; Figure 3 A three-dimensional target positioning system based on two-dimensional segmentation and three-dimensional reconstruction is provided. Figure 4 An electronic device schematic diagram is provided. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0029] In the description of the present application, the words such as setting, installing, connecting, etc. should be understood broadly, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0030] In order to describe the technical solutions of the present application in detail, the content of the present application is described in detail below in combination with the accompanying drawings: The first object of the present application is to provide a three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction, comprising: S01, acquiring a visual perception video, performing key target segmentation on the visual perception video based on a two-dimensional image segmentation model to obtain a video frame, and performing RGB label annotation on the video; In the above scheme, the key target in the video frame is extracted by using the two-dimensional image segmentation model, and each target is assigned a unique RGB label. The label distinguishes different targets by color coding, and provides semantic association for subsequent three-dimensional point cloud screening. The complex scene is simplified to target level processing, reducing the computational complexity of three-dimensional reconstruction.

[0031] S02, inputting the annotated video frame into a three-dimensional reconstruction model to obtain a three-dimensional point cloud model with a mask; In the above scheme, the three-dimensional point cloud is reconstructed based on multi-view geometry or deep learning, and the mask is generated in combination with the two-dimensional label. The mask retains only the target-related point cloud through semantic consistency constraint. The semantic information of two-dimensional segmentation is extended to three-dimensional space to realize target-level point cloud separation.

[0032] S03, screening the point cloud related to the target object in the three-dimensional point cloud model with the mask based on the label value; S04, using a clustering algorithm to analyze the screened point cloud to generate a clustered point set; In the above scheme, the point cloud is filtered according to the RGB label, and the point cloud that is continuous in space and consistent in label is aggregated as an independent target through a clustering algorithm. The problem of target occlusion or dispersion can be solved, and the robustness of multi-target positioning is improved.

[0033] S05, extracting the three-dimensional space center point of different clustered point sets as the three-dimensional position of the segmentation target, and performing coordinate conversion of the three-dimensional position according to the set visual line origin and visual line yaw angle, pitch angle to obtain the spatial position of the target relative to the shooting point.

[0034] In the above scheme, the three-dimensional center of the clustered point cloud is calculated as the target position, and the point cloud coordinates are converted to the shooting point reference system in combination with the camera internal and external parameters (visual line origin, yaw angle, pitch angle). The precise positioning of the target in the real three-dimensional space is realized, which is suitable for robot navigation, AR / VR and other scenes.

[0035] The present application reduces background interference through label screening, focuses on the target area in three-dimensional reconstruction, and reduces the influence of noise. Even if the target is partially occluded or dispersed, the clustering algorithm can still identify the complete target. Two-dimensional segmentation quickly reduces the processing range, and three-dimensional reconstruction only processes target-related frames, reducing the amount of calculation. Label screening and clustering can be performed in parallel, which is suitable for GPU acceleration. It supports independent positioning of multiple targets in complex scenes (such as dense traffic scenes). Combined with video timing information, it can be extended to dynamic target trajectory prediction.

[0036] The three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction of the present application will be described in detail below in combination with specific embodiments and drawings.

[0037] A three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction proposed by the present application, as shown in Figure 2 The principle of the three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction, comprising the following steps: Step 1, video segmentation based on two-dimensional image segmentation model; The visual perception data (video) of the robot for the operation site and the operation target is decomposed into a video frame combination, and the first frame is sent to the Grounding DINO model. The key target is framed according to the human-provided semantic vocabulary (target prompt word). The framing result is sent to the two-dimensional image segmentation model (such as the Segment Anything model) for processing, and the key target segmentation for all video frames is realized through time sequence propagation.

[0038] Among them, the Grounding DINO model is to combine the DINO model (a self-supervised visual representation learning framework) with the visual Grounding task, use DINO to extract image features, and then combine the language model (such as BERT) for cross-modal alignment; or use the features of DINO as input to train a detection or segmentation model to associate visual elements with text labels. The model also needs to consider multi-task learning while optimizing visual and text representations.

[0039] The present application introduces the concept of RGB label. RGB label is a color label used to label target objects to support three-dimensional target positioning after three-dimensional reconstruction. RGB label is actually a kind of mask with preset (R, G, B) values and opaque properties. Generally, the color and transparency of the mask output by the two-dimensional image segmentation model can be obtained by redefining.

[0040] Specifically, the present embodiment can complete the labeling by covering the RGB label on the segmentation result output by the two-dimensional image segmentation model.

[0041] Preferably, the (R, G, B) values of the RGB label should be selected from (255, 0, 0), (0, 255, 0), (0, 0, 255) and other colors that are rarely seen in reality and have distinctive features, which can optimize the subsequent target point cloud screening effect. At the same time, it is generally recommended to unify the RGB label color of the same type of target and distinguish the RGB label color of different types of target, so that the subsequent target type can be distinguished directly according to the RGB label.

[0042] Step 2, three-dimensional reconstruction; The labeled video frames are input into a neural radiance field or Gaussian splatting type three-dimensional reconstruction model to obtain a three-dimensional point cloud model with a mask.

[0043] Preferably, models such as Dust3r and Spann3r that can directly regress three-dimensional point clouds from two-dimensional images are used, which can better restore the three-dimensional shape of the labeled target object according to the shape of the RGB label and the image information around the label.

[0044] Among them, Dust3r (Dense U-Net for Single-view 3D Reconstruction) directly predicts three-dimensional point cloud coordinates through an improved U-Net architecture combined with dense connections and self-attention mechanisms. The model maps two-dimensional images to point distributions in three-dimensional space through end-to-end training.

[0045] Spann3r (Spatial Point Attention Network for 3D Generation) combines spatial transformer and GAN (Generative Adversarial Network) to dynamically generate three-dimensional point cloud through attention mechanism.

[0046] Step 3, key point screening; According to the (R, G, B) value of the RGB label, the point cloud related to the target object is screened. In the reconstruction process, noise points, alignment deviations and other factors will cause the deviation of the (R, G, B) value of the marked point cloud, so it is necessary to screen the appropriate expansion (R, G, B) value range.

[0047] Step 4, clustering identification and noise reduction; The clustering algorithm (such as k-means, DBSCAN, etc.) is used to analyze the screened point cloud to generate a clustering label, and to distinguish multiple targets of the same kind. It is worth noting that the clustering algorithm can effectively filter out noise points and improve the segmentation quality.

[0048] Step 5, three-dimensional positioning; The three-dimensional space center point of different cluster point sets is extracted as the three-dimensional position of the segmentation target, and the coordinate conversion is performed according to the set visual line starting point and visual line yaw angle, pitch angle and other information, so as to obtain the spatial position of the target relative to the shooting point.

[0049] In the method of the present application, since the implementation subject of the segmentation task is a two-dimensional image segmentation model, only LoRa fine-tuning is required to realize the optimization of segmentation accuracy and precision. Compared with the direct three-dimensional segmentation model which needs to apply three-dimensional information for fine-tuning, it has great application advantages. For example, the method of the present application can be applied to live-line robot control, inspection robot navigation, etc.

[0050] As shown in Figure 3 The second object of the present application is to provide a three-dimensional target positioning system based on two-dimensional segmentation and three-dimensional reconstruction, based on the above-mentioned three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction, the system comprises: A video segmentation module 100 is used to obtain a visual perception video, perform key target segmentation on the visual perception video based on a two-dimensional image segmentation model to obtain a video frame, and perform RGB label annotation on the video. A three-dimensional reconstruction module 200 is used to input the annotated video frame into a three-dimensional reconstruction model to obtain a three-dimensional point cloud model with a mask. The key point screening module 300 is configured to screen the point cloud related to the target object from the three-dimensional point cloud model with the mask based on the label value. The clustering identification module 400 is configured to analyze the screened point cloud by using a clustering algorithm to generate a clustering point set. The three-dimensional positioning module 500 is configured to extract the three-dimensional space center point of different clustering point sets as the three-dimensional position of the segmentation target, and perform coordinate conversion on the three-dimensional position according to the set visual line starting point and visual line yaw angle and pitch angle to obtain the spatial position of the target relative to the shooting point.

[0051] In the above scheme, the three-dimensional target positioning system based on two-dimensional segmentation and three-dimensional reconstruction realizes the complete process from video input to three-dimensional space positioning through modular design. The system adopts a five-stage pipeline of two-dimensional segmentation→three-dimensional reconstruction→point cloud screening→clustering analysis→coordinate conversion, gradually maps low-dimensional visual information to high-dimensional space coordinates, extracts target masks (low-dimensional labels) through two-dimensional segmentation, and expands the target masks to three-dimensional point clouds with semantics (high-dimensional geometric information) through three-dimensional reconstruction. Through label screening and clustering analysis, sparse but semantically clear target feature points are extracted from the dense point cloud. The functions of each module are as follows: The video segmentation module decomposes the video into frames, generates target-level masks by using an instance segmentation model (such as Mask R-CNN), and assigns unique RGB labels to different targets. The pixel-level semantic association is established to provide target boundary constraints for subsequent three-dimensional reconstruction and avoid excessive dispersion of point clouds.

[0052] The three-dimensional reconstruction module adopts a monocular three-dimensional reconstruction model (such as Dust3r / Spann3r) to fuse the RGB label and geometric features through an encoder-decoder structure. The point cloud with label attributes is generated, each point contains three-dimensional coordinates (XYZ) and color label (RGB), and the strong association between semantics and geometry is realized.

[0053] The key point screening module filters the point cloud based on the label value and only retains the points corresponding to the target mask. The background noise is eliminated, the subsequent calculation amount is reduced, and the clustering efficiency is improved.

[0054] The clustering identification module adopts a density clustering or spatial segmentation algorithm. The discrete point cloud is aggregated into a compact cluster, each cluster corresponds to an independent target, and the separation problem of occluded or overlapped targets is solved.

[0055] The three-dimensional positioning module obtains the three-dimensional position of the target by calculating the centroid (Centroid) of the clustering cluster. The point cloud coordinates are converted from the camera coordinate system to the world coordinate system by using the camera external parameter (pose).

[0056] Preferably, through semantic-geometric joint modeling, the two-dimensional segmentation result is transferred to the three-dimensional space through the RGB label, avoiding semantic loss caused by independent reconstruction. Lightweight point cloud processing, screening-clustering two-step method significantly reduces data volume and improves real-time performance (compared to full point cloud analysis). Modular design allows replacement of segmentation / reconstruction models (such as replacing with a more efficient Transformer architecture), enhancing scalability.

[0057] Further, in the video segmentation module, the visual perception video obtained at least includes: Obtaining a visual perception video of a robot for an operating site and an operating target.

[0058] Further, the robot at least includes a live-line working robot or a patrol robot.

[0059] The video segmentation module is specifically configured to: The visual perception video is decomposed into a video frame combination, and the first frame is sent to the Grounding DINO model, and the key target is framed according to the human-provided semantic vocabulary; The framing result is sent to the two-dimensional image segmentation model for processing, and the key target segmentation of all video frames is obtained through time sequence propagation.

[0060] Further, the video segmentation module is further configured to: The labeling is completed by covering the RGB label on the segmented video frame output by the two-dimensional image segmentation model.

[0061] The (R, G, B) values of the RGB label are selected as (255, 0, 0), (0, 255, 0), and (0, 0, 255), and the RGB label colors of similar targets are unified, and the RGB label colors of different types of targets are distinguished.

[0062] The three-dimensional reconstruction module is specifically configured to: The labeled video frame is sent to the neural radiance field or Gaussian splatting three-dimensional reconstruction model, and the three-dimensional reconstruction model restores the three-dimensional shape of the labeled target object according to the shape of the RGB label and the information of the image around the label, and obtains a three-dimensional point cloud model with a mask.

[0063] Further, the key point screening module is specifically configured to: According to the (R, G, B) values of the RGB label, the point cloud related to the target object is screened.

[0064] The clustering algorithm includes k-means or DBSCAN.

[0065] As Figure 4As shown, a third object of the embodiments of the present application is to provide an electronic device, comprising a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor, wherein the processor implements the above-mentioned three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction when executing the computer program. The electronic device further comprises a communication interface 703 and a bus 704.

[0066] A fourth object of the embodiments of the present application is to provide a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the above-mentioned three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction.

[0067] A fifth object of the embodiments of the present application is to provide a computer program product, which comprises computer instructions, wherein the computer instructions instruct a computer to execute the above-mentioned three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction.

[0068] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product comprising instruction devices, which implement the functions specified in one or more flows and / or one or more blocks. Figure 1 Figure 1 These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product comprising instruction devices, which implement the functions specified in one or more flows and / or one or more blocks.

[0069] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product comprising instruction devices, which implement the functions specified in one or more flows and / or one or more blocks. Figure 1 Figure 1 These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product comprising instruction devices, which implement the functions specified in one or more flows and / or one or more blocks.

[0070] The present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, readable storage media, optical storage, etc.) containing computer usable program code.

[0071] ​​The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowcharts. Figure 1 The flowchart and / or block diagram in the variations disclosed herein illustrate the architecture, functionality, and operation of possible implementations of apparatuses and computer program products according to various embodiments. In this regard, each flowchart block and / or block in the flowcharts and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable Figure 1 The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowcharts.

[0072] Obviously, the embodiments described above are only part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit it. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced, and any modification or replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction, characterized in that: include: Acquire a visual perception video, segment the visual perception video into key objects based on a two-dimensional image segmentation model to obtain video frames, and annotate the video with RGB labels; Input the annotated video frames into the 3D reconstruction model to obtain a 3D point cloud model with a mask; Based on the label value, the point cloud related to the target object in the masked 3D point cloud model is filtered; Use clustering algorithm to analyze the screened point cloud and generate a set of clustered points; The three-dimensional spatial center points of different cluster point sets are extracted as the three-dimensional position of the segmented target. The three-dimensional position coordinates are transformed according to the set line of sight starting point, line of sight yaw angle, and pitch angle to obtain the spatial position of the target relative to the shooting point.

2. The three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction according to claim 1, characterized in that: The obtaining of the visual perception video at least includes: Obtain the robot's visual perception video of the operation site and operation target.

3. The three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction according to claim 1, characterized in that: The robot at least includes a live working robot or an inspection robot.

4. The three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction according to claim 1, characterized in that: The step of performing key target segmentation on the visual perception video based on a two-dimensional image segmentation model to obtain video frames includes: Decompose the visual perception video into a combination of video frames, and feed the first frame into the Grounding DINO model to select key targets based on manually provided semantic vocabulary; The box selection results are sent to the two-dimensional image segmentation model for processing, and the key targets of all video frames are segmented through temporal propagation to obtain the segmented video frames.

5. The three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction according to claim 1, characterized in that: The step of labeling the video with RGB tags includes: The annotation is completed by overlaying the RGB labels on the segmented video frames output by the 2D image segmentation model.

6. The three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction according to claim 1, characterized in that: The (R, G, B) values ​​of the RGB labels are selected as (255, 0, 0), (0, 255, 0), and (0, 0, 255). The RGB label colors of the same type of targets are unified, while the RGB label colors of different types of targets are distinguished.

7. The three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction according to claim 1, characterized in that: The process of inputting the labeled video frames into the 3D reconstruction model to obtain a 3D point cloud model with a mask includes: The labeled video frames are sent to the neural radiation field or Gaussian splash type 3D reconstruction model. The 3D reconstruction model restores the 3D shape of the labeled target object based on the shape of the RGB label and the information of the image around the label, and obtains a 3D point cloud model with a mask.

8. The three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction according to claim 1, characterized in that: The step of screening the point cloud related to the target object in the masked three-dimensional point cloud model based on the label value includes: Based on the (R, G, B) values ​​of the RGB labels, the point clouds related to the target objects are filtered.

9. The three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction according to claim 1, characterized in that: The clustering algorithm includes k-means or DBSCAN.

10. A three-dimensional target positioning system based on two-dimensional segmentation and three-dimensional reconstruction, characterized in that: include: A video segmentation module is used to obtain visual perception videos, segment the visual perception videos into key objects based on a two-dimensional image segmentation model to obtain video frames, and annotate the videos with RGB labels; The 3D reconstruction module is used to input the annotated video frames into the 3D reconstruction model to obtain a 3D point cloud model with a mask; The key point screening module is used to screen the point clouds related to the target object in the masked 3D point cloud model based on the label value; Cluster recognition module, used to analyze the screened point cloud using clustering algorithm to generate cluster point sets; The three-dimensional positioning module is used to extract the three-dimensional spatial center point of different cluster point sets as the three-dimensional position of the segmented target, and perform three-dimensional position coordinate conversion according to the set line of sight starting point and line of sight yaw angle and pitch angle to obtain the spatial position of the target relative to the shooting point.

11. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction as claimed in any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer instructions, characterized in that: The computer instructions instruct the computer to execute the three-dimensional target positioning method based on two-dimensional segmentation and three-dimensional reconstruction as described in any one of claims 1-9.

Images