Open-source 3D panoramic segmentation methods, devices, storage media, and program products

Abstract

This application provides an open-vocabulary 3D panoramic segmentation method, device, storage medium, and program product. The method includes: acquiring camera parameter information of a target 3D scene at a predicted viewpoint; inputting the camera parameter information of the predicted viewpoint into a preset instance segmentation network model to obtain a predicted instance segmentation result for the predicted viewpoint; determining the object category corresponding to each instance number in the predicted instance segmentation result based on a pre-acquired voting matrix, wherein the voting matrix includes the object categories and instance numbers of multiple objects at different viewpoints; determining a predicted semantic segmentation result for the predicted viewpoint based on each object category; and determining a predicted panoramic segmentation result for the predicted viewpoint based on the predicted instance segmentation result and the predicted semantic segmentation result. This method aims to achieve the technical effect of obtaining accurate panoramic segmentation results.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an open-vocabulary 3D panoramic segmentation method, device, storage medium, and program product. Background Technology

[0002] 3D scene understanding is a crucial technology in robotics. Its goal is to segment 3D object masks of all instances and background objects within a scene and identify their categories. It can also be viewed as open-vocabulary 3D scene panorama segmentation, a key component in building spatial semantic awareness. Furthermore, 3D scene understanding can support tasks such as autonomous driving, visual language robot navigation, and augmented reality. Because open-vocabulary 3D scene panorama segmentation can obtain semantic representations of objects not presented in the training set from visual language models, this task has attracted increasing attention.

[0003] Existing solutions for open-vocabulary 3D scene panoramic segmentation tasks mainly rely on RGB images as input, using multi-object tracking algorithms or feature point matching algorithms for pre-training, and then using new perspective synthesis algorithms for supervised training.

[0004] However, in existing technologies, when RGB images are used as input, the instance segmentation results obtained may have multiple different problems where the same instance segmentation result number is assigned to multiple different problems. At the same time, the semantic segmentation results contain erroneous information that interferes with network training. Therefore, existing technologies for panoramic segmentation suffer from low accuracy. Summary of the Invention

[0005] This application provides an open-vocabulary 3D panoramic segmentation method, device, storage medium, and program product to achieve the technical effect of obtaining accurate panoramic segmentation results.

[0006] In a first aspect, embodiments of this application provide an open vocabulary 3D panoramic segmentation method, including:

[0007] Obtain camera parameter information of the predicted viewpoint in the target 3D scene;

[0008] The camera parameter information of the viewpoint to be predicted is input into a preset instance segmentation network model to obtain the predicted instance segmentation result of the viewpoint to be predicted; wherein, the predicted instance segmentation result includes: multiple objects and the instance number corresponding to each object, and the instance number corresponding to each object is different.

[0009] Based on the pre-acquired voting matrix, the object category corresponding to each instance number in the predicted instance segmentation result is determined. The voting matrix includes the object category and instance number of multiple objects from different perspectives.

[0010] Based on each object category, determine the predicted semantic segmentation result for the viewpoint to be predicted; wherein, the predicted semantic segmentation result includes a category number corresponding to each object category;

[0011] The predicted panoramic segmentation result of the viewpoint to be predicted is determined based on the predicted instance segmentation result and the predicted semantic segmentation result.

[0012] In one possible implementation, the instance segmentation network model is trained in the following manner:

[0013] Acquire RGB images of the target 3D scene from multiple perspectives;

[0014] Based on the unsupervised instance segmentation algorithm, the first instance segmentation result of the RGB image for each viewpoint is obtained;

[0015] The instance number in the first instance segmentation result of the RGB image of each viewpoint is reassigned according to the Hungarian matching algorithm to obtain the instance segmentation result of the reassigned instance number of the RGB image of each viewpoint.

[0016] Based on the instance segmentation results obtained by reassigning instance numbers to the RGB images from each viewpoint, the instance segmentation network model to be trained is iteratively trained to obtain the instance segmentation network model.

[0017] In one possible implementation, based on the instance segmentation results of the reassigned instance numbers of the RGB images from each viewpoint, the instance segmentation network model to be trained is iteratively trained to obtain the instance segmentation network model, including:

[0018] The first stage of training of the instance segmentation network model to be trained is carried out based on the instance segmentation results of the RGB images of each viewpoint with reassigned instance numbers to obtain loss function information. The instance segmentation network model to be trained is then supervised based on multiple loss function information to obtain an intermediate instance segmentation network model. The second instance segmentation result of the RGB images of each viewpoint is obtained based on the instance segmentation network model.

[0019] The instance numbers of the same object under different viewpoints in the first instance segmentation result and the second instance segmentation result are uniformly processed to obtain the third instance segmentation result of the RGB image of each viewpoint; wherein, the third instance segmentation result of the RGB image of each viewpoint includes multiple objects, each object corresponds to an instance number, and the instance number of the same object is consistent in the RGB images of different viewpoints.

[0020] A set of point clouds with the same instance number is determined based on multiple instance numbers in the third instance segmentation results of RGB images from multiple viewpoints; wherein, the set of point clouds includes multiple point clouds, and the number of point clouds with the same instance number is at least 2.

[0021] Multiple point clouds in the point cloud set are fused to determine the fourth instance segmentation result of the RGB image from each viewpoint; wherein, the fourth instance segmentation result of the RGB image from each viewpoint includes multiple objects, each object corresponds to a unique instance number, and the instance number of the same object is consistent in the RGB images from different viewpoints.

[0022] Based on the fourth instance segmentation result of the RGB image from each viewpoint, the intermediate instance segmentation network model is trained in the second stage to obtain cross-entropy loss information. The intermediate instance segmentation network model is then supervised based on multiple cross-entropy loss information to obtain the instance segmentation network model.

[0023] In one possible implementation, multiple point clouds in the point cloud set are fused to determine the fourth instance segmentation result of the RGB image from each viewpoint, including:

[0024] Step a, identify two adjacent point clouds in a set of point clouds with the same instance number; wherein, the two adjacent point clouds include a first point cloud and a second point cloud, the first point cloud includes at least one object, and the second point cloud includes at least one object;

[0025] Step b: Determine whether there are any identical objects among multiple objects in two adjacent point clouds. If so, unify the instance numbers of the identical objects and merge the identical objects to determine a new instance number for the identical objects.

[0026] Step c: Identify multiple different objects between two adjacent point clouds, excluding identical objects.

[0027] Step d: Renumber the instance numbers of each different object to complete the fusion of the first point cloud and the second point cloud. The fused point cloud is the new first point cloud.

[0028] Step e: Determine the adjacent point clouds of the new first point cloud as the new second point cloud. Repeat steps b to e until all point clouds in the point cloud set with the same instance number are merged and determined as the target point cloud; wherein, each object in the target point cloud corresponds to a unique instance number.

[0029] In one possible implementation, there is at least one object in the first point cloud and at least one object in the second point cloud;

[0030] Correspondingly, determining whether there are identical objects among multiple objects in two adjacent point clouds includes:

[0031] Determine the distance between each data point of each object in the first point cloud and each data point of each object in the second point cloud to obtain the distance information between every two data points.

[0032] The nearest neighbor of a data point in the first point cloud is determined based on the distance information of multiple data points; wherein the number of nearest neighbors of a data point is at most 1.

[0033] Determine the number of nearest neighbors for each object in the first point cloud, and determine whether the number of nearest neighbors for each object meets the preset conditions. If so, the two objects are the same object, and there are multiple objects with the same object in the two adjacent point clouds.

[0034] In one possible implementation, after determining the fourth instance segmentation result of the RGB image for each viewpoint, the method further includes:

[0035] Based on the open vocabulary semantic segmentation algorithm, the first semantic segmentation result of the RGB image for each viewpoint is determined. The first semantic segmentation result includes multiple object categories, each object category corresponds to a category number, and the number of category numbers is less than or equal to the number of instance numbers.

[0036] Based on the first semantic segmentation result of the RGB image of each viewpoint, obtain the object category corresponding to each instance number in the fourth instance segmentation result of the RGB image of each viewpoint under different viewpoints;

[0037] A voting matrix is ​​determined based on the object category corresponding to each instance number; the voting matrix is ​​used to determine the number of objects of the same category in each instance number.

[0038] Determine the maximum number among multiple counts belonging to the same object category, and determine the target object category corresponding to the maximum number of the same object category for each instance number;

[0039] Based on each target object category, a second semantic segmentation result is determined for the RGB image of each viewpoint; wherein, the second semantic segmentation result includes a category number corresponding to each object category.

[0040] Secondly, embodiments of this application provide an open vocabulary 3D panoramic segmentation device, comprising:

[0041] The acquisition module is used to acquire camera parameter information of the viewpoint to be predicted in the target 3D scene;

[0042] The first processing module is used to input the camera parameter information of the viewpoint to be predicted into a preset instance segmentation network model to obtain the predicted instance segmentation result of the viewpoint to be predicted; wherein, the predicted instance segmentation result includes: multiple objects and the instance number corresponding to each object, and the instance number corresponding to each object is different.

[0043] The second processing module is used to determine the object category corresponding to each instance number in the predicted instance segmentation result based on the pre-acquired voting matrix. The voting matrix includes the object category and instance number of multiple objects from different perspectives.

[0044] The second processing module is used to determine the predicted semantic segmentation result of the viewpoint to be predicted based on each object category; wherein, the predicted semantic segmentation result includes a category number corresponding to each object category;

[0045] The third processing module is used to determine the predicted panoramic segmentation result of the viewpoint to be predicted based on the predicted instance segmentation result and the predicted semantic segmentation result.

[0046] In one possible implementation, the first processing module is further configured to:

[0047] Acquire RGB images of the target 3D scene from multiple perspectives;

[0048] Based on the unsupervised instance segmentation algorithm, the first instance segmentation result of the RGB image for each viewpoint is obtained;

[0049] The instance numbers in the first instance segmentation results of the RGB images from each viewpoint are reassigned based on the Hungarian matching algorithm to obtain the second instance segmentation results of the RGB images from each viewpoint.

[0050] Based on the first instance segmentation results and the second instance segmentation results of RGB images from multiple perspectives, the instance segmentation network model to be trained is obtained.

[0051] In one possible implementation, the first processing module is further configured to:

[0052] Acquire RGB images of the target 3D scene from multiple perspectives;

[0053] Based on the unsupervised instance segmentation algorithm, the first instance segmentation result of the RGB image for each viewpoint is obtained;

[0054] The instance number in the first instance segmentation result of the RGB image of each viewpoint is reassigned according to the Hungarian matching algorithm to obtain the instance segmentation result of the reassigned instance number of the RGB image of each viewpoint.

[0055] Based on the instance segmentation results obtained by reassigning instance numbers to the RGB images from each viewpoint, the instance segmentation network model to be trained is iteratively trained to obtain the instance segmentation network model.

[0056] In one possible implementation, the first processing module is further configured to:

[0057] The first stage of training of the instance segmentation network model to be trained is carried out based on the instance segmentation results of the RGB images of each viewpoint with reassigned instance numbers to obtain loss function information. The instance segmentation network model to be trained is then supervised based on multiple loss function information to obtain an intermediate instance segmentation network model. The second instance segmentation result of the RGB images of each viewpoint is obtained based on the instance segmentation network model.

[0058] The instance numbers of the same object under different viewpoints in the first instance segmentation result and the second instance segmentation result are uniformly processed to obtain the third instance segmentation result of the RGB image of each viewpoint; wherein, the third instance segmentation result of the RGB image of each viewpoint includes multiple objects, each object corresponds to an instance number, and the instance number of the same object is consistent in the RGB images of different viewpoints.

[0059] A set of point clouds with the same instance number is determined based on multiple instance numbers in the third instance segmentation results of RGB images from multiple viewpoints; wherein, the set of point clouds includes multiple point clouds, and the number of point clouds with the same instance number is at least 2.

[0060] Multiple point clouds in the point cloud set are fused to determine the fourth instance segmentation result of the RGB image from each viewpoint; wherein, the fourth instance segmentation result of the RGB image from each viewpoint includes multiple objects, each object corresponds to a unique instance number, and the instance number of the same object is consistent in the RGB images from different viewpoints.

[0061] Based on the fourth instance segmentation result of the RGB image from each viewpoint, the intermediate instance segmentation network model is trained in the second stage to obtain cross-entropy loss information. The intermediate instance segmentation network model is then supervised based on multiple cross-entropy loss information to obtain the instance segmentation network model.

[0062] In one possible implementation, the first processing module is further configured to:

[0063] Step a, identify two adjacent point clouds in a set of point clouds with the same instance number; wherein, the two adjacent point clouds include a first point cloud and a second point cloud, the first point cloud includes at least one object, and the second point cloud includes at least one object;

[0064] Step b: Determine whether there are any identical objects among multiple objects in two adjacent point clouds. If so, unify the instance numbers of the identical objects and merge the identical objects to determine a new instance number for the identical objects.

[0065] Step c: Identify multiple different objects between two adjacent point clouds, excluding identical objects.

[0066] Step d: Renumber the instance numbers of each different object to complete the fusion of the first point cloud and the second point cloud. The fused point cloud is the new first point cloud.

[0067] Step e: Determine the adjacent point clouds of the new first point cloud as the new second point cloud. Repeat steps b to e until all point clouds in the point cloud set with the same instance number are merged and determined as the target point cloud; wherein, each object in the target point cloud corresponds to a unique instance number.

[0068] In one possible implementation, the first processing module is further configured to:

[0069] The first point is that there is at least one object in the cloud; the second point is that there is at least one object in the cloud.

[0070] Determine the distance between each data point of each object in the first point cloud and each data point of each object in the second point cloud to obtain the distance information between every two data points.

[0071] The nearest neighbor of a data point in the first point cloud is determined based on the distance information of multiple data points; wherein the number of nearest neighbors of a data point is at most 1.

[0072] Determine the number of nearest neighbors for each object in the first point cloud, and determine whether the number of nearest neighbors for each object meets the preset conditions. If so, the two objects are the same object, and there are multiple objects with the same object in the two adjacent point clouds.

[0073] In one possible implementation, the first processing module is further configured to:

[0074] Based on the open vocabulary semantic segmentation algorithm, the first semantic segmentation result of the RGB image for each viewpoint is determined. The first semantic segmentation result includes multiple object categories, each object category corresponds to a category number, and the number of category numbers is less than or equal to the number of instance numbers.

[0075] Based on the first semantic segmentation result of the RGB image of each viewpoint, obtain the object category corresponding to each instance number in the fourth instance segmentation result of the RGB image of each viewpoint under different viewpoints;

[0076] A voting matrix is ​​determined based on the object category corresponding to each instance number; the voting matrix is ​​used to determine the number of objects of the same category in each instance number.

[0077] Determine the maximum number among multiple counts belonging to the same object category, and determine the target object category corresponding to the maximum number of the same object category for each instance number;

[0078] Based on each target object category, a second semantic segmentation result is determined for the RGB image of each viewpoint; wherein, the second semantic segmentation result includes a category number corresponding to each object category.

[0079] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0080] The memory stores computer-executed instructions;

[0081] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0082] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0083] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0084] The open-vocabulary 3D panoramic segmentation method, device, storage medium, and program product provided in this application obtain camera parameter information of the viewpoint to be predicted in the target 3D scene. By inputting the camera parameter information of the viewpoint to be predicted into a preset instance segmentation network model, a predicted instance segmentation result for the viewpoint to be predicted can be obtained. Each object in the predicted instance segmentation result corresponds to a unique instance number. At the same time, based on a pre-acquired voting matrix, the object category corresponding to each instance number in the predicted instance segmentation result is determined. Based on each object category, a predicted semantic segmentation result for the viewpoint to be predicted is determined. The predicted semantic segmentation result includes a category number corresponding to each object category, making the predicted semantic segmentation result more accurate. By determining the predicted panoramic segmentation result for the viewpoint to be predicted through the predicted instance segmentation result and the predicted semantic segmentation result, the technical effect of obtaining a more accurate panoramic segmentation result is achieved. Attached Figure Description

[0085] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0086] Figure 1 Flowchart of the open vocabulary 3D panoramic segmentation method provided in this application Figure 1 ;

[0087] Figure 2Flowchart of the open vocabulary 3D panoramic segmentation method provided in this application Figure 2 ;

[0088] Figure 3 Flowchart of the open vocabulary 3D panoramic segmentation method provided in this application Figure 3 ;

[0089] Figure 4 A schematic diagram of the open vocabulary 3D panoramic segmentation method provided in this application;

[0090] Figure 5 Flowchart of the open vocabulary 3D panoramic segmentation method provided in this application Figure 4 ;

[0091] Figure 6 A schematic diagram of the structure of the open-vocabulary 3D panoramic segmentation device provided in this application;

[0092] Figure 7 A hardware schematic diagram of the open-vocabulary 3D panoramic segmentation device provided in this application.

[0093] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0094] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and approaches consistent with some aspects of this application as detailed in the appended claims.

[0095] First, let me explain the terms used in this application:

[0096] Open vocabulary: In computer vision, open vocabulary tasks typically refer to tasks where models, by combining natural language, can identify and process categories or objects not seen during training. These tasks are crucial for improving the model's generalization and adaptability.

[0097] Instance segmentation: Identifying the exact pixel boundaries of each individual object instance in an image.

[0098] Semantic segmentation: classifying each pixel in an image to achieve a fine understanding of the scene.

[0099] Panoramic segmentation combines the features of semantic segmentation and instance segmentation. Its goal is to assign a category label and instance number to each pixel in the image, generating a global and uniform segmented image.

[0100] Hungarian matching algorithm: The Hungarian matching algorithm is a classic algorithm for solving task assignment problems. It can find the maximum weight matching in a bipartite graph in polynomial time. In practical applications, it is often used to solve assignment problems.

[0101] In existing technologies, panoramic segmentation uses RGB images as input and pre-connects them with multi-object tracking algorithms or feature point matching algorithms. Then, it uses a new perspective synthesis algorithm for supervised training. This can lead to problems such as objects in space being assigned multiple numbers, meaning one object is considered as multiple objects and thus assigned multiple numbers, and the same number being assigned to multiple objects, meaning multiple objects are considered as one object. In addition, for semantic segmentation results, current methods directly train on the semantic information extracted from multiple perspectives. Error information interferes with network training, resulting in inaccurate semantic segmentation results.

[0102] To address the problems in existing technologies, this application provides an open-vocabulary 3D panoramic segmentation method, device, storage medium, and program product. By inputting camera parameter information of the desired viewpoint into a preset instance segmentation network model, a predicted instance segmentation result for the desired viewpoint is obtained. In this predicted instance segmentation result, each object corresponds to a unique number. Based on the predicted instance segmentation result and a pre-acquired voting matrix, the object category corresponding to each instance number is obtained, thereby obtaining a predicted semantic segmentation result that eliminates erroneous information. The predicted panoramic segmentation result for the desired viewpoint is determined based on the predicted instance segmentation result and the predicted semantic segmentation result. This achieves the goal of obtaining a panoramic segmentation result from a new viewpoint using only camera parameter information with a predicted viewpoint. This result has a certain degree of accuracy and reliability.

[0103] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0104] Figure 1 Flowchart of the open vocabulary 3D panoramic segmentation method provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0105] S101. Obtain camera parameter information of the viewpoint to be predicted in the target 3D scene;

[0106] In this embodiment, camera parameter information includes camera intrinsic parameters and pose information. The camera intrinsic parameters include focal length, principal point, and distortion coefficients, while the pose information refers to the position and orientation of the object, robot, or camera in three-dimensional space. In a target three-dimensional scene, to accurately predict the panoramic segmentation result of a new viewpoint, it is necessary to obtain the camera parameter information of the viewpoint to be predicted. Camera parameter information helps improve the performance of panoramic segmentation algorithms in various complex scenes, providing strong support for subsequent image understanding and analysis. The camera parameter information allows for more accurate inference of the image from the viewpoint to be predicted.

[0107] S102. Input the camera parameter information of the viewpoint to be predicted into the preset instance segmentation network model to obtain the predicted instance segmentation result of the viewpoint to be predicted; wherein, the predicted instance segmentation result includes: multiple objects and the instance number corresponding to each object, and the instance number corresponding to each object is different.

[0108] In this embodiment, the camera parameter information of the viewpoint to be predicted is input into a preset instance segmentation network model. Through the model's calculation, the predicted instance segmentation result of the viewpoint to be predicted is obtained. This result includes multiple objects in the image and the instance number corresponding to each object. These instance numbers are unique and used to distinguish different object instances. Each object is accurately segmented, and its boundaries are clearly defined. Through this preset instance segmentation network model, the technical problems of objects in the current space being assigned multiple numbers, i.e., one object is considered as multiple objects and thus assigned multiple numbers, and the same number being assigned to multiple objects, i.e., multiple objects are considered as one object, can be effectively solved. Through this model, highly accurate instance segmentation results can be obtained.

[0109] S103. Based on the pre-acquired voting matrix, determine the object category corresponding to each instance number in the predicted instance segmentation result. The voting matrix includes the object category and instance number of multiple objects from different perspectives.

[0110] In this embodiment, the voting matrix includes the object category and instance number of each object under multiple different viewpoints. Based on the voting matrix, the object category corresponding to each instance segmentation result in the predicted instance segmentation result can be determined.

[0111] S104. Determine the predicted semantic segmentation result for the viewpoint to be predicted based on each object category; wherein, the predicted semantic segmentation result includes a category number corresponding to each object category;

[0112] In this embodiment, each object category is assigned a unique number to determine a unique category number, thereby determining the predicted semantic segmentation result for the viewpoint to be predicted.

[0113] S105. Determine the predicted panoramic segmentation result of the viewpoint to be predicted based on the predicted instance segmentation result and the predicted semantic segmentation result.

[0114] In this embodiment, the complex task of target panoramic segmentation relies not only on the results of instance segmentation but also on the output of semantic segmentation to jointly determine the final predicted panoramic segmentation result. This step involves a detailed analysis and classification of each pixel in the image, aiming to generate a complete segmentation map that contains both object instance information and semantic category information. In the predicted panoramic segmentation result for the viewpoint to be predicted, each pixel in the image is accurately classified into a specific object instance and semantic category. This result provides rich image understanding information and strong support for subsequent image analysis and applications. Predicted panoramic segmentation results play a crucial role in fields such as autonomous driving, medical image analysis, and robot vision.

[0115] The open-vocabulary 3D panoramic segmentation method provided in this application's embodiments obtains the predicted instance segmentation result for the predicted viewpoint by inputting the camera parameter information of the viewpoint to be predicted into a preset instance segmentation network model. In the predicted instance segmentation result, each object corresponds to a unique number. Based on the predicted instance segmentation result and a pre-acquired voting matrix, the object category corresponding to each instance number is obtained, thereby obtaining the predicted semantic segmentation result that eliminates erroneous information. Based on the predicted instance segmentation result and the predicted semantic segmentation result, the predicted panoramic segmentation result for the viewpoint to be predicted is determined. Thus, a panoramic segmentation result under a new viewpoint can be obtained based on camera parameter information with a predicted viewpoint. This result has a certain degree of accuracy and reliability.

[0116] Figure 2 Flowchart of the open vocabulary 3D panoramic segmentation method provided in this application Figure 2 This embodiment is in Figure 1 Based on the embodiments, the open-vocabulary 3D panoramic segmentation method is described in detail. The training process of the instance segmentation network model can be implemented through steps S201-S205; the determination of the voting matrix based on the fourth segmentation result and the first semantic segmentation result can be implemented through steps S206-S207; based on the instance segmentation network model and the voting matrix, the predicted panoramic segmentation result of the viewpoint to be predicted can be obtained in steps S101-S105; as shown... Figure 2 As shown, the open vocabulary 3D panoramic segmentation method provided in this embodiment includes:

[0117] S201. Obtain RGB images of the target 3D scene from multiple perspectives; obtain the first instance segmentation result of the RGB image of each perspective according to the unsupervised instance segmentation algorithm; reassign the instance number in the first instance segmentation result of the RGB image of each perspective according to the Hungarian matching algorithm to obtain the instance segmentation result of the RGB image of each perspective with reassigned instance number.

[0118] In this embodiment, a target 3D scene is determined, and RGB images from multiple perspectives within the target 3D scene are obtained. Camera parameter information for each RGB image is also obtained. An unsupervised instance segmentation algorithm is used to obtain the first instance segmentation result for each perspective's RGB image. The instance numbers in the first instance segmentation result of each perspective's RGB image are then reassigned using a Hungarian matching algorithm, resulting in a reassigned segmentation result for each time. The training data for the instance segmentation network model is determined based on these reassigned instance segmentation results, and the model is iteratively trained using this data. According to the unsupervised instance segmentation algorithm, obtaining the first instance segmentation result for each perspective's RGB image presents two technical problems. One problem is that objects in space may be assigned multiple numbers, meaning one object is considered multiple objects and thus assigned multiple numbers, and the same number may be assigned to multiple objects, meaning multiple objects are considered one object. To address the issue of one object being considered multiple objects and thus assigned multiple numbers, a Hungarian matching algorithm can be used to reassign the instance numbers in each first instance segmentation result, obtaining the reassigned instance segmentation result for each perspective's RGB image.

[0119] Specifically, the process of reallocating the instance number in each first instance segmentation result using the Hungarian matching algorithm is as follows:

[0120] For each viewpoint of the RGB image, there is a set of rays {I1, I2, ..., I...} n In this context, an image contains multiple pixels, and a ray passing through a pixel from the camera center is called a ray. An instance segmentation network model can predict the instance number of each ray, and the probability distribution of the instance number is as follows: Among them, y n ∈[0,1] U U represents the preset number of instances. Using the Hungarian matching algorithm, the predicted instance number can be mapped to each first instance segmentation result. For each RGB image, the first instance segmentation result is {M1, M2, ..., M...}. n}; where M n ∈[0,T n ] H×W H and W are the height and width of the image, and T is the height and width of the image. n It is Mn The maximum index in the matrix, where 0 represents the position where no instance exists, and the cost matrix is:

[0121]

[0122] in It is M n The one-hot encoded version, T n It is M n The largest number in; This indicates taking the corresponding light ray from the t-th channel. The value of the position. The Hungarian matching algorithm can be used to predict... The channel order was adjusted to allow it to work with M. n T in n By aligning the masks, we obtain the instance segmentation results of the RGB images of each viewpoint with reassigned instance numbers.

[0123] S202. The first stage of training of the instance segmentation network model to be trained is carried out based on the instance segmentation results of the reassigned instance numbers of the RGB images of each viewpoint to obtain loss function information. The instance segmentation network model to be trained is supervised based on multiple loss function information to obtain an intermediate instance segmentation network model. The second instance segmentation result of the RGB images of each viewpoint is obtained based on the instance segmentation network model.

[0124] In this embodiment, by using M n T in n After aligning the masks, the instance segmentation network model to be trained is iterated multiple times using the instance segmentation results of the RGB images of each viewpoint with reassigned instance numbers to obtain the loss function information. The formula of the loss function is as follows:

[0125]

[0126] This yields an intermediate instance segmentation network model. This model has learned how to number instances. If the camera parameter information of the viewpoint to be predicted is input into this intermediate instance segmentation network model, instance segmentation results can be obtained. The current instance segmentation results no longer present the technical problem of objects in space being assigned multiple numbers (i.e., one object being considered multiple objects and thus assigned multiple numbers). However, there are still cases where the same number is assigned to multiple objects, meaning multiple objects are considered as one object.

[0127] S203. In the first instance segmentation result and the second instance segmentation result, the instance numbers of the same object under different viewpoints are uniformly processed to obtain the third instance segmentation result of the RGB image of each viewpoint.

[0128] In this embodiment, in order to eliminate the influence of noise in the second instance segmentation result, the third instance segmentation result is determined by the first instance segmentation result and the second instance segmentation result according to the following formula:

[0129]

[0130] in, The second instance segmentation result of the RGB image for each viewpoint; The third instance segmentation result for the RGB image at each viewpoint; M n The first instance segmentation result of the RGB image for each viewpoint; t∈[1,T] n ] for M n Each number that exists in the [the database].

[0131] The above formula can be used to obtain the third instance segmentation result of the RGB image from each viewpoint. Each third instance segmentation result includes multiple objects, and each object corresponds to an instance number. The instance number of the same object is the same in the RGB images from different viewpoints, but there may be cases where the same instance number corresponds to multiple different objects.

[0132] S204. Determine a set of point clouds with the same instance number based on the multiple instance numbers in the third instance segmentation results of RGB images from multiple viewpoints; fuse the multiple point clouds in the point cloud set to determine the fourth instance segmentation result of the RGB image from each viewpoint.

[0133] In this embodiment, in order to eliminate the situation where the same instance number corresponds to multiple objects, a point cloud set with the same instance number is obtained. The point cloud set includes multiple point clouds, and each point cloud contains at least one object. The multiple point clouds in the point cloud set are fused multiple times, and finally the fourth instance segmentation result of the RGB image of each viewpoint is obtained. The fourth instance segmentation result of the RGB image of each viewpoint includes multiple objects, and each object corresponds to a unique instance number. The instance number of the same object is consistent in the RGB images of different viewpoints.

[0134] S205. Based on the fourth instance segmentation result of the RGB image from each viewpoint, perform a second-stage training on the intermediate instance segmentation network model to obtain cross-entropy loss information. Supervise the intermediate instance segmentation network model based on multiple cross-entropy loss information to obtain the instance segmentation network model.

[0135] In this embodiment, the obtained fourth instance segmentation result is used. Supervised training of the network is performed. During sampling, samples are taken from all training images to obtain the network's prediction results corresponding to the sampled rays. The instance number of the corresponding fourth instance segmentation result is Supervision is performed using cross-entropy loss, the cross-entropy loss function of which is shown in the following equation:

[0136]

[0137] Through the above training, the instance segmentation network model can predict accurate instance segmentation results based on the camera parameter information of the viewpoint to be predicted. Each object in the instance segmentation result has a unique instance number, and each instance number corresponds to only one object. The model trained in this way eliminates the erroneous representation between instance numbers and objects, providing a foundation for accurately obtaining panoramic segmentation results of 3D scenes.

[0138] S206. Based on the open vocabulary semantic segmentation algorithm, determine the first semantic segmentation result of the RGB image from each viewpoint. Based on the first semantic segmentation result of the RGB image from each viewpoint, obtain the object category corresponding to each instance number in the fourth instance segmentation result of the RGB image from each viewpoint under different viewpoints.

[0139] In this embodiment, to eliminate the influence of erroneous information in the semantic segmentation results, the semantics of the first semantic segmentation result of the RGB image for each viewpoint, determined according to the open vocabulary semantic segmentation algorithm, are combined with the fourth instance segmentation result to eliminate erroneous information in the first semantic segmentation result. Specifically, it is necessary to obtain the object category corresponding to each instance number in the fourth instance segmentation result of the RGB image for each viewpoint under different viewpoints. The first semantic segmentation result includes multiple object categories, each object category corresponding to a category number, and the number of category numbers is less than or equal to the number of instance numbers.

[0140] S207. Determine the voting matrix based on the object category corresponding to each instance number; the voting matrix determines the number of each identical object category in the object category corresponding to each instance number, determines the maximum number among multiple numbers belonging to the same object category, and determines the target object category corresponding to each instance number based on the maximum number belonging to the same object category; based on each target object category, determine the second semantic segmentation result of the RGB image for each viewpoint.

[0141] In this embodiment, the voting matrix includes the object category and instance number of each object from multiple perspectives. This voting matrix determines the number of instances of the same object category for each instance number. For example, if object K exists in the instance segmentation result, and the instance number of object A is 1, then the object category corresponding to object K in each perspective of the first semantic segmentation result is determined. If there are 6 perspectives, then the object category corresponding to K is determined to be A in the first perspective, B in the second perspective, A in the third perspective, A in the fourth perspective, and B in the fifth perspective. From the sixth perspective, the object category corresponding to K is C. These results are statistically analyzed in the voting matrix. Further, the voting matrix determines the number of objects belonging to the same object category for each instance number. That is, it is determined that the number of objects K (instance number 1) belonging to object category A is 3, the number belonging to object category B is 2, and the number belonging to object category C is 1. Then, the maximum number of objects belonging to the same object category is determined, and the object category corresponding to the maximum number is determined as the target object category for each instance number. That is, the maximum number of objects belonging to object category A in object K is determined, and the target object category corresponding to object K with instance number 1 is A.

[0142] Optionally, in conjunction with the fourth instance segmentation result, the object categories in the first semantic segmentation result are renumbered, and based on each target object category, the second semantic segmentation result of the RGB image for each viewpoint is determined. In this result, erroneous information is eliminated, and a voting matrix is ​​used to determine the unified result of each object category under multiple viewpoints.

[0143] Figure 3 Flowchart of the open vocabulary 3D panoramic segmentation method provided in this application Figure 3 This embodiment is in Figure 2 Based on the embodiments, the process of fusing multiple point clouds in a point cloud set is described in detail; such as Figure 3 As shown, the open vocabulary 3D panoramic segmentation method provided in this embodiment includes:

[0144] S301. Identify two adjacent point clouds in a set of point clouds with the same instance number;

[0145] In this embodiment, based on the segmentation result of each fourth instance and the corresponding depth map, multiple sets of point clouds with the same instance number can be determined. The depth map includes multiple depth information items, and the point cloud information can be determined based on the depth information. A set of point clouds with the same instance number is represented as... P represents a point cloud. During the fusion process, each time two adjacent point clouds in the set are identified, it is determined whether there is a common object between them. The two adjacent point clouds include a first point cloud and a second point cloud. The first point cloud contains at least one object, and the second point cloud contains at least one object. Initially, during the first fusion, both the first and second point clouds contain only one object. After the second and subsequent fusions, both the first and second point clouds must contain at least one object. Figure 4 As shown, in In the first fusion stage, there is only one object in two adjacent point clouds. After fusion through r rounds, there is at least one object in a point cloud.

[0146] S302. Determine whether there are any identical objects among multiple objects in two adjacent point clouds. If so, unify the instance numbers of the identical objects and merge the identical objects to determine a new instance number for the identical objects.

[0147] In this embodiment, if two point clouds contain the same object, the instance numbers of the same object are re-unified, thereby unifying the numbering of the two identical objects. For example, if... Figure 4 As shown in In the stages and Two adjacent point clouds, determine and If two objects are the same object, they will be merged, the merged object will be identified as the new object, and the two objects will be assigned the same number. The new object will correspond to a new instance number, and the instance numbers of the same objects in the new point cloud will be consistent.

[0148] S303. Identify multiple objects between two adjacent point clouds that are different from the same objects.

[0149] In this embodiment, two adjacent point clouds may contain the same object or multiple different objects, for example, Figure 4 As shown, the first point cloud includes two objects, A and B, which are not the same object. The second point cloud includes two objects, C and D, which are not the same object. B and D are the same two objects. Therefore, the two objects are merged into a new object, while A and C are different objects.

[0150] S304. Renumber the instance numbers of each different object to complete the fusion of the first point cloud and the second point cloud. The fused point cloud is the new first point cloud.

[0151] In this embodiment, as Figure 4As shown, for example, when it is determined that A and C are different objects, A and C are renumbered to obtain instance numbers that retain the original values ​​of A and C. Through this round of fusion, a new point cloud is obtained, which includes object A, object C, and object E, which is a new object formed by the fusion of B and D. The fused new point cloud is named the first point cloud. In the new first point cloud, there are no identical objects, and the number corresponding to each object is different.

[0152] S305. Determine the adjacent point clouds of the new first point cloud as the new second point cloud. Repeat steps S302 to S305 until all point clouds in the point cloud set with the same instance number are merged and determined as the target point cloud. Each object in the target point cloud corresponds to a unique instance number.

[0153] In this embodiment, by repeating steps S302 to S305 with the new first point cloud and the new second point cloud, objects with the same instance number can be fused together, and these identical objects are determined to correspond to a unified number. For different objects, new instance numbers are reassigned. In the final target point cloud, each different object corresponds to a unique number, while identical objects have a unified number. Of course, in the target point cloud, all identical objects are fused together to determine that they are fused into one object, which corresponds to a unique instance number. The numbers of identical objects in this object are all the same instance number.

[0154] Figure 5 Flowchart of the open vocabulary 3D panoramic segmentation method provided in this application Figure 4 This embodiment is in Figure 3 Based on the embodiments, the process of determining whether there are identical objects among multiple objects in two adjacent point clouds is described in detail; for example... Figure 5 As shown, the open vocabulary 3D panoramic segmentation method provided in this embodiment includes:

[0155] S501. Determine the distance between each data point of each object in the first point cloud and each data point of each object in the second point cloud, and obtain the distance information between every two data points.

[0156] In this embodiment, there is at least one object in the first point cloud and at least one object in the second point cloud. Each object includes multiple data points, and each data point can be represented by (x, y, z, i), where i represents the correspondence between the data point and the object. The distance between each data point in each object in the first point cloud and each data point in each object in the second point cloud is determined, and the distance information between the data points of the objects in the first point cloud and the data points of the objects in the second point cloud is obtained. For example, there is object A in the first point cloud, and object A has N data points. There are objects B and C in the second point cloud, with M data points in object B and T data points in object C. Then, there are a total of M+T distance information between each data point in object A and the data points in the second point cloud.

[0157] S502. Determine the nearest neighbor point in the first point cloud based on the distance information of multiple data points; wherein the number of nearest neighbors corresponding to the data point is at most 1.

[0158] In this embodiment, a preset threshold is set for the distance information of data points. Each data point distance is then judged to see if it meets the preset threshold. The number of data point distances that meet the preset threshold is greater than or equal to 0. When the number of data point distances that meet the preset threshold is at least 1, the minimum distance among all data point distances is determined to be the nearest neighbor point corresponding to that data point in the first point cloud. For example, the first point cloud contains object A with N data points, and the second point cloud contains objects B and C with M data points in object B and T data points in object C. The Xth data point of object A in the first point cloud has distance information from M data points in object B in the second point cloud. Among these M data points, 3 data points satisfy a preset threshold. These three data points are the Yth, Rth, and Wth data points in the second point cloud. Therefore, the nearest neighbors of the Xth data point in the first point cloud are determined to be the Yth, Rth, and Wth data points in the second point cloud. Then, the minimum value of the distance information of the data points is determined based on the Yth, Rth, and Wth data points, and the point corresponding to the minimum value is the nearest neighbor of the Xth data point. The Xth data point of object A in the first point cloud has distance information from T data points in object C in the second point cloud. Among these T data points, none of the distance information of the data points satisfies the preset threshold. Therefore, the Xth data point of object A has no nearest neighbor in object C.

[0159] S503. Determine the number of nearest neighbors in each object in the first point cloud, and determine whether the number of nearest neighbors in each object meets the preset conditions. If so, the two objects are the same object, and there are multiple objects with the same object in the two adjacent point clouds.

[0160] In this embodiment, by determining the number of nearest neighbors for each object in the first point cloud, it is judged whether the number of nearest neighbors for each object meets a preset condition. If it does, the two objects can be the same object. The preset condition is that the number of nearest neighbors is greater than half the number of data points of any one of the objects in the first and second point clouds, or the number of nearest neighbors is greater than a preset value. For example, there is object A in the first point cloud with N data points, and objects B and C in the second point cloud with M and T data points respectively. After calculation, there are L nearest neighbors between object A and object B, and P nearest neighbors between object A and object C. After judgment, the L nearest neighbors between object A and object B are greater than half the M data points in object B, so object A and object B are the same object. The P nearest neighbors between object A and object C do not meet the preset condition, so the two objects are not the same object.

[0161] One possible implementation is, such as Figure 3 As shown, the first point cloud includes objects A and B, and the second point cloud includes objects C and D. Assume object A contains *a* data points, object B contains *b* data points, object C contains *c* data points, and object D contains *d* data points. Then, calculate the data point distances between object A in the first point cloud and objects C and D in the second point cloud, and between object B in the first point cloud and objects C and D in the second point cloud. This yields *c* data point distances between objects A and C, *d* data point distances between objects A and D, *c* data point distances between objects B and C, and *d* data point distances between objects B and D. Then, calculate the multiple data point distances for each object in the first and second point clouds. The system performs a judgment. It finds that there are 10 nearest neighbors among the data points in objects A and C (c points), 6 nearest neighbors among the data points in objects D (d points), 5 nearest neighbors among the data points in objects B and C (c points), and 40 nearest neighbors among the data points in objects B and D (d points). It then checks if the number of nearest neighbors for each object meets a preset condition. The system finds that the number of nearest neighbors among objects B and D meets the preset condition because there are identical objects between the two point clouds. However, objects A and C, A and D, and B and C are all different objects. Therefore, objects B and D are merged to obtain object H, which contains b+d data points.

[0162] The open-vocabulary 3D panoramic segmentation method provided in this application involves: acquiring RGB images of multiple viewpoints of a target panoramic segmentation scene; obtaining a first instance segmentation result for each viewpoint's RGB image using an unsupervised instance segmentation algorithm; reassigning the numbers in the first instance segmentation results of each viewpoint's RGB image using a Hungarian matching algorithm to obtain a second instance segmentation result for each viewpoint's RGB image; training an instance segmentation network model based on the first and second instance segmentation results of the multiple viewpoints' RGB images to obtain the camera parameter information of the viewpoint to be predicted in the target panoramic segmentation scene; inputting the camera parameter information of the viewpoint to be predicted into the preset instance segmentation network model to obtain the predicted instance segmentation result for the viewpoint to be predicted; determining the object category corresponding to each instance number in the predicted instance segmentation result based on a pre-acquired voting matrix; determining the predicted semantic segmentation result for the viewpoint to be predicted based on each object category; and determining the predicted panoramic segmentation result for the viewpoint to be predicted based on the predicted instance segmentation result and the predicted semantic segmentation result. This achieves accurate labeling of objects in the target 3D scene, making the panoramic segmentation result more accurate, and providing an innovative idea and method for the field of open-vocabulary 3D scene panoramic segmentation.

[0163] Figure 6 A schematic diagram of the structure of the open vocabulary 3D panoramic segmentation device provided in this application is shown below. Figure 6 As shown, the open vocabulary 3D panoramic segmentation device 60 provided in this embodiment includes:

[0164] The acquisition module 601 is used to acquire camera parameter information of the viewpoint to be predicted in the target 3D scene;

[0165] The first processing module 602 is used to input the camera parameter information of the viewpoint to be predicted into a preset instance segmentation network model to obtain the predicted instance segmentation result of the viewpoint to be predicted; wherein, the predicted instance segmentation result includes: multiple objects and the instance number corresponding to each object, and the instance number corresponding to each object is different.

[0166] The second processing module 603 is used to determine the object category corresponding to each instance number in the predicted instance segmentation result based on the pre-acquired voting matrix. The voting matrix includes the object categories and instance numbers of multiple objects from different perspectives.

[0167] The second processing module 603 is further configured to determine the predicted semantic segmentation result of the viewpoint to be predicted based on each object category; wherein the predicted semantic segmentation result includes a category number corresponding to each object category;

[0168] The third processing module 604 is used to determine the predicted panoramic segmentation result of the view to be predicted based on the predicted instance segmentation result and the predicted semantic segmentation result.

[0169] In one possible implementation, the first processing module 602 is further configured to:

[0170] Acquire RGB images of the target 3D scene from multiple perspectives;

[0171] Based on the unsupervised instance segmentation algorithm, the first instance segmentation result of the RGB image for each viewpoint is obtained;

[0172] The instance numbers in the first instance segmentation results of the RGB images from each viewpoint are reassigned based on the Hungarian matching algorithm to obtain the second instance segmentation results of the RGB images from each viewpoint.

[0173] Based on the first instance segmentation results and the second instance segmentation results of RGB images from multiple perspectives, the instance segmentation network model to be trained is obtained.

[0174] In one possible implementation, the first processing module 602 is further configured to:

[0175] Acquire RGB images of the target 3D scene from multiple perspectives;

[0176] Based on the unsupervised instance segmentation algorithm, the first instance segmentation result of the RGB image for each viewpoint is obtained;

[0177] The instance number in the first instance segmentation result of the RGB image of each viewpoint is reassigned according to the Hungarian matching algorithm to obtain the instance segmentation result of the reassigned instance number of the RGB image of each viewpoint.

[0178] Based on the instance segmentation results obtained by reassigning instance numbers to the RGB images from each viewpoint, the instance segmentation network model to be trained is iteratively trained to obtain the instance segmentation network model.

[0179] In one possible implementation, the first processing module 602 is further configured to:

[0180] The first stage of training of the instance segmentation network model to be trained is carried out based on the instance segmentation results of the RGB images of each viewpoint with reassigned instance numbers to obtain loss function information. The instance segmentation network model to be trained is then supervised based on multiple loss function information to obtain an intermediate instance segmentation network model. The second instance segmentation result of the RGB images of each viewpoint is obtained based on the instance segmentation network model.

[0181] The instance numbers of the same object under different viewpoints in the first instance segmentation result and the second instance segmentation result are uniformly processed to obtain the third instance segmentation result of the RGB image of each viewpoint; wherein, the third instance segmentation result of the RGB image of each viewpoint includes multiple objects, each object corresponds to an instance number, and the instance number of the same object is consistent in the RGB images of different viewpoints.

[0182] A set of point clouds with the same instance number is determined based on multiple instance numbers in the third instance segmentation results of RGB images from multiple viewpoints; wherein, the set of point clouds includes multiple point clouds, and the number of point clouds with the same instance number is at least 2.

[0183] Multiple point clouds in the point cloud set are fused to determine the fourth instance segmentation result of the RGB image from each viewpoint; wherein, the fourth instance segmentation result of the RGB image from each viewpoint includes multiple objects, each object corresponds to a unique instance number, and the instance number of the same object is consistent in the RGB images from different viewpoints.

[0184] Based on the fourth instance segmentation result of the RGB image from each viewpoint, the intermediate instance segmentation network model is trained in the second stage to obtain cross-entropy loss information. The intermediate instance segmentation network model is then supervised based on multiple cross-entropy loss information to obtain the instance segmentation network model.

[0185] In one possible implementation, the first processing module 602 is further configured to:

[0186] Step a, identify two adjacent point clouds in a set of point clouds with the same instance number; wherein, the two adjacent point clouds include a first point cloud and a second point cloud, the first point cloud includes at least one object, and the second point cloud includes at least one object;

[0187] Step b: Determine whether there are any identical objects among multiple objects in two adjacent point clouds. If so, unify the instance numbers of the identical objects and merge the identical objects to determine a new instance number for the identical objects.

[0188] Step c: Identify multiple different objects between two adjacent point clouds, excluding identical objects.

[0189] Step d: Renumber the instance numbers of each different object to complete the fusion of the first point cloud and the second point cloud. The fused point cloud is the new first point cloud.

[0190] Step e: Determine the adjacent point clouds of the new first point cloud as the new second point cloud. Repeat steps b to e until all point clouds in the point cloud set with the same instance number are merged and determined as the target point cloud; wherein, each object in the target point cloud corresponds to a unique instance number.

[0191] In one possible implementation, the first processing module 602 is further configured to:

[0192] The first point is that there is at least one object in the cloud; the second point is that there is at least one object in the cloud.

[0193] Determine the distance between each data point of each object in the first point cloud and each data point of each object in the second point cloud to obtain the distance information between every two data points.

[0194] The nearest neighbor of a data point in the first point cloud is determined based on the distance information of multiple data points; wherein the number of nearest neighbors of a data point is at most 1.

[0195] Determine the number of nearest neighbors for each object in the first point cloud, and determine whether the number of nearest neighbors for each object meets the preset conditions. If so, the two objects are the same object, and there are multiple objects with the same object in the two adjacent point clouds.

[0196] In one possible implementation, the first processing module 602 is further configured to:

[0197] Based on the open vocabulary semantic segmentation algorithm, the first semantic segmentation result of the RGB image for each viewpoint is determined. The first semantic segmentation result includes multiple object categories, each object category corresponds to a category number, and the number of category numbers is less than or equal to the number of instance numbers.

[0198] Based on the first semantic segmentation result of the RGB image of each viewpoint, obtain the object category corresponding to each instance number in the fourth instance segmentation result of the RGB image of each viewpoint under different viewpoints;

[0199] A voting matrix is ​​determined based on the object category corresponding to each instance number; the voting matrix is ​​used to determine the number of objects of the same category in each instance number.

[0200] Determine the maximum number among multiple counts belonging to the same object category, and determine the target object category corresponding to the maximum number of the same object category for each instance number;

[0201] Based on each target object category, a second semantic segmentation result is determined for the RGB image of each viewpoint; wherein, the second semantic segmentation result includes a category number corresponding to each object category.

[0202] The open vocabulary 3D panoramic segmentation device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0203] Figure 7 A hardware schematic diagram of the open-vocabulary 3D panoramic segmentation device provided in this application. (For example...) Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0204] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0205] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0206] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0207] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0208] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0209] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0210] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0211] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0212] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0213] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, methods, or units, and may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0216] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0218] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An open-vocabulary 3D panoramic segmentation method, characterized in that, include: Obtain camera parameter information of the predicted viewpoint in the target 3D scene; The camera parameter information of the viewpoint to be predicted is input into a preset instance segmentation network model to obtain the predicted instance segmentation result of the viewpoint to be predicted; wherein, the predicted instance segmentation result includes: multiple objects and the instance number corresponding to each object, and the instance number corresponding to each object is different. Based on the pre-acquired voting matrix, the object category corresponding to each instance number in the predicted instance segmentation result is determined. The voting matrix includes the object category and instance number of multiple objects from different perspectives. Based on each object category, the predicted semantic segmentation result of the viewpoint to be predicted is determined; wherein, the predicted semantic segmentation result includes a category number corresponding to each object category; The predicted panoramic segmentation result of the view to be predicted is determined based on the predicted instance segmentation result and the predicted semantic segmentation result. The instance segmentation network model was trained in the following way: Acquire RGB images of the target 3D scene from multiple perspectives; Based on the unsupervised instance segmentation algorithm, the first instance segmentation result of the RGB image for each viewpoint is obtained; The instance number in the first instance segmentation result of the RGB image of each viewpoint is reassigned according to the Hungarian matching algorithm to obtain the instance segmentation result of the reassigned instance number of the RGB image of each viewpoint. Based on the instance segmentation results of the RGB images from each viewpoint with reassigned instance numbers, the instance segmentation network model to be trained is iteratively trained to obtain the instance segmentation network model.

2. The method according to claim 1, characterized in that, The instance segmentation results based on the reassigned instance numbers of the RGB images from each viewpoint are used to iteratively train the instance segmentation network model to be trained, resulting in the instance segmentation network model, including: The first stage of training of the instance segmentation network model to be trained is performed based on the instance segmentation results of the RGB images of each viewpoint with reassigned instance numbers to obtain loss function information. The instance segmentation network model to be trained is then supervised based on multiple loss function information to obtain an intermediate instance segmentation network model. The second instance segmentation result of the RGB images of each viewpoint is then obtained based on the instance segmentation network model. The instance numbers of the same object under different viewpoints in the first instance segmentation result and the second instance segmentation result are uniformly processed to obtain the third instance segmentation result of the RGB image of each viewpoint; wherein, the third instance segmentation result of the RGB image of each viewpoint includes multiple objects, each object corresponds to an instance number, and the instance number of the same object is consistent in the RGB images of different viewpoints. A set of point clouds with the same instance number is determined based on multiple instance numbers in the third instance segmentation results of RGB images from multiple viewpoints; wherein, the set of point clouds includes multiple point clouds, and the number of point clouds with the same instance number is at least 2. Multiple point clouds in the point cloud set are fused to determine the fourth instance segmentation result of the RGB image from each viewpoint; wherein, the fourth instance segmentation result of the RGB image from each viewpoint includes multiple objects, each object corresponds to a unique instance number, and the instance number of the same object is consistent in the RGB images from different viewpoints. Based on the fourth instance segmentation result of the RGB image from each viewpoint, the intermediate instance segmentation network model is trained in the second stage to obtain cross-entropy loss information. The intermediate instance segmentation network model is then supervised based on multiple cross-entropy loss information to obtain the instance segmentation network model.

3. The method according to claim 2, characterized in that, The process of fusing multiple point clouds in the point cloud set to determine the fourth instance segmentation result of the RGB image from each viewpoint includes: Step a, determine two adjacent point clouds in the set of point clouds with the same instance number; wherein, the two adjacent point clouds include a first point cloud and a second point cloud, the first point cloud includes at least one object, and the second point cloud includes at least one object; Step b: Determine whether there are any identical objects among the multiple objects in the two adjacent point clouds. If so, unify the instance numbers of the identical objects and merge the identical objects to determine a new instance number for the identical objects. Step c: Identify multiple different objects between the two adjacent point clouds, excluding the same objects; Step d: Renumber the instance numbers of each different object to complete the fusion of the first point cloud and the second point cloud. The fused point cloud is the new first point cloud. Step e: Determine the adjacent point clouds of the new first point cloud as the new second point cloud, and repeat steps b to e until all point clouds in the point cloud set with the same instance number are fused and determined as the target point cloud; wherein, each object in the target point cloud corresponds to a unique instance number.

4. The method according to claim 3, characterized in that, There is at least one object in the first point cloud and at least one object in the second point cloud; Accordingly, determining whether there are identical objects among multiple objects in the two adjacent point clouds includes: Determine the distance between each data point in each object in the first point cloud and each data point in each object in the second point cloud to obtain the distance information between every two data points; The nearest neighbor point in the first point cloud is determined based on the distance information of multiple data points; wherein, the number of nearest neighbors corresponding to the data point is at most 1. Determine the number of nearest neighbors for each object in the first point cloud, and determine whether the number of nearest neighbors for each object meets a preset condition. If so, the two objects are the same object, and there are multiple objects with the same object in the two adjacent point clouds.

5. The method according to claim 4, characterized in that, After determining the fourth instance segmentation result of the RGB image for each viewpoint, the method further includes: Based on the open vocabulary semantic segmentation algorithm, a first semantic segmentation result is determined for the RGB image of each viewpoint. The first semantic segmentation result includes multiple object categories, each object category corresponds to a category number, and the number of category numbers is less than or equal to the number of instance numbers. Based on the first semantic segmentation result of the RGB image of each viewpoint, obtain the object category corresponding to each instance number in the fourth instance segmentation result of the RGB image of each viewpoint under different viewpoints; The voting matrix is ​​determined based on the object category corresponding to each instance number; wherein, the voting matrix is ​​used to determine the number of each identical object category in the object category corresponding to each instance number; Determine the maximum number among multiple counts belonging to the same object category, and determine the object category corresponding to the maximum number as the target object category corresponding to each instance number; Based on each target object category, a second semantic segmentation result is determined for the RGB image of each viewpoint; wherein, the second semantic segmentation result includes a category number corresponding to each object category.

6. An open-vocabulary 3D panoramic segmentation device, characterized in that, include: The acquisition module is used to acquire camera parameter information of the viewpoint to be predicted in the target 3D scene; The first processing module is used to input the camera parameter information of the viewpoint to be predicted into a preset instance segmentation network model to obtain the predicted instance segmentation result of the viewpoint to be predicted; wherein, the predicted instance segmentation result includes: multiple objects and the instance number corresponding to each object, and the instance number corresponding to each object is different. The second processing module is used to determine the object category corresponding to each instance number in the predicted instance segmentation result based on the pre-acquired voting matrix. The voting matrix includes the object categories and instance numbers of multiple objects from different perspectives. The second processing module is used to determine the predicted semantic segmentation result of the viewpoint to be predicted based on each object category; wherein, the predicted semantic segmentation result includes a category number corresponding to each object category; The third processing module is used to determine the predicted panoramic segmentation result of the view to be predicted based on the predicted instance segmentation result and the predicted semantic segmentation result. The first processing module is also used for: Acquire RGB images of the target 3D scene from multiple perspectives; Based on the unsupervised instance segmentation algorithm, the first instance segmentation result of the RGB image for each viewpoint is obtained; The instance number in the first instance segmentation result of the RGB image of each viewpoint is reassigned according to the Hungarian matching algorithm to obtain the instance segmentation result of the reassigned instance number of the RGB image of each viewpoint. Based on the instance segmentation results of the RGB images from each viewpoint with reassigned instance numbers, the instance segmentation network model to be trained is iteratively trained to obtain the instance segmentation network model.

7. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images