Instance segmentation method and apparatus

Abstract

Provided in the present application are an instance segmentation method and apparatus in the field of computer vision, which are used for aligning, on the basis of a two-dimensional instance in an open scene, primitives segmented from a three-dimensional model, and then outputting consistent three-dimensional instance segmentation, such that even if the view is switched, the same three-dimensional instance can be segmented. The method comprises: acquiring at least one image of a target scene, a camera view of the at least one image, and a three-dimensional model of the target scene; then performing instance segmentation on the at least one image, so as to obtain a first two-dimensional instance segmentation result, wherein the first two-dimensional instance segmentation result comprises at least one two-dimensional instance; acquiring at least one primitive in the three-dimensional model, wherein the at least one primitive comprises a primitive formed by a corner point and a surface patch in the three-dimensional model; next, on the basis of the camera view, projecting the at least one two-dimensional instance and the at least one primitive, so as to determine at least one three-dimensional instance, wherein each three-dimensional instance comprises one or more primitives; and outputting an instance identifier of each three-dimensional instance.

Description

An instance segmentation method and device

[0001] The present application claims priority to the Chinese Patent Application No. 202410796620.4, filed on June 19, 2024, entitled "Scene Segmentation Method, Device, Computing Device Cluster and Storage Medium", and to the Chinese Patent Application No. 202410865517.0, filed on June 29, 2024, entitled "An Instance Segmentation Method and Device", both of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The present application relates to the field of computer vision, and in particular to an instance segmentation method and device. BACKGROUND

[0003] Reconstruction of an environment is very important for scenarios such as digital city, autonomous driving or robot application, and editability of a scene is very important for applications such as display and editing of a digital world, autonomous driving, and training and testing of robots in diverse environments. This requires that each object in the environment can be extracted individually, that is, each object needs to have a class label and an object number (or referred to as instance annotation) to facilitate editing; an environment with instance annotation can also be used for robot semantic navigation, etc.

[0004] In the instance segmentation scenario for the collected images, the same or different instances can be included between different images. For the instances included in each frame of image, a corresponding instance ID needs to be set to distinguish each instance and to further process by the downstream task. However, since the existing open scene segmentation model relies on a two-dimensional visual language model, that is, the segmentation only gives an instance ID for a single image, the same object is configured with different instance IDs in different frames, thereby resulting in that useful three-dimensional information cannot be provided. Therefore, how to realize the consistency of the instance ID in each perspective image in an open scene becomes a problem to be solved. SUMMARY

[0005] The present application provides an instance segmentation method and device for aligning the primitives segmented from a three-dimensional model based on two-dimensional instances in an open scene, and further outputting three-dimensional instance segmentation with consistency, that is, the same three-dimensional instance can be segmented even if the perspective is switched.

[0006] Therefore, in a first aspect, an example segmentation method is provided, including: obtaining at least one image of a target scene, a camera perspective of the at least one image, and a three-dimensional model of the target scene; then performing instance segmentation on the at least one image to obtain a first two-dimensional instance segmentation result, the first two-dimensional instance segmentation result including at least one two-dimensional instance; obtaining at least one primitive in the three-dimensional model, the at least one primitive including a primitive formed by a vertex and a face in the three-dimensional model; then projecting the at least one two-dimensional instance and the at least one primitive based on the camera perspective to determine at least one three-dimensional instance, the at least one three-dimensional instance including a structure formed by the at least one primitive, and each three-dimensional instance including one or more primitives; and outputting an instance identifier of the at least one three-dimensional instance, the three-dimensional instance being assigned after the three-dimensional instance is determined, or being determined based on an ID of a two-dimensional instance aligned with the three-dimensional instance.

[0007] In the embodiments of the present application, the at least one image is segmented to output two-dimensional instances, and the three-dimensional model is cut to output primitives, and the two-dimensional instances and the primitives are projected to align the two-dimensional instances and the primitives in the same projection space, to determine the primitives belonging to the same entity under the constraint of the two-dimensional instances, to implement instance segmentation combined with the three-dimensional model. Further, when applied to an open scene, three-dimensional consistent instance segmentation in the open scene can be implemented, that is, the consistency of each three-dimensional instance or two-dimensional instance can be maintained in two-dimensional images of different perspectives.

[0008] In a possible implementation, the foregoing determining the at least one three-dimensional instance by projecting the at least one two-dimensional instance and the at least one primitive includes: projecting the at least one two-dimensional instance and the at least one primitive based on the camera perspective, that is, projecting the two-dimensional instances and the primitives into the same projection space based on the camera perspective, and obtaining an overlapping area between each two-dimensional instance and each primitive in the projection space; grouping one or more primitives with an overlapping area greater than a preset area of the same two-dimensional instance into a three-dimensional instance to obtain the at least one three-dimensional instance.

[0009] In the embodiments of the present application, the two-dimensional instances and the primitives segmented from the three-dimensional model can be projected into the same space, and whether the instances and the primitives represent the same entity is identified through the overlapping area between the two-dimensional instances and the primitives, to accurately segment the instances in the three-dimensional model in a geometric manner, and to implement instance consistency in two-dimensional and three-dimensional.

[0010] In a possible implementation, the aforementioned projecting the at least one two-dimensional instance and the at least one primitive, and obtaining an overlapping area between each two-dimensional instance and each primitive in a projection space can include: projecting the at least one primitive into an imaging plane of the two-dimensional instance, and obtaining the overlapping area between each two-dimensional instance and each primitive in the imaging plane. In the implementation of the present application, the primitive can be projected into the plane corresponding to the two-dimensional instance, so that the primitive and the two-dimensional instance are in the same dimension, to facilitate the calculation of the overlapping area between the two-dimensional instance and the primitive, and to accurately identify the primitive representing the same entity, and to realize the three-dimensional instance segmentation.

[0011] In a possible implementation, the aforementioned instance segmentation of the at least one image to obtain the first two-dimensional instance segmentation result can include: obtaining input text corresponding to the at least one image, the input text including text for indicating segmentation of the at least one image, the input text being user input text including prompt words in the instance segmentation, such as the category or name of the entity to be segmented; and then inputting the at least one image and the input text into a large visual language model to output information of the at least one two-dimensional instance. In the implementation of the present application, the language understanding capability of the existing language model can be used for image segmentation, without additional training cost.

[0012] In a possible implementation, the aforementioned method can further include: obtaining a plurality of second two-dimensional instance segmentation results, the same instance in each of the plurality of second two-dimensional instance segmentation results having the same instance identifier; and training a neural network model based on the plurality of second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene. In the implementation of the present application, the three-dimensional implicit representation can be trained in the process of three-dimensional instance segmentation. Generally, the two-dimensional images under different perspectives can have hidden or shape-changed entities, and in the implementation of the present application, the segmentation maps corresponding to multiple images can be accumulated to train the three-dimensional implicit representation, so that the shapes of the entities under different perspectives or different frames can be learned, and thus a more accurate three-dimensional implicit representation including instance segmentation can be obtained, such as a two-dimensional rendering map matching the actual scene even when switching to any different perspective in an open scene.

[0013] In a possible implementation, the aforementioned obtaining the information of the at least one primitive in the three-dimensional model can include: clustering the corner points or the patches in the three-dimensional model according to the normal vectors of the surface of the three-dimensional model, regarding the corner points and / or the patches in a group of clustering categories as one primitive, and obtaining the information of the at least one primitive. In the implementation of the present application, the intersection points or the patches can be segmented based on the normal vectors of the surface of the three-dimensional model, and the primitives can be segmented through clustering, so that the instance segmentation of the primitive granularity is realized, and more accurate three-dimensional instance segmentation results can be output in combination with the entity granularity segmentation under the two-dimensional instance as a constraint.

[0014] In a possible implementation, the aforementioned obtaining the three-dimensional model can include: obtaining the three-dimensional model from the user input data, or obtaining the three-dimensional model through three-dimensional construction based on the multiple frames of images of the target scene. In the implementation of the present application, the three-dimensional model can be directly input by the user, or can be obtained through three-dimensional reconstruction based on the multiple frames of images, so as to adapt to various scenes and have stronger generality.

[0015] In a possible implementation, the aforementioned method further includes: outputting the three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating at least one three-dimensional instance in the three-dimensional model; or outputting at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result including at least one two-dimensional instance, and the instance identifier of each two-dimensional instance being the same as the instance identifier of the three-dimensional instance representing the same entity.

[0016] In the implementation of the present application, the two-dimensional instance segmentation result and / or the three-dimensional instance segmentation result can be output, and the type of output data can be selected according to the actual application scene, so that the method can be applied to more scenes and has stronger generality.

[0017] In a second aspect, the embodiments of the present application provide an instance segmentation device, which includes:

[0018] The input module is configured to obtain at least one image of a target scene, a camera perspective of the at least one image, and a three-dimensional model of the target scene.

[0019] The two-dimensional segmentation module is configured to perform instance segmentation on the at least one image to obtain a first two-dimensional instance segmentation result, the first two-dimensional instance segmentation result including at least one two-dimensional instance.

[0020] The three-dimensional segmentation module is configured to obtain at least one primitive in the three-dimensional model, the at least one primitive including a primitive formed by a corner point and a patch in the three-dimensional model.

[0021] The projection module is configured to project the at least one two-dimensional instance and the at least one primitive through the camera perspective to determine at least one three-dimensional instance, the at least one three-dimensional instance including a structure formed by the at least one primitive.

[0022] an output module configured to output an instance identifier of each three-dimensional instance.

[0023] Effects of the second aspect or any possible implementation of the second aspect can refer to the effects of the first aspect or any possible implementation of the first aspect, which will not be repeated here.

[0024] In a possible implementation, the projection module is configured to: project the at least one two-dimensional instance and the at least one primitive based on a camera view angle, and obtain an overlapping area between each two-dimensional instance and each primitive in a projection space; and group one or more primitives with an overlapping area greater than a preset area to the same two-dimensional instance to obtain at least one three-dimensional instance.

[0025] In a possible implementation, the projection module is specifically configured to: project the at least one primitive into an imaging plane of the two-dimensional instance, and obtain an overlapping area between each two-dimensional instance and each primitive in the imaging plane.

[0026] In a possible implementation, the two-dimensional segmentation module is specifically configured to: obtain an input text, the input text including a text for instructing segmentation of the image; and input the image and the input text as an input of a large visual language model, and output a first two-dimensional instance segmentation result.

[0027] In a possible implementation, the device further includes a training module configured to: obtain a plurality of second two-dimensional instance segmentation results, each second two-dimensional instance segmentation result in the plurality of second two-dimensional instance segmentation results having a same instance identifier of a same instance; and train a neural network model based on the plurality of second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene.

[0028] In a possible implementation, the three-dimensional segmentation module is specifically configured to: cluster corner points or face patches in the three-dimensional model according to normal vectors of surfaces of the three-dimensional model, and obtain the at least one primitive by taking corner points and / or face patches in a group of clustering categories as a primitive.

[0029] In a possible implementation, the input module is specifically configured to: obtain the three-dimensional model from user input data; or perform three-dimensional reconstruction based on a plurality of images of the target scene to obtain the three-dimensional model.

[0030] In a possible implementation, the output module is further configured to: output a three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating the at least one three-dimensional instance in the three-dimensional model; or output at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result including the at least one two-dimensional instance, and each two-dimensional instance having a same instance identifier as an instance identifier of a three-dimensional instance representing a same entity.

[0031] In a third aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device performs the method steps in the first aspect and any implementation manner thereof.

[0032] In a fourth aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method steps in the first aspect and any implementation manner thereof.

[0033] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method in the first aspect or any implementation manner of the first aspect.

[0034] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer program instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method in any implementation manner of the first aspect, the second aspect, the third aspect or the fourth aspect.

[0035] In a seventh aspect, an embodiment of the present application provides a chip, comprising at least one processor and an interface; the at least one processor acquires program instructions or data through the interface; the at least one processor is configured to execute the program instructions to implement the method in the first aspect or any implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0036] FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0037] FIG. 2 is a schematic diagram of a flow of an instance segmentation method provided by an embodiment of the present application;

[0038] FIG. 3 is a schematic diagram of a flow of another instance segmentation method provided by an embodiment of the present application;

[0039] FIG. 4 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0040] FIG. 5 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0041] FIG. 6 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0042] FIG. 7 is a schematic diagram of a structure of an instance segmentation apparatus provided by an embodiment of the present application;

[0043] FIG. 8 is a structural schematic diagram of a computing device provided in the present application;

[0044] FIG. 9 is a structural schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0045] FIG. 10 is a structural schematic diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0047] The method provided in the present application can be applied in an artificial intelligence (AI) scenario. For ease of understanding, some terms or concepts related to the present application are introduced below.

[0048] (1) Large model

[0049] A large model is a large-scale model. The largeness of the large model can be reflected in multiple aspects, such as large data scale, large-scale parallel computing capability, and larger model structure, etc.

[0050] (2) Language model (LM)

[0051] A language model is used for natural language processing (NLP) and plays an important role in NLP. Its task is to predict the probability of a sentence appearing in a language. For example, a language model is usually constructed as a probability distribution p(s) of a string s, where p(s) attempts to reflect the frequency of the string s as a sentence. It can be applied to scenarios such as text recognition or machine translation. In the embodiments of the present application, the NLP model mentioned below can include a language model.

[0052] (3) Large language model (LLM)

[0053] Large Language Model (LLM) refers to a language model containing hundreds of billions (or more) of parameters, which are trained on a large amount of text data. LLM is a deep learning-based natural language processing model. These models can process large amounts of text data, thereby learning the grammar and semantic rules of natural language. LLMs can be applied to text generation, machine translation, question-answering systems, text summarization, sentiment analysis, etc., and have the advantages of strong generation ability, strong adaptability, accurate prediction, and strong scalability. For example, in the movie recommendation scenario, a large language model can generate a description of the movie scene, including movie type, lead actor, plot, etc., so that the system can better recommend similar movies. A large language model can also generate a recommendation reason, such as an e-commerce website can generate a reason for recommending a product through a large language model, such as the quality, price, and functionality of the product, so that users can better understand the value of the product.

[0054] (4) Pre-trained language model (PLM)

[0055] A natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. The training of PLM consists of two stages, pre-training and fine-tuning. In the pre-training stage, the model is trained on large-scale unsupervised text for language model tasks, learning the word representation method. In the fine-tuning stage, the model uses the parameters learned in the pre-training stage as initialization, such as training for a few steps on downstream tasks such as text classification or sequence labeling, so that the semantic information learned in pre-training can be successfully transferred to downstream tasks.

[0056] (5) Large Vision-Language Model (LVLM)

[0057] Also known as visual large language model, visual language large model, etc. It is usually composed of a visual encoder (such as CLIP) and a language model (LM), the former explains the encoded features, and the latter solves downstream tasks. In the embodiments of the present application, the language model in LVLM can be a pre-trained language model or a large language model. LVLM combines the capabilities of large language models (LLMs) to handle complex tasks involving vision and natural language.

[0058] (6) Loss function

[0059] Also known as cost function, a measure of the difference between the predicted output of the machine learning model and the true value (also known as supervised value) of the sample, i.e. for measuring the difference between the predicted output of the machine learning model and the true value of the sample. The loss function can generally include error square mean square, cross entropy, logarithm, exponential, etc. For example, the error mean square can be used as the loss function, defined as Specifically, the specific loss function can be selected according to the actual application scene.

[0060] (7) Gradient: Derivative vector of loss function with respect to parameters.

[0061] (8) Instance segmentation

[0062] Image segmentation refers to dividing an image into several mutually exclusive regions according to features such as gray scale, color, spatial texture, geometric shape, etc. so that these features show consistency or similarity within the same region, and show obvious difference between different regions. Instance segmentation is a combination of object detection and semantic segmentation, which detects objects in the image (object detection), and then labels each pixel (semantic segmentation).

[0063] (9) Graph reasoning

[0064] A graph is usually composed of nodes and edges connecting the nodes, and graph reasoning is to form a connected group of nodes by calculating the edges between the nodes in the graph.

[0065] (10) Three-dimensional reconstruction

[0066] It refers to establishing a mathematical model suitable for computer representation and processing for a three-dimensional object, which is the basis for processing, operating and analyzing its properties in a computer environment, and is equivalent to simulating a three-dimensional entity in a virtual three-dimensional space. That is, based on multiple frames of images and camera parameters corresponding to the images, the pose corresponding to each frame of image is determined, and the coordinates of the pixel points of each frame of image in the three-dimensional space are determined according to the pose, and a three-dimensional model is reconstructed.

[0067] (11) Neural radiance field (NeRF)

[0068] NeRF is a way or model that uses neural networks to implicitly represent a 3D scene. NeRF can be used to learn a scene that is implicitly saved in the parameters of the NeRF neural network, i.e. the implicit representation of the scene. If a new perspective picture is needed, the light and color values of each position of the picture can be calculated using NeRF, and the new perspective picture can be output after rendering.

[0069] (12) Open vocabulary scene

[0070] Also referred to as an open scene, an open world, an open vocabulary three-dimensional scene, etc., refers to a scene of a virtual world reconstructed for an open world, including a model obtained by three-dimensional reconstruction based on three-dimensional data. In an open vocabulary scene, a user can freely view the world from the scene based on any view angle, any position, or any vocabulary, etc. The open vocabulary scene can be applied in a digital city, autonomous driving, a robot, or a game, etc. scene to provide a user with a visual experience simulating a real world.

[0071] In an intelligent scene such as a digital city, autonomous driving, or a robot, the perception of the environment around the robot includes three-dimensional segmentation of instances in any scene in which the robot is located, and assigning IDs to the three-dimensional instances segmented, which is very important for downstream tasks. The downstream task distinguishes instances by the IDs of the instances, and further performs the downstream task based on each instance. For example, for 3D instance segmentation of an open world, the same instance may be assigned different IDs in different images, or different instances may be assigned the same ID in different images, etc., which may cause the downstream task to be inaccurate. Existing solutions are mostly for closed set scene segmentation, lack model generalization, cannot adapt to new scenes, and the training data of the model needs to be manually annotated, which is difficult to obtain. In recent years, with the development of large visual language models, 2D instance segmentation in an open vocabulary scene has reached high accuracy and strong generalization. However, the 2D instance label lacks consistency in three-dimensional space. Existing methods for open vocabulary scene segmentation reconstruction mostly use linear allocation, that is, a loss matrix is constructed according to a feature loss function to reallocate the IDs of different instance masks in each image to align the 2D instance IDs. However, due to the lack of prior information of the three-dimensional scene and the limited two-dimensional constraint view angle, the labels of different instances in space that have never appeared in the same frame are repeated.

[0072] For example, in an existing solution, in the training phase of the segmentation model, linear matching is performed between 2D instances and 3D instances to achieve end-to-end alignment and reconstruction of 2D instance labels. However, the 3D instance ID is completely limited by the number of IDs initialized in the three-dimensional scene. Due to the low randomness of the initialization output of the MLP network constituting the Neural Field, different objects appearing in different frames in the scene have repeated instance IDs. Moreover, the segmentation model usually uses a pre-trained mask2former model, which has poor generalization.

[0073] For another example, in an existing solution, target tracking is performed on each instance to maintain consistency of the instance IDs in the image. However, the effect of target tracking is very limited. When the image view angle changes quickly or the scene folds and past objects are seen again, errors may occur, which in turn affects the accuracy of the downstream task.

[0074] Therefore, the embodiment of the present application provides an instance segmentation method, which uses a three-dimensional model of a scene as geometric prior information, aligns a 2D instance segmented from an image with a primitive segmented from the three-dimensional model, and identifies a three-dimensional instance, so that each instance can be accurately identified and distinguished, and the ID of each instance in the scene is always accurate.

[0075] The system architecture and method steps provided by the present application are described below.

[0076] Firstly, the method provided by the present application can be applied to three-dimensional reconstruction of a scene, and by combining the instance segmentation result of a two-dimensional image, the method aligns the result with a primitive in a three-dimensional model, so as to segment an instance in the three-dimensional model under the constraint of the two-dimensional instance, and realize three-dimensional consistent instance segmentation in an open scene.

[0077] The method provided by the present application can be deployed in various electronic devices, such as a server cluster, a cloud platform, a personal computer, a smart phone, or a smart car.

[0078] In a possible implementation, the method provided by the present application can be deployed in a personal computer, a computer workstation, a smart phone, a tablet computer, a notebook computer, and a smart car, and a user can directly use the computing device to perform the method provided by the embodiment of the present application to realize three-dimensional instance segmentation.

[0079] For example, the method provided by the present application can be applied to a user device, such as a smart phone, a tablet computer, or a device that can deploy OCR recognition. The user can use the user device to scan a scene that needs to be segmented and reconstructed, such as using a camera, a depth camera, or a laser device of the user device to obtain scanning data of the scene that needs to be segmented. Then, the method provided by the present application is used to perform three-dimensional reconstruction and instance segmentation on the scene.

[0080] In a possible implementation, the method provided by the present application can be deployed in a server cluster, a cloud platform, or other devices with computing capability.

[0081] For example, the method provided by the present application can be deployed in a cloud platform, and the cloud service is used to provide services for users.

[0082] For example, FIG. 1 shows a structure schematic diagram of a cloud service system provided by the present application. As shown in FIG. 1, the cloud service system 10 can include a computing device 11 and a client 12.

[0083] The computing device 11 can specifically include a server cluster or a cloud platform, or can be other computing devices. Optionally, the computing device 11 can be combined with other computing devices, such as data storage, routers, load balancers, and the like. The computing device 11 can use data in the data storage system or call program code in the data storage system to implement the method steps provided in the embodiments of the present application.

[0084] The computing device 11 can provide services for users in the form of a client. The user can operate in the client 12 to implement data interaction with the computing device 11 or request the computing device 11 to provide services. The client can be specifically deployed on a personal computer, a computer workstation, a smart phone, a tablet computer, a notebook computer, and a smart car, and the like.

[0085] In an implementation, the computing device 11 is configured to implement the method provided in the embodiments of the present application to implement segmentation of at least one image for the client and feed back information of segmented instances to the client.

[0086] Optionally, the client 12 is an optional device. In actual application scenarios, the computing device can read input data from a storage space of the computing device itself or from a data center, perform three-dimensional instance segmentation on the input data, and output an instance segmentation result containing three-dimensional instances.

[0087] In general, the method provided in the embodiments of the present application can be applied to an electronic device, that is, the aforementioned computing device 11 can be various electronic devices, such as a server cluster, a cloud platform, a personal computer, a smart phone, or a smart car, and the like.

[0088] In combination with the architecture shown in the aforementioned FIG. 1, the method provided in the present application can be executed by the computing device 11, which can specifically include a server cluster or a cloud platform, can provide services for users in the form of a client, or can be other computing devices with computing capabilities.

[0089] For example, the method provided in the present application can be deployed on a cloud platform to provide a user with a three-dimensional consistent image segmentation service or a three-dimensional mesh model instance segmentation service through a client. When the user obtains a three-dimensional mesh model of a target scene through other means, the user can directly input at least one image collected in the target scene and a three-dimensional mesh model of the corresponding scene on the client; if the user cannot directly obtain the three-dimensional mesh model, at least one image and depth data of the target scene can be collected, and a reconstruction option (for example, a button for setting a three-dimensional model reconstruction) provided by the embodiment of the present application is selected to request the generation of a three-dimensional mesh model. The at least one image can include an image of the target scene taken by an image sensor of a user device, and the three-dimensional mesh model can include a depth map calculated by three-dimensional laser scanning or a binocular camera, and the input information is transmitted to the cloud platform, and the cloud platform performs three-dimensional instance segmentation through the method provided in the present application to output an instance identifier of each three-dimensional instance. Optionally, a three-dimensional instance segmentation result and / or a two-dimensional instance segmentation result can also be output synchronously.

[0090] Next, the method provided in the present application will be described in combination with the foregoing architecture.

[0091] Referring to FIG. 2, a flowchart of an instance segmentation method provided in an embodiment of the present application is as follows.

[0092] 201, at least one image of a target scene, a camera view of the at least one image, and a three-dimensional model of the target scene are obtained.

[0093] The at least one image is obtained by collecting the target scene according to the camera view, and the camera views of the multiple images can be the same or different when there are multiple images.

[0094] Based on the system architecture corresponding to the foregoing FIG. 1, the at least one image can be sent to the cloud platform after being input by the user through the client. The at least one image can be obtained by the user by using the client to take a picture, or can be obtained by reading from the storage of the client. For example, in combination with the architecture of the foregoing FIG. 1, the client can be deployed on a terminal of the user, and the user can take a picture through the terminal to obtain the at least one image.

[0095] The three-dimensional model is a three-dimensional reconstruction model of the target scene, and can be used to simulate the entity distribution of the target scene.

[0096] Optionally, the three-dimensional model can be a model included in the user input data, such as a three-dimensional mesh model constructed by the user in other ways, or a three-dimensional mesh model obtained by three-dimensional reconstruction based on one or more frames of images input by the user. For example, the user can construct a three-dimensional model in advance and input it to the client. For another example, the user can collect multiple frames of images under the scene to be reconstructed, each frame of image can carry corresponding depth information, and send the multiple frames of images to the cloud platform through the client; the cloud platform can perform three-dimensional reconstruction on the received multiple frames of images to obtain the corresponding three-dimensional model.

[0097] In a possible scenario, the user can obtain multiple frames of images under the target scene through the client. For example, in combination with the architecture of the foregoing FIG. 1, the client can be deployed on the user terminal, and the user can shoot multiple frames of images carrying depth information through the terminal, and then trigger instance segmentation of the multiple frames of images to the cloud platform. After receiving the multiple frames of images carrying depth information, the cloud platform can construct a three-dimensional mesh model based on the received multiple frames of images to obtain a three-dimensional model of the environment corresponding to the time when the user collects the images.

[0098] In the case where the user inputs multiple frames of images carrying depth information, the cloud platform can perform three-dimensional reconstruction on the multiple frames of images, and simultaneously perform a two-dimensional instance segmentation step on the multiple frames of images.

[0099] 202. Perform instance segmentation on at least one image to obtain a first two-dimensional instance segmentation result.

[0100] After obtaining at least one image in the target scene, the at least one image can be subjected to two-dimensional instance segmentation to obtain a two-dimensional segmentation instance result. For the sake of distinction, this is referred to as a first two-dimensional instance segmentation result. The first two-dimensional instance segmentation result can include one or more two-dimensional instances.

[0101] In a possible scenario, a large visual language model can be utilized to perform the segmentation task of the at least one image. For example, input text corresponding to the at least one image can also be obtained, and the input text can include instruction or prompt text for indicating segmentation of the at least one image, such as a category or name of an instance to be segmented. The input text and the at least one image can be input into the large visual language model, the large visual language model can understand the input text and detect corresponding objects in the image, and output one or a first two-dimensional instance segmentation result, which includes information of multiple two-dimensional instances, such as a category, a detection probability (or confidence), and a segmentation pixel range of each two-dimensional instance. The large visual language model mentioned in the embodiments of the present application can be a large language model or a multi-modal large language model including a visual encoder, which has the processing capability of the at least one image and the language understanding capability, and is usually a trained model, so as to realize zero-training-cost two-dimensional image instance segmentation.

[0102] 203. Obtain at least one primitive in the three-dimensional model.

[0103] Generally, the three-dimensional model can include basic geometric units of various dimensions, such as corner points or face patches, that is, primitives. One or more primitives can be segmented from the three-dimensional model, and information of the one or more primitives can be output. For example, the shape, size, center point coordinate, or corner point coordinate of each primitive.

[0104] The three-dimensional model generally includes one or more instances in a target scene, and each instance can generally be composed of one or more primitives. Therefore, a segmentation algorithm or a segmentation network can be utilized to segment the three-dimensional model, and the primitives in the three-dimensional model can be output, so that subsequent three-dimensional instance segmentation can be implemented in the granularity of the primitives.

[0105] Optionally, the corner points and the face patches in the three-dimensional mesh model can be clustered according to the normal vectors, and the corner points in a group of clustering categories can be taken as a three-dimensional instance segmentation primitive. The three-dimensional instance mentioned below can be a combination shape composed of one or more primitives.

[0106] It should be noted that the execution order of steps 202 and 203 is not limited in the embodiments of the present application. Step 202 can be executed first, step 203 can be executed first, or steps 202 and 203 can be executed simultaneously. The specific execution order can be determined according to actual application scenarios.

[0107] 204. Project the at least one two-dimensional instance and the at least one primitive through a camera view angle to determine at least one three-dimensional instance.

[0108] The primitives can be projected with the two-dimensional instances based on a camera view angle of each two-dimensional image, and the primitives can be divided into one or more three-dimensional instances under supervision of a coverage range of the two-dimensional instances in the projection space, or the at least one three-dimensional instance can include a combined structure formed by at least one primitive of the foregoing, and each three-dimensional instance can include a structure formed by one or more primitives.

[0109] Specifically, the two-dimensional instances and the primitives can be projected into a same projection space based on a camera view angle corresponding to each image. The projection space can be a 2D space or a 3D space, for example, a three-dimensional space corresponding to a three-dimensional model or a 2D space corresponding to at least one image. One or more primitives with an overlapping area greater than a preset area with a same two-dimensional instance can form a three-dimensional instance, and at least one three-dimensional instance can be obtained. This is equivalent to aligning the primitives with the two-dimensional instances, and the primitives aligned with a same two-dimensional instance can form a three-dimensional instance, so that the primitives segmented from the three-dimensional model can be grouped, and the three-dimensional instances can be segmented under supervision of the two-dimensional instances.

[0110] Optionally, the at least one primitive can be projected into a plane corresponding to the two-dimensional instance, and an overlapping area between each two-dimensional instance and each primitive in the plane can be obtained. Therefore, in the embodiments of the present application, the primitives in the three-dimensional model can be projected into a plane where the two-dimensional instances are located, so that the two-dimensional instances and the primitives of the three-dimensional model can be aligned in the same two-dimensional plane. This is equivalent to using the two-dimensional instances as constraints or supervision to determine the primitives belonging to a same entity, so that the three-dimensional instances can be more accurately identified.

[0111] Further, the camera view angle is used to represent a pose of a camera in a target scene (i.e., a three-dimensional scene) when an image is captured, and can specifically include a field of view angle or a focal length of the camera, and the like. For example, the process of capturing an image can be simulated as a process of viewing a scene from a human eye view angle. The scene observed from different view angles can be different, and during the capturing process, entities in a field of view of the camera can be projected from a view direction to an imaging plane. Accordingly, when performing projection, the projection plane, i.e., the imaging plane of the camera, can be determined based on the camera view angle, and the entities in the three-dimensional model can be projected into the imaging plane, so that the primitives and the two-dimensional instances can be in the same plane, and the primitives and the two-dimensional instances can be aligned in the same dimension.

[0112] Of course, the two-dimensional instances can also be projected into a three-dimensional space. For example, if the at least one image carries depth information, the two-dimensional instances can be projected into a three-dimensional space corresponding to the three-dimensional model based on a depth corresponding to the two-dimensional instances, so that the primitives belonging to a same instance in the three-dimensional model can be identified by an overlapping area or a volume of the two-dimensional instances and the primitives in the three-dimensional space after projection, i.e., the three-dimensional instances can be more accurately identified from a geometric perspective.

[0113] Further, after projecting the primitives into the projection space, a graph can be constructed based on the projected primitives, where the constructed graph is an instance segmentation graph, which is composed of nodes and edges connecting the nodes. The center point of each primitive in the projection space is taken as a node in the instance segmentation graph, and the connection lines between the nodes are taken as edges in the instance segmentation graph. According to the overlapping area of each primitive and the corresponding two-dimensional instance, it is determined whether to retain the edge connecting the nodes. For example, if two primitives have no overlapping area or the overlapping area is 0 or the overlapping area is less than a preset area with a two-dimensional instance, the edge between the two primitives can not be retained; that is, the nodes and edges having a connection relationship within the same two-dimensional instance range are taken as a three-dimensional instance, so as to output an instance segmentation graph containing a three-dimensional instance.

[0114] Therefore, in the embodiments of the present application, at least one image and a three-dimensional model are segmented respectively, and the 2D instances and the primitives in the three-dimensional model are aligned from the geometric dimension, so as to identify the 3D instances composed of the primitives under the supervision of the 2D instances, and realize the ID alignment of the instances in the at least one image. Even in images of different perspectives, the 2D instances corresponding to the 3D instances can be identified, so as to realize the ID allocation of different instances in the same image and the ID configuration of the same instance in different images, so as to maintain the consistency of the instance IDs under different perspectives. The downstream task can obtain the two-dimensional instances or three-dimensional instances maintaining consistency under different perspectives, and then the perception ability of the environment can be improved, and the possibility of more complex downstream tasks can be provided.

[0115] 205、outputting the instance identification of each three-dimensional instance.

[0116] Specifically, after identifying the three-dimensional instances from the three-dimensional model, the unique identification of each three-dimensional instance can be determined, so that each three-dimensional instance can be distinguished or identified in the subsequent steps.

[0117] Optionally, when determining the ID of each three-dimensional instance, the ID of the three-dimensional instance can be allocated after identifying the three-dimensional instance, and the ID of the three-dimensional instance is also configured to the two-dimensional instances aligned with the three-dimensional instance, so as to maintain the ID consistency between the aligned three-dimensional instances and two-dimensional instances; if a unique ID is allocated to each two-dimensional instance in advance, after aligning the primitives with the two-dimensional instances, the ID of the two-dimensional instance can be taken as the ID of the three-dimensional instance composed of the primitives aligned with the two-dimensional instance, so as to obtain the ID of the three-dimensional instance.

[0118] Optionally, a three-dimensional instance segmentation result can also be output synchronously, the three-dimensional instance segmentation result indicating at least one three-dimensional instance in the three-dimensional model, and information of each three-dimensional instance can further include information such as a corner point, a position interval, and a three-dimensional mask of each instance, for representing a size, a position, and a pose of the three-dimensional instance in the three-dimensional model. Alternatively, at least one second two-dimensional instance segmentation result is output, and the at least one second two-dimensional instance segmentation result includes at least one two-dimensional instance, and an instance identifier of each two-dimensional instance is the same as an instance identifier of a three-dimensional instance representing the same entity, in other words, an instance identifier of a same instance in each of the plurality of second two-dimensional instance segmentation results is the same.

[0119] In the embodiments of the present application, a two-dimensional instance segmentation result and / or a three-dimensional instance segmentation result can be output, and the type of output data can be selected according to an actual application scenario, which can be applied to more scenarios and has stronger generalization.

[0120] In a possible implementation, a three-dimensional implicit representation can also be trained to represent a target scene through the three-dimensional implicit representation to output a rendering image at an arbitrary view angle, which is more convenient for downstream tasks. Specifically, after obtaining the plurality of second two-dimensional instance segmentation results, a neural network model such as a NeRF model or other three-dimensional reconstruction model is trained based on the plurality of second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene. In the embodiments of the present application, the three-dimensional implicit representation can be trained in the process of three-dimensional instance segmentation. Generally, two-dimensional images at different view angles can have hidden or shape-changed entities, and in the embodiments of the present application, the three-dimensional implicit representation can be trained by accumulating segmentation maps corresponding to multiple frames of images, so that the morphology of each entity at different view angles or different frames can be learned, and a more accurate three-dimensional implicit representation containing instance segmentation can be obtained, for example, in an open scene, even if the view angle is switched to any different view angle, a two-dimensional rendering image matching the actual scene can be output.

[0121] For example, for at least one image captured at different view angles, the at least one image includes entities with different poses or occluded entities, and in the embodiments of the present application, the final output can be set as a three-dimensional implicit representation. After determining an instance segmentation map including a three-dimensional instance (i.e., a three-dimensional instance) based on a two-dimensional instance (i.e., a two-dimensional instance) in each frame of at least one image as supervision, the three-dimensional implicit representation can be updated based on a difference between the instance segmentation map including the three-dimensional instance and a corresponding view angle in the three-dimensional implicit representation. The three-dimensional implicit representation can be input to a downstream task, so that the downstream task can extract a scene image at an arbitrary view angle from the three-dimensional implicit representation.

[0122] More specifically, in a three-dimensional instance segmentation process, after obtaining a three-dimensional instance segmentation map, a corresponding instance segmentation map is rendered from the three-dimensional implicit representation, and a difference between the three-dimensional instance segmentation map and the rendered instance segmentation map is calculated, and the three-dimensional implicit representation is updated based on the difference, i.e., the features in the three-dimensional instance segmentation map are learned into the three-dimensional implicit representation, so that the three-dimensional implicit representation can include richer features, such as more accurate instance segmentation under the two-dimensional instance constraint of different camera perspectives. The downstream task model can perform downstream tasks based on the three-dimensional implicit representation. For example, a two-dimensional segmentation example map under a certain perspective after rendering can be displayed on the client, or a three-dimensional implicit representation can be used for omnidirectional scene perception to make subsequent decisions, etc.

[0123] The foregoing introduces the method flow provided by the embodiments of the present application. In the following, the method steps provided by the embodiments of the present application are further introduced in combination with specific application scenarios.

[0124] The method provided by the present application can be applied to instance segmentation or instance segmentation in a three-dimensional scene, such as instance segmentation in an open scene.

[0125] Taking instance segmentation in an open scene as an example, the method application architecture provided by the present application can be as shown in FIG. 3.

[0126] Among them, the method provided by the present application can be divided into multiple stages, such as the data preprocessing stage, the graph reasoning stage or the output stage shown in FIG. 3, and the detailed steps of each stage are introduced below.

[0127] I. Data preprocessing

[0128] Among them, in the data preprocessing stage, the three-dimensional model and the at least one image can be processed respectively. Taking the mesh model of the target scene as an example for exemplary introduction, and taking any one frame of RGB image as an example for exemplary introduction.

[0129] (1) Preprocessing of mesh model of target scene

[0130] Among them, the mesh model of the target scene can come from user input data, or can be obtained based on multiple frames of RGB images and corresponding depth information. For example, in an open vocabulary scene, the mesh model of the target scene can be generated by the user through other ways, or can be constructed based on user input or by using multiple frames of RGB images and corresponding depth information, so as to obtain the mesh model of the scene.

[0131] The segmentation of the mesh model of the target scene can be performed based on the normal vector of the mesh model of the target scene. The mesh model is divided into a plurality of primitives, i.e., mesh clustering segmentation blocks. Correspondingly, each three-dimensional instance included in the mesh model is divided into one or more segmentation blocks, or one or more segmentation blocks form a three-dimensional instance.

[0132] Specifically, the Segmentator algorithm can be used to cluster the corner points and / or patches in the mesh model according to the surface normal vector of the mesh model. Generally, corner points and patches with similar normal vectors in a local area are grouped into a group, i.e., a segmentation block, and one or more segmentation blocks are output. Each segmentation block has a center point as a representative of the segmentation block. For example, the segmentation result is shown in the mesh structure in FIG. 4, where each color block represents a segmentation block, and the point on each segmentation block is the center point of the block.

[0133] Optionally, when segmenting the mesh model, the number and / or resolution of the segmentation blocks used can be a pre-set number and / or resolution, or a user-set number and / or resolution. For example, the user can set the number of segmentation blocks to 100, and when segmenting the mesh model, 100 primitives can be divided. Generally, the more the number of divided primitives, the finer the granularity of the division, and the more accurate the output result when performing instance segmentation based on primitives as the granularity.

[0134] (2) RGB image preprocessing

[0135] The RGB image can be an image input by the user, such as an image captured by the user through the client or an image input by the user on the client, etc.

[0136] In the case of constructing a mesh model based on a plurality of frames of RGB images input by the user and corresponding depths, the RGB image can be one of the plurality of frames of RGB images input by the user, or an RGB image collected by the user in the scene. The specific determination can be made according to the actual application scenario.

[0137] Specifically, the instance segmentation of the RGB image can use a pre-trained segmentation model, or use a large visual language model (or visual language large model) for instance segmentation. For example, a visual language large model can be used for 2D instance segmentation. For example, GroundedSAM can be used to add annotations to the image. The user inputs a prompt text corresponding to the instance category to be segmented and an RGB image, and GroundedSAM outputs information of the segmented 2D instance, such as a 2D instance segmentation map or a 2D instance mask, etc.

[0138] Generally, the visual language large model has strong generalization, and can realize instance segmentation in an open vocabulary scene without pre-training. For example, the rendering graph corresponding to two-dimensional instance segmentation can be as shown in FIG. 5. For example, the user inputs prompt words such as "chair, table, sofa...". The user input words and the RGB image are input into the GroundedSAM model. The GroundedSAM model can understand the input text of the user, segment the chair, table and sofa in the RGB image, and output the two-dimensional instance segmentation result.

[0139] II. Graph reasoning

[0140] On the basis of the segmentation block and the segmentation result of the 2D instance output in the foregoing preprocessing stage, graph reasoning can be performed based on the segmentation block and the 2D instance, that is, the segmentation block and the 2D instance are aligned, and an instance segmentation result is output. The instance segmentation result can include 3D instances composed of each segmentation block under the supervision of the 2D instance.

[0141] Specifically, the segmentation block and the 2D instance can be projected into the same space, and the segmentation block and the 2D instance can be aligned in the same space, so as to determine the 3D instance composed of each primitive under the constraint of the 2D instance, and obtain the instance segmentation result containing the 3D instance.

[0142] For example, as shown in FIG. 6, the segmentation block segmented from the three-dimensional model can be projected into the imaging plane of the RGB image, the overlapping area (or the intersection area) between the projection area of each segmentation block in the plane and the 2D instance is calculated, and whether the 3D instance and the 2D instance correspond to the same entity is determined based on the area.

[0143] Specifically, graph reasoning can be used to align 2D instances with segmentation blocks, thereby outputting an instance segmentation graph that includes 3D instances. For example, the 3D instances can be projected onto the imaging plane of the 2D instances to construct the instance segmentation graph, with the center points of each primitive as nodes in the graph, connecting each pair of nodes. The algorithm calculates the overlap area, or intersection area, between each primitive projected onto the 2D instance segmentation map and each 2D instance. Based on this overlap area, it determines whether to retain the edges connecting the nodes. For example, if primitive A and primitive B are connected, and the areas of both primitive A and primitive B projected onto a certain 2D instance are greater than a preset area, then the edge connecting primitive A and primitive B can be retained, meaning that primitive A and primitive B belong to the same instance, thus segmenting primitives within the same instance. However, if the overlap area between either primitive A or primitive B projected onto a certain 2D instance is not greater than the preset area, then primitive A and primitive B do not belong to the same instance, and therefore the edge connecting primitive A and primitive B is removed. This process continues until all primitives in the mesh model are traversed, outputting the final instance segmentation map representing the 3D instance.

[0144] For example, a graph can be represented as G(V,E), which is usually composed of nodes V and edges E between nodes.

[0145] Wherein, node V: In this embodiment of the application, the node is the center point of each segmented block (i.e., primitive) output after the mesh model is segmented.

[0146] Edge E: A connection edge is set between each pair of nodes during the initialization phase. For a given frame of RGB image, the node projected onto that image is defined as... Define overlapping U ij For node V i (V i ∈V I When the corresponding segmented block is projected onto the mask of this frame, it corresponds to the two-dimensional instance mask M in image I. j The percentage of the intersection area between them. For U with non-zero values. ij Node set Define the mask M in the segmentation graph j The central node is: For the remaining node V k ∈V I j If the node's U kj If it is greater than the threshold, then for V k and Voting is conducted on the edges between V. I Except for V I j Other nodes will have their values ​​deducted from V. Ij The voting of the edges. By traversing all the nodes and edges in the graph, the resulting graph composed of nodes, edges and the voting of the edges is obtained, and the edges with positive votes are retained, and the graph composed of only the edges with positive votes is output.

[0147] A three-dimensional instance segmentation graph is constructed based on a plurality of interconnected node groups in the graph, and each group of connected nodes can represent a three-dimensional instance. The output includes an instance segmentation graph containing three-dimensional instances.

[0148] Each three-dimensional instance has an ID assigned in the three-dimensional scene, i.e., an instance identifier, i.e., according to the segmentation block of the mesh represented by the node, so that each node of the mesh saves an ID, and the nodes in the same three-dimensional instance save the same ID. The mesh with the ID is rasterized and projected onto a two-dimensional instance segmentation image, and the ID of the corresponding three-dimensional instance projection is re- assigned to the two-dimensional instance according to the IoU (a standard for measuring the accuracy of detecting corresponding objects in a specific dataset) between the projection mask and the original image mask. The alignment between the two-dimensional instance mask ID and the three-dimensional instance ID is achieved, and thus the output includes a segmentation graph containing three-dimensional instances.

[0149] III. Output stage

[0150] After the foregoing stage one and stage two, the three-dimensional instances in the target scene can be segmented, and the instance identifier of each three-dimensional instance is output.

[0151] Further, the three-dimensional instance segmentation result containing the three-dimensional instance and / or the two-dimensional instance segmentation result can be output simultaneously.

[0152] The three-dimensional instance segmentation result can specifically include an instance segmentation graph containing three-dimensional instances, or point cloud data containing three-dimensional instances, or a sequence containing three-dimensional instances, etc. In the case of simultaneously outputting the instance identifier of each three-dimensional instance and the three-dimensional instance segmentation result, the instance identifier of each three-dimensional instance can be directly carried in the three-dimensional instance segmentation result, such as increasing a dimension of data to store the ID of the 3D instance in the corner point or the center point of each 3D instance.

[0153] The two-dimensional instance segmentation result can include an instance segmentation graph after aligning the ID of the two-dimensional instance with the ID of the three-dimensional instance. For example, after segmenting the three-dimensional instance, an ID can be assigned to each three-dimensional instance, and the ID of each three-dimensional instance is assigned to the two-dimensional instance aligned therewith, so that the IDs of the three-dimensional instance and the two-dimensional instance representing the same entity remain consistent. For example, a dimension of data can be added to the pixel points corresponding to each two-dimensional instance in the two-dimensional instance segmentation graph to store the ID of the 2D instance.

[0154] In a possible scenario, a three-dimensional implicit representation can also be output. For example, a NeRF model can be trained to realize the implicit representation of the target scene through the parameters of the NeRF model.

[0155] For example, in the case of inputting multiple frames of RGB by the user, the two-dimensional instance segmentation result of each RGB image is obtained, and after the aforementioned stage one and stage two, the IDs of the two-dimensional instances representing the same entity in the two-dimensional instance segmentation result of each RGB image are the same. The NeRF model is trained using the multiple two-dimensional instance segmentation results of the multiple frames of RGB images, and the trained NeRF model is output, so that the downstream task can cut a rendering image at any viewing angle based on the NeRF.

[0156] It can be understood that in the embodiments of the present application, the implicit representation for representing a three-dimensional scene can be deployed, and after obtaining the instance segmentation graph containing three-dimensional instances based on the two-dimensional instances in each frame of RGB image as supervision, the loss between the instance segmentation graph containing three-dimensional instances and the three-dimensional implicit representation can be calculated, and the three-dimensional implicit representation is updated based on the loss, so that as the number of frames of RGB images increases, the final output three-dimensional implicit representation is also more accurate, thereby outputting a more accurate three-dimensional implicit representation.

[0157] In the embodiments of the present application, the mask corresponding to the two-dimensional instance is segmented by a large visual language model, and due to the object detection-based manner, partial object detection may be missed due to occlusion caused by the viewing angle, but the corresponding shooting viewing angles between multiple frames of images may be different, so the two-dimensional instance obtained by segmentation based on different frames can be used to upgrade the two-dimensional instance to a three-dimensional space and render it again into a two-dimensional segmentation image, so as to improve the accuracy of the instances included in the two-dimensional segmentation image, thereby outputting a more accurate two-dimensional instance segmentation result and avoiding the missing detection in a single viewing angle.

[0158] In addition, the Segmentor algorithm generally roughly segments the three-dimensional instances composed of primitives and aligns them, and the edges of the segmentation result may be accurate. A more accurate implicit representation model is trained to provide more accurate three-dimensional instance segmentation, so as to improve the accuracy of the representation input to the downstream task.

[0159] In the embodiments of the present application, the instance segmentation graph containing three-dimensional instances is constructed based on the three-dimensional reconstruction model of the scene, that is, three-dimensional instance segmentation is performed in combination with the two-dimensional geometric prior of the three-dimensional scene, which can avoid classifying objects at different positions with no common frame observation or at a long distance as the same instance; at the same time, since two-dimensional segmentation is realized based on object detection, the supervision of graph segmentation comes from the two-dimensional segmentation result, thereby ensuring the instance segmentation with object as the granularity. In addition, when performing instance segmentation on at least one image, a large visual language model can be used to realize the three-dimensional instance segmentation and two-dimensional instance segmentation graph generation with zero training samples.

[0160] Further, in specific application scenarios, the output instance ID can be used for downstream tasks, such as applied to environmental perception and understanding of intelligent operating robots, and can also generate a large number of true value labels as training data for the segmentation model, or edit the scene according to the instance segmentation result in combination with a stable diffusion model.

[0161] For example, in an autonomous driving scenario, a vehicle needs to perceive the environment of the real world in which the vehicle is located, and can perform instance segmentation on the collected images by the method provided in the embodiments of the present application, to assign a unique ID to each entity in the environment, so as to identify the same entity in different image frames when perceiving the environment, and further perceive the state change of the entity, thereby making more accurate driving decisions for the vehicle, or rendering and displaying the environment in which the vehicle is located, etc.

[0162] For another example, in the environmental perception scenario of a robot, the robot can perform instance segmentation on the collected images by the method provided in the embodiments of the present application, thereby accurately distinguishing each entity, to perform a required task based on the identification result of each entity, such as a grasping operation task, reconstruction of the scene to accumulate object data assets, etc.

[0163] For another example, the method provided in the present application can be deployed in a cloud platform, to provide original data collected in the real world through a data interface specified by the platform, provide a virtual scene of the real world created by the platform and a scene instance segmentation 3D grid and 2D segmentation image, and provide them to a user, thereby providing instance segmentation services for the user.

[0164] For another example, in an open world application scenario, the method provided in the embodiments of the present application can be used to construct a three-dimensional graph by aligning a three-dimensional model with a two-dimensional instance mask of an open scene, and perform graph reasoning to output an instance segmentation graph containing three-dimensional instances, to realize three-dimensional instance segmentation and three-dimensional consistent two-dimensional instance segmentation in an open world with zero training, and output a more accurate three-dimensional model of the open world for a user.

[0165] The foregoing introduces the method flow provided by the present application, and the following introduces the device structure for executing the foregoing method flow.

[0166] Referring to FIG. 7, the structure schematic diagram of an example instance segmentation device provided by an embodiment of the present application includes:

[0167] The input module 701 is configured to acquire at least one image of a target scene, a camera perspective of the at least one image, and a three-dimensional model of the target scene.

[0168] The two-dimensional segmentation module 702 is configured to perform instance segmentation on the at least one image to obtain a first two-dimensional instance segmentation result, and the first two-dimensional instance segmentation result includes at least one two-dimensional instance.

[0169] The three-dimensional segmentation module 703 is configured to acquire at least one primitive in the three-dimensional model, and the at least one primitive includes a primitive formed by a corner point and a face sheet in the three-dimensional model.

[0170] The projection module 704 is configured to project the at least one two-dimensional instance and the at least one primitive based on the camera perspective to determine at least one three-dimensional instance, and the at least one three-dimensional instance includes a structure formed by the at least one primitive.

[0171] The output module 705 is configured to output an instance identifier of each three-dimensional instance.

[0172] In a possible implementation, the foregoing projection module 704 is configured to: project the at least one two-dimensional instance and the at least one primitive based on the camera perspective to acquire an overlapping area between each two-dimensional instance and each primitive in a projection space; and group one or more primitives with an overlapping area greater than a preset area to the same two-dimensional instance to form a three-dimensional instance to obtain the at least one three-dimensional instance.

[0173] In a possible implementation, the foregoing projection module 704 is specifically configured to: project the at least one primitive into an imaging plane of the two-dimensional instance to acquire an overlapping area between each two-dimensional instance and each primitive in the imaging plane.

[0174] In a possible implementation, the foregoing two-dimensional segmentation module 702 is specifically configured to: acquire input text, and the input text includes text for indicating segmentation of the image; and input the image and the input text as inputs of a large visual language model to output the first two-dimensional instance segmentation result.

[0175] In a possible implementation, the apparatus further includes a training module 706 configured to obtain a plurality of second two-dimensional instance segmentation results, each second two-dimensional instance segmentation result in the plurality of second two-dimensional instance segmentation results having a same instance identifier of a same instance; and train a neural network model based on the plurality of second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene.

[0176] In a possible implementation, the three-dimensional segmentation module 703 is specifically configured to cluster the corner points or the patches in the three-dimensional model according to the normal vectors of the surfaces of the three-dimensional model, and obtain at least one primitive by taking the corner points and / or the patches in a group of clustering categories as a primitive.

[0177] In a possible implementation, the input module 701 is specifically configured to obtain the three-dimensional model from user input data, or perform three-dimensional reconstruction based on a plurality of images of the target scene to obtain the three-dimensional model.

[0178] In a possible implementation, the output module 705 is further configured to output the three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating at least one three-dimensional instance in the three-dimensional model; or output at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result including at least one two-dimensional instance, and each two-dimensional instance having a same instance identifier as a three-dimensional instance representing a same entity.

[0179] Each module in the instance segmentation apparatus can be implemented by software or by hardware. For example, the implementation of the projection module is described as follows. Similarly, the implementation of other modules, such as the input module, the two-dimensional segmentation module, the three-dimensional segmentation module, the output module, or the training module, can refer to the implementation of the projection module.

[0180] As an example of a software functional unit, the projection module can include code running on a computing instance. The computing instance can be at least one of a physical host (computing device), a virtual machine, a container, or the like. Further, the computing device can be one or more. For example, the projection module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region, or can be distributed in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same AZ (a kind of server), or can be distributed in different AZs, each AZ including a data center or multiple data centers in a similar geographical location. Generally, one region can include multiple AZs.

[0181] Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually one VPC is set in one region. Communication between two VPCs in the same region and between VPCs in different regions needs to set a communication gateway in each VPC to realize the interconnection between VPCs through the communication gateway.

[0182] As an example of a hardware functional unit, the projection module can include at least one computing device, such as a server, etc. Alternatively, the projection module can also be a device implemented by a central processing unit (CPU), an application specific integrated circuit (ASIC) for special applications, a programmable logic device (PLD), a complex PLD (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), an embedded neural-network process unit (NPU), a system on chip (SoC), an offload card, an acceleration card, etc. Among them, the above-mentioned PLD can be a CPLD, an FPGA, a GAL or any combination thereof.

[0183] The multiple computing devices included in the projection module can be distributed in the same region or in different regions. The multiple computing devices included in the projection module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the projection module can be distributed in the same VPC or in multiple VPCs. Among them, the multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, acceleration cards, etc.

[0184] As an example of a software functional unit, the instance segmentation apparatus can include code running on a computing instance. The computing instance can be at least one of a physical host (computing device), a virtual machine, a container, and the like. Further, the computing device can be one or more. For example, the instance segmentation apparatus can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the application can be distributed in the same region, or in different regions. The multiple hosts / virtual machines / containers for running the code can be distributed in the same AZ, or in different AZs, each of which includes a data center or multiple data centers in close geographical proximity. Generally, a region can include multiple AZs.

[0185] Similarly, the multiple hosts / virtual machines / containers for running the code can be distributed in the same VPC, or in multiple VPCs. Generally, a VPC is set up within a region. Communication between two VPCs in the same region, and cross-region communication between VPCs in different regions, requires a communication gateway to be set up in each VPC to achieve interconnection between VPCs via the communication gateway.

[0186] As an example of a hardware functional unit, the instance segmentation apparatus can include at least one computing device, such as a server or the like. Alternatively, the instance segmentation apparatus can also be a device implemented using an ASIC, or a PLD, and the like. The PLD can be a CPLD, an FPGA, a GAL, or any combination thereof.

[0187] The multiple computing devices included in the instance segmentation apparatus can be distributed in the same region, or in different regions. The multiple computing devices included in the instance segmentation apparatus can be distributed in the same AZ, or in different AZs. Similarly, the multiple computing devices included in the instance segmentation apparatus can be distributed in the same VPC, or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, and the like.

[0188] The present application also provides a chip system including a processor and a power supply circuit, the power supply circuit being configured to supply power to the processor, and the processor being configured to perform the operation steps corresponding to the method provided by the present application. For brevity, the details are not repeated here. The processor can be implemented by a GPU, or by a DPU, an NPU, an XPU, a SoC, an offload card, an acceleration card, and the like.

[0189] The application also provides a computing device 100. As shown in FIG. 8, the computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 100 is not limited.

[0190] The bus 102 can be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. Among them, the unified bus is also called a smart bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in FIG. 8, but it does not mean that there is only one bus or one type of bus. The bus 104 can include a path for transmitting information between various components (e.g., the memory 106, the processor 104, the communication interface 108) of the computing device 100. Among them, the unified bus can also be called a smart bus.

[0191] The processor 104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, etc. in a computing device.

[0192] The memory 106 can include volatile memory, such as random access memory (RAM). The processor 104 can also include non-volatile memory, such as read-only memory (ROM), Flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). In addition, the memory 106 can also be implemented by storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0193] It is worth noting that the same type of storage medium can be configured to implement the memory 106 function in the same computing device, or two or more types of storage media can be configured to implement the memory 106 function, which is not limited in the present application.

[0194] The memory 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the modules mentioned in the foregoing FIG. 10, thereby implementing the method provided in the present application. That is, the memory 106 stores instructions for executing the method provided in the present application.

[0195] Alternatively, the memory 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the input module, the two-dimensional segmentation module, the three-dimensional segmentation module, the projection module, the output module, or the training module, thereby implementing the method provided in the present application. That is, the memory 106 stores instructions for executing the method provided in the present application.

[0196] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card and a transceiver to implement communication between the computing device 100 and other devices or communication networks.

[0197] As a possible implementation manner, the computing device 100 can also include a chip system including a processor and a power supply circuit for performing power supply to the processor, and the processor is used to perform the operation steps corresponding to the method provided in the present application. For the sake of brevity, it will not be repeated here. The processor can be implemented by a GPU, and can also be implemented by a DPU, NPU, XPU, SoC, offload card, acceleration card, or other computing devices or AI chips.

[0198] As a possible implementation, a plurality of types of processors 104 can be included in the computing device 100, i.e., the computing device 100 is a heterogeneous device, for example, the computing device 100 includes a CPU and a GPU, and the operation steps corresponding to the method provided in the present application can be executed by at least one of the processors 104. For the sake of brevity, details are not repeated here.

[0199] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0200] As shown in FIG. 9, the computing device cluster includes at least one computing device 100. The same instructions for executing the method provided in the present application can be stored in the memory 106 of one or more computing devices 100 in the computing device cluster.

[0201] In some possible implementations, partial instructions for executing the method provided in the present application can also be respectively stored in the memory 106 of one or more computing devices 100 in the computing device cluster. In other words, the combination of one or more computing devices 100 can collectively execute the instructions for executing the method provided in the present application.

[0202] It should be noted that the memories 106 in different computing devices 100 in the computing device cluster can store different instructions, respectively used to execute part of the functions of the joint testing device. That is, the instructions stored in the memories 106 in different computing devices 100 can implement the functions of one or more of the aforementioned input module, two-dimensional segmentation module, three-dimensional segmentation module, projection module, output module, or training module.

[0203] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 10 shows a possible implementation. As shown in FIG. 10, two computing devices 100A and 100B are connected through a network. Specifically, the communication interfaces in the respective computing devices are connected to the network. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the input module, two-dimensional segmentation module, three-dimensional segmentation module, projection module, output module, or training module. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the input module, two-dimensional segmentation module, three-dimensional segmentation module, projection module, output module, or training module.

[0204] It should be understood that the functions of the computing device 100A shown in FIG. 10 can also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be completed by multiple computing devices 100.

[0205] The connection manner between the computing device clusters shown in FIG. 10 can be that a large amount of computing power is required for the method provided in the present application, load balancing needs to be implemented, or a large amount of data storage is required, and the like, so different modules are deployed in different computing devices, such as the functions implemented by the output module are executed by the computing device 100A, and the functions implemented by the projection module are executed by the computing device 100B.

[0206] The present application also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manner of the computing device cluster described with reference to FIG. 9 and FIG. 10. The difference is that the memory 106 in one or more computing devices 100 in the computing device cluster can store the same instructions for executing the method provided in the present application.

[0207] In some possible implementation manners, the memory 106 of one or more computing devices 100 in the computing device cluster can also respectively store part of the instructions for executing the method provided in the present application. In other words, the combination of one or more computing devices 100 can collectively execute the instructions for executing the method provided in the present application.

[0208] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions for executing part of the functions of the instance segmentation apparatus provided in the present application. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of the input module, the two-dimensional segmentation module, the three-dimensional segmentation module, the projection module, the output module, or the training module.

[0209] The present application also provides a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to execute the method provided in the present application.

[0210] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be used to store instructions that can be executed by a computing device, or a data center containing one or more available media or data storage devices. The available medium can be a magnetic medium, (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state hard disk), etc. The computer readable storage medium includes instructions that instruct the computing device to execute the method provided by the present application.

[0211] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An instance segmentation method, characterized by, The method comprises: obtaining at least one image of a target scene, a camera perspective of the at least one image, and a three-dimensional model of the target scene; performing instance segmentation on the at least one image to obtain a first two-dimensional instance segmentation result, the first two-dimensional instance segmentation result comprising at least one two-dimensional instance; obtaining at least one primitive in the three-dimensional model, the at least one primitive comprising a primitive formed by a vertex and a face in the three-dimensional model; projecting the at least one two-dimensional instance and the at least one primitive according to the camera perspective to determine at least one three-dimensional instance in the three-dimensional model, the at least one three-dimensional instance comprising a structure formed by the at least one primitive; outputting an instance identifier of the at least one three-dimensional instance.

2. The method of claim 1, wherein, The method further comprises: obtaining an overlap area between each two-dimensional instance and each primitive in a projection space based on the projection of the at least one two-dimensional instance and the at least one primitive according to the camera perspective; grouping one or more primitives with an overlap area greater than a preset area with the same two-dimensional instance to form a three-dimensional instance, thereby obtaining the at least one three-dimensional instance.

3. The method of claim 2, wherein, The method further comprises: projecting the at least one primitive into an imaging plane of the two-dimensional instance to obtain an overlap area between each two-dimensional instance and each primitive in the imaging plane.

4. The method according to any one of claims 1-3, characterized in that, The method further comprises: obtaining an input text, the input text comprising text for instructing segmentation of the image; inputting the at least one image and the input text into a large visual language model to output the first instance segmentation result.

5. The method according to any one of claims 1-4, characterized in that, The method further comprises: obtaining a plurality of second two-dimensional instance segmentation results, each second two-dimensional instance segmentation result in the plurality of second two-dimensional instance segmentation results having the same instance identifier of the same instance; training a neural network model based on the plurality of second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene.

6. The method according to any one of claims 1-5, characterized in that, The method further comprises: clustering the vertices or the faces in the three-dimensional model according to normal vectors of surfaces of the three-dimensional model, grouping the vertices and / or the faces in a cluster category as a primitive, and obtaining the at least one primitive.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: obtaining the three-dimensional model from user input data; or, performing three-dimensional reconstruction based on a plurality of images of the target scene to obtain the three-dimensional model.

8. The method according to any one of claims 1-6, characterized in that, The method further comprises: outputting a three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating the at least one three-dimensional instance in the three-dimensional model. Or, output at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result comprising the at least one two-dimensional instance, and the instance identifier of each two-dimensional instance being the same as the instance identifier of the three-dimensional instance representing the same entity.

9. An instance segmentation device, comprising: Comprise: An input module configured to obtain at least one image of a target scene, a camera perspective of the at least one image, and a three-dimensional model of the target scene; A two-dimensional segmentation module configured to perform instance segmentation on the at least one image to obtain a first two-dimensional instance segmentation result comprising at least one two-dimensional instance; A three-dimensional segmentation module configured to obtain at least one primitive in the three-dimensional model, the at least one primitive comprising a primitive formed by a vertex and a face in the three-dimensional model; A projection module configured to project the at least one two-dimensional instance and the at least one primitive based on the camera perspective to determine at least one three-dimensional instance, the at least one three-dimensional instance comprising a structure formed by the at least one primitive; An output module configured to output an instance identifier of the at least one three-dimensional instance.

10. The apparatus of claim 9, wherein, The projection module is configured to: Project the at least one two-dimensional instance and the at least one primitive based on the camera perspective to obtain an overlapping area between each two-dimensional instance and each primitive in a projection space; Group one or more primitives with an overlapping area greater than a preset area of the same two-dimensional instance to form a three-dimensional instance to obtain the at least one three-dimensional instance.

11. The apparatus of claim 10, wherein, The projection module is specifically configured to: Project the at least one primitive into an imaging plane of the two-dimensional instance to obtain an overlapping area between each two-dimensional instance and each primitive in the imaging plane.

12. The apparatus of any one of claims 9-11, wherein, The two-dimensional segmentation module is specifically configured to: Obtain an input text comprising text for instructing segmentation of the image; Input the at least one image and the input text into a large visual language model to output the first two-dimensional instance segmentation result.

13. The apparatus of any one of claims 9-12, wherein, The device further comprises a training module configured to: Obtain a plurality of second two-dimensional instance segmentation results, each second two-dimensional instance segmentation result in the plurality of second two-dimensional instance segmentation results having the same instance identifier of the same instance; Train a neural network model based on the plurality of second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene.

14. The device of any one of claims 9-13, wherein: The three-dimensional segmentation module is specifically configured to cluster the vertices or the faces in the three-dimensional model according to normal vectors of surfaces of the three-dimensional model, and group the vertices and / or the faces under a cluster category as a primitive to obtain the at least one primitive.

15. The apparatus of any one of claims 9-14, wherein, The input module is specifically configured to: Obtain the three-dimensional model from user input data; Or, perform three-dimensional reconstruction based on a plurality of images of the target scene to obtain the three-dimensional model.

16. The apparatus of any one of claims 9-15, wherein, The output module is further configured to: Output a three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating the at least one three-dimensional instance in the three-dimensional model; Or, outputting at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result comprising the at least one two-dimensional instance, each two-dimensional instance having an instance identifier identical to an instance identifier of the three-dimensional instance representing the same entity.

17. A cluster of computing devices, characterized in that, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the processing steps of the method of any one of claims 1 to 8.

18. A computer program product comprising instructions, characterized in that, the instructions, when executed by the cluster of computing devices, cause the cluster of computing devices to perform the processing steps of the method of any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, comprising computer program instructions, the computer program instructions, when executed by the cluster of computing devices, cause the cluster of computing devices to perform the processing steps of the method of any one of claims 1 to 8.

Images