Instance segmentation method and apparatus

Abstract

Provided in the present application are an instance segmentation method and apparatus in the field of computer vision, which are used for aligning, on the basis of a two-dimensional instance in an open scene, primitives segmented from a three-dimensional model, and then outputting consistent three-dimensional instance segmentation, such that even if the view is switched, the same three-dimensional instance can be segmented. The method comprises: acquiring at least one image of a target scene, a camera view of the at least one image, and a three-dimensional model of the target scene; then performing instance segmentation on the at least one image, so as to obtain a first two-dimensional instance segmentation result, wherein the first two-dimensional instance segmentation result comprises at least one two-dimensional instance; acquiring at least one primitive in the three-dimensional model, wherein the at least one primitive comprises a primitive formed by a corner point and a surface patch in the three-dimensional model; next, on the basis of the camera view, projecting the at least one two-dimensional instance and the at least one primitive, so as to determine at least one three-dimensional instance, wherein each three-dimensional instance comprises one or more primitives; and outputting an instance identifier of each three-dimensional instance.

Description

An instance segmentation method and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202410796620.4, filed with the State Intellectual Property Office of China on June 19, 2024, entitled "Scene Segmentation Method, Apparatus, Computing Device Cluster and Storage Medium", and to Chinese Patent Application No. 202410865517.0, filed with the State Intellectual Property Office of China on June 29, 2024, entitled "An Instance Segmentation Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer vision, and more particularly to an instance segmentation method and apparatus. Background Technology

[0003] Reconstructing the environment is crucial for applications such as digital cities, autonomous driving, and robotics. The editability of the environment is essential for showcasing and editing the digital world, as well as for training and testing autonomous vehicles and robots in diverse environments. This requires that each object in the environment can be extracted individually; that is, each object needs a category label and object number (or instance annotation) to facilitate editing. Environments with instance annotations can also be used for semantic navigation in robots.

[0004] In instance segmentation scenarios involving acquired images, different images may contain the same or different instances. For each instance included in each frame, a corresponding instance ID needs to be assigned so that downstream tasks can distinguish and further process each instance. However, existing open scene segmentation models rely on 2D visual language models, meaning segmentation only assigns instance IDs to single images. This results in the same object having different instance IDs in different frames, thus failing to provide useful 3D information. Therefore, achieving consistency of instance IDs across different viewpoints in open scenes has become a pressing issue. Summary of the Invention

[0005] This application provides an instance segmentation method and apparatus for aligning primitives segmented from a 3D model based on 2D instances in an open scene, thereby outputting consistent 3D instance segmentation that can segment the same 3D instance even when the viewpoint is switched.

[0006] In view of this, in a first aspect, this application provides an instance segmentation method, comprising: acquiring at least one image of a target scene, a camera view of the at least one image, and a three-dimensional model of the target scene; subsequently performing instance segmentation on the at least one image to obtain a first two-dimensional instance segmentation result, the first two-dimensional instance segmentation result including at least one two-dimensional instance; acquiring at least one primitive in the three-dimensional model, the at least one primitive including primitives formed by corner points and facets in the three-dimensional model; subsequently projecting the at least one two-dimensional instance and the at least one primitive based on the camera view to determine at least one three-dimensional instance, the at least one three-dimensional instance including a structure formed by at least one primitive, each three-dimensional instance including one or more primitives; subsequently outputting an instance identifier of the at least one three-dimensional instance, the three-dimensional instance may be assigned after the three-dimensional instance is determined, or it may be determined based on the ID of a two-dimensional instance aligned with the three-dimensional instance.

[0007] In this embodiment, at least one image is segmented to output two-dimensional instances, and a three-dimensional model is segmented to output primitives. The two-dimensional instances and primitives are then projected to align them in the same projection space. This allows for the identification of primitives belonging to the same entity under the constraints of the two-dimensional instances, thereby achieving instance segmentation combined with the three-dimensional model. Furthermore, when applied to open scenes, consistent three-dimensional instance segmentation can be achieved, meaning that the consistency of each three-dimensional instance or two-dimensional instance can be maintained even in two-dimensional images from different viewpoints.

[0008] In one possible implementation, the aforementioned method of determining at least one three-dimensional instance by projecting at least one two-dimensional instance and at least one primitive includes: projecting at least one two-dimensional instance and at least one primitive based on a camera viewpoint, that is, projecting the two-dimensional instance and the primitive into the same projection space based on the camera viewpoint, and obtaining the overlap area between each two-dimensional instance and each primitive in the projection space; forming a three-dimensional instance by grouping one or more primitives with an overlap area greater than a preset area with the same two-dimensional instance, thereby obtaining at least one three-dimensional instance.

[0009] In this embodiment, two-dimensional instances and primitives segmented from a three-dimensional model can be projected into the same space. The overlap area between the two-dimensional instance and the primitive is used to identify whether the instance and the primitive represent the same entity, thereby achieving accurate segmentation of instances in the three-dimensional model in a geometric way and realizing consistency between instances in two-dimensional and three-dimensional models.

[0010] In one possible implementation, the aforementioned projection of at least one two-dimensional instance and at least one primitive to obtain the overlap area between each two-dimensional instance and each primitive in the projection space may include: projecting at least one primitive onto the imaging plane of the two-dimensional instance, and obtaining the overlap area between each two-dimensional instance and each primitive in the imaging plane. In this embodiment, the primitive can be projected onto the plane corresponding to the two-dimensional instance, thereby placing the primitive and the two-dimensional instance in the same dimension. This facilitates the calculation of the overlap area between the two-dimensional instance and the primitive, thereby accurately identifying primitives representing the same entity and achieving three-dimensional instance segmentation.

[0011] In one possible implementation, the aforementioned instance segmentation of at least one image to obtain a first two-dimensional instance segmentation result may include: acquiring input text corresponding to at least one image, the input text including text indicating the segmentation of at least one image, the input text being user-inputted text including prompts for instance segmentation, such as the category or name of the entity to be segmented; subsequently using at least one image and the input text as input to a large visual language model, and outputting information about at least one two-dimensional instance. In this embodiment, image segmentation can be performed using the language understanding capabilities of existing language models without incurring additional training costs.

[0012] In one possible implementation, the aforementioned method may further include: acquiring multiple second two-dimensional instance segmentation results, wherein the instance identifier of the same instance in each of the multiple second two-dimensional instance segmentation results is the same; and training a neural network model based on the multiple second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene. In this embodiment, a three-dimensional implicit representation can be trained during the three-dimensional instance segmentation process. Typically, two-dimensional images from different perspectives may exhibit partial entity hiding or shape changes. In this embodiment, segmentation maps corresponding to multiple frames can be accumulated to train the three-dimensional implicit representation, thereby learning the morphology of each entity from different perspectives or different frames, thus obtaining a more accurate three-dimensional implicit representation containing instance segmentation. For example, in an open scene, even when switching between any different perspectives, a two-dimensional rendering map matching the actual scene can be output.

[0013] In one possible implementation, the aforementioned acquisition of information about at least one primitive in a 3D model may include: clustering corner points or faces in the 3D model based on the normal vectors of the 3D model surface, and taking a set of corner points and / or faces under a cluster category as a primitive to obtain information about at least one primitive. In this embodiment, intersection points or faces can be segmented based on the normal vectors of the 3D model surface, and primitives can be segmented through clustering, thereby achieving primitive-level instance segmentation. Combining this with entity-level segmentation under 2D instances as a constraint, more accurate 3D instance segmentation results can be output.

[0014] In one possible implementation, the aforementioned acquisition of the 3D model may include: acquiring the 3D model from user input data; or, constructing a 3D model based on multiple frames of images of the target scene. In this embodiment, the 3D model may be directly input by the user or obtained through 3D reconstruction based on multiple frames of images, to adapt to various scenarios and have stronger generalization.

[0015] In one possible implementation, the aforementioned method further includes: outputting a three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating at least one three-dimensional instance in the three-dimensional model; or, outputting at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result including at least one two-dimensional instance, the instance identifier of each two-dimensional instance being the same as the instance identifier of the three-dimensional instance representing the same entity.

[0016] In this embodiment, two-dimensional instance segmentation results and / or three-dimensional instance segmentation results can be output. Specifically, the type of output data can be selected according to the actual application scenario, which can be applied to more scenarios and has stronger generalization.

[0017] Secondly, embodiments of this application provide an instance segmentation apparatus, comprising:

[0018] The input module is used to acquire at least one image of the target scene, the camera view of at least one image, and a 3D model of the target scene;

[0019] A two-dimensional segmentation module is used to perform instance segmentation on at least one image to obtain a first two-dimensional instance segmentation result, wherein the first two-dimensional instance segmentation result includes at least one two-dimensional instance.

[0020] The 3D segmentation module is used to obtain at least one primitive in a 3D model. The at least one primitive includes primitives formed by corner points and facets in the 3D model.

[0021] A projection module is used to project at least one two-dimensional instance and at least one primitive from a camera perspective to determine at least one three-dimensional instance, wherein the at least one three-dimensional instance includes a structure formed by at least one primitive;

[0022] The output module is used to output the instance identifier for each 3D instance.

[0023] The effects achieved by the second aspect or any optional implementation of the second aspect can be referred to the effects of the first aspect or any optional implementation of the first aspect, and will not be repeated hereafter.

[0024] In one possible implementation, the aforementioned projection module is used to: project at least one two-dimensional instance and at least one primitive based on the camera viewpoint, and obtain the overlap area between each two-dimensional instance and each primitive in the projection space; and combine one or more primitives with an overlap area greater than a preset area with the same two-dimensional instance to form a three-dimensional instance, thereby obtaining at least one three-dimensional instance.

[0025] In one possible implementation, the aforementioned projection module is specifically used to: project at least one primitive onto the imaging plane of a two-dimensional instance, and obtain the overlap area between each two-dimensional instance and each primitive in the imaging plane.

[0026] In one possible implementation, the aforementioned two-dimensional segmentation module is specifically used for: acquiring input text, which includes text indicating the segmentation of the image; using the image and the input text as input to a large visual language model, and outputting a first two-dimensional instance segmentation result.

[0027] In one possible implementation, the aforementioned apparatus further includes: a training module, configured to acquire multiple second two-dimensional instance segmentation results, wherein the instance identifier of the same instance in each of the multiple second two-dimensional instance segmentation results is the same; and to train a neural network model based on the multiple second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene.

[0028] In one possible implementation, the aforementioned three-dimensional segmentation module is specifically used to cluster corner points or facets in the three-dimensional model based on the normal vectors of the three-dimensional model surface, and to obtain at least one primitive by taking a set of corner points and / or facets under a cluster category as a primitive.

[0029] In one possible implementation, the aforementioned input module is specifically used to: obtain a 3D model from user input data; or to perform 3D reconstruction based on multiple frames of images of the target scene to obtain a 3D model.

[0030] In one possible implementation, the aforementioned output module is further configured to: output a three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating at least one three-dimensional instance in the three-dimensional model; or, output at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result including at least one two-dimensional instance, the instance identifier of each two-dimensional instance being the same as the instance identifier of the three-dimensional instance representing the same entity.

[0031] Thirdly, embodiments of this application provide a computing device, including a processor and a memory; the processor of at least one computing device is configured to execute instructions stored in the memory of at least one computing device, so that the computing device performs method steps as described in the first aspect and any implementation thereof.

[0032] Fourthly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method steps as described in the first aspect and any implementation thereof.

[0033] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform a method as described in the first aspect or any implementation thereof.

[0034] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform a method as described in any of the first, second, third, or fourth aspects.

[0035] In a seventh aspect, embodiments of this application provide a chip including at least one processor and an interface; at least one processor obtains program instructions or data through the interface; at least one processor is used to execute program line instructions to implement the method in the first aspect or any implementation thereof. Attached Figure Description

[0036] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0037] Figure 2 is a flowchart illustrating an instance segmentation method provided in an embodiment of this application;

[0038] Figure 3 is a flowchart illustrating another instance segmentation method provided in an embodiment of this application;

[0039] Figure 4 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0040] Figure 5 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0041] Figure 6 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0042] Figure 7 is a schematic diagram of an instance segmentation device provided in an embodiment of this application;

[0043] Figure 8 is a schematic diagram of the structure of a computing device provided in this application;

[0044] Figure 9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0045] Figure 10 is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0047] The method provided in this application can be applied to artificial intelligence (AI) scenarios. For ease of understanding, some terms and concepts involved in this application are introduced below.

[0048] (1) Large Model

[0049] Large models are large-scale models. The "large" in large models can be reflected in many aspects, such as large data scale, large-scale parallel computing capabilities, and larger model structures.

[0050] (2) Language model (LM)

[0051] Language models play a crucial role in Natural Language Processing (NLP), where their task is to predict the probability of a sentence occurring in a language. For example, a language model is typically constructed as a probability distribution p(s) of a string s, where p(s) attempts to reflect the frequency of string s as a sentence. It can be applied to scenarios such as text recognition or machine translation. In the embodiments of this application, the NLP models mentioned below include language models.

[0052] (3) Large Language Model (LLM)

[0053] A large language model (LLM) refers to a language model containing hundreds of billions (or more) parameters trained on massive amounts of text data. It is a natural language processing model based on deep learning. These models can process large amounts of text data to learn the grammatical and semantic rules of natural language. LLMs can be applied to text generation, machine translation, question answering systems, text summarization, and sentiment analysis, offering advantages such as strong generative capabilities, high adaptability, accurate prediction, and strong scalability. For example, in movie recommendation scenarios, a large language model can generate descriptions of movie scenes, including genre, main actors, and plot, enabling the system to better recommend similar movies. Large language models can also generate recommendation reasons; for example, e-commerce websites can use large language models to generate reasons for recommending products, such as product quality, price, and features, allowing users to better understand the value of the product.

[0054] (4) Pre-trained language model (PLM)

[0055] A Natural Language Sequence Encoder (PLM) is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. PLM training consists of two phases: pre-training and fine-tuning. In the pre-training phase, the model is trained on large-scale unsupervised text to learn word representations. In the fine-tuning phase, the model is initialized using the parameters learned in the pre-training phase. For example, it can be trained on downstream tasks such as text classification or sequence labeling with fewer steps, successfully transferring the semantic information obtained in pre-training to downstream tasks.

[0056] (5) Large Vision-Language Model (LVLM)

[0057] It can also be called a visual large language model, visual language large model, etc. It typically consists of a visual encoder (such as CLIP) and a language model (LM), the former interpreting the encoded features and the latter solving downstream tasks. In the embodiments of this application, the language model in the LVLM can be a pre-trained language model or a large language model. The LVLM combines the capabilities of large language models (LLMs) to handle complex tasks involving vision and natural language.

[0058] (6) Loss function

[0059] Also known as the cost function, it's a metric that compares the predicted output of a machine learning model with the true value of a sample (also called the supervised value). In other words, it measures the difference between the model's predicted output and the true value. This loss function typically includes loss functions such as mean squared error, cross-entropy, logarithmic loss, and exponential loss. For example, mean squared error can be used as the loss function, defined as... The specific loss function can be selected based on the actual application scenario.

[0060] (7) Gradient: The vector of the derivative of the loss function with respect to the parameters.

[0061] (8) Instance segmentation

[0062] Image segmentation refers to dividing an image into several non-overlapping regions based on features such as grayscale, color, spatial texture, and geometric shape. These features should exhibit consistency or similarity within the same region, while showing significant differences between different regions. Instance segmentation combines object detection and semantic segmentation. It detects objects in the image (object detection) and then labels each pixel (semantic segmentation).

[0063] (9) Graphic Reasoning

[0064] A graph typically consists of nodes and edges connecting the nodes. Graph reasoning involves calculating the edges between nodes in a graph to form multiple connected groups of nodes.

[0065] (10) Three-dimensional reconstruction

[0066] This refers to establishing a mathematical model of a three-dimensional object suitable for computer representation and processing. It forms the basis for processing, manipulating, and analyzing the properties of the object in a computer environment, essentially simulating a three-dimensional entity in a virtual three-dimensional space. Specifically, it involves determining the pose of each frame based on multiple images and the corresponding camera parameters, then using this pose to determine the coordinates of the pixels projected into three-dimensional space, thus reconstructing the three-dimensional model.

[0067] (11) Neural radiance field (NeRF)

[0068] NeRF is a method or model that uses neural networks to implicitly represent 3D scenes. A given scene can be learned using NeRF, and this scene is implicitly stored in the parameters of the NeRF neural network, i.e., the implicit representation of the scene. If a new perspective is needed, NeRF can be used to calculate the light and color values ​​at various locations within this scene, and after rendering, the new perspective can be output.

[0069] (12) Open vocabulary scenarios

[0070] Also known as open scene, open world, or open vocabulary 3D scene, it refers to a virtual world scene reconstructed from an open world, including models obtained through 3D reconstruction based on 3D data. In an open vocabulary scene, users can freely explore the world from any perspective, any location, or with any vocabulary. Open vocabulary scenes can be applied in scenarios such as digital cities, autonomous driving, robotics, or games, providing users with a visual experience that simulates the real world.

[0071] In intelligent scenarios such as digital cities, autonomous driving, and robotics, perceiving the robot's surrounding environment, including 3D segmentation of instances within any given scene and assigning IDs to these segmented instances, is crucial for downstream tasks. Downstream tasks differentiate instances using their IDs and then execute further tasks based on these instances. For example, in open-world 3D instance segmentation, situations may arise where the same instance is configured with different IDs in different images, or different instances are configured with the same ID in different images, potentially leading to inaccuracies in downstream tasks. Existing solutions mostly handle closed-set scene segmentation, lacking model generalization and adaptability to new scenarios. Furthermore, the training data for these models requires manual annotation, which is difficult to obtain. In recent years, with the development of large-scale visual language models, 2D instance segmentation for open-vocabulary scenes has achieved high accuracy and strong generalization. However, 2D instance labels lack consistency across 3D space. Existing open-vocabulary scene segmentation and reconstruction methods often employ linear assignment, constructing a loss matrix based on a feature loss function to redistribute the IDs of different instance masks in each image, thereby aligning 2D instance IDs. However, due to the lack of prior information about the three-dimensional scene and the limited two-dimensional constraint perspective, the labels of different instances in space that have never appeared in the same frame are repeated.

[0072] For example, in an existing approach, during the training phase of the segmentation model, 2D and 3D instances are linearly matched to achieve end-to-end alignment and reconstruction of 2D instance labels. However, the 3D instance IDs are entirely limited by the number of IDs initialized in the 3D scene. Due to the low randomness of the initial output of the MLP network constituting the Neural Field, duplicate instance IDs can occur for different objects appearing in different frames within the scene. Furthermore, segmentation models typically use pre-trained mask2former models, which have poor generalization capabilities.

[0073] For example, one existing approach uses individual target tracking for each instance to maintain consistency of instance IDs in an image. However, the effectiveness of target tracking is very limited; errors can occur when the image viewpoint changes rapidly or when the scene is reversed and past objects are seen again, thus affecting the accuracy of downstream tasks.

[0074] Therefore, this application provides an instance segmentation method that uses the three-dimensional model of the scene as geometric prior information to align the 2D instances segmented from the image with the primitives segmented from the three-dimensional model, thereby identifying the three-dimensional instances. This method can very accurately identify and distinguish each instance, thus ensuring that the ID of each instance in the scene remains accurate.

[0075] The system architecture and methodological steps provided in this application are described below.

[0076] First, the method provided in this application can be applied to 3D reconstruction scenarios. By combining the instance segmentation results of 2D images with the primitives in the 3D model, instances in the 3D model are segmented under the constraints of 2D instances, thus achieving 3D consistent instance segmentation in open scenes.

[0077] The method provided in this application can be deployed in various electronic devices, such as server clusters, cloud platforms, personal computers, smartphones, or smart cars.

[0078] In one possible implementation, the method provided in this application can be deployed on computing devices such as personal computers, computer workstations, smartphones, tablets, laptops, and smart cars. Users can directly use computing devices to execute the method provided in the embodiments of this application to achieve three-dimensional instance segmentation.

[0079] For example, the method provided in this application can be applied to user devices, such as smartphones, tablets, or devices capable of deploying OCR recognition. Users can use these user devices to scan the scene requiring segmentation and reconstruction, for example, using the user device's camera, depth camera, or laser device to obtain scan data of the scene to be segmented. Subsequently, the method provided in this application is used to perform 3D reconstruction and instance segmentation of the scene.

[0080] In one possible implementation, the method provided in this application can be deployed on a server cluster, cloud platform, or other devices with computing capabilities.

[0081] For example, the method provided in this application can be deployed in a cloud platform to provide services to users through cloud services.

[0082] For example, Figure 1 shows a schematic diagram of the structure of a cloud service system provided in this application. As shown in Figure 1, the cloud service system 10 may include a computing device 11 and a client 12.

[0083] The computing device 11 may specifically include a server cluster or a cloud platform, or it may be other devices with computing capabilities. Optionally, the computing device 11 may work in conjunction with other computing devices, such as data storage, routers, load balancers, etc. The computing device 11 may use data from the data storage system or call program code in the data storage system to implement the method steps provided in the embodiments of this application.

[0084] The computing device 11 can provide services to users in the form of a client. Users can operate on the client 12 to achieve data interaction with the computing device 11 or request services from the computing device 11. This client can be deployed on personal computers, computer workstations, smartphones, tablets, laptops, and smart cars, etc.

[0085] In one implementation, the computing device 11 is used to implement the method provided in the embodiments of this application to segment at least one image for a client and to feed back information about the segmented instance to the client.

[0086] Among them, client 12 is an optional device. In actual application scenarios, the computing device can read input data from its own storage space or from the data center, perform three-dimensional instance segmentation on the input data, and output the instance segmentation result containing three-dimensional instances.

[0087] In summary, the methods provided in this application embodiment can be applied to electronic devices, that is, the aforementioned computing device 11 can be various electronic devices, such as server clusters, cloud platforms, personal computers, smartphones or smart cars, etc.

[0088] Based on the architecture shown in Figure 1 above, the method provided in this application can be executed by a computing device 11, which may specifically include a server cluster or a cloud platform, and can provide services to users through a client. Alternatively, the computing device 11 may also be other devices with computing capabilities.

[0089] For example, the method provided in this application can be deployed on a cloud platform to provide users with consistent 3D image segmentation services or 3D mesh model instance segmentation services through a client. When a user obtains a 3D mesh model of a target scene through other means, the user can directly input at least one image captured in the target scene and the corresponding 3D mesh model of the scene on the client. If the user cannot directly obtain the 3D mesh model, they can acquire at least one image and depth data of the target scene and select the reconstruction option provided to the user in this application embodiment (such as a button set for reconstructing a 3D model) to request the generation of a 3D mesh model. The aforementioned at least one image may include an image of the target scene captured by the image sensor of the user's device, and the 3D mesh model may include a depth map calculated by 3D laser scanning or a binocular camera, etc. The input information is transmitted to the cloud platform, and the cloud platform performs 3D instance segmentation using the method provided in this application and outputs the instance identifier of each 3D instance. Optionally, the 3D instance segmentation results and / or 2D instance segmentation results can also be output simultaneously.

[0090] The following section describes the method and process provided in this application, based on the aforementioned architecture.

[0091] Referring to Figure 2, a flowchart of an instance segmentation method provided in an embodiment of this application is shown below.

[0092] 201. Acquire at least one image of the target scene, the camera view of at least one image, and a 3D model of the target scene.

[0093] The at least one image is acquired in the target scene from the camera's perspective. When there are multiple images, the camera perspectives of the multiple images may be the same or different.

[0094] Based on the system architecture corresponding to Figure 1, the at least one image can be input by the user through a client and then sent to the cloud platform. Specifically, the at least one image can be captured by the user using the client, or it can be retrieved from the client's storage. For example, combining the architecture of Figure 1, the client can be deployed on the user's terminal, allowing the user to capture images through the terminal and use these images as at least one image.

[0095] This 3D model is a 3D reconstruction model of the target scene, which can be used to simulate the entity distribution of the target scene.

[0096] Optionally, the 3D model can be a model included in the user input data, such as a 3D mesh model constructed by the user through other means, or a 3D mesh model obtained by 3D reconstruction based on one or more frames of images input by the user. For example, the user can pre-construct a 3D model and input it into the client. Alternatively, the user can acquire multiple frames of images of the scene to be reconstructed, each frame carrying corresponding depth information, and send these multiple frames to the cloud platform via the client; the cloud platform can then perform 3D reconstruction on the received multiple frames to obtain the corresponding 3D model.

[0097] In one possible scenario, a user can acquire multiple frames of images of a target scene through a client. For example, based on the architecture shown in Figure 1 above, the client can be deployed on a user terminal. The user can capture multiple frames of images carrying depth information through the terminal, and then trigger instance segmentation of the multiple frames of images to the cloud platform. After receiving the multiple frames of images carrying depth information, the cloud platform can construct a 3D mesh model based on the received multiple frames of images to obtain a 3D model of the environment corresponding to the user's image acquisition.

[0098] When a user inputs multiple frames of images carrying depth information, the cloud platform can perform 3D reconstruction of these multiple frames and then perform 2D instance segmentation on them.

[0099] 202. Perform instance segmentation on at least one image to obtain the first two-dimensional instance segmentation result.

[0100] After obtaining at least one image from the target scene, two-dimensional instance segmentation can be performed on that image to obtain a two-dimensional segmentation instance result, which is referred to here as the first two-dimensional instance segmentation result for easy distinction. This first two-dimensional instance result may include one or more two-dimensional instances.

[0101] In one possible scenario, a large visual language model can be used to perform the segmentation task of at least one image. For example, input text corresponding to at least one image can be obtained. This input text may include instructions or prompts for segmenting at least one image, such as the category or name of the instance to be segmented. This input text and at least one image are used as input to a large visual language model. The large visual language model can understand the input text and detect the corresponding objects in the image, outputting one or a first two-dimensional instance segmentation result, which includes information about multiple two-dimensional instances, such as the category, detection probability (or confidence level), and segmentation pixel range of each two-dimensional instance. The large visual language model mentioned in the embodiments of this application may be a large language model including a visual encoder or a multimodal large language model, which has the ability to process at least one image and understand language. It is usually a trained model, thereby enabling instance segmentation of two-dimensional images with zero training cost.

[0102] 203. Obtain at least one primitive from the 3D model.

[0103] Typically, a 3D model can include basic geometric units, such as corner points or facets, in multiple dimensions. These are called primitives. One or more primitives can be extracted from the 3D model, and information about one or more primitives can be output. This includes information such as the shape, size, center point coordinates, or corner point coordinates of each primitive.

[0104] The 3D model typically includes one or more instances of the target scene. Each instance can usually be composed of one or more primitives. Therefore, the 3D model can be segmented using segmentation algorithms or segmentation networks to output the primitives in the 3D model, so that subsequent 3D instance segmentation can be achieved at the primitive level.

[0105] Optionally, corner points and faces in the 3D mesh model can be clustered based on normal vectors, and corner points under a set of cluster categories can be used as 3D instance segmentation primitives. The 3D instance mentioned below can be a combination shape composed of one or more primitives.

[0106] It should be noted that the execution order of steps 202 and 203 is not limited in this embodiment. Step 202 can be executed first, or step 203 can be executed first, or steps 202 and 203 can be executed simultaneously. The specific order can be determined according to the actual application scenario.

[0107] 204. Project the at least one two-dimensional instance and at least one primitive from the camera's perspective to determine at least one three-dimensional instance.

[0108] Specifically, based on the camera viewpoint of each two-dimensional image, the primitives and two-dimensional instances can be projected, and under the supervision of the coverage of the two-dimensional instances in the projection space, the primitives can be divided into one or more three-dimensional instances, or the at least one three-dimensional instance can be referred to as a combination structure formed by the aforementioned at least one primitive, and each three-dimensional instance can include a structure formed by one or more primitives.

[0109] Specifically, based on the camera viewpoint corresponding to each image, 2D instances and primitives can be projected into the same projection space. This projection space can be a 2D space or a 3D space, such as the 3D space corresponding to a 3D model or the 2D space corresponding to at least one image. One or more primitives with an overlap area greater than a preset area with the same 2D instance are grouped into a 3D instance, resulting in at least one 3D instance. This is equivalent to aligning primitives with 2D instances; primitives aligned to the same 2D instance can form a 3D instance. Thus, the primitives segmented from the 3D model are grouped, enabling supervised segmentation of 3D instances using 2D instances.

[0110] Optionally, at least one primitive can be projected onto the plane corresponding to the two-dimensional instance to obtain the overlap area between each two-dimensional instance and each primitive in the plane. Therefore, in this embodiment, primitives in the three-dimensional model can be projected onto the plane where the two-dimensional instance is located, thereby aligning the two-dimensional instance and the primitives of the three-dimensional model in the same two-dimensional plane. This is equivalent to using the two-dimensional instance as a constraint or supervisor to determine the primitives belonging to the same entity, thereby more accurately identifying the three-dimensional instance.

[0111] Furthermore, the camera perspective is used to represent the camera's pose in the target scene (i.e., the 3D scene) when capturing an image. Specifically, it can include parameters such as the camera's field of view or focal length. For example, the process of capturing an image can simulate the process of viewing a scene from the perspective of a human eye. The scene observed from different perspectives may be different. During the capturing process, entities within the camera's field of view can be projected from the viewing direction onto the imaging plane. Correspondingly, during projection, the projection plane, i.e., the camera's imaging plane, can be determined based on the camera perspective, and entities in the 3D model can be projected onto this imaging plane, so that the primitives and 2D instances are on the same plane, facilitating alignment of the primitives and 2D instances in the same dimension.

[0112] Of course, two-dimensional instances can also be projected into three-dimensional space. For example, if at least one image carries depth information, the two-dimensional instance can be projected into the three-dimensional space corresponding to the three-dimensional model based on the depth of the two-dimensional instance. Thus, by using the overlapping area or volume of the two-dimensional instance and each primitive in the three-dimensional space after projection, the primitives belonging to the same instance in the three-dimensional model can be identified, that is, the three-dimensional instance can be identified more accurately from a geometric perspective.

[0113] Furthermore, by projecting primitives into the projection space, a graph can be constructed based on the projected primitives. This constructed graph is called an instance segmentation graph, which consists of nodes and edges connecting them. The center point of each primitive in the projection space is taken as a node in the instance segmentation graph, and the connecting lines between these nodes are called edges in the instance segmentation graph. The edge selection is determined based on the overlap area between each primitive and its corresponding 2D instance. For example, if two primitives have no overlap area with a 2D instance, or the overlap area is 0, or the overlap area is less than a preset area, then the edge between these two primitives can be discarded. Thus, nodes and edges connected within the same 2D instance are treated as a single 3D instance, resulting in an instance segmentation graph containing 3D instances.

[0114] Therefore, in this embodiment, at least one image and a 3D model are segmented respectively, and the primitives in the 2D instance and the 3D model are aligned from a geometric dimension to identify the 3D instance composed of primitives under the supervision of the 2D instance, and to achieve ID alignment of instances in at least one image. Even in images from different viewpoints, 2D instances corresponding to 3D instances can be identified, thereby achieving ID allocation for different instances in the same image, and ID configuration for the same instance in different images, to maintain instance ID consistency across different viewpoints. This allows downstream tasks to obtain consistent 2D or 3D instances from different viewpoints, thereby improving environmental perception and enabling more complex downstream tasks.

[0115] 205. Output the instance identifier for each 3D instance.

[0116] Specifically, after identifying 3D instances from a 3D model, a unique identifier for each 3D instance can be determined so that each 3D instance can be distinguished or identified in subsequent steps.

[0117] Optionally, when determining the ID of each 3D instance, the ID can be assigned to the 3D instance after it is identified. At the same time, the ID of the 3D instance can also be configured as the ID of the 2D instance that is aligned with the 3D instance to maintain the ID consistency between the aligned 3D instance and the 2D instance. If a unique ID is assigned to each 2D instance in advance, then after aligning the primitives with the 2D instance, the ID of the 2D instance can be used as the ID of the 3D instance composed of the primitives aligned with it, thereby obtaining the ID of the 3D instance.

[0118] Optionally, 3D instance segmentation results can also be output simultaneously. These 3D instance segmentation results indicate at least one 3D instance in the 3D model. The information for each 3D instance may also include corner points, position ranges, 3D masks, etc., to represent the size, position, and pose of the 3D instance in the 3D model. Alternatively, at least one second 2D instance segmentation result can be output. This second 2D instance segmentation result includes at least one 2D instance, and the instance identifier of each 2D instance is the same as the instance identifier of the 3D instance representing the same entity. In other words, the instance identifier of the same instance in each of the multiple second 2D instance segmentation results is the same.

[0119] In this embodiment, two-dimensional instance segmentation results and / or three-dimensional instance segmentation results can be output. Specifically, the type of output data can be selected according to the actual application scenario, which can be applied to more scenarios and has stronger generalization.

[0120] In one possible implementation, a three-dimensional implicit representation can also be trained to represent the target scene, thereby outputting a rendered image from any viewpoint, which is more convenient for downstream tasks. Specifically, after obtaining multiple second-dimensional instance segmentation results, a neural network model, such as a NeRF model or other 3D reconstruction models, is trained based on these results to obtain a three-dimensional implicit representation of the target scene. In this embodiment, a three-dimensional implicit representation can be trained during the three-dimensional instance segmentation process. Typically, two-dimensional images from different viewpoints may exhibit partial entity hiding or shape changes. In this embodiment, segmentation images corresponding to multiple frames can be accumulated to train the three-dimensional implicit representation, thereby learning the morphology of each entity from different viewpoints or frames, thus obtaining a more accurate three-dimensional implicit representation that includes instance segmentation. For example, in an open scene, even when switching between any different viewpoints, a two-dimensional rendered image matching the actual scene can be output.

[0121] For example, in the case of at least one image captured from different viewpoints that includes entities with different poses or occluded entities, in this embodiment of the application, the final output can be set as a three-dimensional implicit representation. After using the two-dimensional instances (i.e., two-dimensional instances) in at least one image in each frame as supervision to determine the instance segmentation map including the three-dimensional instances (i.e., three-dimensional instances), the three-dimensional implicit representation can be updated based on the difference between the instance segmentation map containing the three-dimensional instances and the corresponding viewpoint in the three-dimensional implicit representation. The three-dimensional implicit representation can be used as input to downstream tasks so that the downstream tasks can extract scene images from any viewpoint from the three-dimensional implicit representation.

[0122] More specifically, in a 3D instance segmentation process, after obtaining the 3D instance segmentation map, the corresponding instance segmentation map is rendered from the 3D implicit representation. The difference between the 3D instance segmentation map and the rendered instance segmentation map is calculated, and the 3D implicit representation is updated based on this difference. That is, the features in the 3D instance segmentation map are learned into the 3D implicit representation, so that the 3D implicit representation can include richer features, such as more accurate instance segmentation under 2D instance constraints from different camera perspectives. Downstream task models can then perform downstream tasks based on the 3D implicit representation. For example, a rendered 2D segmentation example map from a certain perspective can be displayed on the client, or the 3D implicit representation can be used for comprehensive scene perception for subsequent decision-making.

[0123] The foregoing has described the method flow provided in the embodiments of this application. The following will further describe the method steps provided in the embodiments of this application in conjunction with specific application scenarios.

[0124] The method provided in this application can be applied to instance segmentation in 3D scenes, such as instance segmentation in open scenes.

[0125] Taking instance segmentation in an open scenario as an example, the application architecture of the method provided in this application can be illustrated as shown in Figure 3.

[0126] The method provided in this application can be divided into multiple stages, such as the data preprocessing stage, graph inference stage, or output stage shown in Figure 3. The detailed steps of each stage are described below.

[0127] I. Data Preprocessing

[0128] In the data preprocessing stage, the 3D model and at least one image can be processed separately. For example, the 3D model is illustrated using a mesh model of the target scene, and the at least one image is illustrated using any frame of an RGB image from that scene.

[0129] (1) Mesh model preprocessing of the target scene

[0130] The mesh model of the target scene can be derived from user input data or constructed based on multiple frames of RGB images and their corresponding depth information. For example, in an open-vocabulary scenario, the user can generate the mesh model of the target scene through other means, or the mesh scene can be constructed based on user input or using multiple frames of RGB images and their corresponding depth information, thus obtaining the scene's mesh model.

[0131] The mesh model of the target scene can be segmented using the normal vector of the mesh model of the target scene, dividing the mesh model into multiple primitives, i.e., mesh clustering segmentation blocks. Correspondingly, each 3D instance included in the mesh model is also divided into one or more segmentation blocks, or one or more segmentation blocks form a 3D instance.

[0132] Specifically, the Segmentator algorithm can be used to cluster corners and / or faces in the mesh model based on the surface normal vectors of the mesh model. Typically, corners and faces with similar normal vectors in a local area are grouped together, i.e., a segmentation block, and one or more segmentation blocks are output. Each segmentation block has a center point as its representative. For example, the segmentation result is shown in the mesh structure in Figure 4, where each color block represents a segmentation block, and the point on each segmentation block is the center point of that block.

[0133] Optionally, when segmenting the mesh model, the number and / or resolution of the segments can be preset or user-defined. For example, a user can set the number of segments to 100, thus dividing the mesh model into 100 primitives. Generally, the more primitives used, the finer the granularity of the segmentation, resulting in more accurate output when performing instance segmentation at the primitive level.

[0134] (2) RGB image preprocessing

[0135] The RGB image can be an image input by the user, such as an image taken by the user through the client, or an image input by the user through the client.

[0136] In a scenario where a mesh model is constructed based on multiple frames of RGB images input by the user and their corresponding depths, the RGB image can be one frame from the multiple frames of RGB images input by the user, or it can be an RGB image that the user collects separately in the scene. The specific choice can be determined based on the actual application scenario.

[0137] Specifically, instance segmentation for RGB images can be performed using pre-trained segmentation models or based on large visual language models (or large visual language models). For example, a large visual language model can be used for 2D instance segmentation. GroundedSAM can be used to add annotations to the image. The user inputs prompt text corresponding to the instance category to be segmented, along with the RGB image. GroundedSAM then outputs information about the segmented 2D instances, such as a 2D instance segmentation map or a mask corresponding to the 2D instances.

[0138] Typically, large visual language models have strong generalization capabilities and can achieve instance segmentation in open vocabulary scenarios without pre-training. For example, the rendering image corresponding to two-dimensional instance segmentation can be shown in Figure 5. For instance, if a user inputs prompts such as "chair, table, sofa...", the user's input words and RGB images are used as input to the GroundedSAM model. The GroundedSAM model can then understand the user's input text, segment out the chair, table, and sofa in the RGB image, and output the two-dimensional instance segmentation result.

[0139] II. Graphic Reasoning

[0140] Based on the segmentation results of the segmented blocks and 2D instances output in the aforementioned preprocessing stage, graph reasoning can be performed based on the segmented blocks and 2D instances, that is, aligning the segmented blocks and 2D instances and outputting the instance segmentation results. The instance segmentation results may include 3D instances composed of each segmented block under the supervision of 2D instances.

[0141] Specifically, by projecting the segmentation block and the 2D instance into the same space, the segmentation block and the 2D instance can be aligned in the same space, thereby determining the 3D instance composed of each primitive under the constraint of the 2D instance, and obtaining the instance segmentation result containing the 3D instance.

[0142] For example, as shown in Figure 6, the segmented blocks from the 3D model can be projected onto the imaging plane of the RGB image, and the overlap area (or intersection area) between the projected area of ​​each segmented block and the 2D instance can be calculated. Based on this area, it can be determined whether the 3D instance and the 2D instance correspond to the same entity.

[0143] Specifically, graph reasoning can be used to align 2D instances with segmentation blocks, thereby outputting an instance segmentation graph that includes 3D instances. For example, the 3D instances can be projected onto the imaging plane of the 2D instances to construct the instance segmentation graph, with the center points of each primitive as nodes in the graph, connecting each pair of nodes. The algorithm calculates the overlap area, or intersection area, between each primitive projected onto the 2D instance segmentation map and each 2D instance. Based on this overlap area, it determines whether to retain the edges connecting the nodes. For example, if primitive A and primitive B are connected, and the areas of both primitive A and primitive B projected onto a certain 2D instance are greater than a preset area, then the edge connecting primitive A and primitive B can be retained, meaning that primitive A and primitive B belong to the same instance, thus segmenting primitives within the same instance. However, if the overlap area between either primitive A or primitive B projected onto a certain 2D instance is not greater than the preset area, then primitive A and primitive B do not belong to the same instance, and therefore the edge connecting primitive A and primitive B is removed. This process continues until all primitives in the mesh model are traversed, outputting the final instance segmentation map representing the 3D instance.

[0144] For example, a graph can be represented as G(V,E), which is usually composed of nodes V and edges E between nodes.

[0145] Wherein, node V: In this embodiment of the application, the node is the center point of each segmented block (i.e., primitive) output after the mesh model is segmented.

[0146] Edge E: A connection edge is set between each pair of nodes during the initialization phase. For a given frame of RGB image, the node projected onto that image is defined as... Define overlapping U ij For node V i (V i ∈V I When the corresponding segmented block is projected onto the mask of this frame, it corresponds to the two-dimensional instance mask M in image I. j The percentage of the intersection area between them. For U with non-zero values. ij Node set Define the mask M in the segmentation graph j The central node is: For the remaining node V k ∈V I j If the node's U kj If it is greater than the threshold, then for V k and Voting is conducted on the edges between V. I Except for V I j Other nodes will have their values ​​deducted from V. Ij The graph is determined by voting on the edges. By traversing all nodes and edges in the graph, a graph consisting of nodes, edges, and the votes of the edges is obtained. Edges with positive votes are retained, and the output graph consists only of edges with positive votes.

[0147] A 3D instance segmentation graph is constructed based on multiple interconnected node groups in the graph. Each group of connected nodes can represent a 3D instance, and the output is an instance segmentation graph containing 3D instances.

[0148] Each 3D instance has an ID assigned within the 3D scene, i.e., an instance identifier. This ID is determined by the segmentation blocks of the mesh represented by each node, ensuring that each node in the mesh stores an ID, and nodes within the same 3D instance store the same ID. The mesh with these IDs is rasterized and projected onto the 2D instance segmentation image. Based on the IoU (Intersection over Union) between the projected mask and the original image mask (a metric for measuring the accuracy of object detection in a specific dataset), the 2D instances are reassigned corresponding 3D instance projected IDs. This aligns the 2D instance mask IDs with the 3D instance IDs, resulting in a segmentation map containing the 3D instances.

[0149] III. Output Stage

[0150] After going through the aforementioned stages one and two, the three-dimensional instances in the target scene can be segmented, and the instance identifier of each three-dimensional instance can be output.

[0151] Furthermore, it can also simultaneously output 3D instance segmentation results containing 3D instances, and / or 2D instance segmentation results.

[0152] Specifically, the 3D instance segmentation result may include an instance segmentation map containing 3D instances, point cloud data containing 3D instances, or a sequence containing 3D instances. When outputting both the instance identifier and the 3D instance segmentation result for each 3D instance, the instance identifier of each 3D instance can be directly carried in the 3D instance segmentation result. For example, an additional dimension of data can be added to the corner points or center points of each 3D instance to store the ID of the 3D instance.

[0153] The 2D instance segmentation result can include an instance segmentation map after aligning the IDs of the 2D instances with the IDs of the 3D instances. For example, after segmenting the 3D instances, an ID can be assigned to each 3D instance, and this ID can be assigned to the 2D instance it is aligned with, ensuring that the IDs of the 3D instances representing the same entity are consistent with those of the 2D instances. Alternatively, in the 2D instance segmentation map, an additional dimension can be added to the pixels corresponding to each 2D instance to store the 2D instance ID.

[0154] In one possible scenario, a three-dimensional implicit representation can also be output. For example, a NeRF model can be trained, and the parameters of the NeRF model can be used to achieve an implicit representation of the target scene.

[0155] For example, when the user inputs multiple frames of RGB, the two-dimensional instance segmentation result of each RGB image is obtained. After the aforementioned stage one and stage two, the two-dimensional instance segmentation results of each RGB image indicate that the two-dimensional instances representing the same entity have the same ID. The NeRF model is trained using the multiple two-dimensional instance segmentation results of multiple frames of RGB images, and the trained NeRF model is output so that downstream tasks can use the NeRF to capture the rendering image from any viewpoint.

[0156] It can be understood that, in the embodiments of this application, an implicit representation for representing a three-dimensional scene can be deployed. After obtaining an instance segmentation map containing three-dimensional instances based on two-dimensional instances in each frame of RGB image as supervision, the loss between the instance segmentation map containing three-dimensional instances and the three-dimensional implicit representation can be calculated. The three-dimensional implicit representation is updated based on the loss. As the number of frames of RGB images increases, the final output three-dimensional implicit representation becomes more accurate, thus outputting a more accurate three-dimensional implicit representation.

[0157] In this embodiment, a large visual language model is used to segment and output the mask corresponding to the two-dimensional instance. However, since the method is based on object detection, some objects may be missed due to occlusion of objects in the viewpoint. However, the shooting viewpoints corresponding to multiple frames may be different. Therefore, the two-dimensional instances obtained from segmentation based on different frames can be used to upgrade the two-dimensional instances to three-dimensional space and render them again as two-dimensional segmentation images to improve the accuracy of the instances included in the two-dimensional segmentation image, thereby outputting a more accurate two-dimensional instance segmentation result and avoiding the situation of missed detection under a single viewpoint.

[0158] Furthermore, the Segmentor algorithm typically performs a coarse segmentation into 3D instances composed of real-world primitives that are then aligned. The edges of the segmentation results may be accurate. By training a more precise implicit representation model, more accurate 3D instance segmentation can be provided, thereby improving the representation accuracy of input to downstream tasks.

[0159] In this embodiment, the instance segmentation map containing 3D instances is constructed based on the 3D reconstruction model of the scene. That is, 3D instance segmentation is performed by combining the 2D geometric priors of the 3D scene, which avoids classifying objects at distant locations or different locations that have not been observed in the same frame as the same instance. Simultaneously, since 2D segmentation is based on object detection, the supervision of graph segmentation comes from the 2D segmentation results, thus ensuring instance segmentation at the object-level granularity. Furthermore, when performing instance segmentation on at least one image, a large visual language model can be used. This large-scale visual language model empowers the robot's environmental understanding and perception capabilities, generating instance segmentation data from any viewpoint, ensuring the consistency between the 3D and 2D data of the instance segmentation data, thereby achieving 3D instance segmentation and 2D instance segmentation map generation with zero training samples.

[0160] Furthermore, in specific application scenarios, the output instance IDs can be used for downstream tasks, such as environmental perception and understanding of intelligent operating robots. They can also generate a large number of ground truth labels as training data for segmentation models, or generate model editing scenarios based on instance segmentation results combined with stablediffusion, etc.

[0161] For example, in autonomous driving scenarios, vehicles need to perceive the real-world environment in which they are located. The method provided in this application can be used to segment the collected images into instances and assign a unique ID to each entity in the environment. This allows the same entity to be identified in different image frames during environmental perception, and further perception of the entity's state changes, thereby making more accurate driving decisions for the vehicle, or rendering and displaying the environment in which the vehicle is located.

[0162] For example, in the context of robot environmental perception, the robot can use the method provided in the embodiments of this application to perform instance segmentation on the acquired images, thereby achieving accurate differentiation of each entity, and performing the required tasks based on the recognition results of each entity, such as grasping operations, scene reconstruction, and accumulation of object data assets.

[0163] For example, the method provided in this application can be deployed in a cloud platform. Through the data interface specified by the platform, it can provide raw data collected from the real world. The platform can provide the created virtual scene of the real world and the 3D mesh and 2D segmented image of the scene instance segmentation, and provide them to the user, thereby providing the user with instance segmentation service.

[0164] For example, in open-world application scenarios, the method provided in this application can be used to construct a 3D graph by aligning the 3D model with the 2D instance mask of the open scene, and then perform graph inference to output an instance segmentation graph containing 3D instances. This achieves 3D instance segmentation in open scenes with zero training and 2D instance segmentation consistent with 3D, providing users with a more accurate 3D model of the open world.

[0165] The method flow provided in this application has been described above. The structure of the apparatus for performing the above method flow is described below.

[0166] Referring to Figure 7, a schematic diagram of an instance segmentation device provided in an embodiment of this application includes:

[0167] Input module 701 is used to acquire at least one image of the target scene, the camera view of at least one image, and a 3D model of the target scene;

[0168] The two-dimensional segmentation module 702 is used to perform instance segmentation on at least one image to obtain a first two-dimensional instance segmentation result, wherein the first two-dimensional instance segmentation result includes at least one two-dimensional instance.

[0169] The 3D segmentation module 703 is used to obtain at least one primitive in a 3D model, the at least one primitive including primitives formed by corner points and facets in the 3D model;

[0170] Projection module 704 is used to project at least one two-dimensional instance and at least one primitive through a camera viewpoint to determine at least one three-dimensional instance, wherein the at least one three-dimensional instance includes a structure formed by at least one primitive;

[0171] Output module 705 is used to output the instance identifier of each 3D instance.

[0172] In one possible implementation, the aforementioned projection module 704 is used to: project at least one two-dimensional instance and at least one primitive based on the camera viewpoint, and obtain the overlap area between each two-dimensional instance and each primitive in the projection space; and combine one or more primitives with an overlap area greater than a preset area with the same two-dimensional instance to form a three-dimensional instance, thereby obtaining at least one three-dimensional instance.

[0173] In one possible implementation, the aforementioned projection module 704 is specifically used to: project at least one primitive onto the imaging plane of a two-dimensional instance, and obtain the overlap area between each two-dimensional instance and each primitive in the imaging plane.

[0174] In one possible implementation, the aforementioned two-dimensional segmentation module 702 is specifically used for: acquiring input text, which includes text indicating the segmentation of the image; using the image and the input text as input to a large visual language model; and outputting a first two-dimensional instance segmentation result.

[0175] In one possible implementation, the aforementioned apparatus further includes: a training module 706, configured to acquire multiple second two-dimensional instance segmentation results, wherein the instance identifier of the same instance in each of the multiple second two-dimensional instance segmentation results is the same; and to train a neural network model based on the multiple second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene.

[0176] In one possible implementation, the aforementioned three-dimensional segmentation module 703 is specifically used to cluster corner points or facets in the three-dimensional model according to the normal vector of the surface of the three-dimensional model, and take a set of corner points and / or facets under a cluster category as a primitive to obtain at least one primitive.

[0177] In one possible implementation, the aforementioned input module 701 is specifically used to: obtain a three-dimensional model from user input data; or to perform three-dimensional reconstruction based on multiple frames of images of the target scene to obtain a three-dimensional model.

[0178] In one possible implementation, the aforementioned output module 705 is further configured to: output a three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating at least one three-dimensional instance in the three-dimensional model; or, output at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result including at least one two-dimensional instance, the instance identifier of each two-dimensional instance being the same as the instance identifier of the three-dimensional instance representing the same entity.

[0179] Each module in the aforementioned instance segmentation device can be implemented in software or hardware. For example, the implementation of the projection module will be described below. Similarly, the implementation of other modules, such as the input module, 2D segmentation module, 3D segmentation module, output module, or training module, can refer to the implementation of the projection module.

[0180] As an example of a software functional unit, a projection module may include code running on a computing instance. This computing instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Furthermore, the aforementioned computing devices may be one or more. For example, a projection module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed within the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed within the same Availability Zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0181] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0182] As an example of a hardware functional unit, a projection module may include at least one computing device, such as a server. Alternatively, a projection module may also be a device implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), an embedded neural network process unit (NPU), a system-on-chip (SoC), an offloading card, or an accelerator card. The aforementioned PLD can be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0183] The projection module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the projection module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the projection module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0184] As an example of a software functional unit, an instance partitioning device can include code running on compute instances. A compute instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Further, the aforementioned computing devices can be one or more. For example, an instance partitioning device can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same Availability Zone (AZ) or in different AZs, each AZ including one data center or multiple geographically proximate data centers. Typically, a region can include multiple AZs.

[0185] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0186] As an example of a hardware functional unit, an instance partitioning device may include at least one computing device, such as a server. Alternatively, the instance partitioning device may also be a device implemented using an ASIC or a PLD. The aforementioned PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0187] The instance partitioning device includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the instance partitioning device can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the instance partitioning device can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0188] This application also provides a chip system including a processor and a power supply circuit. The power supply circuit supplies power to the processor, which executes the operation steps corresponding to the method provided in this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices such as a DPU, NPU, Scalable Processing Unit (XPU), SoC, offload card, or accelerator card.

[0189] This application also provides a computing device 100. As shown in FIG8, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0190] Bus 102 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in Figure 8, but this does not imply that there is only one bus or one type of bus. Bus 104 can include pathways for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108). The Unified Bus can also be referred to as the Lingqu Bus.

[0191] The processor 104 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0192] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 106 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0193] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 106, or two or more types of storage media can be configured to realize the function of memory 106. This application does not limit this.

[0194] The memory 106 stores executable program code, and the processor 104 executes the executable program code to implement the functions of each module mentioned in FIG10, thereby implementing the method provided in this application. That is, the memory 106 stores instructions for executing the method provided in this application.

[0195] Alternatively, the memory 106 stores executable code, which the processor 104 executes to implement the functions of the aforementioned input module, two-dimensional segmentation module, three-dimensional segmentation module, projection module, output module, or training module, thereby implementing the method provided in this application. That is, the memory 106 stores instructions for executing the method provided in this application.

[0196] The communication interface 103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0197] As one possible implementation, the computing device 100 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the method provided in this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offload card, or accelerator card.

[0198] As one possible implementation, the computing device 100 may include various types of processors 104, meaning the computing device 100 is a heterogeneous device. For example, the computing device 100 may include a CPU and a GPU, and at least one of the processors 104 may execute the operation steps corresponding to the method provided in this application. For the sake of brevity, further details are omitted here.

[0199] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0200] As shown in Figure 9, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for executing the methods provided in this application.

[0201] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the methods provided in this application. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the methods provided in this application.

[0202] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the joint testing device. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more of the aforementioned input module, two-dimensional segmentation module, three-dimensional segmentation module, projection module, output module, or training module.

[0203] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 106 in computing device 100A stores instructions for performing the functions of an input module, a 2D segmentation module, a 3D segmentation module, a projection module, an output module, or a training module. Simultaneously, the memory 106 in computing device 100B stores instructions for performing the functions of an input module, a 2D segmentation module, a 3D segmentation module, a projection module, an output module, or a training module.

[0204] It should be understood that the functions of computing device 100A shown in Figure 10 can also be performed by multiple computing devices 100. Similarly, the functions of computing device 100B can also be performed by multiple computing devices 100.

[0205] The connection method between the computing device clusters shown in Figure 10 can be based on the fact that the method provided in this application requires a large amount of computing power, needs to achieve load balancing, or requires a large amount of data storage. Therefore, different modules are deployed in different computing devices. For example, the functions implemented by the output module are executed by computing device 100A, and the functions implemented by the projection module are executed by computing device 100B.

[0206] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 9 and 10. The difference is that the memory 106 of one or more computing devices 100 in this computing device cluster can store the same instructions for executing the method provided in this application.

[0207] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the methods provided in this application. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the methods provided in this application.

[0208] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions for executing some functions of the instance segmentation apparatus provided in this application. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of an input module, a two-dimensional segmentation module, a three-dimensional segmentation module, a projection module, an output module, or a training module.

[0209] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method provided in this application.

[0210] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the method provided in this application.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. An instance segmentation method, characterized by, The method comprises: obtaining at least one image of a target scene, a camera perspective of the at least one image, and a three-dimensional model of the target scene; performing instance segmentation on the at least one image to obtain a first two-dimensional instance segmentation result, the first two-dimensional instance segmentation result comprising at least one two-dimensional instance; obtaining at least one primitive in the three-dimensional model, the at least one primitive comprising a primitive formed by a vertex and a face in the three-dimensional model; projecting the at least one two-dimensional instance and the at least one primitive according to the camera perspective to determine at least one three-dimensional instance in the three-dimensional model, the at least one three-dimensional instance comprising a structure formed by the at least one primitive; outputting an instance identifier of the at least one three-dimensional instance.

2. The method of claim 1, wherein, The method further comprises: obtaining an overlap area between each two-dimensional instance and each primitive in a projection space based on the projection of the at least one two-dimensional instance and the at least one primitive according to the camera perspective; grouping one or more primitives with an overlap area greater than a preset area with the same two-dimensional instance to form a three-dimensional instance, thereby obtaining the at least one three-dimensional instance.

3. The method of claim 2, wherein, The method further comprises: projecting the at least one primitive into an imaging plane of the two-dimensional instance to obtain an overlap area between each two-dimensional instance and each primitive in the imaging plane.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: obtaining an input text, the input text comprising text for instructing segmentation of the image; inputting the at least one image and the input text into a large visual language model to output the first instance segmentation result.

5. The method according to any one of claims 1-4, characterized in that, The method further comprises: obtaining a plurality of second two-dimensional instance segmentation results, each second two-dimensional instance segmentation result in the plurality of second two-dimensional instance segmentation results having the same instance identifier of the same instance; training a neural network model based on the plurality of second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene.

6. The method according to any one of claims 1-5, characterized in that, The method further comprises: clustering the vertices or the faces in the three-dimensional model according to normal vectors of surfaces of the three-dimensional model, grouping the vertices and / or the faces in a cluster category as a primitive, and obtaining the at least one primitive.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: obtaining the three-dimensional model from user input data; or, performing three-dimensional reconstruction based on a plurality of images of the target scene to obtain the three-dimensional model.

8. The method according to any one of claims 1-6, characterized in that, The method further comprises: outputting a three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating the at least one three-dimensional instance in the three-dimensional model. Or, output at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result including the at least one two-dimensional instance, and the instance identifier of each two-dimensional instance being the same as the instance identifier of the three-dimensional instance representing the same entity.

9. An instance segmentation device, comprising: Comprise: An input module configured to obtain at least one image of a target scene, a camera perspective of the at least one image, and a three-dimensional model of the target scene; A two-dimensional segmentation module configured to perform instance segmentation on the at least one image to obtain a first two-dimensional instance segmentation result, the first two-dimensional instance segmentation result including at least one two-dimensional instance; A three-dimensional segmentation module configured to obtain at least one primitive in the three-dimensional model, the at least one primitive including a primitive formed by a vertex and a face in the three-dimensional model; A projection module configured to project the at least one two-dimensional instance and the at least one primitive based on the camera perspective to determine at least one three-dimensional instance, the at least one three-dimensional instance including a structure formed by the at least one primitive; An output module configured to output an instance identifier of the at least one three-dimensional instance.

10. The apparatus of claim 9, wherein, The projection module is configured to: Project the at least one two-dimensional instance and the at least one primitive based on the camera perspective to obtain an overlapping area between each two-dimensional instance and each primitive in a projection space; Group one or more primitives with an overlapping area greater than a preset area of the same two-dimensional instance to form a three-dimensional instance to obtain the at least one three-dimensional instance.

11. The apparatus of claim 10, wherein, The projection module is specifically configured to: Project the at least one primitive into an imaging plane of the two-dimensional instance to obtain an overlapping area between each two-dimensional instance and each primitive in the imaging plane.

12. The apparatus of any one of claims 9-11, wherein, The two-dimensional segmentation module is specifically configured to: Obtain an input text, the input text including a text for instructing segmentation of the image; Input the at least one image and the input text into a large visual language model to output the first two-dimensional instance segmentation result.

13. The apparatus of any one of claims 9-12, wherein, The device further comprises a training module configured to: Obtain a plurality of second two-dimensional instance segmentation results, the instance identifier of the same instance in each of the plurality of second two-dimensional instance segmentation results being the same; Train a neural network model based on the plurality of second two-dimensional instance segmentation results to obtain a three-dimensional implicit representation of the target scene.

14. The device of any one of claims 9-13, wherein: The three-dimensional segmentation module is specifically configured to cluster the vertices or the faces in the three-dimensional model according to normal vectors of a surface of the three-dimensional model, and group the vertices and / or the faces under a cluster category as a primitive to obtain the at least one primitive.

15. The apparatus of any one of claims 9-14, wherein, The input module is specifically configured to: Obtain the three-dimensional model from user input data; Or, perform three-dimensional reconstruction based on a plurality of images of the target scene to obtain the three-dimensional model.

16. The apparatus of any one of claims 9-15, wherein, The output module is further configured to: Output a three-dimensional instance segmentation result, the three-dimensional instance segmentation result indicating the at least one three-dimensional instance in the three-dimensional model; Or, outputting at least one second two-dimensional instance segmentation result, the at least one second two-dimensional instance segmentation result comprising the at least one two-dimensional instance, each two-dimensional instance having an instance identifier identical to an instance identifier of the three-dimensional instance representing the same entity.

17. A cluster of computing devices, characterized in that, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the processing steps of the method of any one of claims 1 to 8.

18. A computer program product comprising instructions, characterized in that, the instructions, when executed by the cluster of computing devices, cause the cluster of computing devices to perform the processing steps of the method of any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, computer program instructions, which, when executed by the cluster of computing devices, cause the cluster of computing devices to perform the processing steps of the method of any one of claims 1 to 8.

Images