Mask generation method, system and electronic equipment of two-dimensional image and storage medium

Abstract

The application relates to a mask generation method of a two-dimensional image, which comprises the following steps: acquiring scene image information and a three-dimensional grid model of a target object, obtaining an ID graph of a scene image according to the IDs of each face sheet in the scene image information and the three-dimensional grid model, obtaining a face sheet ID set of the target object according to a target image in which the target object is labeled in a scene image, an ID graph of the target image and the three-dimensional grid model, and obtaining a prompt box of the target object in the scene image and a mask in the prompt box based on the ID graph and the face sheet ID set. Through the application, the problem of low efficiency of the mask generation method of the two-dimensional image is solved, the face sheet corresponding to the target object is determined from the three-dimensional grid model based on a small amount of images in which the target object is labeled, the pixel points corresponding to the target object in each scene image are determined, the masks of the scene images are obtained, the efficiency of the two-dimensional image mask generation is improved, and the method only needs light software for displaying and recording clicks.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to methods, systems, electronic devices, and storage media for generating masks for two-dimensional images. Background Technology

[0002] An object instance mask is a pixel-level precise segmentation of a target object. It is a binary image composed of 0s and 1s. When a mask is applied in a function, regions with 1 values ​​are processed, while regions with 0 values ​​are masked out and not included in the calculation. In augmented reality (AR) technology, by obtaining a mask of the set of images in which the target object appears in an AR map, the target object can be identified, and occlusion relationships between the target object and AR content can be established.

[0003] In related technologies, obtaining an image mask requires manually annotating the semantics of each object on a 2D image using specialized software, and then mapping it onto a 3D model of the target scene. Manual annotation requires specialized training and is inefficient. Furthermore, this method requires specialized software and a large amount of manpower, resulting in high costs.

[0004] Currently, no effective solution has been proposed to address the low efficiency of mask generation methods for 2D images in related technologies. Summary of the Invention

[0005] This application provides a method, system, electronic device, and storage medium for generating masks for two-dimensional images, in order to at least solve the problem of low efficiency in related technologies for generating masks for two-dimensional images.

[0006] In a first aspect, embodiments of this application provide a method for generating a mask for a two-dimensional image, the method comprising:

[0007] Acquire scene image information and 3D mesh models of target objects;

[0008] Based on the scene image information and the IDs of each facet in the 3D mesh model, an ID map of the scene image is obtained, wherein the ID map labels the ID of the facet corresponding to each pixel in the scene image;

[0009] Based on the target image, the ID map of the target image, and the 3D mesh model, a set of facet IDs of the target object is obtained, wherein the target image is a scene image labeled with the target object;

[0010] Based on the ID map and the set of patch IDs, the prompt box of the target object in the scene image and the mask within the prompt box are obtained.

[0011] In some embodiments, obtaining the set of facet IDs of the target object based on the target image, the ID map of the target image, and the 3D mesh model includes:

[0012] Generate a mask for the target image based on the annotations in the target image;

[0013] Based on the mask of the target image, the ID of the target object in the ID map of each target image is estimated to obtain the ID prediction set of the target image;

[0014] Based on the connectivity of the patches, the patches corresponding to the ID prediction set are divided into different connected regions to obtain the ID prediction subset of the connected regions;

[0015] Based on the ID prediction subset, determine the set of facet IDs of the target object.

[0016] In some embodiments, determining the set of facet IDs of the target object based on the ID estimation subset includes:

[0017] Determine the target ID prediction subset for each target image from the ID prediction subset;

[0018] Based on all the predicted subsets of target IDs, determine the set of facet IDs of the target object.

[0019] In some embodiments, determining the target ID prediction subset for each target image from the ID prediction subset includes:

[0020] Determine the number of patch IDs for each of the connected regions in the target image;

[0021] The target ID prediction subset is determined from the ID prediction subset based on the maximum value among the number of patch IDs.

[0022] In some embodiments, determining the target ID prediction subset for each target image from the ID prediction subset includes:

[0023] Based on the distance from the vertex of the patch in the target image to the camera device, determine the average distance from the vertex of each connected region patch to the camera device, wherein the camera device is the device that acquires the target image;

[0024] Based on the minimum value of the average distance, a target connected region is determined from the connected regions, and the target ID prediction subset is obtained based on the target connected region.

[0025] In some embodiments, obtaining the ID map of the scene image based on the IDs of each patch in the 3D mesh model includes:

[0026] Obtain the pose of the scene image from the scene image information;

[0027] Based on the pose and the IDs of each facet in the 3D mesh model, the facet IDs corresponding to each pixel in the scene image are obtained, and the ID map is generated.

[0028] In some embodiments, obtaining the tooltip of the target object in the scene image and the mask within the tooltip based on the ID map and the set of patch IDs includes:

[0029] Based on the set of facet IDs, determine the ID element representing the target object in the ID diagram;

[0030] Obtain the maximum x-coordinate, maximum y-coordinate, minimum x-coordinate, and minimum y-coordinate of the ID element;

[0031] Based on the maximum x-coordinate, the maximum y-coordinate, the minimum x-coordinate, and the minimum y-coordinate, a prompt box for the target object is generated in the scene image;

[0032] Based on the ID map and the set of patch IDs, a mask for the scene image within the prompt box is obtained.

[0033] Secondly, embodiments of this application provide a mask generation system for two-dimensional images, the system comprising: an information acquisition module, an ID image generation module, an ID set acquisition module, and a mask generation module, wherein,

[0034] The information acquisition module is used to acquire scene image information and a three-dimensional mesh model of the target object;

[0035] The ID map generation module is used to obtain an ID map of the scene image based on the scene image information and the IDs of each facet in the three-dimensional mesh model, wherein the ID map labels the ID of the facet corresponding to each pixel in the scene image;

[0036] The ID set acquisition module is used to obtain a set of facet IDs of the target object based on the target image, the ID map of the target image, and the three-dimensional mesh model, wherein the target image is a scene image labeled with the target object;

[0037] The mask generation module is used to obtain the prompt box of the target object in the scene image and the mask within the prompt box based on the ID map and the set of patch IDs.

[0038] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the mask generation method for two-dimensional images as described in the first aspect above.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the mask generation method for two-dimensional images as described in the first aspect above.

[0040] Compared to related technologies, the 2D image mask generation method provided in this application obtains scene image information and a 3D mesh model of the target object. Based on the scene image information and the IDs of each facet in the 3D mesh model, an ID map of the scene image is obtained. Based on the target image annotated with the target object, the ID map of the target image, and the 3D mesh model, a set of facet IDs of the target object is obtained. Based on the ID map and the set of facet IDs, a prompt box for the target object in the scene image and the mask within the prompt box are obtained. This solves the problem of low efficiency in 2D image mask generation methods. By using a small number of images annotated with the target object, the facets corresponding to the target object are determined from the 3D mesh model, thereby determining the pixels corresponding to the target object in each scene image and obtaining the mask for each scene image. This improves the efficiency of 2D image mask generation. Furthermore, this method only requires lightweight software for displaying and recording clicks, resulting in low cost. Attached Figure Description

[0041] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0042] Figure 1 This is a schematic diagram of the application environment of the mask generation method for two-dimensional images according to an embodiment of this application;

[0043] Figure 2 This is a flowchart of a two-dimensional image mask generation method according to an embodiment of this application;

[0044] Figure 3 A schematic diagram of a scene image acquisition method according to an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of a target object annotation according to an embodiment of this application;

[0046] Figure 5 It is an ID image of scene image rendering according to an embodiment of this application;

[0047] Figure 6This is a schematic diagram of the target object surface annotation model according to an embodiment of this application;

[0048] Figure 7 This is a schematic diagram of an image mask annotation according to an embodiment of this application;

[0049] Figure 8 This is a structural block diagram of a two-dimensional image mask generation system according to an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0052] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0053] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0054] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0055] Figure 1 This is a schematic diagram illustrating the application environment of the two-dimensional image mask generation method according to an embodiment of this application. The two-dimensional image mask generation method provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, server 10 acquires scene image information and a 3D mesh model of the target object. Based on the scene image information and the IDs of each facet in the 3D mesh model, it obtains an ID map of the scene image. Based on the target image, the ID map of the target image, and the 3D mesh model, it determines the set of facet IDs of the target object. Based on the ID map and the set of facet IDs, it obtains the tooltip of the target object in the scene image and the mask within the tooltip. Server 10 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0056] This embodiment provides a method for generating a mask for a two-dimensional image. Figure 2 This is a flowchart of a two-dimensional image mask generation method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0057] Step S201: Obtain scene image information and a 3D mesh model of the target object.

[0058] Figure 3A schematic diagram of a scene image acquisition method according to an embodiment of this application is shown below. Figure 3 As shown, the scene consists of target objects and non-target objects. Based on the camera trajectory, several scene images and their corresponding image poses are obtained. The scene images include those containing target objects and those not containing target objects.

[0059] Select a preset number of scene images (e.g., 3 or 4) containing observations of the target object from different angles, and annotate the target object to obtain the target image. The target object annotation includes the target object's bounding box and pixel annotations. The target object's bounding box can be obtained through manual annotation, object detection algorithms, or tracking algorithms. The target object's pixel annotations can be obtained by segmenting a large segmentation model based on the bounding box. Only a small number of scene images used as references need to be annotated, and manual annotation of the bounding box is simple and quick, requiring no professional training.

[0060] Figure 4 This is a schematic diagram of target object annotation according to an embodiment of this application, such as... Figure 4 As shown, a small number of scene images were selected for annotation. Each target image is annotated with a target object prompt box and the segmentation result of the target object.

[0061] Scene image information includes the target image, the scene image of the unlabeled target object, and the image poses corresponding to all scene images.

[0062] A 3D mesh model consists of a series of vertices and faces connecting these vertices, used to represent the 3D shape of an object or scene. In a mesh, vertices represent points in 3D space, and "faces" represent planes composed of multiple vertices. A face contains three or more vertices, and a point can belong to multiple faces. Faces containing the same vertices are adjacent faces. Each face has its own unique ID; that is, different faces have different IDs, and one ID corresponds to only one face.

[0063] Step S202: Based on the scene image information and the IDs of each facet in the 3D mesh model, obtain the ID map of the scene image, where the ID map labels the ID of the facet corresponding to each pixel in the scene image.

[0064] In some embodiments, step S202 specifically includes:

[0065] Step S2021: Obtain the pose of the scene image from the scene image information.

[0066] Step S2022: Based on the pose and the ID of each facet in the 3D mesh model, obtain the facet ID corresponding to each pixel in the scene image, and generate an ID map.

[0067] An ID map is an image where each pixel records the ID of a face in a 3D mesh model. Essentially, it renders the faces from the 3D mesh model onto the pixels of an image, with different face IDs corresponding to different grayscale values. Rendering involves finding the correspondence between a 2D image with a specified pose and camera parameters and the mesh. This relationship simulates actual physical optics, mimicking the information of the mesh at the intersection of a ray emanating from the camera's optical center, passing through the pixel's position, and the mesh itself. The most common rendering is the mapping of virtual game content to the RGB representation of the game screen. Rendering processes are often integrated into software like Unity and OpenGL; in this application, rendering can be implemented using a custom rendering program.

[0068] Figure 5 It is an ID image for scene image rendering according to an embodiment of this application, such as Figure 5 As shown in the diagram, different shades of gray represent different face IDs.

[0069] Step S203: Based on the target image, the ID map of the target image, and the 3D mesh model, obtain the set of facet IDs of the target object, where the target image is a scene image labeled with the target object.

[0070] In some embodiments, step S203 specifically includes:

[0071] Step S2031: Generate a mask for the target image based on the annotations of the target image.

[0072] A mask is generated based on the annotation information in the target image. In this mask, "1" represents the pixel corresponding to the target object and "0" represents the pixel corresponding to a non-target object.

[0073] Step S2032: Based on the mask of the target image, estimate the ID of the target object in the ID map of each target image to obtain the ID prediction set of the target images.

[0074] The set of IDs corresponding to pixels with an ID of "1" in the target image mask is denoted as I1, which is the ID prediction set. The faces with IDs belonging to I1 are extracted from the 3D mesh model and combined into a set Mesh1.

[0075] Because there are errors in the accuracy of the 3D mesh model obtained from 3D reconstruction and the pose estimation of the target image, some IDs may correspond to patches near the target object, resulting in the collection Mesh1 possibly containing patches corresponding to some of the background. This embodiment optimizes the collection Mesh1 through steps S2033 and S2034.

[0076] Step S2033: Based on the connectivity of the patches, divide the patches corresponding to the ID prediction set into different connected regions to obtain the ID prediction subset of the connected regions.

[0077] Connectivity refers to the adjacency and connection of faces, which can be determined by examining the adjacency relationships between faces. Two faces are adjacent if they share one or more vertices; a face is connected to its adjacent faces, and faces connected to its adjacent faces are also connected. In other words, if a face can move through adjacency to reach another face, then those two faces are connected. All interconnected faces form a connected region.

[0078] Divide the patches corresponding to the ID prediction set I1 into n connected regions, and the set of these connected regions is Mesh_part(i), i = 1, 2, 3, ..., n. Count the patch IDs corresponding to each connected region to obtain the corresponding ID prediction subset I_part(i).

[0079] Step S2034: Determine the set of face IDs of the target object based on the ID prediction subset.

[0080] In some embodiments, step S2034 specifically includes:

[0081] Step S301: Determine the target ID prediction subset for each target image from the ID prediction subset.

[0082] The connected region i* of the target object is determined from multiple connected regions, and its corresponding set of faces is Mesh_part(i*). The set of face IDs corresponding to Mesh_part(i*) is obtained, which is the target ID prediction subset I_part(i*). The connected region of the target object can be determined based on the number of faces or the distance from the face vertices to the camera device.

[0083] In some embodiments, the connected regions of the target object are determined by the number of facets. Step S301 specifically includes:

[0084] Step S3011: Determine the number of patch IDs for each connected region in the target image.

[0085] Step S3012: Determine the target ID prediction subset from the ID prediction subset based on the maximum value among the number of face IDs.

[0086] Since the 3D mesh model obtained from the 3D reconstruction has a small error, the error only corresponds to the edge parts of the target object. Therefore, the connected regions with the largest area ratio are the connected regions of the target object. Count the number of IDs in each ID prediction subset I_part(i), and the ID prediction subset with the most IDs is the target ID prediction subset I_part(i*).

[0087] In some embodiments, the connected regions of the target object are determined by the distance from the vertex of the facet to the camera device. Step S301 specifically includes:

[0088] Step S3013: Determine the average distance from the vertex of each connected region to the camera device based on the distance from the vertex of the face in the target image to the camera device, wherein the camera device is the device for acquiring the target image;

[0089] Step S3014: Based on the minimum average distance, determine the target connected region from the connected regions, and obtain the target ID prediction subset based on the target connected region.

[0090] Generally, the connected region closest to the camera when capturing an image is usually the connected region corresponding to the target object. The average distance from the camera to each patch in the connected region is calculated, and the connected region with the smallest average distance is the target connected region.

[0091] After determining the target ID prediction subset for each target image in step S301, step S302 is executed.

[0092] Step S302: Determine the set of face IDs of the target object based on all estimated subsets of target IDs.

[0093] The union of the estimated subsets I_part(i*) of target IDs for each target image is the set of facet IDs of the target object.

[0094] Figure 6 This is a schematic diagram of the target object patch annotation model according to an embodiment of this application, such as... Figure 6 As shown, the white 3D model represents the model of an object in the scene. The target 3D model, also known as the black 3D model in the image, is determined from the set of facet IDs of the target object. Based on this target 3D model, the mask corresponding to the scene image can be obtained.

[0095] Continue to refer to Figure 2 After obtaining the set of face IDs of the target object in step S203, step S204 is executed.

[0096] Step S204: Based on the ID map and the set of patch IDs, obtain the cue box of the target object in the scene image and the mask within the cue box.

[0097] In some embodiments, step S204 specifically includes:

[0098] Step S2041: Based on the set of facet IDs, determine the ID element representing the target object in the ID diagram.

[0099] Step S2042: Obtain the maximum x-coordinate, maximum y-coordinate, minimum x-coordinate, and minimum y-coordinate of the ID element.

[0100] Step S2043: Generate a tooltip for the target object in the scene image based on the maximum x-coordinate, maximum y-coordinate, minimum x-coordinate, and minimum y-coordinate.

[0101] Step S2044: Based on the ID map and the set of patch IDs, obtain the mask of the scene image within the prompt box.

[0102] Based on the maximum and minimum x and y coordinates of the target object's corresponding ID in the ID map, determine the position and size of the tooltip in the corresponding scene image. Mark the tooltip in the scene image. Simultaneously, based on the target object's patch ID set and the scene image's ID map, determine the corresponding pixels of the target object in the scene image, and generate a mask for the scene image corresponding to the tooltip.

[0103] Figure 7 This is a schematic diagram of an image mask annotation according to an embodiment of this application, such as... Figure 7 As shown, the target object's tooltip and the mask within the tooltip are marked in the scene image containing the target object, while the information without the tooltip is marked in the scene image without the target object.

[0104] Through the above steps, scene image information and a 3D mesh model of the target object are obtained. Based on the scene image information and the IDs of each facet in the 3D mesh model, an ID map of the scene image is obtained. Based on the target image annotated with the target object in the scene image, the ID map of the target image, and the 3D mesh model, a set of facet IDs of the target object is obtained. Based on the ID map and the set of facet IDs, a tooltip and a mask within the tooltip of the target object in the scene image are obtained, solving the problem of low efficiency in 2D image mask generation methods. Using a small number of images annotated with the target object, the facets corresponding to the target object are determined from the 3D mesh model, thereby determining the pixels corresponding to the target object in each scene image and obtaining the mask for each scene image. This improves the efficiency of 2D image mask generation, and the method only requires lightweight software for displaying and recording clicks, resulting in low cost. Furthermore, the 2D image mask generation method provided in this application is applicable to target objects of any shape.

[0105] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0106] This embodiment also provides a mask generation system for two-dimensional images. This device is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0107] Figure 8 This is a structural block diagram of a two-dimensional image mask generation system according to an embodiment of this application, such as... Figure 8 As shown, the system includes: an information acquisition module 81, an ID graph generation module 82, an ID set acquisition module 83, and a mask generation module 84.

[0108] Information acquisition module 81 is used to acquire scene image information and the three-dimensional mesh model of the target object;

[0109] ID map generation module 82 is used to obtain the ID map of the scene image based on the scene image information and the ID of each facet in the 3D mesh model. The ID map indicates the ID of the facet corresponding to each pixel in the scene image.

[0110] ID set acquisition module 83 is used to obtain the facet ID set of the target object based on the target image, the ID map of the target image and the 3D mesh model, wherein the target image is a scene image with the target object annotated;

[0111] The mask generation module 84 is used to obtain the target object's bounding box and the mask within the bounding box in the scene image based on the ID map and the set of patch IDs.

[0112] In some embodiments, the ID set acquisition module 83 includes: a first mask generation module, an ID prediction module, a region division module, and an ID set determination module.

[0113] The first mask generation module is used to generate a mask for the target image based on the annotations of the target image.

[0114] The ID prediction module is used to estimate the ID of the target object in the ID map of each target image based on the mask of the target image, and obtain the ID prediction set of the target images.

[0115] The region partitioning module is used to divide the ID prediction set into different connected regions based on the connectivity of the patches, thereby obtaining the ID prediction subset of the connected regions.

[0116] The ID set determination module is used to determine the set of face IDs of a target object based on the estimated subset of IDs.

[0117] In some embodiments, the ID set determination module includes a target subset determination module and a set merging module.

[0118] The target subset determination module is used to determine the target ID prediction subset of each target image from the ID prediction subset.

[0119] The set merging module is used to determine the set of face IDs of a target object based on the estimated subset of all target IDs.

[0120] In some embodiments, the target subset determination module includes: a patch quantity calculation module and a first target subset determination module.

[0121] The patch count calculation module is used to determine the number of patch IDs in each connected region of the target image.

[0122] The first target subset determination module is used to determine the target ID estimation subset from the ID estimation subset based on the maximum value among the number of patch IDs.

[0123] In some embodiments, the target subset determination module further includes: an average distance calculation module and a second target subset determination module.

[0124] The average distance calculation module is used to determine the average distance from the vertex of each connected region to the camera device based on the distance from the vertex of the face in the target image to the camera device, wherein the camera device is the device that acquires the target image.

[0125] The second target subset determination module is used to determine the target connected region from the connected regions based on the minimum average distance, and obtain the target ID estimated subset based on the target connected region.

[0126] In some embodiments, the ID graph generation module 82 includes a pose acquisition module and an ID matching module.

[0127] The pose acquisition module is used to obtain the pose of the scene image from the scene image information.

[0128] The ID matching module is used to obtain the face ID corresponding to each pixel in the scene image based on the pose and the ID of each face in the 3D mesh model, and generate an ID map.

[0129] In some embodiments, the mask generation module 84 includes: a target ID determination module, a coordinate determination module, a prompt box generation module, and an in-box mask generation module.

[0130] The target ID determination module is used to determine the ID element representing the target object in the ID diagram based on the set of facet IDs;

[0131] The coordinate determination module is used to obtain the maximum x-coordinate, maximum y-coordinate, minimum x-coordinate, and minimum y-coordinate of an ID element;

[0132] The tooltip generation module is used to generate tooltips for target objects in the scene image based on the maximum x-coordinate, maximum y-coordinate, minimum x-coordinate, and minimum y-coordinate.

[0133] The in-frame mask generation module is used to obtain a mask of the scene image within the prompt box based on the ID map and the set of patch IDs.

[0134] Through the above system, the information acquisition module 81 acquires scene image information and the 3D mesh model of the target object; the ID map generation module 82 obtains the ID map of the scene image based on the scene image information and the IDs of each facet in the 3D mesh model; the ID set acquisition module 83 obtains the facet ID set of the target object based on the target image annotated with the target object, the ID map of the target image, and the 3D mesh model; and the mask generation module 84 obtains the prompt box of the target object in the scene image and the mask within the prompt box based on the ID map and the facet ID set. This solves the problem of low efficiency in 2D image mask generation methods. Based on a small number of images annotated with the target object, the corresponding facet of the target object is determined from the 3D mesh model, thereby determining the pixel points corresponding to the target object in each scene image and obtaining the mask for each scene image. This improves the efficiency of 2D image mask generation, and the method only requires lightweight software for displaying and recording clicks, resulting in low cost.

[0135] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0136] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0137] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0138] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0139] S1, acquire scene image information and the 3D mesh model of the target object.

[0140] S2. Based on the scene image information and the IDs of each facet in the 3D mesh model, obtain the ID map of the scene image. The ID map indicates the ID of the facet corresponding to each pixel in the scene image.

[0141] S3. Based on the target image, the ID map of the target image, and the 3D mesh model, obtain the set of facet IDs of the target object, where the target image is a scene image with the target object annotated.

[0142] S4, based on the ID map and the set of patch IDs, obtain the tooltip of the target object in the scene image and the mask within the tooltip.

[0143] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0144] In one embodiment, Figure 9 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 9 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 9 As shown, this electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a method for generating a mask for a two-dimensional image.

[0145] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0147] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for generating a mask for a two-dimensional image, characterized in that, The method includes: Acquire scene image information and 3D mesh models of target objects; Based on the scene image information and the IDs of each facet in the 3D mesh model, an ID map of the scene image is obtained, wherein the ID map labels the ID of the facet corresponding to each pixel in the scene image; Based on the target image, the ID map of the target image, and the 3D mesh model, a set of facet IDs of the target object is obtained, specifically including: Generate a mask for the target image based on the annotations in the target image; Based on the mask of the target image, the ID of the target object in the ID map of each target image is estimated to obtain the ID prediction set of the target image; Based on the connectivity of the patches, the patches corresponding to the ID prediction set are divided into different connected regions to obtain the ID prediction subset of the connected regions; Based on the ID prediction subset, determine the set of facet IDs of the target object, specifically including: determining the target ID prediction subset of each target image from the ID prediction subset; Based on all the predicted subsets of target IDs, a set of face IDs for the target object is determined; wherein, the ID image is an image in which each pixel records the ID of a face in the corresponding 3D mesh model, that is, rendering the face in the 3D mesh model onto the pixel of the image. Different face IDs correspond to different grayscale values. Rendering is finding the correspondence between a 2D image and a 3D mesh model with a specified pose and camera parameters. This relationship is the information of the 3D model at the intersection of a ray emanating from the optical center of the camera, passing through the pixel object position, and the 3D mesh model; wherein, the target image is a scene image labeled with the target object; Based on the ID map and the set of patch IDs, the prompt box of the target object in the scene image and the mask within the prompt box are obtained.

2. The method according to claim 1, characterized in that, Determining the target ID prediction subset for each target image from the ID prediction subset includes: Determine the number of patch IDs for each of the connected regions in the target image; The target ID prediction subset is determined from the ID prediction subset based on the maximum value among the number of patch IDs.

3. The method according to claim 1, characterized in that, Determining the target ID prediction subset for each target image from the ID prediction subset includes: Based on the distance from the vertex of the patch in the target image to the camera device, determine the average distance from the vertex of each connected region patch to the camera device, wherein the camera device is the device that acquires the target image; Based on the minimum value of the average distance, a target connected region is determined from the connected regions, and the target ID prediction subset is obtained based on the target connected region.

4. The method according to claim 1, characterized in that, The step of obtaining the ID map of the scene image based on the IDs of each facet in the 3D mesh model includes: Obtain the pose of the scene image from the scene image information; Based on the pose and the IDs of each facet in the 3D mesh model, the facet IDs corresponding to each pixel in the scene image are obtained, and the ID map is generated.

5. The method according to claim 1, characterized in that, The step of obtaining the tooltip of the target object in the scene image and the mask within the tooltip based on the ID map and the set of patch IDs includes: Based on the set of facet IDs, determine the ID element representing the target object in the ID diagram; Obtain the maximum x-coordinate, maximum y-coordinate, minimum x-coordinate, and minimum y-coordinate of the ID element; Based on the maximum x-coordinate, the maximum y-coordinate, the minimum x-coordinate, and the minimum y-coordinate, a prompt box for the target object is generated in the scene image; Based on the ID map and the set of patch IDs, a mask for the scene image within the prompt box is obtained.

6. A mask generation system for two-dimensional images, characterized in that, The system includes: an information acquisition module, an ID graph generation module, an ID set acquisition module, and a mask generation module, wherein, The information acquisition module is used to acquire scene image information and a three-dimensional mesh model of the target object; The ID map generation module is used to obtain an ID map of the scene image based on the scene image information and the IDs of each facet in the three-dimensional mesh model, wherein the ID map labels the ID of the facet corresponding to each pixel in the scene image; The ID set acquisition module is used to obtain the set of facet IDs of the target object based on the target image, the ID map of the target image, and the 3D mesh model, specifically including: Generate a mask for the target image based on the annotations in the target image; Based on the mask of the target image, the ID of the target object in the ID map of each target image is estimated to obtain the ID prediction set of the target image; Based on the connectivity of the patches, the patches corresponding to the ID prediction set are divided into different connected regions to obtain the ID prediction subset of the connected regions; Based on the ID prediction subset, determine the set of facet IDs of the target object, specifically including: determining the target ID prediction subset of each target image from the ID prediction subset; Based on all the predicted subsets of target IDs, a set of face IDs for the target object is determined; wherein, the ID image is an image in which each pixel records the ID of a face in the corresponding 3D mesh model, that is, rendering the face in the 3D mesh model onto the pixel of the image. Different face IDs correspond to different grayscale values. Rendering is finding the correspondence between a 2D image and a 3D mesh model with a specified pose and camera parameters. This relationship is the information of the 3D model at the intersection of a ray emanating from the optical center of the camera, passing through the pixel object position, and the 3D mesh model; wherein, the target image is a scene image labeled with the target object; The mask generation module is used to obtain the prompt box of the target object in the scene image and the mask within the prompt box based on the ID map and the set of patch IDs.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the mask generation method for a two-dimensional image as described in any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the mask generation method for a two-dimensional image as described in any one of claims 1 to 5.

Images