Semantic segmentation method, electronic equipment and storage medium

Abstract

The invention discloses a semantic segmentation method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first image of a target scene at a plurality of different visual angles, obtaining a first image set, and obtaining a sparse point cloud of the target scene; according to each first image in the first image set, determining a second image corresponding to the first image to obtain a second image set; determining the three-dimensional points of the same instance tag in the sparse point cloud as an instance semantic tag to obtain a semantic sparse point cloud; wherein the instance label of the three-dimensional point is an instance label of a pixel point corresponding to the three-dimensional point in the second image set; according to instance point sets belonging to the same instance semantic tag in the semantic sparse point cloud, segmenting a segmented image set corresponding to the instance semantic tag from the first image set; and outputting a segmented image set corresponding to the instance semantic tag.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a semantic segmentation method, an electronic device, and a storage medium. Background Technology

[0002] In recent years, advancements in semantic segmentation technology have enabled the accurate identification and differentiation of different categories of ground features in images, providing new insights for the segmentation of individual buildings. However, existing methods primarily focus on semantic segmentation of single-viewpoint images, neglecting the collaborative information inherent in multi-viewpoint images and failing to effectively address the cross-image consistency problem of buildings from different perspectives. While re-identification algorithms can attempt to solve some instance matching problems, the variations in building appearance and the similarity of their local features result in poor re-identification performance. Furthermore, existing methods often require substantial computational resources for complex segmentation processing, lacking specificity and efficiency. Summary of the Invention

[0003] One objective of this invention is to provide a new technical solution for semantic segmentation that can solve the problems of low accuracy and low efficiency in building segmentation in related technologies.

[0004] According to a first aspect of the present invention, a semantic segmentation method is provided, comprising:

[0005] First images of the target scene from multiple different perspectives are obtained to form a first image set, and sparse point cloud of the target scene is obtained.

[0006] Based on each first image in the first image set, a second image corresponding to the first image is determined to obtain a second image set; wherein, the second image is an image obtained by annotating the first image with target instances, and different target instances in the second image have different instance labels;

[0007] Three-dimensional points with the same instance label in the sparse point cloud are identified as an instance semantic label to obtain a semantic sparse point cloud; wherein, the instance label of the three-dimensional point is the instance label of the pixel point corresponding to the three-dimensional point in the second image set;

[0008] Based on the set of instance points belonging to the same instance semantic label in the semantic sparse point cloud, the segmented image set corresponding to the instance semantic label is segmented from the first image set.

[0009] Output the segmented image set corresponding to the semantic labels of the instance.

[0010] Optionally, determining the second image corresponding to each first image in the first image set includes:

[0011] Based on the first image, determine the semantic segmentation map of the first image;

[0012] Based on the semantic label of each pixel in the semantic segmentation map, the pixel type of each pixel in the first image is determined; wherein, the pixel type includes target pixels and non-target pixels, and the target pixels are the pixels corresponding to the target instance;

[0013] The target pixels belonging to the same connected region in the first image are labeled as an instance label to obtain the second image.

[0014] Optionally, labeling target pixels belonging to the same connected region in the first image as an instance label includes:

[0015] The pixel values ​​of the target pixels in the first image are updated to the first pixel value, and the pixel values ​​of the non-target pixels in the first image are updated to the second pixel value to obtain the third image;

[0016] Based on the position coordinates of the target pixel in the third image, at least one connected region of the third image is determined;

[0017] Mark the target pixel in any connected region of the at least one connected region as an instance label.

[0018] Optionally, determining at least one connected region of the third image based on the position coordinates of the target pixel in the third image includes:

[0019] During the sliding process of the convolution kernel in the third image, if the pixel values ​​at the non-center positions of the convolution kernel are all the first pixel values, the pixel values ​​at the center positions of the convolution kernel are updated to the first pixel values ​​to obtain the processed third image.

[0020] Based on the position coordinates of the target pixels in the processed third image, at least one connected region of the processed third image is determined.

[0021] Optionally, determining 3D points with the same instance label in the sparse point cloud as an instance semantic label includes:

[0022] If the number of instance labels in the labels of the pixels corresponding to the three-dimensional point in the second image set is greater than or equal to a number threshold, the instance semantic label corresponding to the three-dimensional point is determined based on the instance labels of the three-dimensional point.

[0023] Optionally, the step of segmenting the segmented image set corresponding to the instance semantic label from the first image set based on the set of instance points belonging to the same instance semantic label in the semantic sparse point cloud includes:

[0024] For any of the multiple different viewpoints, based on the projection area of ​​the instance point set under the viewpoint, an image corresponding to the projection area is cropped from the first image corresponding to the viewpoint and used as the segmentation image of the instance semantic label under the viewpoint.

[0025] Based on the instance semantic label in multiple segmented images corresponding to multiple different viewpoints, a segmented image set corresponding to the instance semantic label is obtained.

[0026] Optionally, obtaining the sparse point cloud of the target scene includes:

[0027] Based on the first image set, the sparse point cloud of the target scene is determined.

[0028] Optionally, determining the semantic segmentation map of the first image based on the first image includes:

[0029] The first image is input into a lightweight semantic segmentation model to obtain a semantic segmentation map of the first image.

[0030] According to a second aspect of the present invention, an electronic device is also provided, comprising a memory and a processor, the memory being configured to store executable instructions; the processor being configured to operate under the control of the instructions to perform the method as described in the first aspect of the present invention.

[0031] According to a third aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed, performs the steps of the method described in the first aspect.

[0032] One beneficial effect of this invention is that by annotating the first images in a first image set with target instances, a second image set is obtained. Then, the instance labels in the second image set are mapped to a sparse point cloud to obtain a semantic sparse point cloud. Finally, based on the semantic sparse point cloud, semantic segmentation is performed on the first image set to obtain a segmented image set corresponding to each instance's semantic label. In this way, accurate semantic segmentation of different instance semantic labels in the first image set can be achieved by combining first images from multiple different perspectives. Furthermore, the segmentation method does not involve complex algorithms, resulting in high semantic segmentation efficiency.

[0033] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0035] Figure 1 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention;

[0036] Figure 2 This is a flowchart illustrating a semantic segmentation method according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0038] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0039] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0040] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0041] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0042] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0043] <Hardware Configuration>

[0044] Figure 1 This is a block diagram of the hardware configuration of an electronic device 1000 according to an embodiment of the present invention.

[0045] Electronic device 1000 can be a terminal device, such as a laptop or PC, or it can be a server; there is no limitation here.

[0046] Electronic device 1000 may include processor 1100, memory 1200, interface device 1300, communication device 1400, display device 1500, input device 1600, speaker 1700, microphone 1800, etc.

[0047] Processor 1100 may be a mobile processor. Memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. Interface device 1300 includes, for example, a USB interface and a headphone jack. Communication device 1400 is capable of wired or wireless communication. Communication device 1400 may include short-range communication devices, such as any device that performs short-range wireless communication based on short-range wireless communication protocols such as Hilink, WiFi (IEEE 802.11), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, and LiFi. Communication device 1400 may also include long-range communication devices, such as any device that performs WLAN, GPRS, or 2G / 3G / 4G / 5G long-range communication. Display device 1500 is, for example, an LCD screen or a touch screen. Input device 1600 may include, for example, a touch screen or a keyboard. Users can input / output voice information through speaker 1700 and microphone 1800.

[0048] In this embodiment, the memory 1200 of the electronic device 1000 is used to store instructions for controlling the processor 1100 to operate in order to at least execute the semantic segmentation method performed by the electronic device 1000 according to any embodiment of the present invention. Those skilled in the art can design the instructions according to the disclosed scheme of the present invention. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0049] Despite Figure 1 The present invention illustrates multiple devices of electronic device 1000, but may refer to only some of these devices; for example, electronic device 1000 may refer only to memory 1200 and processor 1100.

[0050] <Method Implementation>

[0051] Figure 2 This is a flowchart illustrating a semantic segmentation method according to an embodiment of the present invention, which can be implemented by an electronic device 1000.

[0052] according to Figure 2 As shown, the semantic segmentation method of this embodiment may include the following steps S2100 to S2500:

[0053] Step S2100: Obtain first images of the target scene from multiple different perspectives to obtain a first image set, and obtain the sparse point cloud of the target scene.

[0054] In this embodiment, the target scene can be an urban scene, a tourist town scene, a campus scene, a commercial area scene, an industrial park scene, etc., and there are no restrictions here.

[0055] The viewpoint can be an aerial viewpoint captured by a camera. Correspondingly, the first image can also be called an aerial image.

[0056] Sparse point clouds can be a 3D model of the target scene.

[0057] Sparse point clouds can be a pre-established 3D model of the target scene, or a 3D model of the target scene established on the spot based on the first images from multiple different perspectives; there is no limitation here.

[0058] In one embodiment of this application, obtaining the sparse point cloud of the target scene in step S2100 includes:

[0059] Based on the first image set, the sparse point cloud of the target scene is determined.

[0060] In this embodiment, the sparse point cloud of the target scene can be determined by structured light illumination (SfM), multi-view stereo vision (MVS), stereo matching, deep learning methods, etc. The specific construction method of the sparse point cloud of the target scene is not limited here.

[0061] Step S2200: Based on each first image in the first image set, determine the second image corresponding to the first image to obtain the second image set.

[0062] The second image is obtained by annotating the first image with target instances, and different target instances in the second image have different instance labels.

[0063] In this embodiment, the target instance can be a target entity object in the target scene.

[0064] In one example, the target scene can be an urban scene, and the target instance can be a building; that is, the second image is the image obtained by annotating the buildings in the first image.

[0065] For example, if the first image contains three buildings, then all pixels corresponding to the first building in the first image are labeled A1, all pixels corresponding to the second building in the first image are labeled A2, and all pixels corresponding to the third building in the first image are labeled A3, resulting in a second image A with instance labels A1, A2, and A3. If the first image contains two buildings, then all pixels corresponding to the first building in the first image are labeled B1, and all pixels corresponding to the second building in the second image are labeled B2, resulting in a second image B with instance labels B1 and B2.

[0066] It should be noted that when annotating the target instance, all pixels corresponding to the target instance in the first image need to be annotated.

[0067] In one embodiment of this application, step S2200, which determines the second image corresponding to each first image in the first image set, includes steps S2200.1 to S2200.3.

[0068] Step S2200.1: Determine the semantic segmentation map of the first image based on the first image.

[0069] In this embodiment, the semantic segmentation map of the first image includes the real semantic label of each pixel in the first image.

[0070] In the example where the target scene is an urban scene, the semantics included in the semantic segmentation map of the first image can be vehicles, grass, buildings, pedestrians, etc.

[0071] In one example, a first image can be input into a semantic segmentation model to obtain a semantic segmentation map of the first image.

[0072] In one embodiment of this application, in order to save resource consumption and improve the speed of semantic segmentation, step S2200.1, which determines the semantic segmentation map of the first image based on the first image, includes: inputting the first image into a lightweight semantic segmentation model to obtain the semantic segmentation map of the first image.

[0073] For example, a lightweight D-LinkNet network is used to perform semantic segmentation on each input first image to obtain a semantic segmentation map of the first image.

[0074] Step S2200.2: Determine the pixel type of each pixel in the first image based on the semantic label of each pixel in the semantic segmentation map.

[0075] The pixel types include target pixels and non-target pixels. Target pixels are pixels corresponding to target instances, and non-target pixels are pixels corresponding to non-target instances.

[0076] In this embodiment, since the pixels of the semantic segmentation map and the first image are in one-to-one correspondence, the pixel type of the pixel in the first image corresponding to any pixel in the semantic segmentation map can be determined based on the semantic label of any pixel in the semantic segmentation map.

[0077] The semantic labels in the semantic segmentation map can be buildings, grass, pedestrians, vehicles, etc.

[0078] In the example where the target instance is a building, the pixels corresponding to the semantic tags of buildings are determined as target pixels, while the pixels corresponding to semantic tags such as grass, pedestrians, and vehicles are determined as non-target pixels.

[0079] Step S2200.3: Mark the target pixels in the first image that belong to the same connected region as an instance label to obtain the second image.

[0080] In this embodiment, the target pixel can also be called the foreground pixel, so the target pixel is labeled as an instance label.

[0081] In the first image, each target instance corresponds to a connected region. This connected region can be a region consisting of neighboring target pixels in the first image. Furthermore, each connected region is labeled as an instance label.

[0082] In the example where the target instance is a building, if there are three connected regions in the first image, and these three connected regions are labeled with different instance labels, these three connected regions can be labeled with instance labels A1, A2, and A3 respectively to obtain the second image A.

[0083] In one example, the process of labeling target pixels belonging to the same connected region in the first image as an instance label is as follows: Traverse the pixels in the first image. When an unlabeled target pixel is encountered, initiate a depth-first search (DFS) or breadth-first search (BFS) to label the target pixel and all connected target pixels in its neighborhood (i.e., connected region) as having the same instance label. Continue traversing the first image, repeating this process for new unlabeled target pixels, until all target pixels are labeled.

[0084] In one embodiment of this application, step S2200.3 marks the target pixels in the first image that belong to the same connected region as an instance label, including steps S2200.31 to S2200.33.

[0085] Step S2200.31: Update the pixel value of the target pixel in the first image to the first pixel value, and update the pixel value of the non-target pixel in the first image to the second pixel value, to obtain the third image.

[0086] In this embodiment, to facilitate the determination of connected regions, the first image can be binarized to obtain a third image, and then the connected regions can be determined using the third image. The binarization process can involve updating the pixel values ​​of target pixels in the first image to first pixel values, and updating the pixel values ​​of non-target pixels in the first image to second pixel values. The first pixel value is greater than the second pixel value.

[0087] For example, the first pixel value can be the pixel value 255 corresponding to white, and the second pixel value can be the pixel value 0 corresponding to black.

[0088] Step S2200.32: Determine at least one connected region of the third image based on the position coordinates of the target pixel in the third image.

[0089] In this embodiment, target pixels adjacent to the target pixels in the third image can be identified as a connected region based on the position coordinates of the target pixels.

[0090] To improve the accuracy of instance labeling, morphological erosion can be performed on the third image before instance labeling to ensure that relatively separated target instances are not incorrectly identified as a connected region.

[0091] Based on this, in one embodiment of this application, step S2200.32, which determines at least one connected region of the third image according to the position coordinates of the target pixel in the third image, includes steps SA1 to SA2.

[0092] Step SA1: During the sliding process of the convolution kernel in the third image, if the pixel values ​​at the non-center positions of the convolution kernel are all the first pixel values, update the pixel values ​​at the center positions of the convolution kernel to the first pixel values ​​to obtain the processed third image.

[0093] For example, in a 3x3 convolution kernel, if a second pixel value exists at a non-center position (i.e., any position other than the center of the kernel), and the pixel value at the center of the kernel is the first pixel value, then the pixel value at the center of the kernel is updated to the second pixel value; otherwise, the second pixel value remains unchanged. Similarly, if all non-center pixels in the current convolution kernel are the first pixel value, and the pixel value at the center of the kernel is the second pixel value, then the pixel value at the center of the kernel is updated to the first pixel value; otherwise, the first pixel value remains unchanged.

[0094] In this way, pixels at the edges of target instances in the third image can be removed, thereby reducing the connected regions and expanding the background regions, which helps to separate target instances that may be connected together due to noise or segmentation errors.

[0095] Step SA2: Determine at least one connected region of the processed third image based on the position coordinates of the target pixel in the processed third image.

[0096] The method for determining at least one connected region in this step is basically the same as that in step S2200.32 above, and will not be elaborated here.

[0097] Step S2200.33: Mark the target pixel in any connected region of the at least one connected region as an instance label.

[0098] In this embodiment, different instance labels correspond to different connected regions.

[0099] Step S2300: Determine the 3D points with the same instance label in the sparse point cloud as an instance semantic label to obtain a semantic sparse point cloud.

[0100] The instance label of the three-dimensional point is the instance label of the pixel point corresponding to the three-dimensional point in the second image set.

[0101] In this embodiment, a 3D point may be invisible from certain viewpoints. Correspondingly, there will be no corresponding pixel for that 3D point in the second image from that viewpoint. Therefore, when determining the instance label of a 3D point, it is necessary to first determine the pixel corresponding to each 3D point in the second image set. For ease of description, the pixel corresponding to a 3D point in the second image set is referred to as the common-view information of that 3D point.

[0102] In one example, common-view information for each 3D point in a sparse point cloud can be generated using SfM (Structure from Motion).

[0103] Then, traverse each 3D point in the sparse point cloud. If there is a pixel corresponding to the instance label in the common view information of the 3D point, use the instance label as the instance label of the 3D point.

[0104] For example, a certain 3D point has corresponding pixels in second images A, B, and C. The pixel in second image A is labeled as instance label A2, the pixel in second image B is labeled as instance label B3, but the pixel in second image C does not have an instance label. In this case, the instance labels of the 3D point are A2 and B3.

[0105] Finally, 3D points with the same instance label in the sparse point cloud are identified as an instance semantic label.

[0106] Continuing the example above, if the instance labels for a certain 3D point are A2 and B3, and there are 50 3D points in the sparse point cloud with instance labels A2 and B3, then these 50 3D points in the sparse point cloud will be labeled as one instance semantic label 'a'. If there are 30 3D points in the sparse point cloud with instance labels C3, D4, and E1, then these 30 3D points will be labeled as one instance semantic label 'b'.

[0107] In one embodiment of this application, step S2300, which determines 3D points with the same instance label in the sparse point cloud as an instance semantic label, includes:

[0108] If the number of instance labels in the labels of the pixels corresponding to the three-dimensional point in the second image set is greater than or equal to a number threshold, the instance semantic label corresponding to the three-dimensional point is determined based on the instance labels of the three-dimensional point.

[0109] In this embodiment, the quantity threshold can be flexibly set according to the accuracy requirements of instance annotation, and no limitation is made here.

[0110] In one example, the quantity threshold can be 3, and 3D points with instance labels C3, D4, and E1 are labeled with instance semantic label b. If a 3D point has four corresponding pixels in the second image set, including either instance labels D4 and E1 or C3 and D4, then no instance semantic label is applied to that 3D point. If a 3D point has three corresponding pixels in the second image set, including instance labels C3, D4, and E1, then that 3D point is labeled with instance semantic label b.

[0111] Step S2400: Based on the set of instance points belonging to the same instance semantic label in the semantic sparse point cloud, segment the set of segmented images corresponding to the instance semantic label from the first image set.

[0112] For example, a semantically sparse point cloud contains 30 3D points corresponding to instance semantic label b and 50 3D points corresponding to instance semantic label a. In this case, the set of instance points corresponding to instance semantic label b is reprojected onto a first image set from multiple different viewpoints to segment the image set corresponding to instance semantic label b. Similarly, the set of instance points corresponding to instance semantic label a is reprojected onto the first image set from multiple different viewpoints to segment the image set corresponding to instance semantic label a. This achieves fast semantic segmentation of the first image set composed of multiple different viewpoints with high semantic segmentation accuracy.

[0113] In one embodiment of this application, step S2400 involves segmenting the segmented image set corresponding to the instance semantic label from the first image set based on the instance point set belonging to the same instance semantic label in the semantic sparse point cloud, including steps S2400.1 to S2400.2.

[0114] Step S2400.1: For any of the multiple different viewpoints, based on the projection area of ​​the instance point set under the viewpoint, crop the image corresponding to the projection area from the first image corresponding to the viewpoint, and use it as the segmentation image of the instance semantic label under the viewpoint.

[0115] In this embodiment, the projection area can be the smallest bounding box of the instance point set from any viewpoint.

[0116] In one example, if the projection area exceeds the image range of the first image, the projection area can be supplemented by interpolation or other methods.

[0117] Step S2400.2: Based on the instance semantic label in the multiple segmented images corresponding to the multiple different viewpoints, obtain the segmented image set corresponding to the instance semantic label.

[0118] In this embodiment, an instance semantic label may have segmented images from multiple perspectives, and these multiple segmented images constitute the segmented image set corresponding to the instance semantic label.

[0119] Step S2500: Output the segmented image set corresponding to the semantic label of the instance.

[0120] By annotating target instances in the first images of the first image set, a second image set is obtained. Then, the instance labels in the second image set are mapped to a sparse point cloud to obtain a semantic sparse point cloud. Based on this semantic sparse point cloud, semantic segmentation is performed on the first image set to obtain segmented image sets corresponding to the semantic labels of each instance. In this way, accurate semantic segmentation of different instance semantic labels in the first image set can be achieved by combining first images from multiple different perspectives. Furthermore, the segmentation method does not involve complex algorithms, resulting in high semantic segmentation efficiency.

[0121] The semantic segmentation method of this application is illustrated below, taking the target scene as the target city scene and the target instance as a building:

[0122] Step S1: Obtain the first images of the target scene from multiple different perspectives to obtain the first image set.

[0123] In this example, a drone is used to acquire multi-view images of the target city from different angles.

[0124] Step S2: Construct a sparse point cloud of the target city scene based on the first image set of the target city scene.

[0125] Sparse point clouds of the target city scene are generated using SfM technology. The sparse point cloud contains feature points with the same name from multiple viewpoints, providing a basis for global registration of building instances.

[0126] Step S3: For each first image, determine the semantic segmentation map of the first image based on the first image.

[0127] In this example, the D-LinkNet network is applied to each first image for semantic segmentation to extract semantic labels for buildings, roads, and trees.

[0128] Step S4: Determine the pixel type of each pixel in the first image based on the semantic label of each pixel in the semantic segmentation map.

[0129] The pixel types include target pixels and non-target pixels. Target pixels are those corresponding to buildings. Non-target pixels are those not corresponding to buildings.

[0130] Step S5: Update the pixel value of the target pixel in the first image to the first pixel value, and update the pixel value of the non-target pixel in the first image to the second pixel value, to obtain the third image.

[0131] The first pixel value can be, for example, 255, and the second pixel value can be 0.

[0132] Step S6: Label any target pixel in the same connected region in any third image as an instance label to obtain a second image, thereby obtaining the second image set corresponding to the first image set.

[0133] In this example, connected component analysis can be used to extract the individual building instance labels from the third image and separate adjacent buildings in the third image.

[0134] Step S7: Based on the instance label of each 3D point in the sparse point cloud, determine the 3D points with the same instance label in the sparse point cloud as an instance semantic label, and obtain the semantic sparse point cloud.

[0135] The instance label of a 3D point is the instance label of the pixel corresponding to the 3D point in the second image set.

[0136] In this example, common view information in sparse point clouds is used to match building instances from different perspectives, ensuring that all building instances are correctly registered globally and merging multiple viewpoint instances of the same building.

[0137] Step S8: Based on the set of instance points belonging to the same instance semantic label in the semantic sparse point cloud, segment the set of image points corresponding to the instance semantic label from the first image set.

[0138] In this example, for any viewpoint, based on the projection region of the instance point set of an instance semantic label under that viewpoint, an image corresponding to the projection region is cropped from the first image corresponding to that viewpoint, and this image is used as the segmentation image of the instance semantic label under that viewpoint. Based on multiple segmentation images corresponding to the instance semantic label under multiple different viewpoints, a segmentation image set corresponding to the instance semantic label is obtained.

[0139] <Storage Medium Examples>

[0140] This application provides a readable storage medium on which a program or instructions are stored. When the program or instructions are executed by a processor, they implement the steps of the method described in any of the above method embodiments.

[0141] <Electronic Device Examples>

[0142] Figure 3 This is a structural block diagram of an electronic device 300 according to an embodiment of the present invention.

[0143] In this embodiment, as Figure 3 As shown, the electronic device 300 includes a memory 310 and a processor 320. The memory 310 is used to store executable instructions, and the processor 320 is used to operate according to the control of the instructions to execute the methods described in any of the above embodiments.

[0144] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0145] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0146] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0147] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0148] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0149] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0150] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0152] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A semantic segmentation method, characterized in that, The method includes: First images of the target scene from multiple different perspectives are obtained to form a first image set, and sparse point cloud of the target scene is obtained. Based on each first image in the first image set, a second image corresponding to the first image is determined to obtain a second image set; wherein, the second image is an image obtained by annotating the first image with target instances, and different target instances in the second image have different instance labels; Three-dimensional points with the same instance label in the sparse point cloud are identified as an instance semantic label to obtain a semantic sparse point cloud; wherein, the instance label of the three-dimensional point is the instance label of the pixel point corresponding to the three-dimensional point in the second image set; Based on the set of instance points belonging to the same instance semantic label in the semantic sparse point cloud, the segmented image set corresponding to the instance semantic label is segmented from the first image set. Output the segmented image set corresponding to the semantic labels of the instance.

2. The method according to claim 1, characterized in that, The step of determining the second image corresponding to each first image in the first image set includes: Based on the first image, determine the semantic segmentation map of the first image; Based on the semantic label of each pixel in the semantic segmentation map, the pixel type of each pixel in the first image is determined; wherein, the pixel type includes target pixels and non-target pixels, and the target pixels are the pixels corresponding to the target instance; The target pixels belonging to the same connected region in the first image are labeled as an instance label to obtain the second image.

3. The method according to claim 2, characterized in that, The step of labeling target pixels belonging to the same connected region in the first image as an instance label includes: The pixel values ​​of the target pixels in the first image are updated to the first pixel value, and the pixel values ​​of the non-target pixels in the first image are updated to the second pixel value to obtain the third image; Based on the position coordinates of the target pixel in the third image, at least one connected region of the third image is determined; Mark the target pixel in any connected region of the at least one connected region as an instance label.

4. The method according to claim 3, characterized in that, Determining at least one connected region of the third image based on the position coordinates of the target pixel in the third image includes: During the sliding process of the convolution kernel in the third image, if the pixel values ​​at the non-center positions of the convolution kernel are all the first pixel values, the pixel values ​​at the center positions of the convolution kernel are updated to the first pixel values ​​to obtain the processed third image. Based on the position coordinates of the target pixels in the processed third image, at least one connected region of the processed third image is determined.

5. The method according to claim 1, characterized in that, The step of determining 3D points with the same instance label in the sparse point cloud as an instance semantic label includes: If the number of instance labels in the labels of the pixels corresponding to the three-dimensional point in the second image set is greater than or equal to a number threshold, the instance semantic label corresponding to the three-dimensional point is determined based on the instance labels of the three-dimensional point.

6. The method according to claim 1, characterized in that, The step of segmenting the segmented image set corresponding to the instance semantic label from the first image set based on the instance point set belonging to the same instance semantic label in the semantic sparse point cloud includes: For any of the multiple different viewpoints, based on the projection area of ​​the instance point set under the viewpoint, an image corresponding to the projection area is cropped from the first image corresponding to the viewpoint and used as the segmentation image of the instance semantic label under the viewpoint. Based on the instance semantic label in multiple segmented images corresponding to multiple different viewpoints, a segmented image set corresponding to the instance semantic label is obtained.

7. The method according to claim 1, characterized in that, The acquisition of the sparse point cloud of the target scene includes: Based on the first image set, the sparse point cloud of the target scene is determined.

8. The method according to claim 2, characterized in that, Determining the semantic segmentation map of the first image based on the first image includes: The first image is input into a lightweight semantic segmentation model to obtain a semantic segmentation map of the first image.

9. An electronic device comprising a memory and a processor, the memory for storing executable instructions; the processor for operating under the control of the instructions to perform the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium comprising a stored computer program, wherein, When the computer program is run, it performs the method steps of any one of claims 1 to 8.

Images