A semantic label fusion method and device, electronic equipment and storage medium

Abstract

The application provides a semantic label fusion method and device, electronic equipment and a storage medium, and aims at improving the problem that the accuracy of semantic labels obtained in semantic segmentation is low. The method comprises the following steps: obtaining point cloud data scanned for a target object and multiple target images shot from different perspectives; for each target image in the multiple target images, performing semantic segmentation on each two-dimensional pixel point in the target image to obtain the category label of the two-dimensional pixel point; performing semantic segmentation on each three-dimensional pixel point in the point cloud data to obtain the category label of the three-dimensional pixel point; and fusing the category label of the three-dimensional pixel point and the category label of the projected pixel point according to the projection distance of the projected pixel point of the three-dimensional pixel point projected onto the target image to obtain the fused semantic label.

Description

Technical Field

[0001] This application relates to the technical fields of semantic segmentation, image segmentation, and point cloud segmentation, and more specifically, to a semantic tag fusion method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, the common practice for image semantic segmentation is to use neural network models in deep learning methods to predict the semantic segmentation results of target images. However, in practice, it has been found that the difficulty in obtaining training data for neural network models makes it difficult to adequately train them. Therefore, the accuracy of semantic labels obtained when performing semantic segmentation on target images is relatively low. Summary of the Invention

[0003] The purpose of this application is to provide a semantic tag fusion method, apparatus, electronic device, and storage medium to improve the problem of low accuracy of semantic tags obtained during semantic segmentation.

[0004] This application provides a semantic label fusion method, comprising: acquiring point cloud data scanned from a target object and multiple target images taken from different perspectives; for each of the multiple target images, performing semantic segmentation on each two-dimensional pixel in the target image to obtain a category label for the two-dimensional pixel; performing semantic segmentation on each three-dimensional pixel in the point cloud data to obtain a category label for the three-dimensional pixel; and fusing the category label of the three-dimensional pixel and the category label of the projected pixel on the target image according to the projection distance of the three-dimensional pixel onto the projected pixel to obtain a fused semantic label. In the implementation of the above scheme, by acquiring point cloud data scanned from a target object and multiple target images taken from different perspectives, and fusing the category labels of the three-dimensional pixels and the category labels of the two-dimensional pixels in the target images, the fused semantic label fully utilizes the category label information from image semantic segmentation and point cloud semantic segmentation, avoiding the situation where the semantic segmentation results of multiple target images are predicted solely through a neural network model, thereby improving the accuracy of the semantic labels obtained during semantic segmentation.

[0005] Optionally, in this embodiment, semantic segmentation is performed on each 3D pixel in the point cloud data to obtain the category label of the 3D pixel. This includes: extracting features from each 3D pixel in the point cloud data to obtain an initial feature vector for the 3D pixel. The point cloud data includes multiple volume elements, each containing multiple 3D pixels. For each volume element, the initial feature vectors of all 3D pixels within that volume element are averaged to obtain an average feature vector for that volume element. For each 3D pixel within a volume element, the category label of the 3D pixel is predicted based on the average feature vector of that volume element. In the implementation of the above scheme, by first calculating the average feature vector of the volume element and then determining the category label of each 3D pixel in that volume element based on the average feature vector, the problem of inaccurate calculation of the category label of a single 3D pixel leading to a decrease in the accuracy of semantic segmentation is avoided, thus improving the accuracy of the semantic labels obtained during semantic segmentation.

[0006] Optionally, in this embodiment, feature extraction for each three-dimensional pixel in the point cloud data includes: using a pre-trained neural network model to extract features for each three-dimensional pixel in the point cloud data. In the implementation of the above scheme, by using a pre-trained neural network model to extract features for each three-dimensional pixel in the point cloud data, the need for manual extraction of three-dimensional pixel features is avoided, thus improving the accuracy of semantic labels obtained during semantic segmentation.

[0007] Optionally, in this embodiment, the category labels of the 3D pixel and the projected pixel are fused based on the projection distance between the 3D pixel and the projected pixel on the target image. This includes: calculating the fusion weight of the category labels of the projected pixel on the target image based on the projection distance; and fusing the category labels of the 3D pixel and the projected pixel based on the fusion weight. In the implementation of the above scheme, by calculating the fusion weight of the category labels of the projected pixel on the target image based on the projection distance, and fusing the category labels of the 3D pixel and the projected pixel based on the fusion weight, the situation of predicting the semantic segmentation results of multiple target images solely through a neural network model is avoided, thereby improving the accuracy of the semantic labels obtained during semantic segmentation.

[0008] Optionally, in this embodiment, before calculating the fusion weight of the category labels of the projected pixels on the target image based on the projection distance, the method further includes: obtaining camera parameters of the target image; projecting each three-dimensional pixel in the point cloud data onto the projected pixels of the target image based on the camera parameters, and calculating the projection distance between the three-dimensional pixel and the projected pixel. In the implementation of the above scheme, by projecting each three-dimensional pixel in the point cloud data onto the projected pixels of the target image based on the camera parameters and calculating the projection distance between the three-dimensional pixel and the projected pixel, the problem of being unable to calculate the projection distance is avoided, effectively improving the accuracy of the projection distance between the three-dimensional pixel and the projected pixel.

[0009] Optionally, in this embodiment, calculating the projection distance between the 3D pixel and the projected pixel includes: if multiple projection distances are calculated between the same projected pixel and different 3D pixels, then the shortest projection distance is selected from the multiple projection distances. In the implementation of the above scheme, by selecting the shortest projection distance from the multiple projection distances when multiple projection distances between the same projected pixel and different 3D pixels are calculated, the problem of the longest projection distance affecting the semantic segmentation accuracy is avoided, thereby improving the accuracy of the semantic tags obtained during semantic segmentation.

[0010] Optionally, in this embodiment, after obtaining the fused semantic tags, the method further includes: using the fused semantic tags to extract the region image of the target object from at least one of the multiple target images, and / or using the fused semantic tags to extract the three-dimensional pixels of the target object from the point cloud data. In the implementation of the above scheme, by using the fused semantic tags to extract the target object from the target image and / or point cloud data, semantic understanding of the image and / or point cloud data is achieved, avoiding the problem of difficulty in extracting target objects of the same category from the image and / or point cloud data, and effectively realizing the function of extracting target objects of the same category from the image and / or point cloud data.

[0011] This application also provides a semantic label fusion device, including: a point cloud image acquisition module, used to acquire point cloud data scanned for a target object and multiple target images taken from different perspectives; an image semantic segmentation module, used to perform semantic segmentation on each two-dimensional pixel in each of the multiple target images to obtain a category label for the two-dimensional pixel; a point cloud semantic segmentation module, used to perform semantic segmentation on each three-dimensional pixel in the point cloud data to obtain a category label for the three-dimensional pixel; and a semantic label acquisition module, used to fuse the category label of the three-dimensional pixel and the category label of the projected pixel based on the projection distance of the three-dimensional pixel onto the projected pixel on the target image to obtain a fused semantic label.

[0012] Optionally, in this embodiment, the point cloud semantic segmentation module includes: a point cloud feature extraction submodule, used to extract features from each three-dimensional pixel in the point cloud data to obtain an initial feature vector of the three-dimensional pixel, wherein the point cloud data includes: multiple volume elements, and each volume element includes multiple three-dimensional pixels; a feature vector averaging submodule, used to average the initial feature vectors of all three-dimensional pixels within each of the multiple volume elements to obtain an average feature vector of the volume element; and a category label prediction submodule, used to predict the category label of each three-dimensional pixel within each volume element based on the average feature vector of the volume element.

[0013] Optionally, in this embodiment of the application, the point cloud feature extraction submodule includes: a pixel feature extraction unit, used to extract features from each three-dimensional pixel in the point cloud data using a pre-trained neural network model.

[0014] Optionally, in this embodiment, the semantic label acquisition module includes: a fusion weight calculation submodule, used to calculate the fusion weight of the category label of the projected pixel on the target image based on the projection distance; and a category label fusion submodule, used to fuse the category label of the three-dimensional pixel and the category label of the projected pixel based on the fusion weight.

[0015] Optionally, in this embodiment, the semantic tag acquisition module further includes: a camera parameter acquisition submodule, used to acquire camera parameters of the target image; and a projection distance calculation submodule, used to project each three-dimensional pixel in the point cloud data onto the projection pixel of the target image according to the camera parameters, and calculate the projection distance between the three-dimensional pixel and the projection pixel.

[0016] Optionally, in this embodiment of the application, the projection distance calculation submodule includes: a projection distance calculation unit, used to select the shortest projection distance from the multiple projection distances if multiple projection distances between the same projection pixel and different three-dimensional pixel points are calculated.

[0017] Optionally, in this embodiment of the application, the semantic tag fusion device further includes: a target object extraction module, used to extract the region image of the target object from at least one of the multiple target images using the fused semantic tags, and / or to extract the three-dimensional pixels of the target object from point cloud data using the fused semantic tags.

[0018] This application also provides an electronic device, including a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions, when executed by the processor, perform the method described above.

[0019] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to perform the methods described above.

[0020] Other features and advantages of embodiments of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of this application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 The flowchart shown is a schematic diagram of the semantic tag fusion method provided in the embodiments of this application;

[0023] Figure 2 The illustration shows a schematic diagram of semantic segmentation of a target image provided in an embodiment of this application;

[0024] Figure 3 The diagram shown is a structural schematic of the semantic tag fusion device provided in an embodiment of this application;

[0025] Figure 4 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of this application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0027] It is understood that the terms "first" and "second" in the embodiments of this application are used to distinguish similar objects. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0028] Before introducing the semantic tag fusion method provided in the embodiments of this application, let's first introduce some concepts involved in the embodiments of this application:

[0029] Semantic segmentation, literally, means enabling computers to segment images based on their semantics. In speech recognition, semantics refers to the meaning of speech, while in the image domain, semantics refers to the content of an image and the understanding of its meaning. Segmentation means separating different objects in an image from the perspective of pixels, classifying and labeling each pixel in the original image.

[0030] It should be noted that the semantic tag fusion method provided in this application embodiment can be executed by an electronic device. Here, an electronic device refers to a device terminal or server with the function of executing computer programs. Device terminals include, for example, smartphones, personal computers, tablets, personal digital assistants, or mobile internet devices. A server refers to a device that provides computing services through a network. Servers include, for example, x86 servers and non-x86 servers. Non-x86 servers include, for example, mainframes, minicomputers, and UNIX servers.

[0031] The following describes the applicable scenarios for this semantic label fusion method. These scenarios include, but are not limited to, fields such as computer vision and 3D semantic understanding. For example, when acquiring images and point cloud data of a target object (such as a target building in the construction industry or an aircraft in the aviation industry) from different perspectives, this semantic label fusion method can be used to perform semantic segmentation on the 2D pixels in the image and the 3D pixels in the point cloud data, and then fuse the category labels obtained from the semantic segmentation of the 2D pixels and the 3D pixels to obtain the fused semantic label. This fully utilizes the information from the category labels of both and effectively improves the accuracy of the fused semantic label.

[0032] Please see Figure 1 The illustration shows a flowchart of the semantic tag fusion method provided in this application embodiment; this application embodiment provides a semantic tag fusion method, including:

[0033] Step S110: Acquire point cloud data of the target object scan and multiple target images taken from different perspectives.

[0034] Point cloud data refers to the collection of three-dimensional pixels obtained by point cloud acquisition devices from a target object (such as a target building in the construction industry or an aircraft in the aviation field). Examples of point cloud acquisition devices include depth cameras, 3D coordinate measuring machines, lidar sensors, 3D laser scanners, or photographic scanners.

[0035] There are many ways to acquire point cloud data in step S110 above. These implementation methods include: First, other terminal devices collect point cloud data of the target object, then send the collected point cloud data to the electronic device, and finally the electronic device receives the point cloud data sent by the other terminal devices. These other terminal devices include: depth cameras, laser sensors, robots with depth cameras or laser sensors, robots controlled by robotic arms to collect point cloud data, etc.; Second, after receiving the point cloud data sent by other terminal devices, the electronic device stores the point cloud data in a file system or database, and retrieves the pre-stored point cloud data from the file system or database when the data is needed.

[0036] There are many ways to acquire multiple target images in step S110 above. These methods include: First, a camera matrix can be used to simultaneously or separately capture images of the target object from different perspectives, obtaining multiple target images. Here, a camera matrix, also known as a camera array, refers to a calibrated camera array consisting of multiple cameras fixed on a frame to maximize the acquisition of images or videos of the target object from different angles. Second, multiple image acquisition devices carried by drones can be used to simultaneously or separately capture images of the target object, obtaining multiple target images. These image acquisition devices include, for example, cameras, video recorders, or color cameras.

[0037] Step S120: For each of the multiple target images, perform semantic segmentation on each two-dimensional pixel in the target image to obtain the category label of the two-dimensional pixel.

[0038] Please see Figure 2The illustration shows a schematic diagram of semantic segmentation of a target image provided by an embodiment of this application. The number of category labels in the diagram can be set according to specific circumstances. For example, the number of category labels in the diagram can be set to three: car, tree, and background, or it can be set to four: car, tree, road, and background, etc. After semantic segmentation of the target image, each two-dimensional pixel in the target image is classified into the above three or four categories, which are called the category label of the two-dimensional pixel. An example implementation of step S120 is as follows: For each target image in multiple target images, a trained semantic segmentation network model is used to perform semantic segmentation on each two-dimensional pixel in the target image to obtain the semantic segmentation result of the two-dimensional pixel (i.e., the specific category to which the two-dimensional pixel belongs). The semantic segmentation result here is the category label of the two-dimensional pixel. The semantic segmentation network model here can be a DeepLab V1 model, a DeepLab V2 model, or a DeepLab V3 model, etc.

[0039] Step S130: Perform semantic segmentation on each 3D pixel in the point cloud data to obtain the category label of the 3D pixel.

[0040] Step S140: Based on the projection distance of the 3D pixel point onto the projected pixel point on the target image, fuse the category label of the 3D pixel point and the category label of the projected pixel point to obtain the fused semantic label.

[0041] In the above implementation process, point cloud data of the target object scanned and multiple target images taken from different perspectives are acquired, and the category labels of three-dimensional pixels and two-dimensional pixels in the target image are fused. Since the fused semantic label makes full use of the category label information of image semantic segmentation and point cloud semantic segmentation, it avoids the situation of predicting the semantic segmentation results of multiple target images by simply using a neural network model, thereby improving the accuracy of the semantic label obtained during semantic segmentation.

[0042] As an optional implementation of step S130 above, the implementation of semantic segmentation of point cloud data may include:

[0043] Step S131: Extract features from each three-dimensional pixel in the point cloud data to obtain the initial feature vector of the three-dimensional pixel. The point cloud data includes multiple volume elements, and each volume element includes multiple three-dimensional pixels.

[0044] A volume element, also known as a voxel, is a point cloud unit that divides point cloud data according to a preset volume space size. Here, the preset volume space can be set to a cube with a length, width, and height of 640 centimeters.

[0045] It is understood that there are many ways to implement step S131 above, therefore, the implementation of step S131 will be described in detail below. In the specific implementation process, in order to reduce the computational load of feature extraction of point cloud data, the point cloud data can be divided into multiple scene blocks. The size of the scene blocks can be set according to specific circumstances, for example, the size of the scene blocks can be set to a cube with a length, width and height of 100 meters. Then, each scene block in the multiple scene blocks is divided into multiple volume elements (also known as voxels), where each volume element includes multiple three-dimensional pixels. In the above implementation process, by dividing each scene block in the multiple scene blocks into multiple volume elements (also known as voxels), then calculating the average feature vector of each volume element, and finally determining the category label of each three-dimensional pixel in the volume element based on the average feature vector, the direct division of very large point cloud data into volume elements is avoided. This can effectively complete the category label calculation process of each three-dimensional pixel in each scene block under the condition of limited hardware computing resources, and complete the semantic segmentation process of point cloud data with the idea of ​​trading running time for storage space, thereby reducing the use and waste of storage space resources.

[0046] Step S132: For each volume element among multiple volume elements, average the initial feature vectors of all three-dimensional pixels within that volume element to obtain the average feature vector of that volume element.

[0047] For example, the implementation of step S132 above is as follows: For each volume element among multiple volume elements, calculate the average value of the initial feature vectors of all three-dimensional pixels within that volume element, thereby obtaining the average feature vector of that volume element.

[0048] Step S133: For each three-dimensional pixel within a volume element, predict the category label of the three-dimensional pixel based on the average feature vector of that volume element.

[0049] For example, the implementation of step S133 above can be as follows: the average feature vector of the volume element is predicted using a sparse convolutional neural network model to obtain the category label of the volume element; for each three-dimensional pixel in the volume element, the category label of the volume element can be determined as the category label of the three-dimensional pixel (that is, the category label of the volume element is back-projected onto each three-dimensional pixel in the volume element), thereby obtaining the category label of each three-dimensional pixel.

[0050] As a first optional implementation of step S131 above, the implementation of using a neural network model to extract features from three-dimensional pixels may include:

[0051] Step S131a: Use a pre-trained neural network model to extract features from each three-dimensional pixel in the point cloud data to obtain the initial feature vector of the three-dimensional pixel.

[0052] For example, the above step S131a can be implemented by using a neural network model to extract features from each three-dimensional pixel in the point cloud data. The existing neural network models obtained here include, but are not limited to: Feature Fusion Single Shot Multibox Detector (FSSD), LeNet network, AlexNet network, GoogLeNet network, VGG network, ResNet network, Wide ResNet network, and Inception network.

[0053] As a second optional implementation of step S131 above, the implementation of using machine learning algorithms to extract features from three-dimensional pixels may include:

[0054] Step S131b: Use a machine learning algorithm to extract features from each three-dimensional pixel in the point cloud data to obtain the initial feature vector of the three-dimensional pixel.

[0055] For example, the implementation of step S131b above involves using machine learning algorithms to extract features from each three-dimensional pixel in the point cloud data. These machine learning algorithms include, but are not limited to, decision trees, Bayesian learning, instance-based learning, genetic algorithms, rule learning, interpretation-based learning, and histogram of oriented gradients feature extraction algorithms.

[0056] As an optional implementation of step S140 above, the implementation of fusing the category labels of the three-dimensional pixel and the projected pixel according to the projection distance may include:

[0057] Step S141: Calculate the fusion weight of the category labels of the projected pixels on the target image based on the projection distance.

[0058] Understandably, when calculating the projection distance, each 3D pixel in the point cloud data needs to be projected onto the projection pixel of the target image based on camera parameters (e.g., external parameters). For a 3D pixel to be correctly projected onto the target image, preset conditions must be met to confirm that the 3D pixel has been correctly projected onto the target image. These preset conditions include: the 3D pixel is projected within the range of the target image; the 3D pixel is in front of the camera; and the 3D pixel has the shortest projection distance. This shortest projection distance can be obtained, for example, by calculating multiple projection distances between the same projection pixel and different 3D pixels, and then selecting the shortest projection distance from these multiple projection distances.

[0059] An example implementation of step S141 above is: using the formula The projection distance of a 3D pixel correctly projected onto the target image is calculated to obtain the fusion weight of the class label of the projected pixel on the target image; where w j Let represent the fusion weight of the class label of the projected pixel on the j-th target image, e represent the natural constant, σ represent the average of all projected distances, and p i I represents the i-th 3D pixel in the point cloud data. j Let d(p) represent the j-th target image. i ,I j ) represents the projection distance between the i-th 3D pixel in the point cloud data and the j-th target image.

[0060] Step S142: Merge the category labels of the 3D pixel and the projected pixel according to the fusion weight to obtain the fused semantic label.

[0061] The specific method for fusing semantic tags in step S142 above is, for example, by using the formula. The fusion weights, the category labels of the 3D pixels, and the category labels of the projected pixels are calculated to obtain the fused semantic label. Where p(l i =k) ​​represents a three-dimensional pixel p i The probability of belonging to category k, where λ represents the smoothing hyperparameter between the semantic segmentation result of the point cloud data (i.e., the category labels of 3D pixels) and the semantic segmentation result of the target image (i.e., the category labels of projected pixels). This smoothing hyperparameter can be set according to specific circumstances, for example, setting the smoothing hyperparameter to 0.5. This represents the semantic segmentation result of point cloud data (i.e., the category labels of 3D pixels). w represents the semantic segmentation result (i.e., the category label of the projected pixel) of the j-th target image. jThis represents the fusion weight of the category label of the projected pixel on the j-th target image.

[0062] As an optional implementation of step S140 above, before calculating based on the projection distance in step S141, the projection distance can be calculated first. The calculation method for the projection distance can include:

[0063] Step S143: Obtain the camera parameters of the target image.

[0064] Camera parameters refer to the relative parameters between the camera and the point cloud acquisition device. These camera parameters can include both external and internal camera parameters.

[0065] The methods for obtaining camera parameters in step S143 above include: a first method, receiving camera parameters of a target image sent by another terminal device and storing the camera parameters of the target image in a file system, database, or mobile storage device; a second method, obtaining camera parameters of a pre-stored target image, specifically, for example, obtaining camera parameters of the target image from a file system, database, or mobile storage device; and a third method, using software such as a browser to obtain camera parameters of a target image on the Internet, or using other applications to access the Internet to obtain camera parameters of the target image.

[0066] Step S144: Project each 3D pixel in the point cloud data onto the projection pixel of the target image according to the camera parameters, and calculate the projection distance between the 3D pixel and the projection pixel.

[0067] As an optional implementation of step S144 above, when calculating the projection distance, if multiple three-dimensional pixels are projected onto the same pixel, then only the shortest projection distance between the two points can be taken. This implementation may include:

[0068] Step S144a: Determine whether the projection distances between the same projection pixel and different 3D pixels have been calculated.

[0069] For example, the implementation of step S144a above involves using an executable program compiled or interpreted in a preset programming language to determine whether multiple projection distances between the same projection pixel and different three-dimensional pixels have been calculated. The programming languages ​​that can be used include C, C++, Java, BASIC, JavaScript, LISP, Shell, Perl, Ruby, Python, and PHP, etc.

[0070] Step S144b: If multiple projection distances between the same projected pixel and different 3D pixels are calculated, then the shortest projection distance is selected from the multiple projection distances.

[0071] For example, in the implementation of step S144b above: if multiple projection distances between the same projection pixel and different three-dimensional pixels are calculated (i.e., multiple three-dimensional pixels are projected onto the same projection pixel on the target image, there will be multiple projection distances), then the shortest projection distance is selected from the multiple projection distances.

[0072] As a first method of using semantic tags in the aforementioned semantic tag fusion method, after obtaining the fused semantic tags, the target object in the target image can also be extracted based on the semantic tags. This implementation may include:

[0073] Step S150: Extract the region image of the target object from at least one of the multiple target images using the fused semantic labels.

[0074] An example of implementing step S150 above is as follows: assuming the target image is... Figure 2 , Figure 2 There are cars and trees in the text, and the target object is trees. Because the fused semantic tags are... Figure 2 If each two-dimensional pixel in the image has a category label (car, tree, or background), then all two-dimensional pixels with the category label belonging to trees can be extracted from the target image. The set of these two-dimensional pixels is the region image of the tree as the target object.

[0075] As a second way of using semantic tags in the above-mentioned semantic tag fusion method, after obtaining the fused semantic tags, the target object in the point cloud data can also be extracted based on the semantic tags. This implementation method may include:

[0076] Step S160: Extract the 3D pixels of the target object from the point cloud data using the fused semantic labels.

[0077] For example, the implementation of step S160 above is as follows: Suppose the point cloud data is a set of three-dimensional pixels including cars, trees and background, and the target object is trees. Since the fused semantic label is the category label (car, tree or background) of each three-dimensional pixel, then all three-dimensional pixels with category labels belonging to trees can be extracted from the point cloud data. The set of these three-dimensional pixels is the set of three-dimensional pixels of trees as the target object.

[0078] As a third way of using semantic tags in the above-mentioned semantic tag fusion method, after obtaining the fused semantic tags, target pairs in the target image and target objects in the point cloud data can be extracted based on the semantic tags. This implementation may include:

[0079] Step S170: Use the fused semantic labels to extract the region image of the target object from at least one of the multiple target images, and use the fused semantic labels to extract the three-dimensional pixels of the target object from the point cloud data.

[0080] The implementation principle and implementation method of step S170 are similar to those of steps S150 and S160. Therefore, the implementation principle and implementation method will not be described here. If there is anything unclear, please refer to the description of steps S150 and S160.

[0081] Please see Figure 3 The diagram shown is a structural schematic of the semantic tag fusion device provided in the embodiments of this application; the embodiments of this application provide a semantic tag fusion device 200, including:

[0082] The point cloud image acquisition module 210 is used to acquire point cloud data scanned for the target object and multiple target images taken from different perspectives.

[0083] The image semantic segmentation module 220 is used to perform semantic segmentation on each two-dimensional pixel in each of multiple target images to obtain the category label of the two-dimensional pixel.

[0084] The point cloud semantic segmentation module 230 is used to perform semantic segmentation on each three-dimensional pixel in the point cloud data to obtain the category label of the three-dimensional pixel.

[0085] The semantic label acquisition module 240 is used to fuse the category label of the three-dimensional pixel and the category label of the projected pixel based on the projection distance of the three-dimensional pixel onto the projected pixel on the target image, so as to obtain the fused semantic label.

[0086] Optionally, in this embodiment of the application, the point cloud semantic segmentation module includes:

[0087] The point cloud feature extraction submodule is used to extract features from each three-dimensional pixel in the point cloud data to obtain the initial feature vector of the three-dimensional pixel. The point cloud data includes multiple volume elements, and each volume element includes multiple three-dimensional pixels.

[0088] The feature vector averaging submodule is used to average the initial feature vectors of all 3D pixels within each of the multiple volume elements to obtain the average feature vector of that volume element.

[0089] The category label prediction submodule is used to predict the category label of each 3D pixel within a volume element based on the average feature vector of that volume element.

[0090] Optionally, in this embodiment of the application, the point cloud feature extraction submodule includes:

[0091] The pixel feature extraction unit is used to extract features from each three-dimensional pixel in the point cloud data using a pre-trained neural network model.

[0092] Optionally, in this embodiment of the application, the semantic tag acquisition module includes:

[0093] The fusion weight calculation submodule is used to calculate the fusion weight of the category labels of the projected pixels on the target image based on the projection distance.

[0094] The category label fusion submodule is used to fuse the category labels of the 3D pixel and the projected pixel according to the fusion weight.

[0095] Optionally, in this embodiment of the application, the semantic tag acquisition module further includes:

[0096] The camera parameter acquisition submodule is used to acquire the camera parameters of the target image.

[0097] The projection distance calculation submodule is used to project each 3D pixel in the point cloud data onto the projection pixel of the target image according to the camera parameters, and calculate the projection distance between the 3D pixel and the projection pixel.

[0098] Optionally, in this embodiment of the application, the projection distance calculation submodule includes:

[0099] The projection distance calculation unit is used to select the shortest projection distance from the multiple projection distances between the same projection pixel and different 3D pixels if multiple projection distances are calculated.

[0100] Optionally, in this embodiment of the application, the semantic tag fusion device further includes:

[0101] The target object extraction module is used to extract the region image of the target object from at least one of multiple target images using fused semantic labels, and / or to extract the three-dimensional pixels of the target object from point cloud data using fused semantic labels.

[0102] It should be understood that this device corresponds to the semantic tag fusion method embodiment described above and is capable of performing the various steps involved in the above method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0103] Please see Figure 4 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0104] This application embodiment also provides a computer-readable storage medium 330, on which a computer program is stored, and the computer program is executed by a processor 310 to perform the above method.

[0105] The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0106] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0107] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending primarily on the functions involved.

[0108] Furthermore, the functional modules of each embodiment in this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0109] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0110] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A semantic tag fusion method, characterized in that, include: Acquire point cloud data of the target object and multiple target images taken from different perspectives; For each of the multiple target images, semantic segmentation is performed on each two-dimensional pixel in the target image to obtain the category label of the two-dimensional pixel; Semantic segmentation is performed on each three-dimensional pixel in the point cloud data to obtain the category label of the three-dimensional pixel; Based on the projection distance of the three-dimensional pixel point onto the projected pixel point on the target image, the category label of the three-dimensional pixel point and the category label of the projected pixel point are fused to obtain the fused semantic label. Before fusing the category labels of the three-dimensional pixel and the projected pixel based on the projection distance from the three-dimensional pixel to the projected pixel on the target image, the method further includes: Obtain the camera parameters of the target image; Based on the camera parameters, each 3D pixel in the point cloud data is projected onto the projection pixel of the target image, and the projection distance between the 3D pixel and the projection pixel is calculated; wherein, if multiple projection distances between the same projection pixel and different 3D pixels are calculated, the shortest projection distance is selected from the multiple projection distances. The step of fusing the category label of the 3D pixel and the category label of the projected pixel based on the projection distance of the 3D pixel onto the target image includes: The fusion weights of the category labels of the projected pixels on the target image are calculated based on the projection distance; The category labels of the 3D pixel and the category labels of the projected pixel are fused according to the fusion weight.

2. The method according to claim 1, characterized in that, The step of semantic segmentation of each 3D pixel in the point cloud data to obtain the category label of the 3D pixel includes: Each three-dimensional pixel in the point cloud data is subjected to feature extraction to obtain the initial feature vector of the three-dimensional pixel. The point cloud data includes multiple volume elements, and each volume element includes multiple three-dimensional pixels. For each of the plurality of volume elements, the initial feature vectors of all three-dimensional pixels within that volume element are averaged to obtain the average feature vector of that volume element. For each three-dimensional pixel within the volume element, the category label of the three-dimensional pixel is predicted based on the average feature vector of the volume element.

3. The method according to claim 2, characterized in that, The feature extraction of each three-dimensional pixel in the point cloud data includes: Feature extraction is performed on each 3D pixel in the point cloud data using a pre-trained neural network model.

4. The method according to any one of claims 1-3, characterized in that, After obtaining the fused semantic tags, the process also includes: The region image of the target object is extracted from at least one of the multiple target images using the fused semantic tags, and / or, the three-dimensional pixels of the target object are extracted from the point cloud data using the fused semantic tags.

5. A semantic tag fusion device, characterized in that, include: The point cloud image acquisition module is used to acquire point cloud data scanned for the target object and multiple target images taken from different perspectives; The image semantic segmentation module is used to perform semantic segmentation on each two-dimensional pixel in each of the multiple target images to obtain the category label of the two-dimensional pixel; The point cloud semantic segmentation module is used to perform semantic segmentation on each three-dimensional pixel in the point cloud data to obtain the category label of the three-dimensional pixel. The semantic label acquisition module is used to fuse the category label of the three-dimensional pixel and the category label of the projected pixel based on the projection distance of the three-dimensional pixel projected onto the target image, so as to obtain the fused semantic label. The semantic tag acquisition module also includes: The camera parameter acquisition submodule is used to acquire the camera parameters of the target image; The projection distance calculation submodule is used to project each 3D pixel in the point cloud data onto the projection pixel of the target image according to the camera parameters, and calculate the projection distance between the 3D pixel and the projection pixel; if multiple projection distances between the same projection pixel and different 3D pixels are calculated, the shortest projection distance is selected from the multiple projection distances. The semantic tag acquisition module also includes: The fusion weight calculation submodule is used to calculate the fusion weight of the category labels of the projected pixels on the target image based on the projection distance; The category label fusion submodule is used to fuse the category label of the 3D pixel and the category label of the projected pixel according to the fusion weight.

6. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1 to 4.

Images