Image semantic segmentation method, mobile platform and storage medium

Abstract

The embodiment of the present invention provides an image semantic segmentation method, a mobile platform and a storage medium, which obtains point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image; obtains a depth map according to the point cloud data; obtains a fused image of the depth map and the image; identifies the scene category information of each image block in the fused image; and annotates the scene category information of each image block into the fused image. By fusing the depth map obtained by the laser radar point cloud data with the image taken by the camera, various targets can be accurately separated when performing image semantic segmentation, and the semantic segmentation has a better effect on images with complex background information and images with occlusion or overlap and similar textures, and can improve the accuracy of image semantic segmentation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing, and in particular to an image semantic segmentation method, a movable platform and a storage medium. Background Art

[0002] Image semantic segmentation is a basic task in computer vision. In semantic segmentation, images need to be divided into different semantically interpretable categories, that is, to identify the content and location information in the image. It is widely used in autonomous driving, medical image analysis, robotics and other fields.

[0003] In the prior art, a camera is usually used to take an image and then input it into a neural network model, which then performs semantic segmentation on the image. However, the prior art image semantic segmentation method cannot accurately segment images of some scenes, such as images with complex background information such as car and airplane scenes. It is difficult for the prior art to separate all the targets. For example, it is also difficult for the prior art to distinguish images with occlusion or overlap and similar textures. Summary of the invention

[0004] The embodiments of the present invention provide an image semantic segmentation method, a movable platform and a storage medium to improve the accuracy of image semantic segmentation.

[0005] A first aspect of an embodiment of the present invention is to provide an image semantic segmentation method, comprising:

[0006] Acquire point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image;

[0007] Acquire a depth map according to the point cloud data;

[0008] Obtaining a fused image of the depth map and the image;

[0009] Identifying scene category information of each image block in the fused image;

[0010] The scene category information of each image block is annotated into the fused image.

[0011] A second aspect of an embodiment of the present invention is to provide a movable platform, including: a laser radar, a camera, a memory, and a processor;

[0012] The memory is used to store program code;

[0013] The processor calls the program code, and when the program code is executed, it is used to perform the following operations:

[0014] Acquire point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image;

[0015] Acquire a depth map according to the point cloud data;

[0016] Obtaining a fused image of the depth map and the image;

[0017] Identifying scene category information of each image block in the fused image;

[0018] The scene category information of each image block is annotated into the fused image.

[0019] A third aspect of an embodiment of the present invention is to provide a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the method as described in the first aspect.

[0020] The image semantic segmentation method, movable platform and storage medium provided by the embodiments of the present invention obtain point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image; obtain a depth map according to the point cloud data; obtain a fused image of the depth map and the image; identify the scene category information of each image block in the fused image; and annotate the scene category information of each image block into the fused image. By fusing the depth map obtained by the laser radar point cloud data with the image taken by the camera, various targets can be accurately separated when performing image semantic segmentation, and the semantic segmentation has a better effect on images with complex background information and images with occlusion or overlap and similar textures, and can improve the accuracy of image semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of an image semantic segmentation method provided by an embodiment of the present invention;

[0023] Figure 2a A schematic diagram of an application scenario of the image semantic segmentation method provided by an embodiment of the present invention;

[0024] Figure 2b for Figure 2a Schematic diagram of the depth map and image of the scene;

[0025] Figure 3a A schematic diagram of an application scenario of an image semantic segmentation method provided by another embodiment of the present invention;

[0026] Figure 3b for Figure 3a Schematic diagram of the depth map and image of the scene;

[0027] Figure 4a A schematic diagram of an application scenario of an image semantic segmentation method provided by another embodiment of the present invention;

[0028] Figure 4b for Figure 4a Schematic diagram of the depth map and image of the scene;

[0029] Figure 5 A flowchart of an image semantic segmentation method provided by another embodiment of the present invention;

[0030] Figure 6 A flowchart of an image semantic segmentation method provided by another embodiment of the present invention;

[0031] Figure 7 A flowchart of an image semantic segmentation method provided by another embodiment of the present invention;

[0032] Figure 8 A structural diagram of a movable platform provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there can be a central component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there can be a central component at the same time.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0036] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0037] An embodiment of the present invention provides an image semantic segmentation method. Figure 1 Flow chart of the image semantic segmentation method provided by the embodiment of the present invention. Figure 1 As shown, the image semantic segmentation method in this embodiment may include:

[0038] Step S101, obtaining point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image.

[0039] In this embodiment, a laser radar and a camera are set on a movable platform such as a vehicle, a drone, or a robot. The laser radar can collect point cloud data, and the camera can capture images (e.g., RGB images), wherein the point cloud data corresponds to the image, and the point cloud data can overlap with the image when projected into the image, that is, the laser radar and the camera can scan the same target area, and the line of sight direction of the laser radar is the same as or close to the shooting direction of the camera, which can be achieved by jointly calibrating the laser radar and the camera. That is, before obtaining the point cloud data collected by the laser radar and the image captured by the camera in S101, the laser radar and the camera are calibrated so that the point cloud data collected by the laser radar corresponds to the image captured by the camera. Of course, the point cloud data can also be made to correspond to the image after the point cloud data and the image are obtained through rotation and translation processing.

[0040] In this embodiment, the laser radar and the camera can be fixedly installed on the movable platform, or they can be installed on the movable platform through a rotating component to realize the collection of point cloud data and images within a certain range.

[0041] Step S102: Acquire a depth map according to the point cloud data.

[0042] In this embodiment, after acquiring the point cloud data, a depth map can be acquired based on the point cloud data, wherein each pixel value of the depth map represents the distance from the object to the laser radar, and the point cloud data has the orientation and distance information of the object, and the depth map can be acquired based on the orientation and distance information of the object in the point cloud data. In this embodiment, the point cloud data can be converted into a depth map through an existing algorithm, or the point cloud data can be converted into a depth map through a pre-trained convolutional neural network model.

[0043] Step S103: Obtain a fused image of the depth map and the image.

[0044] In this embodiment, after the depth map is obtained, the depth map and the image can be fused, so that a fused image of the depth map and the image can be obtained. Specifically, since the point cloud data corresponds to the image, the depth map can be directly projected into the image to form the fused image. In this embodiment, feature points can be extracted from the depth map and the image respectively, and the corresponding feature points in the depth map and the image can be obtained by comparison. Based on the corresponding feature points, the coordinate system of the depth map is converted to the coordinate system of the image through operations such as rotation, translation, and cropping, and then the depth map is projected into the image coordinate system, so that each pixel of the depth map is aligned with the image, and then the depth map is fused with the image at the data level to obtain a fused image.

[0045] In addition, in this embodiment, the point cloud data and the image can be aligned first, for example, by using a grayscale area-based registration method, a feature-based registration method, a registration method combining line features and point features, etc., and then a depth map is obtained based on the point cloud data. At this time, the depth map obtained is aligned with the image, and then the depth map can be directly fused with the image to obtain a fused image.

[0046] Step S104: identifying scene category information of each image block in the fused image.

[0047] In this embodiment, multiple preset scene categories can be predefined, and by identifying the scene category of any image block in the fused image, it is determined which preset scene category the image block belongs to, and then the scene category information of the image block can be obtained. More specifically, the probability that the image block belongs to each preset scene category among the multiple preset scene categories can be obtained, and the preset scene category corresponding to the maximum probability is used as the scene category information of the image block.

[0048] In this embodiment, the preset scene categories may include, for example, car, sky, road, static obstacle, dynamic obstacle, etc.; of course, more detailed scene categories may also be included, for example, cars may be specifically divided into cars, trucks, buses, trains, RVs, etc., static obstacles may be specifically divided into buildings, walls, guardrails, electric poles, traffic lights, traffic signs, etc., and dynamic obstacles may include pedestrians, bicycles, motorcycles, etc. In this embodiment, the above-mentioned car, sky, road, static obstacle, dynamic obstacle, etc. categories may be used as primary scene categories, and more detailed scene categories may be used as secondary scene categories, that is, the multiple preset scene categories described in this embodiment may include at least one primary scene category and at least one secondary scene category, wherein any of the primary scene categories has at least one of the secondary scene categories as a subcategory.

[0049] Furthermore, in this embodiment, the recognition of scene category information of image blocks can be achieved by using a convolutional neural network. Of course, other methods can also be used for recognition, which will not be described in detail here.

[0050] In another embodiment, the depth map and the image may be aligned first, and then the scene category information of each image block in the image may be identified based on the correspondence between the depth map and the image. Then, the depth map and the image may be fused to obtain a fused image, and the scene category information of each image block may be annotated into the fused image.

[0051] Step S105: annotate the scene category information of each image block into the fused image.

[0052] In this embodiment, after identifying the scene category information of each image block in the fused image, the scene category information can be annotated into the fused image to generate a semantic map, thereby completing the image semantic segmentation. In this embodiment, the annotation of the scene category information can only be annotated with a mark corresponding to the scene category, or the image block can be marked with a corresponding color according to the scene category information.

[0053] The image semantic segmentation method provided in this embodiment can be applied to semantic segmentation of images with occlusion, such as Figure 2a As shown in the top view, when cars A and B are in the positions shown in the figure, the movable platform equipped with laser radar and camera is viewed from the direction of the arrow. Car A is partially blocked by car B. After collecting point cloud data along the viewing angle by laser radar, the following can be obtained: Figure 2b The depth map shown in the figure above (where different filling patterns represent different depths) can be obtained by taking pictures along the observation angle with a camera. Figure 2b The image shown in the figure below can distinguish the two vehicles at different distances in front by fusing the depth map with the image and performing speech segmentation, and can also further distinguish the type of vehicle (that is, scene category information).

[0054] Of course, this embodiment can also distinguish more complex occlusion situations, such as Figure 3a In the top view shown in the figure, the movable platform equipped with laser radar and camera is viewed from the direction of the arrow. Car C blocks part of car B and car A, and car B blocks part of car A. After collecting point cloud data along the viewing angle through the laser radar, we can get Figure 3b The depth map shown in the figure above can be obtained by taking pictures along the observation angle with a camera. Figure 3b The image shown in the figure below fuses the depth map with the image and performs speech segmentation, which can distinguish different occlusion relationships in front and further distinguish the type of vehicle (that is, scene category information).

[0055] The image semantic segmentation method provided in this embodiment can also be applied to semantic segmentation of images with objects with similar textures, such as Figure 4a In the top view shown in the figure, there is a corner wall in front of the movable platform equipped with laser radar and camera. Wall D is closer to the movable platform than wall E, and wall D and wall E have similar textures. After collecting point cloud data along the observation angle through laser radar, it can be obtained Figure 4b The depth map shown in the figure above can be obtained by taking pictures along the observation angle with a camera. Figure 4b In the image shown in the figure below, by fusing the depth map with the image and performing speech segmentation, the front-to-back relationship between wall D and wall E can be distinguished, and the scene category information can be further distinguished as a wall with a corner.

[0056] The image semantic segmentation method provided in this embodiment obtains point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image; obtains a depth map based on the point cloud data; obtains a fused image of the depth map and the image; identifies the scene category information of each image block in the fused image; and annotates the scene category information of each image block into the fused image. By fusing the depth map obtained by the laser radar point cloud data with the image taken by the camera, various targets can be accurately separated when performing image semantic segmentation, and the semantic segmentation has a better effect on images with complex background information and images with occlusion or overlap and similar textures, and can improve the accuracy of image semantic segmentation.

[0057] An embodiment of the present invention provides an image semantic segmentation method. Figure 5 and Figure 6 FIG. 1 is a flowchart of an image semantic segmentation method provided by another embodiment of the present invention. Figure 5 and Figure 6 As shown, based on the above embodiment, the image semantic segmentation method in this embodiment may include:

[0058] Step S201, obtaining point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image.

[0059] Step S202: Acquire a depth map according to the point cloud data.

[0060] Wherein, steps S201 and S202 may refer to S101 and S102 in the above embodiment, and will not be described in detail here. In this embodiment, the following second model is used to obtain a depth map according to the point cloud data.

[0061] Step S203: input the depth map and the image into a pre-trained first model to obtain a fused image of the depth map and the image.

[0062] Step S204: obtaining, through the first model, the probability that any image block in the fused image belongs to each of the plurality of preset scene categories, and taking the preset scene category corresponding to the maximum probability as the scene category information of the image block.

[0063] In this embodiment, the first model may be a convolutional neural network model, such as Figure 7 As shown, the first model includes a plurality of first processing units connected in sequence, and the first processing unit includes a convolution layer, a batch normalization layer, and an activation layer. Further, the plurality of first processing units connected in sequence in the first model can prevent gradient dissipation through skip connection.

[0064] In this embodiment, after inputting the corresponding depth map and image into the first model, the first model first fuses the depth map with the image to obtain a fused image of the depth map and the image, and then the first model performs scene category recognition on each image block of the fused image. For any image block, the probability of it belonging to each preset scene category among multiple preset scene categories is obtained, and the preset scene category corresponding to the maximum probability is used as the scene category information of the image block.

[0065] It should be noted that for the training of the first model, a plurality of fused images marked with preset scene categories can be first obtained as training sets and test sets. Through training, the characteristics of image blocks of each preset scene category can be obtained, so that when a new fused image is input, the probability of each image block in the new fused image belonging to each preset scene category in multiple preset scene categories can be analyzed, and then the preset scene category corresponding to the maximum probability can be used as the scene category information of the image block. The specific training process will not be repeated here.

[0066] Based on any of the above embodiments, the multiple preset scene categories include at least one primary scene category and at least one secondary scene category, wherein any of the primary scene categories has at least one of the secondary scene categories as a subcategory.

[0067] Furthermore, step S204 may specifically include:

[0068] The probability of any image block in the fused image belonging to each of the secondary scene categories is obtained through the first model, and the secondary scene category corresponding to the maximum probability or the primary scene category to which the secondary scene category belongs is used as the scene category information of the image block; or

[0069] The probability that any image block in the fused image belongs to each of the first-level scene categories is obtained through the first model, and the first-level scene category corresponding to the maximum probability is used as the scene category information of the image block.

[0070] In the present embodiment, since the preset scene categories include primary scenes and secondary scene categories, when the probability that any image block belongs to each preset scene category among multiple preset scene categories is obtained through the first model, the secondary scene category can be used as a measurement standard to determine the probability that the image block belongs to each secondary scene category, and then the secondary scene category corresponding to the maximum probability can be used as the scene category information of the image block. Of course, the scene category to which the secondary scene category belongs can also be used as the scene category information of the image block. For example, if the secondary scene category corresponding to the maximum probability is pedestrians, pedestrians or dynamic obstacles can be used as the scene category information of the image block. In addition, in the present embodiment, the primary scene category can also be directly used as a measurement standard to determine the probability that the image block belongs to each primary scene category, and the primary scene category corresponding to the maximum probability can be used as the scene category information of the image block.

[0071] Step S205: annotate the scene category information of each image block into the fused image.

[0072] In this embodiment, step S205 may refer to S105 in the above embodiment, and will not be described in detail here.

[0073] On the basis of any of the above embodiments, after obtaining the probability that any image block in the fused image belongs to each preset scene category among multiple preset scene categories and taking the preset scene category corresponding to the maximum probability as the scene category information of the image block, the method may further include:

[0074] If the maximum probability is lower than a preset probability threshold, the scene category information of the image block is ignored in the fused image.

[0075] In this embodiment, since there may be certain errors in the recognition of scene category information, the judgment of misrecognition can be achieved by comparing the above-mentioned maximum probability with the preset probability threshold. That is, when the maximum probability is lower than the preset probability threshold, it is determined that the image block may be misrecognized. For image blocks that are misrecognized, the scene category information of the image block can be directly ignored to avoid affecting the semantic segmentation results.

[0076] Based on any of the above embodiments, the method further includes:

[0077] The scene category information of any image block is compared with the scene category information of multiple adjacent image blocks in terms of similarity. If the similarity is lower than a preset similarity threshold, the scene category information of the image block is ignored in the fused image.

[0078] In this embodiment, if the scene category information recognition result of a certain image block is significantly different from the scene category information of multiple adjacent image blocks, it indicates that there is a possibility of misidentification, and the scene category information of the image block also needs to be ignored. In this embodiment, the scene category information of any image block is compared with the scene category information of multiple adjacent image blocks in terms of similarity. When there is a significant difference in the similarity, that is, the similarity is lower than a preset similarity threshold, the scene category information of the image block is judged to be misidentified, and the scene category information of the image block is ignored to avoid affecting the semantic segmentation result.

[0079] The image semantic segmentation method provided in this embodiment obtains point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image; obtains a depth map based on the point cloud data; obtains a fused image of the depth map and the image; identifies the scene category information of each image block in the fused image; and annotates the scene category information of each image block into the fused image. By fusing the depth map obtained by the laser radar point cloud data with the image taken by the camera, various targets can be accurately separated when performing image semantic segmentation, and the semantic segmentation has a better effect on images with complex background information and images with occlusion or overlap and similar textures, and can improve the accuracy of image semantic segmentation.

[0080] On the basis of any of the above embodiments, obtaining a depth map according to the point cloud data in S102 and S202 may specifically include:

[0081] The point cloud data is input into a pre-trained second model to generate the depth map.

[0082] Wherein, the second model can be a convolutional neural network model, such as Figure 7 As shown, the second model includes a plurality of second processing units connected in sequence, and the second processing units include a convolution layer, a batch normalization layer, and an activation layer. Further, the plurality of second processing units connected in sequence in the second model can prevent gradient dissipation through skip connections.

[0083] In this embodiment, the conversion of point cloud data into depth map is realized by a pre-trained second model, wherein the training of the second model can obtain multiple groups of corresponding point cloud data and depth maps as training sets and test sets to train the second model, and the specific training process will not be repeated here.

[0084] Furthermore, the step of inputting the point cloud data into a pre-trained first model to generate the depth map includes:

[0085] If the density of the point cloud data is higher than a preset density threshold, converting the point cloud data into the depth map by using the second model; or

[0086] If the density of the point cloud data is not higher than the preset density threshold, the point cloud data is densified using the second model, and the densified point cloud data is converted into the depth map.

[0087] In this embodiment, the density of the point cloud data affects the accuracy of the depth information in the depth map, which will greatly affect the accuracy of the image semantic segmentation. Therefore, when converting the point cloud data into a depth map, it is necessary to determine whether the density of the point cloud data meets the requirements. When the density of the point cloud data is higher than the preset density threshold, it means that the density of the point cloud data meets the requirements, and the point cloud data can be directly converted into the depth map through the second model; when the density of the point cloud data is not higher than the preset density threshold, it means that the density of the point cloud data does not meet the requirements, and the point cloud data needs to be densified. In this embodiment, the second model can be trained so that the second model has the function of densifying the point cloud data. When converting the point cloud data into a depth map through the second model, the point cloud data is first densified and then converted to the depth map. Of course, other algorithms can also be used to achieve the densification of point cloud data, such as a point cloud densification algorithm based on sparse matching, which will not be repeated here.

[0088] An embodiment of the present invention provides a movable platform. Figure 8 A structural diagram of a movable platform provided by an embodiment of the present invention, such as Figure 8 As shown, the movable platform 30 includes: a laser radar 31, a camera 32, a processor 33 and a memory 34.

[0089] The memory 34 is used to store program codes;

[0090] The processor 33 calls the program code, and when the program code is executed, it is used to perform the following operations:

[0091] Acquire point cloud data collected by the laser radar 31 and an image captured by the camera 32, wherein the point cloud data corresponds to the image;

[0092] Acquire a depth map according to the point cloud data;

[0093] Obtaining a fused image of the depth map and the image;

[0094] Identifying scene category information of each image block in the fused image;

[0095] The scene category information of each image block is annotated into the fused image.

[0096] On the basis of any of the above embodiments, when the processor 33 obtains a fused image of the depth map and the image, the processor 33 is configured to:

[0097] The depth map and the image are input into a pre-trained first model to obtain a fused image of the depth map and the image.

[0098] On the basis of any of the above embodiments, when the processor 33 identifies the scene category information of each image block in the fused image, the processor 33 is configured to:

[0099] The probability that any image block in the fused image belongs to each preset scene category among multiple preset scene categories is obtained through the first model, and the preset scene category corresponding to the maximum probability is used as the scene category information of the image block.

[0100] Based on any of the above embodiments, the multiple preset scene categories include at least one primary scene category and at least one secondary scene category, wherein any of the primary scene categories has at least one of the secondary scene categories as a subcategory.

[0101] On the basis of any of the above embodiments, when the processor 33 obtains the probability that any image block in the fused image belongs to each preset scene category among multiple preset scene categories through the first model, and uses the preset scene category corresponding to the maximum probability as the scene category information of the image block, the processor 33 is configured as follows:

[0102] The probability of any image block in the fused image belonging to each of the secondary scene categories is obtained through the first model, and the secondary scene category corresponding to the maximum probability or the primary scene category to which the secondary scene category belongs is used as the scene category information of the image block; or

[0103] The probability that any image block in the fused image belongs to each of the first-level scene categories is obtained through the first model, and the first-level scene category corresponding to the maximum probability is used as the scene category information of the image block.

[0104] On the basis of any of the above embodiments, after the processor 33 uses the preset scene category corresponding to the maximum probability as the scene category information of the image block, the processor 33 is further configured to:

[0105] If the maximum probability is lower than a preset probability threshold, the scene category information of the image block is ignored in the fused image.

[0106] Based on any of the above embodiments, the processor 33 is further configured to:

[0107] The scene category information of any image block is compared with the scene category information of multiple adjacent image blocks in terms of similarity. If the similarity is lower than a preset similarity threshold, the scene category information of the image block is ignored in the fused image.

[0108] Based on any of the above embodiments, the first model is a convolutional neural network model, the first model includes a plurality of first processing units connected in sequence, and the first processing unit includes a convolutional layer, a batch normalization layer and an activation layer.

[0109] On the basis of any of the above embodiments, when the processor 33 obtains the depth map according to the point cloud data, the processor 33 is configured to:

[0110] The point cloud data is input into a pre-trained second model to generate the depth map.

[0111] On the basis of any of the above embodiments, when the processor 33 inputs the point cloud data into the pre-trained first model to generate the depth map, the processor 33 is configured as follows:

[0112] If the density of the point cloud data is higher than a preset density threshold, converting the point cloud data into the depth map by using the second model; or

[0113] If the density of the point cloud data is not higher than the preset density threshold, the point cloud data is densified using the second model, and the densified point cloud data is converted into the depth map.

[0114] Based on any of the above embodiments, the second model is a convolutional neural network model, the second model includes a plurality of second processing units connected in sequence, and the second processing unit includes a convolution layer, a batch normalization layer and an activation layer.

[0115] On the basis of any of the above embodiments, before the processor 33 acquires the point cloud data collected by the laser radar 31 and the image taken by the camera 32, the processor 33 is further configured as follows:

[0116] The laser radar 31 and the camera 32 are calibrated so that the point cloud data collected by the laser radar 31 corresponds to the image taken by the camera 32 .

[0117] Based on any of the above embodiments, the movable platform 30 includes: at least one of a vehicle, a drone, and a robot.

[0118] The specific principles and implementation methods of the movable platform 30 provided in this embodiment are similar to those in the above embodiment and will not be described in detail here.

[0119] The mobile platform provided in this embodiment obtains point cloud data collected by the laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image; obtains a depth map based on the point cloud data; obtains a fused image of the depth map and the image; identifies scene category information of each image block in the fused image; and annotates the scene category information of each image block into the fused image. By fusing the depth map obtained by the laser radar point cloud data with the image taken by the camera, various targets can be accurately separated when performing image semantic segmentation, and the semantic segmentation has a better effect on images with complex background information and images with occlusion or overlap and similar textures, and can improve the accuracy of image semantic segmentation.

[0120] In addition, this embodiment further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the image semantic segmentation method described in the above embodiment.

[0121] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0123] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0124] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0125] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image semantic segmentation, characterized in that: include: Acquire point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image; Acquire a depth map according to the point cloud data; Obtaining a fused image of the depth map and the image; Identifying scene category information of each image block in the fused image; marking the scene category information of each image block into the fused image; The obtaining of a depth map according to the point cloud data comprises: Inputting the point cloud data into a pre-trained second model, and if the density of the point cloud data is higher than a preset density threshold, converting the point cloud data into the depth map through the second model; or If the density of the point cloud data is not higher than the preset density threshold, the point cloud data is densified using the second model, and the densified point cloud data is converted into the depth map.

2. The method according to claim 1, characterized in that: The obtaining a fused image of the depth map and the image includes: The depth map and the image are input into a pre-trained first model to obtain a fused image of the depth map and the image.

3. The method according to claim 1 or 2, characterized in that: The identifying the scene category information of each image block in the fused image includes: The probability that any image block in the fused image belongs to each preset scene category among multiple preset scene categories is obtained through the first model, and the preset scene category corresponding to the maximum probability is used as the scene category information of the image block.

4. The method according to claim 3, characterized in that The plurality of preset scene categories include at least one primary scene category and at least one secondary scene category, wherein any of the primary scene categories has at least one of the secondary scene categories as a subcategory.

5. The method according to claim 4, characterized in that The obtaining, by using the first model, the probability that any image block in the fused image belongs to each preset scene category among a plurality of preset scene categories, and taking the preset scene category corresponding to the maximum probability as the scene category information of the image block, comprises: Obtaining the probability that any image block in the fused image belongs to each of the secondary scene categories through the first model, and taking the secondary scene category corresponding to the maximum probability or the primary scene category to which the secondary scene category belongs as the scene category information of the image block; or The probability that any image block in the fused image belongs to each of the first-level scene categories is obtained through the first model, and the first-level scene category corresponding to the maximum probability is used as the scene category information of the image block.

6. The method according to claim 3, characterized in that After taking the preset scene category corresponding to the maximum probability as the scene category information of the image block, the method further includes: If the maximum probability is lower than a preset probability threshold, the scene category information of the image block is ignored in the fused image.

7. The method according to claim 2, characterized in that: Also includes: The scene category information of any image block is compared with the scene category information of multiple adjacent image blocks in terms of similarity. If the similarity is lower than a preset similarity threshold, the scene category information of the image block is ignored in the fused image.

8. The method according to claim 2, characterized in that: The first model is a convolutional neural network model, which includes a plurality of first processing units connected in sequence, and the first processing units include a convolutional layer, a batch normalization layer and an activation layer.

9. The method according to claim 1, characterized in that: The second model is a convolutional neural network model, which includes a plurality of second processing units connected in sequence, and the second processing units include a convolution layer, a batch normalization layer and an activation layer.

10. The method according to claim 1, characterized in that Before obtaining the point cloud data collected by the laser radar and the image taken by the camera, the method further includes: The laser radar and the camera are calibrated so that the point cloud data collected by the laser radar corresponds to the image taken by the camera.

11. A movable platform, characterized in that: include: LiDAR, cameras, memory, and processors; The memory is used to store program code; The processor calls the program code, and when the program code is executed, it is used to perform the following operations: Acquire point cloud data collected by a laser radar and an image taken by a camera, wherein the point cloud data corresponds to the image; Acquire a depth map according to the point cloud data; Obtaining a fused image of the depth map and the image; Identifying scene category information of each image block in the fused image; marking the scene category information of each image block into the fused image; The obtaining of a depth map according to the point cloud data comprises: Inputting the point cloud data into a pre-trained second model, and if the density of the point cloud data is higher than a preset density threshold, converting the point cloud data into the depth map through the second model; or If the density of the point cloud data is not higher than the preset density threshold, the point cloud data is densified using the second model, and the densified point cloud data is converted into the depth map.

12. The movable platform according to claim 11, characterized in that: When the processor obtains a fused image of the depth map and the image, the processor is configured to: The depth map and the image are input into a pre-trained first model to obtain a fused image of the depth map and the image.

13. The movable platform according to claim 11 or 12, characterized in that: When the processor identifies the scene category information of each image block in the fused image, the processor is configured to: The probability that any image block in the fused image belongs to each preset scene category among multiple preset scene categories is obtained through the first model, and the preset scene category corresponding to the maximum probability is used as the scene category information of the image block.

14. The movable platform according to claim 13, characterized in that: The plurality of preset scene categories include at least one primary scene category and at least one secondary scene category, wherein any of the primary scene categories has at least one of the secondary scene categories as a subcategory.

15. The movable platform according to claim 14, characterized in that: When the processor obtains the probability that any image block in the fused image belongs to each preset scene category among multiple preset scene categories through the first model, and uses the preset scene category corresponding to the maximum probability as the scene category information of the image block, the processor is configured to: Obtaining the probability that any image block in the fused image belongs to each of the secondary scene categories through the first model, and taking the secondary scene category corresponding to the maximum probability or the primary scene category to which the secondary scene category belongs as the scene category information of the image block; or The probability that any image block in the fused image belongs to each of the first-level scene categories is obtained through the first model, and the first-level scene category corresponding to the maximum probability is used as the scene category information of the image block.

16. The movable platform according to claim 13, characterized in that: After the processor uses the preset scene category corresponding to the maximum probability as the scene category information of the image block, the processor is further configured to: If the maximum probability is lower than a preset probability threshold, the scene category information of the image block is ignored in the fused image.

17. The movable platform according to claim 12, characterized in that: The processor is further configured to: The scene category information of any image block is compared with the scene category information of multiple adjacent image blocks in terms of similarity. If the similarity is lower than a preset similarity threshold, the scene category information of the image block is ignored in the fused image.

18. The movable platform according to claim 12, characterized in that: The first model is a convolutional neural network model, which includes a plurality of first processing units connected in sequence, and the first processing units include a convolutional layer, a batch normalization layer and an activation layer.

19. The movable platform according to claim 12, characterized in that: The second model is a convolutional neural network model, which includes a plurality of second processing units connected in sequence, and the second processing units include a convolution layer, a batch normalization layer and an activation layer.

20. The movable platform according to claim 11, characterized in that: Before the processor acquires the point cloud data collected by the laser radar and the image taken by the camera, the processor is further configured to: The laser radar and the camera are calibrated so that the point cloud data collected by the laser radar corresponds to the image taken by the camera.

21. The movable platform according to claim 11, characterized in that: The movable platform includes at least one of a vehicle, a drone, and a robot.

22. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 10.

Images