An aerial view semantic segmentation method, device, equipment and storage medium

Abstract

The application relates to the technical field of image processing, and discloses an aerial view semantic segmentation method, device and equipment and a storage medium, wherein the method comprises the following steps: extracting image features of images of different perspectives of a vehicle surrounding environment; mapping the image features to corresponding position points in hyper voxels, and compressing the hyper voxels in a height dimension to obtain a first aerial view space feature map; calculating feature weights between the position points in the first aerial view space feature map to construct a feature connected graph; traversing all the position points in the feature connected graph, fusing features of all the position points and other position points having a connected relationship with the position points, updating feature vectors of all the position points, and obtaining a second aerial view space feature map containing correlation information. The second aerial view space feature map is subjected to a decoding operation to obtain a semantic segmentation result. The application can associate feature information in an aerial view space map, improves the precision of a semantic segmentation result, and improves the robustness of a semantic segmentation model effect.

Description

[0001] Technology Neighborhood

[0002] This invention relates to the field of image processing technology, and more specifically, to a method, apparatus, device, and storage medium for semantic segmentation of bird's-eye view images. Background Technology

[0003] Target semantic segmentation is the process of subdividing a digital image into multiple image sub-regions, where features within the same sub-region have a certain degree of similarity, such as... Figure 1 As shown, this illustrates the general process of target semantic segmentation. In the field of autonomous driving, target semantic segmentation technology based on multi-camera visual perception is rapidly developing. Multi-camera sensors can observe 360-degree visual information around the vehicle, which can solve the ill-conditioned problems of single-camera spatial positioning to a certain extent, acquiring richer information and providing more meaningful measurements of spatial position and attitude. Among them, target segmentation technology in a bird's-eye view (BEV) based on multi-camera image content inference has become an important and popular research direction. The model design for BEV target segmentation is as follows: Figure 2 As shown.

[0004] However, current methods for generating semantic segmentation results from the BEV perspective based on multi-camera images often project the multi-camera images into the BEV space and then directly decode them based on BEV features. The resulting semantic segmentation prediction map does not take into account the correlation information of BEV space features and does not fully integrate multi-camera information. As a result, the semantic segmentation map has low accuracy and the overall method has low robustness. Summary of the Invention

[0005] To overcome the shortcomings of low robustness and low accuracy of semantic segmentation results in existing technologies, this invention proposes the following technical solution:

[0006] In a first aspect, the present invention proposes a semantic segmentation method for bird's-eye view images, comprising:

[0007] Images of the vehicle's surrounding environment from different perspectives are acquired using multiple cameras, and feature extraction is performed on these images to obtain image features.

[0008] Generate a three-dimensional cubic hypervoxel. Each element in the hypervoxel represents a location point.

[0009] Based on the intrinsic and extrinsic parameters of the multi-camera sensor, image features are mapped to corresponding locations in a supervoxel, and the supervoxels are compressed in the height dimension to obtain a first bird's-eye view spatial feature map. Each location in the first bird's-eye view spatial feature map is accompanied by an image feature vector.

[0010] Calculate the feature weights between location points in the first bird's-eye view spatial feature map, and construct a feature connectivity graph based on the feature weights.

[0011] Traverse all locations in the feature connectivity graph, fuse features of all locations with other locations that are connected to each other, update the feature vectors of all locations, and obtain a second bird's-eye view spatial feature map containing correlation information.

[0012] The second bird's-eye view spatial feature map is decoded to obtain the semantic segmentation result.

[0013] Secondly, the present invention also proposes a bird's-eye view semantic segmentation device, comprising:

[0014] The acquisition module is used to acquire images of the vehicle's surrounding environment from different perspectives using multiple cameras.

[0015] The extraction module is used to extract features from the image to obtain image features.

[0016] The generation module is used to generate three-dimensional cubic hypervoxels. Each element in the hypervoxel represents a location point.

[0017] The mapping module is used to map image features to corresponding locations in the supervoxel based on the intrinsic and extrinsic parameters of the multi-camera sensor.

[0018] The compression module is used to compress the supervoxels mapped with image features in the height dimension to obtain a first bird's-eye view spatial feature map. Each location point in the first bird's-eye view spatial feature map is accompanied by an image feature vector.

[0019] A construction module is used to calculate the feature weights between location points in the first bird's-eye view spatial feature map, and construct a feature connectivity graph based on the feature weights.

[0020] The update module is used to traverse all location points in the feature connectivity graph, fuse the features of all location points with other location points that are connected to each other, update the feature vectors of all location points, and obtain a second bird's-eye view spatial feature map containing correlation information.

[0021] The decoding module is used to perform decoding operations on the second bird's-eye view spatial feature map to obtain semantic segmentation results.

[0022] Thirdly, the present invention also proposes a computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform operations performed by the bird's-eye view semantic segmentation method as described in any of the embodiments of the first aspect.

[0023] In a fourth aspect, the present invention also proposes a computer-readable storage medium on which a program is stored, the program being executed by a processor as performed by the bird's-eye view semantic segmentation method as described in any of the embodiments of the first aspect.

[0024] Compared with the prior art, the beneficial effects of the technical solution of the present invention include: The present invention constructs a feature connectivity graph based on the feature weights between location points with feature vectors in the bird's-eye view spatial feature map, and fuses the features of all location points with other location points that have a connection relationship with them, updates the feature vectors of all location points, and obtains a second bird's-eye view spatial feature map containing correlation information. The second bird's-eye view spatial feature map is then decoded to obtain a semantic segmentation result. This invention can associate feature information in the bird's-eye view spatial map, establish close connections between regional features with the same semantic meaning, weaken the correlation between semantic features in different regions, improve the accuracy of semantic segmentation results, and enhance the robustness of the semantic segmentation model. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the principle of traditional target semantic segmentation models in the background technology.

[0026] Figure 2 This is a schematic diagram of the BEV target segmentation model in the background technology.

[0027] Figure 3 This is a flowchart illustrating the bird's-eye view semantic segmentation method provided in an embodiment of this application.

[0028] Figure 4 This is the principle behind the high-dimensional compression of hypervoxels in the embodiments of this application. Figure I .

[0029] Figure 5 This is the principle behind the high-dimensional compression of hypervoxels in the embodiments of this application. Figure II .

[0030] Figure 6 The neighboring location point P in the embodiment of this application is P. n A diagram showing the positional relationships.

[0031] Figure 7 This is a schematic diagram illustrating the principle of obtaining a second bird's-eye view spatial feature map containing correlation information in an embodiment of this application.

[0032] Figure 8 This is an architectural diagram of a bird's-eye view semantic segmentation device according to an embodiment of this application.

[0033] Figure 9 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0034] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention and are not intended to limit the scope of protection of the present invention.

[0035] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0036] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0037] Specifically, Figure 3 This is a flowchart illustrating the bird's-eye view semantic segmentation method provided in an embodiment of this application.

[0038] like Figure 3 As shown, the bird's-eye view semantic segmentation method includes the following steps:

[0039] S101: Acquire images of the vehicle's surrounding environment from different perspectives using a multi-camera sensor, and extract features from the images to obtain image features.

[0040] S102: Generate a three-dimensional cubic hypervoxel; wherein each element in the hypervoxel is a location point.

[0041] Optionally, in one embodiment of this application, a three-dimensional cubic hypervoxel is generated around the multi-camera sensor or the vehicle body.

[0042] S103: Based on the intrinsic and extrinsic parameters of the multi-camera sensor, the image features are mapped to the corresponding location points in the super voxels, and the super voxels are compressed in the height dimension to obtain the first bird's-eye view spatial feature map; each location point in the first bird's-eye view spatial feature map is accompanied by an image feature vector.

[0043] like Figure 4 and Figure 5 As shown, Figure 4 and Figure 5 The present application describes the principle of high-dimensional compression of hypervoxels in the embodiments of this application. Figure I and Ⅱ In the bird's-eye view space, each small cube is a supervoxel with an additional image feature vector, representing a corresponding location point. Even after compressing the supervoxel in the height dimension, each small cube is still a supervoxel with an additional image feature vector, representing a corresponding location point.

[0044] Optionally, in one embodiment of this application, the compression of the supervoxels in the height dimension includes using summation, averaging, maximization, or 3D convolution methods to compress the supervoxels in the height dimension.

[0045] S104: Calculate the feature weights between location points in the first bird's-eye view spatial feature map, and construct a feature connectivity graph based on the feature weights.

[0046] Optionally, in one embodiment of this application, the feature weights between location points in the first bird's-eye view spatial feature map are represented by cosine similarity or Euclidean distance.

[0047] Optionally, in one embodiment of this application, feature weights are calculated between location points in the first bird's-eye view spatial feature map, and a feature connectivity graph is constructed based on the feature weights. Specific steps include:

[0048] S1041: Initialize the feature weights between all locations and establish an edge with a weight of 0 between every two locations;

[0049] S1042: For a location point P in the first bird's-eye view spatial feature map, calculate the feature vector V of location point P. x,y Its neighboring location point P n eigenvectors The feature weights; where x, x n ∈[0,W), y,y1∈[0,H), V x,y , n is the number of neighboring locations of location P; W and H are the columns and rows of the supervoxels in the first bird's-eye view spatial feature map, respectively; and d is the dimension of the feature vector.

[0050] S1043: If σ is the feature weight threshold, then the location point P and the location point P n Given a connectivity relationship, construct location point P and location point P'. n The feature weights between them are The neighborhood path; otherwise, location point P and location point P n There is no connectivity.

[0051] S1044: For two non-neighboring points P and Q... m If and only if position point P and position point Q m When there are continuous neighborhood paths between location points P and Q, then location points P and Q are... m Given that there is a connection, construct location points P and Q. m The feature weights between them are The non-neighborhood path; where m is the number of non-neighborhood locations of location point P;

[0052] S1045: Traverse all location points in the first bird's-eye view spatial feature map and construct a feature connectivity graph based on the neighborhood paths and non-neighborhood paths between all location points.

[0053] In this embodiment, Euclidean distance is used to represent the feature weights between location points in the first bird's-eye view feature map. For example, the feature vector V of location point P is used. x,y Its neighboring location point P n eigenvectors The expression for the Euclidean distance is as follows:

[0054]

[0055] The eigenvector V of location point P x,y With position point Q m eigenvectors The expression for the Euclidean distance is as follows:

[0056]

[0057] in, This represents the Euclidean distance operation.

[0058] Optionally, in one embodiment of this application, such as Figure 6 As shown, Figure 6 The neighboring location point P in the embodiment of this application is P. n A diagram illustrating the positional relationships of point P, showing the neighborhood points P1 and P2. n It includes eight neighboring locations of location point P: the top left neighbor P1, the top neighbor P2, the top right neighbor P3, the right neighbor P4, the bottom right neighbor P5, the bottom neighbor P6, the bottom left neighbor P7, and the left neighbor P8. Each neighboring location is a supervoxel with a feature vector.

[0059] If the feature similarity between location point P and its right neighbor location point P1 is greater than a threshold, it indicates that there is a connectivity relationship between P and P1, and a neighborhood path between P and P1 is constructed.

[0060] The lower right neighbor point P5 and the upper left neighbor point P1 are not neighbors. However, if point P is connected to the upper left neighbor point P1 and to the lower right neighbor point P5, it means that the lower right neighbor point P5 and the upper left neighbor point P1 are connected. A non-neighbor path between the lower right neighbor point P5 and the upper left neighbor point P1 is then constructed.

[0061] S105: Traverse all points in the feature connectivity graph, fuse features of all points with other points that are connected to each other, update the feature vectors of all points, and obtain a second bird's-eye view spatial feature map containing correlation information. For example... Figure 7 As shown, Figure 7 This is a schematic diagram illustrating the principle of obtaining a second bird's-eye view spatial feature map containing correlation information in an embodiment of this application.

[0062] Optionally, in one embodiment of this application, traversing all location points in the feature connectivity graph, fusing features of all location points with other location points that are connected to each other, and updating the feature vectors of all location points, specifically includes:

[0063] Transform the feature connected graph into a matrix M∈R HW×HW The first bird's-eye view spatial feature map is then transformed into a matrix F∈R. d ×HW .

[0064] Let matrix M∈R HW×HW And matrix F∈R d×HW Perform matrix multiplication and then perform feature processing on the result to obtain a second bird's-eye view spatial feature map containing correlation information.

[0065] S106: Decode the second bird's-eye view spatial feature map to obtain the semantic segmentation result.

[0066] Next, the bird's-eye view semantic segmentation apparatus proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0067] Figure 8 This is an architectural diagram of a bird's-eye view semantic segmentation device according to an embodiment of this application.

[0068] like Figure 8 As shown, the semantic segmentation device includes: an acquisition module 100, an extraction module 200, a generation module 300, a mapping module 400, a compression module 500, a construction module 600, an update module 700, and a decoding module 800.

[0069] The system comprises the following modules: an acquisition module 100 acquires images of the vehicle's surrounding environment from different perspectives using a multi-camera sensor; an extraction module 200 extracts features from the images to obtain image features; a generation module 300 generates three-dimensional cubic hypervoxels, where each element in the hypervoxel is a location point; a mapping module 400 maps the image features to the corresponding location points in the hypervoxels based on the intrinsic and extrinsic parameters of the multi-camera sensor; a compression module 500 compresses the hypervoxels mapped with image features in the height dimension to obtain a first bird's-eye view spatial feature map; each location point in the first bird's-eye view spatial feature map is accompanied by an image feature vector; a construction module 600 calculates the feature weights between location points in the first bird's-eye view spatial feature map and constructs a feature connectivity graph based on the feature weights; an update module 700 traverses all location points in the feature connectivity graph, fuses the features of all location points with other location points that have a connection relationship, updates the feature vectors of all location points, and obtains a second bird's-eye view spatial feature map containing correlation information; and a decoding module 800 decodes the second bird's-eye view spatial feature map to obtain a semantic segmentation result.

[0070] It should be noted that the foregoing explanation of the bird's-eye view semantic segmentation method embodiment also applies to the bird's-eye view semantic segmentation device of this embodiment, and will not be repeated here.

[0071] Figure 9 This is a schematic diagram of the structure of a computing device 900 provided in an embodiment of this application. The computing device 900 includes: a memory 901, a processor 902, and a computer program stored in the memory 901 and executable on the processor 902.

[0072] When processor 902 executes the program, it implements the bird's-eye view semantic segmentation method provided in the above embodiments.

[0073] Furthermore, the computing device 900 also includes a communication interface 903 for communication between the memory 901 and the processor 902.

[0074] The memory 901 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage device.

[0075] If the memory 901, processor 902, and communication interface 903 are implemented independently, then the communication interface 903, memory 901, and processor 902 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0076] Optionally, in a specific implementation, if the memory 901, processor 902, and communication interface 903 are integrated on a single chip, then the memory 901, processor 902, and communication interface 903 can communicate with each other through an internal interface.

[0077] The processor 902 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0078] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described bird's-eye view semantic segmentation method.

[0079] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0080] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0081] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which the embodiments of this application pertain.

[0082] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0083] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0084] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A semantic segmentation method for bird's-eye view images, characterized in that, include: Images of the vehicle's surrounding environment from different perspectives are acquired using multiple cameras, and feature extraction is performed on these images to obtain image features. Generate a three-dimensional cubic hypervoxel; wherein each element in the hypervoxel is a location point; Based on the intrinsic and extrinsic parameters of the multi-camera sensor, image features are mapped to corresponding locations in the super-voxels, and the super-voxels are compressed in the height dimension to obtain the first bird's-eye view spatial feature map; each location in the first bird's-eye view spatial feature map is accompanied by an image feature vector; Calculate the feature weights between location points in the first bird's-eye view spatial feature map, and construct a feature connectivity graph based on the feature weights; Traverse all locations in the feature connectivity graph, fuse all locations with other locations that are connected, update the feature vectors of all locations, and obtain a second bird's-eye view spatial feature map containing correlation information. The second bird's-eye view spatial feature map is decoded to obtain the semantic segmentation result.

2. The bird's-eye view semantic segmentation method according to claim 1, characterized in that, Calculate the feature weights between location points in the first bird's-eye view spatial feature map, and construct a feature connectivity graph based on the feature weights. The specific steps include: Initialize the feature weights between all location points; For a location point in the first bird's-eye view spatial feature map P Calculate the location point eigenvectors Its neighboring location points eigenvectors Feature weights ;in, , , , n Location point P The number of neighborhood location points, and These represent the columns and rows of the supervoxels in the first bird's-eye view spatial feature map. d The dimension of the feature vector. Represents the Euclidean distance operation; like , If the feature weight threshold is applied, then the location point... and location point If there is a connection, construct the location points. and location point The feature weights between them are Neighborhood pathways; otherwise, location points and location point There is no connectivity. For two non-neighboring points and location point if and only if the location point and location point When there are continuous neighborhood paths between them, then the location point and location point If there is a connection, construct the location points. and location point The feature weights between them are Non-neighborhood pathways; among which, m Location point P The number of non-neighborhood location points; Traverse all locations in the first bird's-eye view spatial feature map and construct a feature connectivity graph based on the neighborhood paths and non-neighborhood paths between all locations.

3. The bird's-eye view semantic segmentation method according to claim 2, characterized in that, Location point neighborhood location points Including location points P top left neighbor location point P 1. Upper neighbor location point P 2. Top right neighboring location point P 3. Right Neighbor Location Point P 4. Lower right neighboring location point P 5. Next Neighbor Location Point P 6. Lower left neighboring location point P 7 and left neighbor location points P 8.

4. The bird's-eye view semantic segmentation method according to claim 2, characterized in that, Traverse all points in the feature connectivity graph, fuse the features of each point with other points that are connected to it, and update the feature vectors of all points. Specifically, this includes: Transform the feature connected graph into a matrix And transform the first bird's-eye view spatial feature map into a matrix. ; matrix sum matrix Perform matrix multiplication and then perform feature processing on the result to obtain a second bird's-eye view spatial feature map containing correlation information.

5. The bird's-eye view semantic segmentation method according to claim 1, characterized in that, The feature weights between location points in the first bird's-eye view spatial feature map are represented by cosine similarity or Euclidean distance.

6. The bird's-eye view semantic segmentation method according to claim 1, characterized in that, The high-dimensional compression of supervoxels includes using summation, averaging, maximization, or 3D convolution methods to compress the supervoxels in a high dimension.

7. The bird's-eye view semantic segmentation method according to any one of claims 1 to 6, characterized in that, Generate a three-dimensional cubic hypervoxel around multiple cameras or the vehicle body.

8. A bird's-eye view semantic segmentation device, characterized in that, include: The acquisition module is used to acquire images of the vehicle's surrounding environment from different perspectives using multiple cameras; The extraction module is used to extract features from the image to obtain image features; A generation module is used to generate three-dimensional cubic hypervoxels; wherein each element in the hypervoxel is a location point; The mapping module is used to map image features to corresponding location points in the hypervoxel based on the intrinsic and extrinsic parameters of the multi-camera sensor. The compression module is used to compress the supervoxels mapped with image features in the height dimension to obtain a first bird's-eye view spatial feature map; each location point in the first bird's-eye view spatial feature map is attached with an image feature vector. The construction module is used to calculate the feature weights between location points in the first bird's-eye view spatial feature map and construct a feature connectivity graph based on the feature weights. The update module is used to traverse all location points in the feature connectivity graph, fuse all location points with other location points that are connected to each other, update the feature vectors of all location points, and obtain a second bird's-eye view spatial feature map containing correlation information. The decoding module is used to perform decoding operations on the second bird's-eye view spatial feature map to obtain semantic segmentation results.

9. A computing device, characterized in that, The computing device includes a memory, a processor, and storage devices. A computer program stored in memory and executable on a processor, wherein when the processor executes the program... Implement the operations performed by the bird's-eye view semantic segmentation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that is executed by a processor as described in any one of claims 1 to 7, using the bird's-eye view semantic segmentation method.

Images