Method and system for implementing bird's eye semantic segmentation based on implicit set prediction

Abstract

The application relates to a method and system for realizing bird's-eye view semantic segmentation based on implicit set prediction, which comprises the following steps: preprocessing labeled data in a high-precision map, converting a semantic layer of the high-precision map into a labeled polygon layer; extracting semantic features in the labeled polygon layer, performing prediction head analysis on the semantic features, realizing splicing and fusion of multi-view cameras, and finally obtaining a polygon segmentation frame through implicit set prediction; and converting an image plane into a perspective of a bird's-eye view, and completing implicit prediction based on a transformer. The application can make a network simultaneously perceive dynamic objects and static roads, and reduce the calculation amount of the perception network; and can be applied in the field of automatic driving.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a method and system for semantic segmentation of bird's-eye view images based on implicit set prediction using computer vision and autonomous driving surround-view cameras. Background Technology

[0002] Autonomous vehicles need to accurately understand objects and static elements in traffic scenarios. Environmental perception is generally achieved through multiple sensors, including LiDAR, cameras, and millimeter-wave radar. Surround-view multi-camera environmental perception technology includes tasks such as pose estimation, object detection, and bird's-eye view semantic segmentation. Due to the lack of depth information in cameras, object detection in 3D environments is difficult and the low accuracy does not meet the perception requirements of high-level autonomous driving. However, since vehicles generally operate on the ground, the representation of objects in the bird's-eye view can directly affect the vehicle's cognition and decision-making system. Moreover, the transformation relationship from the camera plane to the geodetic reference plane is relatively clear. Therefore, bird's-eye view semantic segmentation is an important task of surround-view camera systems and an important prerequisite for the latest local map construction. For example, Tesla's FSD system can build an understanding of the semantic attributes of lane lines and lanes in road scenes by processing data from surround-view cameras on roads without maps.

[0003] The key stages of semantic segmentation of bird's-eye view images are image feature extraction, perspective transformation from the image viewpoint to the bird's-eye viewpoint, multi-camera image stitching and fusion, and the techniques for feature extraction and prediction heads for bird's-eye view images. Current technologies utilize advanced segmentation or detection backbone networks for image feature extraction. Perspective transformation and depth estimation are closely related. There are various methods for perspective transformation. The basic method uses inverse perspective transformation of the camera, which only requires the camera's intrinsic parameter model, but its strong planar assumption limits its universality. A popular method uses pseudo-LiDAR, estimating the depth of each pixel in the image and projecting it onto a point cloud, then directly projecting the 3D point cloud onto the bird's-eye view. The difficulty of this method lies in the difficulty of obtaining and training pixel depths. A corresponding improvement is to use depth distribution instead of deterministic depth prediction. However, the performance of this method is easily affected by erroneous depth estimation. A third method uses a convolutional network to predict the correspondence between each pixel and the corresponding grid in the bird's-eye view. This method is still limited by the fuzziness of the smallest unit correspondence in perception and scene specificity; its scene universality has not been systematically studied.

[0004] In multi-camera object detection tasks, similar viewpoint transformation units with different techniques as described above are also used, including methods such as directly predicting 3D bounding boxes from images, converting images into pseudo-point clouds and outputting bounding boxes through point cloud prediction heads, and multi-task learning of image depth estimation and bounding box prediction. For surround-view cameras, there is an implicit prediction method based on a transformer. This method does not require exact pixel-by-pixel matching and very precise correspondence between intrinsic and extrinsic parameters. It generates bounding box queries end-to-end through a transformer structure, transforming the localization problem in detection into an ensemble prediction problem. It is simple and effective in design, but depends on the consistency of the output and the design of the ensemble. Summary of the Invention

[0005] To address the aforementioned problems, the purpose of this invention is to provide a method and system for semantic segmentation of bird's-eye view images based on implicit set prediction, which enables the network to simultaneously perceive dynamic objects and static roads, thereby reducing the computational load of the perception network.

[0006] To achieve the above objectives, in a first aspect, the technical solution adopted by the present invention is as follows: a method for semantic segmentation of bird's-eye view based on implicit ensemble prediction, comprising: preprocessing the labeled data in a high-precision map to transform the semantic layer of the high-precision map into a labeled polygon layer; extracting semantic features from the labeled polygon layer, performing predictive head analysis on the semantic features, realizing the stitching and fusion of multi-view cameras, so as to predict the final polygon segmentation box by implicit ensemble prediction; and converting the perspective of the image plane and the top view to complete the implicit prediction based on the transformer.

[0007] Furthermore, the labeled data in the high-precision map is preprocessed, including dynamic target preprocessing and static road element preprocessing;

[0008] Dynamic target preprocessing includes: squeezing the target from the height layer to form a ground occupancy mask, forming a polygon on the ground plane, and assigning corresponding semantic labels to the mask;

[0009] Static road feature preprocessing includes: it does not need to be completely covered by a high-precision map when the vehicle is running; only a portion of the map data is needed as training labels during training; and it relies on semantic map training to enable effective semantic segmentation results to be generalized to any structured road.

[0010] Furthermore, the static road element preprocessing also includes: allowing for a pre-defined error in the linear map information, and assigning a pre-defined width to the polyline so that the polyline can be expanded into a long and narrow polygon for training.

[0011] Furthermore, predictive head analysis is performed on the semantic features, including:

[0012] The prediction head regresses the image features obtained from the pre-processor network into the final required physical quantity, compares them with the ground truth labels to obtain the loss function, and then backpropagates the loss function.

[0013] The transmission loss function consists of three parts: two-sided loss, classification loss, and crossover ratio loss.

[0014] Furthermore, the prediction head is a single-stage, anchorless FCOS3D model that classifies each pixel and obtains a two-dimensional polygon through a centering operation, and then directly guides the generation of the three-dimensional polygon region through the two-dimensional polygon.

[0015] Furthermore, the perspective of the image plane is transformed from that of the top view, including:

[0016] An implicit mapping in the perspective transformation is performed using a transformer. The input of the transformer is a dimension-reduced feature map, and the output is the center point information of the polygons in the bird's-eye view.

[0017] The feature map of the image is compressed from (batch size, height, width, number of channels) to (batch size, height multiplied by width, number of channels), input into the encoder unit of the transformer, and a position encoding rule based on trigonometric functions is added. The output of the decoder is the center point of a set of polygons as the query of the polygon.

[0018] The two-dimensional reference points related to the polygons in the top view are back-projected onto the corresponding positions on the image plane according to the camera's intrinsic and extrinsic parameters, and their corresponding positions in the original image and adjacent feature maps are found.

[0019] Features are sampled from the feature map using bilinear interpolation and introduced into the polygon query. A multi-head attention mechanism is used to describe the interaction between the center point of the polygon and the corresponding position in the feature map. Then, these feature mapping parts are used to refine the query through a successive transformer layer.

[0020] For queries that are corrected after the transformer output, two multilayer perceptron branches are introduced to output category information and polygon vertex information, respectively. A fitted fixed multi-vertex polygon is used to fit complex semantic region features. The polygons in the prediction set are compared with the polygons in the ground value set to form a loss function and the gradient is backpropagated.

[0021] Furthermore, the stitching of multi-view cameras includes: for queries involving multiple cameras, it is only necessary to inversely map the reference points of the bird's-eye view onto the feature map of each camera, use the feature maps of several cameras simultaneously in the query refinement process, and then input them into the encoding side of the transformer; if the reference points of the bird's-eye view are not in the image plane of some cameras after inverse phase projection of intrinsic and extrinsic parameters, then the adjustment of the feature map corresponding to this camera for the query is ignored.

[0022] Secondly, the technical solution adopted by the present invention is as follows: a system for semantic segmentation of bird's-eye view based on implicit ensemble prediction, comprising: a preprocessing module for preprocessing the labeled data in a high-precision map and converting the semantic layer of the high-precision map into a labeled polygon layer; a prediction head module for extracting semantic features from the labeled polygon layer, performing prediction head analysis on the semantic features, realizing the stitching and fusion of multi-view cameras, so as to predict the final polygon segmentation box by implicit ensemble prediction; and a transformation module for transforming the viewpoint of the image plane and the top view to complete the implicit prediction based on the transformer.

[0023] Thirdly, the technical solution adopted by the present invention is: a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0024] Fourthly, the technical solution adopted by the present invention is: a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0025] The present invention has the following advantages due to the adoption of the above technical solutions:

[0026] 1. This invention models the transformation relationship and region of interest from image plane to bird's-eye view through implicit multi-head attention mechanism, without requiring pixel-by-pixel depth estimation or transformation correspondence.

[0027] 2. This invention directly uses the original semantic map polygon annotations for training, and only one regression is needed in the network, which greatly reduces the computational load of data preprocessing and postprocessing.

[0028] 3. This invention can share the backbone and neck network with the target detection network, but the prediction head is different, which enables the network to perceive both dynamic objects and static roads at the same time, reducing the computational load of the perception network. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the implicit set prediction process in an embodiment of the present invention;

[0030] Figure 2 This is a flowchart illustrating the network design and iteration method in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of perspective transformation and map element generation in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0033] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0034] To address the problem of semantic segmentation of bird's-eye view images in existing technologies, this invention provides an improved method and system for semantic segmentation of bird's-eye view images based on multi-camera implicit ensemble prediction. Using images from vehicle-mounted multi-camera systems as the information source, a novel perspective conversion module is implemented to directly transfer image depth features to polygonal semantic regions under the bird's-eye view. This is then combined with polygon label regression from the original semantic map to form a polygonal ensemble prediction problem. A two-level loss function of matching and intersection-union ratio is designed, compatible with the backbone network for object detection, and different shared prediction heads are designed to simultaneously perceive dynamic objects and static roads, forming a multi-camera single-network dynamic and static perception system.

[0035] In one embodiment of the present invention, a method for semantic segmentation of bird's-eye view images based on implicit set prediction is provided. In this embodiment, as... Figure 1 As shown, the method includes the following steps:

[0036] 1) Preprocess the labeled data in the high-precision map to convert the semantic layer of the high-precision map into a labeled polygon layer;

[0037] 2) Extract semantic features from the labeled polygon layers, perform predictive head analysis on the semantic features, realize the stitching and fusion of multi-view cameras, and predict the final polygon segmentation box by implicit set.

[0038] 3) Transform the perspective of the image plane and the top view to complete the implicit prediction based on the transformer.

[0039] In step 1) above, the labeled data in the high-precision map is preprocessed, including dynamic target preprocessing and static road element preprocessing; wherein:

[0040] Dynamic target preprocessing includes: squeezing the target from the height layer to form a ground occupancy mask, forming a polygon on the ground plane, and assigning corresponding semantic labels to the mask;

[0041] Specifically, dynamic targets are generally labeled objects in datasets primarily used for object detection tasks. These typically include over ten subcategories of targets, such as people and vehicles. The original labels in the dataset are usually 3D bounding boxes, whose attributes include length, width, height, 3D position (x, y, z), yaw angle, and semantic attributes. In this embodiment, the data processing of the bird's-eye view mainly involves compressing the targets from the height layer to form a ground occupancy mask, creating polygons (usually quadrilaterals) on the ground plane, and assigning corresponding semantic labels to the mask.

[0042] Static road feature preprocessing includes: it does not need to be completely covered by a high-precision map when the vehicle is running; only a portion of the map data is needed as training labels during training; and it relies on semantic map training to enable effective semantic segmentation results to be generalized to any structured road.

[0043] In the above embodiments, the static road element preprocessing further includes: allowing for a pre-defined error in the linear map information, and assigning a pre-defined width to the polylines to expand them into long and narrow polygons for training. Preferably, the pre-defined width can be 0.3 meters.

[0044] Specifically, static road element preprocessing is generally provided by a high-precision map of the area. Note that this invention does not require complete high-precision map coverage during vehicle operation; only partial map data is needed as training labels during training. Relying on semantic map training, it can generalize to effective semantic segmentation on any structured road during testing. Road elements consist of area information, represented by polygons, including drivable areas, sidewalks, crosswalks, lanes, parking areas, and road blocks; and line information, expressed as multi-segment polylines, including road dividing lines, lane dividing lines, and stop lines. In this invention, considering the inherent width of line map information in the real world and allowing for small errors, a small width (0.3 meters) is assigned to polylines, allowing them to expand into long, narrow polygons for training. In summary, this preprocessing method allows commonly used high-precision map semantic layers to use their labeled polygon layers without needing to be converted into dense semantic coverage information.

[0045] In step 2) above, semantic feature extraction specifically involves: Image semantic feature representation learning is an ongoing research problem, generally divided into backbone networks and neck networks. Backbone networks can be designed using convolutional neural networks such as VGG, ResNet, and VovNet, or transformer-based methods like the Swing Transformer. The final feature map has dimensions of (width, height, embedding dimension), with the embedding dimension typically set to 256 to maximize GPU performance. Neck networks generally use multi-layer feature pyramids (FPN) or their improved versions. The number of feature channels is typically distributed across four layers (256, 512, 1024, 2048). The introduction of feature pyramids allows the network to simultaneously focus on large and small targets, achieving a balance in attention across different scales.

[0046] In step 2) above, the semantic features are analyzed by the prediction head. Specifically, the features of the image obtained by the preceding network are regressed into the final required physical quantity through the prediction head, and the loss function is obtained by comparing it with the ground truth label and then back-propagating the loss function.

[0047] In this embodiment, the prediction head adopts the single-stage, anchor-free FCOS3D model. It classifies each pixel and obtains a two-dimensional polygon through a centering operation, and then directly guides the generation of the three-dimensional polygon region through the two-dimensional polygon.

[0048] The transmission loss function consists of three parts: two-sided loss, classification loss, and crossover ratio loss.

[0049] Specifically, the design of the loss function is a crucial factor in ensuring the convergence of neural network regression. In this invention, the loss function consists of three parts:

[0050] The first part is the two-sided loss. Since both the predicted results and the ground truth labels are semantically meaningful polygons in a top-down view, based on the ensemble prediction theory of the transformer, the Hungarian algorithm is first used to calculate the bipartite matching between the predictions and the ground truth. The bipartite matching process is to obtain the minimum value of the following terms:

[0051]

[0052] In the formula, σ represents the permutation of true values ​​matching predicted values, P represents the set of all possible permutations, M represents the number of elements representing the prediction set, and c j Elements representing predictions, This indicates that the predicted result is an empty set of elements. This indicates that the j-th prediction σ(j) belongs to category c. j The probability, L polygon_iou b represents the intersection-union ratio of the predicted and ground truth polygons. j Represents a true polygon. This represents a polygon representing predicted values.

[0053] Each true value is forced to match a predicted value, and the bilateral loss is calculated based on the weights of the Hungarian matching. The set-to-set loss function is calculated using the following formula:

[0054]

[0055] In the formula, Loss represents the loss function of geometric prediction, and N represents the number of true and predicted values ​​that are matched.

[0056] The second part is the classification loss. Generally, the content to be segmented is divided into several semantic categories according to the needs of local map construction, such as semantic labels for lanes, sidewalks, and crosswalks. Then, the biased focal cross-entropy loss function is used to calculate it.

[0057] The third part is the Cross-Union Ratio (CUI) loss, which characterizes the similarity between the predicted polygon and the ground truth polygon of a single match. The formula for calculating the two-dimensional CUI is:

[0058] IoU=(A∩B) / (A∪B).

[0059] In the formula, IoU represents the intersection-union ratio, A represents the area covered by graph A, B represents the area covered by graph B, A∩B represents the area simultaneously within both graphs A and B, and A∪B represents the total area covered by either A or B.

[0060] The intersection-union ratio (IoU) ranges from [0,1]. Predicted boxes with higher overlap are more reasonable. IoU loss regresses the four boundary information of the candidate boxes as a whole, achieving accurate and efficient localization with good scale invariance. To address the issue of non-differentiability of the IoU metric, a negative Ln norm is introduced to indirectly calculate the IoU loss.

[0061] In step 3) above, the perspective of the image plane is converted to that of the top view, such as... Figure 2 , Figure 3 As shown, it includes the following steps:

[0062] In this embodiment, a transformer is used for implicit mapping in viewpoint transformation. The input of the transformer is a dimension-reduced feature map, and the output is the center point information of the polygons in the bird's-eye view.

[0063] Because cameras lack explicit depth information, the 2D to 3D viewpoint conversion presents significant challenges, especially since the estimation of the height direction heavily relies on implicit regression in neural networks. Since there is no direct perceptual evidence, considering only the top-down view eliminates the uncertainty in the height direction. The conversion between the image plane and the top-down view can be achieved through inverse projection transformation, but the accuracy of this method is highly dependent on intrinsic parameters. The distortion of a real camera is difficult to fully express with a simple intrinsic parameter model. Therefore, an implicit estimation method is needed, requiring only a less precise intrinsic parameter for approximate conversion. In this invention, a transformer is used to implement the implicit mapping process in viewpoint conversion.

[0064] The iterative process of predicting polygons from a single camera's 2D feature map to a bird's-eye view, with backpropagation of the loss function, is carried out in a loop through the following four steps: 3.1) to 3.4).

[0065] 3.1) The feature map of the image is compressed from (batch size, height, width, number of channels) to (batch size, height multiplied by width, number of channels), input into the encoder unit of the transformer, and the transformer general position encoding rule based on trigonometric functions is added. The output of the decoder is the center point of a set of polygons as the query of the polygon.

[0066] The query generates two-dimensional reference points in the top view through a simple convolutional network. Note that the number of reference points is proportional to both the number of transformer layers and the number of queries.

[0067] 3.2) Inversely project the two-dimensional reference points related to the polygons in the top view onto the corresponding positions on the image plane according to the camera's intrinsic and extrinsic parameters, and find their corresponding positions in the original image and adjacent feature maps.

[0068] 3.3) Features of the feature map are sampled using bilinear interpolation and introduced into the polygon query. A multi-head attention mechanism is used to describe the interaction between the center point of the polygon and the corresponding position of the feature map. Then, these feature mapping parts are used to refine the query through the transform layer.

[0069] The multi-head attention module adopts the implementation of the standard multi-head attention module.

[0070] 3.4) For queries that are corrected after the output of the transformer, two multilayer perceptron branches are introduced to output category information and polygon vertex information respectively. The fitted fixed multi-vertex polygons are used to fit complex semantic region elements. The polygons of the prediction set are compared with the polygons of the ground value set to form a loss function and the gradient is backpropagated.

[0071] In the above embodiments, the stitching of multi-view cameras is specifically as follows: the viewpoint conversion method based on the implicit transformer attention mechanism does not require explicit reconstruction of the 3D scene. Therefore, for queries involving multiple cameras, it is only necessary to inversely map the reference points of the bird's-eye view onto the feature map of each camera, and use the feature maps of several cameras simultaneously in the query refinement process, and then input them into the encoding side of the transformer. If the reference points of the bird's-eye view are not in the image plane of some cameras after inverse phase projection of intrinsic and extrinsic parameters, the adjustment of the feature map corresponding to this camera for the query is ignored.

[0072] In real-world vehicles, such as those used in the nuScenes dataset or Baidu Apollo vehicles, surround-view vision is achieved by six cameras working simultaneously. After inverse projection, this results in M×L×NUM_CAM center points. A compromise considering network performance and efficiency is to use a hidden layer dimension M=900, a transformer hidden layer number L=6, and a camera number NUM_CAM=6.

[0073] In one embodiment of the present invention, a system for semantic segmentation of bird's-eye view images based on implicit set prediction is provided, comprising:

[0074] The preprocessing module preprocesses the labeled data in the high-precision map, converting the semantic layer of the high-precision map into a labeled polygon layer;

[0075] The prediction head module extracts semantic features from the labeled polygon layers, performs prediction head analysis on the semantic features, and realizes the stitching and fusion of multi-view cameras to predict the final polygon segmentation box by implicit set.

[0076] The transformation module converts the perspective of the image plane to that of the top view, thus completing implicit prediction based on the transformer.

[0077] The above preprocessing module preprocesses the labeled data in the high-precision map, including dynamic target preprocessing and static road element preprocessing.

[0078] Dynamic target preprocessing includes: squeezing the target from the height layer to form a ground occupancy mask, forming a polygon on the ground plane, and assigning corresponding semantic labels to the mask;

[0079] Static road feature preprocessing includes: it does not need to be completely covered by a high-precision map when the vehicle is running; only a portion of the map data is needed as training labels during training; and it relies on semantic map training to enable effective semantic segmentation results to be generalized to any structured road.

[0080] In this embodiment, the static road element preprocessing further includes: allowing for a pre-defined error in the linear map information, and assigning a pre-defined width to the polyline so that the polyline is expanded into a long and narrow polygon for training.

[0081] The prediction head module described above performs prediction head analysis on semantic features, including:

[0082] The prediction head regresses the image features obtained from the pre-processor network into the final required physical quantity, compares them with the ground truth labels to obtain the loss function, and then backpropagates the loss function.

[0083] The transmission loss function consists of three parts: two-sided loss, classification loss, and crossover ratio loss.

[0084] In the aforementioned prediction head module, the prediction head adopts a single-stage, anchor-free approach from FCOS3D. It classifies each pixel and obtains a two-dimensional polygon through a centering operation, which then directly guides the generation of the three-dimensional polygon region.

[0085] The conversion module described above converts the perspective between the image plane and the top view, including:

[0086] An implicit mapping in the perspective transformation is performed using a transformer. The input of the transformer is a dimension-reduced feature map, and the output is the center point information of the polygons in the bird's-eye view.

[0087] The feature map of the image is compressed from (batch size, height, width, number of channels) to (batch size, height multiplied by width, number of channels), input into the encoder unit of the transformer, and a position encoding rule based on trigonometric functions is added. The output of the decoder is the center point of a set of polygons as the query of the polygon.

[0088] The two-dimensional reference points related to the polygons in the top view are back-projected onto the corresponding positions on the image plane according to the camera's intrinsic and extrinsic parameters, and their corresponding positions in the original image and adjacent feature maps are found.

[0089] Features are sampled from the feature map using bilinear interpolation and introduced into the polygon query. A multi-head attention mechanism is used to describe the interaction between the center point of the polygon and the corresponding position in the feature map. Then, these feature mapping parts are used to refine the query through a successive transformer layer.

[0090] For queries that are corrected after the transformer output, two multilayer perceptron branches are introduced to output category information and polygon vertex information, respectively. A fitted fixed multi-vertex polygon is used to fit complex semantic region features. The polygons in the prediction set are compared with the polygons in the ground value set to form a loss function and the gradient is backpropagated.

[0091] In the above embodiments, the stitching of multi-view cameras is specifically as follows: for queries involving multiple cameras, it is only necessary to inversely map the reference points of the bird's-eye view onto the feature map of each camera, use the feature maps of several cameras simultaneously in the query refinement process, and then input them to the encoding side of the transformer; if the reference points of the bird's-eye view are not in the image plane of some cameras after inverse phase projection of intrinsic and extrinsic parameters, then the adjustment of the feature map corresponding to this camera for the query is ignored.

[0092] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0093] In one embodiment of the present invention, a computing device structure is provided. This computing device can be a terminal, which may include: a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. When executed by the processor, the computer program implements a method for semantic segmentation of bird's-eye view based on implicit set prediction. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.

[0094] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.

[0096] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.

[0097] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0098] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for semantic segmentation of bird's-eye view images based on implicit set prediction, characterized in that, include: Preprocess the labeled data in the high-precision map to convert the semantic layer of the high-precision map into a labeled polygon layer; Semantic features are extracted from the labeled polygon layers, and predictive head analysis is performed on the semantic features to realize the stitching and fusion of multi-view cameras, so as to predict the final polygon segmentation box by implicit set. Transform the perspective of the image plane and the top view to complete the implicit prediction based on the transformer; Among them, the prediction head analysis of semantic features is: the prediction head regresses the image features obtained by the preorder network into the final required physical quantity, compares them with the ground truth label to obtain the loss function, and then backpropagates the loss function; Converting the perspective between the image plane and the top view includes: An implicit mapping in the perspective transformation is performed using a transformer. The input of the transformer is a dimension-reduced feature map, and the output is the center point information of the polygons in the bird's-eye view. The feature map of the image is compressed from (batch size, height, width, number of channels) to (batch size, height multiplied by width, number of channels), input into the encoder unit of the transformer, and a position encoding rule based on trigonometric functions is added. The output of the decoder is the center point of a set of polygons as the query of the polygon. The two-dimensional reference points related to the polygons in the top view are back-projected onto the corresponding positions on the image plane according to the camera's intrinsic and extrinsic parameters, and their corresponding positions in the original image and adjacent feature maps are found. Features are sampled from the feature map using bilinear interpolation and introduced into the polygon query. A multi-head attention mechanism is used to describe the interaction between the center point of the polygon and the corresponding position in the feature map. Then, these feature mapping parts are used to refine the query through a successive transformer layer. For queries that are corrected after the transformer output, two multilayer perceptron branches are introduced to output category information and polygon vertex information, respectively. A fitted fixed multi-vertex polygon is used to fit complex semantic region features. The polygons in the prediction set are compared with the polygons in the ground value set to form a loss function and the gradient is backpropagated.

2. The method for semantic segmentation of bird's-eye view based on implicit set prediction as described in claim 1, characterized in that, Preprocessing of labeled data in high-precision maps includes dynamic target preprocessing and static road element preprocessing; Dynamic target preprocessing includes: squeezing the target from the height layer to form a ground occupancy mask, forming a polygon on the ground plane, and assigning corresponding semantic labels to the mask; Static road feature preprocessing includes: it does not need to be completely covered by a high-precision map when the vehicle is running; only a portion of the map data is needed as training labels during training; and it relies on semantic map training to enable effective semantic segmentation results to be generalized to any structured road.

3. The method for semantic segmentation of bird's-eye view based on implicit set prediction as described in claim 2, characterized in that, Static road feature preprocessing also includes: allowing for pre-defined errors in linear map information, and assigning a pre-defined width to polylines so that they can be expanded into long and narrow polygons for training.

4. The method for semantic segmentation of bird's-eye view based on implicit set prediction as described in claim 1, characterized in that, The backpropagation loss function consists of three parts: bilateral loss, classification loss, and crossover ratio loss.

5. The method for semantic segmentation of bird's-eye view based on implicit set prediction as described in claim 1, characterized in that, The prediction head is a single-stage, anchorless FCOS3D model that classifies each pixel and obtains a 2D polygon through a centering operation. The 2D polygon is then used to directly guide the generation of the 3D polygon region.

6. The method for semantic segmentation of bird's-eye view based on implicit set prediction as described in claim 1, characterized in that, Multi-view camera stitching includes: for queries involving multiple cameras, it is only necessary to inversely map the reference points of the bird's-eye view onto the feature map of each camera, use the feature maps of several cameras simultaneously in the query refinement process, and then input them into the encoding side of the transformer; if the reference points of the bird's-eye view are not in the image plane of some cameras after inverse phase projection of intrinsic and extrinsic parameters, then the adjustment of the feature map corresponding to this camera for the query is ignored.

7. A system for semantic segmentation of bird's-eye view images based on implicit set prediction, used to implement the method for semantic segmentation of bird's-eye view images based on implicit set prediction as described in any one of claims 1 to 6, characterized in that, include: The preprocessing module preprocesses the labeled data in the high-precision map, converting the semantic layer of the high-precision map into a labeled polygon layer; The prediction head module extracts semantic features from the labeled polygon layers, performs prediction head analysis on the semantic features, and realizes the stitching and fusion of multi-view cameras to predict the final polygon segmentation box by implicit set. The transformation module converts the perspective of the image plane to that of the top view, thus completing implicit prediction based on the transformer.

8. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in claims 1 to 6.

9. A computing device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described in claims 1 to 6.

Images