Road element detection result optimization method, system and device and storage medium

Abstract

The embodiment of the invention provides a road element detection result optimization method, system and device and a storage medium, and the method comprises the steps: obtaining a road element detection result under a BEV aerial view space based on a multi-modal road element detection model, the road element detection result comprising prediction coordinates of broken line segment nodes of elements; obtaining projection points of the predicted coordinates of the broken line segment nodes based on the predicted coordinates of the broken line segment nodes; and performing iterative optimization on the projection point based on the trained detection result optimization module to obtain an optimized target node coordinate.

Description

Technical Field

[0001] This specification relates to the field of road perception technology, and in particular to a method, system, device and storage medium for optimizing road element detection results. Background Art

[0002] The development of artificial intelligence and autonomous driving technologies has had a significant impact on the automotive industry, social development, and mobility systems. Autonomous driving requires extracting road elements to perceive the vehicle's surroundings. Rapid and accurate extraction of road elements facilitates accurate driving control.

[0003] Road elements can be detected and extracted online using on-board sensors and supporting on-board computing equipment, and corresponding vectorized representations can be obtained. Due to the camera's ability to capture color and texture details and the lidar's ability to capture three-dimensional structural information of the scene, these two sensors often complement each other. The feature information extracted from these two modal data is converted to a consistent BEV space for fusion to achieve multimodal road element detection. However, the conversion process from the perspective viewpoint to the BEV space of the image acquired by the camera introduces distortion, and the calibration parameters between the camera, lidar, and vehicle body may be inaccurate or change with the movement of the vehicle body, which exacerbates the distortion of the converted features and makes it difficult to obtain accurate and stable detection results.

[0004] Therefore, it is necessary to provide a road element detection result optimization method, system, device and storage medium for obtaining accurate and stable road element detection results. Summary of the Invention

[0005] One or more embodiments of the present specification provide a method for optimizing road element detection results, the method comprising: obtaining road element detection results in a BEV bird's-eye view space based on a multimodal road element detection model, the road element detection results including predicted coordinates of broken line segment nodes of the elements; obtaining projection points of the predicted coordinates of the broken line segment nodes based on the predicted coordinates of the broken line segment nodes; and iteratively optimizing the projection points based on a trained detection result optimization module to obtain optimized target node coordinates.

[0006] In some embodiments, obtaining the projection point of the predicted coordinates of the broken line segment node based on the predicted coordinates of the broken line segment node includes: projecting the predicted coordinates of the broken line segment node onto a fused perspective view feature map to obtain the projection point of the predicted coordinates of the broken line segment node, and the fused perspective view feature map is obtained based on the lidar point cloud and the perspective view camera image feature map.

[0007] In some embodiments, the trained detection result optimization module iteratively optimizes the projection point to obtain the optimized target node coordinates, including: determining the sampling features of the projection point based on the trained detection result optimization module; performing a regression operation on the sampling features to determine the correction amount of the broken line segment node; and obtaining the optimized target node coordinates based on the predicted coordinates of the broken line segment node and the correction amount.

[0008] In some embodiments, the loss function during the training of the detection result optimization module is determined based on the positional relationship between the broken line segment node and a preset number of other nodes before and after it.

[0009] In some embodiments, the method further includes: based on an initial extrinsic parameter matrix, projecting the point-by-point features of the lidar point cloud onto the perspective camera image feature map to obtain a pseudo image feature map; processing the perspective camera image feature map and the pseudo image feature map through a supplementary transformation model to determine a supplementary transformation matrix; based on the supplementary transformation matrix and the initial extrinsic parameter matrix, determining a corrected extrinsic parameter matrix, the corrected extrinsic parameter matrix being used to project the predicted coordinates of the broken line segment nodes.

[0010] In some embodiments, the fused perspective view feature map is obtained based on the lidar point cloud and the perspective view camera image feature map, including: projecting the lidar point cloud onto the perspective view camera image feature map to obtain the three-dimensional coordinates corresponding to each pixel on the perspective view camera image feature map; splicing the three-dimensional coordinates corresponding to each pixel onto the perspective view camera image feature map to obtain the fused perspective view feature map.

[0011] One or more embodiments of the present specification provide a road element detection result optimization system, the system comprising: a prediction module for obtaining road element detection results in a BEV bird's-eye view space based on a multimodal road element detection model, the road element detection results including predicted coordinates of the element's broken line segment nodes; a projection module for obtaining projection points of the predicted coordinates of the broken line segment nodes based on the predicted coordinates of the broken line segment nodes; and an optimization module for iteratively optimizing the projection points based on the trained detection result optimization module to obtain optimized target node coordinates.

[0012] In some embodiments, the optimization module is further used to: determine the sampling features of the projection point based on the trained detection result optimization module; perform regression operation on the sampling features to determine the correction amount of the broken line segment node; and obtain the optimized target node coordinates based on the predicted coordinates of the broken line segment node and the correction amount.

[0013] One or more embodiments of this specification provide a road element detection result optimization device, which includes at least one processing device and at least one memory; the at least one memory is used to store computer instructions; and the at least one processing device is used to execute at least part of the computer instructions to implement the road element detection result optimization method described in any of the above embodiments.

[0014] One or more embodiments of this specification provide a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the road element detection result optimization method described in any one of the above embodiments.

[0015] According to the above embodiments, accurate and stable road element detection results can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] This specification will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same numbers represent the same structures, wherein:

[0017] Figure 1 is a schematic diagram of an application scenario of a road element detection result optimization system according to some embodiments of this specification;

[0018] Figure 2 is an exemplary module diagram of a road element detection result optimization system according to some embodiments of this specification;

[0019] Figure 3 is an exemplary flow chart of a method for optimizing road element detection results according to some embodiments of this specification;

[0020] Figure 4 is an exemplary flow chart for determining optimized target node coordinates according to some embodiments of this specification;

[0021] Figure 5 This is an exemplary flowchart for determining a modified extrinsic parameter matrix according to some embodiments of this specification. DETAILED DESCRIPTION

[0022] To more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly describes the drawings required for describing the embodiments. Obviously, the drawings described below are merely examples or embodiments of this specification. Those skilled in the art can apply this specification to other similar scenarios based on these drawings without inventive effort. Unless otherwise apparent from the context or otherwise noted, the same reference numerals in the figures represent the same structure or operation.

[0023] It should be understood that the terms "system," "device," "unit," and / or "module" used herein are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0024] As used in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not refer to the singular but also include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0025] Flowcharts are used throughout this specification to illustrate the operations performed by systems according to embodiments of this specification. It should be understood that preceding or following operations do not necessarily need to be performed in exact order. Instead, the steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0026] Figure 1 This is a schematic diagram of an application scenario of the road element detection result optimization system shown in some embodiments of this specification. The systems and methods in some embodiments of this specification can be applied to vehicle movement control scenarios, such as autonomous driving scenarios, pure unmanned driving scenarios without a driver, and auxiliary driving modules in vehicles. The auxiliary driving module in the vehicle can autonomously control the vehicle's movement and can also output driving prompts to the driver. Figure 1 As shown, the application scenario 100 of the road element detection result optimization system may include a processing device 110 , a vehicle 120 , a terminal device 130 , a storage device 140 , a network 150 , a lidar device 160 and an image acquisition device 170 .

[0027] The processing device 110 can be used to process data and / or information from at least one component of the application scenario 100 of the road element detection result optimization system or an external data source (e.g., a cloud data center). For example, the processing device 110 can obtain road element detection results in a BEV bird's-eye view space based on a multimodal road element detection model. For another example, the processing device 110 can obtain the projection points of the predicted coordinates of the broken line segment nodes based on the predicted coordinates of the broken line segment nodes. For another example, the processing device 110 can iteratively optimize the projection points based on the trained detection result optimization module to obtain optimized target node coordinates.

[0028] In some embodiments, the processing device 110 may include a central processing unit (CPU), a digital signal processing device (DSP), a system on a chip (SoC), a microcontroller unit (MCU), a computer, a user console, or the like, or any combination thereof. In some embodiments, the processing device 110 may include a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processing device 110 may be local or remote. In some embodiments, the processing device 110 may be implemented on a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, or the like, or any combination thereof. In some embodiments, the processing device 110 may be integrated into the vehicle 120.

[0029] The vehicle 120 may be a service vehicle for the application scenario 100 of the road element detection result optimization system. In some embodiments, the vehicle 120 may include a passenger car, an autonomous vehicle, and the like.

[0030] Terminal device 130 may be a terminal device for displaying road element detection results. In some embodiments, terminal device 130 may be installed on vehicle 120 or be a component of vehicle 120. In some embodiments, terminal device 130 may include a mobile device, a tablet computer, a laptop computer, other devices with input and / or output functions, or any combination thereof.

[0031] The storage device 140 can be used to store data, instructions, and / or any other information. For example, the storage device can store lidar point clouds and perspective camera images. In some embodiments, the storage device may include random access memory (RAM), read-only memory (ROM), mass storage, removable storage, volatile read-write memory, or any combination thereof. In some embodiments, the storage device may be integrated with or included in one or more other components of the road element detection result optimization system application scenario 100 (e.g., processing device 110, vehicle 120, terminal device 130).

[0032] Network 150 can facilitate the exchange of information and / or data. In some embodiments, one or more components of the application scenario 100 of the road element detection result optimization system (e.g., processing device 110, vehicle 120, terminal device 130, storage device 140, lidar device 160, image acquisition device 170) can transmit information and / or data to other components of the application scenario 100 of the road element detection result optimization system via network 150. In some embodiments, network 150 can include any one or more of a wired network or a wireless network. In some embodiments, network 150 can include a cable network, a fiber optic network, a telecommunications network, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, near-field communication (NFC), an intra-device bus, an intra-device line, a cable connection, or any combination thereof. In some embodiments, the network connections between the components of the application scenario 100 of the road element detection result optimization system can adopt one of the aforementioned methods, or multiple methods. In some embodiments, the network may be of various topologies such as point-to-point, shared, centralized, etc., or a combination of multiple topologies.

[0033] The laser radar device 160 may be a device for acquiring a laser radar point cloud. For example, the laser radar device 160 may include a solid-state laser radar, a gas laser radar, a semiconductor laser radar, etc. In some embodiments, the laser radar device 160 may be installed on the vehicle 120.

[0034] Image acquisition device 170 may be a device for acquiring perspective camera images. For example, image acquisition device 170 may include a long-range camera, a medium-range camera, a short-range camera, etc. In some embodiments, image acquisition device 170 may be installed on vehicle 120. In some embodiments, vehicle 120 may be equipped with multiple image acquisition devices 170 of the same or different types to acquire perspective camera images at different angles and / or viewing distances.

[0035] The systems and methods in some embodiments of this specification can also be used to construct high-precision maps, update local areas of high-definition maps, match and locate road elements around vehicles, etc.

[0036] It should be noted that the application scenario 100 of the road element detection result optimization system is provided for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art will appreciate that various modifications and variations can be made based on the description of this specification. For example, the application scenario 100 of the road element detection result optimization system may also include a database, information source, and the like. For another example, the application scenario 100 of the road element detection result optimization system may be implemented on other devices to achieve similar or different functionality. However, such variations and modifications do not deviate from the scope of this specification.

[0037] Figure 2 2 is an exemplary module diagram of a road element detection result optimization system according to some embodiments of this specification. In some embodiments, the road element detection result optimization system 200 may include a prediction module 210 , a projection module 220 , and an optimization module 230 .

[0038] The prediction module 210 may be configured to obtain road element detection results in the BEV bird's-eye view space based on a multimodal road element detection model. The road element detection results include predicted coordinates of the polyline segment nodes of the elements.

[0039] The projection module 220 may be configured to obtain a projection point of the predicted coordinates of the polyline segment node based on the predicted coordinates of the polyline segment node.

[0040] In some embodiments, the projection module 220 is further used to project the predicted coordinates of the broken line segment node onto the fused perspective view feature map to obtain the projection points of the predicted coordinates of the broken line segment node, and the fused perspective view feature map is obtained based on the lidar point cloud and the perspective view camera image.

[0041] In some embodiments, the projection module 220 is also used to project the point-by-point features of the lidar point cloud onto the perspective camera image based on the initial extrinsic parameter matrix to obtain a pseudo-image feature map; the perspective camera image and the pseudo-image feature map are processed by a supplementary transformation model to determine the supplementary transformation matrix; based on the supplementary transformation matrix and the initial extrinsic parameter matrix, a corrected extrinsic parameter matrix is ​​determined, and the corrected extrinsic parameter matrix is ​​used to project the predicted coordinates of the broken line segment nodes.

[0042] In some embodiments, the projection module 220 is further used to project the lidar point cloud onto the perspective camera image to obtain the three-dimensional coordinates corresponding to each pixel on the perspective camera image; and splice the three-dimensional coordinates corresponding to each pixel onto the perspective camera image to obtain a fused perspective feature map.

[0043] The optimization module 230 can be used to iteratively optimize the projection points based on the trained detection result optimization module to obtain optimized target node coordinates.

[0044] In some embodiments, the optimization module 230 is further used to determine the sampling features of the projection point based on the trained detection result optimization module; perform regression operations on the sampling features to determine the correction amount of the broken line segment node; and obtain the optimized target node coordinates based on the predicted coordinates and correction amount of the broken line segment node.

[0045] For more information about the prediction module, see Figure 3 For more information about the projection module, see Figure 3 and Figure 5 For more information about the optimization module, see Figure 3 and Figure 4 and its related descriptions.

[0046] It should be noted that the above description of the road element detection result optimization system 200 and its modules is for convenience only and does not limit this specification to the scope of the embodiments. It is understood that after understanding the principles of the system, those skilled in the art may arbitrarily combine the modules or form subsystems connected with other modules without departing from the principles. In some embodiments, Figure 2 The prediction module 210, projection module 220, and optimization module 230 disclosed herein may be different modules within a system, or a single module may implement the functions of two or more of the aforementioned modules. For example, the modules may share a storage module, or each module may have its own storage module. Such variations are within the scope of protection of this specification.

[0047] Figure 3 is an exemplary flow chart of a road element detection result optimization method according to some embodiments of this specification. In some embodiments, process 300 can be executed by the processing device 110 or the road element detection result optimization system 200. For example, process 300 can be stored in a storage device in the form of a program or instruction, and when the processing device 110 or the road element detection result optimization system 200 executes the instruction, process 300 can be implemented. The operational diagram of process 300 presented below is illustrative. In some embodiments, the process can be completed using one or more additional operations not described and / or one or more operations not discussed. In addition, Figure 3 The order in which the operations of flow 300 are illustrated and described below is not intended to be limiting.

[0048] Step 310 : Acquire road element detection results in the BEV bird's-eye view space based on the multimodal road element detection model.

[0049] The multimodal road element detection model may be a machine learning model for determining road element detection results. For example, the multimodal road element detection model may include a graph neural network (GNN) model.

[0050] In some embodiments, the input of the multimodal road element detection model may include a lidar point cloud, a perspective camera image, and a corrected extrinsic parameter matrix; and the output may include a road element detection result.

[0051] A LiDAR point cloud is data related to a point in space acquired through scanning by a LiDAR device. For example, a LiDAR point cloud can include the 3D coordinates of the point in space, the laser reflection intensity, and more.

[0052] In some embodiments, the processing device can scan the vehicle's driving road using a laser radar device to obtain a laser radar point cloud. For more information about laser radar devices, see Figure 1 and its related descriptions.

[0053] The perspective camera image refers to an image related to the road obtained by scanning by an image acquisition device.

[0054] In some embodiments, the processing device can scan the road the vehicle is traveling on through an image acquisition device to obtain a perspective camera image. For more information about the image acquisition device, see Figure 1 and its related descriptions.

[0055] The corrected extrinsic matrix is ​​the final matrix used to convert the lidar point cloud and the perspective camera image. For more information about the corrected extrinsic matrix, see Figure 5 and its related descriptions.

[0056] Road element detection results refer to data related to road elements. For example, the road element detection results may include the type of road element (e.g., lane markings, stop signs, crosswalks, drivable area boundaries, etc.), length, etc. In some embodiments, the road element detection results also include the predicted coordinates of the element's polyline segment nodes.

[0057] In some embodiments, a multimodal road element detection model can be trained using multiple first training samples with first labels. For example, multiple first training samples with first labels can be input into an initial multimodal road element detection model. A loss function is constructed using the first labels and the results of the initial multimodal road element detection model. The parameters of the initial multimodal road element detection model are iteratively updated based on the loss function. Model training is completed when the loss function of the initial multimodal road element detection model meets pre-set iteration conditions, resulting in a trained multimodal road element detection model. Pre-set iteration conditions may include convergence of the loss function or a threshold number of iterations.

[0058] In some embodiments, the first training samples may include a sample LiDAR point cloud, a sample perspective camera image, and a sample corrected extrinsic parameter matrix of a sample vehicle; the first labels may include sample road element detection results corresponding to the first training samples. In some embodiments, the first training samples may be obtained based on historical data, and the first labels may be obtained through manual annotation.

[0059] Step 320: Obtain the projection point of the predicted coordinates of the broken line segment node based on the predicted coordinates of the broken line segment node.

[0060] The projection point of the predicted coordinates of the broken line segment node refers to the projection point of the broken line segment node on the two-dimensional image.

[0061] In some embodiments, the processing device may multiply the corrected extrinsic parameter matrix by the predicted coordinates of the broken line segment node to obtain the projection point of the predicted coordinates of the broken line segment node.

[0062] In some embodiments, the processing device may project the predicted coordinates of the broken line segment node onto the fused perspective view feature map to obtain the projection point of the predicted coordinates of the broken line segment node.

[0063] The fused perspective feature map refers to the image obtained by fusing the lidar point cloud and the perspective camera image.

[0064] In some embodiments, the processing device may obtain a fused perspective feature map based on the lidar point cloud and the perspective camera image.

[0065] In some embodiments, the processing device may project the lidar point cloud onto the perspective camera image to obtain the three-dimensional coordinates corresponding to each pixel in the perspective camera image; and splice the three-dimensional coordinates corresponding to each pixel onto the perspective camera image to obtain a fused perspective feature map. For example, the processing device may multiply the corrected extrinsic parameter matrix by the three-dimensional coordinates corresponding to the lidar point cloud to obtain the projection point of the lidar point cloud on the perspective camera image, use the three-dimensional coordinates of the projection point as the three-dimensional coordinates of the pixel corresponding to the projection point, and splice the three-dimensional coordinates corresponding to each pixel onto the perspective camera image to obtain a fused perspective feature map.

[0066] In some embodiments of this specification, a fused perspective feature map is obtained by using a lidar point cloud and a perspective camera image, which not only fully utilizes the features of the perspective camera image, but also obtains the three-dimensional coordinates corresponding to each pixel, which is convenient for subsequent determination of the optimized target node coordinates.

[0067] In some embodiments of this specification, by projecting the predicted coordinates of the broken line segment nodes onto the fused perspective feature map, the predicted coordinates and three-dimensional coordinates of each pixel point can be obtained simultaneously, which facilitates subsequent regression operations and determines the optimized target node coordinates.

[0068] Step 330 : Iteratively optimize the projection points based on the trained detection result optimization module to obtain optimized target node coordinates.

[0069] The target node coordinates refer to the three-dimensional coordinates of the finally determined polyline segment node.

[0070] The detection result optimization module may be a machine learning model for optimizing the target node coordinates. For example, the detection result optimization module may include a deep neural network (DNN) model.

[0071] In some embodiments, the input of the detection result optimization module may include the predicted coordinates of the broken line segment nodes and the fused perspective view feature map; the output may include the optimized target node coordinates.

[0072] In some embodiments, the detection result optimization module can be trained using a plurality of second training samples with second labels. The training method of the detection result optimization module can refer to the training method of the multimodal road element detection model.

[0073] In some embodiments, the detection result optimization module and the multimodal road element detection model can be obtained through joint training.

[0074] In some embodiments, the loss function during the training of the detection result optimization module can be determined based on the positional relationship between the broken line segment node and a preset number of other nodes before and after it. As an example only, the processing device can calculate the positional structural relationship between the broken line segment node and a preset number of other nodes before and after it, and supervise the current node through the preset number of other nodes before and after it. For example, the processing device can calculate the positional relationship between the center of gravity of the connecting line between the first node and the third node and the second node, and then supervise the position of the second node based on the positions of the first node and the third node, wherein the first node and the third node can be multiple nodes.

[0075] In some embodiments of the present specification, the loss function during the training of the detection result optimization module is determined based on the positional relationship between the broken line segment node and a preset number of other nodes before and after it. It not only considers the accuracy of the predicted coordinates of the broken line segment node, but also considers the structural positional relationship between the broken line segment node and the other surrounding nodes, so that the optimized target node coordinates finally outputted are more accurate.

[0076] In some embodiments, the second training samples may include sample predicted coordinates of sample polyline segment nodes; the second labels may include sample target node coordinates corresponding to the set of second training samples. In some embodiments, the second training samples may also include sample fused perspective view feature maps of the sample polyline segment nodes; the second labels may also include sample projection point coordinates corresponding to the set of second training samples. In some embodiments, the second training samples may be obtained from historical data, and the second labels may be obtained through manual annotation.

[0077] In some embodiments, the processing device can determine the sampling features of the projection point based on the trained detection result optimization module; perform regression operations on the sampling features to determine the correction amount of the broken line segment node; and obtain the optimized target node coordinates based on the predicted coordinates and correction amount of the broken line segment node. For more information about the above embodiments, please refer to Figure 4 and its related descriptions.

[0078] In some embodiments of the present specification, by obtaining road element detection results and iteratively optimizing the projection points of the predicted coordinates based on the trained detection result optimization module, the optimized target node coordinates are obtained, which fully utilizes the distortion-free characteristics of the perspective camera image, making the determined target node coordinates more accurate and enabling end-to-end direct generation of high-precision maps, thereby improving generation speed and flexibility.

[0079] It should be noted that the above description of process 300 is for illustration and purpose only and does not limit the scope of application of this specification. Those skilled in the art may make various modifications and alterations to process 300 under the guidance of this specification. However, such modifications and alterations are still within the scope of this specification.

[0080] Figure 4 This is an exemplary flow chart for determining the optimized target node coordinates according to some embodiments of this specification. In some embodiments, process 400 can be executed by the processing device 110 or the road element detection result optimization 200. For example, process 400 can be stored in a storage device in the form of a program or instruction, and when the processing device 110 or the road element detection result optimization 200 executes the instruction, process 400 can be implemented. The operational diagram of process 400 presented below is illustrative. In some embodiments, the process can be completed using one or more additional operations not described and / or one or more operations not discussed. In addition, Figure 4 The order in which the operations of flow 400 are illustrated and described below is not intended to be limiting.

[0081] Step 410: Optimize the module based on the trained detection results to determine the sampling features of the projection points.

[0082] Sampling features refer to features related to the position information of the projection points.

[0083] Step 420: Perform regression operation on the sampled features to determine the correction amount of the broken line segment node.

[0084] The correction amount refers to the parameter used to correct the predicted coordinates of the projection point.

[0085] In some embodiments, the processing device may perform regression processing on the sampled features based on a multi-layer perceptron to determine the correction amount of the broken line segment node.

[0086] Step 430 : Obtain optimized target node coordinates based on the predicted coordinates and correction values ​​of the broken line segment nodes.

[0087] For more information about target node coordinates, see Figure 3 and its related descriptions.

[0088] In some embodiments, the processing device may calculate the sum of the predicted coordinates and the correction amount of the polyline segment node, and determine the calculated sum as the optimized target node coordinates.

[0089] In some embodiments of the present specification, a correction amount is determined by performing a regression operation on the sampling features of the projection points, and the optimized target node coordinates are determined based on the predicted coordinates of the broken line segment nodes and the correction amount. This fully utilizes the distortion-free feature of the perspective camera image, reduces the distortion of the perspective camera image during the conversion process from perspective to BEV space, and makes the final determined target node coordinates more accurate.

[0090] It should be noted that the above description of process 400 is for illustration and purpose only and does not limit the scope of application of this specification. Those skilled in the art may make various modifications and variations to process 400 under the guidance of this specification. However, such modifications and variations are still within the scope of this specification.

[0091] Figure 5 This is an exemplary flow chart for determining a modified extrinsic parameter matrix according to some embodiments of this specification. In some embodiments, process 500 can be executed by the processing device 110 or the road element detection result optimization 200. For example, process 500 can be stored in a storage device in the form of a program or instruction, and when the processing device 110 or the road element detection result optimization 200 executes the instruction, process 500 can be implemented. The operational diagram of process 500 presented below is illustrative. In some embodiments, the process can be completed using one or more additional operations not described and / or one or more operations not discussed. In addition, Figure 5 The order in which the operations of flow 500 are illustrated and described below is not intended to be limiting.

[0092] Step 510 : Based on the initial extrinsic parameter matrix, the point-by-point features of the lidar point cloud are projected onto the perspective camera image to obtain a pseudo image feature map.

[0093] In some embodiments, the processing device may obtain point-by-point features of the lidar point cloud through a backbone network, such as a Visual Geometry Group Net (VGGNet) network, a residual network, or a start network.

[0094] The initial extrinsic parameter matrix refers to the matrix initially determined for the mutual conversion between the lidar point cloud and the perspective camera image.

[0095] In some embodiments, the processing device may determine an initial extrinsic parameter matrix based on the three-dimensional coordinates of the image acquisition device, the shooting direction, and the three-dimensional coordinates of the lidar device.

[0096] In some embodiments, there is an initial external parameter matrix corresponding to each image acquisition device and laser radar device.

[0097] In some embodiments, the processing device may multiply the initial extrinsic parameter matrix by the point-by-point features of the lidar point cloud, project it onto the perspective camera image, and obtain a pseudo image feature map.

[0098] Step 520 : Process the perspective camera image and the pseudo image feature map using a supplementary transformation model to determine a supplementary transformation matrix.

[0099] The supplementary transformation model may be a machine learning model for determining the supplementary transformation matrix. For example, the supplementary transformation model may include a graph neural network (GNN) model.

[0100] In some embodiments, the input to the supplemental transformation model may include a perspective camera image and a pseudo image feature map; the output may include a supplemental transformation matrix.

[0101] In some embodiments, for multiple perspective camera images and corresponding multiple pseudo image feature maps, the supplementary transformation model can input a perspective camera image and a corresponding pseudo image feature map respectively to obtain a supplementary transformation matrix corresponding to the perspective camera image.

[0102] In some embodiments, the supplementary transformation model can be obtained by training a plurality of third training samples with third labels. For the training method of the supplementary transformation model, please refer to Figure 3 Training method of multimodal road element detection model.

[0103] In some embodiments, the third training sample may include a sample perspective camera image and a sample pseudo image feature map; the third label may include a sample supplementary transformation matrix. In some embodiments, the third training sample may be obtained from historical data, and the third label may be obtained through manual annotation.

[0104] Step 530: Determine a modified extrinsic parameter matrix based on the supplementary transformation matrix and the initial extrinsic parameter matrix.

[0105] The modified extrinsic parameter matrix refers to the final matrix used to convert the lidar point cloud and the perspective camera image. In some embodiments, the modified extrinsic parameter matrix is ​​used to project the predicted coordinates of the polyline segment nodes.

[0106] In some embodiments, the processing device may left-multiply the supplementary transformation matrix by the initial extrinsic parameter matrix, and determine the calculated new matrix as the modified extrinsic parameter matrix.

[0107] In some embodiments of the present specification, a supplementary transformation matrix is ​​determined through a pseudo-image feature map, and a corrected extrinsic parameter matrix is ​​determined based on the initial extrinsic parameter matrix and the supplementary transformation matrix. The corrected extrinsic parameter matrix effectively reduces the calculation errors caused by inaccurate calibration parameters between the image acquisition device, the lidar equipment and the vehicle, or changes with the movement of the vehicle body.

[0108] It should be noted that the above description of process 500 is for illustration and purpose only and does not limit the scope of application of this specification. Those skilled in the art may make various modifications and variations to process 500 under the guidance of this specification. However, such modifications and variations are still within the scope of this specification.

[0109] One or more embodiments of this specification provide a road element detection result optimization device, which includes at least one processing device and at least one memory; the at least one memory is used to store computer instructions; and the at least one processing device is used to execute at least part of the computer instructions to implement the road element detection result optimization method described in any of the above embodiments.

[0110] One or more embodiments of this specification provide a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the road element detection result optimization method described in any one of the above embodiments.

[0111] While the basic concepts have been described above, it will be apparent to those skilled in the art that the detailed disclosure is merely illustrative and does not limit this specification. Although not explicitly stated herein, various modifications, improvements, and revisions to this specification may be made by those skilled in the art. Such modifications, improvements, and revisions are suggested in this specification and remain within the spirit and scope of the exemplary embodiments of this specification.

[0112] This specification also uses specific terms to describe the embodiments of this specification. For example, "one embodiment," "an embodiment," and / or "some embodiments" refer to a feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "one embodiment," "an embodiment," or "an alternative embodiment" two or more times in different locations in this specification do not necessarily refer to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of this specification may be appropriately combined.

[0113] In addition, unless expressly stated in the claims, the order of the processing elements and sequences, the use of alphanumeric characters, or the use of other names described in this specification are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some of the invention embodiments currently considered useful through various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the spirit and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0114] Similarly, it should be noted that, in order to simplify the presentation of this specification and thus facilitate understanding of one or more embodiments of the invention, the foregoing descriptions of the embodiments of this specification sometimes combine multiple features into a single embodiment, figure, or description thereof. However, this disclosure method does not imply that the subject matter of this specification requires more features than those recited in the claims. In fact, an embodiment may have fewer features than all of the features of a single disclosed embodiment.

[0115] In some embodiments, numbers are used to describe the quantity of components and attributes. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise stated, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the description and claims are approximate values, which may change according to the required characteristics of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the settings of such numerical values ​​are as accurate as possible within the feasible range.

[0116] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, and documents, cited in this specification is hereby incorporated by reference in its entirety. This includes application history documents that are inconsistent with or conflict with the content of this specification, as well as documents (currently or subsequently attached to this specification) that limit the broadest scope of the claims of this specification. It should be noted that if the descriptions, definitions, and / or terminology used in the accompanying materials are inconsistent or conflicting with the content of this specification, the descriptions, definitions, and / or terminology used in this specification will control.

[0117] Finally, it should be understood that the embodiments described in this specification are intended only to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly described and illustrated in this specification.

Claims

1. A method for optimizing road element detection results, characterized in that: The method comprises: Acquire a road element detection result in a BEV bird's-eye view space based on a multimodal road element detection model, wherein the road element detection result includes predicted coordinates of a polyline segment node of the element; Based on the predicted coordinates of the broken line segment node, obtaining a projection point of the predicted coordinates of the broken line segment node; The projection points are iteratively optimized based on the trained detection result optimization module to obtain optimized target node coordinates.

2. The method according to claim 1, characterized in that The step of obtaining the projection point of the predicted coordinates of the broken line segment node based on the predicted coordinates of the broken line segment node comprises: The predicted coordinates of the broken line segment nodes are projected onto a fused perspective view feature map to obtain projection points of the predicted coordinates of the broken line segment nodes, wherein the fused perspective view feature map is obtained based on a laser radar point cloud and a perspective view camera image.

3. The method according to claim 1, characterized in that The trained detection result optimization module iteratively optimizes the projection points to obtain optimized target node coordinates, including: Determine the sampling characteristics of the projection points based on the trained detection result optimization module; Performing a regression operation on the sampling features to determine the correction amount of the broken line segment node; Based on the predicted coordinates of the broken line segment nodes and the correction amount, the optimized target node coordinates are obtained.

4. The method according to claim 3, characterized in that The loss function during the training of the detection result optimization module is determined based on the positional relationship between the broken line segment node and a preset number of other nodes before and after it.

5. The method according to claim 2, characterized in that: The method further comprises: Based on the initial extrinsic parameter matrix, projecting the point-by-point features of the laser radar point cloud onto the perspective camera image to obtain a pseudo image feature map; Processing the perspective camera image and the pseudo image feature map by means of a supplementary transformation model to determine a supplementary transformation matrix; Based on the supplementary transformation matrix and the initial extrinsic parameter matrix, a modified extrinsic parameter matrix is ​​determined, and the modified extrinsic parameter matrix is ​​used to project the predicted coordinates of the polyline segment nodes.

6. The method according to claim 2, characterized in that The fused perspective view feature map is obtained based on the laser radar point cloud and the perspective view camera image, and includes: Projecting the laser radar point cloud onto the perspective camera image to obtain the three-dimensional coordinates corresponding to each pixel on the perspective camera image; The three-dimensional coordinates corresponding to each pixel are spliced ​​onto the perspective camera image to obtain the fused perspective feature map.

7. A road element detection result optimization system, characterized in that: The system comprises: A prediction module, used to obtain a road element detection result in a BEV bird's-eye view space based on a multimodal road element detection model, wherein the road element detection result includes a predicted coordinate of a polyline segment node of the element; A projection module, used for obtaining a projection point of the predicted coordinates of the broken line segment node based on the predicted coordinates of the broken line segment node; The optimization module is used to iteratively optimize the projection points based on the trained detection result optimization module to obtain optimized target node coordinates.

8. The system according to claim 7, characterized in that The optimization module is further used for: Determine the sampling characteristics of the projection points based on the trained detection result optimization module; Performing a regression operation on the sampling features to determine the correction amount of the broken line segment node; Based on the predicted coordinates of the broken line segment nodes and the correction amount, the optimized target node coordinates are obtained.

9. A road element detection result optimization device, characterized in that: The apparatus comprises at least one processing device and at least one memory; The at least one memory is used to store computer instructions; The at least one processing device is used to execute at least part of the computer instructions to implement a road element detection result optimization method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions. When the computer reads the computer instructions in the storage medium, the computer executes a road element detection result optimization method as described in any one of claims 1 to 6.

Images