High-precision map real-time updating method and system based on road end camera picture

Abstract

The invention relates to a high-precision map real-time updating method and system based on road end camera pictures. The method comprises the following steps: acquiring an initial image acquired by a road end camera, and determining the position and attitude parameters of the road end camera in a three-dimensional space based on the initial image; acquiring a real-time image acquired by a road end camera, wherein the real-time image comprises environment and map information of the current road section; based on the position and attitude parameters, enabling a reference point in a three-dimensional space to correspond to a reference pixel coordinate of the real-time image, and obtaining a position corresponding relation between the three-dimensional space and the image; performing feature extraction on the real-time image to obtain road end feature representation information; based on the road end feature representation information and feature representation information of an initial high-precision map, generating fusion feature information; and based on the query vector and the fusion feature information, updating the coordinate points of the initial high-precision map to realize real-time updating of the high-precision map. According to the method, the problems of poor timeliness, high manpower labeling cost and the like of a high-precision map updating method in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the fields of high-precision maps and autonomous driving technologies, and in particular to a method and system for real-time updating of high-precision maps based on road-side camera images. Background Art

[0002] As a key component of the autonomous driving system, high-precision maps can provide accurate geographic information for vehicle positioning, navigation and decision-making. Compared with ordinary maps, high-precision maps can provide more detailed lane information, traffic sign information and location information of infrastructure such as overpasses.

[0003] The existing high-precision map collection method mainly relies on professional surveying vehicles to collect data while driving on the road, and then manually annotate and process it. Although this method can obtain high-quality and high-precision map data, and the information is relatively comprehensive, since all information needs to be collected on-site, the update cycle is long. Therefore, for dynamic road conditions such as temporarily closed lanes, construction areas, and newly installed traffic signs, the existing high-precision maps cannot be updated in real time, and have the defect of poor timeliness. In addition, the manual annotation process also brings high time and labor costs. Summary of the invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the problems of poor timeliness and high labor annotation costs existing in the high-precision map updating method in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a method for real-time updating of high-precision maps based on road-side camera images, comprising:

[0006] Acquire an initial image captured by a roadside camera, and determine the position and attitude parameters of the roadside camera in three-dimensional space based on the initial image;

[0007] Acquire real-time images collected by roadside cameras, wherein the real-time images include the environment and map information of the current road section;

[0008] Based on the position and posture parameters, the reference point in the three-dimensional space is mapped to the reference pixel coordinates of the real-time image to obtain a position correspondence relationship between the three-dimensional space and the image;

[0009] Based on the position correspondence between the three-dimensional space and the image, feature extraction is performed on the real-time image to obtain road end feature representation information;

[0010] generating fused feature information based on the road end feature representation information and the feature representation information of the initial high-precision map;

[0011] Based on the query vector and the fused feature information, the coordinate points of the initial high-precision map are updated to achieve real-time update of the high-precision map; wherein the map includes a map query vector and a new element query vector, the map query vector is a vector generated by taking the coordinate points of the initial high-precision map as prior coordinates and encoding through a multi-layer perceptron, and the new element query vector is a vector updated as the real-time update network of the high-precision map is updated in real time.

[0012] In one embodiment of the present invention, determining the position and attitude parameters of the roadside camera in three-dimensional space based on the initial image includes:

[0013] Based on the camera parameter prediction network, the image information of the initial image is input into the camera parameter prediction network to obtain the camera's extrinsic parameter matrix; wherein the extrinsic parameter matrix includes a translation vector (position parameter) and a rotation matrix (attitude parameter), which are used to convert the coordinates in the world coordinate system into the coordinates in the camera coordinate system.

[0014] In one embodiment of the present invention, the external parameter matrix is ​​used to convert the coordinates in the world coordinate system into the coordinates in the camera coordinate system, which is obtained by transformation according to the following matrix formula:

[0015]

[0016] Among them, K is the camera's intrinsic parameter matrix, including focal length, principal point (optical center) coordinates, distortion coefficient, etc.;

[0017] R is the rotation matrix in the external parameter matrix;

[0018] T is the translation vector in the extrinsic matrix;

[0019] P c is the coordinate of the camera coordinate system;

[0020] P w The coordinates are in the world coordinate system.

[0021] In one embodiment of the present invention, extracting features from the real-time image to obtain roadside feature representation information includes:

[0022] Based on a convolutional neural network, extracting image features of the real-time image;

[0023] Based on the position and posture parameters, correspond the reference point in the three-dimensional space to the reference pixel coordinates of the real-time image;

[0024] The image features are used as the input of the Transformer neural network model, and the reference pixel coordinates corresponding to the real-time image and the picture features of the surrounding positions are fused using the cross-attention mechanism to obtain the road end feature representation information.

[0025] In one embodiment of the present invention, the generating of fused feature information based on the road end feature representation information and the feature representation information of the initial high-precision map includes:

[0026] The road end feature representation information and the feature representation information of the initial high-precision map are spliced ​​to form a fused feature vector;

[0027] The fused feature vector is input into a multi-layer perceptron, and the high-dimensional fused feature vector is mapped to a lower dimension by the multi-layer perceptron to obtain the fused feature information.

[0028] In one embodiment of the present invention, the updating of the coordinate points of the initial high-precision map based on the query vector and the fused feature information to achieve real-time updating of the high-precision map includes:

[0029] Input the query vector and the fused feature information into a prediction network based on a Transformer decoder, predict and output a coordinate change value, a coordinate change type, a new element coordinate, and a confidence level corresponding to the coordinate change type; wherein the new element coordinate is predicted based on the new element query vector; and the coordinate change type includes addition, modification, and deletion;

[0030] According to the coordinate change value, the new element coordinates and the coordinate change type, determine whether the confidence corresponding to the coordinate change type is greater than a preset threshold value. If greater than, update the prior coordinates of the coordinate point of the initial high-precision map and / or add the new element coordinates to obtain the updated target point coordinates; for each query vector, generate an updated high-precision map element framework line vector, and obtain the updated high-precision map element framework line according to the high-precision map element framework line vector.

[0031] In one embodiment of the present invention, the prediction network based on the Transformer decoder performs enhancement processing on the initial high-precision map data during training, including:

[0032] Adding random coordinate noise, including modifying the coordinate values ​​in an existing high-precision map or offsetting the positions of coordinate points on an existing high-precision map;

[0033] Delete some map elements, including deleting some frame lines in existing high-precision maps.

[0034] The present invention also provides a high-precision map real-time update system based on road-side camera images, comprising:

[0035] A position and attitude parameter acquisition module, used to acquire an initial image captured by a road-side camera, and determine the position and attitude parameters of the road-side camera in three-dimensional space based on the initial image;

[0036] A real-time image acquisition module is used to acquire real-time images collected by roadside cameras, wherein the real-time images include the environment and map information of the current road section;

[0037] A spatial position correspondence acquisition module, used to correspond the reference point in the three-dimensional space to the reference pixel coordinates of the real-time image based on the position and posture parameters, so as to obtain the position correspondence between the three-dimensional space and the image;

[0038] A road end feature representation information acquisition module is used to extract features from the real-time image to obtain road end feature representation information;

[0039] A fusion feature information acquisition module, used to generate fusion feature information based on the road end feature representation information and the feature representation information of the initial high-precision map;

[0040] A high-precision map real-time update module is used to update the coordinate points of the initial high-precision map based on the query vector and the fused feature information to achieve real-time update of the high-precision map; wherein the map includes a map query vector and a new element query vector, the map query vector is a vector generated by using the coordinate points of the initial high-precision map as prior coordinates and encoding through a multi-layer perceptron, and the new element query vector is a vector updated as the high-precision map is updated in real time.

[0041] The above technical solution of the present invention has the following advantages compared with the prior art:

[0042] The method and system for real-time updating of high-precision maps based on road-side camera images described in the present invention use real-time road-side camera data to add, delete and modify existing high-precision map elements based on existing high-precision maps, thereby obtaining a more accurate real-time high-precision map. By collecting data using road-side cameras and updating high-precision maps in real time using computer vision methods, the timeliness of high-precision maps is enhanced, while saving human resources for data collection and annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0044] Figure 1 It is a flow chart of the method for real-time updating of high-precision maps based on road-side camera images of the present invention.

[0045] Figure 2 It is a schematic diagram of predicting the external parameters of a camera based on a picture taken by a roadside camera in the present invention.

[0046] Figure 3 It is a schematic diagram of the present invention performing feature extraction on real-time images to obtain road-end feature representation information.

[0047] Figure 4 It is a schematic diagram of the present invention realizing real-time update of high-precision maps.

[0048] Figure 5 It is a schematic diagram of high-precision map training and reasoning of the high-precision map real-time update method based on road-side camera pictures of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0050] In the present invention, "several" means one or more, "multiple" means more than two, "greater than", "less than", "exceed" and the like are understood to exclude the number itself; "above", "below", "within" and the like are understood to include the number itself. In the description of the present invention, if there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0051] Example 1

[0052] Reference Figure 1 As shown, a method for real-time updating of a high-precision map based on road-side camera images in this embodiment includes:

[0053] S1. Obtain an initial image captured by a roadside camera, and determine the position and attitude parameters of the roadside camera in three-dimensional space based on the initial image. Figure 2 As shown, specifically including:

[0054] Use a roadside camera to take one or a group of pictures as initial images;

[0055] Based on the camera parameter prediction network, the image information of the initial image is input into the camera parameter prediction network to obtain the external parameter matrix of the camera.

[0056] The external parameter matrix includes a translation vector (position parameter) and a rotation matrix (attitude parameter), which is used to transform the coordinates in the world coordinate system into the coordinates in the camera coordinate system. It is obtained by the following matrix formula transformation:

[0057]

[0058] Among them, K is the intrinsic parameter matrix of the camera, including focal length, principal point (optical center) coordinates, distortion coefficient, etc.;

[0059] R is the rotation matrix in the external parameter matrix;

[0060] T is the translation vector in the extrinsic matrix;

[0061] P c is the coordinate of the camera coordinate system;

[0062] P w The coordinates are in the world coordinate system.

[0063] Through this process, the camera's external parameters are predicted based on the images taken by the road-side camera, and the position information of the road-side camera can be accurately obtained, which provides a basis for subsequent three-dimensional reconstruction of images and updating of high-precision maps.

[0064] As part of the intelligent transportation infrastructure, roadside cameras can collect image data of the road environment in real time, with the characteristics of strong timeliness and wide coverage. By processing the data collected by roadside cameras, some information of high-precision maps can be updated in real time, thereby significantly improving the dynamics and accuracy of the map. Using roadside cameras for automatic processing and updating can also reduce long labor costs and time costs.

[0065] S2. Acquire a real-time image collected by a roadside camera. The real-time image includes the environment and map information of the current road section.

[0066] S3. Based on the position and posture parameters, correspond the reference point in the three-dimensional space to the reference pixel coordinates of the real-time image to obtain the position correspondence between the three-dimensional space and the image.

[0067] Specifically, at each time point t, the real-time image P collected by the roadside camera r , P r ={p t1 ,p t2 ,…,p tn}, input the real-time image into the camera parameter prediction network to predict the camera's position and posture; according to the aforementioned coordinate transformation formula, correspond the reference point in the three-dimensional space to the reference pixel coordinates of the real-time image.

[0068] The camera parameter prediction network can adopt the existing convolutional neural network (CNN) or other architectures to extract image features, identify key points and visual clues in the image, and then infer the camera's posture.

[0069] S4. Based on the position correspondence between the three-dimensional space and the image, extract features from the real-time image to obtain roadside feature representation information. Figure 3 shown.

[0070] In this embodiment, a convolutional neural network or a neural network of other architectures is used to extract features of real-time images, and then these features are input into a feature information extraction network, and a neural network based on a Transformer architecture is used to process feature information of multiple images through a self-attention mechanism and a cross-attention mechanism. Specifically:

[0071] Based on the convolutional neural network, the image features of the real-time image are extracted; a multi-dimensional feature vector is generated, and the convolutional neural network slides in the image through its convolution kernel to extract feature information with local correlation, and the feature information includes edges, textures, shapes, etc.

[0072] The main workflow of the convolutional neural network is as follows:

[0073] Convolution operation: extract local information of the image through the convolution kernel. The convolution kernel is a small window that slides on the image and performs weighted summation of local pixel values ​​to generate a feature map.

[0074] Activation function: ReLU activation function is usually used to increase the nonlinearity of the network to capture complex features.

[0075] Pooling: The pooling layer is used to downsample feature maps, reduce dimensions and improve computational efficiency. The most common pooling method is Max Pooling.

[0076] Generate multi-dimensional feature vectors: After multiple layers of convolution operations and pooling, CNN will eventually generate a feature map containing multiple channels, each channel represents different feature information. These feature maps are flattened and output as multi-dimensional feature vectors, containing local related information such as edges, textures, shapes, etc. of the image.

[0077] Based on the position and posture parameters, the reference point in the three-dimensional space is mapped to the reference pixel coordinates of the real-time image.

[0078] The image features are used as the input of the Transformer neural network model. The Transformer model fuses the reference pixel coordinates corresponding to multiple images and the image features of the surrounding positions through the self-attention mechanism and the cross-attention mechanism to obtain the road end feature representation information F r , the specific form is a multi-dimensional feature vector. The specific process is as follows:

[0079] Input multi-dimensional feature vector: The multi-dimensional feature vector generated by the convolutional neural network is used as the input of the Transformer.

[0080] For each image, Transformer processes its feature vector through a self-attention mechanism to ensure that each feature point can interact with other feature points. This can integrate the global information within the image.

[0081] In order to process data from different images, the Transformer's cross-attention mechanism integrates image features from different perspectives or time points. For example, different road-side cameras may shoot the same scene from multiple angles, and the cross-attention mechanism can combine the feature information of these different images to generate a more accurate road-side feature representation.

[0082] After processing self-attention and cross-attention, Transformer outputs a feature vector that integrates global and cross-image relationships, and obtains road-end feature representation information, which is a multi-dimensional vector that covers key information such as spatial layout, object shape, and positional relationship in the scene.

[0083] S5. Generate fusion feature information based on the road-end feature representation information and the feature representation information of the initial high-precision map (optionally). Specifically:

[0084] The road end feature representation information F and the feature representation information F of the initial high-precision map m Concatenate to form a fusion feature vector [F r ,F m ];

[0085] The fused feature vector [F r ,F m ] is input into a multi-layer perceptron (MLP), through which the high-dimensional fusion features are mapped to a lower dimension to obtain the fusion feature information F fuse Among them, F fuse =MLP([F r ,F m ]).

[0086] Multilayer Perceptron (MLP) is a fully connected neural network consisting of multiple linear layers, which is often used to reduce the dimensionality of high-dimensional data. Through the multilayer perceptron, the system can extract more refined feature information for subsequent map updates. High-precision maps are composed of points and frame lines connecting the points, containing geographic information such as roads and buildings.

[0087] The working process of MLP is roughly as follows:

[0088] Input layer: Input the concatenated fusion feature vector [F r ,F m ];

[0089] Hidden layer: Through a series of linear transformations and nonlinear activation functions, MLP maps high-dimensional feature vectors to lower dimensions. Activation functions include ReLU, Sigmoid, etc.

[0090] Output layer: output fusion feature information F after dimension reduction fuse .

[0091] S6, based on the query vector and the fusion feature information generated in step S5, updating the coordinate points of the initial high-precision map to achieve real-time update of the high-precision map; Figure 4 As shown, the process operates using a Transformer decoder-based prediction network.

[0092] Wherein, the map includes a map query vector e m and the new element query vector e n , the map query vector e m The coordinate points of the initial high-precision map are used as the prior coordinates C m , and the vector generated by the multilayer perceptron encoding, and the new element query vector e n It is a high-dimensional vector that is updated in real time with the HD map. The update of the new element query vector is dynamic and is constantly adjusted with the update of the HD map. The two are concatenated to form the input query vector e in =[e m ,e n ].

[0093] The query vector e m And fusion feature information F fuse Input into the Transformer decoder to obtain the updated query vector, and then predict the coordinate change value C delta , coordinate change type T, new element coordinates C n and the corresponding confidence level P. There are three types of coordinate changes: addition (adding new map elements), modification (adjusting the coordinates of existing elements), and deletion (removing elements that no longer exist).

[0094] According to the coordinate change value C delta , coordinate change type T, new element coordinates C n And the corresponding confidence P, to determine whether the confidence is greater than the preset threshold (θ). If the confidence meets the requirements, the coordinate points in the high-precision map are updated to generate a new target coordinate set C u This step also includes the prediction and insertion of new elements.

[0095] It can be understood that each coordinate change carries a confidence value, which indicates the credibility of the change. When the confidence exceeds a preset threshold value θ, the update is performed to ensure the accuracy of the map update.

[0096] The details are explained below.

[0097] In step S6, based on the query vector e m and the fusion feature information e n , update the coordinate points of the initial high-precision map to achieve real-time update of the high-precision map, including:

[0098] The query vector e m and the fusion feature information e n Input to the prediction network based on Transformer decoder, predict and output the coordinate change value C delta , coordinate change type T, new element coordinates C n and a confidence level P corresponding to a coordinate change type T; wherein the coordinates of the new element are predicted based on the new element query vector; the coordinate change type T includes addition, modification and deletion;

[0099] Based on the coordinate change value C delta , coordinate change type T, new element coordinates C n And the corresponding confidence level P, update the coordinate points of the initial high-precision map.

[0100] It should be noted that a high-precision map is composed of points and frame lines connecting the points; the coordinate points of the initial high-precision map (i.e., the points that make up the high-precision map) are used as the prior coordinates C m ={(x1,y1),(x2,y2),…,(x n ,y n )};

[0101] Concatenate the prior coordinates of each frame line of the initial high-precision map into [x1′, y1′, x2′, y2′, …, x n ′,y n ′], which is input into the multilayer perceptron for encoding to obtain the map query vector e m ={e m1 ,e m2 ,…,e mn In order to consider the newly emerged elements, a high-dimensional new element query vector e is introduced which can be continuously updated with training. n ={e n1 ,e n2 ,…,e nN}, the two are concatenated to get the input e in =[e m ,en ]. in and F fuse As the input of the prediction network based on the Transformer decoder, the coordinate change value C is predicted delta ={(Δx1,Δy1),(Δx2,Δy2),…,(Δx n ,Δy n )}, and the coordinate change type T of each coordinate is T = {t1, t2, ..., t n} and confidence P=={p1,p2,…,p n}.

[0102] The coordinate change value C is obtained delta , coordinate change type T, new element coordinates C n And the corresponding confidence level P, to update the coordinate points of the initial high-precision map, including:

[0103] According to the coordinate change value C delta , the new element coordinates C n and the coordinate change type T, determine whether the confidence P corresponding to the coordinate change type T is greater than a preset threshold θ, and if so, update the prior coordinates C of the coordinate point of the initial high-precision map m And / or add the new element coordinates C n , get the updated target point coordinates C in ;

[0104] Specifically, for the prior coordinates C m ={(x1,y1),(x2,y2),…,(x n ,y n )}, update or delete according to the change type, and obtain the updated target point coordinate set C u ={(x1+Δx1,y1+Δy1),(x2+Δx2,y2+Δy2),…,(x n +Δx n ,y n +Δy n )}. For new element queries, predict the new element coordinates C n ={(x1′,y1′),(x2′,y2′),…,(x n ′,y n ′)}. Finally, only the coordinate points with confidence greater than the threshold θ are retained. According to this method, an updated high-precision map element frame line vector is generated for each query vector, and the specific form is [x′1,y1′,x2′,y2′,…,x n ′,y n′], and the coordinates of the points on this frame line are obtained by using this vector as (x1′, y1′), (x2′, y2′),…, (x n ′,y n ′), draw the frame line to realize real-time update of the map.

[0105] By selectively accepting additions, deletions, and modifications based on confidence, we ensure that the quality of the updated map meets actual needs.

[0106] The above method automatically adds, deletes or modifies the elements of the existing high-precision map through image input from different angles of the road-side camera, and generates a real-time updated high-precision map.

[0107] In addition, in order to improve the robustness of the model in the face of new environments and different data modes, the prediction network based on the Transformer decoder performs enhanced processing on the initial high-precision map data during training, including:

[0108] Add random coordinate noise This includes modifying the coordinate values ​​in an existing high-precision map or offsetting the positions of coordinate points on an existing high-precision map;

[0109] Delete some map elements, including deleting some frame lines in the existing HD map. In HD maps, map elements are usually composed of coordinate points and the frame lines formed by their connections. Delete some frame lines to simulate incomplete maps or missing data in the real world.

[0110] The purpose of adding random coordinate noise and deleting some map elements is to use the existing high-precision map to modify the elements to build an "initial map", using the unenhanced map as the true value to train the model's coordinate update and generation capabilities.

[0111] In summary, refer to Figure 5 As shown in the figure, the real-time update method of high-precision maps based on road-side camera images constructs an "initial map" through data enhancement (such as adding coordinate noise, removing some map elements, etc.), takes the unenhanced map as the true value, trains the coordinate update and generation capabilities of the model, and selectively accepts additions, deletions, and modifications based on confidence to ensure that the quality of the updated map meets actual needs.

[0112] Example 2

[0113] Based on the same inventive concept, this embodiment provides a high-precision map real-time update system based on road-side camera images. The principle of solving the problem is similar to the high-precision map real-time update method based on road-side camera images, and the repeated parts will not be repeated.

[0114] This embodiment provides a high-precision map real-time update system based on road-side camera images, including:

[0115] A position and attitude parameter acquisition module, used to acquire an initial image captured by a road-side camera, and determine the position and attitude parameters of the road-side camera in three-dimensional space based on the initial image;

[0116] A real-time image acquisition module is used to acquire real-time images collected by roadside cameras, wherein the real-time images include the environment and map information of the current road section;

[0117] A spatial position correspondence relationship acquisition module, used to convert the real-time image into corresponding three-dimensional coordinates in the three-dimensional space based on the position and posture parameters, and obtain the position correspondence relationship between the three-dimensional space and the image;

[0118] A road end feature representation information acquisition module is used to extract features from the real-time image to obtain road end feature representation information;

[0119] A fusion feature information acquisition module, used to generate fusion feature information based on the road end feature representation information and the feature representation information of the initial high-precision map;

[0120] A high-precision map real-time update module is used to update the coordinate points of the initial high-precision map based on the query vector and the fused feature information to achieve real-time update of the high-precision map; wherein the map includes a map query vector and a new element query vector, the map query vector is a vector generated by using the coordinate points of the initial high-precision map as prior coordinates and encoding through a multi-layer perceptron, and the new element query vector is a vector updated as the high-precision map is updated in real time.

[0121] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] Finally, it should be noted that the above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for real-time updating of high-precision maps based on roadside camera images, characterized in that: include: Acquire an initial image captured by a roadside camera, and determine the position and attitude parameters of the roadside camera in three-dimensional space based on the initial image; Acquire real-time images collected by roadside cameras, wherein the real-time images include the environment and map information of the current road section; Based on the position and posture parameters, the reference point in the three-dimensional space is mapped to the reference pixel coordinates of the real-time image to obtain a position correspondence relationship between the three-dimensional space and the image; Based on the position correspondence between the three-dimensional space and the image, feature extraction is performed on the real-time image to obtain road end feature representation information; Generating fused feature information based on the road end feature representation information and the feature representation information of the initial high-precision map; Based on the query vector and the fused feature information, the coordinate points of the initial high-precision map are updated to achieve real-time update of the high-precision map; wherein the map includes a map query vector and a new element query vector, the map query vector is a vector generated by taking the coordinate points of the initial high-precision map as prior coordinates and encoding through a multi-layer perceptron, and the new element query vector is a vector updated as the real-time update network of the high-precision map is updated in real time.

2. The method for real-time updating of high-precision maps based on roadside camera images according to claim 1, characterized in that: The determining, based on the initial image, the position and attitude parameters of the roadside camera in the three-dimensional space includes: Based on the camera parameter prediction network, the image information of the initial image is input into the camera parameter prediction network to obtain the camera's extrinsic parameter matrix; wherein the extrinsic parameter matrix includes a translation vector, i.e., a position parameter, and a rotation matrix, i.e., a posture parameter, which are used to convert the coordinates in the world coordinate system into the coordinates in the camera coordinate system.

3. The method for real-time updating of high-precision maps based on roadside camera images according to claim 2 is characterized in that: The external parameter matrix is ​​used to convert the coordinates in the world coordinate system into the coordinates in the camera coordinate system, which is obtained according to the following matrix formula: Among them, K is the intrinsic parameter matrix of the camera, including focal length, coordinates of the principal point (i.e., optical center), and distortion coefficient; R is the rotation matrix in the external parameter matrix; T is the translation vector in the extrinsic matrix; P c is the coordinate of the camera coordinate system; P w The coordinates are in the world coordinate system.

4. The method for real-time updating of high-precision maps based on roadside camera images according to claim 2 is characterized in that: The step of corresponding the reference point in the three-dimensional space to the reference pixel coordinates of the real-time image based on the position and posture parameters to obtain the position correspondence between the three-dimensional space and the image includes: At each time point, inputting the real-time image into the camera parameter prediction network; Based on the extrinsic parameter matrix, the reference points in the three-dimensional space are mapped to the reference pixel coordinates of the real-time image to obtain the positional correspondence between the three-dimensional space and the image.

5. The method for real-time updating of high-precision maps based on roadside camera images according to claim 1, characterized in that: The step of extracting features from the real-time image to obtain roadside feature representation information includes: Based on a convolutional neural network, extracting image features of the real-time image; Based on the position and posture parameters, correspond the reference point in the three-dimensional space to the reference pixel coordinates of the real-time image; The image features are used as the input of the Transformer neural network model, and the reference pixel coordinates corresponding to the real-time image and the picture features of the surrounding positions are fused using the cross-attention mechanism to obtain the road end feature representation information.

6. The method for real-time updating of high-precision maps based on roadside camera images according to claim 1, characterized in that: The generating fused feature information based on the road end feature representation information and the feature representation information of the initial high-precision map includes: The road end feature representation information and the feature representation information of the initial high-precision map are spliced ​​to form a fused feature vector; The fused feature vector is input into a multi-layer perceptron, and the high-dimensional fused feature vector is mapped to a lower dimension by the multi-layer perceptron to obtain the fused feature information.

7. The method for real-time updating of high-precision maps based on roadside camera images according to claim 1, characterized in that: The updating of the coordinate points of the initial high-precision map based on the query vector and the fused feature information to achieve real-time updating of the high-precision map includes: Input the query vector and the fused feature information into a prediction network based on a Transformer decoder, predict and output a coordinate change value, a coordinate change type, a new element coordinate, and a confidence level corresponding to the coordinate change type; wherein the new element coordinate is predicted based on the new element query vector; and the coordinate change type includes addition, modification, and deletion; According to the coordinate change value, the new element coordinates and the coordinate change type, determine whether the confidence corresponding to the coordinate change type is greater than a preset threshold value. If greater than, update the prior coordinates of the coordinate point of the initial high-precision map and / or add the new element coordinates to obtain the updated target point coordinates; for each query vector, generate an updated high-precision map element framework line vector, and obtain the updated high-precision map element framework line according to the high-precision map element framework line vector.

8. The method for real-time updating of high-precision maps based on roadside camera images according to claim 7, characterized in that: During training, the prediction network based on the Transformer decoder performs enhancement processing on the initial high-precision map data, including: Adding random coordinate noise, including modifying the coordinate values ​​in an existing high-precision map or offsetting the positions of coordinate points on an existing high-precision map; Delete some map elements, including deleting some frame lines in existing high-precision maps.

9. A high-precision map real-time update system based on roadside camera images, characterized in that: include: A position and attitude parameter acquisition module, used to acquire an initial image captured by a road-side camera, and determine the position and attitude parameters of the road-side camera in three-dimensional space based on the initial image; A real-time image acquisition module is used to acquire real-time images collected by roadside cameras, wherein the real-time images include the environment and map information of the current road section; A spatial position correspondence acquisition module, used to correspond the reference point in the three-dimensional space to the reference pixel coordinates of the real-time image based on the position and posture parameters, so as to obtain the position correspondence between the three-dimensional space and the image; A road end feature representation information acquisition module is used to extract features from the real-time image to obtain road end feature representation information; A fusion feature information acquisition module, used to generate fusion feature information based on the road end feature representation information and the feature representation information of the initial high-precision map; A high-precision map real-time update module is used to update the coordinate points of the initial high-precision map based on the query vector and the fused feature information to achieve real-time update of the high-precision map; wherein the map includes a map query vector and a new element query vector, the map query vector is a vector generated by using the coordinate points of the initial high-precision map as prior coordinates and encoding through a multi-layer perceptron, and the new element query vector is a vector updated with the real-time update of the high-precision map real-time update network.

Images