AR-HUD navigation guide line generation method and related device
Through the road center line information query model to extract and match the target road center line, an AR-HUD navigation guide line that fits the real scene is generated, which solves the problem that the navigation guide line and the real scene are not in line with the existing technology, and improves the accuracy of navigation and security.
Patent Information
- Application Number
- CN202411208952.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-06
AI Technical Summary
The existing AR-HUD navigation guide lines do not fit the real scene, making it difficult for drivers to navigate accurately, which may lead to wrong lane entry or obstacle collision, affecting road traffic safety.
By obtaining the real scene image of the vehicle, the road center line information query model is used to extract the point column coordinates and TBT direction information of the road center line, and the navigation information is combined with the navigation information to match the target road center line to generate a navigation guide line that fits the real scene. The model performs feature extraction and prediction through a graph encoder and graph decoder, and outputs information of the road centerline to generate an AR-HUD navigation guide line.
It improves the fit between the navigation guide lines and the real scene, ensures the accuracy and safety of navigation guidance, reduces the risk of drivers entering the lane incorrectly or colliding, and improves driving experience and travel efficiency.
Smart Images

Figure CN119091409A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle navigation technology, and in particular to an AR-HUD navigation guide line generation method and related devices. Background Art
[0002] Augmented Reality (AR) technology is a technology that superimposes computer-generated virtual information onto the real environment around the user, so that the user can be convinced from a sensory effect that the virtual environment is a part of the real environment around him. Head-Up Display (HUD) technology is a technology that can project important information (such as vehicle speed, navigation information, safety warnings, etc.) directly onto a transparent display or windshield in front of the driver's line of sight. With HUD, the driver can get the required information in real time without having to look down at the dashboard or navigation screen during driving, thereby improving driving safety and convenience. HUD can also be translated into Chinese as "head-up display" or "head-up display system."
[0003] Augmented Reality Head-Up Display (AR-HUD) technology is a combination of AR technology and HUD technology. AR-HUD uses AR imaging technology to overlay digital images on the real scene seen by the driver, so that the information projected by the HUD is integrated with the real environment outside the car. The basic principle of AR-HUD is similar to that of HUD, that is, the light information emitted by the projector is projected onto the windshield after a series of refractions and reflections. The human eye can see the information projected on it, and it feels like the information is suspended in front. However, AR-HUD achieves a high degree of integration of virtual information and the real environment through more complex optical systems and algorithms. At present, AR-HUD has been mass-produced on many models. With the continuous maturity of technology and the reduction of costs, AR-HUD is expected to become an important part of the smart cockpit in the future.
[0004] In the prior art, navigation guide lines can be projected by using AR-HUD technology. As an example, the navigation guide line can be in the form of a one-way arrow with a direction attribute. The display of the navigation guide line guides the driver to drive his vehicle into the recommended lane according to the navigation path. However, in current practical applications, there is a problem that the AR-HUD navigation guide line does not fit the real scene. Figure 1 The current technology uses AR-HUD to project navigation guide lines in real scenes. The green arrows represent navigation guide lines. Figure 1 As shown, the current navigation guide line suggests that the driver drive to the right front and enter the ramp. Figure 1It can be seen that the navigation guide line points to the edge of the lane line of the right front ramp, which makes the navigation guide line inconsistent with the real scene and has a poor visual effect. If the driver drives according to the navigation guide line, the driving judgment is likely to be affected by the navigation guide line that does not match the real scene, and it is easy to drive into the wrong lane or collide with other obstacles, causing road traffic safety problems. Summary of the invention
[0005] Based on the above problems, the present application provides an AR-HUD navigation guide line generation method and related devices, with the aim of generating navigation guide lines that fit the real scene, improving the visual effect of the navigation guide lines displayed by the AR-HUD, and improving the effectiveness and safety of the navigation guide lines in guiding the driver's driving.
[0006] The embodiments of the present application disclose the following technical solutions:
[0007] The first aspect of the present application provides a method for generating an AR-HUD navigation guide line, the method comprising:
[0008] Acquire a real-life image of the vehicle;
[0009] The image is used as an input of a road centerline information query model, and the road centerline information query model processes and analyzes the image and outputs information of multiple road centerlines contained in the image; the road centerline information includes point column coordinates on the road centerline and turn-by-turn navigation TBT direction information of the road centerline;
[0010] According to the navigation information of the vehicle and the information of the multiple road center lines, determining a target road center line that matches the TBT direction information in the navigation information from the multiple road center lines;
[0011] The target road centerline is used to generate an AR-HUD navigation guide line.
[0012] In a first optional implementation of the first aspect, the training step of the road centerline information query model includes:
[0013] A training sample is obtained from a visual perception data set and input into a model to be trained; the training sample has corresponding annotation data; the training sample includes a sample image taken by a surround view camera of a sample vehicle and internal and external parameter calibration data of the surround view camera; the annotation data includes the actual coordinates of a point column on a road traffic marking line contained in the map data corresponding to the sample image, and TBT direction annotation information of a road centerline in the road traffic marking line; the model to be trained includes a graph encoder and a graph decoder connected front and back;
[0014] Performing feature extraction on the sample image in the training sample by using the image encoder;
[0015] Performing inverse perspective transformation according to the features extracted by the image encoder and the internal and external parameter calibration data to obtain real scene bird's-eye view features;
[0016] Constructing a line query vector, a point query vector and a TBT direction query vector based on the real scene bird's-eye view feature;
[0017] The graph decoder performs learning based on the line query vector, the point query vector and the TBT direction query vector, and outputs a prediction result; the prediction result includes a first prediction content and a second prediction content, the first prediction content includes the predicted coordinates of the point column on the road traffic marking line, and the second prediction content includes the TBT direction prediction result of the road centerline;
[0018] Based on the difference between the first prediction content and the actual coordinates of the point column on the corresponding road traffic marking line in the annotated data, as well as the difference between the second prediction content and the TBT direction annotation information of the corresponding road centerline in the annotated data, it is determined whether the training cutoff condition has been met. If so, the training is stopped to obtain a road centerline information query model; if not, after adjusting the parameters of the model to be trained, the steps of obtaining training samples from the visual perception data set and inputting them into the model to be trained are repeated until the training cutoff condition is met.
[0019] In a first optional implementation of the first aspect, the graph decoder includes L self-attention layers and L cross-attention layers that are cross-arranged; L is an integer greater than 1;
[0020] The graph decoder performs learning based on the line query vector, the point query vector and the TBT direction query vector, and outputs a prediction result, including:
[0021] combining the line query vector, the point query vector and the TBT direction query vector into an initial hierarchical query representation;
[0022] The initial hierarchical query representation is input into the first self-attention layer, and the prediction result is output through the learning of L cross-attention layers and L cross-attention layers in sequence; the learning result output by the i-th self-attention layer is used as the hierarchical query representation input to the i-th cross-attention layer, and i is any integer in the range of 1 to L.
[0023] In a second optional implementation of the first aspect, the training step of the road centerline information query model includes:
[0024] A training sample is obtained from a visual perception data set and input into a model to be trained; the training sample has corresponding annotation data; the training sample includes a sample image taken by a surround view camera of a sample vehicle and internal and external parameter calibration data of the surround view camera; the annotation data includes the actual coordinates of a point column on a road traffic marking line contained in the map data corresponding to the sample image, and TBT direction annotation information of a road centerline in the road traffic marking line; the model to be trained includes a graph encoder, a graph decoder and a multi-layer perceptron connected in sequence;
[0025] Performing feature extraction on the sample image in the training sample by using the image encoder;
[0026] Performing inverse perspective transformation according to the features extracted by the image encoder and the internal and external parameter calibration data to obtain real scene bird's-eye view features;
[0027] Constructing a line query vector and a point query vector based on the real scene bird's-eye view feature;
[0028] The graph decoder performs learning based on the line query vector and the point query vector to obtain an N-dimensional output result; the N-dimensional output result includes a first prediction content, and the first prediction content includes predicted coordinates of a point sequence on a road traffic marking line;
[0029] Mapping the N-dimensional output result into an 8-dimensional TBT direction prediction vector through the multi-layer perceptron; each dimension in the 8-dimensional TBT direction prediction vector corresponds to a TBT direction;
[0030] Based on the difference between the first prediction content and the actual coordinates of the point column on the corresponding road traffic marking in the annotated data, as well as the difference between the 8-dimensional TBT direction prediction vector and the TBT direction annotation information of the corresponding road centerline in the annotated data, it is determined whether the current training cutoff condition has been met. If so, the training is stopped to obtain a road centerline information query model; if not, after adjusting the parameters of the model to be trained, the steps of obtaining training samples from the visual perception data set and inputting them into the model to be trained are repeated until the training cutoff condition is met.
[0031] Optionally, determining, from the multiple road centerlines according to the navigation information of the vehicle and the information of the multiple road centerlines, a target road centerline that matches the TBT direction information in the navigation information includes:
[0032] Adding the road center lines among the plurality of road center lines that are consistent with the TBT direction information in the navigation information to the set of road center lines to be selected;
[0033] If the number of road centerlines included in the set of road centerlines to be selected is greater than 1, a target road centerline is determined from the set of road centerlines to be selected according to the current driving direction of the vehicle and / or the current position information of the vehicle.
[0034] Optionally, determining a target road centerline from the set of to-be-selected road centerlines according to the current driving direction of the vehicle and / or the current position information of the vehicle includes:
[0035] A road centerline in the set of candidate road centerlines that is consistent with the current driving direction of the vehicle and closest to the current position of the vehicle is used as the target road centerline.
[0036] Optionally, determining a target road centerline from the set of to-be-selected road centerlines according to the current driving direction of the vehicle and / or the current position information of the vehicle includes:
[0037] From the set of road centerlines to be selected, a road centerline that minimizes the calculated value of the expression is used as the target road centerline; the expression is:
[0038] ||dir(argmin d(p,p car ))-dir car ||+λmin d(p,p car )
[0039] Wherein, d(p, pcar) is the distance between the point on the center line of the road and the current position of the vehicle;
[0040] The dir(argmin d(p,p car )) is the center line of the road so that d(p,p car ) reaches a minimum point relative to the direction of the vehicle's current position;
[0041] The dir car is the current driving direction of the vehicle;
[0042] The min d(p,p car ) is the minimum distance between a point on the road centerline and the current position of the vehicle;
[0043] The λ is the value for min d(p,p car )’s configuration weight.
[0044] The second aspect of the present application provides an AR-HUD navigation guide line generation device, the device comprising:
[0045] An image acquisition module, used to acquire a real-scene image of a vehicle;
[0046] A centerline information acquisition module, used to use the image as an input of a road centerline information query model, and the road centerline information query model processes and analyzes the image and then outputs information of multiple road centerlines contained in the image; the road centerline information includes the coordinates of the point columns on the road centerline and the turn-by-turn navigation TBT direction information of the road centerline;
[0047] A centerline matching module, configured to determine, from the multiple road centerlines, a target road centerline that matches the TBT direction information in the navigation information based on the navigation information of the vehicle and the information of the multiple road centerlines;
[0048] A guide line generation module is used to generate an AR-HUD navigation guide line using the target road center line.
[0049] A third aspect of the present application provides a processor for running a computer program, which, when running, executes the AR-HUD navigation guide line generation method introduced in any implementation of the first aspect.
[0050] The fourth aspect of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the AR-HUD navigation guide line generation method described in any implementation method of the first aspect is implemented.
[0051] The technical solution of the present application provides an AR-HUD navigation guide line generation method, device, processor and storage medium. In the AR-HUD navigation guide line generation method, firstly, a real scene image of the vehicle is obtained; then, the obtained image is used as the input of the road centerline information query model, and the road centerline information query model processes and analyzes the image and outputs the information of multiple road centerlines contained in the image; wherein, the information of the road centerline includes the point column coordinates on the road centerline and the turn-by-turn navigation TBT direction information of the road centerline. Thereafter, according to the navigation information of the vehicle and the information of the multiple road centerlines, the target road centerline that matches the TBT direction information in the navigation information is determined from the multiple road centerlines; finally, the AR-HUD navigation guide line is generated using the target road centerline. The information of the road centerline output by the road centerline query model contains TBT direction information, and the TBT direction information of these road centerlines is matched with the TBT direction information in the navigation information. The target road centerline finally determined, on the one hand, conforms to the position of the road centerline in the real scene, and on the other hand, can better demonstrate the navigation guidance function from the position. Therefore, the generated AR-HUD navigation guide line is more closely aligned with the real scene. Compared with the navigation guide line generated by the prior art, it is more accurate in guiding the driver's driving lane and driving direction, and has a more obvious auxiliary effect on driving safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0053] Figure 1 This is a schematic diagram of the effect of using AR-HUD to project navigation guide lines in real scenes using current technology;
[0054] Figure 2 A flowchart of a method for generating an AR-HUD navigation guide line provided in an embodiment of the present application;
[0055] Figure 3 A schematic diagram of a process for training a road centerline information query model provided in an embodiment of the present application;
[0056] Figure 4 A schematic diagram of the structure of a road centerline information query model provided in an embodiment of the present application;
[0057] Figure 5 A schematic diagram of another process of training a road centerline information query model provided in an embodiment of the present application;
[0058] Figure 6 A schematic diagram of the structure of another road centerline information query model provided in an embodiment of the present application;
[0059] Figure 7 A schematic diagram of the structure of an AR-HUD navigation guide line generating device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0060] AR-HUD technology has a prominent application in the field of vehicle navigation technology. By projecting navigation guide lines, it can provide guidance for drivers to drive their vehicles, for example, guiding drivers to drive into the recommended lane. However, in current practical applications, the generated AR-HUD navigation guide lines often have problems with not fitting the real scene well enough, such as Figure 1 The effect shown. Figure 1 The green arrow-shaped navigation guide line shown in the figure is driving in the corresponding direction, which is likely to cause the vehicle to enter the ramp along the edge of the lane line and collide with other obstacles on the roadside. It can be seen that although AR-HUD navigation guide lines are currently provided, drivers still need to judge and adjust in combination with traffic rules and personal driving experience. If the judgment is insufficient or the adjustment is not timely enough, there may still be problems of driving into the wrong lane or colliding with obstacles, causing road traffic safety hazards. In addition, because the AR-HUD navigation guide line does not fit the real scene, the driver may also miss the opportunity to drive into the lane he should have entered, resulting in unnecessary lengthening of the journey, detours, and other problems, delaying the driving itinerary plan, and affecting the driving and riding experience of the driver and passengers on the car.
[0061] After research, the inventor proposed an AR-HUD navigation guide line generation method and related devices. Based on the real-scene image of the vehicle, the road centerline information query model is used to obtain the information of multiple road centerlines contained in the image, such as point column coordinates and TBT direction information. Based on this, the target road centerline that matches the TBT direction information in the navigation information can be determined from multiple road centerlines in combination with the TBT direction information in the navigation information, and then the target road centerline is used to generate the current AR-HUD navigation guide line. On the one hand, the target road centerline conforms to the position of the road centerline in the real scene, and on the other hand, it can better demonstrate the navigation guidance function from the position. The navigation guide line has a higher degree of fit with the real scene, the driving guidance effect on the driver is more accurate, and the auxiliary effect of driving safety is more obvious.
[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0063] See also Figure 2 , which is a flow chart of a method for generating an AR-HUD navigation guide line provided in an embodiment of the present application. Figure 2 As shown, the AR-HUD navigation guide line generation method includes:
[0064] S201: Acquire a real-life image of a vehicle.
[0065] The technical solution of this application can provide AR-HUD navigation guide lines for vehicles equipped with surround-view cameras. The surround-view camera can collect real-scene images on the vehicle and provide a real-scene image data basis for subsequent inquiries about the information of the center line of the road. In practical applications, the surround-view camera installed on the vehicle can use a wide-angle camera or a fisheye camera to capture real-scene images with a wider range and larger coverage. The surround-view camera can be specifically set at the vehicle's B-pillar, vehicle fender, rear windshield of the parking space, rear license plate frame, roof and other positions. Installing surround-view cameras at these positions will not block the driver's line of sight to observe the road conditions outside the vehicle on the one hand, and on the other hand, it can capture scenes in the driver's blind spots, providing safe driving assistance from the image level.
[0066] In actual applications, the number of surround view cameras can be configured according to demand. For example, 6, 8 or even 12 surround view cameras can be configured. The real-life images captured by the surround view cameras can not only be used to generate AR-HUD navigation guide lines in the technical solution of this application, but can also assist in realizing vehicle reversing images, panoramic images, lane departure warnings, blind spot detection, automatic parking and other functions.
[0067] S202: Using the image as input to a road centerline information query model, and having the road centerline information query model process and analyze the image to output information of multiple road centerlines contained in the image.
[0068] In the technical solution of this application, a pre-trained road centerline information query model is used to implement the query function of the road centerline. In the AR-HUD navigation guide line technical solution generated by the relevant technology, the relevant information of the road centerline is not effectively combined, resulting in the AR-HUD navigation guide line not fitting the real scene, and even the AR-HUD navigation guide line pointing to the edge of the lane or even a roadblock. In this solution, the AR-HUD navigation guide line is generated in combination with the road centerline.
[0069] In the specific implementation of this step, the image collected in the previous step S201 is input into the road centerline information query model. The model can process the image, for example, after extracting the features of the real scene image, perform operations such as inverse perspective transformation, and finally obtain the bird's-eye view features of the real scene. That is, through the changes in this stage, the information of the panoramic perspective is converted into the information of the overall bird's-eye view, and this transformation is consistent with the map perspective.
[0070] To facilitate understanding of the functions of the road centerline information query model, two optional model training implementation methods are introduced. In the two model training implementation methods introduced below, the trained models are both graph encoder-graph decoder architectures. The graph encoder converts the real-scene image input into a real-scene bird's-eye view feature representation, and the graph decoder predicts and learns map elements through a hierarchical query method. Finally, the decoding outputs the road centerline information, including the point column coordinates on the road centerline and the turn-by-turn navigation TBT direction information of the road centerline. The key to training the road centerline information query model in the technical solution of this application is to enable the trained model to have the ability to output the TBT direction information of the road centerline. For the basic structure of the model, it can be designed and modified with reference to the structure of the MapTRv2 model. The following is a detailed introduction.
[0071] The first model training implementation method:
[0072] Figure 3 A schematic diagram of a process for training a road centerline information query model provided in an embodiment of the present application. Figure 4 A schematic diagram of the structure of a road centerline information query model provided in an embodiment of the present application. Figure 4 As shown, the road centerline information query model includes a graph encoder and a graph decoder, which are connected front and back, that is, the output end of the graph encoder is connected to the input end of the graph decoder.
[0073] like Figure 3 As shown in Figure 1, the process of training the road centerline information query model includes:
[0074] S301, obtaining training samples from a visual perception dataset and inputting them into a model to be trained.
[0075] In this scheme, the vehicle from which the data for training purposes collected in the early stage originated is called a sample vehicle, and the real-life image captured by the surround-view camera of the sample vehicle is called a sample image. As an example, the sample vehicle is equipped with 6 surround-view cameras. The visual perception data set mentioned in S301 may include: sample images captured by the 6 surround-view cameras of the sample vehicle, and internal and external parameter calibration data of each of the 6 surround-view cameras. It should be noted that the sample images captured by the 6 surround-view cameras can be grouped according to the shooting time. For example, sample images of 6 different surround-view cameras captured at the same shooting time can be marked as the same group. The visual perception data also includes map data (which contains the actual coordinates of the point columns of various road traffic markings) and annotation information on the TBT direction of the road centerline in the map data.
[0076] In this step, in order to train the model to be trained, training samples are extracted from the visual perception data set, wherein each training sample has corresponding annotation data. As an example, the training samples include sample images taken by multiple surround view cameras of the sample vehicle at the same time and the internal and external parameter calibration data of each surround view camera. The annotation data includes: the actual coordinates of the point column on the road traffic markings contained in the map data corresponding to the sample image, and the TBT direction annotation information of the road centerline in the road traffic markings.
[0077] That is to say, in the embodiment of the present application, the training samples mainly include the real-life images collected by the image sensors represented by the surround view camera, and the annotation data mainly include the position information of the road traffic markings within the shooting range, and the TBT direction information of the road centerline. Combining the training samples and the annotation data, it can be seen that the purpose of using the model of the present application is to determine the position of the road traffic markings and the TBT direction of the road centerline in the environment that affect the accuracy and safety of navigation guidance through the real-life images.
[0078] TBT navigation is a function that guides users along a selected route in real time. After obtaining the user's current location and destination, the navigation system will plan the best path from the starting point to the end point based on map data and algorithms. During the navigation process, the system will continuously present the current direction of travel to the user in the form of verbal or visual instructions. These instructions usually include whether to turn left, turn right, go straight, and the street name and distance of the next intersection. TBT is currently widely used in GPS navigation devices. In the current related technology, the AR-HUD navigation guide line fails to match the TBT direction information in the navigation information provided by the map manufacturer, which leads to deviations in the display effect of navigation guidance and navigation planning. This solution took this into consideration in the early annotation stage, and the TBT direction was marked for the centerline of the road in the map.
[0079] S302: extract features from sample images in training samples through a graph encoder.
[0080] As mentioned above, the basic structure of the model in this solution includes a graph encoder and a graph decoder connected front and back. When the training sample is input into the model to be trained, the graph encoder extracts features from the sample image. It should be noted that the data input into the graph encoder is not limited to the sample image. Since the advanced driver assistance system (ADAS) of the sample vehicle may also be equipped with various types of auxiliary driving sensors such as radar and lidar, taking lidar as an example, the data input into the graph encoder may also include radar point cloud data collected at the same time as the sample image.
[0081] The map encoder extracts features from the data collected by the above sensors (including surround-view cameras). Because the features come from different sensors (such as surround-view cameras from 6 shooting angles), they need to be further integrated into a unified perspective to facilitate the recognition and modeling of map elements.
[0082] S303: Perform inverse perspective transformation according to the features extracted by the image encoder and the internal and external parameter calibration data to obtain real scene bird's-eye view features.
[0083] In this step, the image encoder can combine the internal and external calibration data of the surround view camera to perform inverse perspective mapping (IPM) based on the features extracted in S302. IPM is an image processing technology that is mainly used to restore the image after perspective transformation (i.e., the image deformation caused by the camera perspective) to an image that is closer to the actual situation or normal perspective. IPM is the inverse process of perspective transformation. It maps the image from the image coordinate system to the world coordinate system through calculation, thereby eliminating the interference and error of perspective effect on image detection and recognition tasks. It is mainly used to restore the actual situation of scenes such as roads and buildings for more accurate measurement, analysis and recognition.
[0084] exist Figure 4In the figure, we can see that two structural blocks are displayed in the image encoder, one before and one after, respectively represented as: 2Dbackbone and PV to BEV. Among them, the Chinese meaning of 2D backbone is two-dimensional backbone network. In the field of deep learning and computer vision, it refers to a network structure used to extract image features. The network can be combined through a series of convolutional layers, pooling layers, non-linear activation functions, etc. The 2D backbone completes the work of S302. In S303, PV to BEV is mainly used to realize the process of converting two-dimensional perspective images such as sample images taken by the surround view camera to bird's-eye view features in three-dimensional space. The meaning of PV to BEV is Perspective View to Bird'sEye View, that is, the conversion from perspective view to bird's-eye view.
[0085] S304: construct a line query vector, a point query vector and a TBT direction query vector based on the real scene bird's-eye view features.
[0086] In the present application, the graph encoder of the model to be trained supports hierarchical query, and the graph decoder hierarchical query refers to querying and processing the input data in a hierarchical manner inside the graph decoder. This hierarchical structure helps the graph decoder to extract features more efficiently, process complex relationships, and generate more accurate outputs. During the decoding process, each layer may focus on different tasks or features, and achieve the final goal in a progressive manner layer by layer. The hierarchical structure can adjust the parameters and complexity of each layer as needed to optimize the overall performance of the decoder. This helps to reduce computational complexity and resource consumption while maintaining high accuracy. The graph decoder hierarchical query is used for modeling and representation of map elements. Through the embedding scheme of hierarchical query, the graph decoder can optimize the representation of map elements through iterative updates and interactive BEV features. In this application, the graph decoder can update the query vector through self-attention and cross-attention mechanisms, so as to better realize the modeling and representation of map elements during model training and model application.
[0087] Instance query and point query in MapTRv2 model are important components of its hierarchical query embedding scheme, which are used to explicitly encode and model map elements. Instance query is used in MapTRv2 to represent and model each independent element in the map (such as lanes, sidewalks, intersections, etc.). Each instance query corresponds to a map element and contains the overall characteristics and attributes of the element. In MapTRv2's hierarchical query embedding scheme, instance queries are higher-level queries, which are responsible for capturing global information such as the category and location of map elements. Point queries are used to represent specific points or nodes in map elements, which together constitute the shape and outline of map elements.
[0088] In MapTRv2, each map element corresponds to a set of point queries, which are explicitly encoded into the model through a hierarchical query embedding scheme. Point queries are responsible for capturing the precise location and local features of each point in a map element, thereby achieving a fine description of the shape of the map element. The hierarchical query embedding scheme adopted by MapTRv2 combines instance queries and point queries to explicitly encode map elements. Specifically, MapTRv2 defines a set of instance-level queries and a set of point-level queries shared by all instances. Each map element corresponds to a set of hierarchical queries, including one instance query and multiple point queries. This hierarchical query embedding scheme enables MapTRv2 to flexibly handle map elements with arbitrary shapes and complex structures. During the training process of the MapTRv2 model, instance-level matching (Instance-LevelMatching) and point-level matching (Point-Level Matching) are performed to find the optimal instance-level allocation and point-level allocation.
[0089] Different from the inherent MapTRv2 implementation, in this application, three types of query vectors are constructed based on the real-life bird's-eye view feature (BEV feature), namely line query vector, point query vector and TBT direction query vector. Among them, the line query vector corresponds to the instance query introduced above, with road traffic markings as independent elements in the map; the point query vector corresponds to the point query introduced above. In addition, the TBT direction query vector is a new part added to the model processing flow. Through the construction of the TBT direction query vector, the flexibility and comprehensiveness of the hierarchical query are increased, and the map elements can be processed from the TBT direction level to capture the TBT direction features of the map elements.
[0090] S305 , the graph decoder performs learning based on the line query vector, the point query vector and the TBT direction query vector, and outputs a prediction result.
[0091] In the embodiment of the present application, the graph decoder includes L self-attention layers and L cross-attention layers that are cross-set; L is an integer greater than 1. L can be set according to actual needs, for example, L=20 can be set, that is, the graph encoder includes 20 self-attention layers and 20 cross-attention layers that are cross-set.
[0092] In this implementation, the graph decoder learns based on the line query vector, the point query vector and the TBT direction query vector, and outputs the prediction result, including: combining the line query vector, the point query vector and the TBT direction query vector into an initial hierarchical query representation; inputting the initial hierarchical query representation into the first self-attention layer, and sequentially learning through L self-attention layers and L cross-attention layers that are cross-set, and outputting the prediction result; wherein the learning result output by the i-th self-attention layer is used as the hierarchical query representation input to the i-th cross-attention layer, and i is any integer in the range of 1 to L. It can be understood that, since the self-attention layer and the cross-attention layer are interlaced, the learning result output by the i-th cross-attention layer is used as the hierarchical query representation input to the i+1-th self-attention layer.
[0093] Figure 4 The structure of the graph decoder with self-attention layers and cross-attention layers interlaced is shown in Figure 1. The combination of line query vectors, point query vectors, and TBT direction vectors into the initial hierarchical query representation is shown on the left side of the first self-attention layer. A single blue square represents a point query vector, a single green square represents a line query vector, and a purple square represents a TBT direction query vector added for each line. Figure 4 The combination of three different colored blocks represents the initial hierarchical query representation.
[0094] At each self-attention layer, the TBT direction query is modified by the point sequence on the line. At each cross-attention layer, the shape of the point sequence on the line is modified by the bird's-eye view features of the real scene. Finally, after multiple self-attention layers and cross-attention layers, the model outputs the coordinates of the point sequence on each line and the predicted results of the TBT direction of each line.
[0095] Specifically, the prediction result includes a first prediction content and a second prediction content. The first prediction content includes the predicted coordinates of the point series on the road traffic marking line, and the second prediction content includes the TBT direction prediction result of the road centerline. Next, S306 is used to determine whether the current model has been trained to achieve the expected effect.
[0096] S306: Based on the difference between the actual coordinates of the point sequence on the corresponding road traffic line in the annotated data and the difference between the second predicted content and the TBT direction annotation information of the corresponding road centerline in the annotated data, determine whether the training cutoff condition is currently met. If yes, proceed to S307; if not, adjust the model parameters and proceed to S301.
[0097] The training cutoff condition can be constructed by different losses. For example, three different loss functions are constructed to supervise the accuracy of the first prediction content and the second prediction content respectively. In addition, the training cutoff condition can also be set as a threshold in combination with the number of iterations of model training.
[0098] If the training cutoff condition is met, it means that the model can currently achieve the expected effect and no further iteration training is required. If the model still cannot meet the training cutoff condition, new sample images and corresponding annotated data can be used for retraining and parameter adjustment.
[0099] S307: Stop training and obtain a road centerline information query model.
[0100] In the first model training implementation described above, by constructing a TBT direction query vector, and combining the difference between the actual coordinates of the point column on the corresponding road traffic marking in the annotation data, and the difference between the second prediction content and the TBT direction annotation information of the corresponding road centerline in the annotation data, the model parameters are adjusted during the training process. The entire training takes into account the value of the accuracy of the TBT direction of the road centerline, and improves the accuracy and comprehensiveness of the road centerline information provided after the model training is completed.
[0101] The second model training implementation method:
[0102] Figure 5 A schematic diagram of a process for training another road centerline information query model provided in an embodiment of the present application. Figure 6 This is a schematic diagram of another road centerline information query model provided in an embodiment of the present application. Figure 6 As shown in FIG. 1 , the road centerline information query model includes a graph encoder, a graph decoder, and a multilayer perceptron (MLP) connected in sequence. The output of the graph encoder is connected to the input of the graph decoder, and the output of the graph decoder is connected to the input of the multilayer perceptron.
[0103] like Figure 5 As shown in Figure 1, the process of training the road centerline information query model includes:
[0104] S501, obtaining training samples from a visual perception dataset and inputting them into a model to be trained.
[0105] The training samples have corresponding annotation data; the training samples include sample images taken by the surround-view camera of the sample vehicle and the internal and external parameter calibration data of the surround-view camera; the annotation data include the actual coordinates of the point column on the road traffic marking contained in the map data corresponding to the sample image, and the TBT direction annotation information of the center line of the road in the road traffic marking.
[0106] S502: Extract features from sample images in training samples using a graph encoder.
[0107] S503: Perform inverse perspective transformation according to the features extracted by the image encoder and the internal and external parameter calibration data to obtain real scene bird's-eye view features.
[0108] For the specific implementation of S501 to S503, please refer to the relevant content of S301 to S303 introduced above. Since the implementation methods are basically the same, they will not be repeated here.
[0109] S504: construct a line query vector and a point query vector based on the real scene bird's-eye view features.
[0110] In the second training model implementation method introduced in this embodiment, two types of query vectors are constructed, namely line query vectors and point query vectors. The structure of the MapTRv2 model can be applied to this application. Compared with the first training model implementation method, the model modification amount is smaller, the model training time is reduced, and the difficulty of model training is reduced.
[0111] S505 : The graph decoder performs learning based on the line query vector and the point query vector to obtain an N-dimensional output result.
[0112] In the technical solution of the present application, the line query vector and the point query vector can be first combined into an initial hierarchical query representation. Figure 6 In the model shown in , individual blue squares represent point query vectors and individual green squares represent line query vectors. Figure 6 The combination of blue and green blocks in the figure represents the initial hierarchical query representation.
[0113] The structure of the graph decoder can be similar to that described in the previous embodiment, and the graph decoder has L self-attention layers and L cross-attention layers that are cross-set. Finally, the last cross-attention layer of the graph decoder outputs an N-dimensional output result. The output result includes a first prediction content, and the first prediction content includes the predicted coordinates of the point series on the road traffic marking line.
[0114] S506 , mapping the N-dimensional output result into an 8-dimensional TBT direction prediction vector through a multi-layer perceptron.
[0115] TBT directions can generally be divided into eight types, such as straight direction, left front direction, right front direction, left turn direction, right turn direction, left U-turn direction, left rear direction and right rear direction. In the present application, the mapping of dimensions is realized by connecting a multi-layer perceptron at the output end of the graph decoder, and the N-dimensional output result is mapped to an 8-dimensional TBT direction prediction vector, wherein each dimension corresponds to a TBT direction. For example, [1 0 0 0 0 0 0 0] represents the straight direction in the TBT direction, and [0 1 0 0 00 0 0] represents the left front direction in the TBT direction. In addition, in other possible implementations, the values of each dimension in the TBT direction prediction vector can also be represented by a probability value between 0 and 1. For example, it is expressed as [0.05 0.65 0.1 0.050.05 0.05 0.01 0.04]. In the specific implementation, a 2-layer MLP can be set at the output end of the graph decoder of the model to be trained. Next, S507 is used to determine whether the current model has been trained to achieve the expected effect.
[0116] S507. Based on the difference between the actual coordinates of the point column on the corresponding road traffic marking in the annotation data and the difference between the 8-dimensional TBT direction prediction vector and the TBT direction annotation information of the corresponding road centerline in the annotation data, determine whether the training cutoff condition has been met. If yes, execute S508; if not, adjust the model parameters and enter S501.
[0117] S508: Stop training and obtain a road centerline information query model.
[0118] In the second model training implementation method introduced above, by adding MLP at the end of the model, the prediction result can be mapped from N dimensions to 8 dimensions, so as to obtain the predicted TBT direction, which is convenient for comparing it with the annotated TBT direction. During the training process, the model parameters are adjusted based on the difference between the actual coordinates of the point column on the corresponding road traffic marking in the annotated data and the difference between the 8-dimensional TBT direction prediction vector and the TBT direction annotation information of the corresponding road centerline in the annotated data. The entire training takes into account the value of the accuracy of the TBT direction of the road centerline, and improves the accuracy and comprehensiveness of the road centerline information provided after the model training is completed. In addition, the amount of changes to the model is relatively small, especially there is no need to modify the structure of the graph encoder and graph decoder of the MapTRv2 model, thereby improving the efficiency of model training and saving model training time.
[0119] In the first and second model training implementations described above, the self-attention layer used in the graph decoder can obtain the relationship between lines and between points within a line through the attention mechanism between instances and within instances. The advantage of the self-attention layer designed in this way is that it has a small amount of calculation and requires less data.
[0120] In the first and second model training implementations described above, the cross attention layer used in the graph decoder can use both the real-scene bird's-eye view features and the features in the real-scene image captured by the surround view camera. Compared with the learning scheme that only uses the real-scene bird's-eye view features or the features in the real-scene image captured by the surround view camera, its advantage is that no information is lost.
[0121] Combining the two implementation methods of training the road centerline information query model introduced above, the function of the model can be understood more clearly and intuitively. The image obtained in S201 is used as the input of the road centerline information query model. After the road centerline information query model processes and analyzes the image, it can output the information of multiple road centerlines contained in the image, including the point column coordinates on the road centerline and the turn-by-turn navigation TBT direction information of the road centerline.
[0122] S203: According to the navigation information of the vehicle and the information of the multiple road center lines, determine a target road center line that matches the TBT direction information in the navigation information from the multiple road center lines.
[0123] In order to enable the generated AR-HUD navigation guide line to achieve accurate and effective navigation guidance functions for the relevant information of the multiple road center lines output by the S202 model, it is also necessary to determine a target road center line from the multiple road center lines. In this stage, the determination of the target road center line can be completed in combination with the navigation information.
[0124] In the navigation system, the destination information and current location provided by the user (driver or passenger) are generally used to plan a recommended driving path for the vehicle and the driver. During the driving process, the driver is also provided with driving tips through TBT direction information, such as reminding the driver to take corresponding driving operations through voice prompts and the like. The navigation information also includes TBT direction information recommended to the driver based on the planned driving path. In order to match a target road centerline, in this application, the road centerlines consistent with the TBT direction information in the navigation information among multiple road centerlines are first added to the set of road centerlines to be selected. In this way, all road centerlines in the set of road centerlines to be selected are consistent with the TBT direction in the navigation information. Through this step, the initial screening of the road centerlines is completed, and the initial screening results are retained in the set of road centerlines to be selected. It can be understood that if only one road centerline is included in the road centerlines to be selected, the only one road centerline can be confirmed as the target road centerline. If the number of road centerlines included in the set of road centerlines to be selected is greater than 1, it is necessary to further combine the vehicle's driving direction in that year and / or the vehicle's current location information for further detailed screening.
[0125] If the number of road centerlines included in the set of road centerlines to be selected is greater than 1, a target road centerline is determined from the set of road centerlines to be selected based on the vehicle's current driving direction and / or the vehicle's current position information. The specific implementation method may be: a road centerline in the set of road centerlines to be selected that is consistent with the vehicle's current driving direction and closest to the vehicle's current position is used as the target road centerline.
[0126] At this stage, the following expression can be used to calculate, and from the set of candidate road centerlines, the road centerline that makes the calculated value of the expression reach the minimum is used as the target road centerline.
[0127] ||dir(argmin d(p,p car ))-dir car ||+λmin d(p,p car )
[0128] Where d(p,pcar) is the distance between the point on the road centerline and the current position of the vehicle; dir(argmin d(p,p car )) is the center line of the road so that d(p,p car ) The direction of the point where the minimum is reached relative to the vehicle's current position; dir car is the current driving direction of the vehicle; the |||| symbol indicates the norm. min d(p,p car) is the minimum distance between a point on the road centerline and the current position of the vehicle; λ is the minimum distance between the point on the road centerline and the current position of the vehicle; car )’s configuration weight.
[0129] The smaller the distance between the point on the road centerline and the current position of the vehicle, and the smaller the distance between the point on the road centerline and the current position of the vehicle, the smaller the distance between the point on the road centerline and the current position of the vehicle. car ) reaches the minimum. The smaller the deviation between the direction of the vehicle's current position and the vehicle's current driving direction, the smaller the calculated value of the above expression. Therefore, combining the above expression, a target road centerline that best matches and is closest to the vehicle's current position and driving direction can be determined from the set of candidate road centerlines.
[0130] S204: Generate an AR-HUD navigation guide line using the target road centerline.
[0131] In the technical solution of the present application, after the target road centerline is determined through S201 to S203, the AR-HUD navigation guide line can be rendered and drawn according to the target road centerline and projected for the user to watch.
[0132] The technical solution of the present application provides an AR-HUD navigation guide line generation method, device, processor and storage medium. In the AR-HUD navigation guide line generation method, firstly, a real scene image of the vehicle is obtained; then, the obtained image is used as the input of the road centerline information query model, and the road centerline information query model processes and analyzes the image and outputs the information of multiple road centerlines contained in the image; wherein, the information of the road centerline includes the point column coordinates on the road centerline and the turn-by-turn navigation TBT direction information of the road centerline. Thereafter, according to the navigation information of the vehicle and the information of the multiple road centerlines, the target road centerline that matches the TBT direction information in the navigation information is determined from the multiple road centerlines; finally, the AR-HUD navigation guide line is generated using the target road centerline. The information of the road centerline output by the road centerline query model contains TBT direction information, and the TBT direction information of these road centerlines is matched with the TBT direction information in the navigation information. The target road centerline finally determined, on the one hand, conforms to the position of the road centerline in the real scene, and on the other hand, can better demonstrate the navigation guidance function from the position. Therefore, the generated AR-HUD navigation guide line is more closely aligned with the real scene. Compared with the navigation guide line generated by the prior art, it is more accurate in guiding the driver's driving lane and driving direction, and has a more obvious auxiliary effect on driving safety.
[0133] Based on the AR-HUD navigation guide line generation method described in the above method embodiment, the present application also provides an AR-HUD navigation guide line generation device. Figure 7 Provide explanation.
[0134] Figure 7 This is a schematic diagram of the structure of an AR-HUD navigation guide line generation device provided in an embodiment of the present application. Figure 7 As shown, the AR-HUD navigation guide line generating device 70 includes:
[0135] An image acquisition module 71 is used to acquire a real-life image of a vehicle;
[0136] The centerline information acquisition module 72 is used to use the image as an input of a road centerline information query model, and the road centerline information query model processes and analyzes the image and then outputs information of multiple road centerlines contained in the image; the road centerline information includes the coordinates of the points on the road centerline and the turn-by-turn navigation TBT direction information of the road centerline;
[0137] A centerline matching module 73, configured to determine, from the plurality of road centerlines, a target road centerline that matches the TBT direction information in the navigation information, based on the navigation information of the vehicle and the information of the plurality of road centerlines;
[0138] The guide line generation module 74 is used to generate an AR-HUD navigation guide line using the target road center line.
[0139] The road centerline information output by the road centerline query model contains TBT direction information. The TBT direction information of these road centerlines is matched with the TBT direction information in the navigation information. The target road centerline finally determined not only conforms to the position of the road centerline in the real scene, but also can better demonstrate the navigation guidance function from the position. Therefore, the generated AR-HUD navigation guide line has a higher degree of fit with the real scene. Compared with the navigation guide line generated by the existing technology, it has a more accurate guidance effect on the driver's driving lane and driving direction, and a more obvious auxiliary effect on driving safety.
[0140] In an optional implementation, the AR-HUD navigation guide line generating device 70 further includes a model training module 75, which is used to train a road centerline information query model. The model training module 75 is specifically used to:
[0141] A training sample is obtained from a visual perception data set and input into a model to be trained; the training sample has corresponding annotation data; the training sample includes a sample image taken by a surround view camera of a sample vehicle and internal and external parameter calibration data of the surround view camera; the annotation data includes the actual coordinates of a point column on a road traffic marking line contained in the map data corresponding to the sample image, and TBT direction annotation information of a road centerline in the road traffic marking line; the model to be trained includes a graph encoder and a graph decoder connected front and back;
[0142] Performing feature extraction on the sample image in the training sample by using the image encoder;
[0143] Performing inverse perspective transformation according to the features extracted by the image encoder and the internal and external parameter calibration data to obtain real scene bird's-eye view features;
[0144] Constructing a line query vector, a point query vector and a TBT direction query vector based on the real scene bird's-eye view feature;
[0145] The graph decoder performs learning based on the line query vector, the point query vector and the TBT direction query vector, and outputs a prediction result; the prediction result includes a first prediction content and a second prediction content, the first prediction content includes the predicted coordinates of the point column on the road traffic marking line, and the second prediction content includes the TBT direction prediction result of the road centerline;
[0146] Based on the difference between the first prediction content and the actual coordinates of the point column on the corresponding road traffic marking line in the annotated data, as well as the difference between the second prediction content and the TBT direction annotation information of the corresponding road centerline in the annotated data, it is determined whether the training cutoff condition has been met. If so, the training is stopped to obtain a road centerline information query model; if not, after adjusting the parameters of the model to be trained, the steps of obtaining training samples from the visual perception data set and inputting them into the model to be trained are repeated until the training cutoff condition is met.
[0147] In an optional implementation, the graph decoder includes L self-attention layers and L cross-attention layers arranged in a cross-arrangement; L is an integer greater than 1;
[0148] The model training module 75 is specifically used for:
[0149] combining the line query vector, the point query vector and the TBT direction query vector into an initial hierarchical query representation;
[0150] The initial hierarchical query representation is input into the first self-attention layer, and the prediction result is output through the learning of L cross-attention layers and L cross-attention layers in sequence; the learning result output by the i-th self-attention layer is used as the hierarchical query representation input to the i-th cross-attention layer, and i is any integer in the range of 1 to L.
[0151] In an optional implementation, the process of training the model by the model training module 75 is different from the above. The model training module 75 is specifically used for:
[0152] A training sample is obtained from a visual perception data set and input into a model to be trained; the training sample has corresponding annotation data; the training sample includes a sample image taken by a surround view camera of a sample vehicle and internal and external parameter calibration data of the surround view camera; the annotation data includes the actual coordinates of a point column on a road traffic marking line contained in the map data corresponding to the sample image, and TBT direction annotation information of a road centerline in the road traffic marking line; the model to be trained includes a graph encoder, a graph decoder and a multi-layer perceptron connected in sequence;
[0153] Performing feature extraction on the sample image in the training sample by using the image encoder;
[0154] Performing inverse perspective transformation according to the features extracted by the image encoder and the internal and external parameter calibration data to obtain real scene bird's-eye view features;
[0155] Constructing a line query vector and a point query vector based on the real scene bird's-eye view feature;
[0156] The graph decoder performs learning based on the line query vector and the point query vector to obtain an N-dimensional output result; the N-dimensional output result includes a first prediction content, and the first prediction content includes predicted coordinates of a point sequence on a road traffic marking line;
[0157] Mapping the N-dimensional output result into an 8-dimensional TBT direction prediction vector through the multi-layer perceptron; each dimension in the 8-dimensional TBT direction prediction vector corresponds to a TBT direction;
[0158] Based on the difference between the first prediction content and the actual coordinates of the point column on the corresponding road traffic marking in the annotated data, as well as the difference between the 8-dimensional TBT direction prediction vector and the TBT direction annotation information of the corresponding road centerline in the annotated data, it is determined whether the current training cutoff condition has been met. If so, the training is stopped to obtain a road centerline information query model; if not, after adjusting the parameters of the model to be trained, the steps of obtaining training samples from the visual perception data set and inputting them into the model to be trained are repeated until the training cutoff condition is met.
[0159] In an optional implementation, the centerline matching module 73 is specifically configured to:
[0160] Adding the road center lines among the plurality of road center lines that are consistent with the TBT direction information in the navigation information to the set of road center lines to be selected;
[0161] If the number of road centerlines included in the set of road centerlines to be selected is greater than 1, a target road centerline is determined from the set of road centerlines to be selected according to the current driving direction of the vehicle and / or the current position information of the vehicle.
[0162] In an optional implementation, determining a target road centerline from the set of candidate road centerlines according to the current driving direction of the vehicle and / or the current position information of the vehicle includes:
[0163] A road centerline in the set of candidate road centerlines that is consistent with the current driving direction of the vehicle and closest to the current position of the vehicle is used as the target road centerline.
[0164] In an optional implementation, the centerline matching module 73 is specifically configured to:
[0165] From the set of road centerlines to be selected, a road centerline that minimizes the calculated value of the expression is used as the target road centerline; the expression is:
[0166] ||dir(argmin d(p,p car ))-dir car ||+λmin d(p,p car )
[0167] Wherein, d(p, pcar) is the distance between the point on the center line of the road and the current position of the vehicle;
[0168] The dir(argmin d(p,p car )) is the center line of the road so that d(p,p car ) reaches a minimum point relative to the direction of the vehicle's current position;
[0169] The dir car is the current driving direction of the vehicle;
[0170] The min d(p,p car ) is the minimum distance between a point on the road centerline and the current position of the vehicle;
[0171] The λ is the value for min d(p,p car )’s configuration weight.
[0172] Based on the method embodiments and device embodiments described above, the present application further provides a processor for running a computer program, which, when running, executes the AR-HUD navigation guide line generation method described in any implementation manner in the method embodiments.
[0173] Based on the method embodiments and device embodiments introduced above, the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the AR-HUD navigation guide line generation method described in any implementation of the method embodiment is implemented.
[0174] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.
[0175] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for generating AR-HUD navigation guide lines, It is characterized in that include: Acquire a real-life image of the vehicle; The image is used as an input of a road centerline information query model, and the road centerline information query model processes and analyzes the image and outputs information of multiple road centerlines contained in the image; the road centerline information includes point column coordinates on the road centerline and turn-by-turn navigation TBT direction information of the road centerline; According to the navigation information of the vehicle and the information of the multiple road center lines, determining a target road center line that matches the TBT direction information in the navigation information from the multiple road center lines; The target road centerline is used to generate an AR-HUD navigation guide line.
2. The method according to claim 1, It is characterized in that The training steps of the road centerline information query model include: A training sample is obtained from a visual perception data set and input into a model to be trained; the training sample has corresponding annotation data; the training sample includes a sample image taken by a surround view camera of a sample vehicle and internal and external parameter calibration data of the surround view camera; the annotation data includes the actual coordinates of a point column on a road traffic marking line contained in the map data corresponding to the sample image, and TBT direction annotation information of a road centerline in the road traffic marking line; the model to be trained includes a graph encoder and a graph decoder connected front and back; Performing feature extraction on the sample image in the training sample by using the image encoder; Performing inverse perspective transformation according to the features extracted by the image encoder and the internal and external parameter calibration data to obtain real scene bird's-eye view features; Constructing a line query vector, a point query vector and a TBT direction query vector based on the real scene bird's-eye view feature; The graph decoder performs learning based on the line query vector, the point query vector and the TBT direction query vector, and outputs a prediction result; the prediction result includes a first prediction content and a second prediction content, the first prediction content includes the predicted coordinates of the point column on the road traffic marking line, and the second prediction content includes the TBT direction prediction result of the road centerline; Based on the difference between the first prediction content and the actual coordinates of the point column on the corresponding road traffic marking line in the annotated data, as well as the difference between the second prediction content and the TBT direction annotation information of the corresponding road centerline in the annotated data, it is determined whether the training cutoff condition has been met. If so, the training is stopped to obtain a road centerline information query model; if not, after adjusting the parameters of the model to be trained, the steps of obtaining training samples from the visual perception data set and inputting them into the model to be trained are repeated until the training cutoff condition is met.
3. The method according to claim 2, It is characterized in that The graph decoder includes L self-attention layers and L cross-attention layers arranged crosswise; L is an integer greater than 1; The graph decoder performs learning based on the line query vector, the point query vector and the TBT direction query vector, and outputs a prediction result, including: combining the line query vector, the point query vector and the TBT direction query vector into an initial hierarchical query representation; The initial hierarchical query representation is input into the first self-attention layer, and the prediction result is output through the learning of L cross-attention layers and L cross-attention layers in sequence; the learning result output by the i-th self-attention layer is used as the hierarchical query representation input to the i-th cross-attention layer, and i is any integer in the range of 1 to L.
4. The method according to claim 2, It is characterized in that The training steps of the road centerline information query model include: A training sample is obtained from a visual perception data set and input into a model to be trained; the training sample has corresponding annotation data; the training sample includes a sample image taken by a surround view camera of a sample vehicle and internal and external parameter calibration data of the surround view camera; the annotation data includes the actual coordinates of a point column on a road traffic marking line contained in the map data corresponding to the sample image, and TBT direction annotation information of a road centerline in the road traffic marking line; the model to be trained includes a graph encoder, a graph decoder and a multi-layer perceptron connected in sequence; Performing feature extraction on the sample image in the training sample by using the image encoder; Performing inverse perspective transformation according to the features extracted by the image encoder and the internal and external parameter calibration data to obtain real scene bird's-eye view features; Constructing a line query vector and a point query vector based on the real scene bird's-eye view feature; The graph decoder performs learning based on the line query vector and the point query vector to obtain an N-dimensional output result; the N-dimensional output result includes a first prediction content, and the first prediction content includes predicted coordinates of a point sequence on a road traffic marking line; Mapping the N-dimensional output result into an 8-dimensional TBT direction prediction vector through the multi-layer perceptron; each dimension in the 8-dimensional TBT direction prediction vector corresponds to a TBT direction; Based on the difference between the first prediction content and the actual coordinates of the point column on the corresponding road traffic marking in the annotated data, as well as the difference between the 8-dimensional TBT direction prediction vector and the TBT direction annotation information of the corresponding road centerline in the annotated data, it is determined whether the current training cutoff condition has been met. If so, the training is stopped to obtain a road centerline information query model; if not, after adjusting the parameters of the model to be trained, the steps of obtaining training samples from the visual perception data set and inputting them into the model to be trained are repeated until the training cutoff condition is met.
5. The method according to any one of claims 1 to 4, It is characterized in that The step of determining, based on the navigation information of the vehicle and the information of the multiple road center lines, a target road center line that matches the TBT direction information in the navigation information from the multiple road center lines comprises: Adding the road center lines among the plurality of road center lines that are consistent with the TBT direction information in the navigation information to the set of road center lines to be selected; If the number of road centerlines included in the set of road centerlines to be selected is greater than 1, a target road centerline is determined from the set of road centerlines to be selected according to the current driving direction of the vehicle and / or the current position information of the vehicle.
6. The method according to claim 5, It is characterized in that The step of determining a target road centerline from the set of road centerlines to be selected according to the current driving direction of the vehicle and / or the current position information of the vehicle includes: A road centerline in the set of candidate road centerlines that is consistent with the current driving direction of the vehicle and closest to the current position of the vehicle is used as the target road centerline.
7. The method according to claim 5, It is characterized in that The step of determining a target road centerline from the set of road centerlines to be selected according to the current driving direction of the vehicle and / or the current position information of the vehicle includes: From the set of road centerlines to be selected, a road centerline that minimizes the calculated value of the expression is used as the target road centerline; the expression is: ||dir(argmin d(p,p car ))-you car ||+λmin d(p,p car ) Wherein, d(p, pcar) is the distance between the point on the center line of the road and the current position of the vehicle; The dir(argmin d(p,p car )) is the center line of the road so that d(p,p car ) reaches a minimum point relative to the direction of the vehicle's current position; The dir car is the current driving direction of the vehicle; The min d(p,p car ) is the minimum distance between a point on the road centerline and the current position of the vehicle; The λ is the value for min d(p,p car )’s configuration weight.
8. An AR-HUD navigation guide line generation device, It is characterized in that include: An image acquisition module, used to acquire a real-scene image of a vehicle; A centerline information acquisition module, used to use the image as an input of a road centerline information query model, and the road centerline information query model processes and analyzes the image and then outputs information of multiple road centerlines contained in the image; the road centerline information includes the coordinates of the point columns on the road centerline and the turn-by-turn navigation TBT direction information of the road centerline; A centerline matching module, configured to determine, from the multiple road centerlines, a target road centerline that matches the TBT direction information in the navigation information based on the navigation information of the vehicle and the information of the multiple road centerlines; A guide line generation module is used to generate an AR-HUD navigation guide line using the target road center line.
9. A processor, It is characterized in that Used to run a computer program, which, when running, executes the AR-HUD navigation guide line generation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the AR-HUD navigation guide line generation method according to any one of claims 1 to 7 is implemented.
