A cognitive-driven intelligent vehicle visual positioning method and system
By collecting multimodal driving data and driver attention information, and combining visual and cognitive feature fusion, a multi-task loss function is constructed, which solves the problem of decreased visual positioning accuracy in complex urban environments and achieves high-precision, low-cost vehicle visual positioning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2026-04-21
- Publication Date
- 2026-07-31
AI Technical Summary
Existing visual positioning methods rely on high-precision maps and sensors in complex urban environments such as densely built-up areas, tunnel entrances and exits, and underground/semi-underground roads, resulting in decreased positioning accuracy and high costs, and are unable to quickly adapt to new environments.
A cognitive-driven intelligent vehicle visual localization method is adopted. By collecting multimodal driving data, combining driver eye-tracking data and Gaussian kernel mapping, visual and cognitive features are extracted using image encoders and cognitive encoders. Spatiotemporal fusion is performed by combining visual spatial self-attention mechanism and cognitive-driven cross-attention mechanism to construct a multi-task joint loss function, update encoder and decoder parameters, and realize vehicle visual localization.
High-precision vehicle visual positioning is achieved without relying on high-precision maps and high-precision sensors. It has strong applicability, low cost, and good practicality and application prospects.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent vehicle perception and positioning technology, and in particular to a cognitively driven intelligent vehicle visual positioning method and system. Background Technology
[0002] With the rapid development of science and technology, artificial intelligence has had a profound impact on the field of autonomous driving. The Navigate on Autopilot (NOA) system has made a significant contribution to improving travel efficiency and reducing traffic accidents.
[0003] In the field of urban NOA (Noise, Arrival, and Orientation), the goal of intelligent vehicle localization is to accurately estimate vehicle pose, which is an essential foundation and prerequisite for subsequent target planning and navigation. However, in complex urban environments such as densely built-up areas, tunnel entrances and exits, and underground / semi-underground roads, the availability of satellite positioning signals is poor, necessitating reliance on visual signals for positioning. Traditional visual positioning methods rely on high-precision maps and high-precision sensors such as LiDAR and combined inertial navigation systems. These methods are typically costly and are susceptible to factors such as map update delays, unstable visual features, and abnormal sensor drift in complex urban scenarios, leading to decreased positioning accuracy.
[0004] Therefore, there is an urgent need to develop a visual positioning method and system for intelligent vehicles to improve visual positioning accuracy without relying on high-precision maps and high-precision sensors, which is of great significance for improving the safety and interpretability of autonomous driving. Summary of the Invention
[0005] The purpose of this application is to provide a visual positioning method and system for intelligent vehicles, which can effectively solve the problems of existing visual positioning methods relying on high-precision maps and high-precision sensors, being unable to quickly locate in new environments, and having high costs and high energy consumption.
[0006] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a cognitive-driven visual localization method for intelligent vehicles, comprising: Collect vehicle multimodal driving data, which includes vehicle front-view camera image sequences, real vehicle position, vehicle motion state sequences, and real vehicle orientation angles; Driver eye movement fixation data is acquired and Gaussian kernel mapping is performed to obtain the driver attention ground map; Visual features are extracted based on an image encoder; Cognitive features are extracted based on a cognitive encoder; these cognitive features are obtained by fusing motion features and semantic features. Based on the visual spatial self-attention mechanism and the cognitive-driven cross-attention mechanism, the visual features and cognitive features are spatiotemporally fused to obtain spatiotemporally fused features. The spatiotemporal fusion features are input into the attention decoder to obtain the driver attention prediction map; The motion features are input into the state decoder to obtain the predicted vehicle position coordinates and predicted vehicle orientation angle in the global coordinate system. A multi-task joint loss function is constructed based on the driver attention ground truth map, driver attention prediction map, real vehicle position coordinates, predicted vehicle position coordinates, real vehicle orientation angle, and predicted vehicle orientation angle. The weight parameters in the image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder, and state decoder are updated based on the multi-task joint loss function until a preset number of iterations is reached. Based on the updated image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder, and state decoder, the position coordinates and orientation angle of the vehicle are predicted to achieve vehicle visual positioning.
[0007] Optionally, the cognitive encoder includes a motion encoder and a semantic encoder.
[0008] Optionally, based on the cognitive encoder, cognitive features are extracted, specifically including the following steps: Motion features are extracted from the vehicle motion state sequence based on the motion encoder; Based on the semantic encoder, semantic features are extracted from the vehicle forward-looking camera image sequence; The motion features and semantic features are fused by element-wise addition to obtain cognitive features.
[0009] Optionally, extracting semantic features from the vehicle's forward-looking camera image sequence based on the semantic encoder specifically includes the following steps: Based on a semantic segmentation network, a vehicle front-view semantic image sequence is extracted from the vehicle front-view camera image sequence; Based on the semantic encoder, semantic features are extracted from the vehicle forward-looking semantic image sequence.
[0010] Optionally, based on the visual-spatial self-attention mechanism and the cognitive-driven cross-attention mechanism, the visual features and cognitive features are spatiotemporally fused to obtain spatiotemporally fused features, specifically including the following steps: The visual features are calculated based on the visual spatial self-attention mechanism to obtain the visual spatial features; Based on the cognitive-driven cross-attention mechanism, the cognitive features and the visual spatial features are calculated to obtain spatiotemporal fusion features.
[0011] Optionally, the visual features are calculated based on the visual spatial self-attention mechanism to obtain visual spatial features, specifically including the following steps: The visual features are projected onto a linear feature space to obtain a first query matrix, a first key matrix, and a first value matrix. Based on the first query matrix and the first key matrix, calculate the first correlation weight matrix between different pixel regions in the vehicle front-view camera image; The first correlation weight matrix and the first value matrix are subjected to feature weight aggregation to obtain the first aggregated feature; The first aggregated features are nonlinearly mapped based on the first multilayer perceptron to obtain visual spatial features.
[0012] Optionally, based on a cognitive-driven cross-attention mechanism, the cognitive features and the visual-spatial features are calculated to obtain spatiotemporal fusion features, specifically including the following steps: The visual spatial features are projected onto a linear feature space to obtain a second query matrix; The cognitive features are projected onto a linear feature space to obtain a second key matrix and a second value matrix. Based on the second query matrix and the second key matrix, calculate the second relevance weight matrix of each state in the historical cognitive features to the current positioning task; The second correlation weight matrix and the second value matrix are subjected to feature weight aggregation to obtain the second aggregated feature; The second aggregated feature is nonlinearly mapped based on the second multilayer perceptron to obtain the spatiotemporal fusion feature.
[0013] Optionally, the spatiotemporal fusion features are input into the attention decoder to obtain the driver attention prediction map, specifically including the following steps: The spatiotemporal fusion features are upsampled using bilinear interpolation to obtain the interpolated and amplified fusion features; The interpolated and amplified fused features are decoded using an attention decoder to obtain the decoded fused features. A learnable two-dimensional Gaussian kernel is constructed based on local spatial coordinates, where the local spatial coordinates represent the horizontal and vertical pixel offsets of each point within the two-dimensional Gaussian kernel relative to the kernel center. Based on the two-dimensional Gaussian kernel, the decoded fused features are smoothed and filtered to obtain the smoothed fused features. The smoothed and filtered fused features are normalized in the spatial dimension to generate a driver attention prediction map.
[0014] Optionally, a multi-task joint loss function is constructed based on the driver attention ground truth map, driver attention prediction map, real vehicle position coordinates, predicted vehicle position coordinates, real vehicle orientation angle, and predicted vehicle orientation angle, specifically including the following steps: Calculate the vehicle positioning loss based on the actual vehicle position coordinates and the predicted vehicle position coordinates; Calculate the vehicle orientation loss based on the actual vehicle orientation angle and the predicted vehicle orientation angle; The driver's attention loss is calculated based on the driver attention truth map and the driver attention prediction map. Based on the vehicle positioning loss, vehicle orientation loss, and driver attention loss, a multi-task joint loss function is constructed.
[0015] Secondly, this application provides a cognitive-driven intelligent vehicle vision positioning system, comprising: The acquisition module is used to acquire multimodal driving data of the vehicle, including the vehicle's forward-looking camera image sequence, the actual vehicle position, the vehicle motion state sequence, and the actual vehicle orientation angle. The driver attention ground truth map determination module is used to acquire driver eye movement fixation data and perform Gaussian kernel mapping to obtain the driver attention ground truth map. The visual feature extraction module is used to extract visual features based on the image encoder; A cognitive feature extraction module is used to extract cognitive features based on a cognitive encoder; the cognitive features are obtained by fusing motion features and semantic features. The spatiotemporal fusion module is used to perform spatiotemporal fusion of the visual features and cognitive features based on the visual spatial self-attention mechanism and the cognitive-driven cross-attention mechanism to obtain spatiotemporal fused features. The driver attention prediction map determination module is used to input the spatiotemporal fusion features into the attention decoder to obtain the driver attention prediction map; The prediction module is used to input the motion features into the state decoder to obtain the predicted vehicle position coordinates and the predicted vehicle orientation angle in the global coordinate system. The multi-task joint loss function construction module is used to construct a multi-task joint loss function based on the driver attention truth map, driver attention prediction map, real vehicle position coordinates, predicted vehicle position coordinates, real vehicle orientation angle and predicted vehicle orientation angle. The update module is used to update the weight parameters in the image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder and state decoder based on the multi-task joint loss function, until a preset number of iterations is reached; The visual positioning module is used to predict the position coordinates and orientation angle of the vehicle based on the updated image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder and state decoder, so as to realize vehicle visual positioning.
[0016] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a cognitive-driven intelligent vehicle visual positioning method and system. By collecting multimodal driving data of the vehicle, driver spatial cognition and driver attention information are introduced as additional supervision signals. Finally, updates are guided by a multi-task joint loss function. High-precision vehicle visual positioning is achieved without relying on high-precision maps and high-precision sensors. It has strong applicability, low cost, and good practicality and application prospects. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 A flowchart illustrating the cognitive-driven intelligent vehicle visual positioning method provided in an embodiment of this application; Figure 2 A schematic diagram of the overall logical architecture of the cognitive-driven intelligent vehicle visual positioning method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the KL divergence between the driver attention truth map and the driver attention prediction map provided in the embodiments of this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0021] like Figure 1 As shown, a cognitive-driven visual localization method for intelligent vehicles is provided, including the following steps: S1: Collect multimodal driving data of the vehicle, including the vehicle's forward-view camera image sequence, the actual vehicle position, the vehicle motion state sequence, and the actual vehicle orientation angle.
[0022] In a specific embodiment, the vehicle's forward-looking camera image sequence Data acquired by a forward-facing monocular camera mounted on the vehicle's roof bracket, defined as... , where the sequence length Set to 50 frames per second. For the first t A frame of a vehicle's forward-facing camera image. For the real number field, and These represent the height and width of the vehicle's forward-facing camera image, respectively, and are both set to 448 pixels after cropping and resizing.
[0023] In a specific embodiment, the actual vehicle position is obtained by a high-precision global satellite navigation system installed on the vehicle's roof with an unobstructed view, denoted as [missing information]. ,in The x-coordinate represents the actual location of the vehicle. The vertical coordinate represents the actual position of the vehicle.
[0024] In a specific embodiment, the vehicle motion state sequence and the actual vehicle heading angle The data is acquired jointly by an Inertial Measurement Unit (IMU) installed near the vehicle's center of gravity and chassis data collected via the Controller Area Network (CAN) bus. The vehicle motion state sequence is defined as follows: ,in For sequence length, For the real number field, For the first t The frame shows the vehicle's motion state. , for The linear velocity of the vehicle at any given moment. The actual vehicle orientation angle relative to the global coordinate system, with a range set within... .
[0025] S2: Obtain driver eye movement fixation data and perform Gaussian kernel mapping to obtain the driver attention ground truth map.
[0026] In a specific simulated driving environment or a real vehicle environment, the driver's gaze is first tracked using infrared corneal reflection to directly obtain the original two-dimensional discrete gaze coordinates in the current driving scene image. ,in Here are the horizontal pixel coordinates of the gaze point in the current driving scene image. The vertical pixel coordinates of the gaze point in the current driving scene image.
[0027] Subsequently, Gaussian kernels were used to evaluate the discrete gaze coordinates. Spatial mapping is performed. This mapping process aims to simulate the real-world physiological distribution characteristics of human vision, also known as the foveal visual mechanism. Because when humans observe a scene, they focus only on the focal area, which is the clearest, while the peripheral areas are relatively blurry. Therefore, by introducing a Gaussian kernel, the originally discrete fixation points can be visualized. The value is transformed into a continuous two-dimensional probability distribution map that conforms to the characteristics of human visual attenuation, and this map is used as the true value map of driver attention. The calculation formula is as follows: in, This represents the coordinates of a pixel in the truth map of driver attention. This represents the diffusion coefficient corresponding to the human foveal angle, in this embodiment. The value is 0.1.
[0028] S3: Based on an image encoder, extract visual features.
[0029] In a specific embodiment, see Figure 2 The topmost part of the image encoder uses ResNet50 as its skeleton to process the sequence of images from the vehicle's forward-looking camera. Encode to obtain visual features The calculation formula is as follows: in, This indicates a convolution operation.
[0030] S4: Based on a cognitive encoder, cognitive features are extracted; these features are obtained by fusing motion and semantic features. See also Figure 2 The middle part specifically includes the following steps: S401: Extracts motion features from vehicle motion state sequences based on a motion encoder.
[0031] In a specific embodiment, a motion encoder with a Long Short-Term Memory (LSTM) network as its backbone is used to process the vehicle motion state sequence. Encode to obtain the hidden state vector Motion features are further obtained by mapping using a multi-layer perceptron (MLP). The calculation formula is as follows: S402: Extract semantic features from vehicle forward-looking camera image sequences based on a semantic encoder.
[0032] In a specific embodiment, a semantic segmentation network with ResNet50 as its backbone is first used to extract the vehicle front-view semantic image sequence from the vehicle front-view camera image sequence. Defined as ,in For the first t Frame of vehicle forward-looking semantic image.
[0033] Then, based on a semantic encoder with ResNet50 as the skeleton, from the vehicle's forward-looking semantic image sequence Extract semantic features The calculation formula is as follows: in, This indicates a convolution operation.
[0034] S403: Utilizing element-wise addition to characterize motion features and semantic features By fusing the data, cognitive features can be obtained. The calculation formula is as follows: in, This indicates an element-wise addition operation.
[0035] S5: Image sequence from the vehicle's forward-looking camera Vehicle forward-looking semantic image sequence Vehicle location and vehicle motion state sequence Align the timestamps.
[0036] S6: Based on the visual-spatial self-attention mechanism and the cognitive-driven cross-attention mechanism, visual features and cognitive features are spatiotemporally fused to obtain spatiotemporally fused features. See also Figure 2 The right half of the text specifically includes the following steps: S601: Visual features are calculated based on a visuospatial self-attention mechanism to obtain visuospatial features. Specifically, this includes: (1) Visual features By performing linear feature space projection, we obtain the first query matrix, the first key matrix, and the first value matrix, calculated as follows: in, , , These represent the learnable weight matrices used in the visual spatial self-attention mechanism to map the original visual features to three independent feature spaces: query, key, and value. The first query matrix represents the context feature vector that the current pixel region needs to find in order to understand the global scene; The first key matrix represents the index feature vectors that can provide matching information for other pixel regions in the image; This is the first-value matrix, representing the substantial visual semantic content inherent in each pixel region.
[0037] (2) Based on the first query matrix and the first key matrix Calculate the first correlation weight matrix between different pixel regions in the vehicle's forward-looking camera image, denoted as: ,in This is the feature dimension scaling factor.
[0038] (3) For the first correlation weight matrix and the first value matrix We perform weighted feature aggregation to obtain the first aggregated feature, denoted as: .
[0039] (4) Based on the first multilayer perceptron, the first aggregated features Perform nonlinear mapping to obtain visual spatial features The calculation formula is as follows: This visual space feature By incorporating global contextual relevance information from a single frame image and enhancing visual feature representation with spatial attention, the network not only focuses on local pixels but also perceives the topological structure of the global scene.
[0040] S602: Based on a cognitive-driven cross-attention mechanism, focusing on cognitive features and visual spatial features Calculations are performed to obtain spatiotemporal fusion features. Specifically, these include: (1) Visual spatial features The second query matrix is obtained by projecting onto the linear feature space, and the calculation formula is as follows: in, Indicates visual spatial features The learnable weight matrix mapped to the second query space. The second query matrix is composed of visual spatial features. The learnable weight matrix Linear projection yields the spatial topology that physically represents the current moment's forward-looking scene of the vehicle.
[0041] (2) Cognitive characteristics By performing linear feature space projection, we obtain the second bond matrix and the second value matrix, calculated as follows: in, Indicates cognitive characteristics The learnable weight matrix mapped to the second bond space. Indicates cognitive characteristics Learnable weight matrix mapped to the second value space; The second key matrix is composed of cognitive features. Learnable weight matrix Obtained by linear projection; This is a second-valued matrix, composed of cognitive features. Learnable weight matrix Obtained by linear projection. Second key matrix. Second-value matrix Physically, they represent the matching index of historical temporal cognitive state and the actual temporal cognitive experience content, respectively.
[0042] (3) Based on the second query matrix Second bond matrix Calculate the second relevance weight matrix of each state in the historical cognitive features to the current localization task, denoted as: ,in This is the feature dimension scaling factor.
[0043] (4) For the second correlation weight matrix Second-value matrix Perform feature weighted aggregation to obtain the second aggregated feature, denoted as: This section simulates, at the physical level, the biomimetic mechanism by which human drivers, after observing current road conditions, draw upon the most relevant driving memories and spatial cognitive experiences in their brains to make decisions.
[0044] (5) Based on the second multilayer perceptron, the second aggregated features Nonlinear mapping is performed to obtain spatiotemporal fusion features. The calculation formula is as follows: S7: Input the spatiotemporal fusion features into the attention decoder to obtain the driver attention prediction map. For example... Figure 2As shown, the specific steps include: S701: Features of spatiotemporal fusion Perform bilinear interpolation upsampling at a specified ratio to restore the feature map resolution, obtaining the interpolated and magnified fused features. The calculation formula is as follows: in, This indicates the upsampling factor, in this embodiment... The value is 4; For the process Fusion features after multiplication and amplification; Indicates that the upsampling factor is included. The bilinear interpolation function.
[0045] S702: Based on an attention decoder with ResNet50 as the backbone, it performs interpolation-amplified fusion feature processing. Decode the data to obtain the fused features, denoted as... ,in This represents the convolution operation of the attention decoder.
[0046] S703: Constructing a learnable 2D Gaussian kernel based on local spatial coordinates Local spatial coordinates represent the horizontal and vertical pixel offsets of each point within the 2D Gaussian kernel relative to the kernel center. 2D Gaussian kernel The calculation formula is as follows: in, and Here, are local spatial coordinates, representing the horizontal and vertical pixel offsets of a point within the two-dimensional Gaussian kernel relative to the center of the Gaussian kernel, respectively. denoted as the standard deviation of the Gaussian distribution.
[0047] S704: Based on a two-dimensional Gaussian kernel Decoded fusion features Perform smoothing filtering to obtain the smoothed fused features, denoted as: ,in This represents the convolution operator.
[0048] S705: Fusion features after smoothing filtering The driver attention prediction map is generated by normalizing the data in the spatial dimension. The calculation formula is as follows: in, For the final generated driver attention prediction map, This indicates that normalization is performed in the spatial dimension.
[0049] The KL divergence diagrams of the driver attention prediction map and the driver attention truth map generated in this embodiment are shown below. Figure 3 As shown, this illustrates the difference in viewpoint distribution between the two.
[0050] S8: Input the motion features into the state decoder to obtain the predicted vehicle position coordinates and predicted vehicle orientation angle in the global coordinate system. The calculation formula is as follows: in, This indicates the predicted vehicle position coordinates. This indicates the predicted vehicle heading angle. This refers to the characteristics of motion.
[0051] S9: Construct a multi-task joint loss function based on the driver attention ground truth map, driver attention prediction map, real vehicle position coordinates, predicted vehicle position coordinates, real vehicle orientation angle, and predicted vehicle orientation angle. Specifically, this includes the following steps: S901: Based on the actual vehicle position coordinates and the predicted vehicle position coordinates, the vehicle positioning loss is calculated using the Smooth L1 loss function. The calculation formula is as follows: in, This indicates a loss of vehicle location information. To predict the vehicle's position coordinates, These are the actual vehicle location coordinates.
[0052] S902: Based on the actual vehicle orientation angle and the predicted vehicle orientation angle, cosine similarity is used to calculate the vehicle orientation loss. The calculation formula is as follows: in, Indicates vehicle orientation loss. This indicates the predicted vehicle heading angle. This is the actual orientation angle.
[0053] S903: Based on the driver attention ground truth map and the driver attention prediction map, the driver attention loss is calculated using the KL divergence (Kullback-Leibler Divergence) formula as follows: in, This indicates impaired driver attention. This represents a truth map of driver attention. This represents a driver attention prediction map. To prevent division by zero of extremely small constants, This represents the coordinates of a pixel in the driver attention truth map.
[0054] S904: Based on vehicle positioning loss Vehicle orientation loss and driver attention loss Construct a joint loss function for multiple tasks, and calculate it using the following formula: in, This represents the joint loss function for multiple tasks. , and These are the weight coefficients for the vehicle positioning loss, vehicle orientation loss, and driver attention loss, respectively. Considering that joint optimization requires the gradients of each loss sub-item to be numerically close, the weight coefficients are set to 1.0, 10.0, and 5.0, respectively.
[0055] S10: Based on multi-task joint loss function Update the weight parameters in the image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder, and state decoder until the preset number of iterations is reached.
[0056] S11: Based on the updated image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder and state decoder, the vehicle's position coordinates and orientation angle are predicted to achieve vehicle visual positioning.
[0057] This application also provides a cognitive-driven intelligent vehicle vision positioning system, the system comprising: The acquisition module is used to collect multimodal driving data of the vehicle, including the vehicle's forward-looking camera image sequence, the actual vehicle position, the vehicle motion state sequence, and the actual vehicle orientation angle. The driver attention ground truth map determination module is used to acquire driver eye movement fixation data and perform Gaussian kernel mapping to obtain the driver attention ground truth map. The visual feature extraction module is used to extract visual features based on the image encoder; The cognitive feature extraction module is used to extract cognitive features based on the cognitive encoder; the cognitive features are obtained by fusing motion features and semantic features. The spatiotemporal fusion module is used to spatiotemporally fuse visual features and cognitive features based on the visual spatial self-attention mechanism and the cognitive-driven cross-attention mechanism to obtain spatiotemporal fused features. The driver attention prediction map determination module is used to input spatiotemporal fusion features into the attention decoder to obtain the driver attention prediction map; The prediction module is used to input motion features into the state decoder to obtain the predicted vehicle position coordinates and predicted vehicle orientation angle in the global coordinate system. The multi-task joint loss function construction module is used to construct a multi-task joint loss function based on the driver attention ground truth map, driver attention prediction map, real vehicle position coordinates, predicted vehicle position coordinates, real vehicle orientation angle and predicted vehicle orientation angle. The update module is used to update the weight parameters in the image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder and state decoder based on the multi-task joint loss function, until the preset number of iterations is reached; The visual positioning module is used to predict the vehicle's position coordinates and orientation angle based on the updated image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder and state decoder, so as to achieve vehicle visual positioning.
[0058] Compared with existing technologies, this embodiment utilizes ample multimodal input data, such as vehicle forward-looking camera image sequences and motion state sequences, and introduces driver attention information as an additional supervisory signal to achieve high-precision visual positioning based on deep learning training. The visual positioning method of this application incorporates driver spatial cognition, does not rely on high-precision maps and sensors, has strong applicability, low cost, and excellent practicality.
[0059] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. In summary, the content of this specification should not be construed as a limitation of this application.
Claims
1. A cognitive-driven intelligent vehicle visual positioning method, characterized in that, The cognitively driven intelligent vehicle visual localization method includes: Collect vehicle multimodal driving data, which includes vehicle front-view camera image sequences, real vehicle position, vehicle motion state sequences, and real vehicle orientation angles; Driver eye movement fixation data is acquired and Gaussian kernel mapping is performed to obtain the driver attention ground map; Visual features are extracted based on an image encoder; Cognitive features are extracted based on a cognitive encoder; these cognitive features are obtained by fusing motion features and semantic features. Based on the visual spatial self-attention mechanism and the cognitive-driven cross-attention mechanism, the visual features and cognitive features are spatiotemporally fused to obtain spatiotemporally fused features. The spatiotemporal fusion features are input into the attention decoder to obtain the driver attention prediction map; The motion features are input into the state decoder to obtain the predicted vehicle position coordinates and predicted vehicle orientation angle in the global coordinate system. A multi-task joint loss function is constructed based on the driver attention ground truth map, driver attention prediction map, real vehicle position coordinates, predicted vehicle position coordinates, real vehicle orientation angle, and predicted vehicle orientation angle. The weight parameters in the image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder, and state decoder are updated based on the multi-task joint loss function until a preset number of iterations is reached. Based on the updated image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder, and state decoder, the position coordinates and orientation angle of the vehicle are predicted to achieve vehicle visual positioning.
2. The cognitive-driven intelligent vehicle visual positioning method of claim 1, wherein, The cognitive encoder includes a motion encoder and a semantic encoder.
3. The cognitive-driven intelligent vehicle visual positioning method of claim 2, wherein, Based on a cognitive encoder, cognitive features are extracted, specifically including the following steps: Motion features are extracted from the vehicle motion state sequence based on the motion encoder; Based on the semantic encoder, semantic features are extracted from the vehicle forward-looking camera image sequence; The motion features and semantic features are fused by element-wise addition to obtain cognitive features.
4. The cognitive-driven intelligent vehicle visual positioning method of claim 3, wherein, Extracting semantic features from the vehicle's forward-looking camera image sequence based on the semantic encoder specifically includes the following steps: Based on a semantic segmentation network, a vehicle front-view semantic image sequence is extracted from the vehicle front-view camera image sequence; Based on the semantic encoder, semantic features are extracted from the vehicle forward-looking semantic image sequence.
5. The cognitively driven intelligent vehicle visual positioning method according to claim 1, characterized in that, Based on the visual spatial self-attention mechanism and the cognitive-driven cross-attention mechanism, the visual features and cognitive features are spatiotemporally fused to obtain spatiotemporally fused features, specifically including the following steps: The visual features are calculated based on the visual spatial self-attention mechanism to obtain the visual spatial features; Based on the cognitive-driven cross-attention mechanism, the cognitive features and the visual spatial features are calculated to obtain spatiotemporal fusion features.
6. The cognitively driven intelligent vehicle visual positioning method according to claim 5, characterized in that, The visual features are calculated based on the visual spatial self-attention mechanism to obtain visual spatial features, specifically including the following steps: The visual features are projected onto a linear feature space to obtain a first query matrix, a first key matrix, and a first value matrix. Based on the first query matrix and the first key matrix, calculate the first correlation weight matrix between different pixel regions in the vehicle front-view camera image; The first correlation weight matrix and the first value matrix are subjected to feature weight aggregation to obtain the first aggregated feature; The first aggregated features are nonlinearly mapped based on the first multilayer perceptron to obtain visual spatial features.
7. The cognitively driven intelligent vehicle visual positioning method according to claim 5, characterized in that, Based on a cognitive-driven cross-attention mechanism, the cognitive features and the visual-spatial features are calculated to obtain spatiotemporal fusion features, specifically including the following steps: The visual spatial features are projected onto a linear feature space to obtain a second query matrix; The cognitive features are projected onto a linear feature space to obtain a second key matrix and a second value matrix. Based on the second query matrix and the second key matrix, calculate the second relevance weight matrix of each state in the historical cognitive features to the current positioning task; The second correlation weight matrix and the second value matrix are subjected to feature weight aggregation to obtain the second aggregated feature; The second aggregated feature is nonlinearly mapped based on the second multilayer perceptron to obtain the spatiotemporal fusion feature.
8. The cognitively driven intelligent vehicle visual positioning method according to claim 1, characterized in that, The spatiotemporal fusion features are input into the attention decoder to obtain the driver attention prediction map, specifically including the following steps: The spatiotemporal fusion features are upsampled using bilinear interpolation to obtain the interpolated and amplified fusion features; The interpolated and amplified fused features are decoded using an attention decoder to obtain the decoded fused features. A learnable two-dimensional Gaussian kernel is constructed based on local spatial coordinates, where the local spatial coordinates represent the horizontal and vertical pixel offsets of each point within the two-dimensional Gaussian kernel relative to the kernel center. Based on the two-dimensional Gaussian kernel, the decoded fused features are smoothed and filtered to obtain the smoothed fused features. The smoothed and filtered fused features are normalized in the spatial dimension to generate a driver attention prediction map.
9. The cognitively driven intelligent vehicle visual positioning method according to claim 1, characterized in that, Based on the driver attention ground truth map, driver attention prediction map, real vehicle position coordinates, predicted vehicle position coordinates, real vehicle orientation angle, and predicted vehicle orientation angle, a multi-task joint loss function is constructed, specifically including the following steps: Calculate the vehicle positioning loss based on the actual vehicle position coordinates and the predicted vehicle position coordinates; Calculate the vehicle orientation loss based on the actual vehicle orientation angle and the predicted vehicle orientation angle; The driver's attention loss is calculated based on the driver attention truth map and the driver attention prediction map. Based on the vehicle positioning loss, vehicle orientation loss, and driver attention loss, a multi-task joint loss function is constructed.
10. A cognitively driven intelligent vehicle vision positioning system, characterized in that, The cognitively driven intelligent vehicle vision positioning system includes: The acquisition module is used to acquire multimodal driving data of the vehicle, including the vehicle's forward-looking camera image sequence, the actual vehicle position, the vehicle motion state sequence, and the actual vehicle orientation angle. The driver attention ground truth map determination module is used to acquire driver eye movement fixation data and perform Gaussian kernel mapping to obtain the driver attention ground truth map. The visual feature extraction module is used to extract visual features based on the image encoder; A cognitive feature extraction module is used to extract cognitive features based on a cognitive encoder; the cognitive features are obtained by fusing motion features and semantic features. The spatiotemporal fusion module is used to perform spatiotemporal fusion of the visual features and cognitive features based on the visual spatial self-attention mechanism and the cognitive-driven cross-attention mechanism to obtain spatiotemporal fused features. The driver attention prediction map determination module is used to input the spatiotemporal fusion features into the attention decoder to obtain the driver attention prediction map; The prediction module is used to input the motion features into the state decoder to obtain the predicted vehicle position coordinates and the predicted vehicle orientation angle in the global coordinate system. The multi-task joint loss function construction module is used to construct a multi-task joint loss function based on the driver attention truth map, driver attention prediction map, real vehicle position coordinates, predicted vehicle position coordinates, real vehicle orientation angle and predicted vehicle orientation angle. The update module is used to update the weight parameters in the image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder and state decoder based on the multi-task joint loss function, until a preset number of iterations is reached; The visual positioning module is used to predict the position coordinates and orientation angle of the vehicle based on the updated image encoder, cognitive encoder, visual spatial self-attention mechanism, cognitive-driven cross-attention mechanism, attention decoder and state decoder, so as to realize vehicle visual positioning.