Automatic driving task processing method and device, equipment and medium

Abstract

The invention relates to the technical field of automatic driving, in particular to an automatic driving task processing method and device, equipment and a medium. According to the method, firstly, the navigation guidance information is acquired and directly incorporated into beyond visual range contents (such as remote steering and distance), so that the limitation of the sensing range of a vehicle-mounted sensor is broken through, and the problem that an existing system lacks beyond visual range information is solved. And secondly, semantic analysis is performed based on navigation information and multi-view visual data to obtain text probability distribution, so that association of global navigation context and real-time scenes is realized, and a semantic basis is provided for humanization-like planning. Then, performing cross-modal fusion on the aerial view angle features and the text probability features, so that global navigation logic and real-time environment perception are deeply coupled, and decision is ensured to consider both a short-term environment and long-term planning. And finally, fusing the features and inputting the features into a task network, so that sensing, prediction and planning are more suitable for an actual scene, the utilization capability of beyond-visual-range information in a complex scene is improved, and the adaptability and reasoning capability of the system are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a processing method and device of automatic driving task, equipment and medium. BACKGROUND

[0002] With the rapid development of artificial intelligence and intelligent transportation technology, automatic driving systems have gradually become an important direction to improve traffic efficiency and driving safety. Current automatic driving systems mainly realize environmental perception and decision planning through two types of technical paths: one is a question and answer system based on visual language model (VLM), which can understand the semantic of the visual scene around the vehicle and generate corresponding text description or answer; the other is an end-to-end driving model, which directly outputs specific control instructions of the vehicle through the linkage of perception, prediction and planning modules. These systems obtain the surrounding environment information through vehicle-mounted sensors (such as cameras, laser radars, etc.), and the perception range is usually within 100 to 150 meters, and based on the visual data within this range, the identification of obstacles, traffic participants, lane lines and other elements is realized, and then short-term driving decisions are completed.

[0003] In actual driving scenarios, the driver's decision depends not only on the environmental information within the current field of view, but also on the global navigation context for long-term planning. For example, when using navigation applications such as Google Maps, human drivers will obtain key information beyond the existing field of view, including upcoming turning operations (left turn, right turn, straight ahead, etc.), the distance corresponding to the operation, the type of intersection to pass through, and whether there is a traffic signal, etc. Among them, such distance information beyond the visual perception range of vehicle-mounted sensors is defined as over-the-horizon information, which is of great significance for the driver to adjust the speed in advance, select the lane, and predict the traffic strategy of complex intersections.

[0004] However, the existing automatic driving technology has significant deficiencies in the fusion and application of global context information. On the one hand, the decision basis of visual language model and end-to-end driving model is mainly limited to environmental data within the current visual range, which is difficult to incorporate over-the-horizon navigation information, resulting in a lack of human-like proactive reasoning ability in long-term planning; on the other hand, existing related data sets and model researches focus on frame-by-frame perception and short-term prediction tasks, at most extending to the supervision of the next frame state, failing to cover the global navigation context required for comprehensive scene understanding, making the automatic driving system limited in generalization and adaptability when facing unfamiliar complex scenarios, and difficult to achieve safe and efficient long-term path planning. SUMMARY

[0005] Therefore, the embodiments of the present application provide a method and device for processing an autonomous driving task, and a medium, to solve the problem that the existing autonomous driving system and related model lack effective use of over-the-horizon information, cannot be integrated into a global navigation context to support human-like long-term planning, and thus the adaptability and reasoning ability of the system in complex scenarios are limited.

[0006] In a first aspect, the embodiments of the present application provide a method for processing an autonomous driving task, the method comprising:

[0007] obtaining multi-view visual data collected by a vehicle during driving, and obtaining navigation guide information of the vehicle;

[0008] generating bird's-eye view features based on visual features in the multi-view visual data, and performing semantic analysis of a driving scene based on the navigation guide information and the multi-view visual data to obtain a text probability distribution;

[0009] performing cross-modal fusion based on the bird's-eye view features and processed text probability features to obtain fusion features, and inputting the fusion features into a plurality of autonomous driving task networks to obtain autonomous driving decision information output by each autonomous driving task network.

[0010] Further, the obtaining of the navigation guide information of the vehicle comprises:

[0011] obtaining a navigation video of the vehicle driving from a source point to a destination point, wherein the navigation video is generated based on a navigation route between the source point and the destination point and continuous videos on the navigation route;

[0012] extracting a plurality of navigation images from the navigation video;

[0013] generating a plurality of candidate navigation descriptions based on the continuous image frames and the corresponding guide texts, calculating a consistency evaluation score corresponding to each candidate navigation description using a similarity index, and taking the candidate navigation description with the highest consistency evaluation score as the navigation guide information, wherein the consistency evaluation score represents the matching degree between the candidate navigation description and the actual navigation scene requirement.

[0014] Further, the obtaining of the navigation video of the vehicle driving from the source point to the destination point comprises:

[0015] obtaining a navigation route of the vehicle from the source point to the destination point from a navigation map, and obtaining continuous images on the navigation route;

[0016] predicting a driving distance and a driving time length based on the navigation route;

[0017] arranging the continuous images on the navigation route in a sequence according to the order of the route, to obtain an image sequence, and determining a playing rate of each frame of the continuous images based on the driving distance and the driving time length;

[0018] dynamically concatenating the continuous images according to the image sequence and the playing rate of each frame of the continuous images, to generate a navigation video of the vehicle driving from the source point to the destination along the navigation route.

[0019] Further, the calculating of the consistency evaluation score corresponding to each candidate navigation description by using the similarity index comprises:

[0020] obtaining directional keywords in each candidate navigation description, and comparing the directional keywords in each candidate navigation description to obtain target keywords with intersection;

[0021] calculating an intersection similarity corresponding to each candidate navigation description based on the number of target keywords in each candidate navigation description and the total number of directional keywords corresponding to all candidate navigation descriptions;

[0022] obtaining distance data in each candidate navigation description, and calculating a distance similarity corresponding to each candidate navigation description based on the difference of the distance data between the candidate navigation descriptions;

[0023] extracting words in each candidate navigation description, and counting the number of words corresponding to the common appearing words in the candidate navigation descriptions;

[0024] calculating a word similarity corresponding to each candidate navigation description based on the number of words in each candidate navigation description and the total number of words corresponding to all candidate navigation descriptions;

[0025] calculating a consistency evaluation score corresponding to each candidate navigation description by using the intersection similarity, the distance similarity and the word similarity.

[0026] Further, the generating of the bird's eye view perspective feature based on the visual features in the multi-view visual data comprises:

[0027] extracting basic visual features under each perspective in the multi-view visual data;

[0028] mapping the basic visual features under each perspective to the bird's eye view perspective based on a spatial mapping relationship between different perspectives and the bird's eye view perspective, to obtain the bird's eye view perspective feature.

[0029] Further, the performing of the driving scene semantic analysis based on the navigation guide information and the multi-view visual data to obtain a text probability distribution comprises:

[0030] input the navigation guide information and the multi-view visual data into a trained visual language model;

[0031] resolve a semantic intention of the navigation guide information through the visual language model, understand a driving scene in combination with the multi-view visual data, and output a text probability distribution corresponding to an automatic driving action.

[0032] Further, the resolving the semantic intention of the navigation guide information through the visual language model, understanding the driving scene in combination with the multi-view visual data, and outputting the text probability distribution corresponding to the automatic driving action include:

[0033] identifying a semantic instruction intention corresponding to the navigation guide information through the visual language model;

[0034] determining a current driving scene of the vehicle based on the multi-view visual data, and analyzing a degree of applicability between the semantic instruction intention and the driving scene;

[0035] predicting a probability value of at least one automatic driving action based on the degree of applicability, and constructing a text probability distribution adapted to the navigation guide instruction and the current scene based on the probability value.

[0036] In a second aspect, an embodiment of the present application provides a processing device for an automatic driving task, and the device includes:

[0037] an acquisition module configured to acquire multi-view visual data collected in a driving process of a vehicle and navigation guide information;

[0038] an analysis module configured to generate a bird's-eye view feature based on a visual feature in the multi-view visual data, and perform semantic analysis on a driving scene based on the navigation guide information and the multi-view visual data to obtain a text probability distribution;

[0039] a processing module configured to perform cross-modal fusion based on the bird's-eye view feature and the processed text probability feature to obtain a fusion feature, and input the fusion feature into a plurality of automatic driving task networks to obtain automatic driving decision information output by each automatic driving task network.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory and a processor, the memory and the processor are communicatively connected with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the method in the first aspect or any of the corresponding embodiments.

[0041] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the method in the first aspect or any of the corresponding embodiments.

[0042] The present application firstly breaks through the range limitation of vehicle-mounted sensor perception by directly incorporating the over-the-horizon content (such as far turning, distance, etc.) into the navigation guidance information, solving the problem of lack of over-the-horizon information in the existing system. Secondly, based on the semantic analysis of navigation information and multi-view visual data to obtain text probability distribution, the association of global navigation context and real-time scene is realized, providing semantic basis for human-like planning. Then, the bird's eye view features and text probability features are cross-modal fused, which makes the global navigation logic and real-time environment perception deeply coupled, ensuring that the decision takes into account the short-term environment and long-term planning. Finally, the task network is inputted with the fused features, making the perception, prediction and planning more suitable for the actual scene, improving the utilization of over-the-horizon information in complex scenes, and enhancing the adaptability and reasoning ability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the specific embodiments or the prior art of the present application, the drawings needed to be used in the specific embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0044] Figure 1 is a flowchart of a processing method of an automatic driving task according to some embodiments of the present application;

[0045] Figure 2 is a comparison diagram of visual language model reasoning with or without navigation guidance according to some embodiments of the present application;

[0046] Figure 3 is a flowchart of another processing method of an automatic driving task according to some embodiments of the present application;

[0047] Figure 4 is a generation process diagram of navigation guidance information according to some embodiments of the present application;

[0048] Figure 5 is a working diagram of an automatic driving task processing framework according to some embodiments of the present application;

[0049] Figure 6 is a structural block diagram of a processing device of an automatic driving task according to an embodiment of the present application;

[0050] Figure 7 is a hardware structure diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0052] According to the embodiments of the present application, a processing method, device and equipment of an autonomous driving task and a medium are provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0053] In the embodiments of the present application, a processing method of an autonomous driving task is provided, Figure 1 is a flowchart of a processing method of an autonomous driving task according to the embodiments of the present application, as Figure 1 shown, the flow includes the following steps:

[0054] In step S101, multi-view visual data collected in the driving process of the vehicle is acquired, and navigation guidance information of the vehicle is acquired.

[0055] In the embodiments of the present application, the vehicle is equipped with multiple high-resolution cameras, including front-view, rear-view, left-view, right-view and surround-view cameras. These cameras continuously collect environmental images around the vehicle at a preset frame rate (such as 30 frames per second).

[0056] The front-view camera mainly captures information such as lane lines, traffic lights, pedestrians and front vehicles on the front road; the rear-view camera focuses on the road conditions behind the vehicle, including the following car distance and the following car; the side-view camera is responsible for monitoring the lane-changing situation of vehicles on both sides of the lane and the roadside obstacles; and the surround-view camera forms a 360-degree panoramic image through splicing technology to eliminate the visual dead zone.

[0057] The collected image data will be preprocessed, such as distortion correction (eliminating lens distortion according to camera internal parameters), exposure adjustment (ensuring the clarity of images under different light conditions), time synchronization (ensuring that the images collected by different cameras at the same time correspond to the same driving state), and finally forming a structured multi-view visual data set, which is stored in the local storage unit of the vehicle or the cloud server.

[0058] Then, the current real-time position (source point) and the latitude and longitude coordinates of the destination (end point) of the vehicle are determined by the positioning system; then the map service API is called to obtain the detailed navigation route from the source point to the end point, including the road names, turning nodes, driving distances, estimated time lengths, etc. on the way; then the navigation system will convert these route information into natural language form of navigation guide instructions, such as "turn left into XX road 500 meters ahead" and "drive straight along the current road for 2 kilometers".

[0059] At the same time, in order to improve the accuracy of navigation information, real-time traffic data (such as congested sections, traffic accidents, etc.) can also be combined to dynamically update the navigation guide information, ensuring that the vehicle can adjust the driving plan according to the latest road conditions. These multi-view visual data and navigation guide information will be transmitted to the central processing unit of the vehicle, preparing for subsequent feature extraction and semantic analysis.

[0060] Step S102, generating bird's eye view features based on visual features in multi-view visual data, and performing semantic analysis on driving scenes based on navigation guide information and multi-view visual data to obtain a text probability distribution.

[0061] In the embodiments of the present application, generating bird's eye view features based on visual features in multi-view visual data includes the following steps A1-A2:

[0062] Step A1, extracting basic visual features under each view in the multi-view visual data.

[0063] Specifically, first, the multi-view visual data needs to be preprocessed to eliminate noise and redundant information. For the images captured by the front, rear, left, right and surround view cameras, distortion correction (using camera intrinsic matrix and distortion coefficient, through polynomial fitting to eliminate image distortion caused by lens optical distortion), illumination normalization (using histogram equalization or CLAHE algorithm to balance the image brightness and contrast under different light conditions, avoiding the influence of backlight or shadow on feature extraction) and size standardization (uniformly scaling the image to a preset size, such as 384x288 pixels, to ensure the data format of the input feature extraction network consistent) are performed respectively.

[0064] After pre-processing, a pre-trained deep convolutional neural network (CNN) is used as a feature extractor, such as ResNet-50, DINOv2, or MobileNetV3, etc. These networks perform layer-by-layer feature extraction on images through the stacking of multiple convolutional layers, pooling layers, and activation functions (such as ReLU): shallow convolutional layers (such as layers 1-2) mainly capture low-level visual features such as edges, colors, and textures, for example, the white edges of lane lines and the contour textures of vehicles; middle convolutional layers (such as layers 3-4) extract target component features, such as the shape of traffic light clusters and the limb structure of pedestrians; and deep convolutional layers (such as layer 5 and above) output high-level features containing semantic information, such as feature vectors of abstract concepts such as "lane line", "vehicle", "pedestrian", and "traffic sign".

[0065] To adapt to multi-view scenarios, a feature extraction branch needs to be deployed for each camera view to ensure that the feature extraction parameters for different views are fine-tuned for their shooting ranges (such as the front-view camera focusing on long-distance roads and the rear-view camera focusing on close-range vehicle tail environments). For example, the feature extraction network of the front-view camera will enhance the feature response to long-distance lane lines and traffic signs, while the network of the rear-view camera will focus on capturing features of close-range obstacles. Finally, after each view's image passes through the feature extraction network, a uniform-dimension base visual feature map (such as a 2048x16x12 feature tensor) is output, which retains spatial position information and contains rich semantic content, providing a reliable feature basis for subsequent mapping to the bird's eye view.

[0066] Step A2, based on the spatial mapping relationship between different views and the bird's eye view, the base visual features under each view are mapped to the bird's eye view to obtain bird's eye view features.

[0067] Specifically, a spatial mapping relationship model between different views and the bird's eye view is obtained. Camera calibration parameters include intrinsic parameters (focal length, principal point coordinates) and extrinsic parameters (rotation matrix, translation vector), which are obtained through Zhang Zhengyou calibration method or laser radar auxiliary calibration, and are used to describe the installation position (such as height, pitch angle, yaw angle) and imaging optical characteristics of the camera in the vehicle coordinate system. Based on these parameters, a conversion formula from pixel coordinates to three-dimensional space coordinates can be constructed: for a pixel point (u, v) in a certain view image, the ray direction in the camera coordinate system is obtained through the inverse operation of the intrinsic matrix, and then converted to the vehicle coordinate system (with the vehicle center as the origin, x-axis forward, y-axis left, z-axis upward) by combining the extrinsic parameters, and finally projected onto the two-dimensional plane of the bird's eye view (ignoring the z-axis height, retaining the x and y coordinates), forming a "pixel point-bird's eye coordinate" mapping table.

[0068] After obtaining the mapping relationship, a bird's-eye view mapping is performed on the basic visual features of each viewpoint. For the feature map of the front-view camera, the long-distance road features it captures (such as lane lines at 50-100 meters) need to be mapped to the corresponding x-coordinate region in front of the vehicle in the bird's-eye view; the left lane features captured by the left-view camera are mapped to the y-coordinate region to the left of the vehicle in the bird's-eye view. During the mapping process, bilinear interpolation is used to handle the pixel correspondence in the feature map: when a certain bird's-eye coordinate corresponds to feature pixels from multiple views, multi-view features are fused by weighted averaging (the weight is inversely proportional to the pixel distance) to avoid feature conflicts; for bird's-eye coordinates without corresponding pixels (such as blind spots), zero-padding or context-based feature prediction is used for completion.

[0069] After mapping, the bird's-eye view features from each perspective undergo initial alignment and filtering. Non-Maximum Suppression (NMS) is used to remove redundant features from repeated mappings (such as lane lines being repeatedly mapped in left-view and surround-view features), retaining the feature points with the highest confidence. Simultaneously, vehicle motion information (such as vehicle speed and steering angle provided by the inertial measurement unit) is used to smooth the bird's-eye view features over time, reducing feature shifts caused by inter-frame jitter. Finally, the bird's-eye view features corresponding to each perspective are obtained.

[0070] In this embodiment of the application, semantic analysis of the driving scene is performed based on navigation guidance information and multi-view visual data to obtain the text probability distribution, including the following steps B1-B2:

[0071] Step B1: Input the navigation guidance information and multi-view visual data into the trained visual language model.

[0072] Step B2 involves parsing the semantic intent of navigation guidance information using a visual language model, understanding the driving scenario by combining multi-view visual data, and outputting the text probability distribution corresponding to autonomous driving actions.

[0073] As an example, such as Figure 2 As shown, in non-navigational visual language model reasoning (Figure a), based solely on multi-view visual data and the question "What action should be taken?", the model arrives at the answer "Continue straight." However, in navigation-guided visual language model reasoning (Figure b), by combining multi-view visual data, navigation guidance ("Go straight for 123.5 meters and then turn right; What action should be taken?"), and the question, the model can more accurately provide the answer "Turn right into the right-hand turning lane." This demonstrates the important role of navigation guidance in visual language model reasoning, facilitating semantic analysis that is more closely aligned with real-world scenarios.

[0074] Specifically, the semantic intention of the navigation guide information is analyzed by a visual language model, the driving scene is understood in combination with multi-view visual data, and a text probability distribution corresponding to an automatic driving action is output, including the following steps B201-B203:

[0075] Step B201, the semantic instruction intention corresponding to the navigation guide information is identified by a visual language model.

[0076] Specifically, when the visual language model (Vision-Language Model, VLM) identifies the semantic instruction intention of the navigation guide information, it first performs deep semantic analysis on the input navigation guide prompt text. For example, the instruction in the figure “Go straight for 53.6 meters, then turn left at the traffic light. Continue straight for 80 meters on this road”, the text processing module inside the model will first perform word segmentation operation, splitting the continuous text into “go straight”, “53.6 meters”, “then”, “turn left at xxx”, “traffic light”, “continue”, “80 meters”, “on this road” and other lexical units. Then, a pre-trained word embedding model is used to convert these words into high-dimensional vector representations, which can capture the semantic information of the words themselves, such as the vector corresponding to “go straight” is associated with the semantic concept of “continuous forward driving”, and the vector corresponding to “turn left” is associated with the semantic of “change driving direction to left”.

[0077] At the same time, the model will assist semantic understanding in combination with multi-view visual data. For visual data, the visual language model will first extract visual features through a convolutional neural network to identify lane lines, traffic lights, road signs and other elements in the image. When processing the “traffic light” in the navigation instruction, the model will locate the position and state (such as red light, green light) of the traffic light in the visual features, and associate the text semantics with the visual elements. Then, with the help of attention mechanism, the text semantic analysis process is focused on the key visual elements, such as when analyzing “turn left at the traffic light”, the attention is focused on the image area where the traffic light is located, and the state of the traffic light is determined whether it allows left turn, etc., so as to more accurately understand the instruction intention.

[0078] On the basis of the understanding of the semantic vectors of the words and the visual association, the model will perform sequence modeling through the encoder of the Transformer architecture to capture the timing and logical relationship between the words. For example, the action sequence and logical constraints of “first go straight for 53.6 meters, then turn left at the traffic light, and then go straight for 80 meters” are sorted out to build a complete semantic instruction intention representation, which clearly indicates the required driving action sequence, distance condition, and environmental trigger condition (such as traffic light), etc. The natural language navigation guide information is converted into a semantic instruction intention structure that can be understood by machines and used for subsequent decision-making.

[0079] Step B202, determine the current driving scene of the vehicle based on multi-view visual data, and analyze the applicability between the semantic instruction intention and the driving scene.

[0080] Specifically, when determining the current driving scene of the vehicle, multi-view visual data is the core basis. The image collected by the front-view camera is used to identify information such as the type of the road ahead (such as urban road, highway), the number of lanes, the style of lane lines (solid line, dashed line), the state of traffic signal lights, and the front obstacles (vehicles, pedestrians); the side-view camera data is used to assist in judging the situation of adjacent lanes and roadside environment (such as whether there is a parking area, a non-motor vehicle lane); and the rear-view camera data is used to know the traffic situation behind. The visual language model will first extract features from these multi-view images, and then through feature fusion (such as splicing fusion, attention fusion), the overall visual feature representation of the environment around the vehicle is constructed.

[0081] Then, the driving scene is classified and identified based on the extracted visual features. For example, by comparing the pre-trained scene feature library, it is determined whether the current scene is a “city intersection scene” or a “highway straight driving scene”. In the city intersection scene, the traffic signal lights and the layout of the intersection turning lanes are focused on; in the highway scene, the lane spacing, speed limit signs, and distant road conditions are focused on. At the same time, combined with the vehicle's own state information (such as speed, driving direction, which can be calculated from vehicle sensors or visual data, such as calculating speed by lane line relative displacement), the scene description is further refined, such as “city intersection green light straight driving scene (speed 30km / h)”.

[0082] Then, the obtained semantic instruction intention features (including action sequence, distance, environmental conditions, etc.) and the current driving scene features (road type, traffic facility state, vehicle state) are multi-dimensionally matched. For example, the semantic instruction requires “turn left at the traffic signal light”, and the current scene is “city intersection green light, and there is a left turning lane”, so the matching degree in the “turning action feasibility” dimension is high; if the current scene is “traffic signal light red light, no left turning lane”, then the matching degree in this dimension is low.

[0083] By calculating the matching scores of multiple dimensions (such as action execution condition matching degree, distance adaptation degree, environmental constraint matching degree), and then weighting and summing according to the weights (such as high weight of action execution condition, because it involves traffic rules), the overall applicability of the semantic instruction intention and the driving scene is obtained. For example, the similarity between the semantic feature vector and the scene feature vector is calculated by using the cosine similarity, or a rule-based matching engine is constructed (such as prohibition of left turn at red light, corresponding to low applicability), and the adaptation degree between the two is comprehensively judged, which provides a basis for subsequent action probability prediction.

[0084] Step B203, predicting a probability value of at least one automatic driving action based on the applicability, and constructing a text probability distribution adapted to the navigation guide instruction and the current scene based on the probability value.

[0085] Specifically, according to the semantic instruction intention and the driving scene, possible actions are determined, such as for the "turn left at the traffic signal" instruction and the intersection scene, the candidate actions can be "keep the current lane and drive straight to the intersection stop line", "turn on the left turn signal in advance", "turn left at the intersection", and "adjust the driving strategy due to the change of the signal light (such as stopping and waiting for the red light)".

[0086] For each candidate action, a mapping relationship between the applicability and the action probability is constructed. The probability of the action execution can be output by a pre-trained probability prediction model (such as logistic regression, neural network) by inputting the semantic-scene applicability. For example, when the semantic-scene applicability is high (such as green light, left turn lane, applicability 0.9), the probability of the "turn left at the intersection" action will be relatively high, and the probability of the "adjust the driving strategy due to the change of the signal light" will be relatively low. At the same time, the logical constraints and correlations between actions are considered, such as when the "turn on the left turn signal in advance" action probability is high, the "turn left at the intersection" action probability will also be affected (usually increased), because turning on the turn signal is a pre-action of left turn.

[0087] When constructing the text probability distribution, the probability values of all candidate automatic driving actions are normalized. The sum of all action probabilities is ensured to be 1, forming a probability distribution. For example, the probability of candidate action A is 0.6, the probability of action B is 0.3, and the probability of action C is 0.1, and the total is 1. The probability distribution can reflect the possibility of executing each automatic driving action in the current scene, and is consistent with the semantic intention of the navigation guide instruction.

[0088] During the construction process, traffic rules and safe driving constraints are also introduced. For example, the probability of the "turn left at the intersection" action will be forced to be low (close to 0) when the light is red, even if the semantic-scene applicability in other dimensions is high. At the same time, the probability distribution is dynamically adjusted according to the applicability, and the probability distribution is updated in real time when the applicability changes (such as the signal light turning red), ensuring that the output text probability distribution meets the requirements of the navigation instruction and is consistent with the actual constraints of the current driving scene, providing accurate probabilistic decision basis for subsequent automatic driving task networks (such as planning and decision making), so that the vehicle can choose the most suitable driving action for the scene and the instruction.

[0089] Step S103, performing cross-modal fusion based on the bird's eye view features and the processed text probability features to obtain fusion features, and inputting the fusion features into multiple automatic driving task networks to obtain automatic driving decision information output by each automatic driving task network.

[0090] In the embodiment of the present application, in the cross-modal fusion stage, first, the bird's eye view feature and the processed text probability feature are preprocessed to ensure consistency in dimension and spatial scale. The bird's eye view feature usually exists in the form of a two-dimensional feature map (such as a tensor with a size of 200x200xC, where C is the number of feature channels), which contains spatial layout information of the vehicle's surrounding environment, such as lane line position, obstacle distribution, road boundary, etc.; the processed text probability feature is a one-dimensional vector (such as a vector with a dimension of D) after dimensionality reduction by sparsification, which covers semantic information such as navigation instruction execution probability, action feasibility score, etc., such as key indicators of "left turn feasibility 0.85" and "straight distance adaptability 0.92".

[0091] Then, the two types of features are fused in the channel dimension through a concatenation operation, that is, the text probability feature is broadcast to the same spatial dimension as the bird's eye view feature, and then concatenated with the channel of the bird's eye view feature to form a mixed feature with a dimension of 200x200x(C+D). At the same time, an element-wise addition (⊕) operation is performed to weight and sum the semantic information related to the spatial position (such as "left turn probability 50 meters ahead") in the text probability feature and the spatial feature (such as the lane line turning feature 50 meters ahead) at the corresponding position in the bird's eye view feature, to strengthen the relevance of semantics and space. For example, when the "left turn feasibility" score in the text probability feature is high, the feature response of the left lane area in the bird's eye view feature will be enhanced.

[0092] Subsequently, the mixed feature is input into a feature fusion network composed of multiple layers of perceptron (MLP). The network includes multiple fully connected layers and activation functions (such as ReLU), which extract deep associations between the two types of features through nonlinear transformation. The first layer of multilayer perceptron compresses the high-dimensional feature after concatenation to the target dimension, reducing redundant information; the middle layer introduces a cross-channel attention mechanism to allow features from different channels (such as "lane line channel" and "left turn probability channel") to interact and dynamically adjust feature weights - when the navigation instruction requires a left turn, the weight of the channel related to the left turn will be increased, while the weight of the irrelevant channel (such as the right lane feature) will be suppressed; the last layer outputs a unified fusion feature (such as a tensor with a size of 200x200xF), which retains both the global spatial layout of the bird's eye view and the semantic decision logic of the text probability, achieving deep coupling of visual spatial information and semantic probability information.

[0093] Finally, the fusion feature is input into three task-specific networks: perception, prediction, and planning, which perform targeted processing based on the characteristics of the fusion feature.

[0094] In the perception network, the fusion feature first extracts local detailed features through the convolutional layer, combines the spatial positioning ability of the bird's eye view and the semantic guidance of the text probability, and accurately identifies the traffic elements. For example, when the fusion feature contains semantic probability information such as “turning left soon”, the network will focus on the key elements such as the state of the traffic signal (red light / green light) at the intersection, the position and speed of the oncoming straight vehicle, and the possibility of pedestrians crossing the road. Through the multi-scale feature pyramid network (Feature Pyramid Network, FPN) to fuse features of different levels, the detection accuracy of small targets (such as distant pedestrians) and fuzzy targets (such as lane lines in rainy weather) is improved, and the perception results containing traffic signal state, obstacle category and coordinates, lane type, etc. Information is output.

[0095] In the prediction network, a time series modeling module (such as long short-term memory network or Transformer) is used, combined with the historical trajectory features of traffic participants in the bird's eye view (such as the position change of the vehicle in front of the vehicle in the past 3 seconds) and the scene semantics in the text probability (such as “in the left turn scene, the oncoming vehicle is likely to slow down and yield”), to predict the behavior of other traffic participants in the next 5-10 seconds. For example, for pedestrians at an intersection, if the fusion feature shows “pedestrian crossing green light” and “high left turn probability”, the network will predict that the pedestrian is likely to cross the road, and output the probability distribution of its motion trajectory; for vehicles in adjacent lanes, combined with the semantic probability of “current lane congestion”, the possibility of lane changing intention is predicted, providing a prediction basis for subsequent planning.

[0096] In the planning network, the driving scheme is generated combined with the spatial constraints and semantic requirements in the fusion feature. The network first constructs a road topology graph through the graph neural network (GNN), converts the lane, intersection and other spatial elements in the bird's eye view into graph nodes, and then uses the text probability feature to assign weights to the nodes (such as the “left turn lane” node weight increases with the increase of the left turn instruction probability). Subsequently, reinforcement learning or sampling-based method is used to search for the optimal path, which considers safety (distance from obstacles), compliance (compliance with traffic rules) and navigation fit (matching “left turn” “straight” instructions) under the premise of meeting vehicle dynamics constraints (such as maximum steering angle, acceleration limit), to generate multiple candidate driving trajectories. Finally, the optimal trajectory is selected through the evaluation function (based on the semantic probability score in the fusion feature), and specific control instructions are output, such as “steering angle 30 degrees, acceleration 1.2 m / s 2 , keep driving in the current lane for 20 meters and then enter the left turn lane”, realizing the complete conversion from fusion feature to automatic driving decision.

[0097] In this embodiment, a processing method of an automatic driving task is provided, Figure 3is a flowchart of a processing method of an automatic driving task according to an embodiment of the present application, as shown in the figure, the flow includes the following steps: Figure 3

[0098] In step S201, a navigation video of a vehicle driving from a source point to a destination is obtained, wherein the navigation video is generated based on a navigation route from the source point to the destination and continuous videos on the navigation route.

[0099] In the embodiments of the present application, the navigation video of the vehicle driving from the source point to the destination includes the following steps C1-C4:

[0100] Step C1, obtaining the navigation route of the vehicle from the source point to the destination from the navigation map, and obtaining the continuous images on the navigation route.

[0101] Specifically, first, based on the calculated latitude and longitude coordinates of the source point and the destination and the direction API of Google map is called to obtain the navigation route. The direction API will plan the optimal path according to the starting point, ending point coordinates and the preset travel mode (such as driving), and return the structured data containing road segment information, turning instruction, passing point coordinates, etc. For example, the route data may contain detailed road segment description such as "from the source point, drive eastward along XX road for 1.2 kilometers, turn right into YY road at XX intersection", as well as the starting point, ending point latitude and longitude and road type (such as trunk road, secondary trunk road) of each road segment.

[0102] After obtaining the navigation route, the continuous images on the route are collected through the static map API of the vehicle-mounted map. The static map API allows a single map image to be generated by specifying center coordinates, zoom level, image size and other parameters. In order to cover the complete route, the sampling points need to be set at fixed intervals (such as every 50 meters) according to the latitude and longitude sequence of the route, and each sampling point is used as the center coordinates of the static map to generate an image containing the position and surrounding environment. For example, for a 1-kilometer-long route, 20 sampling points need to be set, and each sampling point generates an image with a resolution of 640x480 pixels, and the current position and forward direction (such as identified by an arrow) are marked on the image through API parameters.

[0103] ​To ensure the continuity and consistency of the images, the static map of all sampling points needs to use the same zoom level (such as zoom level 18, ensuring that each pixel corresponds to a consistent actual distance), viewing angle (such as a 45-degree inclined viewing angle, enhancing the stereoscopic effect), and layer settings (such as displaying lane lines, traffic signs, buildings, and other details). After the collection is complete, the continuous images are preprocessed, including removing duplicate or blurred images (such as similar images caused by too close sampling points), unifying image size and color space, and finally forming a continuous image set that covers the entire navigation route and can clearly display changes in the road environment.

[0104] Step C2, predicting the driving distance and driving time based on the navigation route.

[0105] Specifically, when predicting the driving distance, the route data returned by the distance matrix API and the direction API of the vehicle-mounted map are mainly relied on. The distance matrix API directly returns the driving distance between two points (such as "5.2 kilometers") by inputting the source point and the end point coordinates, which has considered the actual length of the route (such as bends, detours, etc.) rather than the straight-line distance. At the same time, combined with the road segment information returned by the direction API, the distance is verified again: the route is divided into multiple continuous road segments, the distance of each road segment can be calculated by using the spherical distance formula (such as the Haversine formula) based on the latitude and longitude of the start and end points of the road segment, and then the total driving distance is obtained by adding up all the road segment distances. For example, a route contains 3 road segments with distances of 1.5 km, 2.3 km, and 1.4 km respectively, the total driving distance is 5.2 km, which is consistent with the result returned by the distance matrix API, ensuring the accuracy of the distance data.

[0106] When predicting the driving time, the driving distance, road type, speed limit information, and real-time traffic conditions need to be considered comprehensively. First, based on the speed limit data of the road segment returned by the direction API (such as 100 km / h for expressways and 60 km / h for urban roads), the theoretical driving time of each road segment (road segment distance / speed limit) is calculated, and the total driving time is obtained after accumulation. Second, the distance matrix API is called to obtain the estimated time containing traffic conditions, which will modify the theoretical time based on real-time traffic data (such as the driving speed of congested road segments). For example, the theoretical time of a road segment is 10 minutes, but due to congestion, the actual estimated time is 15 minutes. Finally, the two kinds of time are weighted and fused (such as the distance matrix API result weight 0.7, the theoretical calculation result weight 0.3), and the final driving time is obtained. For scenarios without real-time traffic data (such as simulation environment), the average driving speed is preset according to the road type (such as 40 km / h for urban roads and 60 km / h for rural roads), and the time is calculated by "driving distance / average speed", ensuring that the time prediction conforms to the actual driving rules.

[0107] Step C3, arrange the continuous images on the navigation route in the order of the route to obtain an image sequence, and determine the playing rate of each frame of image in the continuous images based on the driving distance and the driving time length.

[0108] Specifically, when arranging the continuous images to obtain the image sequence, the continuous images collected in step C1 are sorted in the order of the positions of the sampling points on the route based on the spatial order of the navigation route. Each sampling point corresponds to a specific position on the route (such as 50 meters, 100 meters, 150 meters from the source point), and the images are arranged in the order of “source point→ passing point 1→ passing point 2→…→ end point” by comparing the cumulative distances of the sampling points (the distances from the source point to the sampling points). For example, a route contains 20 sampling points, and the cumulative distances are 50 meters, 100 meters, …, 1000 meters, respectively. The corresponding images are arranged in this order to form an image sequence that can intuitively reflect the progress of the route. During the arrangement process, the correctness of the image order needs to be verified by the turning information of the navigation route (such as “turn right at 1000 meters”) to ensure that the images before and after the right turn can continuously show the change of the road direction.

[0109] When determining the playing rate of each frame of image, the playing interval of the image needs to be calculated in combination with the driving distance and the driving time length. First, the total number of frames of the navigation video is calculated, i.e. the number of continuous images in step C1 (such as 20 frames). The total driving time length (such as T seconds) is divided by the total number of frames to obtain the basic playing interval (such as T / 20 seconds / frame). However, since the driving speeds on different road sections of the route may be different (such as fast on a straight road and slow on a curved road), the playing interval needs to be finely adjusted: the route is divided into road sections, and the number of frames of each road section is the number of sampling points contained in the road section (such as a 1-kilometer road section containing 20 sampling points). The driving time length of the road section (such as t seconds) is divided by the number of frames of the road section to obtain the playing interval of each frame of image in the road section (such as t / 20 seconds / frame). For example, the driving speed on a straight road section is fast (5 seconds in length and 10 frames of image), and the playing interval is 0.5 seconds / frame; the driving speed on a curved road section is slow (10 seconds in length and 10 frames of image), and the playing interval is 1 second / frame. Through this segmented adjustment, the image playing rate is consistent with the simulated driving speed, and the change rhythm of the road environment in the video is consistent with the actual driving experience.

[0110] Step C4, dynamically concatenate the continuous images according to the image sequence and the playing rate of each frame of image in the continuous images to generate a navigation video of the vehicle driving from the source point to the end point along the navigation route.

[0111] Specifically, first, based on the obtained image sequence and the playing interval of each frame image, the time sequence track of the video is constructed. Using a video editing library (such as FFmpeg), the images are imported into the track in sequence order, and the dwell time of each frame image is set to the corresponding playing interval (such as 0.5 seconds for a straight road segment image and 1 second for a curved road segment image). To avoid the appearance of a jump when switching between images, a transition effect is added between adjacent images: when the viewing angle change between two frames of images is small (such as continuous driving on a straight road), a fade-in and fade-out transition (lasting 0.1 seconds) is used; when the viewing angle change is large (such as before and after a turn), a dynamic blur transition is used to simulate the visual changes when the vehicle turns, enhancing the coherence of the video.

[0112] Second, navigation guide elements are superimposed in the video to improve the intuitiveness of navigation information. According to the turning instructions returned by the direction API (such as "turn left in 500 meters"), a text prompt and an arrow icon are added at the specified position of the corresponding image frame (such as the upper right corner of the screen): when the vehicle approaches the turning point, the arrow icon gradually enlarges to emphasize the prompt; after the turning operation is completed, the prompt element automatically disappears. At the same time, a progress bar is added at the bottom of the video to display the progress of the current position in the entire route in real time (such as "30% traveled"), and the remaining distance and estimated arrival time are marked, so that the viewer can clearly understand the driving state.

[0113] Finally, the video is encoded and rendered to generate a navigation video that conforms to the standard format. The video resolution is set to be consistent with the static image (such as 640x480 pixels), the frame rate is calculated according to the average playing interval (such as 2fps for an average of 0.5 seconds per frame), and the encoding format is H.264 to ensure compatibility and compression efficiency. After rendering, the video is quality checked to check whether the image sequence is coherent, the playing rate is consistent with the driving speed, and the navigation guide is accurately superimposed, finally generating a navigation video that can realistically simulate the entire process of the vehicle driving from the source point to the destination along the navigation route, providing high-quality dynamic visual materials for subsequent visual language model processing.

[0114] Step S202, a plurality of navigation images are extracted from the navigation video.

[0115] In the embodiment of the present application, when multiple navigation images are extracted from the navigation video, first, the total number of frames F to be extracted is determined, which needs to be set according to the video duration and the accuracy requirement of subsequent text generation (for example, if the video duration is 60 seconds, F is set to 30, that is, one frame is extracted every 2 seconds). The video is uniformly sampled according to the time axis by using a video analysis tool, and image frames are selected at a fixed time interval (total duration / F) from the starting frame to the ending frame of the video, so as to ensure that the extracted frames can cover the key scenes of the entire navigation route (such as the starting point, the ending point, the intersection, the turning point, etc.). For example, for a 60-second video containing the process of “straight ahead—left turn—straight ahead”, if F = 6, one frame is extracted at the 10th second (at the beginning of straight ahead), the 20th second (near the intersection), the 30th second (at the moment of left turn), the 40th second (at the beginning of straight ahead after left turn), the 50th second (near the ending point), and the 60th second (at the ending point). The extracted image frames need to retain the original resolution and color information, and remove the navigation prompt icons (such as arrows and text) in the video, and only retain the original visual content of the road environment to provide pure image input for subsequent visual language model processing.

[0116] In step S203, multiple candidate navigation descriptions are generated based on the continuous image frames and the corresponding guide text, and a consistency evaluation score corresponding to each candidate navigation description is calculated by using a similarity index, and the candidate navigation description with the highest consistency evaluation score is taken as the navigation guide information, wherein the consistency evaluation score is used to represent the matching degree between the candidate navigation description and the actual navigation scene requirement.

[0117] In the embodiment of the present application, when multiple candidate navigation descriptions are generated based on the continuous image frames and the corresponding guide text, the F frames of images are arranged in time sequence to form a continuous frame sequence, which is input into a visual language model (VLM), and combined with the corresponding guide text (such as “analyze the intersection and road direction in the image to determine the driving direction; estimate the driving distance according to the actual distance corresponding to the frame interval, and generate natural language navigation instructions”), so as to prompt the model to output N candidate navigation descriptions (such as N = 5, each description contains key information such as direction and distance). Then, the consistency evaluation score of each candidate navigation description is calculated.

[0118] In the embodiment of the present application, the consistency evaluation score corresponding to each candidate navigation description is calculated by using a similarity index, including the following steps D1-D6:

[0119] In step D1, the directional keywords in each candidate navigation description are obtained, and the target keywords with intersection are obtained by comparing the directional keywords in each candidate navigation description.

[0120] Specifically, first, for each candidate navigation description, the directional keywords are extracted by natural language processing tools (such as word segmenter, part-of-speech tagger). Directional keywords are usually verbs or noun phrases that describe the driving direction, such as "turn left", "turn right", "go straight", "turn around", "drive on the left", "drive on the right", etc. For example, for the candidate navigation description "300 meters ahead, turn left into the main road, then go straight for 500 meters", the extracted directional keywords are "turn left" and "go straight"; for the candidate navigation description "drive forward for 200 meters and turn right, then go straight along the current road", the extracted keywords are "go straight" and "turn right".

[0121] After extracting the directional keywords of all candidate navigation descriptions, the keyword set is constructed. Assuming that there are N candidate navigation descriptions, the keyword set of each candidate is denoted as K1, K2,..., K n , where K i = {k i1 , k i2 ,..., k im} (m is the number of keywords of the i-th candidate). Then, the intersection of all candidate keyword sets is calculated, that is, the target keyword set T = K1∩K2∩…∩K n . The target keyword needs to appear in all candidate navigation descriptions, for example, if all candidates contain "go straight", "go straight" belongs to the target keyword; if a candidate contains "turn left" and other candidates do not mention it, "turn left" does not belong to the target keyword.

[0122] In actual operation, the situation of synonyms or near synonyms needs to be handled, for example, "turn left" and "left turn" are essentially the same, which need to be normalized to a unified keyword through a synonym dictionary or semantic similarity calculation (such as Word2Vec vector cosine similarity) to avoid misjudgment of the intersection due to differences in expression. For example, "turn left" and "left turn" are unified as "left turn" to ensure the consistency of keyword extraction. The final target keyword set reflects the consensus of all candidate navigation descriptions in terms of direction, which is the core basis for subsequent evaluation of the accuracy of directional keywords.

[0123] Step D2, based on the number of target keywords in each candidate navigation description and the total number of directional keywords corresponding to all candidate navigation descriptions, the intersection similarity corresponding to each candidate navigation description is calculated.

[0124] Specifically, the number of target keywords contained in each candidate navigation description is counted. For the i-th candidate navigation description, the number of target keywords C i is the number of intersection elements of the keyword set K i of the candidate and the target keyword set T, that is, C i = |K i|T|. For example, if the target keyword set T = {“go straight” “turn right”}, a certain candidate’s keyword set K i = {“go straight” “turn right” “turn left”}, then C i = 2 (contains two target keywords “go straight” and “turn right”); if another candidate’s K j = {“go straight” “turn around”}, then C j = 1 (contains only “go straight”).

[0125] Next, the total number of directional keywords corresponding to all candidate navigation descriptions is calculated. The total number of keywords M is the number of elements in the union of all candidate keyword sets, i.e. M = |K1∪K2∪…∪K n |. For example, if K1 = {“turn left” “go straight”}, K2 = {“go straight” “turn right”}, and K3 = {“go straight”}, then the union is {“turn left” “go straight” “turn right”} and M = 3.

[0126] The formula for calculating the intersection similarity S_inter is the ratio of the number of target keywords in this candidate to the total number of keywords. The higher the ratio, the more consensus directional keywords the candidate contains, and the stronger the consistency with other candidates in directional description.

[0127] For example, in the above example, the first candidate’s S_inter = 2 / 3 ≈ 0.67 and the second candidate’s S_inter = 1 / 3 ≈ 0.33, obviously the first candidate is more accurate in directional keywords. During the calculation, if the total number of keywords M is 0 (i.e. all candidates have no directional keywords), the intersection similarity is 0, indicating that the candidate navigation description has defects.

[0128] Step D3, obtain the distance data in each candidate navigation description, and calculate the distance similarity corresponding to each candidate navigation description based on the difference in distance data between candidate navigation descriptions.

[0129] Specifically, the distance data is extracted from each candidate navigation description. Distance data is usually a phrase containing a numerical value and a unit, such as “300 meters” “1.5 kilometers” “50 meters”, etc. It needs to be extracted from the text through regular expressions and unified units (such as converting “kilometers” to “meters”, 1 kilometer = 1000 meters) to obtain standardized distance values, denoted as d i (the distance value of the i-th candidate). For example, the distance data extracted from the candidate navigation description “go straight for 500 meters and turn left” is 500 meters, and the distance data extracted and converted from the candidate navigation description “go straight for 0.3 kilometers and turn right” is 300 meters.

[0130] For N candidate navigation descriptions, the distance difference between each two candidates is calculated. The distance difference between the ith candidate and the jth candidate can be represented by absolute difference or relative difference, where the relative difference is adopted to reflect the proportional relationship: Ad(i,j) = 1 - |d i -d j | / max(d i ,d j )(when d i and d j are not both 0). If d i or d j is 0, then Ad(i,j) = 0. For example, d i = 500 meters, d j = 450 meters, Ad(i,j) = 1 - |500 - 450| / 500 = 1 - 0.1 = 0.9, indicating that the distance difference between them is small; if d j = 1000 meters, then Ad(i,j) = 1 - 500 / 1000 = 0.5, the difference is larger.

[0131] The distance similarity S_dist of each candidate is the average of the distance differences between the candidate and all other candidates. For example, if the Ad of a candidate with 3 other candidates are 0.9, 0.8, and 0.7 respectively, then S_dist = (0.9 + 0.8 + 0.7) / 3 = 0.8. The closer this value is to 1, the higher the consistency of the distance data of the candidate with other candidates, and the more accurate the distance precision conforms to the consensus.

[0132] Step D4, extract the vocabulary in each candidate navigation description, and count the number of common vocabulary in the candidate navigation description.

[0133] Specifically, text preprocessing is performed on each candidate navigation description: punctuation marks (such as commas, periods) are removed, stop words (such as “of”, “in”, “after” and other words without actual meaning) are removed, and morphological reduction (such as “traveling” and “traveling”) is performed to obtain a list of purified words. For example, the candidate navigation description “300 meters ahead, turn left into the main road, then straight ahead for 500 meters” is preprocessed, and the word list is {“front” “300 meters” “left turn” “enter” “main road” “then” “straight” “500 meters”}.

[0134] After extracting the word list of all candidates, a global word set U is constructed, which is the union of all candidate word lists, U = {w1, w2, …, w k} (k is the total number of global vocabulary). Then, the frequency of each vocabulary appearing in the candidate navigation description is counted, and the common appearing vocabulary is screened out - if a vocabulary w appears in the vocabulary list of at least two candidates, it is determined as a common appearing vocabulary. For example, if "turn left" appears in candidate navigation description 1 and candidate navigation description 2, and "go straight" appears in candidate navigation description 1, candidate navigation description 2 and candidate 3, then "turn left" and "go straight" are common appearing vocabularies.

[0135] For each candidate i, the number of intersection of its vocabulary list and the common appearing vocabulary set, i.e. the number of common vocabularies contained in the candidate V i = |vocabulary list i ∩ common vocabulary set | is counted. For example, the vocabulary list of candidate navigation description 1 contains "turn left", "go straight" and "300 meters", and the common vocabulary set is { "turn left", "go straight", "enter"}, so V i = 2 (contains "turn left" and "go straight"). At the same time, the total number of all common appearing vocabularies V_total, i.e. the number of elements of the common vocabulary set, is recorded, for example, V_total of the above common vocabulary set is 3. These two values will be used as core parameters for calculating the vocabulary similarity in step D5.

[0136] Step D5, based on the number of vocabularies in each candidate navigation description and the total number of vocabularies corresponding to all candidate navigation descriptions, the vocabulary similarity corresponding to each candidate navigation description is calculated.

[0137] Specifically, the vocabulary similarity of each candidate is calculated based on the number of vocabularies, and the consistency at the vocabulary level is quantified by using the Jaccard index. The calculation formula of the Jaccard index is: for two sets A and B, the Jaccard index J(A, B) = |A∩B| / |A∪B|. In the vocabulary similarity calculation, the Jaccard index of each candidate with all other candidates is calculated, and the average value is taken as the vocabulary similarity of the candidate.

[0138] Specifically, for candidate i and candidate j, their vocabulary sets are A i and A j (respectively, the preprocessed vocabulary list in step D4), the number of intersection of the two vocabularies is |A i ∩A j |, the number of union is |A i ∪A j |, and the Jaccard index J(i, j) = |A i ∩A j | / |A i ∪A j |. For example, A i = { "turn left", "go straight", "300 meters"}, A j= { "right turn" "go straight" "500 meters"}, then |A i ∩A j | = 1 ("go straight"), |A i ∪A j | = 4 ("left turn" "go straight" "300 meters" "right turn" "500 meters" after removing duplicates is 4), J(i,j) = 1 / 4 = 0.25.

[0139] For candidate i, the word similarity S_word is the average of the Jaccard index of this candidate with all other candidates. For example, candidate i has J values of 0.25, 0.5, 0.75 with 3 other candidates, then S_word = (0.25 + 0.5 + 0.75) / 3 = 0.5. The higher this value, the more overlapping the word selection of this candidate is with other candidates, and the more consistent the language expression style is. If the word set of a certain candidate has no intersection with other candidates (J value is 0), then S_word = 0, indicating that its word usage deviates greatly from the consensus.

[0140] Step D6, using the intersection similarity, distance similarity and word similarity, calculate the consistency evaluation score corresponding to each candidate navigation description.

[0141] Specifically, first, assign weights η1, η2, η3 to the intersection similarity, distance similarity and word similarity, and satisfy η1> η2> η3 (such as η1= 0.5, η2= 0.3, η3= 0.2), to reflect that the priority of direction accuracy is the highest, followed by distance accuracy, and finally the word similarity.

[0142] For each candidate i, calculate its overall similarity S_over(i,j) with other candidate j = η1x S_inter(i,j) + η2x S_dist(i,j) + η3x S_word(i,j), where S_inter(i,j) is the intersection similarity of candidate i and j, S_dist(i,j) is the distance similarity of the two, and S_word(i,j) is the Jaccard index of the two.

[0143] For example, S_inter of candidate i and j is 0.6, S_dist is 0.8, and S_word is 0.5, then S_over(i,j) = 0.5x 0.6 + 0.3x 0.8 + 0.2x 0.5 = 0.3 + 0.24 + 0.1 = 0.64.

[0144] Subsequently, calculate the cumulative similarity of each candidate, that is, the consistency evaluation score:

[0145] S_total(i) = Σ S_over(i,j) (j ≠ i).

[0146] For example, the S_over of candidate i and 3 other candidates are 0.64, 0.72, 0.58, respectively, and S_total(i) = 0.64 + 0.72 + 0.58 = 1.94. The higher the score, the stronger the overall consistency of the candidate with other candidates in the direction, distance, and vocabulary dimensions, and the higher the matching degree with the true navigation scene requirements. Finally, the candidate navigation description with the maximum S_total(i) is selected as the optimal navigation guidance information.

[0147] As an example, as shown in Figure 4

[0148] First, the coordinates of the source point (starting position latitude and longitude) and the terminal point (target position latitude and longitude) are calculated using the three-dimensional translation vector and the latitude and longitude of the starting point. The navigation route is planned by calling the map API, and multiple frames of images are obtained by uniformly sampling the navigation video corresponding to the planned route, thereby generating multiple frames of navigation images.

[0149] Next, these multiple frames of navigation images are input into a visual language model (VLM) together with specially designed prompt words (such as "analyze the changes in the intersections on the path and generate navigation instructions containing distance, direction adjustment, etc."). The model determines the changes in the intersections on the path based on the road scene presented by the image (such as whether there are changes in the intersections, traffic lights, or other traffic signals) and the requirements of the prompt words. If there are no changes in the intersections, the expression "continue driving on the current road (total_length meters)" is generated. If there are changes in the intersections, the distance to this point is calculated (= (frame interval to this point / frame interval to this point - frame interval to the previous point) * total_length meters), and combined with the traffic signals and other conditions, N candidate navigation descriptions (Candidate1 to CandidateN, each containing different expressions of driving instructions such as "turn left after driving X meters due to changes in the intersection ahead") are generated.

[0150] Subsequently, the candidate navigation descriptions are screened through self-consistency evaluation (comparing the logical consistency between the candidate navigation descriptions and the scene adaptation degree), calculating the semantic similarity (using a text matching algorithm to measure the degree of overlap of each candidate in distance, direction adjustment, etc.), and selecting the candidate that best fits the scene and has the most accurate semantics as the final navigation guidance information.

[0151] Step S204, based on the visual features in the multi-view visual data, generate bird's eye view features, and based on the navigation guidance information and the multi-view visual data, perform semantic analysis of the driving scene to obtain a text probability distribution.

[0152] ​In the embodiment of the present application, when generating the bird's eye view feature, first, feature extraction is performed on the multi-view visual data: the images collected by the front view, side view and surround view cameras are processed by a convolutional neural network such as ResNet, and high-dimensional visual features (such as feature maps containing lane lines, obstacles and traffic signs) of each view are output. Subsequently, a view conversion model is established based on the camera calibration parameters (internal parameter matrix and external parameter matrix) to map the features of each view to a bird's eye view coordinate system with the vehicle as the origin (the x-axis is the longitudinal distance and the y-axis is the lateral distance). The mapping process uses an inverse perspective transformation algorithm. Through the conversion formula of pixel coordinates and three-dimensional space coordinates, the road features (such as inclined lane lines) in the perspective view are converted into parallel features in the bird's eye view. Then, through feature stitching and attention weighting (strengthening the features of close-range obstacles and lane lines), a uniform size bird's eye view feature map is generated to realize the global spatial representation of the 360° environment around the vehicle.

[0153] When performing driving scene semantic analysis, the navigation guide information and multi-view visual data are processed by a visual language model. The navigation guide information (such as “turn left in 500 meters”) is encoded into a text feature vector, which is fused with the multi-view visual features through cross-modal fusion: the text features guide the visual features to focus on key areas such as “500-meter range” and “left turn intersection”, and the visual features constrain the text semantic analysis in the opposite direction (such as correcting the feasibility of the text instruction by recognizing the “no left turn” sign).

[0154] After fusion, the model outputs a text probability distribution through a fully connected layer and a softmax function, which quantifies the adaptability of each semantic label with a probability value, covering scene types (such as “intersection” and “straight road”), action constraints (such as “allow left turn” and “need to slow down”), distance intervals (such as “400-600 meters”), etc. For example, when a green light at the intersection is detected and there is a left turn lane, the probability value of “allow left turn” increases to 0.8 or above, and the probability value of “no left turn” decreases to 0.1 or below, forming a semantic probability output that highly matches the current scene, providing a text-level decision basis for subsequent cross-modal fusion.

[0155] In step S205, cross-modal fusion is performed based on the bird's eye view feature and the processed text probability feature to obtain a fusion feature, and the fusion feature is input into multiple automatic driving task networks to obtain automatic driving decision information output by each automatic driving task network.

[0156] In the embodiment of the present application, when performing cross-modal fusion, first, the bird's eye view feature and the processed text probability feature are dimensionally aligned: the bird's eye view feature (such as 512x512xC) is converted into a one-dimensional vector (dimension D) through global average pooling, and the text probability feature (such as dimension M) is mapped to the same dimension D through a fully connected layer, ensuring that the two can be arithmetically operated.

[0157] The fusion process adopts a double-path strategy of "splicing + weighted addition": the splicing path combines the aligned two types of features into a 2D dimensional vector in the dimension, preserving the integrity of the features; the weighted addition path dynamically fuses the two types of features through a learnable parameter a (initial value 0.5) (a x bird's eye view feature + (1-a) x text feature), achieving adaptive balance of spatial information and semantic information (such as increasing the weight of bird's eye view feature when driving at high speed, and increasing the weight of text probability when passing through complex intersections). The fused features are processed by a fusion network composed of multiple layers of perception mechanisms, and through ReLU non-linear activation and batch normalization, a fusion feature vector is generated that combines spatial layout details (such as lane width, obstacle position) and semantic decision logic (such as "left turn feasibility 80%").

[0158] When the fusion features are input into the automatic driving task network, special sub-networks are designed for different tasks: the perception network adopts the YOLOv8 architecture, uses the bird's eye view spatial positioning and text semantic guidance (such as "pay attention to pedestrians at the intersection") in the fusion features to improve the detection accuracy of traffic signal light status (red / green), pedestrian position (coordinate error ≤0.5 meters), and vehicle distance (relative error ≤5%); the prediction network is based on the Transformer architecture, and through the global bird's eye view layout (such as the trajectory of the vehicle in front) and semantic probability (such as "probability of changing lanes 60%") in the fusion features, it predicts the behavior (such as changing lanes, slowing down, stopping) of other road users within the next 5 seconds, and outputs the behavior probability distribution; the planning network combines the Model Predictive Control (MPC) algorithm, generates multiple candidate paths under the constraint of fusion features (such as lane width limit, navigation instruction "turn left"), selects the optimal path through the evaluation function (comprehensive safety, comfort, navigation fit), and outputs specific control instructions (such as steering angle ±30°, acceleration -2 to 2 m / s 2 ). Finally, the outputs of each task network are integrated by the decision fusion module to form complete automatic driving decision information covering environment perception, behavior prediction, and path planning, supporting the vehicle to realize safe and compliant autonomous driving.

[0159] As a complete example, as Figure 5 shown:

[0160] When the vehicle is driving, it first captures the surrounding scene through different orientation cameras (front view, rear view, side view, etc.) to collect multi-view visual data, and at the same time acquires navigation guidance information (such as text of explicit driving instructions such as "straight ahead for 53.6 meters, turn left at the traffic signal, then straight ahead on the road for 80 meters").

[0161] The multi-view visual data is input into the bird's eye view encoder and converted into bird's eye view features, presenting a global road layout containing lane lines, obstacles, and other spatial distribution information.

[0162] The navigation guide information is input into the visual language model together with the multi-view visual data. The model outputs a probability result (such as a probability value of left turn feasibility, straight driving adaptability) after understanding the scene semantics. The result is processed by a sparse reduction multi-layer perceptron to simplify the key decision information. Then, the bird's eye view features and the processed probability features are spliced and added, and input into a feature fusion multi-layer perceptron to generate fusion features with spatial layout and semantic decision logic.

[0163] The fusion features are input into an autonomous driving task network. The perception network accurately identifies road elements such as traffic lights and pedestrians based on the fusion features. The prediction network judges the motion trend of surrounding vehicles and pedestrians based on the fusion features. The planning network plans a driving path based on the fusion features and outputs an autonomous driving decision such as "safely turn left into the corresponding lane at the traffic light".

[0164] In this embodiment, a processing apparatus for an autonomous driving task is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0165] The embodiment provides a processing apparatus for an autonomous driving task, as shown in Figure 6 , comprising:

[0166] The acquisition module 601 is configured to acquire multi-view visual data collected by a vehicle during driving, and acquire navigation guide information of the vehicle.

[0167] The analysis module 602 is configured to generate bird's eye view features based on visual features in the multi-view visual data, and perform driving scene semantic analysis based on the navigation guide information and the multi-view visual data to obtain a text probability distribution.

[0168] The processing module 603 is configured to perform cross-modal fusion based on the bird's eye view features and the processed text probability features to obtain fusion features, and input the fusion features into a plurality of autonomous driving task networks to obtain autonomous driving decision information output by each autonomous driving task network.

[0169] In the embodiment of the present application, the acquisition module 601 is specifically configured to acquire a navigation video of a vehicle driving from a source point to a destination, wherein the navigation video is generated based on a navigation route between the source point and the destination and continuous videos on the navigation route; extract a plurality of navigation images from the navigation video; generate a plurality of candidate navigation descriptions based on the continuous images and corresponding guide texts, and calculate a consistency evaluation score corresponding to each candidate navigation description by using a similarity index, and take the candidate navigation description with the highest consistency evaluation score as the navigation guide information, wherein the consistency evaluation score is used to represent the matching degree between the candidate navigation description and the actual navigation scene requirement.

[0170] In the embodiment of the present application, the acquisition module 601 is specifically configured to acquire a navigation route of a vehicle from a source point to a destination from a navigation map, and acquire continuous images on the navigation route; predict a driving distance and a driving time length based on the navigation route; arrange the continuous images on the navigation route in a sequence according to the order of the route to obtain an image sequence, and determine a playing rate of each image in the continuous images based on the driving distance and the driving time length; dynamically concatenate the continuous images according to the image sequence and the playing rate of each image in the continuous images to generate a navigation video of the vehicle driving from the source point to the destination along the navigation route.

[0171] In the embodiment of the present application, the acquisition module 601 is configured to acquire directional keywords in each candidate navigation description, compare the directional keywords in each candidate navigation description to obtain target keywords with intersection, calculate intersection similarity corresponding to each candidate navigation description based on the number of target keywords in each candidate navigation description and the total number of directional keywords corresponding to all candidate navigation descriptions, acquire distance data in each candidate navigation description, and calculate distance similarity corresponding to each candidate navigation description based on the difference of the distance data between the candidate navigation descriptions, extract words in each candidate navigation description, count the number of words corresponding to the common appearing words in the candidate navigation descriptions, calculate word similarity corresponding to each candidate navigation description based on the number of words in each candidate navigation description and the total number of words corresponding to all candidate navigation descriptions, and calculate consistency evaluation score corresponding to each candidate navigation description by using the intersection similarity, the distance similarity and the word similarity.

[0172] In the embodiment of the present application, the analysis module 602 is specifically configured to extract basic visual features under each perspective in the multi-perspective visual data; map the basic visual features under each perspective to a bird's eye perspective to obtain bird's eye perspective features based on a spatial mapping relationship between different perspectives and the bird's eye perspective.

[0173] In the embodiment of the present application, the analysis module 602 is specifically configured to input the navigation guide information and the multi-view visual data into the trained visual language model; analyze the semantic intention of the navigation guide information through the visual language model, understand the driving scene in combination with the multi-view visual data, and output the text probability distribution corresponding to the automatic driving action.

[0174] In the embodiment of the present application, the analysis module 602 is specifically configured to identify the semantic instruction intention corresponding to the navigation guide information through the visual language model; determine the current driving scene of the vehicle based on the multi-view visual data, and analyze the applicability between the semantic instruction intention and the driving scene; predict the probability value of at least one automatic driving action based on the applicability, and construct the text probability distribution adapted to the navigation guide instruction and the current scene based on the probability value.

[0175] Please refer to Figure 7 , Figure 7 is a structural schematic diagram of an electronic device provided by an optional embodiment of the present application, as shown in Figure 7 , the electronic device includes one or more processors 10, a memory 20, and an interface for connecting various components, including a high-speed interface and a low-speed interface. Various components are communicatively connected to each other by using different buses, and can be installed on a common mainboard or in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in the memory or graphics information of the memory to display a GUI on an external input / output device, such as a display device coupled to the interface. In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0176] The processor 10 can be a central processor, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic gate array, a general array logic, or any combination thereof.

[0177] The memory 20 stores instructions executable by the at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0178] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system and applications required by at least one function. The data storage area can store data created by the use of the electronic device according to the presentation of a small program landing page, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one of a magnetic disk storage device, a flash memory device, or other non-transitory solid state memory device. In some alternative embodiments, the memory 20 can optionally include a memory that is remotely located with respect to the processor 10, and these remote memories can be connected to the electronic device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0179] The memory 20 can include a volatile memory such as a random access memory, and can further include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk, and combinations thereof.

[0180] The electronic device further includes a communication interface 30 for communication of the electronic device with other devices or communication networks.

[0181] The embodiments of the present application also provide a computer readable storage medium. The above-mentioned method according to the embodiments of the present application can be implemented in hardware, firmware, or as computer code recorded on a storage medium, or be stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded to a local storage medium through a network, and thus the method described herein can be processed by such software using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, and the like. Further, the storage medium can include combinations of the above-mentioned kinds of storage media. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above-mentioned embodiments is implemented.

[0182] Although the embodiments of the present application have been described with reference to the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes are intended to fall within the scope of the appended claims.

Claims

1. A method for processing autonomous driving tasks, characterized in that, The method includes: Acquire multi-view visual data collected during vehicle operation, and acquire navigation guidance information of the vehicle; Bird's-eye view features are generated based on the visual features in the multi-view visual data, and semantic analysis of the driving scene is performed based on the navigation guidance information and the multi-view visual data to obtain the text probability distribution. Cross-modal fusion is performed based on bird's-eye view features and processed text probability features to obtain fused features. These fused features are then input into multiple autonomous driving task networks to obtain autonomous driving decision information output by each autonomous driving task network.

2. The method according to claim 1, characterized in that, The process of obtaining the navigation guidance information of the vehicle includes: Acquire navigation video of the vehicle traveling from the source point to the destination, wherein the navigation video is generated based on the navigation route between the source point and the destination and continuous video along the navigation route; Extract multiple frames of navigation images from the navigation video; Multiple candidate navigation descriptions are generated based on consecutive image frames and corresponding guidance text. A consistency evaluation score is calculated for each candidate navigation description using a similarity index. The candidate navigation description with the highest consistency evaluation score is used as the navigation guidance information. The consistency evaluation score is used to represent the degree of matching between the candidate navigation description and the requirements of the real navigation scenario.

3. The method according to claim 2, characterized in that, The acquisition of navigation video of the vehicle traveling from the source point to the destination includes: Obtain the navigation route from the source point to the destination corresponding to the vehicle from the navigation map, and obtain continuous images along the navigation route; Predict driving distance and driving time based on the navigation route; The continuous images on the navigation route are arranged in chronological order to obtain an image sequence, and the playback rate of each frame in the continuous images is determined based on the driving distance and the driving time. According to the image sequence and the playback rate of each frame in the continuous images, the continuous images are dynamically concatenated to generate a navigation video of the vehicle traveling from the source point to the destination along the navigation route.

4. The method according to claim 2, characterized in that, The calculation of the consistency evaluation score for each candidate navigation description using a similarity index includes: Obtain the directional keywords in each candidate navigation description, compare the directional keywords in each candidate navigation description, and obtain the target keywords that have intersection; Based on the number of target keywords in each candidate navigation description and the total number of directional keywords corresponding to all candidate navigation descriptions, the intersection similarity of each candidate navigation description is calculated. Obtain distance data from each candidate navigation description, and calculate the distance similarity between each candidate navigation description based on the difference in distance data between the candidate navigation descriptions; Extract the words from each candidate navigation description and count the number of words corresponding to the common words in the candidate navigation descriptions; Based on the number of words in each candidate navigation description and the total number of words corresponding to all candidate navigation descriptions, calculate the word similarity corresponding to each candidate navigation description. The consistency evaluation score corresponding to each candidate navigation description is calculated using the intersection similarity, the distance similarity, and the lexical similarity.

5. The method according to claim 1, characterized in that, The generation of bird's-eye view features based on visual features in the multi-view visual data includes: Extract the basic visual features from each viewpoint in the multi-view visual data; Based on the spatial mapping relationship between different viewpoints and the bird's-eye view, the basic visual features of each viewpoint are mapped to the bird's-eye view to obtain the bird's-eye view features.

6. The method according to claim 1, characterized in that, The step of performing semantic analysis of the driving scene based on the navigation guidance information and the multi-view visual data to obtain the text probability distribution includes: The navigation guidance information and the multi-view visual data are input into the trained visual language model; The visual language model is used to parse the semantic intent of the navigation guidance information, and multi-view visual data is combined to understand the driving scenario, outputting the text probability distribution corresponding to the autonomous driving actions.

7. The method according to claim 6, characterized in that, The process of parsing the semantic intent of the navigation guidance information using the visual language model, understanding the driving scenario by combining multi-view visual data, and outputting the text probability distribution corresponding to autonomous driving actions includes: The visual language model is used to identify the semantic instruction intent corresponding to the navigation guidance information; The current driving scenario of the vehicle is determined based on the multi-view visual data, and the applicability between the semantic instruction intent and the driving scenario is analyzed. Based on the applicability, predict the probability value of at least one autonomous driving action, and construct a text probability distribution that is adapted to navigation guidance instructions and the current scene based on the probability value.

8. A processing device for autonomous driving tasks, characterized in that, The device includes: The acquisition module is used to acquire multi-view visual data collected by the vehicle during driving, as well as to acquire the navigation guidance information of the vehicle. The analysis module is used to generate bird's-eye view features based on the visual features in the multi-view visual data, and to perform semantic analysis of the driving scene based on the navigation guidance information and the multi-view visual data to obtain the text probability distribution. The processing module is used to perform cross-modal fusion based on bird's-eye view features and processed text probability features to obtain fused features, and input the fused features into multiple autonomous driving task networks to obtain autonomous driving decision information output by each autonomous driving task network.

9. An electronic device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

Images