Vision mark based autonomous driving space planning enhancement method

Abstract

The application discloses an automatic driving space planning enhancement method based on visual markers, comprising the following steps: acquiring an original image and text input; processing the original image to obtain image features; processing the text input to obtain text features; generating text output with visual markers by using the image features and the text features; converting the text output with visual markers to obtain text output with coordinates; significantly improving the accuracy and semantic consistency of space understanding in the automatic driving scene, realizing high synchronization of visual perception and semantic expression, and effectively solving the problem of semantic split between vision and language modal in the existing method. Not only greatly improves the analysis accuracy of object position, motion state and interaction relationship in the automatic driving question and answer task, but also can significantly enhance the decision reliability and planning naturalness in the complex driving scene.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving technology and artificial intelligence technology, and in particular to an enhanced method for autonomous driving spatial planning based on visual tags. Background Technology

[0002] With the rapid development of autonomous driving technology, accurately perceiving spatial relationships (such as object localization, motion state, and interaction intent) in complex traffic scenarios has become a core challenge for achieving safe decision-making and planning. As a key innovation in intelligent transportation systems and autonomous driving, it requires real-time analysis of environmental semantics through multimodal data to construct interpretable driving logic. However, existing autonomous driving visual question answering (AD-VQA) methods based on multimodal large language models (MLLMs) have significant bottlenecks in spatial information representation. Directly encoding coordinate information in text form leads to semantic disconnect between visual representation and language description, which not only increases the model's expressive complexity but also easily causes cross-modal alignment errors, ultimately affecting the reliability of perception, prediction, and planning tasks.

[0003] There are specific technical challenges in spatial understanding of autonomous driving technology. Generating spatial semantic descriptions directly from multimodal inputs (visual images and text commands) has ambiguity in cross-modal alignment. Since there is a one-to-many dynamic mapping relationship between language description logic and visual coordinate features (e.g., "vehicle on the left" may correspond to multiple candidate coordinate regions), this process is prone to overgeneralization of scene parsing, which manifests as fuzzy object localization, misjudgment of motion state, and inaccurate description of spatial relationships.

[0004] While existing text-based coordinate-based methods can enhance target localization accuracy, their reliance on direct embedding of numerical coordinates leads to semantic fragmentation between visual representation and language modality, significantly increasing the complexity of cross-modal alignment and representational errors. Furthermore, although methods based on multimodal large language models have achieved breakthroughs in autonomous driving visual question answering tasks, they often fail to capture fine-grained spatiotemporal relationships when dealing with dynamic interactions in dense traffic flows (such as vehicle entry intention prediction or pedestrian obstacle avoidance planning). In particular, when spatial understanding needs to be strictly synchronized with real-time decision-making logic (such as lane change priority judgment and multi-objective risk classification), the outputs of existing methods often exhibit semantic fragmentation or overly conservative tendencies, resulting in impaired spatiotemporal continuity of the planning process.

[0005] Therefore, to bridge the semantic gap between visual coordinate representation and linguistic description, and to improve the interpretability and robustness of spatial reasoning, it is of urgent significance to research and develop an autonomous driving enhancement framework that integrates dual-granularity visual cues. This will not only strengthen environmental perception and decision-making reliability in complex scenarios, but also promote the paradigm upgrade of autonomous driving systems from "perception-driven" to "cognition-driven," providing core technological support for human-vehicle collaboration and advanced intelligent driving. Summary of the Invention

[0006] (1) Technical problems to be solved

[0007] This invention discloses an enhanced spatial planning method for autonomous driving based on visual tags, aiming to solve the problem that existing text coordinate-based methods rely on the direct embedding of numerical coordinates, which increases the cross-modal alignment complexity and expression error of the model; at the same time, methods based on multimodal large language models cannot capture fine-grained spatiotemporal correlations when dealing with dynamic interactions in dense traffic flows.

[0008] (2) Technical solution

[0009] This invention discloses a visual tagging-based method for enhancing spatial planning in autonomous driving, comprising the following steps:

[0010] Step 1: Obtain the original image and text input;

[0011] Step 2: Process the original image to obtain image features;

[0012] Step 3: Process the text input to obtain text features;

[0013] Step 4: Generate text output with visual markers using image and text features;

[0014] Step 5: Transform the text output with visual markers to obtain text output with coordinates.

[0015] Further, the process of processing the original image to obtain image features includes the following steps:

[0016] Step 201: Using the original image, generate a visually labeled image through a detection expert model;

[0017] Step 202: Generate scene-level features using the original image and the visually labeled image;

[0018] Step 203: Generate instance-level features using scene-level features;

[0019] Step 204: Generate image features using scene-level features and instance-level features respectively.

[0020] Furthermore, the specific steps for processing the text input to obtain text features are as follows:

[0021] Text features are extracted using the text input from step 1 via a token encoder.

[0022] Furthermore, the step of generating text output with visual markers using image features and text features includes the following steps:

[0023] Step 401: Input the text features and image features into the large language model;

[0024] Step 402: The large language model generates text output with visual markers.

[0025] Furthermore, the transformation of the text output with visual markers to obtain text output with coordinates includes the following steps:

[0026] Step 501: Extract object numbers from the text output with visual markers;

[0027] Step 502: Construct a coordinate mapping table, and query the coordinate mapping table according to the object number to obtain the object coordinates;

[0028] Step 503: Replace the object number in the text output with visual markers with the object coordinates;

[0029] Step 504: Output text with coordinates.

[0030] Furthermore, the step of generating visually labeled images using the original images through a detection expert model includes the following steps:

[0031] Step 2011: Use the StreamPETR detection expert model to identify traffic objects in the original image I according to the specified object categories;

[0032] Step 2012: The detection expert model generates detection masks for k objects based on the identified k traffic objects, represented as a binary mask set R = [r1, r2, ..., r...]. k ], where r k ∈{0,1} H×W Let H represent the k-th detection mask, where H is the height of the mask and W is the width of the mask.

[0033] Step 2013, for each r k Calculate its average centroid coordinates c k =(x k ,y k ), x k Represented as the x-axis, y k Represented as the ordinate;

[0034] Step 2014: In the original image, find the centroid coordinates c of each detected object. k Mark the corresponding index k at this location;

[0035] Step 2015: Cover the original image I with a semi-transparent detection mask r of the corresponding size. k To describe the boundaries of an object;

[0036] In step 2016, k detection masks and their corresponding indices are superimposed on the original image I to generate the visually labeled image I. m .

[0037] Furthermore, the process of generating scene-level features using the original image and visually labeled image includes the following steps:

[0038] Step 2021: Freeze the parameters θ of the original visual encoder E, and simultaneously create a new encoder with parameters θ. c Trainable copy E c ;

[0039] Step 2022: The original image I is encoded by the original visual encoder E, and the visually labeled image I is obtained. m Through trainable copy E c Obtain the encoding of the visually labeled image;

[0040] Step 2023, the encoding of the visually labeled image is performed through a process with parameter θ Z The zero-linear network Z outputs scene-level features of the visually labeled image; where the weights and biases of the zero-linear network Z are initialized to 0.

[0041] Step 2024: The scene-level features of the visually labeled image and the encoded elements of the original image are summed to obtain the scene-level features y of the original image. s The expression is:

[0042] y s =E(I;θ)+Z(E c (I m ;θ c );θ Z );

[0043] Among them, y s ∈R H×W×C H is the height, W is the width, and C is the number of channels.

[0044] Furthermore, the step of generating instance-level features using scene-level features includes the following steps:

[0045] First, through scene-level features y of the original image s And the detection mask r for the k-th object k For the detection mask r k Scaling up to the scene-level feature y s Same size;

[0046] Next, masked average pooling (MAP) is used to obtain the instance-level features of the k-th object.

[0047]

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] By employing a detection expert model to annotate the original images and generating two fine-grained image features through a visual encoder and MLP, and combining them with text features generated from text input via a token encoder, a large language model is used to generate action planning that combines image and text input. This significantly improves the accuracy and semantic consistency of spatial understanding in autonomous driving scenarios, achieving a high degree of synchronization between visual perception and semantic expression. It effectively solves the problem of semantic separation between visual and language modalities in existing methods. This not only greatly improves the parsing accuracy of object position, motion state, and interaction relationships in autonomous driving question answering tasks, but also significantly enhances the reliability of decision-making and the naturalness of planning in complex driving scenarios, making it more suitable for applications such as autonomous driving systems. Attached Figure Description

[0050] Figure 1 This is an overall flowchart of the present invention;

[0051] Figure 2 This is a flowchart of step 2 in the present invention, which involves processing the original image and obtaining image features.

[0052] Figure 3 This is a flowchart of step 5 in the present invention, which converts the text output with visual markers to obtain the text output with coordinates. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] For the sake of clarity and reference, the technical terms, abbreviations, or acronyms used below are summarized and explained as follows:

[0055] Visual encoder: A model component that extracts key visual features from an image and encodes them efficiently.

[0056] MLP: A feedforward neural network composed of fully connected layers that uses non-linear activation functions to handle complex tasks.

[0057] Large Language Model: A deep learning system trained on massive amounts of text data, capable of understanding and generating natural language.

[0058] Text token encoder: A component that segments text into basic units (tokens) and converts them into numerical vector representations for machine learning models to process and analyze.

[0059] refer to Figure 1 This invention proposes a visual tagging-based method for enhancing spatial planning in autonomous driving, specifically including the following steps:

[0060] Step 1: Obtain the original image and text input;

[0061] Step 2: Process the original image to obtain image features;

[0062] Step 3: Process the text input to obtain text features;

[0063] Step 4: Generate text output with visual markers using image and text features;

[0064] Step 5: Transform the text output with visual markers to obtain text output with coordinates.

[0065] The following example illustrates the process of a visual tagging-based method for enhancing spatial planning in autonomous driving:

[0066] Step 1, obtaining the original image and text input, specifically includes the following steps:

[0067] Step 101: The original image can be acquired by a camera or other acquisition device and processed by an upstream algorithm to obtain the image input;

[0068] Step 102: Text input can be collected by the user through a data acquisition device and processed by an upstream algorithm to obtain text input, or the system can directly input text.

[0069] Step 2: Process the original image to obtain image features, referencing... Figure 2 ;

[0070] Specifically, the following algorithm flow is included:

[0071] Step 201: Using the original image, generate a visually labeled image through a detection expert model;

[0072] Step 202: Generate scene-level features using the original image and the visually labeled image;

[0073] Step 203: Generate instance-level features using scene-level features;

[0074] Step 204: Generate image features using scene-level features and instance-level features respectively.

[0075] Specifically, step 201 involves generating a visually labeled image from the original image using a detection expert model as follows:

[0076] Step 2011: Use the StreamPETR detection expert model to identify traffic objects (including but not limited to cars, trucks, and buses) in the original image I according to the specified object categories;

[0077] Step 2012: The detection expert model generates detection masks for k objects based on the identified k traffic objects, represented as a binary mask set R = [r1, r2, ..., r...]. k ], where r k ∈{0,1} H×W Let H represent the k-th detection mask, where H is the height of the mask and W is the width of the mask.

[0078] Step 2013, for each r k Calculate its average centroid coordinates c k =(x k ,y k ), x k Represented as the x-axis, y k Represented as the ordinate;

[0079] Step 2014: In the original image, find the centroid coordinates c of each detected object. k Mark the corresponding index k at this location;

[0080] Step 2015: Cover the original image I with a semi-transparent detection mask r of the corresponding size. k To describe the boundaries of an object;

[0081] In step 2016, k detection masks and their corresponding indices are superimposed on the original image I to generate the visually labeled image I. m .

[0082] In addition, if the text input contains new coordinate references c new It will be assigned an index k+1, and a semi-transparent detection mask r of the corresponding size will be overlaid on the original image I. k+1 To maintain consistency between visual and textual patterns.

[0083] Specifically, in step 202, the process of generating scene-level features using the original image and the visually labeled image is as follows:

[0084] Step 2021: Freeze the parameters θ of the original visual encoder E, and simultaneously create a new encoder with parameters θ. c Trainable copy E c ;

[0085] Step 2022: The original image I is encoded by the original visual encoder E, and the visually labeled image I is obtained. m Through trainable copy E c Obtain the encoding of the visually labeled image;

[0086] Step 2023, the encoding of the visually labeled image is performed through a process with parameter θ Z The zero-linear network Z outputs scene-level features of the visually labeled image; where the weights and biases of the zero-linear network Z are initialized to 0.

[0087] Step 2024: The scene-level features of the visually labeled image and the encoded elements of the original image are summed to obtain the scene-level features y of the original image. s The expression is:

[0088] y s =E(I;θ)+Z(E c (I m ;θ c );θ Z );

[0089] Among them, y s ∈R H×W×C H is the height, W is the width, and C is the number of channels.

[0090] Specifically, the visual encoder used in step 202 is the InternViT-300M-448px model, whose core is an improved structure based on VisionTransformer, supporting dynamic resolution input and pixel shuffling technology. In the initial stage of signal processing, the input image is divided into several 448×448 pixel patches using a dynamic block-segmentation strategy. Each patch is downsampled to 1 / 4 of its original size through pixel shuffling, and the 256 visual tokens corresponding to a single patch are input into the visual encoder. Through a multi-layer self-attention mechanism, the model extracts global scene-level features and combines them with an object mask generated by the detection algorithm. Bilinear interpolation is used to scale the mask to the same size as the scene features, and then mask average pooling is used to extract local instance-level features.

[0091] Specifically, in step 203, instance-level features are generated using scene-level features. The specific steps are as follows:

[0092] First, through scene-level features y of the original image s And the detection mask r for the k-th object k For the detection mask r k Scaling up to the scene-level feature y s Same size;

[0093] Next, masked average pooling (MAP) is used to obtain the instance-level features of the k-th object.

[0094]

[0095] Specifically, step 204 involves generating image features using scene-level features and instance-level features respectively:

[0096] First, scene-level features and instance-level features are concatenated, and then mapped to a 4096-dimensional latent space aligned with text features via MLP to obtain image features.

[0097] In step 3, the text input is processed to obtain text features. The specific steps are as follows: the text input in step 1 is used to extract text features through a token encoder.

[0098] In step 4, text output with visual markers is generated using image and text features. The specific steps are as follows:

[0099] Step 401: Input the text features and image features into the large language model;

[0100] Step 402: The large language model generates text output with visual markers.

[0101] Specifically, the large language model used in step 4 is the InternLM2.5-7B-Chat model, which includes the following structure:

[0102] First, the InternLM2.5-7B-Chat is based on a multi-layer Transformer architecture and uses a Grouped Query Attention (GQA) mechanism to group query headers and share key / value headers, thereby reducing video memory usage while maintaining 8K context window processing capabilities.

[0103] Secondly, dynamic gating is used for cross-modal interaction. Dynamic gating cross-attention modules are inserted at layers 16, 24, and 32 of the Transformer. Visual features are used as key-value pairs (K / V), and text features are used as queries (Q). The contribution of visual information is adjusted by learnable gating weights (ranging from 0 to 1), preventing irrelevant image regions from interfering with text generation.

[0104] Step 5 involves transforming the text output with visual markers to obtain text output with coordinates. This includes the following steps, which can be seen... Figure 3 :

[0105] Step 501: Extract object numbers from the text output with visual markers;

[0106] Step 502: Construct a coordinate mapping table, and query the coordinate mapping table according to the object number to obtain the object coordinates;

[0107] Step 503: Replace the object number in the text output with visual markers with the object coordinates;

[0108] Step 504: Output text with coordinates.

[0109] Specifically, step 501 involves extracting object numbers from the text output with visual markers:

[0110] Pre-trained detection expert models (such as StreamPETR) perform multi-object detection on driving scene images, identifying traffic objects (vehicles, pedestrians, etc.) and generating unique numerical identifiers. The detection expert model first outputs a set of binary masks R = [r1, r2, r3] containing the bounding boxes of the traffic objects. 2, ...,r k Then, each detected object is assigned a consecutive integer number (e.g., ID1, ID2), and the number is embedded as text into the text output with visual tags.

[0111] Specifically, step 502 involves querying the coordinate table based on the object number to obtain the object's coordinates:

[0112] During the detection phase, a coordinate mapping table is constructed synchronously, where each object number corresponds to its geometric center coordinate c. k =(x k ,y k The coordinate table is stored in key-value pairs (e.g., ID 1 → (864.2, 468.3)). When the model generates text containing numbers, it parses the object numbers in the text using a string matching algorithm and quickly retrieves the corresponding coordinate values ​​based on a hash table retrieval mechanism, achieving a precise mapping from semantic labels to spatial locations.

[0113] Specifically, step 503 involves replacing the object number in the visually marked text output with the object coordinates:

[0114] The text is located using regular expression-based pattern recognition technology.<ID k> The marker is a string containing k, where k is an integer. By iterating through the replacement queue, each marker is dynamically replaced with its corresponding coordinate string. <x k ,y k The replacement process preserves the original text's grammatical structure while maintaining the original text's ">". A coordinate format verification module is incorporated into the replacement process to ensure that numerical accuracy and coordinate range conform to image spatial resolution constraints.

[0115] Specifically, step 504 involves outputting text with coordinates:

[0116] The replaced text sequence is input into a language model for grammatical correction and format standardization, ultimately generating spatial description text that conforms to natural language norms. The output text strictly follows the triple structure of "<object identifier (mean centroid), coordinates, camera view>" (e.g., ...).<c1,1020.0,515.8,CAM_FRONT> Meanwhile, the attention mechanism enhances the module to ensure the consistency between spatial description and contextual semantics.

[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0118] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The technical features of the above embodiments can be combined arbitrarily. Furthermore, the numbering of each step is not intended to constrain the order of the steps; their order is permissible as long as there are no strict constraints on the sequence. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, any combination of these technical features that does not contradict each other should be considered within the scope of this specification.

[0120] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A visual marker based autonomous driving space planning enhancement method, characterized in that: The method comprises the following steps: Step 1, obtaining an original image and a text input; Step 2, processing the original image to obtain image features; Step 3, processing the text input to obtain text features; Step 4, generating a text output with visual markers using the image features and the text features; Step 5, converting the text output with visual markers to obtain a text output with coordinates; The processing of the original image to obtain image features comprises the following steps: Step 201, generating a visual marker image by detecting an expert model using the original image; Step 202, generating scene-level features using the original image and the visual marker image; Step 203, generating instance-level features using the scene-level features; Step 204, generating image features using the scene-level features and the instance-level features; The generation of scene-level features using the original image and the visual marker image comprises the following steps: Step 2021, freeze the original visual encoder's parameters while creating a trainable copy with parameters ;​​ Step 2022, original image through the original visual encoder get encoding of the original image, visual marker image through the trainable replica get encoding of the visual marker image; Step 2023, the encoding of the visually labeled image is performed through a parameter... Zero linear network It outputs scene-level features of visually labeled images; among them, the zero-linear network The weights and biases are initialized to 0; Step 2024, the scene-level features of the visual mark image and the encoded elements of the original image are accumulated to obtain the scene-level features of the original image The expression is: ； wherein, H is the height, W is the width, and C is the number of channels. The generation of instance-level features using the scene-level features comprises the following steps: First, through scene-level features of the original image and the Detection mask for each object For the detection mask Scaling up to the scene-level features Same size; Next, using a mask average pooling get the instance-level features of the object ; 。 2.The method of claim 1, wherein: The processing of the text input to obtain text features comprises the following steps: Extracting text features from the text input in step 1 using a token encoder. 3.The method of claim 1, wherein: The generation of a text output with visual markers using the image features and the text features comprises the following steps: Step 401, inputting the text features and the image features into a large language model; Step 402, the large language model generating a text output with visual markers. 4.The method of claim 1, wherein: The conversion of the text output with visual markers to obtain a text output with coordinates comprises the following steps: Step 501, extracting object numbers in the text output with visual markers; Step 502, constructing a coordinate mapping table and querying the coordinate mapping table according to the object numbers to obtain object coordinates; Step 503, replacing the object numbers in the text output with visual markers with the object coordinates; Step 504, outputting the text output with coordinates.

5. The method of claim 1, wherein: The generation of a visual marker image by detecting an expert model using the original image comprises the following steps: Step 2011: Use the StreamPETR detection expert model to identify objects in the original image according to the specified object categories. Traffic objects in; Step 2012, detecting that the expert model generates a detection mask of the identified object according to the identified object, denoted as a set of binary masks , where denotes the th detection mask, H is the height of the mask, and W is the width of the mask; Step 2013, for each the average centroid coordinates thereof are calculated , denoted as the horizontal coordinate, denoted as the vertical coordinate; Step 2014, in the original image, at the centroid coordinate of each detected object mark the corresponding index ; Step 2015, in the original image overlaid with a corresponding size of a translucent detection mask to describe the object boundary; Step 2016, The detection mask and its corresponding index will be superimposed on the original image to generate a visual marker image .

Images