An automatic driving space planning enhancement method based on semantic region focusing

Abstract

The application discloses an automatic driving space planning enhancement method based on semantic region focusing, comprising the following steps: acquiring input information, including images of an automatic driving scene and texts of a space planning problem; designing a dynamic text library and constructing cross-modal correlation mapping of the text library and the images to generate a region-activated attribution graph; performing spatial visual coding on the images and fusing the attribution graph to construct an explicit spatial coding prompt, constructing a geometric feature enhancement prompt and combining the explicit spatial coding prompt to generate a structured prompt; inputting the input information and the structured prompt into a multi-modal large model to obtain an answer to the space planning problem; and introducing region attention prior through the explicit visual prompt and the text prompt, improving the attention degree of an automatic driving model to a key semantic region in a complex automatic driving scene, thereby enhancing the space planning capability of the automatic driving system.

Description

An Enhanced Spatial Planning Method for Autonomous Driving Based on Semantic Region Focusing Technical Field

[0001] This invention relates to the fields of autonomous driving technology and artificial intelligence technology, and in particular to an enhanced method for spatial planning of autonomous driving based on semantic region focusing. Background Technology

[0002] Autonomous driving technology is a revolutionary breakthrough in the transportation field, reshaping transportation systems around the world in multiple dimensions and having a significant impact on road safety, traffic efficiency, and energy consumption. With the rapid development of artificial intelligence technology, high-performance multimodal large models (MLLMs) have emerged, demonstrating significant advantages in the field of autonomous driving due to their powerful cross-modal semantic alignment and open-world cognitive capabilities. However, in driving scenarios, MLLMs not only need to complete the semantic parsing of visual information, but also need to construct interpretable relationships between objects, spaces, and behaviors in the driving scenario. This requires the model to focus on key objects and regions in the autonomous driving scenario related to the input question, and to carry out spatial planning and cognitive reasoning. This poses an unprecedented challenge to the spatial planning capabilities of MLLMs and has become a key bottleneck restricting the further improvement of the performance of autonomous driving systems.

[0003] Existing methods improve the driving scene understanding capabilities of MLLMs through domain-adaptive fine-tuning, but due to the scarcity of high-quality labeled data, the models still lack the ability to establish accurate problem-visual region associations. Furthermore, this data-driven implicit attention mechanism lacks explicit guidance, leading to irrational attention distribution in long-tailed scenarios. To alleviate data bottlenecks, some studies have attempted to introduce prior knowledge from detection expert models to guide the model's focus on specific target instances; however, their performance is severely limited by the predefined category set and detection accuracy of the detection model, making it difficult to handle undefined categories or detection failures. Secondly, these methods only focus on discrete object instances, failing to model region-level semantics, resulting in structural defects in the overall understanding of the scene. In addition, the lack of attributable association between detection priors and problem semantics prevents the model from dynamically adjusting the attention distribution to relevant targets and regions according to problem requirements, making it difficult to adapt to complex and ever-changing driving scenarios.

[0004] Therefore, in order to improve the semantic region focusing ability of MLLMs in autonomous driving scenarios and thus enhance their spatial planning capabilities, it is of urgent significance to research and develop an autonomous driving enhancement framework based on semantic region focusing. This will not only break through the performance bottleneck of autonomous driving systems in complex scenarios and improve decision reliability, but also promote the paradigm upgrade of autonomous driving systems from "perception-driven" to "cognition-driven", providing core technical support for high-level intelligent driving. Summary of the Invention

[0005] (1) Technical problems to be solved

[0006] This invention discloses an enhanced method for autonomous driving spatial planning based on semantic region focusing, which aims to solve the problem that MLLMs have difficulty focusing on key semantic regions in complex autonomous driving scenarios.

[0007] (2) Technical solution

[0008] This invention discloses an enhanced spatial planning method for autonomous driving based on semantic region focusing, comprising the following steps:

[0009] Step 1: Obtain input information, including images of the autonomous driving scenario and text of the spatial planning problem;

[0010] Step 2: Design a dynamic text library and construct a cross-modal correlation mapping between the text library and the image to generate an attribution map of region activation;

[0011] Step 3: Perform spatial visual encoding on the image and fuse the attribution map to construct an explicit spatial encoding cue, construct a geometric feature enhancement cue and combine it with the explicit spatial encoding cue to generate a structured cue;

[0012] Step 4: Combine the input information with the structured prompts to obtain the answer to the spatial planning problem in the multimodal large model.

[0013] Furthermore, step 2 includes the following steps:

[0014] Step 201: Dynamically generate a dynamic text library that is positively and negatively correlated with the spatial planning problem using domain knowledge of the driving scenario;

[0015] Step 202: Perform adaptive block resolution optimization on the input image;

[0016] Step 203: The Transformer-MM framework is used to construct the optimized cross-modal correlation mapping between the image and the text library to generate positive text attribution maps and negative text attribution maps respectively.

[0017] Step 204: Merge the positive text attribution graph and the negative text attribution graph.

[0018] Furthermore, step 3 includes the following steps:

[0019] Step 301: Perform spatial visual encoding on the image and fuse it with the attribution map;

[0020] Step 302, construct explicit spatial encoding hints;

[0021] Step 303: Construct geometric feature enhancement cues;

[0022] Step 304: Combine the explicit spatial encoding cue with the geometric feature enhancement cue to generate a structured cue.

[0023] Furthermore, in step 201, the dynamic generation employs a rule-based prior generation method and an open-domain semantic extension method based on a large language model (LLM). The specific generation steps are as follows:

[0024] Step 20101: Construct the text library using the domain knowledge of the driving scenario through a rule-based prior generation method. The text library includes a basic object library, a question enhancement library, and a negative sample library.

[0025] The basic object library covers the core entities of the driving scenario;

[0026] The question enhancement library contains explicit entities and implicit entities associated with actions extracted from the spatial planning question text;

[0027] The negative sample library includes interfering entities that are low in correlation with the decision-making process of the spatial planning problem.

[0028] In this context, entities in the basic object library and the problem enhancement library are all considered as positive text labels, and entities in the negative sample library are all considered as negative text labels.

[0029] Step 20102: Using the open-domain semantic extension method based on the Large Language Model (LLM), design a hierarchical progressive hint framework for the spatial planning problem and input it into the LLM to dynamically generate positive / negative text labels related to the spatial planning problem, thereby expanding the text library.

[0030] Further, step 203 includes the following steps:

[0031] Step 20301: The image and text label are simultaneously fed into the CLIP network in the Transformer-MM framework for encoding to obtain image features and text features. The matching score between the image and the text label is obtained after the dot product of the image features and text features.

[0032] Step 20302: Initialize the self-attention correlation mapping matrix of the visual encoder in the CLIP network;

[0033] Step 20303: Iterate through the attention modules of each Transformer network layer of the visual encoder layer by layer and calculate the gradient-weighted attention of the corresponding layer.

[0034] Step 20304: Aggregate the correlations of each layer along the depth direction of the visual encoder and update the correlation mapping matrix;

[0035] Step 20305: Take the correlation row vector corresponding to the [CLS] token in the correlation mapping matrix of the final output layer, and the column vector corresponding to the image block obtained after CLIP encoding of the image. Then, normalize the vector using the minimum-maximum normalization algorithm and upsample it to the original image resolution to obtain the positive or negative text attribution map corresponding to the text tag.

[0036] Further, step 204 includes the following steps:

[0037] Step 20401: Calculate the attribution graphs for the positive text set and the negative text set respectively.

[0038]

[0039] in, This is an attribution diagram for the main text set. Attribution diagram for negative text sets The number of attribution maps in the main text. The number of negative text attribution maps. Indicates the first A negative text attribution graph Indicates the first A text attribution graph;

[0040] Step 20402: Attribution graphs of the positive and negative text sets are merged using a weighted fusion method.

[0041]

[0042] in, This is the fused attribution map.

[0043] Further, step 301 includes the following steps:

[0044] First, the input image is divided into grids and encoded. The image is uniformly divided into a regular grid containing several image blocks, and each image block is assigned a unique spatial code.

[0045] Next, the grid is aligned with the attribution graph, and masked average pooling is used to extract the local response of the attribution graph for each image block to obtain the relevance score corresponding to each image block.

[0046] Further, step 302 includes the following steps:

[0047] First, role definition suggestion: Define the system role as an autonomous driving perception and analysis framework;

[0048] Secondly, the grid division rule prompt is: Establish a standard for orthogonal grid division of rows and columns, and adopt the coordinate system of the origin at the top left corner of the image block;

[0049] Then, the encoding rule prompts: Establish linear encoding rules and derive the mathematical mapping relationship between row and column coordinates and image block encoding. Beneficial effects.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] This invention proposes an enhanced spatial planning method for autonomous driving based on semantic region focusing. This method leverages the image-text alignment capability of CLIP (Open Set Pre-trained) to generate attribution maps of semantically relevant region activations (explicit visual cues), thereby obtaining reliable region attention priors. Furthermore, it constructs gridded visual cues through gridded visual encoding and builds hierarchical text cues (text cues) based on the attention priors of the attribution maps. This improves model scalability while enhancing the model's ability to focus on relevant semantic regions, effectively solving the problem that existing methods struggle to adaptively focus on key regions in complex driving scenarios, thus affecting response reliability and decision rationality. This invention not only significantly improves the spatial planning capability of autonomous driving systems but also significantly enhances the reliability of planning in complex driving scenarios. Attached Figure Description

[0052] Figure 1 is a flowchart of the overall process of the present invention;

[0053] Figure 2 is a flowchart of step 2 of the present invention;

[0054] Figure 3 is a flowchart of step 3 of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] For the sake of clarity and reference, the technical terms, abbreviations, or acronyms used below are summarized and explained as follows:

[0057] LLM: Large Language Model, refers to a class of artificial intelligence models based on deep learning technology, designed to process and generate natural language text.

[0058] MLLMs: Multimodal Large Models, refers to a generative model that combines large language models (LLM) with multimodal information processing capabilities. Its core feature is that it can simultaneously process and integrate data from multiple modalities such as text, images, audio, and video, thereby achieving cross-modal understanding and generation.

[0059] Attribution mapping: a technique that uses heatmaps to visually analyze the contribution of different regions in input data (such as images) to the model's prediction results. Highlighted regions indicate high contribution and are often used to explain the decision-making basis of deep learning models.

[0060] Transformer: A neural network architecture based on attention mechanisms for processing sequential data.

[0061] Transformer-MM: A general method for interpretability of attention models, designed to generate attribution graphs with correlations to input features for various Transformer architectures.

[0062] CLIP: A multimodal model that uses contrastive learning to jointly train text and images to achieve cross-modal semantic matching, bringing matching text and images closer together and pushing unmatched text and images further apart.

[0063] The primary objective of this invention is to propose an enhanced spatial planning method for autonomous driving based on semantic region focusing.

[0064] Refer to Figure 1:

[0065] An enhanced spatial planning method for autonomous driving based on semantic region focusing includes the following steps:

[0066] Step 1: Obtain input information, including images of the autonomous driving scenario and text of the spatial planning problem;

[0067] Step 2: Design a dynamic text library and construct a cross-modal correlation mapping between the text library and the image to generate an attribution map of region activation;

[0068] Step 3: Perform spatial visual encoding on the image and fuse the attribution map to construct an explicit spatial encoding cue, construct a geometric feature enhancement cue and combine it with the explicit spatial encoding cue to generate a structured cue;

[0069] Step 4: Combine the input information with the structured prompts to obtain the answer to the spatial planning problem in the multimodal large model.

[0070] The following example illustrates a process for enhancing spatial planning in autonomous driving based on semantic region focusing:

[0071] Step 1: Obtain input information, including images of the autonomous driving scenario and text of the spatial planning problem. This specifically includes the following steps:

[0072] Step 101: The image information of the autonomous driving scenario can be obtained by acquisition devices such as cameras and processed by upstream algorithms;

[0073] Step 102: The text information of the spatial planning problem can be obtained by the user through the acquisition device and processed by the upstream algorithm.

[0074] Step 2: Design a dynamic text library and construct a cross-modal correlation mapping between the text library and the image to generate an attribution map of region activation. Referring to Figure 2, this specifically includes the following steps:

[0075] Step 201: Dynamically generate a dynamic text library that is positively and negatively correlated with the spatial planning problem using domain knowledge of the driving scenario; wherein, the domain knowledge of the driving scenario is open-source knowledge commonly used in autonomous driving tasks; and the dynamic generation adopts a rule-based prior generation method and an open-domain semantic extension method based on a large language model (LLM), the specific generation steps are as follows:

[0076] Step 20101: Construct the text library using the domain knowledge of the driving scenario through a rule-based prior generation method. The text library includes a basic object library, a question enhancement library, and a negative sample library.

[0077] The basic object library covers core entities in driving scenarios, such as vehicles (car / truck / bus), traffic facilities (traffic light / sign), and pedestrians.

[0078] The question enhancement library contains explicit entities and implicit entities associated with actions extracted from the spatial planning question text. For example, when the spatial planning question text is "the truck's turning intention", the explicit entity "truck" is automatically captured, and the implicit entity "intersection" associated with the turning action is extracted. Then, it is expanded to generate a contextualized text description of "a photo of a truck in intersection".

[0079] The negative sample library includes interfering entities that are low in correlation with the spatial planning problem decision (such as sky, vegetation, building facades, billboards, ground textures, etc.); wherein, the interfering entities are obtained by processing a large number of images of autonomous driving scenarios through a model to obtain a large number of attention heatmaps about driving scenarios, and then identifying and analyzing them.

[0080] It should be noted that the entities in the basic object library and the problem enhancement library are all regarded as positive text labels, and the entities in the negative sample library are all regarded as negative text labels.

[0081] Step 20102: Using the open-domain semantic extension method based on the LLM large language model, design a hierarchical progressive prompting framework for the spatial planning problem and input it into the LLM to dynamically generate positive / negative text labels related to the spatial planning problem in order to expand the text library;

[0082] The specific prompt framework design is as follows:

[0083] Role definition hints:

[0084] You are an autonomous driving analyst. Your task is to identify critical objects related to the question, considering:

[0085] (1)Direct causal factors (eg, vehicles, traffic signals)

[0086] (2)Indirect contextual cues (eg, road conditions, weather)

[0087] (3)Non-relevant distractors (eg, decorative objects)

[0088] In other words, you are an autonomous driving analyst. Your task is to identify key objects related to the problem and consider the following factors:

[0089] (1) Direct causal factors (such as vehicles, traffic signals)

[0090] (2) Indirect contextual clues (such as road conditions, weather)

[0091] (3) Irrelevant distractions (such as decorations)

[0092] Input: {Question}, which is the text of the spatial planning problem.

[0093] Output requirements:

[0094] - Return in JSON format: {"positive": ["text1", ...], "negative": ["text2", ...]}

[0095] - Each text must comply with the CLIP template specifications

[0096] - Exclude entries that already exist in the rule base

[0097] That is: - Returns in JSON format: {"positive sample": ["text1", ...], "negative sample": ["text2",...]}

[0098] - Each piece of text must conform to the CLIP template specification.

[0099] - Exclude entries that already exist in the rule base;

[0100] Specifically, replace the above {Question} with a question that needs to be entered, such as "Can I turn left in the current scene?", and then input the above prompt frame content into the LLM to output the corresponding positive text label: "traffic light", "vehicle" and negative text label: "sky"), and automatically recognize the positive / negative text label;

[0101] Through the above prior generation and expansion process, the model automatically identifies and labels positive / negative text, which facilitates the subsequent step 203 to generate attribution maps corresponding to positive and negative text respectively; wherein, negative text labels include interference entities in the negative sample library and negative text labels output by LLM, while the text library contains positive text labels except for the negative text labels.

[0102] Meanwhile, through the design of the prompting framework, the LLM outputs positive / negative text labels that better meet the requirements, thereby increasing the number of text labels in the text library and further improving the model's recognition accuracy and the accuracy of the attribution graph.

[0103] Step 202: Perform adaptive block resolution optimization on the input image;

[0104] Specifically, the image is divided into multiple sub-graphs by short side reference. After performing attribution analysis independently on each sub-graph, a weighted average strategy is used to fuse features in the overlapping areas. This effectively expands the feature coverage of the CLIP model in asymmetric resolution scenarios while ensuring geometric properties, and significantly improves the spatial consistency of the attribution graph.

[0105] Step 203: Construct an optimized cross-modal correlation mapping between the image and the text library using the Transformer-MM framework to generate positive and negative text attribution maps, respectively; the specific steps are as follows:

[0106] Step 20301, transfer the image and the text label Simultaneously, the image features are fed into the CLIP network within the Transformer-MM framework for encoding, yielding the image features. and text features After feature multiplication, the matching score between the image and the text label is obtained:

[0107]

[0108] in, The score represents the matching degree between the image and the corresponding text label. It is a transpose operator;

[0109] Step 20302: Initialize the self-attention correlation mapping matrix of the visual encoder in the CLIP network;

[0110] Since the CLIP network encodes image and text modalities separately, the correlation mapping matrix is ​​initialized as an identity matrix to avoid introducing prior bias. The specific initialization is as follows:

[0111]

[0112] in, This indicates the length of the token in the visual encoder (the image is encoded into multiple tokens by the visual encoder). for The identity matrix, Represents the correlation mapping matrix. Represents the row and column indices of the correlation mapping matrix;

[0113] Step 20303: Iterate through the attention modules of each Transformer network layer of the visual encoder, and calculate the corresponding gradient-weighted attention. For the ... The gradient-weighted attention of the layer is calculated as follows:

[0114]

[0115] in, This represents the matching score between the image and the corresponding text label. Indicates the first The attention score matrix of the layer, To find the gradient sign, Indicates the first Gradient of the attention score matrix of the layer This means that only positive contributions are retained. This indicates element-wise multiplication. Indicates the first All attention heads in the layer The average value, Indicates the first Gradient-weighted attention for layers;

[0116] Step 20304: Aggregate the correlations of each layer along the depth direction of the visual encoder (accumulating layer by layer from the input layer to the output layer), and update the correlation mapping matrix:

[0117]

[0118] in, Indicates the first Gradient-weighted attention of layers, Indicates the first The correlation mapping matrix of the layer, Indicates the first The correlation mapping matrix of the layer;

[0119] Step 20305: Take the relevance row vector corresponding to the [CLS] token (representing the classification token, a meta-label, where [CLS] token is its proper noun) in the relevance mapping matrix of the final output layer, and the column vector corresponding to the image patch obtained after CLIP encoding of the image. Then, normalize the vector using the minimum-maximum normalization algorithm and upsample it to the original image resolution to obtain the attribution map corresponding to the text label. :

[0120]

[0121] in, Represents the upsampling function. This indicates the index corresponding to the [CLS] token. This indicates taking the correlation mapping matrix. All elements in the CLS row, and The input images are respectively Height and width, This is the attribution graph corresponding to the text label;

[0122] That is, when the input text label is a body text label, This is an attribution graph for positive text; when the input text labels are negative text labels, Attribution diagram for negative text;

[0123] Step 204: Merge the positive text attribution map and the negative text attribution map, and enhance the activation regions of the positive text attribution map while suppressing the activation regions of the negative text attribution map. The specific process is as follows:

[0124] Step 20401: Calculate the attribution graphs for the positive text set (target semantics) and the negative text set (interference semantics) respectively.

[0125]

[0126] in, This is an attribution diagram for the main text set. Attribution diagram for negative text sets The number of attribution maps in the main text. The number of negative text attribution maps. Indicates the first A negative text attribution graph Indicates the first A text attribution graph;

[0127] Step 20402: Attribution graphs of the positive and negative text sets are merged using a weighted fusion method.

[0128]

[0129] in, This is the fused attribution map.

[0130] It is worth noting that, in response to the problem that the model has difficulty focusing on key semantic regions in autonomous driving scenarios, the process of building a dynamic text library in step 2 obtains the semantic information that the current driving scenario and spatial planning problem should focus on. At the same time, by constructing a correlation mapping between images and the text library to obtain an attribution map of positive / negative text, the semantic information of the problem text is transformed into a more intuitive region heatmap, providing explicit and attributable visual region guidance for the model to improve its semantic focusing ability.

[0131] Step 3 involves spatially visually encoding the image and fusing it with the attribution map to construct an explicit spatial encoding cue. A geometric feature enhancement cue is then constructed and combined with the explicit spatial encoding cue to generate a structured cue. Referring to Figure 3, this step specifically includes the following steps:

[0132] Step 301: Perform spatial visual encoding on the image and fuse it with the attribution map. The specific steps are as follows:

[0133] First, the input image is divided into grids and encoded. The image is evenly divided into a regular grid containing n×m image blocks (e.g., 4×8, which is divided into 4 rows and 8 columns, forming 32 image blocks). Each image block is assigned a unique spatial code (e.g., the upper left corner is marked with a number from "1" to "32").

[0134] Next, the grid is aligned with the attribution map, and masked average pooling is used for each image block to extract the local response of the corresponding attribution map:

[0135]

[0136] in, This is represented as the fused attribution graph. In pixels The response value; These are respectively represented as the image blocks. The corresponding horizontal and vertical coordinates of the pixel; It is an image block The area; These represent the row and column indices of the image patch within the grid, respectively. Represented as the image block The corresponding relevance score, the distribution of the relevance score can accurately reflect the semantic relevance between the local area of ​​the image and the question text. For example, when the question text contains "car" in the prior, the image patch where the vehicle is located in the grid will show a significantly higher relevance score than the background area.

[0137] Step 302: Construct explicit spatial encoding hints based on step 301;

[0138] Specifically, by representing the gridded visual encoding rules and their fusion process with the attribution map in a text prompt format that LLM can understand and inputting it into the LLM, the problem of MLLMs' biased understanding of the discretization of continuous coordinate systems is effectively solved. The specific prompt design includes the following:

[0139] Role definition prompt (defining the system role as an autonomous driving perception and analysis framework):

[0140] As an autonomous driving perception system, analyze the scene undergrid-based spatial reference:

[0141] In other words, as an autonomous driving perception system, it analyzes the scene under a grid-based spatial reference:

[0142] Mesh partitioning rule prompt (Establish a standard for orthogonal row and column mesh partitioning, using the coordinate system of the top left corner origin in the image patch):

[0143] Image is divided into {n_rows}×{n_cols} grids with uniqueidentifiers.

[0144] Block numbering starts from top-left corner (Block 1 at row 1, column1).

[0145] Sequential numbering follows left-to-right, top-down order.

[0146] That is: (1) The image is divided into {n_rows}×{n_cols} image blocks with unique spatial codes.

[0147] (2) Image block encoding starts from the top left corner (image block 1 in the first row and first column).

[0148] (3) The encoding order follows the order from left to right and from top to bottom.

[0149] Encoding rule prompts (establish linear encoding rules and derive the mathematical mapping relationship between row and column coordinates and image block encoding):

[0150] -Row 1: Blocks 1-{n_cols}.

[0151] -Row 2: Blocks{n_cols+1}-{2*n_cols}.

[0152] That is: Line 1: Image block 1-{n_cols}.

[0153] - Line 2: Image block {n_cols+1}-{2*n_cols}.

[0154] Output examples illustrate the logical correspondence between spatial location and encoding through typical examples:

[0155] Example: Block at row 3, column 2 is numbered {2*n_cols+2}

[0156] For example: the unique spatial code of the image patch in the 3rd row and 2nd column is {2*n_cols+2};

[0157] In other words, when the image is divided into 4*4 image blocks, n=4, and the unique spatial code of the image block in the 2nd row and 4th column is (2-1)*4+4=8;

[0158] This can be understood as the unique spatial code corresponding to each image block = (row number - 1) * total number of columns + column number;

[0159] Step 303: Construct geometric feature enhancement prompts based on step 302;

[0160] Specifically, step 303 is actually a further enhancement of step 302, used to help the LLM better understand the positional relationships between the divided mesh blocks;

[0161] Specifically, by constructing a "grid index-center coordinates" mapping dictionary, spatial quantization information is further introduced to present the spatial location information of each image patch in a numerical manner:

[0162] "Grid coordinates (unit: pixel).Note: Coordinates correspond to blockcenters

[0163] {1: ({x1}, {y1}), 2: ({x2}, {y1}), ...}"

[0164] That is: "Grid coordinates (unit: pixels). Note: The coordinates correspond to the center of the image patch."

[0165] {1: (x1, y1), 2: (x2, y1), ...}"

[0166] In other words, the rows and columns corresponding to the image blocks in the grid are represented by pixels, and the unique spatial code of the image block is defined as the coordinates of the center point of the image block;

[0167] For example: Suppose an image is 800 px wide and 600 px high, and is cut into 16 image blocks of 4×4, each image block being 200 px wide and 150 px high;

[0168] Therefore, the center coordinates of image patch 1 are (100, 75);

[0169] The center coordinates of image patch 2 are (300, 75);...

[0170] The center coordinates of image block 16 are (700, 525);

[0171] The unique spatial code of the image patch → center pixel coordinates is listed as a dictionary, resulting in {1:(100, 75),2:(300, 75),...,16:(700, 525)}.

[0172] Step 304: Combine the explicit spatial coding cue with the geometric feature enhancement cue to generate a structured cue;

[0173] Specifically, the spatial gridded visual encoding in step 301 is used and fused with the attribution map obtained in step 2 to construct an explicit spatial encoding cue. Then, a structured cue is generated by combining the geometric feature enhancement cue. After the structured cue is input into the LLM, a relevant cue is output based on the relevance score of each image patch and the unique spatial encoding, so that the model focuses on the region with a high relevance score.

[0174] Here is an example of outputting relevance prompts:

[0175] "Attention weights (0-1 scale) per block: {1: 0.82, 2: 0.15, ..., 32:0.03}; block 1 should be paid close attention to."

[0176] That is: "Attention weights for each image block (range 0-1): {1: 0.82, 2: 0.15, ..., 32: 0.03}; Block 1 should be closely monitored."

[0177] It is worth noting that, in order to address the problem that the model has difficulty focusing on key semantic regions in autonomous driving scenarios, step 3 performs spatial visual encoding of the image and integrates attribution maps to construct explicit spatial encoding cues. Then, it combines geometric feature enhancement cues to generate structured cues. By providing explicit and attributable visual region guidance to MLLMs, the model's ability to focus semantically in autonomous driving scenarios is further enhanced, thereby improving its spatial planning capabilities.

[0178] Step 4: Combine the input information with the structured prompts to obtain the answer to the spatial planning problem in the multimodal large model.

[0179] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0180] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] The technical features of the above embodiments can be combined arbitrarily. Furthermore, the numbering of each step is not intended to constrain the order of the steps; their order is permissible as long as there are no strict constraints on the sequence. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, any combination of these technical features that does not contradict each other should be considered within the scope of this specification.

[0182] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for enhancing spatial planning in autonomous driving based on semantic region focusing, characterized in that: Includes the following steps: Step 1: Obtain input information, including images of the autonomous driving scenario and text of the spatial planning problem; Step 2: Design a dynamic text library and construct a cross-modal correlation mapping between the text library and the image to generate an attribution map of region activation; Step 3: Perform spatial visual encoding on the image and fuse the attribution map to construct an explicit spatial encoding cue, construct a geometric feature enhancement cue and combine it with the explicit spatial encoding cue to generate a structured cue; Step 4: Input the input information and the structured cue into a multimodal large model to obtain the answer to the spatial planning problem; Step 2 includes the following steps: Step 201, dynamically generating a dynamic text library that is positively and negatively correlated with the spatial planning problem using domain knowledge of the driving scenario; Step 202, optimizing the resolution of the input image through adaptive block segmentation; Step 203, constructing a cross-modal correlation mapping between the optimized image and the text library using the Transformer-MM framework to generate positive and negative text attribution maps respectively; Step 204, fusing the positive and negative text attribution maps; In Step 201, the dynamic generation adopts rule-based prior generation. The method and the open-domain semantic expansion method based on the Large Language Model (LLM) are described below, with the following specific generation steps: Step 20101, using a rule-based prior generation method, the text library is constructed using domain knowledge of the driving scenario. The text library includes a basic object library, a question enhancement library, and a negative sample library. The basic object library covers the core entities of the driving scenario. The question enhancement library contains explicit entities extracted from the spatial planning problem text and implicit entities associated with actions. The negative sample library includes interfering entities that are low in relevance to the spatial planning problem decision. The basic object library and the question enhancement library... Entities in the library are all considered positive text labels, and entities in the negative sample library are all considered negative text labels; Step 20102, using the open-domain semantic expansion method based on the Large Language Model (LLM), a hierarchical progressive hint framework for the spatial planning problem is designed and input into the LLM to dynamically generate positive / negative text labels related to the spatial planning problem, thereby expanding the text library; Step 203 includes the following steps: Step 20301, the image and text labels are simultaneously fed into the CLIP network in the Transformer-MM framework for encoding to obtain image features and text features. After multiplying the image features and text features, the matching score between the image and the text label is obtained; Step 20302, the self-attention relevance mapping matrix of the visual encoder in the CLIP network is initialized; Step 20303, the attention module of each Transformer network layer of the visual encoder is traversed layer by layer, and the gradient weighted attention of the corresponding layer is calculated; Step 20304, the relevance of each layer is aggregated along the depth direction of the visual encoder and the relevance mapping matrix is ​​updated; Step 20305, the relevance row vector corresponding to the [CLS] token in the relevance mapping matrix of the final output layer, and the column vector corresponding to the image patch obtained after CLIP encoding of the image are taken, and then normalized by the minimum-maximum normalization algorithm and upsampled to the original image resolution to obtain the positive or negative text attribution map corresponding to the text label.

2. The method for enhancing autonomous driving spatial planning based on semantic region focusing according to claim 1, characterized in that: Step 3 includes the following steps: Step 301, spatially visually encoding the image and fusing the attribution map; Step 302, constructing explicit spatial encoding cues; Step 303, constructing geometric feature enhancement cues; Step 304, combining the explicit spatial encoding cues and the geometric feature enhancement cues to generate structured cues.

3. The method for enhancing spatial planning for autonomous driving based on semantic region focusing according to claim 1, characterized in that: Step 204 includes the following steps: Step 20401, calculating the attribution graphs for the positive text set and the negative text set respectively: ;in, This is an attribution diagram for the main text set. Attribution diagram for negative text sets The number of attribution maps in the main text. The number of negative text attribution maps. Indicates the first A negative text attribution graph Indicates the first Step 20402: Merge the attribution graphs of the positive and negative text sets using a weighted fusion method. ;in, This is the fused attribution map.

4. The method for enhancing autonomous driving spatial planning based on semantic region focusing according to claim 2, characterized in that: Step 301, The process includes the following steps: First, the input image is divided into grids and encoded. The image is uniformly divided into a regular grid containing several image blocks, and each image block is assigned a unique spatial code. Next, the grid is aligned with the attribution graph, and masked average pooling is used to extract the local response of the attribution graph for each image block to obtain the relevance score corresponding to each image block.

5. The method for enhancing spatial planning for autonomous driving based on semantic region focusing according to claim 4, characterized in that: Step 302 includes the following steps: First, role definition prompt: define the system role as an autonomous driving perception analysis framework; second, grid partitioning rule prompt: establish a row and column orthogonal grid partitioning standard and adopt the coordinate system of the origin at the top left corner of the image block; then, encoding rule prompt: formulate linear encoding rules and derive the mathematical mapping relationship between row and column coordinates and image block encoding.

Images