Data processing method, device, medium and program product based on map information

Abstract

Embodiments of the present application provide a data processing method and device based on map information, a medium and a program product. The method comprises: acquiring multi-modal data and map slice information corresponding to a self-vehicle pose; the multi-modal data comprises image data and point cloud data; performing feature extraction on the multi-modal data, deeply fusing obtained image features and point cloud features to obtain first perception features; taking the map slice information as geometric prior, generating four-dimensional grid occupancy information aligned with the map information based on the first perception features under the constraint of the map slice information. To some extent, the difficulty and cost of data alignment are reduced, and the accuracy and reliability of the perception results obtained from the above data are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, medium and program product based on map information. Background Technology

[0002] As autonomous driving shifts from rule-driven to data-driven and even model-driven approaches, it no longer relies on developers writing code logic. Instead, the driving system learns how to drive the vehicle through massive amounts of driving data. Therefore, it is necessary to collect massive amounts of roadside data to enable the autonomous driving system to learn how to drive the vehicle.

[0003] When teaching autonomous driving systems how to drive, massive amounts of road data need to be processed. In the processing of this data, inconsistencies in the coordinate spaces of different data sources make data alignment difficult and result in low accuracy in perception based on this data. Summary of the Invention

[0004] This application provides a data processing method, electronic device, storage medium, and program product based on map information. Under a unified map information coordinate space reference, it achieves pixel-level or voxel-level alignment of road data collected from different sources, which can reduce the difficulty of data alignment and help improve the accuracy of perception analysis based on this data.

[0005] In a first aspect, embodiments of this application provide a data processing method based on map information, comprising: acquiring multimodal data and map tile information corresponding to vehicle pose; the multimodal data including image data and point cloud data; performing feature extraction on the multimodal data, and performing deep fusion of the obtained image features and point cloud features to obtain a first perceptual feature; using the map tile information as a geometric prior, and generating four-dimensional grid occupancy information aligned with the map information based on the first perceptual feature under the constraint of the map tile information.

[0006] In one possible implementation, using the map tile information as a geometric prior, and under the constraint of the map tile information, generating four-dimensional grid occupancy information aligned with the map information based on the first perceptual feature, includes: encoding the map tile information to generate map features; modulating the first perceptual feature using the map features; and generating the four-dimensional grid occupancy information based on the modulated first perceptual feature.

[0007] In one possible implementation, modulating the first sensing feature using the map features includes: for each location in the first sensing feature, determining the map vector element closest to the location; generating transformation parameters based on the local map features corresponding to the map vector element and the second sensing feature corresponding to the location; and applying the transformation parameters to the second sensing feature to obtain the modulation result of the second sensing feature.

[0008] Alternatively, modulating the first perceptual feature using the map features includes: modulating the first perceptual feature input to at least one layer of the decoding network in the decoder using the map features; the decoder belongs to a preset model; and generating the four-dimensional grid occupancy information based on the modulated first perceptual feature includes: inputting the modulated first perceptual feature to the occupancy prediction head of the decoder, and outputting the four-dimensional grid occupancy information through the occupancy prediction head.

[0009] In one possible implementation, the decoder is trained based on a first loss function, wherein the first loss function comprises one or more of the following sub-loss functions: occupancy-map geometric consistency loss, semantic-occupancy projection consistency loss, and vector-direction consistency loss; wherein the occupancy-map geometric consistency loss is used to compare the geometric differences between the predicted four-dimensional grid occupancy information and the static occupancy grid indicated by the map tile information; the semantic-occupancy projection consistency loss is used to constrain the projection of the occupancy voxels classified as drivable areas onto the horizontal plane to be within the drivable area boundary defined by the map tile information; and the vector-direction consistency loss is used to constrain the spatial orientation of the predicted lane line voxel sequence to be consistent with the direction vector of the corresponding lane line in the map tile information.

[0010] In one possible implementation, the step of extracting features from the multimodal data and deeply fusing the obtained image features and point cloud features to obtain a first perceptual feature includes: projecting the image features and point cloud features onto a top view plane; and fusing the image features and point cloud features projected onto the top view plane to obtain the first perceptual feature.

[0011] Alternatively, the method further includes encoding the four-dimensional grid occupancy information and the image features using a multimodal visual encoder based on a preset model to obtain a first vector; mapping the first vector to a language representation space to obtain a second vector; and inputting the second vector and map features enhanced by map knowledge into the backbone network of the preset model, with the backbone network outputting multidimensional scene description information.

[0012] In one possible implementation, the multimodal visual encoder based on a preset model encodes the four-dimensional grid occupancy information and the image features to obtain a first vector, including: dividing the image features into blocks to obtain a first block result; generating a first block vector based on the pixel values ​​of the first block result; dividing the four-dimensional grid occupancy information into blocks to obtain a second block result; generating a second block vector based on the height value and channel value of the second block result; concatenating the first block vectors of multiple first block results and the second block vectors of multiple second block results, and encoding the concatenated result to obtain the first vector;

[0013] Alternatively, it may also include: querying a map prior knowledge base based on the map features to obtain at least one corresponding map prior knowledge vector; associating the at least one map prior knowledge vector with the map features to obtain the map features enhanced by the map knowledge.

[0014] Alternatively, it may also include: performing cross-validation based on the multi-dimensional scene description information and the map tile information; and determining the candidate difficult cases corresponding to the multimodal data in response to the cross-validation result meeting preset conditions.

[0015] In one possible implementation, the cross-validation based on the multi-dimensional scene description information and the map tile information includes one or more of the following: comparing the perceptual information of physical entities in the multi-dimensional scene description information with the map tile information; comparing the perceptual information of physical entities with the scene semantic description text in the multi-dimensional scene description information; comparing the scene semantic description text with the map tile information; and comparing the trajectory planning information in the multi-dimensional scene description information with the prediction information in the four-dimensional grid occupancy information.

[0016] Alternatively, it may also include: in response to determining the candidate difficult cases corresponding to the multimodal data, retrieving associated historical difficult cases from the map memory bank based on the map tile information; writing the information of the associated historical difficult cases, the cross-validation information of the candidate difficult cases, and the thought chain used to guide the model to perform interpretable logical reasoning into a prompt template to obtain prompt information; the information of the associated historical difficult cases includes corresponding coping strategies; inputting at least one of the second vector, the four-dimensional grid occupancy information, and the text instruction, along with the prompt information, into the preset model, and having the preset model output judgment information.

[0017] In one possible implementation, the method further includes: in response to the determination information indicating that the multimodal data corresponds to a difficult example, constructing difficult example causal data based on the interpretable inference chain and type corresponding to the difficult example; generating extended difficult example data based on the difficult example causal data using a generative model; wherein the extended difficult examples and the difficult examples are used to incrementally train the preset model.

[0018] In one possible implementation, constructing difficult example causal data based on the interpretable inference chain and type corresponding to the difficult example includes: extracting causal elements from the interpretable inference chain and the type, constructing causal tuples based on the causal elements; converting the causal tuples into natural language descriptions to obtain the difficult example causal data; generating extended difficult example data using a generative model based on the difficult example causal data includes: encoding the difficult example causal data to obtain a condition vector; inputting the condition vector into the generative model, and having the generative model generate the extended difficult example data based on the condition vector.

[0019] Secondly, embodiments of this application provide a data processing method based on map information, comprising: acquiring four-dimensional grid occupancy information, image features, and map tile information; encoding the four-dimensional grid occupancy information and image features to obtain a first vector; mapping the first vector to a language representation space to obtain a second vector; inputting the second vector and map features enhanced based on map knowledge into the backbone network of a preset model, and having the backbone network output multi-dimensional scene description information; performing cross-validation based on the multi-dimensional scene description information and the map tile information; and determining the scene corresponding to the multimodal data as a candidate difficult example based on the cross-validation result; wherein, the four-dimensional grid occupancy information is generated based on a first perceptual feature under the constraint of the map tile information, using the map tile information as a geometric prior; the first perceptual feature is obtained by deep fusion of the image features and point cloud features; and the image features and point cloud features are obtained by feature extraction from the multimodal data collected by a vehicle.

[0020] Thirdly, embodiments of this application provide a data processing method based on map information, comprising: in response to determining candidate difficult cases corresponding to multimodal data collected by a vehicle, retrieving associated historical difficult cases from a map memory bank based on map tile information; the candidate difficult cases are determined by cross-validation based on multidimensional scene description information and the map tile information; the multidimensional scene description information is determined based on a second vector and map features enhanced by map knowledge; writing the information of the associated historical difficult cases, the cross-validation information, and a thought chain for guiding the model to perform interpretable logical reasoning into a prompt template to obtain prompt information; the information of the associated historical difficult cases includes corresponding coping strategies; the prompt information also includes a thought chain; and the second vector, four-dimensional grid occupancy information, and the vehicle-to-vehicle pair are used to determine the candidate difficult cases. At least one of the corresponding text instructions, along with the prompt information, is input into a preset model, and the preset model outputs the judgment information of the candidate difficult example; the judgment information includes one or more of the following: difficult example judgment result, difficult example type, text description of an interpretable reasoning chain, and coping strategy; wherein, the second vector is obtained by mapping the first vector to a language representation space, the first vector is obtained by encoding four-dimensional grid occupancy information and image features, the four-dimensional grid occupancy information is generated based on a first perceptual feature under the constraint of map tile information, using map tile information as a geometric prior; the first perceptual feature is obtained by deep fusion of the image features and point cloud features; the image features and point cloud features are obtained by feature extraction from the multimodal data.

[0021] Fourthly, embodiments of this application provide a data processing apparatus based on map information, comprising: a first acquisition unit, configured to acquire multimodal data and map tile information corresponding to a vehicle's pose; the multimodal data includes image data and point cloud data; a first feature extraction unit, configured to extract features from the multimodal data and deeply fuse the obtained image features and point cloud features to obtain a first perceptual feature; and a first generation unit, configured to use the map tile information as a geometric prior and, under the constraint of the map tile information, generate four-dimensional grid occupancy information aligned with the map information based on the first perceptual feature.

[0022] In one possible implementation, the first generation unit is further configured to: encode map tile information to generate map features; modulate the first sensing features using the map features; and generate four-dimensional grid occupancy information based on the modulated first sensing features.

[0023] In one possible implementation, the first generation unit is further configured to: for each location in the first perceptual feature, determine the map vector element closest to the location; generate transformation parameters based on the local map features corresponding to the map vector element and the second perceptual feature corresponding to the location; apply the transformation parameters to the second perceptual feature to obtain the modulation result of the second perceptual feature; or, further configured to: modulate the first perceptual feature input to at least one layer of the decoding network in the decoder using map features; the decoder belongs to a preset model; and, based on the modulated first perceptual feature, generate four-dimensional grid occupancy information, including: inputting the modulated first perceptual feature to the occupancy prediction head of the decoder, and outputting four-dimensional grid occupancy information through the occupancy prediction head.

[0024] In one possible implementation, the decoder is trained based on a first loss function, which comprises one or more of the following sub-loss functions: occupancy-map geometric consistency loss, semantic-occupancy projection consistency loss, and vector-direction consistency loss; wherein the occupancy-map geometric consistency loss is used to compare the geometric differences between the predicted four-dimensional grid occupancy information and the static occupancy grid indicated by the map tile information; the semantic-occupancy projection consistency loss is used to constrain the projection of the occupancy voxels classified as drivable areas onto the horizontal plane to lie within the drivable area boundary defined by the map tile information; and the vector-direction consistency loss is used to constrain the spatial orientation of the predicted lane line voxel sequence to be consistent with the direction vector of the corresponding lane line in the map tile information.

[0025] In one possible implementation, the first feature extraction unit is further configured to: project image features and point cloud features onto a top-view plane; fuse the image features and point cloud features projected onto the top-view plane to obtain a first perceptual feature; or, the device further includes a second generation unit, which is configured to: encode four-dimensional grid occupancy information and image features based on a multimodal visual encoder of a preset model to obtain a first vector; map the first vector to a language representation space to obtain a second vector; input the second vector and map features enhanced based on map knowledge into the backbone network of the preset model, and output multi-dimensional scene description information from the backbone network.

[0026] In one possible implementation, the second generating unit is further configured to:

[0027] The image features are divided into blocks to obtain a first block result; a first block vector is generated based on the pixel values ​​of the first block result; the four-dimensional raster occupancy information is divided into blocks to obtain a second block result; a second block vector is generated based on the height and channel values ​​of the second block result; the first block vectors of multiple first block results and the second block vectors of multiple second block results are concatenated, and the concatenated result is encoded to obtain a first vector; or, further used for: querying a map prior knowledge base based on the map features to obtain at least one corresponding map prior knowledge vector; associating at least one map prior knowledge vector with the map features to obtain map knowledge-enhanced map features;

[0028] Alternatively, the device may also include a verification unit, which is used to: perform cross-verification based on multi-dimensional scene description information and map tile information; and determine candidate difficult cases corresponding to multimodal data in response to the cross-verification result meeting preset conditions.

[0029] In one possible implementation, the verification unit is further configured to perform one or more of the following: perform a consistency comparison between the perceived information of physical entities in the multi-dimensional scene description information and the map information; perform a consistency comparison between the perceived information of physical entities and the scene semantic description text in the multi-dimensional scene description information.

[0030] The device performs a consistency comparison between the scene semantic description text and map tile information; it also performs a consistency comparison between the trajectory planning information in the multi-dimensional scene description information and the prediction information in the four-dimensional grid occupancy information; or, the device further includes a secondary judgment unit, which is used to: in response to determining the candidate difficult cases corresponding to the multimodal data, retrieve associated historical difficult cases from the map memory bank according to the map tile information; write the information of associated historical difficult cases, the cross-validation information of candidate difficult cases, and the thought chain used to guide the model to perform interpretable logical reasoning into a prompt template to obtain prompt information; the information of associated historical difficult cases includes the corresponding coping strategies; and input at least one of the second vector, the four-dimensional grid occupancy information, and the text instruction, along with the prompt information, into a preset model, and the preset model outputs judgment information.

[0031] In one possible implementation, the apparatus further includes a second generation unit, which is configured to: in response to a determination information indicating a difficult example corresponding to multimodal data, construct difficult example causal data based on the interpretable inference chain and type corresponding to the difficult example; and generate extended difficult example data based on the difficult example causal data using a generative model; wherein the extended difficult examples and the difficult examples are used for incremental training of a preset model.

[0032] In one possible implementation, the second generation unit is further configured to: extract causal elements from interpretable reasoning chains and types, construct causal tuples based on the causal elements; convert the causal tuples into natural language descriptions to obtain difficult example causal data; and generate extended difficult example data based on the difficult example causal data using a generative model, including: encoding the difficult example causal data to obtain a condition vector; and inputting the condition vector into the generative model, which then generates extended difficult example data based on the condition vector.

[0033] Fifthly, embodiments of this application provide a data processing device based on map information, comprising: a second acquisition unit, configured to acquire four-dimensional grid occupancy information, image features, and map tile information, and encode the four-dimensional grid occupancy information and image features to obtain a first vector; a mapping unit, configured to map the first vector to a language representation space to obtain a second vector; a third generation unit, configured to input the second vector and map features enhanced based on map knowledge into the backbone network of a preset model, and output multi-dimensional scene description information from the backbone network; a cross-validation unit, configured to perform cross-validation based on the multi-dimensional scene description information and map tile information; and a determination unit, configured to determine the scene corresponding to the multimodal data as a candidate difficult example based on the cross-validation result; wherein, the four-dimensional grid occupancy information is generated based on a first perceptual feature under the constraint of map tile information, using map tile information as a geometric prior; the first perceptual feature is obtained by deep fusion of the image features and point cloud features; and the image features and point cloud features are obtained by feature extraction from multimodal data collected by a vehicle.

[0034] Sixthly, embodiments of this application provide a data processing apparatus based on map information, comprising: a retrieval unit, configured to, in response to determining candidate difficult examples corresponding to multimodal data collected by a vehicle, retrieve associated historical difficult examples from a map memory bank based on map tile information; the candidate difficult examples are determined based on cross-validation of multidimensional scene description information and map tile information; the multidimensional scene description information is determined based on a second vector and map features enhanced by map knowledge; and a writing unit, configured to write the information of the associated historical difficult examples, the cross-validation information, and a thought chain for guiding the model to perform interpretable logical reasoning into a prompt template to obtain prompt information; the information of the associated historical difficult examples includes corresponding coping strategies; the prompt information also includes thought chains for interpretable logical reasoning. A decision unit is configured to input at least one of the second vector, four-dimensional grid occupancy information, and the text instruction corresponding to the vehicle, along with the prompt information, into a preset model, and output the decision information of the candidate difficult case from the preset model; wherein, the second vector is obtained by mapping the first vector to the language representation space, the first vector is obtained by encoding the four-dimensional grid occupancy information and image features, the four-dimensional grid occupancy information is generated based on the first perceptual feature under the constraint of the map tile information, using map tile information as a geometric prior; the first perceptual feature is obtained by deep fusion of the image features and point cloud features; the image features and point cloud features are obtained by feature extraction of the multimodal data.

[0035] In a seventh aspect, embodiments of this application provide an electronic device / computer-readable storage medium / computer program product, comprising: a memory, a processor; the memory storing computer-executable instructions; the processor executing the computer-executable instructions stored in the memory, causing the processor to perform the first aspect, second aspect, third aspect and / or various possible implementations of the first aspect, second aspect and third aspect as described above; the computer-readable storage medium storing computer-executable instructions, which, when executed by the processor, are used to implement the first aspect, second aspect, third aspect and / or various possible implementations of the first aspect, second aspect and third aspect as described above; the computer program product comprising a computer program, which, when executed by the processor, implements the first aspect, second aspect, third aspect and / or various possible implementations of the first aspect, second aspect and third aspect as described above.

[0036] The map-based data processing method, electronic device, storage medium, and program product provided in this application, through a decoding process that uses map tile information as the first perceptual feature to constrain geometric priors, naturally aligns the generated four-dimensional raster occupancy information with the map coordinate system. This allows for the acquisition of the world coordinates and map association identifiers for each voxel without post-processing coordinate transformation. Simultaneously, by deeply fusing image features and point cloud features, complementary information between texture semantics and geometric structure is preserved, improving the accuracy and robustness of occupancy prediction. Therefore, this scheme can construct a spatially consistent and semantically rich four-dimensional environment representation within a unified framework, providing a directly quantifiable benchmark for subsequent cross-modal verification and hard example mining. This reduces the difficulty and cost of data alignment to a certain extent and improves the accuracy and reliability of the perceptual results obtained from the aforementioned data. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] Figure 1 A flowchart illustrating the map-constrained perception data processing method provided in this application. Figure 1 ;

[0039] Figure 2 This is a schematic diagram of a perception data processing flow based on map information.

[0040] Figure 3 This is a schematic diagram of another perception data processing flow based on map information;

[0041] Figure 4 This is a schematic diagram of the map information-based data processing method provided in this disclosure. Figure 2 ;

[0042] Figure 5 A schematic diagram of a process for generating multi-dimensional scene description information;

[0043] Figure 6 A flowchart illustrating the map information-based data processing method provided in this disclosure. Figure 3 ;

[0044] Figure 7 This is a flowchart illustrating the cross-validation process;

[0045] Figure 8 A flowchart illustrating the map information-based data processing method provided in this disclosure. Figure 4 ;

[0046] Figure 9 A flowchart illustrating the process of determining a candidate difficult case;

[0047] Figure 10 A schematic diagram of the process for generating an extended hard case;

[0048] Figure 11 A flowchart illustrating the process of difficult example discovery and incremental model training;

[0049] Figure 12 This is a schematic flowchart of a data processing method based on map information provided in an embodiment of this application;

[0050] Figure 13 This is a schematic flowchart of a data processing method based on map information provided in an embodiment of this application;

[0051] Figure 14 A schematic diagram of a map-based data processing device provided in this application;

[0052] Figure 15 A schematic diagram of a map-based data processing device provided in this application;

[0053] Figure 16 A schematic diagram of a map-based data processing device provided in this application;

[0054] Figure 17 A schematic diagram of the structure of the electronic device provided in this application.

[0055] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0057] First, let me explain the terms used in this application:

[0058] Multimodal data refers to data from different vehicle-mounted sensors or different vehicle-mounted information sources, which differ in physical properties, acquisition methods, or representation formats. For example, multimodal data may include, but is not limited to: two-dimensional image sequences acquired by image acquisition devices, three-dimensional spatial point clouds acquired by lidar, and reflected signals acquired by millimeter-wave radar. The types and quantities of multimodal data can be adjusted according to the actual configuration.

[0059] Map vector data refers to structured information about the road traffic environment stored in vector format. It describes static prior knowledge such as road geometry, topological relationships, and traffic rules. Map vector data typically contains multiple logical layers, such as: lane center lines, lane boundary lines (solid lines, dashed lines, double yellow lines, etc.), curbs, stop lines, pedestrian crossings, and the precise locations and attributes of traffic signs (speed limits, turn arrows, traffic light color sequences, etc.). This data has clearly defined geometric coordinates in a predefined coordinate system (e.g., an absolute coordinate system).

[0060] Map tile information is map vector data obtained from a high-precision map within a preset range, centered on the vehicle, based on the vehicle's pose.

[0061] Four-dimensional grid occupancy refers to a structured data representation formed by discretizing three-dimensional space (composed of length, width, and height dimensions) and time dimension. In this representation, each set of spatial coordinates (X, Y, Z) corresponds to a volume unit (called a voxel), and each voxel records whether the corresponding physical location is occupied by an object at a specific time (e.g., occupancy probability). The introduction of the time dimension allows this representation to describe the dynamic changes of the scene over time, thus forming a four-dimensional data structure of "three-dimensional space + time". Each voxel can also carry additional information such as semantic category, motion vector, and map association markers for fine-grained perception and modeling of the environment.

[0062] In the process of learning how to drive using road-collected data for autonomous driving systems, it is required to effectively associate and retrieve data from different sources and modalities (such as image point clouds, maps, and text descriptions). Some related technologies first model dynamic scenes based on the target vehicle's trajectory and then use a non-linear classifier to identify dynamic scene labels such as "vehicle entry" and "vehicle exit." Subsequently, object detection algorithms are used to identify static objects in the images (such as special vehicles and traffic cones) to supplement the static scene labels. However, these approaches process dynamic trajectories and static images separately, without unifying them into a fixed physical coordinate system for fusion analysis. Dynamic trajectory modeling is based on the relative distance to the vehicle, while static object detection is based on image pixels; these two approaches are disconnected, making spatiotemporal consistency verification difficult. Furthermore, the accuracy of analysis results obtained from this data is relatively low.

[0063] In some related technologies, offline videos are preprocessed to generate frame-level understanding descriptions, video summary descriptions, and multi-dimensional standardized video tags, and an offline feature library is constructed. During retrieval, cross-modal matching is performed using Euclidean distance, combining the spatiotemporal parameters and semantic content of the user query, and supporting dynamic knowledge-enhanced question answering and interactive enhanced analysis based on retrieval enhancement. This scheme only extracts tags (such as "intersection" and "sunny day") at the image semantic level, but it does not align with map information and lacks geometric constraints in a three-dimensional world coordinate system. Therefore, it cannot spatially match or verify the perceived results with precise locations on the map (such as lane lines and roadside coordinates).

[0064] Some related technologies utilize large models to perform multi-round question-and-answer sessions on scene images, extracting rich image features and semantic descriptions to construct an "image-text" multimodal matching model, enabling cross-modal retrieval by searching images based on text or images. However, these technologies primarily focus on image-text alignment, completely ignoring the crucial information source of high-precision maps.

[0065] In summary, the processing of road-collected data in the field of autonomous driving faces the following problems: multimodal data lacks unified spatiotemporal alignment with the physical world. That is, the processing and analysis of perception (four-dimensional grid occupancy, detection), mapping (lane lines, topology), and language description data are often performed between their respective independent coordinate spaces or feature spaces. For example, perception results are typically expressed in the vehicle's coordinate system, map information exists in an absolute geographic coordinate system, while language descriptions are completely detached from physical measurements. When attempting cross-modal verification, such as determining whether a vehicle traveling in the wrong direction as described in the language description is actually on a one-way street marked on the map, this analysis becomes extremely difficult or even impossible due to the lack of coordinate space consistency. Even if analysis is possible, the accuracy of the results is low.

[0066] To address the aforementioned issues, the solution provided in this application uses high-precision maps as the absolute anchor point in the physical world. Under a unified map information coordinate space benchmark, pixel-level or voxel-level alignment of road data collected from different sources is achieved, which can reduce the difficulty of data analysis and help improve the accuracy of data analysis results.

[0067] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0068] Figure 1 A flowchart illustrating the map-constrained perception data processing method provided in this application. Figure 1 ,like Figure 1 As shown, the method includes:

[0069] S101. Obtain multimodal data and map tile information corresponding to the vehicle's pose; multimodal data includes image data and point cloud data.

[0070] The entity executing the map-based perception data processing method can be a data processing device (such as a computing platform or cloud server deployed on a vehicle with autonomous driving capabilities).

[0071] Image acquisition devices and LiDAR devices can be installed on the vehicle body (e.g., vehicles equipped with autonomous driving functions). These image acquisition devices may include, for example, 6-channel and 8-channel ultra-wide-angle cameras mounted around the vehicle, covering a 360-degree horizontal field of view. Multiple ultra-wide-angle cameras can be controlled to simultaneously acquire image data. In some examples, the resolution of the multiple ultra-wide-angle cameras can be 1920×1080 pixels, and the frame rate can be 30 Hz. To balance computational load and information density, the image data can be downsampled to 1280×768 pixels using bilinear interpolation before mapping to a four-dimensional grid.

[0072] The aforementioned point cloud acquisition device can be one or more LiDARs mounted on the top of the vehicle, such as a 360-degree mechanically rotating LiDAR or an equivalent 360-degree perception system composed of multiple solid-state LiDARs. This device can acquire multiple frames of point cloud data, with each frame containing approximately 150,000 to 300,000 points, at a rate of 10 frames per second (10Hz). Each point contains three-dimensional coordinates (x, y, z), reflection intensity, and a hardware timestamp. When the vehicle is traveling at high speed, the vehicle's position changes within one frame scan time (approximately 100ms). Therefore, before mapping the data to a four-dimensional grid, motion compensation of the point cloud can be performed using Inertial Measurement Unit (IMU) and wheel speed sensor data, projecting all point clouds onto the vehicle's coordinate system corresponding to the center moment of that frame scan. This process can be implemented using linear interpolation or IMU-based continuous-time trajectory estimation.

[0073] The data processing equipment can also calculate the vehicle's current pose (position and orientation) in a preset coordinate system (such as a map coordinate system) in real time by fusing information from the Global Positioning System (GPS), Inertial Measurement Unit (INS), and wheel speedometer. Based on this pose, the data processing equipment can request map tile information centered on the vehicle and within a 150-meter radius from a high-precision map server. The map tile information can include multi-layered, highly structured vector data packets, including but not limited to the following logical layers: road geometry layer, traffic facility layer, and semantic attribute layer. The road geometry layer includes the 3D spline curve parameters of lane center lines and lane boundary lines (curb, guardrail), as well as the road topology connections defined by nodes and edges. Nodes represent specific coordinate points in the road network, and edges connect nodes and define the geometry and topology of road segments. The traffic facility layer can include, for example, precise 3D bounding boxes containing traffic signs and traffic lights and their orientations, as well as polygonal areas for stop lines, pedestrian crossings, etc. The semantic attribute layer may include, for example, logical attributes such as the type of each lane (e.g., regular motor vehicle lane, bus lane, emergency lane), direction (one-way / two-way), speed limit, and turning arrows (straight, left turn, right turn, etc.).

[0074] In some examples, the multimodal data described above also includes sequences of text terms derived from natural language instructions. These natural language instructions include navigation instructions and / or driver-generated language instructions.

[0075] S102: Extract features from multimodal data, and deeply fuse the obtained image features and point cloud features to obtain the first perceptual features.

[0076] The data processing device extracts features from the multimodal data acquired in step S101, and then deeply fuses these features into a unified first perception feature.

[0077] In some embodiments, step S102 includes:

[0078] Features are extracted from image data and point cloud data respectively to obtain image features and point cloud features;

[0079] Project image features and point cloud features onto the top view plane;

[0080] The image features and point cloud features projected onto the top view plane are fused to obtain the first perceptual features.

[0081] In one example, feature extraction from the image data includes encoding the panoramic image data to obtain image features. Specifically, each image path is input into a feature extraction network based on a convolutional neural network or a VisionTransformer (ViT) to obtain multi-scale image feature maps. For example, a moving window self-attention mechanism can be used to efficiently capture local details and establish long-range global dependencies. The six images are each independently processed by the backbone network, outputting multi-scale feature maps. We select the output feature maps {Fimg2, Fimg3} from the second and third stages, with resolutions of 1 / 16 and 1 / 32 of the original image, respectively, to form the image feature pyramid.

[0082] For point cloud data, the data processing device can first remove distortion, and then perform voxel rasterization on the distorted point cloud data in a predefined 3D space (e.g., voxel size 0.2m × 0.2m × 0.4m) to obtain several voxel units. Within each non-empty voxel unit, all point cloud features contained therein are aggregated (e.g., the mean of all point cloud features is calculated) to obtain the initial feature vector (sparse voxel features) of that voxel unit. Then, a sparse submanifold convolutional layer is used to perform convolution operations on the voxel unit and its neighboring voxel units. The characteristic of this convolutional layer is that it only calculates voxel units that contain point cloud data and their neighboring voxel units that also contain data, and does not calculate empty voxel units. Furthermore, during the operation, the output position of empty voxel units remains empty, thus maintaining the sparse mode of the voxel grid. The above sparse submanifold convolutional layers are stacked continuously for four layers, with the output of the previous layer as the input for each layer, progressively expanding the receptive field of each voxel unit and extracting higher-level local geometric features. After being encoded by four layers of stacked 3D sparse sub-manifestation convolutions, each non-empty voxel unit obtains a high-dimensional feature vector. The feature vectors of all non-empty voxel units and their spatial indices together constitute the point cloud voxel feature map F_pts_voxel.

[0083] Next, the data processing device can project these features from different modalities onto the same two-dimensional top-view (BEV) plane to obtain image BEV features. For example, a depth-based projection method (such as the Lift-Splat-Shoot (LSS) paradigm) can be used to project each pixel in the image features onto the BEV plane. Specifically, for each pixel location (u,v) in each image feature, a discrete point depth distribution P∈R^D is predicted, where P represents a vector representing the depth probability distribution at that pixel location; D is the number of depth intervals (e.g., 1 meter to 60 meters, with each interval being 1 meter) (D is an integer greater than or equal to 1). Then, the semantic feature c of the pixel is outer-producted with the depth distribution P to obtain a three-dimensional feature point cloud {c*Pi|i=1,…,D} within a frustum. Finally, these three-dimensional feature point clouds are mapped onto the BEV plane according to the camera's intrinsic and extrinsic parameters.

[0084] Simultaneously, the height dimension of the point cloud voxel feature map F_pts_voxel is flattened, and the BEV feature from the point cloud perspective is obtained through a simple concatenation or summation operation. Specifically, for all height layer voxel features at each horizontal position, they are aggregated according to preset rules (e.g., concatenating the feature vectors of each height layer, or adding corresponding elements of the feature vectors of each height layer) to obtain a comprehensive feature vector representing that horizontal position. Through this processing, the point cloud voxel features, which originally had a three-dimensional spatial structure, are transformed into a two-dimensional feature map that retains only horizontal spatial information; this feature map is the top-down view feature from the point cloud perspective. This flattening operation can reduce the computational dimensionality of subsequent processing while preserving key geometric information.

[0085] Then, the image BEV features and point cloud BEV features are concatenated along the channel dimension. The data processing device then passes the concatenated image BEV features and point cloud BEV features through an adaptive gated fusion network (e.g., a lightweight convolutional module that learns a fusion weight for each BEV location). This network learns a fusion weight for features from different sources and spatial locations. Finally, through element-wise multiplication and summation, a unified BEV feature map (F{bev}) with complementary information is obtained as the first perceptual feature. The gated fusion mechanism enables the network to autonomously learn how to optimally combine multimodal information in different environments (e.g., relying on vision in good lighting, or relying on lasers at night / in rain / fog). The first perceptual feature is an intermediate representation that can characterize the current scene environment information, obtained through feature extraction and deep fusion of multimodal data. This representation can be in the form of a two-dimensional bird's-eye view (BEV) feature map, i.e., a grid-like data structure with spatial height and width dimensions, where each grid cell corresponds to a horizontal region in the real world and contains a high-dimensional feature vector. This feature vector integrates information from different modalities (such as image texture features and point cloud geometric features), providing a foundation for subsequent processing. The spatial size of the first perceptual feature is X×Y (e.g., 200×200 grids), with each grid corresponding to a 0.2m×0.2m region in the real world, and each grid location contains a multi-dimensional feature vector (e.g., 256-dimensional).

[0086] In the above implementation, by extracting features from each modality of data separately, projecting these features onto the same top-view plane, and fusing them on this plane to obtain the first perceptual feature, efficient integration of multi-source heterogeneous data under a unified spatial reference is achieved. This process enables image texture information and point cloud geometric information, which were originally difficult to correlate directly in the original coordinate system, to be aligned and combined according to their actual spatial positions from a top-view perspective, thereby avoiding post-registration errors caused by inconsistencies in coordinate systems. Compared to processing each modality's original space separately and then performing result-level fusion, spatial alignment and information complementarity are achieved at the feature level. The generated first perceptual feature retains the image's rich descriptive ability of object categories and boundaries, while also incorporating the point cloud's precise measurement information of distance and 3D shape. This improves the robustness of the perceptual feature to complex environments (such as changes in illumination, missing textures, or occlusion) to a certain extent, and provides a richer and more spatially consistent input foundation for subsequent tasks based on this feature, such as obstacle detection, map alignment, or occupancy prediction.

[0087] S103: Using map tile information as a geometric prior, and under the constraint of map tile information, generate four-dimensional raster occupancy information aligned with map information based on the first perceptual features.

[0088] High-precision map information cannot be simply stitched together as another input channel, but must be used as a strong geometric prior to deeply constrain the training and inference of the four-dimensional grid occupancy network from multiple levels.

[0089] The data processing device takes the first perceptual feature obtained in step S102 and the map tile information obtained in step S101 as input, performs a decoding process constrained by the map tile information, and finally outputs four-dimensional grid occupancy information aligned with the map information. In the four-dimensional grid occupancy information, each voxel grid corresponds to a five-dimensional tensor: T×X×Y×Z×C, where: T is the number of time frames (e.g., the past 5 frames, the current 1 frame, and the future 10 frames); X and Y represent the size of the spatial horizontal grid; Z represents the number of voxel layers in the vertical direction (e.g., 10 layers, covering a range of 0 to 4 meters above the ground); C represents the information channel carried by each voxel, which may include, for example, occupancy probability, semantic label, three-dimensional motion flow vector, precise world coordinates, and map association identifier.

[0090] In some implementations, each voxel in the four-dimensional grid occupancy information includes one or more of the following information:

[0091] The occupancy probability of a voxel; the semantic category identifier of a voxel; the 3D motion flow vector of a voxel; the 3D spatial coordinates of a voxel grid; and the map element association identifier of a voxel.

[0092] The occupancy probability corresponding to the aforementioned voxel can be the occupancy probability at time t and spatial location (x, y, z), which can be denoted as O(t, x, y, z), a scalar with an occupancy probability of 0 or 1. The semantic category identifier corresponding to the voxel can be the semantic category identifier at time t and spatial location (x, y, z), which can be denoted as Sem_ID(t, x, y, z), such as "vehicle", "pedestrian", "lane line", "drivable area", etc. The three-dimensional motion flow vector corresponding to the voxel can be the three-dimensional motion flow vector at time t and spatial location (x, y, z), which can be denoted as Flow(t, x, y, z), representing the motion flow vector in three-dimensional space at time t, representing the motion of the voxel from time t to t+1. The three-dimensional spatial coordinates of the voxel grid, denoted as (x_global, y_global, z_global), are the precise coordinates of the voxel in the map coordinate system. These coordinates are uniquely determined by the voxel's position within the grid and the origin of the map coordinate system. They represent the precise three-dimensional spatial coordinates of the voxel grid in the absolute map coordinate system and can be directly mapped to the world coordinate system from the grid origin (the vehicle's pose in the absolute coordinate system). The map element association identifier corresponding to the voxel represents the map element association identifier corresponding to the voxel at time t and spatial position (x, y, z), denoted as Map_Assoc(t, x, y, z). This identifier can be obtained by querying the nearest map element, determining whether the voxel belongs to "Intersection 001" or "Lane 021".

[0093] In these implementations, because each voxel simultaneously records occupancy probability, semantic category, 3D motion flow, spatial coordinates, and map association identifiers in the four-dimensional grid occupancy information, this data structure can carry multi-level information such as perception, semantics, motion, localization, and map association within a unified framework. The 3D spatial coordinates carried by the voxel establish a direct mapping to the absolute coordinate system, avoiding coordinate transformations during subsequent use; the map association identifier pre-associates each voxel with specific map elements (such as lanes and intersections), providing a direct search basis for spatial consistency verification; and the motion flow vector characterizes the voxel's changing trend over time. The integration of this information reduces, to some extent, the additional computational and association costs required for cross-modal analysis (such as perception and map comparison, semantic and location verification), and improves the information density and usability of the four-dimensional grid occupancy information as input data for subsequent hard example mining and scene understanding tasks.

[0094] The aforementioned four-dimensional raster occupancy information is not only a three-dimensional spatial voxel grid, but also includes a voxel state flow (“occupancy flow”) in the time dimension. All elements, whether perceived vehicles and pedestrians, lane lines in the map, or objects that may be referred to in future text descriptions, are registered in the same quantifiable world coordinate system, thus unifying the homogeneous space reference system. Furthermore, since the network output is map-aligned, the model automatically obtains the precise map coordinates and map semantic attributes of each voxel (e.g., “this voxel is located within the passable area of ​​intersection _001”) while predicting the occupancy flow, thus enabling automatic annotation of the aforementioned four-dimensional raster occupancy information. This significantly reduces the post-processing cost of offline association between perception results and maps.

[0095] In step S103, map vector features are used as geometric priors to constrain the training and inference of four-dimensional raster occupancy information. A map element vectorization encoder encodes various static map vector elements (including lane lines, curbs, intersection boundaries, etc.) in the high-precision map tiles into a compact set of feature embeddings carrying geometric attributes, denoted as F{map}. Each map vector element (e.g., a lane line) has a map vector feature vector that includes not only the element's category attribute (e.g., solid or dashed line) but also its detailed geometric description information, such as spline curve parameters and spatial direction vectors.

[0096] In each layer of the occupancy prediction head's decoding process, BEV features are modulated using map vector feature vectors, thereby forcing the four-dimensional raster occupancy to reference the map's geometric layout when predicting occupancy probabilities. Specifically, conditional batch normalization or spatially guided feature linear modulation techniques can be used to modulate BEV features using map vector features. For example, when predicting occupancy near the curb, it is strongly guided by the curb's geometric information, causing the predicted occupancy boundary to naturally tend towards the map's curb, achieving implicit alignment.

[0097] It is understandable that the above-mentioned four-dimensional grid occupancy can use an explicit occupancy network, or it can use a dense point cloud after depth completion based on a high-precision map, or project the point cloud onto the BEV plane and align it with the map at the feature level, as a way to achieve "unified alignment space" four-dimensional grid occupancy.

[0098] The aforementioned scheme, by using map tile information as the decoding process for the first perceptual feature to constrain geometric priors, naturally aligns the generated four-dimensional raster occupancy information with the map coordinate system. This allows for the acquisition of the world coordinates and map association identifiers for each voxel without post-processing coordinate transformation. Simultaneously, through deep fusion of image and point cloud features, complementary information between texture semantics and geometric structure is preserved, improving the accuracy and robustness of occupancy prediction. Therefore, this scheme can construct a spatially consistent and semantically rich four-dimensional environment representation within a unified framework, providing a directly quantifiable benchmark for subsequent cross-modal verification and hard example mining. This reduces the difficulty and cost of data alignment to some extent and improves the accuracy and reliability of the perceptual results obtained from the aforementioned data. By introducing map information as an absolute anchor point, multi-source information such as perception, map, and language is unified under the same physical world coordinate system, laying a data foundation for achieving high-precision scene reconstruction and logical verification, and solving the problem of data fragmentation.

[0099] The aforementioned scheme constructs a unified four-dimensional raster occupancy space with the absolute coordinate system of a high-precision map as the sole physical anchor. This scheme enables each perceptual voxel and each map element to obtain unique, computable physical coordinates and geometric relationships within this space. Specifically, since the four-dimensional roxel occupancy information is naturally aligned with the map coordinate system, the perception results (such as vehicles, pedestrians, and lane lines) can be spatially compared with the high-precision map at the pixel or voxel level without post-processing. This achieves automatic spatial association between the perception output and the static structure of the map, meaning that the precise map coordinates and semantic attributes of each voxel can be obtained simultaneously with the generation of occupancy information. Therefore, this scheme, to a certain extent, eliminates the coordinate transformation and offline association costs required for cross-modal verification, providing a directly computable data foundation for subsequent semantic conflict detection based on spatial consistency.

[0100] In some implementations, step S103 above includes the following sub-steps:

[0101] First, the map tile information is encoded to generate map feature vectors.

[0102] Secondly, the first perceived features are modulated using map feature vectors.

[0103] Finally, based on the modulated first sensing features, four-dimensional grid occupancy information is generated.

[0104] Specifically, the data processing device can first convert each vector element (lane line, curb, stop line, etc.) in the map vector data into a feature vector with geometric attributes that can be used by the neural network. For example, a lane line spline curve can be mapped into a fixed-length feature vector (called a map feature vector) through an encoder composed of a multi-layer graph neural network. This feature vector implicitly expresses the spatial shape, local context, and semantic category of the vector element. Specifically, the feature vector of each map element (such as a lane line) not only contains its category information (solid line / dashed line) but also its spline curve parameters, direction vector, and other geometric descriptions. The set of feature vectors corresponding to all map vector elements constitutes a map feature library. All encoded map feature vectors are stored in association with the spatial location (or spatial influence range) of their corresponding vector elements, forming a spatially indexable feature library. For example, the map feature vector of each lane line is bound to its sampling point coordinates or minimum bounding box.

[0105] The data processing device can acquire a first sensing feature, which is a top-view (BEV) feature map with dimensions H×W×C, where H and W are the height and width of the top-view grid, and C is the number of feature channels. Each grid cell corresponds to a rectangular region in three-dimensional space (e.g., 0.2m×0.2m) and has a center point coordinate (x,y). The data processing device can modulate the first sensing feature using conditional batch normalization or spatially guided feature linear modulation methods.

[0106] Specifically, the data processing device performs the following modulation operation on each spatial location on the BEV feature map:

[0107] First, find the nearest map feature vector. For the current BEV feature map location p=(x,y), the data processing device can calculate the spatial distance between this location and each map vector element (e.g., the shortest distance from a point to a spline curve, or the shortest distance from a point to a bounding box). Select the map vector element with the smallest distance and obtain the corresponding map feature vector e. If the minimum distance exceeds a preset threshold (e.g., 2 meters), a default zero vector is used or the modulation at that location is ignored.

[0108] Second, generate affine transformation parameters. The data processing device can input the map feature vector e into a lightweight parameter prediction network (e.g., a small neural network consisting of two fully connected layers), which outputs two scalars: a scaling factor γ and a translation factor β. In some implementations, γ and β can be generated independently for each feature channel. For example, at a location near a lane line (p{bev}), based on the nearest lane line in the query (F{map}), a scaling factor γ and a translation factor β are generated, and an affine transformation is performed on the features at that location.

[0109] Specifically, the data processing device extracts a second perceptual feature (e.g., a vector f of dimension C) at position (p{bev}) in the first perceptual feature, and performs an element-wise affine transformation on it: f' = γ·f + β; where multiplication is scalar-vector multiplication and addition is broadcast addition. The transformed feature vector f' replaces the value of the original feature vector f at position (p{bev}). This modulation operation guides the spatial distribution of the decoded features to map geometric information, enhancing the aforementioned features while suppressing features in drivable areas (such as off-road areas).

[0110] The data processing device performs the above operations on all locations in the BEV feature map to obtain a modulated perceptual feature map. This method forces the network to "reference" the geometric layout of the map when predicting occupancy probabilities. For example, when predicting occupancy near the curb, the network is strongly guided by the geometric information of the curb, causing the predicted occupancy boundary to naturally tend towards the map curb, achieving implicit alignment.

[0111] The data processing device can input the modulated first perceptual features into the decoder. The decoder may include a multi-layer neural network, such as one consisting of multiple upsampling layers and convolutional layers stacked together, with the first perceptual features (BEV feature map) as input and four-dimensional grid occupancy information as output.

[0112] In these implementations, map feature vectors are generated by encoding each vector element in the map vector data. These feature vectors are then used to modulate the first perceptual feature. Based on the modulated first perceptual feature, four-dimensional raster occupancy information aligned with the map information is generated. This transforms discrete, structured map prior knowledge into a vector form that can participate in numerical calculations and deeply embeds it into the decoding process of the perceptual feature. Compared to related technologies that treat map information as an independent layer or apply weak constraints only to the loss function, this scheme uses modulation operations to allow map geometric information to directly guide the spatial distribution of the perceptual response at the feature level. This ensures that the generated occupancy information maintains geometric consistency with the map coordinate system at the source, reducing the coordinate transformation and association matching steps required for post-processing alignment and lowering cross-modal verification errors caused by coordinate system inconsistencies.

[0113] In some implementations, the above-mentioned modulation of the first perceptual feature using map feature vectors includes: modulating the first perceptual feature input to at least one layer of the decoder using map feature vectors;

[0114] Furthermore, based on the modulated first perceptual features, four-dimensional grid occupancy information is generated, including:

[0115] The first perceptual feature, modulated by at least one layer of neural network, is input into the occupancy prediction head of the decoder, and the occupancy prediction head outputs four-dimensional grid occupancy information.

[0116] In each layer of the above-mentioned at least one network, the spatial location feature values ​​in the first perceptual feature can be purposefully adjusted using map feature vectors so that the adjusted features maintain geometric consistency with the map structure in terms of spatial distribution.

[0117] After processing all network layers, the data processing device can input the final output of the aforementioned coding layers into an occupancy prediction head (e.g., one or more convolutional layers). This occupancy prediction head maps the modulated perceptual feature map into four-dimensional raster occupancy information. The occupancy information for each voxel in the four-dimensional raster occupancy information is a five-dimensional tensor, represented as T×X×Y×Z×C, where T represents the time frame number (e.g., the past 5 frames or the predicted 8 frames).

[0118] In these implementations, by modulating the first perceptual feature using map feature vectors in at least one layer of the decoder's neural network, and using the modulated feature as the input to the occupancy prediction head to output four-dimensional grid occupancy information, a multi-layered, progressive injection of map geometric constraints into the decoding process is achieved. Since the modulation operation is not limited to a single fixed layer of the decoder but occurs in at least one network layer, the map prior can repeatedly guide the spatial distribution of the perceptual response at different stages of progressive feature space resolution recovery. This enhances the geometric consistency between the predicted occupancy information and the map coordinate system to a certain extent and reduces alignment errors caused by feature degradation after a single modulation. Simultaneously, directly inputting the modulated feature into the occupancy prediction head for the final output allows the entire generation process to be completed within a unified framework, avoiding additional post-processing alignment steps and thus improving the efficiency of cross-modal data processing.

[0119] In some examples, the decoder is trained based on a first loss function, which comprises one or more of the following sub-loss functions: occupancy-map geometric consistency loss, semantic occupancy projection consistency loss, and vector direction consistency loss. The occupancy-map geometric consistency loss compares the geometric differences between the predicted four-dimensional raster occupancy information and the static occupancy raster rendered from map vector data. The semantic occupancy projection consistency loss constrains the projection of occupancy voxels classified as drivable areas onto the horizontal plane to lie within the drivable area boundary defined by the map vector data. The vector direction consistency loss constrains the spatial orientation of the predicted lane line voxel sequence to be consistent with the direction vector of the corresponding lane line in the map vector data.

[0120] In order for the occupancy prediction head to modulate the BEV feature map using map vector features, the occupancy prediction head can be trained using a pre-constructed first loss function, which includes the following: occupancy-land Figure 1 The system includes consistency loss, semantic occupancy consistency loss, and vector direction consistency loss. Among these, the occupancy-map geometric consistency loss is used to compare the geometric differences between the predicted four-dimensional raster occupancy information and the static occupancy raster indicated by the map tile information; the semantic-occupancy projection consistency loss is used to constrain the projection of occupancy voxels classified as drivable areas onto the horizontal plane to lie within the drivable area boundary defined by the map tile information; and the vector-direction consistency loss is used to constrain the spatial orientation of the predicted lane line voxel sequence to be consistent with the direction vector of the corresponding lane line in the map tile information.

[0121] Occupy land Figure 1 The consistency loss L1 refers to the comparison between the predicted 3D occupancy raster O_pred∈R^{X×Y×Z} and the 3D static occupancy raster O_map rendered from the high-precision map (which renders lane lines, curbs, buildings, etc. as occupancy voxels). The relevant formula (1) is as follows:

[0122] L1=Dice Loss(O_pred, O_map)+0.5×Focal Loss(O_pred, O_map) (1);

[0123] Among them, Dice Loss is sensitive to the geometry of the foreground, while Focal Loss can handle severe class imbalance.

[0124] Occupy land Figure 1 The consistency loss L1 serves to penalize any spatial misalignment between the predicted occupancy and the static map structure, providing a fundamental guarantee for spatial alignment. It compares the predicted 3D occupancy raster with the 3D static occupancy raster rendered from the high-precision map (rendering lane lines, curbs, buildings, etc., as occupancy voxels). The geometric Dice Loss or Focal Loss between the two is calculated, penalizing areas where the predicted occupancy does not match the static map structure.

[0125] The semantic occupancy consistency loss L2 refers to the requirement that the projection of occupancy voxels classified as "drivable areas" onto the horizontal plane (BEV) must strictly lie within the drivable area enclosed by high-precision map vectors. The derivation of the relevant formula is as follows: First, through a differentiable operation Project(O_pred), the occupancy voxels of the "drivable" category are logically ORed along the Z-axis to obtain the projection mask M_pred_bev on the BEV plane. At the same time, the accurate mask M_map_bev on the drivable area BEV plane is rendered using map vectors. The relevant formula (2) obtained is as follows:

[0126] L2 = 1 - 2 × |M_pred_bev∩M_map_bev| / (|M_pred_bev|+|M_map_bev|) (2); where “∩” is the intersection operation and “+” is the union operation. That is, this loss is achieved by calculating the intersection-union ratio of the projection of the predicted occupancy voxels and the map drivable area mask. The role of semantic occupancy consistency loss L2 is to force semantic alignment, ensuring that the model’s cognition of “where you can drive” is absolutely consistent with the physical world.

[0127] The vector direction consistency loss L3 refers to the spatial curve formed by a set of four-dimensional grid occupancy sequences O_lane that are successfully detected as "lane" instances, which should be locally consistent with the direction of the nearest lane line in the map. The formula derivation process is as follows: For each voxel in O_lane, the cosine similarity between the direction of the three-dimensional spatial curve (v_map) formed by the occupancy sequence predicted as a "lane" instance and the predicted lane line direction vector (v_pred). Simultaneously, the direction vector of the nearest lane line segment in the map is queried. The formula (3) for calculating the vector direction consistency loss L3 is as follows:

[0128] L3=Mean[(1-Cosine_Similarity(v_pred, v_map)) 2 (3);

[0129] Here, `Cosine_Similarity()` is the cosine similarity function, and `mean()` is the mean function. The vector direction consistency loss is used to fine-tune the orientation of the predicted target. Alignment at complex topological structures such as intersections and curves is crucial. For example, for an occupancy sequence predicted as a "lane line" instance, the direction of its resulting 3D spatial curve, after being projected onto the BEV, should be consistent with the tangent direction of the nearest lane line in the map. This is achieved by applying a cosine similarity loss between the predicted occupancy sequence and the map vector.

[0130] The following example illustrates this. Suppose a vehicle is traveling on a city road with a solid left lane line and a dashed right lane line. During the prediction head decoding process, for a grid cell (i,j) on the BEV located near the left lane line, the map feature map M_map indicates that the cell is only 0.1 meters away from the solid left lane line, the lane line type is "solid line," and the direction is 10 degrees east of north. The map feature m{ij} is processed by a fully connected network to generate a large scaling factor γ=1.5 and a small translation factor β=0.1. After being applied to the BEV feature of this cell, subsequent convolutional layers at this location are more inclined to predict outputs with "high occupancy probability and semantic category of curb / lane line." Simultaneously, in the temporal dimension, since the position of this lane line is fixed across multiple consecutive frames, in frames t-2 to t+2 of the prediction head output, the occupancy probability and semantic label of this voxel remain consistent, and the motion flow vector is zero, reflecting the spatiotemporal consistency of static map elements. In the final output of the four-dimensional raster occupancy information, the voxel at this location will be clearly marked as "lane line (solid line)" and associated with the map identifier "Lane_01", and will be aligned with the map location in different time frames.

[0131] The aforementioned four-dimensional raster occupancy information includes not only three-dimensional spatial voxel rasters but also voxel state flows ("occupancy flows") in the time dimension. This unifies the spatial reference system and enables automatic annotation. All elements, whether perceived vehicles and pedestrians, lane lines on the map, or objects that may be referred to in future text descriptions, are registered in the same quantifiable world coordinate system. Because the raster occupancy information is map-aligned, the precise map coordinates and map semantic attributes of each voxel (e.g., "this voxel is located within the passable area of ​​intersection _001") are automatically obtained while predicting the occupancy flow. This reduces the post-processing costs of offline association between perceived data and maps.

[0132] Figure 2 This is a schematic diagram of a perception data processing flow based on map information, such as... Figure 2As shown, the input layer receives multi-source heterogeneous data, including panoramic images (multi-channel video streams), point cloud data, and high-precision map tiles. The panoramic images are processed by an image backbone network to extract multi-scale image features; the point cloud data undergoes voxelization and 3D sparse convolution to obtain point cloud voxel features; and the map tiles are encoded into map feature vectors by a map vector element encoder. The feature fusion and projection layer converts image features into image BEV features through a 2D projection onto a top-down view (e.g., LSS depth projection), and simultaneously flattens the point cloud voxel features along the height to obtain point cloud BEV features. Both are then fused through an encoding layer and heterogeneous features (e.g., adaptive gating fusion) to generate a unified BEV feature map. It can be understood that the backbone network can be a pure Transformer-based autoregressive model or a state-space model-based network, as long as it can uniformly process multimodal inputs and has multi-dimensional output capabilities such as text, occupancy, and trajectory.

[0133] The map prior injection module modulates the unified BEV feature map using map feature vectors (such as conditional batch normalization or spatial feature linear modulation), injecting geometric priors into the decoding process. Simultaneously, network training is supervised through map alignment loss functions (including geometric consistency, semantic projection consistency, and vector direction consistency), forcing the prediction results to align with the map.

[0134] The four-dimensional occupancy prediction head (decoder) in the core layer receives the modulated features and finally outputs four-dimensional occupancy information that is spatiotemporally aligned with the map coordinate system. Each voxel contains occupancy probability, semantic label, motion flow vector, world coordinates and map association identifier.

[0135] This process achieves spatiotemporal alignment of multimodal data using high-precision maps as absolute anchors, paving the way for subsequent sensing-and-ground processing. Figure 1 Consistency checks and hard case mining provide a unified spatial benchmark.

[0136] Figure 3 Here is a schematic diagram of another map-based perception data processing flow, such as Figure 3 As shown, the input layer receives three types of multi-source heterogeneous data: surround view image data, point cloud data, and high-precision map tiles. In the encoding layer, the surround view image extracts multi-scale features through an image backbone network; the point cloud data is voxelized and 3D sparse convolution is used to extract voxel features. The map element vectorization encoder encodes vector elements (such as lane lines, curbs, and intersection topology) in the map tiles into map feature vectors. In the feature fusion stage, image features are converted into image BEV features through 2D projection to BEV (such as LSS depth projection), and point cloud voxel features are flattened along the height to obtain point cloud BEV features. The two are then fused in a unified BEV space (such as adaptive gating fusion) to generate a unified BEV feature map.

[0137] The map prior injection module modulates the BEV features of each layer in the multi-scale occupancy decoder using map feature vectors (e.g., conditional batch normalization or spatial feature linear modulation), while introducing a temporal loop (temporal fusion module) to process temporal information between consecutive frames. The loss supervision layer employs three types of map alignment losses: occupancy-map geometric consistency loss (comparing the predicted occupancy with the static occupancy of the rendered map using Dice / Focal Loss), semantic occupancy projection consistency loss (constraining the BEV projection of the drivable area to be within the map boundary), and vector direction consistency loss (constraining the predicted lane line direction to be consistent with the map direction). The network is trained through backpropagation after a weighted sum of these losses.

[0138] Finally, the decoder outputs four-dimensional raster occupancy information that is spatiotemporally aligned with the absolute coordinate system of the map. Each voxel contains occupancy probability, semantic label, motion flow vector, world coordinates and map association identifier, realizing pixel / voxel-level alignment of multimodal data under a unified physical anchor point.

[0139] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the map information-based data processing method provided in this disclosure. Figure 2 ,like Figure 4 As shown, the method includes, in addition to, with Figure 1 In addition to steps S401-S403 that are the same as or similar to steps S101-S103 in the illustrated embodiment, the following steps are also included:

[0140] S404: A multimodal visual encoder based on a preset model encodes the four-dimensional raster occupancy information and image features to obtain the first vector.

[0141] The data processing device can input image features (such as panoramic image features) and four-dimensional raster occupancy information into a multimodal visual encoder, which outputs a first vector. The multimodal visual encoder can simultaneously receive panoramic image features and map-aligned four-dimensional raster occupancy features as input. Treating four-dimensional raster occupancy as a special type of "three-dimensional image sequence," a unified visual token sequence (T_{vision}) is learned through a spatiotemporal attention mechanism that can both represent static scene structure and capture the motion of dynamic objects.

[0142] In some examples, step S504 above includes the following sub-steps:

[0143] First, the image features are divided into blocks to obtain the first block result; the first block vector is generated based on the pixel values ​​of the first block result.

[0144] The data processing device can divide the panoramic image feature map into several non-overlapping rectangular regions (also called patches) in a spatial dimension according to a preset patch size (e.g., 16×16 pixels). Each rectangular region is called a patch result. For each patch result, the data processing device can extract the pixel values ​​or feature values ​​at all positions within the patch result and expand it into a one-dimensional sequence according to spatial order (e.g., line-by-line scanning) to obtain the original vector of the image patch. Subsequently, the data processing device can input the original vector into a linear projection layer (i.e., a fully connected layer) to map it to a preset target feature dimension (e.g., 768 dimensions) to generate the first patch vector corresponding to the image patch. In one example, if the original patch contains 16×16×3 values, the linear projection layer compresses or expands it into a 768-dimensional floating-point vector. Each image patch performs the above operation independently to obtain several first patch vectors, each of which represents the appearance semantic information of a local image region.

[0145] Second, the four-dimensional grid occupancy information is divided into blocks to obtain the second block result; the second block vector is generated based on the height value and channel value of the second block result.

[0146] The data processing device can treat the four-dimensional raster occupancy information O∈R^{T×X×Y×Z×C} as a spatiotemporal continuum. It can use a spatiotemporal voxel patching layer to divide the information into multiple non-overlapping spatiotemporal voxel blocks according to a preset spatiotemporal block size. Each spatiotemporal voxel block represents a segmentation result. Specifically, the spatiotemporal voxel patching layer can segment the four-dimensional raster occupancy information along the time dimension T, spatial length dimension X, and spatial width dimension Y. The size of each spatiotemporal voxel block is tblock×xblock×yblock, where tblock represents the time window length, and xblock and yblock represent the block size in the horizontal length and width directions, respectively. For each spatiotemporal voxel block, the data processing device can retain the complete data of its internal height dimension Z and channel dimension C, and flatten this data within the block along the height and channel dimensions, that is, sequentially concatenating the height layer features and channel features of all voxels into a one-dimensional sequence. The flattened one-dimensional sequence is used as the second block vector corresponding to the spatiotemporal voxel block. Each spatiotemporal voxel block performs the above operation independently, thereby obtaining several second block vectors. Each second block vector represents the scene dynamic occupancy information within a specific time window and a specific horizontal spatial range.

[0147] Third, the first block vectors of multiple first block results and the second block vectors of multiple second block results are concatenated, and the concatenated result is encoded to obtain the first vector.

[0148] The data processing device can merge all first block vectors and all second block vectors into a single sequence. During merging, the device arranges all first block vectors in a predetermined order to obtain image patch tokens, and arranges all second block vectors to obtain voxel block vectors (voxel raster occupancy tokens). The image patch tokens and voxel raster occupancy tokens are then concatenated to obtain the first vector. Before concatenation, the device adds a spatial location code to each first block vector, representing the spatial position of the corresponding image patch in the original image. Simultaneously, the device can add a spatiotemporal location code to each second block vector, representing the position of the corresponding spatiotemporal voxel block on the time axis and horizontal spatial axis. After the location codes are merged with the corresponding block vector information, they participate in sequence concatenation. Subsequently, the device can input the concatenated sequence into an encoder network, which consists of multiple stacked Transformer layers. In each Transformer layer, each block vector in the sequence interacts with all other block vectors through a self-attention mechanism, fusing information from different spatial locations, time windows, and modalities. The encoder network outputs an encoded sequence, which is the first vector. The first vector is a unified visual token sequence T_vision with rich spatiotemporal awareness, where each element is an enhanced feature representation that fuses local image appearance information and local spatiotemporal occupancy information, which can be subsequently used for tasks such as scene understanding, hard example discrimination, or trajectory planning.

[0149] The above scheme generates a first block vector based on image feature segmentation, a second block vector based on four-dimensional grid occupancy segmentation, concatenates the first block vectors from multiple first block results with the second block vectors from multiple second block results, and encodes the concatenated result to obtain a first vector. This achieves the fusion representation of image appearance information and spatiotemporal occupancy information in a unified feature space. Since the information from both modalities is encoded into vectors in the same sequence and interacts through a self-attention mechanism, the model can simultaneously perceive local texture details and local dynamic occupancy structures, improving the alignment ability and semantic consistency of cross-modal features to a certain extent. The final first vector integrates multi-scale, multi-modal scene information, providing a richer feature foundation for subsequent tasks such as environment understanding, object detection, or scene retrieval.

[0150] S405: Map the first vector to the language representation space to obtain the second vector.

[0151] After obtaining the first vector T_vision, the data processing device can input the first vector T_vision into a spatial projector, which will then map the first vector onto the language representation space to obtain the second vector. The second vector is a set of visual lexical units within the language space.

[0152] In some examples, the spatial projector can consist of one or more fully connected layers, whose input dimension matches the feature dimension of the first vector, and whose output dimension is the same as the lexical vector dimension used by the language model. Through linear or nonlinear transformations by the spatial projector, the first vector is mapped into the language representation space used by the language model, resulting in a second vector. The second vector can be a visual token (T{vis-lang}), which resides in the same semantic space as the text lexical vector in terms of vector dimension and numerical distribution, allowing subsequent large language models to treat it as an understandable input unit.

[0153] S406: Input the second vector and map features enhanced by map knowledge into the backbone network of the preset model, and output multi-dimensional scene description information from the backbone network.

[0154] The map features enhanced by map knowledge mentioned above are feature representations obtained by encoding the original map vector data and fusing prior map knowledge. They carry information such as road geometry, topological relationships and traffic rules corresponding to the current scene.

[0155] The data processing device can input the second vector and map features enhanced by map knowledge together into the backbone network of a pre-defined model. This backbone network, for example, is built based on a pre-trained language model, and its input interface is extended to accept feature vector sequences from different modalities. During input, the second vector serves as a word vector of the visual modality, and the map features serve as a structured prior knowledge word vector; the two are concatenated in a pre-defined order to form a multimodal input sequence.

[0156] After receiving the input sequence, the backbone network deeply fuses the information from the two modalities in its internal Transformer layers through self-attention and cross-attention mechanisms. Specifically, the backbone network can associate and interact with the visual semantic information in the second vector and the geometric prior information in the map features in a unified semantic space, enabling the model to simultaneously refer to image content, occupancy status, and road structure constraints when understanding the current scene.

[0157] Subsequently, multiple dedicated output heads of the pre-defined model can output multi-dimensional scene description information per word in an autoregressive generation manner. The multi-dimensional scene description information can be presented in a structured format. The aforementioned multiple dedicated output heads may include, for example, a text head H1 (outputting scene semantic description text), a 3D occupancy head H2 (outputting perceptual information of physical entities), an occupancy flow head H3 (outputting 4D raster occupancy prediction information), and a behavior diffusion head H4 (outputting trajectory planning information).

[0158] The text header, for example, could be a standard autoregressive language decoder, whose initial state is initialized by the hidden states of the backbone network. It can generate scene descriptions in natural language form with spatiotemporal references and logical reasoning. Its output is a JSON object containing multiple structured fields, such as: `general_description: "Vehicle is entering a traffic light intersection"`, `dynamic_objects: [{"id": "ego", "behavior": "Slowing down and preparing to turn right"}, {"id": "obj_001", "type": "SUV", "3d_box": "x1, y1, z1, ...", "behavior": "Cutting in from the left lane, intending to go straight", "relative_distance": "12.3m"}, ...]`, `static_environment: {"road_type": "Urban intersection", "traffic_light_state": "Green light for straight, red light for left"}`, `risk_assessment: "Potential conflict risk with Obj_001, as it may force a lane change."` All object identifiers can be found in the H2 output as their corresponding 3D bounding boxes or occupancy masks, containing precise world coordinates. This provides a physical link for subsequent consistency checks.

[0159] The 3D occupancy head H2 receives perception-related hidden states from the backbone network and detailed features from the encoder, and connects to a 3D instance segmentation network. It can provide the most accurate perception results, serving as a "fact benchmark." Its output includes one or more of the following: 3D bounding boxes (center point, length, width, height, heading angle) for all dynamic and static targets; fine-grained occupancy masks for each foreground target instance; and precise velocity vectors (vx, vy, vz) for each dynamic target instance.

[0160] Occupation flow H3 (a probabilistic generative four-dimensional world model), for example, could be a world model based on a variational autoencoder or a diffusion model. It receives the historical four-dimensional raster occupancy sequence O_{tT:t} and the backbone network's understanding of the scene h_t as conditional variables. It outputs a predicted four-dimensional raster occupancy probability distribution P(O_{t+1:t+K}|O_{tT:t}, h_t) for the next K frames. This is a probability density cloud, not a deterministic single-frame prediction. This head not only predicts multiple possible futures for a dynamic target (e.g., whether a vehicle accelerates or decelerates), but also quantifies the uncertainty of the prediction by the overall entropy of the prediction distribution. A high-entropy output (i.e., a highly uncertain future) is itself a strong signal of interest.

[0161] The behavior diffusion head H4 (safety-aware multimodal trajectory planning) can be a conditional denoising diffusion probability model. It generates the vehicle's future trajectory iteratively through an iterative denoising process, conditioned on scene understanding h_t, perception results O_det, and predicted occupancy flow O_future. This head can have multimodal outputs, such as generating N possible trajectories τ_i, like ["constant speed following", "accelerating lane change", "emergency right yield"]. Furthermore, this head can combine the future prediction from the occupancy flow head to calculate a comprehensive collision risk probability CRP(τ_i) for each trajectory τ_i. The CRP is obtained by calculating the expected intersection of the trajectory path voxels and the predicted future occupancy voxels in space and time.

[0162] In some examples, multidimensional scene description information includes one or more of the following: scene semantic description text; physical entity perception information; four-dimensional grid occupancy prediction information; trajectory planning information.

[0163] The scene semantic description text refers to the structured description of the current scene in natural language. Specifically, it includes: an overall semantic description of the scene (such as road type, lighting conditions, and weather conditions), a description of the behavior of dynamic targets (such as "a car on the left is accelerating to change lanes"), static environmental information (such as lane line type and traffic light status), and potential risk assessments or driving suggestions. This description is presented in human-readable text or JSON format to facilitate subsequent semantic reasoning and analysis.

[0164] Perceptual information of physical entities refers to the fine-grained, structured descriptive information obtained after perceiving each physical entity (such as vehicles, pedestrians, cyclists, traffic signs, etc.) in a scene. The perceptual information of each physical entity includes at least: the entity's 3D bounding box in absolute coordinates (center point coordinates, length, width, height, and heading angle), the voxel-level mask occupied by the entity (i.e., instance-level occupancy information), the entity's 3D motion velocity vector, and the entity's unique instance identifier and category label. This information serves as an objective factual benchmark in the scene and can be used for subsequent consistency checks or hard-case discrimination.

[0165] Four-dimensional grid occupancy prediction information refers to the results obtained by the model predicting the four-dimensional occupancy of the grid at multiple consecutive future time points. This prediction can be output in the form of a probability distribution, providing the probability value that each voxel (spatial unit) will be occupied by an object at each future time point. Simultaneously, this information also includes the overall entropy value of the prediction distribution, serving as a measure of uncertainty; the higher the entropy value, the more uncertain the model's judgment on future evolution. This information can be used to assess the dynamic complexity and potential risks of a scene.

[0166] Trajectory planning information refers to the multiple candidate future driving trajectories planned by the multimodal language model for the autonomous vehicle, along with the corresponding safety score for each trajectory. Each trajectory consists of a series of discrete waypoints (position and orientation), representing different driving intentions (such as maintaining a constant speed while following another vehicle, changing lanes to overtake, or slowing down to yield). The safety score for each trajectory is a quantitative indicator (e.g., collision risk probability), used to measure the degree of risk of collision with other objects during future driving. This information provides subsequent decision-making with selectable multimodal planning schemes and their safety basis.

[0167] The aforementioned scheme achieves multi-level information integration from semantics to geometry, from the present to the future, and from perception to decision-making by simultaneously outputting scene semantic description, physical entity perception information, four-dimensional grid occupancy prediction, and trajectory planning information. Semantic description provides interpretable scene understanding, perception information provides accurate instance-level geometric and motion data, occupancy prediction provides probabilistic evolution in future spatiotemporal contexts, and trajectory planning provides vehicle behavior and its safety assessment. These four types of information complement and cross-validate each other, enabling subsequent hard-case discrimination to be cross-validated from multiple dimensions such as semantic consistency, geometric consistency, temporal consistency, and behavioral safety. This improves the coverage and accuracy of anomaly detection in complex scenarios to a certain extent and provides rich data resources for interpretability analysis.

[0168] In some examples, the method also includes the following steps:

[0169] First, based on the map features, a query is performed from the map prior knowledge base to obtain at least one corresponding map prior knowledge vector;

[0170] Secondly, at least one map prior knowledge vector is associated with map features to obtain map features enhanced with map knowledge.

[0171] The data processing device can pre-build and store a map prior knowledge vector library, which contains multiple map prior knowledge vectors T_map-mem∈R^{M×D}, where M is the number of map prior knowledge vectors (memory tokens) and D is the embedding dimension of the pre-defined model. These vectors are stored and updated as part of the model parameters. Each map prior knowledge vector is obtained through training and is used to encode general map topology rules, road structure common sense, or traffic behavior priors (e.g., "At a T-junction, the lane where this vehicle is located is usually for left turns and straight ahead," "Solid line lanes prohibit lane changes," "There is a pedestrian crossing at the intersection, so be careful of pedestrians," etc.). Each map prior knowledge vector has a fixed vector dimension, which can be consistent with the word vector dimension of a language model.

[0172] In each forward propagation, multiple map prior knowledge vectors T_map-mem are injected as a fixed set of keys and values ​​into the multi-head attention mechanism of the Transformer layer inside the pre-defined model.

[0173] The data processing device can project the map features and context F_map_context generated from the current high-precision map tiles onto the BEV as a query vector and perform cross-attention interaction with T_map-mem. Based on the cross-attention interaction result, at least one map prior knowledge vector with the highest similarity can be selected as the query result. Each selected map prior knowledge vector corresponds to a map prior rule or prior knowledge most relevant to the current scene. For example, when focusing on an intersection area, the relevant map prior knowledge vector will be activated, providing internal knowledge information such as "this is a T-junction, and the lane where this vehicle is located is left turn + straight," thereby supporting a more robust and logical understanding and behavior. In addition, the data processing device can also record the similarity value of each selected vector as the matching confidence of that vector. That is to say, when the preset model performs any reasoning (such as describing a scene or planning a path), its core can not only see the current visual input, but also dynamically retrieve relevant topological rules and prior knowledge from the current map context through these memory tokens.

[0174] The data processing device can associate and fuse at least one map prior knowledge vector obtained from the query with the original map features. The fusion methods include, but are not limited to, one or more combinations of the following: concatenating at least one map prior knowledge vector with map features along the feature channel dimension to form a higher-dimensional joint feature vector; weighting and summing at least one map prior knowledge vector according to its similarity weights to obtain an aggregated prior vector, and then adding or concatenating this aggregated prior vector with the map features; or using at least one map prior knowledge vector as additional conditional input, and generating an enhanced feature representation through cross-attention interaction with the map features.

[0175] The new feature vector obtained after the above correlation and fusion is the map knowledge-enhanced map feature. This feature contains both the instantaneous map vector information of the current scene and general map rules and prior knowledge retrieved from the prior knowledge base. This allows the subsequent multimodal language model to utilize both instance-level map data and abstracted prior knowledge when processing this feature. This map knowledge-enhanced map feature is then input into the multimodal language model to generate scene descriptions or perform semantic reasoning.

[0176] The above scheme achieves an organic combination of instance-level map information and general map prior rules by querying at least one map prior knowledge vector matching the current map features from the map prior knowledge vector library and fusing it with the map features. Since map prior knowledge vectors encode abstract rules such as road topology and traffic behavior common sense, while map features describe the specific geometric and topological information of the current scene, the fusion of the two allows the map features enhanced with map knowledge to retain both the instantaneous characteristics of the current scene and supplement general prior knowledge that transcends a single scene. This scheme can, to some extent, alleviate the inference bias caused by data sparsity or insufficient local information when relying solely on current map features, and improve the understanding ability and semantic accuracy of subsequent multimodal language models for complex scenes (such as rare intersections and temporary construction).

[0177] Figure 5 This is a schematic diagram of a process for generating multi-dimensional scene description information. For example... Figure 5As shown, the data processing device can receive panoramic image data and point cloud data, inputting these data into the input layer of a preset model. After encoding in the input layer, a visual vector, the first vector Tvision, is generated. This vector is then mapped to a language space visual vector, the second vector Tvis−lang, through a spatial projector. This vector, along with the text instruction vector Ttext, is then input into the backbone network of the preset model. The backbone network interacts with the map memory enhancement module through a query-key attention mechanism to dynamically retrieve prior map knowledge. The backbone network connects in parallel to four output heads: a text head (H1) outputting scene semantic description text; a 3D occupancy head (H2) outputting fine-grained perceptual structured results (including 3D bounding boxes, instance masks, and velocity vectors); an occupancy flow head (H3) based on a generative four-dimensional world model outputting future occupancy prediction sequences and uncertainty entropy values; and a behavior diffusion head (H4) outputting multimodal planning trajectories and safety scores. This architecture achieves end-to-end unified modeling from multimodal perception to semantic understanding, future prediction, and behavioral decision-making.

[0178] In related technologies, hard example mining techniques are mostly "single-modal driven" or "rule-driven." For example, hard examples are determined solely by the uncertainty of the model's output confidence level, or by filtering scenarios based on a few manually defined rules (such as "sudden speed change" or "following distance too close"). These methods can only capture "explicit" anomalies at the sensor level or simple dynamic levels. However, the most dangerous and valuable hard examples facing autonomous driving are often caused by "implicit" cross-modal semantic conflicts. For example: Perception-map conflict: The perception system detects a solid lane, but the high-precision map shows it as a dashed line, suggesting a road rerouting, outdated map, or a serious perception error. Perception-language conflict (illusion): The visual language model incorrectly describes a pedestrian that does not exist in the current scene, revealing a serious flaw in the model that misjudges background texture as a key target. Language-map conflict (logical fallacy): The visual language model describes "the vehicle ahead is preparing to turn left into the oncoming lane," but the map clearly shows a T-junction ahead that only allows right turns, exposing the model's insufficient logical reasoning ability. These deep-seated conflicts are completely inaccessible to traditional single-dimensional mining methods. Figure 6 The illustrated embodiment constructs a multi-dimensional semantic hard example mining engine that integrates "map-perception-language". Through an aligned unified space, it automatically and systematically cross-validates the consistency of these three modalities, thereby accurately capturing these high-value semantic hard examples.

[0179] Please refer to Figure 6 , Figure 6 A flowchart illustrating the map information-based data processing method provided in this disclosure. Figure 3 .like Figure 6 As shown, the method includes, in addition to Figure 4In addition to steps S601-S606 that are the same as or similar to steps S401-S406, the following steps are also included:

[0180] S607: Perform cross-validation based on multi-dimensional scene description information and map tile information;

[0181] In some examples, step S607 above includes one or more of the following.

[0182] The perceptual information of physical entities in the multi-dimensional scene description information is compared with the map tile information for consistency.

[0183] The perceptual information of physical entities is compared with the semantic description text of the scene in the multi-dimensional scene description information for consistency.

[0184] Perform a consistency comparison between the scene semantic description text and the map tile information;

[0185] The trajectory planning information in the multi-dimensional scene description information is compared with the prediction information in the four-dimensional grid occupancy information for consistency.

[0186] Multidimensional scene description information includes, but is not limited to: scene semantic description text; physical entity perception information; four-dimensional grid occupancy prediction information; trajectory planning information.

[0187] The data processing device performs a spatial consistency comparison between the 3D bounding box position and instance-level occupancy mask of each detected target (including vehicles, pedestrians, cyclists, traffic signs, etc.) in the perceived information of physical entities and the road geometry elements (including lane lines, curbs, stop lines) and traffic facility elements (including the 3D positions of traffic signs and traffic lights) in the map tile information. For example, this may include: determining whether the perceived vehicle or pedestrian is located within the drivable area defined by the map; determining whether the perceived lane line type (such as solid or dashed lines) is consistent with the type recorded on the map; and determining whether the Euclidean distance between the perceived 3D position of the traffic sign or traffic light and the coordinates recorded on the map is less than a preset threshold. If any of these determinations is negative, a first verification anomaly signal is triggered.

[0188] The data processing device can construct a list of perceived targets from the perceived information of physical entities. Each target includes a category, a 3D bounding box, an instance identifier, and a velocity vector. Simultaneously, it extracts all referred entities and their spatial constraints (e.g., "the blue truck about 15 meters to the left") from the scene semantic description text using named entity recognition or attention weighting. The data processing device can perform a two-way comparison: for each salient target in the perceived target list (e.g., a target with a volume greater than 1 cubic meter), it checks if there is a matching entity in the scene semantic description text with the same category and a positional deviation within the allowable range; if not, it is determined to be a perceived omission. For each entity extracted from the scene semantic description text, it checks if there is a matching target in the perceived target list; if not, it further searches the corresponding spatial region in the four-dimensional raster occupancy information; if still not found, it is determined to be a hallucination. Any anomaly triggers a second anomaly signal.

[0189] The data processing device extracts a set of logical assertions from the map vector data. These assertions include traffic rules such as legal driving directions in lanes, no-turn signs at intersections, and lane function divisions (straight, left-turn, right-turn). Simultaneously, it extracts behavior triples from the scene semantic description text. Each behavior triple includes a subject, behavior type, and spatial or logical constraints (e.g., "a black sedan is traveling from south to north"). The data processing device can then logically match these behavior triples with the map's logical assertions to determine whether the behavior described in the text violates the traffic rules defined on the map. If a logical conflict is detected (e.g., the text describes a vehicle driving in the wrong direction while the map shows the lane as one-way, or the text describes a vehicle turning left while the map shows left turns are prohibited at the intersection), a third anomaly signal is triggered.

[0190] The data processing device can acquire multiple candidate trajectories and a safety score for each trajectory from the trajectory planning information, and simultaneously acquire the probability distribution of four-dimensional occupancy grids for multiple consecutive future time moments from the four-dimensional grid occupancy prediction information. For each candidate trajectory, the data processing device can project the vehicle voxels on the trajectory onto the corresponding future time moment's occupancy grid, calculate the spatiotemporal overlap between the trajectory and the predicted occupancy, and obtain the collision risk probability. If the collision risk probability of a candidate trajectory exceeds a preset safety threshold, or if all candidate trajectories have a high risk, a fourth verification anomaly signal is triggered.

[0191] The aforementioned scheme achieves systematic consistency verification between perception results, semantic descriptions, planned trajectories, and static map priors by cross-validating multi-dimensional scene description information and map tile information. Since the verification covers spatial comparison of physical entities and maps, referential comparison of physical entities and semantic text, logical comparison of semantic text and map rules, and risk comparison of planned trajectories with future occupancy, this scheme can simultaneously detect anomalies from multiple levels, including geometric location, semantic reference, traffic rules, and dynamic safety. Any inconsistency or conflict triggered in any dimension will mark the current scene as a candidate difficult example, thus overcoming to some extent the limitation of single-dimensional judgment in easily missing deep semantic conflicts (such as perceptual-map contradictions, linguistic illusions, logical fallacies, or high-risk planning), and improving the coverage and accuracy of difficult example mining.

[0192] S608: In response to the cross-validation result meeting the preset conditions, determine the scenario corresponding to the multimodal data as a candidate difficult case.

[0193] The aforementioned preset conditions can indicate a verification anomaly for any cross-validation result. The data processing device integrates the results of the above verifications. If any of the above cross-validation results triggers an anomaly signal, the scenario corresponding to the current multimodal data is identified as a candidate difficult case (candidate semantic-level difficult case), and the triggered verification type and corresponding conflict information are recorded.

[0194] To uncover more extreme and complex challenging cases, an enhanced mode can be set. For example, when any two alignment results indicate an alignment conflict, the difficulty level of the scenario will be increased. For instance, when both the first and second verification anomaly signals appear simultaneously, the difficulty level of the scenario will be increased because the aforementioned verification results indicate that the perceived uncertainty has directly affected driving safety.

[0195] Figure 7 This is a flowchart illustrating the cross-validation process, such as... Figure 7 As shown, the input sources include: physical entity perception information (3D bounding box, instance mask, etc.) provided by the 3D occupancy head H2, (lane lines, curbs, topology rules, etc.), scene semantic description output by the text head H1, planning trajectory and safety score output by the behavior diffusion head H4, and future occupancy prediction sequence output by the occupancy flow head H3.

[0196] Map-perception consistency verification involves the data processing device spatially associating the perceived information of physical entities (lane lines, intersection boundaries, traffic signs) output by the 3D occupancy head H2 with map elements in the map prior knowledge base I4 to achieve map element matching. Then, it compares the attribute differences between matched element pairs to obtain attribute difference metrics. These metrics can include: lane line type (solid / dashed); topological form (intersection over union (IoU) of intersection shapes, lane connectivity), and target location (3D Euclidean distance of traffic signs, traffic lights), etc. If any difference exceeds a preset threshold, the decision-maker can output a first verification anomaly signal. For example, a lane line type difference: if the confidence score of a detected dashed line is greater than 0.8, but the corresponding location on the map is a solid line, this indicates a lane line type difference, and a first verification anomaly signal is output. Another example is if the IoU between the perceived intersection boundary shape and the shape recorded on the map is less than a preset threshold, or if the detected lane connectivity contradicts the map topology, this indicates a topological structure difference, and a first verification anomaly signal is output. For example, if the detected 3D position of a traffic light or stop line has an Euclidean distance greater than a dynamic threshold compared to its recorded position on the map (e.g., greater than 0.5m in urban areas, greater than 1m on highways), or if the detected dashed line has a confidence level greater than 0.8 but the corresponding location on the map is a solid line, this indicates a discrepancy in the location of a key target, and a first verification anomaly signal can be output. This first verification anomaly signal can be triggered when any discrepancy exceeds a preset threshold. This signal may be caused by the following reasons: ① sensor malfunction or perception model defect; ② outdated or incorrect high-precision map data; ③ permanent changes to the real environment (e.g., road construction and rerouting).

[0197] The perception-language consistency check (dimension 2 check) first constructs a perception list by extracting perception information (category, 3D bounding box, identifier) ​​for all physical entities (visible and detected targets) from the 3D occupancy head H2. Specifically, from the output of the occupancy head H2, all detected dynamic targets (vehicles, pedestrians, cyclists) and key static targets (cones, construction signs) are constructed into a structured list, with each entry containing category, ID, 3D position, size, etc.

[0198] Language entity extraction is performed on the text information (scene semantic description) output by the text header H1. Specifically, a lightweight named entity recognition model or the self-attention weights of the generative model are used to extract all explicitly referred to dynamic entities and their referential locations from the JSON format description output by the text header H1. For example, from "blue truck about 15 meters to the left", the following can be extracted: {referential anchor: left, distance: ~15m, category: truck, color: blue}.

[0199] The data processing equipment can then perform spatial constraint omission detection and illusion detection. Omission detection involves identifying physical entities in the perception list that cannot be matched in the linguistic entities. For example, for each target obj_perc in Set_perc, does the description output in the text header H1 contain any extracted entity whose category and location match the 3D bounding box of obj_perc (matching category, and the referencing location and the 3D box projection location are within a preset deviation)? If obj_perc is a salient target with a size greater than 1m³ but no corresponding linguistic entity can be found, it is judged as a "perceptual omission difficult case".

[0200] Hallucination detection is performed if, for example, the extracted language entity from the description JSON output from the text header H1 cannot be matched in the perception list and occupies space in the four-dimensional grid, indicating the presence of a hallucination. For instance, for each entity ent_desc extracted from the above description, does a target exist in Set_perc whose category and location match the entity description? If not, the entity is marked as a candidate hallucination. To further confirm, a secondary search can be performed based on the entity's indicated location (e.g., "50 meters directly ahead"), using the corresponding world coordinate region occupied by the generated four-dimensional grid. If no corresponding category is found within this spatial region and the confidence level exceeds a preset threshold, then a hallucination is confirmed.

[0201] If an omission or hallucination is determined, the second-dimension check triggers a second check anomaly signal. Hallucination is then determined.

[0202] Map-language consistency verification (dimensional three verification) extracts a set of logical assertions from the topological and traffic rule information (such as lane direction and prohibited turns) of the map prior knowledge base I4. Specifically, a set of logical assertions can be automatically extracted from the current map tile information. For example, the assertion "The legal driving direction of this road segment (LaneID:L_01) is from north to south" can be extracted from the one-way street attribute and lane direction attribute.

[0203] Behavior extraction is performed on the text information (scene semantic description) output from the text header H1, that is, behavior triples are extracted from the text header H1. For example, from "A black sedan is speeding from south to north", the behavior triples are extracted as follows: <subject: black sedan>, <behavior: speeding>, <direction: from south to north>.

[0204] Then, logical consistency reasoning is performed based on map rule extraction and language line extraction, that is, determining whether the behavior violates the map rules. Specifically, language behavior can be logically matched with map rules. For example, the driving direction of the black car (from south to north) conflicts with the legal direction of Lane L_01 (from north to south). If any logical fallacy exists, when any form of logical conflict is detected, a "logical fallacy / reasoning error" hard case is triggered, that is, the third check outputs a third check anomaly signal. This type of signal is crucial for training the model's causal reasoning and common sense judgment capabilities.

[0205] The planning-prediction consistency check (dimensional four check) can perform trajectory collision detection and spatiotemporal conflict calculation. It can project the vehicle's candidate planned trajectories onto the spatiotemporal grid occupancy predicted by the occupancy flow head for the next N seconds. It can acquire multiple candidate trajectories output by the behavior diffusion head H4, and perform spatiotemporal overlap calculation between the vehicle voxels of each trajectory and the future occupancy predictions output by the occupancy flow head H3 to obtain the Collision Risk Probability (CRP). For example, it can calculate whether there is any intersection between the voxels occupied by the vehicle and the voxels occupied by other dynamic / static objects at any future time T. It also calculates the comprehensive collision risk probability for each trajectory.

[0206] If the collision risk probability of all trajectories exceeds the safety threshold, it is considered a high-risk scenario. If the collision risk probabilities of multiple high-scoring trajectories differ significantly, then decision oscillation occurs. In high-risk scenarios or decision oscillation situations, the fourth dimension verification outputs a fourth verification anomaly signal.

[0207] If a verification anomaly signal is triggered in any of the four dimensions, it can be identified as a candidate difficult case. The triggering dimension label is recorded. The triggering dimension label can be a single label or multiple labels.

[0208] When multiple dimension validation results are triggered simultaneously (such as dimension one and dimension three), the difficulty level is increased, indicating that the conflict has directly affected security.

[0209] When identifying a difficult case corresponding to multimodal data, an indication message indicating that the data is a candidate difficult case and a trigger dimension label can be output.

[0210] The aforementioned scheme achieves systematic consistency verification between perception results, semantic descriptions, planned trajectories, and static map priors by cross-validating multi-dimensional scene description information and map tile information. Since the verification covers spatial comparison of physical entities and maps, referential comparison of physical entities and semantic text, logical comparison of semantic text and map rules, and risk comparison of planned trajectories and future occupancy, this scheme can simultaneously detect anomalies from multiple levels, including geometric location, semantic reference, traffic rules, and dynamic safety. Any inconsistency or conflict triggered in any dimension marks the current scene as a candidate difficult example, thus overcoming to some extent the limitation of single-dimensional discrimination in easily missing deep semantic conflicts (such as perception-map contradictions, language illusions, logical fallacies, or high-risk planning), improving the coverage and accuracy of difficult example mining. This scheme upgrades the mining method from single-dimensional or label matching to a three-in-one cross-modal logical consistency verification of "map-perception-language," thereby enabling the discovery of deep semantic-level difficult examples such as "model illusions" and "map inference errors."

[0211] In some related technologies, difficult examples are identified, but they are merely labeled without explaining "why" they are difficult examples or "how to avoid them next time." This results in low-value feedback signals and unclear directions for model optimization. Furthermore, traditional data augmentation or difficult example generation methods lack an understanding of the causal logic of the scenario, resulting in fragmented and physically inconsistent data that fails to effectively improve the model's ability to handle complex, long-tail problems in the real world. Figure 8 The embodiment shown establishes a retrieval-enhanced generation-driven, chain-thinking reasoning-based interpretable difficult example judgment system, and links it with a four-dimensional world model to form a complete interpretable closed loop from semantic mining to causal generation to model optimization.

[0212] Please refer to Figure 8 , Figure 8 A flowchart illustrating the map information-based data processing method provided in this disclosure. Figure 4 This method, in addition to including... Figure 6 In addition to steps S801-S808 that are the same as or similar to steps S601-S608 shown, the following steps are also included:

[0213] S809: In response to determining the candidate difficult cases corresponding to the multimodal data, retrieve the associated historical difficult cases from the map memory based on the map tile information.

[0214] After obtaining the information of candidate difficult cases corresponding to the multimodal data in step S808, the data processing device can acquire the current map tile identifier and related information of the candidate difficult cases. The related information of the candidate difficult cases may include, for example, difficulty triggering dimension information, image features, point cloud features, etc. The data processing device can use the map tile identifier as the primary index to perform a retrieval in a pre-built map memory.

[0215] The process of storing map tile information into the map memory can be as follows: If a new map tile and multimodal data are provided, and a four-dimensional raster occupancy information is generated based on the map tile and multimodal data, if it is determined that the multimodal data is a hard case, or if the map tile is not stored in the map memory, then a structured storage entry corresponding to the multimodal data is generated, and the structured storage entry is stored in the map memory.

[0216] The map memory can include multiple entries, each keyed by a map tile identifier, with a value including a multimodal scene context package. This multimodal scene context package includes a scene summary list and statistics. The scene summary list can include a 4D grid occupancy snapshot of the historical scene (4d_occ_snapshot, a fast-loading, spatiotemporally compressed thought grid occupancy), semantic description text vlm_desc (e.g., a structured semantic description of the scene in JSON format), hard case type hard_case_type, reference inference chain (if it is a hard case, recording the dimension in which it was triggered (e.g., "map-perc_conflict") and the inference chain text ultimately generated by the preset model), and a coping strategy expert_strategy (which can be a summary of the driver's actions (e.g., "slowly stop and wait for pedestrians to cross")), or a safe driving trajectory from a safe driver or simulation system. For non-hard cases, correct driving behaviors can be recorded.

[0217] The aggregates statistics for this map tile include the distribution of difficult cases and the frequency of physical entities.

[0218] During retrieval, the data processing device can first perform an exact match in the memory based on the map tile identifier of the current scene: if the tile identifier exists in the memory, then all historical scene summaries stored under that tile are extracted to form an initial candidate set. For example, when a hard case signal is triggered in real-time inference, the current vehicle's location identifier will be converted into a map tile identifier. Using this map tile identifier, a hash lookup with constant time complexity (O(1) complexity) is performed to directly extract all scene context information under that tile from the memory. It is understandable that, in addition to direct retrieval based on map tile identifiers, graph-based retrieval can also be used, using the topological relationships between scenes (such as upstream intersections and downstream road segments) as retrieval paths to retrieve a wider range of related scenes.

[0219] If the current map tile identifier does not exist in the memory, or the number of historical scenes stored under this map tile is less than a preset threshold (e.g., less than 3), the data processing device can initiate a topological neighbor retrieval. Specifically, it can utilize the graph structure of the high-precision map to perform a 1-hop or 2-hop breadth-first search along the road topological connections (upstream / downstream) starting from the current tile to retrieve historical scenes of adjacent map tiles. All retrieval results can be merged into an initial candidate set. After obtaining the initial candidate set, the data processing device can perform lightweight semantic filtering.

[0220] The data processing device can acquire the preliminary descriptive text of the current candidate difficult example (e.g., trigger dimension and difference description), and vectorize it with the semantic descriptive text of each historical scene in the initial candidate set. A pre-trained lightweight text matching model (e.g., a Siamese network based on BERT) is used to calculate the cosine similarity between the two. The data processing device can sort the historical scenes from high to low similarity, selecting the top K historical scenes with the highest similarity and type matching (e.g., both related to "intersection illusion") as the final associated historical difficult examples (e.g., K=3). If the initial candidate set is empty or all similarities are below a preset threshold, an empty result is returned or a default example is used.

[0221] Through the above retrieval process, the data processing device can obtain the most similar historical difficult examples in terms of geographical location and semantic content to the current candidate difficult examples, which can be used to construct chain-like thinking prompts in the future.

[0222] S810: Write the information of related historical difficult cases, the cross-validation information of candidate difficult cases, and the thought chain used to guide the model to perform interpretable reasoning into the prompt template to obtain prompt information; the information of related historical difficult cases includes the corresponding coping strategies.

[0223] The data processing device can acquire the set of associated historical difficulty examples obtained in step S809 above. Each structured entry of an associated historical difficulty example includes a four-dimensional grid occupancy snapshot of the historical scene, semantic description text, difficulty example type, reference inference chain, and corresponding coping strategy. At the same time, the data processing device can acquire the cross-validation information of the current candidate difficulty examples. This information includes the discrimination dimension that triggers the difficulty example (such as map-perception conflict, perception-language illusion, map-language logic fallacy, or planning-prediction high risk) and difference measures (such as location deviation, intersection-union ratio difference, and logical violation description).

[0224] The data processing device can construct a prompt message to guide a pre-defined model in generating interpretable inference chains for difficult examples. The prompt message uses a structured template and includes at least the following four parts:

[0225] The first part describes the role setting of the data processing device. The data processing device can be programmed with an instruction requiring the pre-defined model to act as an expert in autonomous driving scenario analysis, performing rigorous step-by-step reasoning based on provided visual and spatial occupancy information, and adhering to a few-sample reasoning style. For example, [SYSTEM_ROLE]: "You are a top expert in autonomous driving scenario analysis. You must conduct a secondary review of an initial difficult-case alert based on the provided visual and spatial occupancy evidence. Please strictly follow the principles of 'example learning' and 'step-by-step reasoning' to ultimately provide a structured conclusion."

[0226] The second part is the few-shot example embeddings [FEW-SHOT_EXAMPLES]. The data processing device can write at least one instance from the retrieved associated historical difficult cases as a few-shot example into the prompt words. For each example, the data processing device sequentially writes the scene description, trigger dimension, reference inference chain (i.e., the step-by-step analysis process), and the final coping strategy adopted for that historical difficult case. This part is not a handwritten example, but is automatically generated directly from the Top-K analog samples retrieved from the memory. For example, the format of the examples is uniformly: "Historical Case X: Scene Description, ...; Reference Inference Chain (Explainable Inference Chain), ...; Coping Strategy, ...". These examples are used to guide the model to learn how to perform analogical reasoning. The above scene description can be, for example, "In the complex traffic of map tile [intersection_045], the model initially detected a 'child suddenly rushing out from behind a bus', but the 3D occupancy display does not show this target." A possible inference chain is: "After a detailed analysis of the occupancy of the four-dimensional raster, it was discovered that the object is a red fire hydrant partially obscured by a bus on the roadside. In the VLM's 2D image understanding, due to the similarity of the background texture and the contextual guidance from the occluding object, its shape is incorrectly associated and completed as a 'child in red clothes.' This is a typical semantic illusion caused by occlusion and low-quality textures." A possible countermeasure is: "Improve the VLM's uncertainty estimation of occluded objects."

[0227] The third part involves injecting information about the current candidate difficult case [CURRENT_CASE]. The data processing device converts the cross-validation information of the current candidate difficult case into a natural language description, including the triggering dimension, conflict type, difference value, relevant spatiotemporal coordinates, etc., and writes it into prompt words. Simultaneously, the data processing device can selectively write map tile identifiers for the current scene, key voxel states of the four-dimensional occupancy grid, or local region descriptions of the panoramic image, for the model to examine as evidence. For example, the information corresponding to the current candidate difficult case is as follows: "Map tile ID: [Tunnel_012]", "Dimension of initial alarm trigger: Perception-language inconsistency (objects not detected by physical entity perception information appear in the scene semantic description text)", "Suspicious description fragment in the scene semantic description text: 'A moving humanoid shadow appears on the tunnel wall'", "Four-dimensional raster occupancy evidence for the corresponding area: Loading for you...".

[0228] Part Four, the thought chain task instruction [TASK_AND_QUESTION]. The data processing device writes a clear, step-by-step inference task instruction, mandating that the model perform logical analysis in the following order:

[0229] The first step is evidence review: analyze the provided four-dimensional grid occupancy and image data to confirm whether there is a physical entity or space occupant.

[0230] The second step is to generate and verify causal hypotheses: propose at least two possible causal hypotheses (such as sensor noise, outdated maps, environmental interference, or model defects), and use spatiotemporal information to verify the rationality of each hypothesis.

[0231] The third step is analogy and risk assessment: referring to the historical difficult cases in the above few sample examples, assess the similarities and differences between the current scenario and the historical difficult cases, determine whether the current anomaly belongs to the same type of problem (e.g., whether it belongs to the same type of visual illusion or hallucination), and assess its impact on driving safety.

[0232] The fourth step is the final judgment: Based on the above reasoning, a structured judgment conclusion is output, including whether it is a real difficult case (e.g., whether it is a real anomaly or a known environmental interference), the type of difficult case, the complete reasoning chain text (e.g., a detailed text of the step-by-step reasoning process), the uncertainty estimate, and the acceptable coping strategies.

[0233] Specifically, the above-mentioned thought chain task instructions include the following: "Please perform rigorous chain reasoning according to the following steps: 1. Evidence Examination: Carefully analyze the provided four-dimensional grid occupancy and panoramic image sequence. In the tunnel wall area described by VLM, is there an independent, dynamic occupant with a clear three-dimensional outline? 2. Causal Hypothesis Generation and Verification: If there is no independent three-dimensional object, combining the environmental characteristics of 'Tunnel_012' (usually dim, with directional light sources, and a large number of dynamic shadows), propose at least two physical hypotheses about the cause of the shadow (e.g., is it a moving shadow cast by a car's headlights? Is it a stain on the tunnel wall?), and verify them using spatiotemporal occupancy information. 3. Analogy and Risk Assessment: Referring to the above historical cases, assess the risk of the same type of 'visual illusion / hallucination' occurring in the current scenario. Will the current situation affect safe driving? 4. Final Judgment: Based on the above reasoning, output a structured JSON object containing:

[0234] -is_confirmed_anomaly(boolean): Is this a genuine anomaly or a known environmental disturbance?

[0235] -anomaly_type(string): The final classification of difficult cases.

[0236] - reasoning_chain (list of strings): Detailed text of the step-by-step reasoning process.

[0237] -uncertainty_estimate(float, 0-1): An estimate of the uncertainty of the final decision.

[0238] -recommended_action (string): Recommended action for this scenario.

[0239] S811: Input at least one of the second vector, four-dimensional grid occupancy information and text instruction, along with the prompt information, into the preset model, and the preset model outputs the judgment information.

[0240] The pre-defined model can be, for example, a visual-language-action model. This model is built upon a pre-trained language model backbone and extended with a visual encoder, a spatial projector, and four dedicated output heads (text head, 3D occupancy head, occupancy stream head, and behavior diffusion head). During the input preparation phase, the data processing device uses the second vector (T_vis-lang) as the word sequence of the visual modality, flattens or samples the key voxel features of the current and historical frames in the 4D raster occupancy information as auxiliary input for the spatiotemporal modality, and uses the text instruction vector (T_text) (if present) as the word sequence of the text modality.

[0241] The data processing device concatenates these inputs with the text content in the prompt message `system_prompt` in a preset order to form a unified multimodal input sequence. After receiving this input sequence, the visual language action model performs deep fusion of the multimodal information in its internal Transformer layer using self-attention and cross-attention mechanisms. Guided by the context, the visual language action model performs a secondary inference task oriented towards a specific difficulty example.

[0242] First, for conflict information in the cross-validation results, the corresponding voxel semantic labels and occupancy probabilities can be found in the four-dimensional raster occupancy information based on the location of the conflict information. Then, visual features near the location can be found to verify whether the conflict information matches the actual situation.

[0243] Then, causal hypotheses are generated and validated using relevant historical difficult examples; finally, a comprehensive evaluation is performed and decision information is output. The model generates structured text output in an autoregressive manner through its text header (H1). The decision information can be presented in JSON format and includes one or more of the following: difficult example decision result (s_confirmed_anomaly), difficult example type (anomaly_type), explainable reasoning chain (reasoning_chain), uncertainty estimate (uncertainty_estimate), and recommended action (recommended_action).

[0244] The difficulty example determination result can be a Boolean value or an enumeration value, indicating whether the current candidate difficulty example is confirmed as a real anomaly or whether manual review is required. The difficulty example type can be a string describing the specific category of the difficulty example, such as "visual illusion," "perception-map position offset," "map-language logic reversal," "planning-predicting collision risk," or "environmental interference." The explainable inference chain can be a string or a list of strings, containing the step-by-step inference process generated by the model according to the thought chain steps, such as "Step 1: Check that no independent 3D object is found in the 4D grid area near coordinates (10,5,0); Step 2: Rule out sensor malfunction, assuming it is a dynamic shadow cast by vehicle headlights; Step 3: Compare with the historical case 'Tunnel_012', confirming it as a visual illusion; Step 4: Determine it as a non-real difficulty example, and suggest ignoring it." The response strategy can be a string or a structured suggestion to guide subsequent processing, such as "Supplement training data for shadow interference in this scenario" or "Update the stop line position at this intersection in the map." For example, a determination message might look like this:

[0245] F_final={is_confirmed_anomaly,anomaly_type,reasoning_chain,uncertainty_estimate,unertainty_estimate, recommended_action}.

[0246] The above hard case determination results indicate whether the above multimodal data corresponds to a hard case. The hard case type can correspond to the branches of the above cross-validation, for example, hard case types include map perception inconsistency, perception language inconsistency, map language inconsistency, and planning and prediction inconsistency, etc.

[0247] The data processing device parses the above JSON format output, extracts each field, and provides complete judgment information.

[0248] The existence of reasoning chains allows data users to clearly understand the model's decision-making process, enabling them to make targeted improvements to the model architecture or supplement training data. For example, if multiple reasoning chains all point to "misjudging dynamic shadows," then such data should be specifically constructed in the training set.

[0249] Furthermore, the aforementioned data processing equipment can identify multimodal data as difficult examples based on the judgment results, and store the structured information corresponding to the multimodal data into the map memory to enrich the map memory. It can also be used to generate generative extended difficult examples based on the aforementioned difficult examples.

[0250] Figure 9 A flowchart illustrating the process of determining a candidate difficult case, as shown below. Figure 9As shown, when the current multimodal data corresponds to a candidate difficult case, the process of difficult case judgment and interpretation can be triggered. Specifically, the data processing device can obtain the triggering dimension information of the aforementioned candidate difficult cases, the current map tile, the current four-dimensional raster occupancy snapshot, and visual features (such as the second vector). Then, using the current map tile identifier as the key, a hash lookup mechanism is used to query the map memory database; if the historical scenes (associated historical difficult cases) stored under the current tile are insufficient, the search can be optionally expanded to spatially adjacent tiles. The map memory database uses key-value storage, where the key is the map tile identifier and the value is a structured scene package. Each scene package contains a historical four-dimensional raster occupancy snapshot, semantic description, enumeration of common difficult case types, and security response strategy text. The K most similar historical difficult cases to the current scene are retrieved from the memory database as analogy samples. Then, the key information of the triggering signal of the current candidate difficult case is extracted, and the scene description and response strategy from the retrieved historical analogy samples are selected and filled into a pre-built prompt template. Next, the filled-in prompts are injected into the pre-defined model as context (including few-sample examples and analogical reasoning). Additionally, the second vector corresponding to the current four-dimensional grid snapshot and visual features is input into the model, which then performs secondary reasoning: a step-by-step logical analysis following a chain-like thought process, outputting structured intermediate results. Finally, the model determines whether the example is a genuine hard case and generates suggested coping strategies. The entire process outputs complete judgment information, including the judgment result of whether it is a genuine hard case, the type of hard case, the reasoning chain, uncertainty estimation, and coping strategies, thus automating the process from candidate hard cases to interpretable judgment results.

[0251] The above-described scheme retrieves relevant historical difficult examples and inputs this information, along with current cross-validation information and thought chain hints, into a pre-defined model. The output includes determination information for candidate difficult examples and enhanced interpretability of the determination. Because relevant historical difficult examples provide validated reasoning information and coping strategies, the thought chain guides the model through step-by-step structured analysis. This results in the pre-defined model output containing not only the difficult example determination results but also progressive logical reasoning text and uncertainty estimates, thus overcoming, to some extent, the black-box nature and lack of interpretability inherent in traditional difficult example mining. Furthermore, the reuse of historical coping strategies provides a direct operational basis for subsequent data correction or model optimization, improving the automation level of the difficult example processing workflow.

[0252] Please refer to some embodiments, where the method further includes the following steps:

[0253] First, in response to the judgment information indicating the difficult cases corresponding to the multimodal data, causal data of the difficult cases is constructed based on the interpretable reasoning chain and type corresponding to the difficult cases;

[0254] Second, generative models are used to generate extended difficult example data based on difficult example causal data; the extended difficult examples and difficult examples are used to incrementally train the preset model.

[0255] In some implementations, the above-mentioned construction of causal data for difficult cases based on the interpretable reasoning chain and type corresponding to the difficult case includes: extracting causal elements from the interpretable reasoning chain and type, and constructing causal tuples based on the causal elements;

[0256] Convert causal tuples into natural language descriptions to obtain difficult-example causal data;

[0257] The above-mentioned use of generative models to generate extended difficult example data based on difficult example causal data includes:

[0258] Encode the causal data of difficult cases to obtain a condition vector;

[0259] The condition vector is input into the generative model, which then generates extended hard example data based on the condition vector.

[0260] The data processing device can construct causal data for difficult examples based on the interpretable inference chain and the type of difficult example. Specifically, the data processing device can employ a lightweight natural language processing module (such as a rule-based information extractor or a finely tuned small language model) to extract the following five causal elements from the inference chain text (reasoning_chain) and the difficult example type (anomaly_type): Place (i.e., the road type or map region where the scene is located, such as "tunnel"), Core Object (i.e., the target that is misperceived or described, such as "moving shadow"), Confuser (i.e., the environmental factor that causes the error, such as "dynamic shadow of car headlights"), Error Mechanism (i.e., the root cause of the model's error, such as "mis-associated visual texture"), and Result (i.e., the final anomaly, such as "reporting a false dynamic target"). These five elements constitute a structured causal tuple for difficult examples. Taking "tunnel illusion" as an example: the causal tuple is as follows: T_causal=(place: "tunnel", Core_Object: "moving shadow", Confuser: "vehicle headlight dynamic shadow / tunnel wall texture", Error_Mechanism: "visual illusion / texture misassociation", Result: "VLM reports false dynamic target").

[0261] The data processing device can convert causal tuples into natural language descriptions (e.g., "In a dimly lit tunnel environment, the dynamic projection of vehicle headlights creates a fast-moving shadow on the tunnel wall, leading to its misidentification as a moving pedestrian"), and encode this statement into a fixed-dimensional conditional vector (C_semantic) using a pre-trained text encoder, serving as part of the hard example causal data. The hard example causal data also includes map tile identifiers of the original scene, a four-dimensional occupancy snapshot that triggered the hard example, and timestamp indexes of the original perceptual data.

[0262] Data processing devices can input the conditional vector (C_semantic) from the hard example causal data along with the static background of the current scene (e.g., an occupancy grid without dynamic objects rendered from map tiles) into a generative model (e.g., a generative four-dimensional world model). This generative model can be a conditional diffusion model or a conditional variational autoencoder. Guided by the conditional vector, the generative model generates a controlled and diverse set of extended hard example data that satisfies the same causal logic through iterative denoising or latent variable sampling processes. For example, within the framework of a 3D static scene of a "tunnel," the generative model begins predicting and rendering future four-dimensional grid occupancy based on the semantic instructions of "dynamic lighting" and "moving shadows." Internally, the generative model simulates the propagation of light and the projection of shadows, generating a series of physically reasonable spatiotemporal occupancy sequences constrained by the map. The core feature of these sequences is the presence of a humanoid moving occupant on the tunnel wall caused by dynamic lighting.

[0263] The aforementioned diverse generation methods, such as adjusting the guiding strength of the conditional vector or adding different noise to the diffusion process, allow generative models to generate multiple extended hard examples that satisfy the same causal logic but are distinct, such as "different movement speeds of shadows," "different light intensities inside the tunnel," and "different driving behaviors of the main vehicle." Each extended hard example contains a four-dimensional occupancy raster sequence within a continuous time window, along with corresponding semantic descriptions, causal labels, and coping strategies.

[0264] The data processing device can merge the expanded hard example data with the original hard example data (i.e., the current multimodal data identified as hard examples) to form an enhanced fine-tuned dataset. Each sample in this dataset contains input data (such as image sequences, point cloud sequences, map tiles) and corresponding target outputs (such as corrected 3D occupancy ground truth, correct semantic descriptions, and safe planned trajectories), along with interpretable inference chain text as auxiliary supervision signals. The data processing device can then use this dataset to incrementally train a pre-defined model, such as a multimodal visual encoder and backbone network, using methods such as instruction fine-tuning or preference alignment, to better avoid similar hard examples in subsequent processing. After training, the pre-defined model deploys the updated model parameters to an online inference environment, thus forming a closed-loop iterative process from hard example discovery and causal generation to model optimization.

[0265] Figure 10 A flowchart illustrating the process of generating an extended hard case, such as... Figure 10 As shown, after obtaining the original data confirmed as genuine hard cases with accompanying inference chains, the data processing device enters the semantic condition extraction and encoding stage: semantic condition tuples are extracted from the inference chains and hard case information, containing location, event, causal logic, and phenomenon descriptions. These semantic condition tuples are then converted into condition vectors by a condition encoder. Following this, the device enters the semantic condition-driven hard case generation stage. In this stage, the data processing device can inject the condition vectors into a generative model (such as a diffusion model or an autoregressive model). Guided by the condition vectors, this model generates diverse future occupancy sequences and evaluates and filters out extended hard case data that conforms to causal logic, while retaining the original genuine hard cases and their inference chains. These generated extended hard cases, along with the original hard cases, enter the data standardization process and are packaged into a fine-tuning dataset. Subsequently, this dataset is used to fine-tune or align preferences in a pre-defined model (e.g., a backbone network) to update model parameters and improve model capabilities. The entire process forms a closed-loop iteration, enabling the model to continuously learn from the mined and generated hard cases and continuously optimize its performance.

[0266] Figure 11 A flowchart illustrating the process of hard example discovery and incremental model training, as follows: Figure 11 As shown, the data processing device can receive the following main input data: map tile information, scene text description, and four-dimensional raster occupancy. The scene text description can be the output of the occupancy prediction head H1. The scene text description provides a textual description of the current environment. The map tile information provides basic geographic and environmental data. Four-dimensional raster occupancy typically refers to the three-dimensional spatial occupancy including the time dimension, used to perceive dynamic obstacles. The data processing device can cross-validate and evaluate the input data from four dimensions to determine if any anomalies exist.

[0267] Dimension 1 is map perception consistency verification, which mainly compares map tile information with 4D raster occupancy to determine if there are discrepancies / conflicts. If so, a difficult case signal is triggered; otherwise, it is ignored.

[0268] Dimension two involves perceptual language consistency verification, primarily cross-checking the scene text description and the occupancy of the four-dimensional grid. Based on the cross-check results, it determines whether there are illusions / omissions (i.e., perceiving non-existent objects or omitting existing objects). If so, a hard case signal is triggered; otherwise, it is ignored.

[0269] Dimension three is map language consistency verification, which mainly performs logical verification on map tile information and scene text descriptions. The results of this logical verification determine whether any logical fallacies exist. If so, a hard case signal is triggered; otherwise, it is ignored.

[0270] Dimension four is the planning safety assessment, which mainly relies on map tile information and scene text descriptions to conduct risk assessment. Specifically, it determines the trajectory risk assessment value based on the map tile information and scene text descriptions, and then checks whether the trajectory risk assessment value exceeds a threshold. If it does, a difficult case signal is triggered; otherwise, it is ignored.

[0271] If the data processing device detects a difficult case signal triggered in any of the above dimensions, it can retrieve associated historical difficult cases from the map memory using map tile information. Based on these associated historical difficult cases and descriptions of candidate difficult cases, it constructs a prompt message. The associated historical difficult cases include response strategies. The data processing device can input the current scene's visual characteristics, 4D grid occupancy, map features, and the constructed prompt message into a preset model. The preset model performs secondary reasoning and outputs judgment information. This judgment information includes one or more of the following: difficult case judgment result, difficult case type, textual description of an interpretable reasoning chain, and response strategy.

[0272] If the hard example determination result indicates that the input data corresponds to a hard example, the data processing device can extract semantic condition tuples from the text description and type of the interpretable inference chain. A semantic vector is generated based on the semantic condition elements, and this semantic vector is used as the semantic condition input to the generative model. Based on the semantic condition, the generative model generates extended hard example data. The data processing device can package the extended hard example data generated in the above process with the hard examples corresponding to the above determination information into an incremental dataset. The incremental dataset is used to incrementally train the preset model, updating the backbone network of the preset model.

[0273] The aforementioned scheme, by constructing causal data based on the interpretable reasoning chains and types of difficult examples, transforms abstract error causes into structured conditions. This enables generative models to generate diverse extended difficult example data according to causal logic, rather than random parameter perturbations. Furthermore, the extended difficult examples, along with the original difficult examples, are used together for incremental training of the pre-set model, achieving targeted reinforcement of the model's weak points. This scheme overcomes, to some extent, the problems of traditional data generation methods lacking causal explanations and struggling to specifically repair model defects, thereby improving the utilization efficiency of difficult example data and the targeting of model iteration, forming a closed-loop mechanism from mining and generation to optimization. This scheme upgrades from "data mining" to "semantic difficult example mining," using semantics as a condition to drive the generation of data from the world model. This ensures that the generated data possesses genuine "difficult example" causal logic, rather than simple parameter deformation, and can more effectively feed back into model training, substantially improving the safety and intelligence of autonomous driving systems.

[0274] Please refer to Figure 12 , Figure 12 A schematic flowchart of a map-based data processing method provided in this application embodiment includes the following steps:

[0275] S1201: Obtain four-dimensional grid occupancy information, image features, and map tile information; encode the four-dimensional grid occupancy information and image features to obtain the first vector;

[0276] S1202: Map the first vector to the language representation space to obtain the second vector;

[0277] S1203: Input the second vector and map features enhanced by map knowledge into the backbone network of the preset model, and output multi-dimensional scene description information from the backbone network;

[0278] S1204: Perform cross-validation based on multi-dimensional scene description information and map tile information;

[0279] S1205: Based on the cross-validation results, the scenario corresponding to the multimodal data is determined as a candidate difficult case.

[0280] The four-dimensional grid occupancy information is generated based on the first perceptual feature under the constraint of map tile information, using map tile information as a geometric prior. The first perceptual feature is obtained by deep fusion of image features and point cloud features. The image features and point cloud features are obtained by feature extraction from multimodal data collected by the vehicle.

[0281] For specific implementation methods, please refer to Figure 1 , Figure 4 , Figure 6 and Figure 8 The relevant descriptions of the embodiments are not repeated here.

[0282] The aforementioned scheme, because the four-dimensional raster occupancy information is generated based on the first perceptual feature of deep fusion of image and point cloud under the prior geometric constraints of map tiles, naturally aligns the occupancy information with the map coordinate system, providing a spatially consistent benchmark for subsequent cross-modal verification. Encoding the four-dimensional raster occupancy and image features together and mapping them to the language space, then inputting them into the backbone network along with map knowledge-enhanced vector features, achieves multi-level fusion of perception, occupancy, map, and semantics. Furthermore, cross-validation based on multi-dimensional scene descriptions and map vector data can automatically detect inconsistencies between perception and map, and between language and map, from multiple dimensions such as geometric location, semantic logic, and topological rules. This improves the comprehensiveness and accuracy of candidate hard case mining to a certain extent and reduces the risk of missed and false detections caused by inconsistent coordinate systems or fragmented information.

[0283] Please refer to Figure 13 , Figure 13 A schematic flowchart of a map-based data processing method provided in this application embodiment includes the following steps:

[0284] S1301: In response to determining the candidate hard cases corresponding to the multimodal data collected by the vehicle, retrieve the associated historical hard cases from the map memory based on the map tile information; the candidate hard cases are determined by cross-validation based on multidimensional scene description information and map tile information; the multidimensional scene description information is determined based on the second vector and map features based on map knowledge enhancement.

[0285] S1302: Write the information related to historical difficult cases, the information of cross-validation, and the thought chain used to guide the model to perform interpretable logical reasoning into the prompt template to obtain the prompt information; the information related to historical difficult cases includes the corresponding coping strategies; the prompt information also includes the thought chain.

[0286] S1303: Input at least one of the second vector, four-dimensional grid occupancy information, and the text command corresponding to the vehicle, along with the prompt information, into a preset model. The preset model then outputs the judgment information for candidate difficult cases. The second vector is obtained by mapping the first vector to the language representation space. The first vector is obtained by encoding the four-dimensional grid occupancy information and image features. The four-dimensional grid occupancy information is generated based on the first perceptual features under the constraint of map tile information, using map tile information as a geometric prior. The first perceptual features are obtained by deep fusion of image features and point cloud features. The image features and point cloud features are obtained by feature extraction from multimodal data. For specific implementation methods, please refer to... Figure 1 , Figure 4 , Figure 6 and Figure 8 The relevant descriptions of the embodiments are not repeated here.

[0287] The above scheme retrieves relevant historical difficult examples from a map tile retrieval memory and writes them, along with cross-validation information, into a prompt template containing a thought chain. This template, along with visual and instructional information, is then input into the model backbone, enabling analogical reasoning based on historical experience and interpretable difficult example identification. Because relevant historical difficult examples provide validated reasoning paradigms and coping strategies, the thought chain guides the model to analyze logically, resulting in output judgment information that includes progressive reasoning text, uncertainty estimates, and coping suggestions. This enhances the interpretability, reliability, and knowledge reuse efficiency of difficult example identification to a certain extent, overcoming the shortcomings of traditional difficult example mining, such as its black-box nature and lack of causal analysis.

[0288] Figure 14 A schematic diagram of the structure of the map information-based data processing device provided in this application, such as... Figure 4 As shown, the map information-based data processing device 140 provided in this embodiment includes:

[0289] The first acquisition unit 1401 is used to acquire multimodal data and map tile information corresponding to the vehicle's pose; the multimodal data includes image data and point cloud data;

[0290] The first feature extraction unit 1402 is used to extract features from multimodal data and deeply fuse the obtained image features and point cloud features to obtain the first perceptual features.

[0291] The first generation unit 1403 is used to generate four-dimensional grid occupancy information aligned with the map information based on the map tile information and the first perceptual features under the constraint of the map tile information, using the map tile information as a geometric prior.

[0292] In some embodiments, the first generating unit 1403 is further configured to:

[0293] Encode map tile information to generate map features;

[0294] Modulate the first perceived features using map features;

[0295] Based on the modulated first sensing features, four-dimensional grid occupancy information is generated.

[0296] In some implementations, the first generation unit 1403 is further configured to: for each location in the first perceived feature, determine the map vector element closest to that location;

[0297] Transformation parameters are generated based on the local map features corresponding to the map vector elements and the second perceptual features corresponding to the location.

[0298] The transformation parameters are applied to the second sensing feature to obtain the modulation result of the second sensing feature;

[0299] Alternatively, it can be further used to: modulate the first perceptual features of the input to at least one layer of the decoding network in the decoder using map features; the decoder belongs to a pre-defined model;

[0300] Furthermore, based on the modulated first perceptual features, four-dimensional grid occupancy information is generated, including:

[0301] The modulated first perceptual feature is input into the occupancy prediction head of the decoder, and the occupancy prediction head outputs four-dimensional grid occupancy information.

[0302] In some implementations, the decoder is trained based on a first loss function, wherein the first loss function comprises one or more of the following sub-loss functions: occupancy-map geometry consistency loss, semantic-occupancy-projection consistency loss, and vector-direction consistency loss;

[0303] Among them, the occupancy-map geometric consistency loss is used to compare the geometric differences between the predicted four-dimensional grid occupancy information and the static occupancy grid indicated by the map tile information; the semantic-occupancy projection consistency loss is used to constrain the projection of the occupancy voxels classified as drivable areas onto the horizontal plane to be within the drivable area boundary defined by the map tile information; and the vector-direction consistency loss is used to constrain the spatial orientation of the predicted lane line voxel sequence to be consistent with the direction vector of the corresponding lane line in the map tile information.

[0304] In some implementations, each voxel grid in the four-dimensional grid occupancy information includes one or more of the following information:

[0305] The occupancy probability corresponding to a voxel;

[0306] Semantic category identifiers corresponding to voxels;

[0307] The three-dimensional motion flow vector corresponding to the voxel;

[0308] The three-dimensional spatial coordinates of a voxel raster;

[0309] The map element association identifier corresponding to the voxel.

[0310] In some embodiments, the first feature extraction unit 1402 is further configured to: project image features and point cloud features onto a top view plane;

[0311] The image features and point cloud features projected onto the top view plane are fused to obtain the first perceptual features;

[0312] Alternatively, the device 140 may further include a second generating unit 1404, the second generating unit 1404 being used for:

[0313] A multimodal visual encoder based on a pre-defined model encodes four-dimensional grid occupancy information and image features to obtain a first vector;

[0314] Mapping the first vector onto the language representation space yields the second vector;

[0315] The second vector and map features enhanced by map knowledge are input into the backbone network of the preset model, and the backbone network outputs multi-dimensional scene description information.

[0316] In some implementations, the multi-dimensional scene description information includes:

[0317] Scene semantic description text;

[0318] Perceived information of physical entities;

[0319] Four-dimensional grid occupancy prediction information;

[0320] Trajectory planning information.

[0321] In some embodiments, the second generating unit 1404 is further configured to:

[0322] The image features are divided into blocks to obtain the first block result; the first block vector is generated based on the pixel values ​​of the first block result.

[0323] The four-dimensional grid occupancy information is divided into blocks to obtain the second block result; the second block vector is generated based on the height value and channel value of the second block result;

[0324] The first block vector of multiple first block results and the second block vector of multiple second block results are concatenated, and the concatenated result is encoded to obtain the first vector;

[0325] Alternatively, it can be further used to: query the map prior knowledge base based on map features to obtain at least one corresponding map prior knowledge vector;

[0326] By associating at least one map prior knowledge vector with map features, map knowledge-enhanced map features are obtained.

[0327] Alternatively, the device 140 may further include a verification unit 1405, which is used for:

[0328] Cross-validation is performed based on multi-dimensional scene description information and map tile information;

[0329] In response to the cross-validation results meeting the preset conditions, candidate difficult cases corresponding to the multimodal data are determined.

[0330] In some implementations, the verification unit 1405 is further configured to perform one or more of the following:

[0331] The perceptual information of physical entities in the multi-dimensional scene description information is compared with the map information for consistency.

[0332] The perceptual information of physical entities is compared with the semantic description text of the scene in the multi-dimensional scene description information for consistency.

[0333] Perform a consistency comparison between the scene semantic description text and the map tile information;

[0334] The trajectory planning information in the multi-dimensional scene description information is compared with the prediction information in the four-dimensional grid occupancy information for consistency.

[0335] Alternatively, the device 140 may further include a secondary determination unit 1406, which is used for:

[0336] In response to identifying candidate difficult cases corresponding to multimodal data, associated historical difficult cases are retrieved from the map memory based on map tile information;

[0337] Information about related historical difficult cases, cross-validation information of candidate difficult cases, and the thought process used to guide the model to perform interpretable logical reasoning are written into the prompt template to obtain prompt information; the information about related historical difficult cases includes the corresponding coping strategies;

[0338] Input at least one of the second vector, four-dimensional grid occupancy information, and text instructions, along with the prompt information, into the preset model, and the preset model outputs the judgment information.

[0339] In some implementations, the determination information includes one or more of the following: the difficulty determination result, the difficulty type, the textual description of the interpretable reasoning chain, and the response strategy.

[0340] In some embodiments, the apparatus 140 further includes a second generating unit 1407, the second generating unit 1407 being used for:

[0341] In response to the judgment information indicating the difficult case corresponding to the multimodal data, causal data of the difficult case is constructed based on the interpretable reasoning chain and type corresponding to the difficult case;

[0342] Generative models are used to generate extended difficult example data based on difficult example causal data; the extended difficult examples and difficult examples are used to incrementally train the preset model.

[0343] In some implementations, the second generation unit 1407 is further configured to: extract causal elements from interpretable reasoning chains and types, and construct causal tuples based on the causal elements;

[0344] Convert causal tuples into natural language descriptions to obtain difficult-example causal data;

[0345] Generative models are used to generate extended difficult case data based on difficult case causal data, including:

[0346] Encode the causal data of difficult cases to obtain a condition vector;

[0347] The condition vector is input into the generative model, which then generates extended hard example data based on the condition vector.

[0348] The map-based data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0349] Figure 15 A schematic diagram of the structure of the map information-based data processing device provided in this application, such as... Figure 15 As shown, the device 150 includes:

[0350] The second acquisition unit 1501 is used to acquire four-dimensional grid occupancy information, image features and map tile information, and encode the four-dimensional grid occupancy information and image features to obtain a first vector.

[0351] Mapping unit 1502 is used to map the first vector to the language representation space to obtain the second vector;

[0352] The third generation unit 1503 is used to input the second vector and map features enhanced by map knowledge into the backbone network of the preset model, and the backbone network outputs multi-dimensional scene description information.

[0353] Cross-validation unit 1504 is used to perform cross-validation based on multi-dimensional scene description information and map tile information;

[0354] The determination unit 1505 is used to determine the scenario corresponding to the multimodal data as a candidate difficult case based on the cross-validation results;

[0355] Among them, the four-dimensional grid occupancy information is generated based on the first perceptual feature under the constraint of the map tile information, using map tile information as the geometric prior; the first perceptual feature is obtained by deep fusion of image features and point cloud features; the image features and point cloud features are obtained by feature extraction from multimodal data collected by the vehicle.

[0356] The map-based data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0357] Figure 16 A schematic diagram of the structure of the map information-based data processing device provided in this application, such as... Figure 16 As shown, the device 160 includes:

[0358] The retrieval unit 1601 is used to retrieve associated historical difficult examples from the map memory based on map tile information in response to determining the candidate difficult examples corresponding to the multimodal data collected by the vehicle; the candidate difficult examples are determined by cross-validation based on multidimensional scene description information and map tile information; the multidimensional scene description information is determined based on the second vector and map features based on map knowledge enhancement.

[0359] The writing unit 1602 is used to write information related to historical difficult cases, cross-validation information, and thought chains used to guide the model to perform interpretable logical reasoning into the prompt template to obtain prompt information; the information related to historical difficult cases includes corresponding coping strategies; the prompt information also includes thought chains;

[0360] The determination unit 1603 is used to input at least one of the second vector, the four-dimensional grid occupancy information and the text instruction corresponding to the vehicle, along with the prompt information, into the preset model, and the preset model outputs the determination information of the candidate difficult cases.

[0361] The second vector is obtained by mapping the first vector to the language representation space. The first vector is obtained by encoding the four-dimensional grid occupancy information and image features. The four-dimensional grid occupancy information is generated based on the first perceptual feature under the constraint of the map tile information, using map tile information as a geometric prior. The first perceptual feature is obtained by deep fusion of image features and point cloud features. The image features and point cloud features are obtained by feature extraction from multimodal data.

[0362] The map-based data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0363] Figure 17 A schematic diagram of the structure of the electronic device provided in this application. Figure 17 As shown, the electronic device 170 provided in this embodiment includes at least one processor 1701 and a memory 1702. Optionally, the device 170 further includes a communication component 1703. The processor 1701, memory 1702, and communication component 1703 are connected via a bus.

[0364] In a specific implementation, at least one processor 1701 executes computer execution instructions stored in memory 1702, causing at least one processor 1701 to perform the above-described method.

[0365] The specific implementation process of processor 1701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0366] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0367] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0368] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0369] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0370] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0371] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0372] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0373] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0374] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0375] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0376] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0377] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0378] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method based on map information, characterized in that, include: Acquire multimodal data and map tile information corresponding to the vehicle's pose; the multimodal data includes image data and point cloud data; Feature extraction is performed on the multimodal data, and the obtained image features and point cloud features are deeply fused to obtain the first perceptual features; Using the map tile information as a geometric prior, and under the constraint of the map tile information, four-dimensional grid occupancy information aligned with the map information is generated based on the first perceptual feature.

2. The method according to claim 1, characterized in that, Using the map tile information as a geometric prior, and under the constraints of the map tile information, based on the first perceptual feature, four-dimensional raster occupancy information aligned with the map information is generated, including: The map tile information is encoded to generate map features; The first perceived feature is modulated using the map features; Based on the modulated first sensing feature, the four-dimensional grid occupancy information is generated.

3. The method according to claim 2, characterized in that, The modulation of the first perceived feature using the map features includes: For each location in the first perceived feature, determine the map vector element closest to that location; Transformation parameters are generated based on the local map features corresponding to the map vector elements and the second perception features corresponding to the location. Applying the transformation parameters to the second sensing feature yields a modulation result of the second sensing feature; or, modulating the first sensing feature using the map features includes: The first perceptual feature input to at least one layer of the decoding network in the decoder is modulated using the map feature; the decoder belongs to a preset model. And, generating the four-dimensional grid occupancy information based on the modulated first perceptual features includes: The modulated first perceptual feature is input to the occupancy prediction head of the decoder, and the four-dimensional grid occupancy information is output through the occupancy prediction head.

4. The method according to claim 3, characterized in that, The decoder is trained based on the following first loss function, wherein the first loss function comprises one or more of the following sub-loss functions: occupancy-map geometry consistency loss, semantic-occupancy-projection consistency loss, and vector-direction consistency loss; The occupancy-map geometric consistency loss is used to compare the geometric differences between the predicted four-dimensional grid occupancy information and the static occupancy grid indicated by the map tile information; the semantic-occupancy projection consistency loss is used to constrain the projection of the occupancy voxels classified as drivable areas onto the horizontal plane to be within the drivable area boundary defined by the map tile information; and the vector-direction consistency loss is used to constrain the spatial orientation of the predicted lane line voxel sequence to be consistent with the direction vector of the corresponding lane line in the map tile information.

5. The method according to any one of claims 1-4, characterized in that, The step of extracting features from the multimodal data and deeply fusing the obtained image features and point cloud features to obtain the first perceptual features includes: The image features and point cloud features are projected onto the top view plane; The image features and point cloud features projected onto the top view plane are fused to obtain the first perceptual feature; or, the method further includes: A multimodal visual encoder based on a preset model encodes the four-dimensional grid occupancy information and the image features to obtain a first vector; Mapping the first vector to the language representation space yields the second vector; The second vector and map features enhanced by map knowledge are input into the backbone network of the preset model, and the backbone network outputs multi-dimensional scene description information.

6. The method according to claim 5, characterized in that, The multimodal visual encoder based on a preset model encodes the four-dimensional raster occupancy information and the image features to obtain a first vector, including: The image features are divided into blocks to obtain a first block result; a first block vector is generated based on the pixel values ​​of the first block result. The four-dimensional grid occupancy information is divided into blocks to obtain a second block result; a second block vector is generated based on the height value and channel value of the second block result; Concatenate multiple first block results into first block vectors and multiple second block results into second block vectors, encode the concatenated result to obtain the first vector; or, further include: Based on the map features, a query is performed from the map prior knowledge base to obtain at least one corresponding map prior knowledge vector; Associating the at least one map prior knowledge vector with the map features to obtain the map knowledge-enhanced map features; or, further comprising: Cross-validation is performed based on the multi-dimensional scene description information and the map tile information; In response to the cross-validation result meeting the preset conditions, the candidate difficult cases corresponding to the multimodal data are determined.

7. The method according to claim 6, characterized in that, The cross-validation based on the multi-dimensional scene description information and the map tile information includes one or more of the following: The perceptual information of physical entities in the multi-dimensional scene description information is compared with the map tile information for consistency. The perceptual information of the physical entity is compared with the semantic description text of the scene in the multi-dimensional scene description information for consistency. The semantic description text of the scene is compared with the map tile information for consistency. The trajectory planning information in the multi-dimensional scene description information is compared with the prediction information in the four-dimensional grid occupancy information for consistency. Or, it may also include: In response to determining the candidate difficult cases corresponding to the multimodal data, associated historical difficult cases are retrieved from the map memory based on the map tile information; The information of the associated historical difficult cases, the cross-validation information of the candidate difficult cases, and the thought chain used to guide the model to perform interpretable logical reasoning are written into the prompt template to obtain prompt information; the information of the associated historical difficult cases includes the corresponding coping strategies; The second vector, the four-dimensional grid occupancy information, and the text instruction are input into the preset model along with the prompt information, and the preset model outputs the determination information.

8. The method according to claim 7, characterized in that, The method further includes: In response to the determination information indicating the difficult case corresponding to the multimodal data, causal data of the difficult case is constructed according to the interpretable reasoning chain and type corresponding to the difficult case; Generative models are used to generate extended difficult example data based on the difficult example causal data; wherein, the extended difficult examples and the difficult examples are used to incrementally train the preset model.

9. The method according to claim 8, characterized in that, The step of constructing causal data for difficult examples based on the interpretable reasoning chain and type corresponding to the difficult examples includes: Extract causal elements from the interpretable reasoning chain and the type, and construct causal tuples based on the causal elements; The causal tuples are converted into natural language descriptions to obtain the difficult case causal data; The generation of extended difficult example data based on difficult example causal data using a generative model includes: The causal data of the difficult cases are encoded to obtain a condition vector; The condition vector is input into the generative model, which then generates the extended hard example data based on the condition vector.

10. A data processing method based on map information, characterized in that, include: Obtain four-dimensional grid occupancy information, image features, and map tile information; encode the four-dimensional grid occupancy information and image features to obtain a first vector; Mapping the first vector to the language representation space yields the second vector; The second vector and map features enhanced by map knowledge are input into the backbone network of the preset model, and the backbone network outputs multi-dimensional scene description information. Cross-validation is performed based on the multi-dimensional scene description information and the map tile information; Based on the cross-validation results, the scenarios corresponding to the multimodal data were identified as candidate difficult cases; The four-dimensional grid occupancy information is generated based on a first perceptual feature under the constraint of the map tile information, using the map tile information as a geometric prior. The first perception feature is obtained by deep fusion of the image features and point cloud features; the image features and point cloud features are obtained by feature extraction from the multimodal data collected by the vehicle.

11. A data processing method based on map information, characterized in that, include: In response to identifying candidate difficult cases corresponding to the multimodal data collected by the vehicle, associated historical difficult cases are retrieved from the map memory based on map tile information; The candidate difficult examples are determined by cross-validation based on multi-dimensional scene description information and map tile information; the multi-dimensional scene description information is determined based on a second vector and map features enhanced by map knowledge. The information of the associated historical difficult cases, the information of the cross-validation, and the thought chain used to guide the model to perform interpretable logical reasoning are written into the prompt template to obtain the prompt information; The information related to historical difficult cases includes corresponding coping strategies; the prompt information also includes thought processes. The second vector, the four-dimensional grid occupancy information, and the text instruction corresponding to the vehicle are input into a preset model along with the prompt information. The preset model then outputs the determination information for the candidate difficult case. The determination information includes one or more of the following: difficult case determination result, difficult case type, text description of the interpretable reasoning chain, and coping strategy. Wherein, the second vector is obtained by mapping the first vector to the language representation space, the first vector is obtained by encoding four-dimensional grid occupancy information and image features, the four-dimensional grid occupancy information is generated based on the first perceptual feature under the constraint of the map tile information, with map tile information as the geometric prior; the first perceptual feature is obtained by deep fusion of the image features and point cloud features; the image features and point cloud features are obtained by feature extraction of the multimodal data.

12. A data processing device based on map information, characterized in that, include: The first acquisition unit is used to acquire multimodal data and map tile information corresponding to the vehicle's pose; the multimodal data includes image data and point cloud data; The first feature extraction unit is used to extract features from the multimodal data and deeply fuse the obtained image features and point cloud features to obtain the first perceptual features. The first generation unit is used to generate four-dimensional grid occupancy information aligned with the map information, based on the first perceptual feature and using the map tile information as a geometric prior, under the constraint of the map tile information.

13. A data processing device based on map information, characterized in that, include: The second acquisition unit is used to acquire four-dimensional grid occupancy information, image features and map tile information, and encode the four-dimensional grid occupancy information and image features to obtain a first vector. A mapping unit is used to map the first vector to a language representation space to obtain a second vector; The third generation unit is used to input the second vector and map features enhanced based on map knowledge into the backbone network of the preset model, and the backbone network outputs multi-dimensional scene description information. A cross-validation unit is used to perform cross-validation based on the multi-dimensional scene description information and the map tile information; The determination unit is used to determine the scenario corresponding to the multimodal data as a candidate difficult case based on the cross-validation results; The four-dimensional grid occupancy information is generated based on a first perceptual feature under the constraint of the map tile information, using the map tile information as a geometric prior. The first perception feature is obtained by deep fusion of the image features and point cloud features; the image features and point cloud features are obtained by feature extraction from the multimodal data collected by the vehicle.

14. A data processing device based on map information, characterized in that, include: The retrieval unit is used to retrieve associated historical difficult cases from the map memory based on map tile information in response to determining the candidate difficult cases corresponding to the multimodal data collected by the vehicle. The candidate difficult examples are determined by cross-validation based on multi-dimensional scene description information and map tile information; the multi-dimensional scene description information is determined based on a second vector and map features enhanced by map knowledge. The writing unit is used to write the information of the associated historical difficult cases, the information of the cross-validation, and the thought chain used to guide the model to perform interpretable logical reasoning into the prompt template to obtain prompt information; The information related to historical difficult cases includes corresponding coping strategies; the prompt information also includes thought processes. The determination unit is used to input at least one of the second vector, the four-dimensional grid occupancy information, and the text instruction corresponding to the vehicle, along with the prompt information, into a preset model, and the preset model outputs the determination information of the candidate difficult case; wherein, the second vector is obtained by mapping the first vector to the language representation space, the first vector is obtained by encoding the four-dimensional grid occupancy information and image features, the four-dimensional grid occupancy information is generated based on the first perceptual feature under the constraint of the map tile information, using map tile information as a geometric prior; the first perceptual feature is obtained by deep fusion of the image features and point cloud features; the image features and point cloud features are obtained by feature extraction of the multimodal data.

15. An electronic device / computer-readable storage medium / computer program product, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-11; the computer-readable storage medium stores computer execution instructions, which, when executed by the processor, are used to implement the method as described in any one of claims 1-11; the computer program product includes a computer program that, when executed by the processor, implements the method as described in any one of claims 1-11.

Images