Method and device for dynamically identifying and monitoring odc state based on visual language model perception

Abstract

The application provides a dynamic identification and monitoring method and device for ODC state based on visual language model perception, the dynamic identification and monitoring method comprises the following steps: establishing an ODC state item screening method suitable for VLM, selecting the semantic state item most suitable for being identified by VLM from the candidate ODC state items; establishing a semantic ODC state identification and generation method, so that VLM can obtain a structured output field for a specific state item; and dynamically monitoring, scheduling and structurally updating VLM in view of resource constraints. The application converts the semantic understanding result of VLM on a complex traffic scene into structured element data that can be received, analyzed and called by an ODC monitoring system, and can effectively enhance the input integrity and boundary perception ability of the dynamic ODC monitoring system.

Description

Technical Field

[0001] This invention belongs to the field of automotive autonomous driving control technology, and in particular relates to a method and device for dynamic identification and monitoring of ODC status based on visual language model perception. Background Technology

[0002] With the continuous development of Advanced Driver Assistance Systems (ADAS) and Autonomous Driving Systems (ADS), the operating scenarios of autonomous vehicles are gradually expanding to highways, urban roads, and complex mixed traffic environments. This increased scenario openness means that autonomous driving systems must contend with long-tail scenarios such as irregularly shaped obstacles, occluded targets, and complex traffic participant behaviors, leading to unreasonable risks. This makes operating condition identification and boundary monitoring key issues in the safe development of autonomous driving systems.

[0003] Lee CW, Nayeer N, Garcia DE, et al. Identifying the operational design domain for an automated driving system through assessed risk[C] / / 2020IEEE Intelligent Vehicles Symposium (IV). IEEE, 2020: 1317-1322. This paper proposes an ODD boundary identification method for automated driving systems based on statistical data and risk tolerance. This method can express the operational range of the automated driving system in the form of a map region. However, its boundary mainly comes from offline data collection, simulation evaluation and risk statistics results. During runtime, it is more suitable for judging whether the vehicle is located in a predefined geographical area, but it is difficult to reflect the state of dynamic elements in the current road scene in real time.

[0004] Therefore, to further optimize the scope of scenarios limiting the safe operation of autonomous driving systems, Operational Design Conditions (ODCs) are used to describe the internal and external environmental conditions that allow the autonomous driving system to operate. ODCs not only cover external operating environments such as road type, weather conditions, and lighting conditions, but also internal operating conditions such as vehicle motion state, system functional state, and passenger status. Existing research and standards related to ODCs have formed relatively complete classifications and boundary expressions, but their effectiveness largely depends on whether the perception module can accurately and timely provide key ODC elements in the current scenario. Traditional autonomous driving front-end perception typically relies on explicit fixed algorithms to output structured information such as lane lines, distance, speed, and heading angle relatively well. However, for semantic ODC elements such as non-standard obstacles, abnormal behavior of traffic participants, and road surface quality, traditional perception algorithms often suffer from insufficient category coverage, limited semantic understanding capabilities, and insufficient open-world generalization ability.

[0005] To improve perception performance, Wu D, Liao MW, Zhang WT, et al. Yolop: You only look once for panoptic driving perception[J]. Machine Intelligence Research, 2022, 19(6): 550-562 proposed a multi-task network, YOLOP, for front-end visual perception in autonomous driving, which simultaneously performs three basic perception tasks: traffic target detection, drivable area segmentation, and lane line detection. This method uses a shared feature extraction network and multiple task output branches to generate perception results in parallel under the same image input, thereby improving perception efficiency. Although the multi-task shared network reduces the amount of repetitive computation under traditional multi-model deployment, its perception capability is still limited by pre-defined task branches, fixed category labels, and training data range. Moreover, when the system needs to add semantic ODD / ODC elements such as non-standard obstacles, abnormal behavior of traffic participants, and road surface quality, the traditional approach usually requires additional design of detection categories, segmentation branches, or dedicated recognition models, along with supplementary labeled data and retraining of the model, leading to a continuous increase in engineering costs and incremental computing power during system expansion. Meanwhile, the output of this method is mainly detection boxes, segmentation masks and lane line results, which are biased towards geometric location and basic category information, and lack the ability to understand complex traffic semantics, abnormal states and potential risk intentions.

[0006] Visual-Language Models (VLMs) possess the ability to jointly understand image content and language semantics, enabling the identification and description of target attributes, traffic relationships, abnormal states, and semantic risks in complex road scenarios. Therefore, introducing VLMs into autonomous driving perception modules can compensate for the shortcomings of traditional perception modules in recognizing semantic ODD / ODC elements. However, current applications of VLMs in autonomous driving are mostly concentrated on scene question answering, image description, driving explanation, or planning assistance. Their output is typically natural language text, lacking field constraints, structured expressions, and interface adaptation mechanisms for ODC elements, making it difficult to integrate with backend dynamic ODC monitoring systems. For example, the graph-based visual question answering method DriveLM proposed in the paper "Drivevlm: The convergence of autonomous driving and large vision-language models[J].arXiv preprint arXiv:2402.12289, 2024" introduces a large visual language model into the end-to-end autonomous driving scene understanding and driving reasoning process. This method breaks down the driving task into stages such as perception, prediction, planning, behavior, and trajectory, and describes the logical dependencies between different stages through graph-structured question answering. It utilizes the image understanding, language expression, and multi-step reasoning capabilities of VLM to enable the model to further infer the future state of the target, potential interaction relationships, and the safe behaviors that the vehicle should take. VLM can enhance the semantic understanding and interpretation capabilities of autonomous driving systems for complex traffic scenarios, and is particularly suitable for handling complex semantic understanding and reasoning problems. However, the output of this method mainly serves visual question answering, behavior prediction, and trajectory planning, and is still mainly based on natural language answers, behavior categories, or trajectory results. It has not established ODD / ODC element fields, value ranges, confidence thresholds, and interface formats. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] The technical problem to be solved by the present invention is to provide a method and device for dynamic identification and monitoring of ODC status based on visual language model perception. By converting the semantic understanding results of VLM on complex road traffic scenarios into structured element data that can be received, parsed and called by the ODC monitoring system, the input integrity and boundary perception capability of the dynamic ODC monitoring system are enhanced.

[0009] (II) Technical Solution

[0010] This invention discloses a dynamic identification and monitoring method for ODC status based on visual language model perception. It includes establishing a semantic ODC status item screening method applicable to the visual language model (VLM) to select the most suitable status item for VLM recognition from candidate semantic ODC status items; establishing a semantic ODC status item recognition and generation method to enable VLM to generate structured output fields that can be parsed by the ODC monitoring system for specific status items; and performing dynamic monitoring, scheduling and structured updates of VLM based on resource constraints.

[0011] Preferably, further detailed description of the present invention can be found in the following detailed embodiments.

[0012] The present invention also discloses an ODC status dynamic identification and monitoring device based on VLM perception, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to realize the aforementioned ODC status dynamic identification and monitoring method based on visual language model perception.

[0013] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned method for dynamic identification and monitoring of ODC status based on visual language model perception.

[0014] The present invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for dynamic identification and monitoring of ODC status based on visual language model perception.

[0015] (III) Beneficial Effects

[0016] The present invention provides a dynamic ODC state recognition and monitoring method based on visual language model perception. First, it establishes a semantic ODC state item filtering method applicable to the Visual Language Model (VLM) to select the most suitable state item for VLM recognition from candidate semantic ODC state items, thus avoiding the indiscriminate use of VLM for all ODC state perception tasks. Then, it establishes a semantic ODC state item recognition and generation method, enabling VLM to generate structured output fields that the ODC monitoring system can parse for specific state items. Finally, it performs dynamic monitoring scheduling and structured updates of VLM under resource constraints to address the problems of high VLM inference overhead and limited vehicle-side computing resources. In summary, this invention converts the semantic understanding results of VLM for complex road traffic scenarios into structured element data that the ODC monitoring system can receive, parse, and call, effectively enhancing the input integrity and boundary awareness capabilities of the dynamic ODC monitoring system.

[0017] Furthermore, for the task of state perception and dynamic monitoring of multi-source heterogeneous ODC elements under limited computing resources, this invention forms a dynamic identification and monitoring method for ODC state based on visual language model perception by screening applicable state items using VLM, enhancing prompt constraints through retrieval, using LoRA for lightweight fine-tuning, generating structured output fields, and dynamic monitoring scheduling based on rate-distortion loss. This method can transform semantic states that are difficult for traditional perception models to stably cover, such as road construction states, non-standard obstacle states, abnormal road states, and accident scene states, into parsable field-based results for the ODC monitoring system, enhancing the semantic input integrity of the dynamic ODC monitoring system. Simultaneously, through a mutual information-based MRMR screening mechanism, this method only includes high-risk, low-redundancy state items suitable for structured recognition within the VLM perception range, avoiding redundant replacement of mature perception results such as vehicle speed, lane lines, and target distance. Furthermore, this method dynamically allocates the VLM call frequency based on the state item risk value, risk fluctuation coefficient, target risk distortion, and computing resource budget, so that high-risk and high-fluctuation state items receive a higher call frequency, while low-risk or stable state items receive a lower call frequency, thereby improving the response timeliness and monitoring efficiency of key ODC state changes under limited vehicle-side computing resources. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below:

[0019] Figure 1 is a diagram of the overall architecture of the VLM front-end sensing system for dynamic ODC monitoring in this invention.

[0020] Figure 2 is a flowchart of the ODC status dynamic identification and monitoring method based on visual language model perception involved in the present invention;

[0021] Figure 3 is a flowchart of the ODC status item screening method applicable to VLM in this invention;

[0022] Figure 4 is a flowchart of the semantic ODC state recognition and generation method based on LoRA lightweight fine-tuning and structured Prompt constraints in this invention.

[0023] Figure 5 This is a code diagram illustrating the VLM dynamic monitoring and scheduling process based on rate-distortion loss in this invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Addressing the challenge of sensing and monitoring the state of multi-source heterogeneous ODC elements under limited computing resources, this invention aims to resolve the problems of fragmented perception results from VLM semantic understanding and inference, difficulty in accurately identifying the state of key dynamic elements, and the contradiction between high monitoring and identification costs and limited resources. Existing ODC monitoring methods can determine whether system operating conditions are approaching or exceeding design operating boundaries, but this relies on the perception module providing complete, accurate, and structured ODC state input. When semantic risks arise, traditional perception modules, due to performance limitations, prevent dynamic ODC monitoring systems with boundary judgment capabilities from fully utilizing these potential changes in operating conditions.

[0026] Compared to the "ODD boundary recognition method for autonomous driving systems based on statistical data and risk tolerance" mentioned in the background technology, this invention does not redesign the ODC boundary learning model or backend failure prediction model. Instead, it addresses the problem of insufficient front-end input by proposing a method for perceiving ODC based on VLM. Furthermore, compared to the "YOLOP multi-task network for front-end visual perception in autonomous driving" mentioned in the background technology, this invention is not limited to conventional visual perception tasks such as vehicles, lane lines, and drivable areas. Instead, it addresses semantic elements closely related to changes in ODC boundaries, such as road anomalies, non-standard obstacles, and abnormal behavior of traffic participants, improving the perception's coverage of long-tail scenarios and complex semantic risks. Additionally, compared to the "graph-based visual question answering method for autonomous driving" mentioned in the background technology, this invention does not use VLM as a decision model. Instead, it positions VLM as a perception interface before the dynamic ODC monitoring system, solving the problem that its natural language output is difficult for the backend system to directly parse and call.

[0027] Therefore, the purpose of this invention is to establish a dynamic identification and monitoring method for ODC status based on visual language model perception, without reconstructing the judgment logic of existing dynamic ODC monitoring systems. (See [link to relevant documentation]). Figure 1This method describes non-standard obstacles, road construction, accident scenes, and visual anomalies as dynamic ODC state items. These state items are the state variables required for ODC monitoring. This invention uses a Visual Language Model (VLM) with a new interface for output, thereby broadening the range of ODC elements and improving system capabilities. Instead of the primary classification elements in the traditional ODD standard, for states with high real-time requirements such as traffic conditions, digital information states, and meteorological data, which already have mature sensor or traditional model outputs, this invention does not require replacement by the Visual Language Model (VLM). Furthermore, states such as target states, vehicle motion states, and driver / passenger state information can be replaced by the Visual Language Model (VLM) to save on algorithms and optimize the structure. Based on this, the ODC state dynamic identification and monitoring method based on Visual Language Model perception disclosed in this invention mainly includes the following steps.

[0028] Step S1, as follows Figure 2 As shown, the current road scene data of the vehicle and candidate ODC status items are obtained.

[0029] Step S2: Establish a semantic ODC state item selection method applicable to the visual language model (VLM), and select the state item most suitable for VLM recognition from the candidate semantic ODC state items.

[0030] To avoid the indiscriminate application of VLM to all ODC state awareness tasks, this invention first filters candidate state items. Preferably, the problem of filtering state items suitable for VLM is modeled as a minimum redundancy maximum relevance selection problem based on mutual information. Specifically, the MRMR (Minimum Redundancy Maximum Relevance) feature selection algorithm is used to filter candidate state items. This algorithm can improve the correlation between candidate state items and ODC boundary state risk while reducing the degree of information redundancy among selected state items, thus obtaining a set of state items more suitable for VLM identification. For the specific filtering process, please refer to [link to relevant documentation]. Figure 3 :

[0031] Step S201: Construct a set of candidate state items The status items are semantic ODC status items, including road construction status, non-standard obstacle status, abnormal road status, and accident scene status, and the candidate status item set. The expression is:

[0032] (14)

[0033] in, Indicates the first There are n candidate state items, including road construction status, non-standard obstacle status, and abnormal road status; n is the number of candidate state items.

[0034] Step S202: Obtain a set of road scene samples with ODC boundary state labels. The samples include road scene images, observations of candidate state items, and ODC boundary state labels.

[0035] Specifically, let's set express The corresponding observed random variable, Indicates the ODC boundary status label. Depending on the specific dynamic ODC monitoring system, it can be defined as a discrete state such as "normal", "close to the boundary" or "beyond the boundary".

[0036] Step S203: Calculate the mutual information between the candidate state item and the ODC boundary state label to obtain the risk correlation between the two.

[0037] For any candidate state item Its boundary status label with ODC Risk correlation between Mutual information can be represented as:

[0038] (15)

[0039] in, express Corresponding observed random variable ODC Boundary Status Label Mutual information between them; the larger the value, the stronger the mutual information. The greater the contribution to distinguishing whether the current operating conditions are normal, close to the boundary, or beyond the boundary, the more suitable it is as a VLM identification object.

[0040] The expression for mutual information is:

[0041] (16)

[0042] in, Indicates the observed values ​​of the state term. With boundary state labels The joint probability distribution, and They represent and The marginal probability distribution is shown above. This probability distribution can be obtained statistically from road scene samples with ODC boundary state labels.

[0043] Step S204: Initialize the set of selected state items The set is empty. When the set of selected states is empty, the state with the highest risk relevance is selected from the candidate states and added as the first one. The state item in the text.

[0044] Step S205: Calculate the set of candidate state items and the set of selected state items. The average redundancy between the states is calculated, and the MRMR gain of the candidate state terms is calculated. The candidate state terms with the largest MRMR gain are added to the set of selected state terms. middle.

[0045] When filtering based solely on risk relevance, multiple semantically similar and informationally redundant state items may be selected. For example, road construction status, temporary cone status, and lane closure status may have a strong correlation. Including all these state items in VLM identification would result in redundant monitoring and state representation. Therefore, this invention also needs to calculate the average redundancy between the candidate state items and the set of selected state items.

[0046] Let the set of currently selected state items be... There are already multiple state items, and candidate state items. With the set of selected state items The average redundancy between them is defined as:

[0047] (17)

[0048] in, Represents the candidate state item Corresponding observed random variable With the selected status item Corresponding observed random variable Mutual information between them; The larger the value, the higher the degree of information overlap between the candidate state item and the selected state item, and the lower the priority for the candidate state item to be selected.

[0049] Based on this, the present invention employs the minimum redundancy maximum correlation criterion to perform MRMR gain screening on candidate state terms. Candidate state terms In the set of selected status items The MRMR gain under the given conditions is defined as follows:

[0050] (1)

[0051] For each round of selection, the state with the largest MRMR gain is chosen from the candidate state items that have not yet been selected as the selected state item for the current round. And add it to the set of selected state items. middle.

[0052] By using the above MRMR gain screening criteria, this invention can avoid selecting only state items that are highly correlated with but overlap with the ODC boundary state, thereby obtaining a set of state items applicable to VLM with more comprehensive risk information coverage and lower redundancy.

[0053] Step S206: Repeat step S105 until the number of selected status items reaches the preset number. Or the maximum MRMR gain of the candidate state item is lower than the preset minimum gain threshold. This yields the final set of selected state items applicable to the VLM. .

[0054] like Figure 2 As shown, step S2 mainly determines the applicable ODC state items for VLM based on risk relevance, average redundancy, computing power cost, and structured feasibility. This process transforms the screening process for applicable VLM state items from manual experience-based judgment to an incremental state item selection process based on mutual information. This process also considers the contribution of state items to distinguishing ODC boundary states and the degree of information redundancy between state items, so that the finally selected state items can reflect changes in operating condition boundaries and reduce the interference of duplicate semantic items on the VLM perception task.

[0055] In one exemplary implementation, the status item filtering results can be obtained as shown in Table 1 below:

[0056] Table 1

[0057]

[0058] It is worth mentioning that the values ​​in Table 1 are normalized values. These values ​​are only used to illustrate the calculation process of the screening method of the present invention and do not limit the specific values ​​in actual deployment.

[0059] In another embodiment, the present invention may also employ expert rule scoring, feature importance ranking, correlation coefficient screening, or a method based on classifier feature contribution to select the most suitable semantic state item for VLM recognition from candidate state items.

[0060] Step S3: Establish a semantic ODC status item identification and generation method, so that VLM can generate structured output fields that the ODC monitoring module can parse for specific status items.

[0061] After completing the screening of state items applicable to VLM, this invention further performs structured identification on the screened dynamic state items. Preferably, this invention constructs dynamic ODC state identification as a combined method of "ODC sample library retrieval enhancement prompts + efficient parameter supervision and adaptation + structured constraint generation," that is, it uses a method based on LoRA lightweight fine-tuning and structured Prompt constraints to identify and generate semantic ODC states. Simply put, as Figure 2 As shown, the process involves first creating an element configuration table, then generating element-level prompts based on the ODC status items applicable to the VLM, and finally inputting them into the VLM that has been lightly tweaked by LORA to obtain structured output fields. This process specifically includes:

[0062] Step S301: Establish a semantic ODC state recognition method based on ODC state similarity sample retrieval enhancement and element configuration table: Establish an element configuration table for state items applicable to each type of VLM; establish an ODC state item sample library containing road scene images, state items, structured output fields, visual encoding vectors, and semantic descriptions; perform similar sample retrieval in the sample library based on the current image input and the state item to be recognized, and generate a retrieval enhancement structured Prompt in combination with the element configuration table, so that the VLM recognition task is transformed from an open image description to a conditional structured generation task oriented towards specified state items.

[0063] 1) Problem modeling.

[0064] like Figure 4 As shown, for the set of selected state items obtained through the aforementioned filtering... For any state item in the data, the system obtains the corresponding image input from the sensor cache or road scene data stream. This input can be a single-frame image, a continuous-frame image, a multi-view image, or a multimodal visual input formed by combining single-frame images, continuous-frame images, and multi-view images.

[0065] This invention defines semantic ODC state recognition as a multimodal conditional structured generation task. Its goal is to generate the structured output field corresponding to a given road scene image, state item, and retrieval-enhanced structured prompt, as shown in the following formula:

[0066] (18)

[0067] in, This represents the visual language model after parameter fine-tuning. The underlying VLM parameters indicate the freeze state. This represents the trainable LoRA adaptation parameters; State item The corresponding road scene image; This indicates an enhanced structured retrieval prompt; For state items The corresponding structured output fields must include at least the following fields:

[0068] (19)

[0069] The meanings of each field are shown in Table 2 below.

[0070] Table 2

[0071]

[0072] Through the above modeling, semantic ODC state recognition is no longer an open image description task, but a constrained state recognition task oriented towards specified ODC state items. Here, result and state can be regarded as classification fields, region as a region field, and evidence as a visual evidence field. The overall output must meet a preset structured format.

[0073] 2) Construction of ODC status sample library.

[0074] To avoid arbitrary selection of Prompt reference examples based on human experience, this invention establishes an ODC state example library to store labeled road scene images, ODC state items, structured output fields, visual encoding vectors, and semantic descriptions. The semantic descriptions are textual or structured expressions of a state item or example based on key configuration items in an element configuration table. The element configuration table includes state item definitions, state item value ranges, judgment criteria, risk evidence, and region descriptions. The ODC state example library is denoted as... As shown in the following formula:

[0075] (20)

[0076] Among them, the bar sample Represented as:

[0077] (twenty one)

[0078] in, This indicates the ODC status item corresponding to this example; State item The corresponding number A road scene image; For state items The corresponding structured output fields obtained through manual annotation or rule-based annotation; This represents the sample vector obtained by the visual encoder or visual language model encoder. This table represents the element configuration table corresponding to the example, including the state item definition, judgment basis, risk evidence, and area description. Taking a non-standard obstacle state as an example, it is shown in Table 3 below.

[0079] Table 3

[0080]

[0081] In one alternative implementation, sub-sample libraries can be established for different ODC status items:

[0082] (twenty two)

[0083] in, express The corresponding subsample library, and These represent the target state item and the state item corresponding to the j-th sample record in the subsample library, respectively.

[0084] Each subsample corpus contains at least positive, negative, and uncertain samples to prevent the model from learning only a single existing state. Specifically, taking non-standard obstacles as an example, examples of subsample corpora are shown in Table 4 below.

[0085] Table 4

[0086]

[0087] Each structured output field in the sample library A consistent field structure should be followed. For example, a sample for "non-standard obstacle status" can be represented as shown in Table 5 below.

[0088] Table 5

[0089]

[0090] 3) ODC status similar sample retrieval.

[0091] When the system needs to identify the current road scene image The specified ODC status item When doing so, the current query vector must first be calculated. :

[0092] (2)

[0093] in, This refers to a visual encoder or visual language model encoder. This encoder can simultaneously utilize image content and state item text definitions to enable query vectors... It includes both visual information of the road scene and semantic information of the state items to be identified.

[0094] Subsequently, the system in the status item Corresponding subsample library Perform a similarity sample retrieval to obtain the most similar sample to the current input. A reference example:

[0095] (3)

[0096] in, State item A set of similar examples; Indicates in the subsample library middle The first K values ​​of the sample; Indicates belonging to the subsample library The first in Sample; Represents the current query vector With the Sample vectors of bar samples Cosine similarity between them.

[0097] In one alternative implementation, the similarity can be expressed as cosine similarity:

[0098] (4)

[0099] In this way, the reference examples in the Prompt are not subjectively selected by humans, but are obtained by retrieving the vector similarity between the current scene image and the example library. This allows the VLM to obtain visual references and structured output field references that are closer to the current road scene during inference.

[0100] 4) Retrieve enhanced structured Prompt generation.

[0101] After obtaining the set of similar samples Subsequently, the present invention generates a search-enhanced structured Prompt based on fixed modules and an element configuration table;

[0102]

[0103] (twenty three)

[0104] in, This indicates an enhanced structured retrieval prompt; For the status item name, For input type, For the applicable perspective, To identify the target, evidence serves as the basis for judgment. To specify the output fields, This is a static output item.

[0105] It should be noted that the fixed module, or common prompt word, includes recognition output rules and common constraints, as detailed below:

[0106] Identification output rules:

[0107] 1. Output only valid JSON (JavaScript Object Notation, Lightweight Data Interchange Format), without outputting explanations, Markdown, comments, or additional text.

[0108] 2. Only matched_rules and odc_elements are allowed at the top level.

[0109] 3. odc_elements must contain exactly 14 items, one for each category, in a fixed order: road_surface_state, road_marking_state, traffic_sign_state, traffic_signal_state, obstacle, motor_vehicle, non_motor_vehicle, pedestrian, ambient_illumination, rainfall_intensity, fog_visibility, road_construction, accident_scene, traffic_control.

[0110] 4. Each item must first output element_name and need_recognition.

[0111] 5. If the element belongs to selected: need_recognition=true, continue to output the other fields required by the configuration of the element.

[0112] 6. If an element is not selected: need_recognition=false, and the element object is only allowed to contain element_name and need_recognition, do not output any other fields such as result / state / severity / region.

[0113] 7. result=absent indicates that the element was selected and identified but did not appear in the current frame; when a reliable judgment cannot be made, result=uncertain and uncertainty_reason must not be none.

[0114] 8. Result indicates the recognition result; severity indicates the degree of visual anomaly; confidence indicates the recognition confidence level.

[0115] 9. The output must be in vertical format, using 2 spaces for indentation.

[0116] 10. Judgments should be based solely on visible evidence within the current single-frame forward-looking image, without using information outside the image; speculation about the target's true intentions, precise speed, precise distance, rainfall, illuminance, visibility in meters, or remaining traffic light duration is prohibited; when a reliable judgment cannot be made, "uncertain" must be used, and "uncertainty_reason" must be entered; "confidence" is a decimal between 0 and 1, representing only the confidence level of identification, not the severity of the risk; "evidence" must be a string array, containing only observable evidence from the image.

[0117] Public constraints:

[0118] result=present,absent,uncertai;

[0119] severity=not_applicable,none,slight,moderate,severe,uncertain;

[0120] region=front_lane,left_lane,right_lane,adjacent_lane,roadside,intersection,overhead,far_ahead,global_scene,unknown;

[0121] reason=none, image_unclear, object_too_small, occluded, ambiguous_category, insufficient_visual_evidence, outside_camera_view, other; confidence is a decimal between 0 and 1.

[0122] `evidence` is an array of strings consisting of visible evidence in the current image.

[0123] Based on this, VLM is limited to identifying specified state items and outputting structured output fields, excluding driving decisions, control suggestions, or generalized descriptions unrelated to the state item. Taking "Obstruction and Blind Spot Risk State" as an example, its enhanced structured prompt is as follows:

[0124] { } = Non-standard obstacle state;

[0125] {input_type} = Single frame / Continuous frame / Multiple viewpoints;

[0126] {view_type} = Front view, Side front view, Ring view;

[0127] {recognition_target} = Determines whether there are non-standard obstacles in the current scene and whether they affect the area where vehicles can pass;

[0128] {evidence} = irregular objects on the road surface, fallen goods, construction debris, abnormal placement of cones or water-filled barriers, animals or unknown objects, non-vehicle and non-pedestrian targets occupying lanes, shoulders or intersection passage areas.

[0129] {output_fields}=element_name,result,state,region,confidence,evidence;

[0130] {forbidden_outputs} = Driving decisions, planning suggestions, and control commands.

[0131] Step S302: Establish a semantic ODC state generation method based on LoRA lightweight fine-tuning and structured output constraints.

[0132] To adapt the VLM to semantic ODC state recognition tasks in road scenes, this invention employs LoRA for parameter fine-tuning. During fine-tuning, the original parameters of the VLM are frozen, and trainable low-rank increment matrices are injected only into some linear layers or multimodal projection layers. For the weights of a certain linear layer of the VLM... Its adapted weight Represented as:

[0133] (5)

[0134] in, and For trainable low-rank increment matrices, Let be the rank of the matrix. This is the scaling factor. Using this method, VLM only needs to update a small number of parameters to learn the ODC state recognition task, reducing training memory usage and sample requirements.

[0135] This invention constructs an ODC structured instruction fine-tuning dataset. :

[0136] (twenty four)

[0137] in, State terms used for training The retrieval enhancement structured Prompt; N represents the number of state terms trained.

[0138] The training objective employs a standard supervised generative loss, enabling the VLM to learn the mapping from road scene images and the retrieved augmented structured prompt to the ODC fielded answer. The VLM's loss function... Represented as:

[0139] (6)

[0140] in, Indicates the first The length of the token in the structured output field; N represents the number of state items trained; State item The corresponding structured output fields The first in the sequence One token; State item The corresponding number A road scene image; State item Enhanced structured Prompt for retrieval; Indicates the first The structured output field content that was previously generated for each token; The conditional generation probability model is jointly determined by the VLM basic parameter θ and the LoRA adaptation parameter ϕ, representing the probability of the model generating the current target token given a road scene image, a structured Prompt, and historical output tokens.

[0141] Therefore, VLM only needs to learn to output structured, predefined fields for specific state items under the conditions of input, state item definition, retrieval examples, and output format constraints given a road scene image.

[0142] In another embodiment, the present invention may also employ an open vocabulary target detection model, a multi-task visual perception network, a CLIP-type image-text matching model, a dedicated classifier, or a visual detection model to identify semantic ODC states, enabling the VLM to output structured and standardized fields for specific state items.

[0143] Step S4: Dynamically monitor, schedule, and structurally update the VLM based on resource constraints.

[0144] After completing the selection of ODC state items applicable to VLM and the semantic identification of ODC states, this invention addresses the problems of high VLM inference overhead and limited vehicle-side computing resources by dynamically monitoring, scheduling, and structurally updating VLM based on rate-distortion loss. Figure 2 As shown, based on the state risk value, risk volatility coefficient, and computational resource budget, the VLM call frequency of each semantic ODC state item is dynamically adjusted (to give high-risk, high-volatility state items a higher call frequency) and the ODC monitoring system is updated. This process specifically includes:

[0145] Step S401: Semantic ODC status item risk quantification.

[0146] Let the set of selected state items obtained through the aforementioned step S1 be... There are several state terms. At time t, any state term... Structured output fields obtained by VLM recognition .

[0147] To enable the aforementioned structured output fields to be incorporated into the subsequent risk quantification model, this invention first uses a state variable extraction function. Convert it into a state vector :

[0148] (25)

[0149] in, State item At time t Each risk monitoring dimension can be derived from... , , , or The data is obtained through field encoding. For non-standard obstacle statuses, monitoring dimensions can include "whether an obstacle exists," "whether a key passageway is occupied," "degree of insufficient confidence," and "amount of risk evidence." This indicates the number of risk monitoring dimensions. The above transformation does not change the meaning of the VLM identification results; it only converts the fielded output into a computable boundary risk input.

[0150] For any risk monitoring dimension in the state vector Based on its ODC boundary constraint type, establish the corresponding normalized risk mapping function. Let... The boundary constraints are , Normalized risk mapping function Represented as:

[0151] (26)

[0152] in, The element-level risk at time t; From the perspective of risk monitoring The type of boundary constraint is determined. Under different types of boundary constraints, the following normalization methods can be used to map the corresponding element-level risks:

[0153] (1) Binary boundary constraints

[0154] When risk monitoring dimensions only represent two states, "safe" or "risk," a binary boundary risk mapping can be used. Let... Where 0 represents safe and 1 represents unsafe, the Sigmoid function can be used to achieve a smooth transition:

[0155] (27)

[0156] in, Binary boundary risk represents the risk monitoring dimension. Element-level risk under binary boundary constraints; For boundary thresholds; This is a parameter for adjusting the steepness of the curve. This mapping is applicable to... The status consists of two parts: fields, key area occupancy flags, and whether risk evidence has appeared.

[0157] (2) One-sided threshold boundary constraint

[0158] When there is only one maximum permissible boundary for risk monitoring dimensions, let , For risk monitoring dimensions The maximum boundary can be normalized by the ratio of the absolute value to the maximum boundary:

[0159] (28)

[0160] in, For one-sided threshold boundary risk, it represents the risk monitoring dimension. Element-level risk under one-sided threshold boundary constraints; when hour, This mapping is applicable to risk monitoring dimensions with one-sided upper bounds, such as the number of pieces of evidence, the number of anomalous targets, and the degree of low confidence.

[0161] (3) Bilateral interval boundary constraints

[0162] When risk monitoring dimensions have a safe range hour, For risk monitoring dimensions The minimum boundary can be determined by using the midpoint of the interval as a benchmark and calculating the risk based on the degree of deviation.

[0163] (29)

[0164] The risk at the two-sided interval boundary, i.e., the risk monitoring dimension. Element-level risk under bilateral interval boundary constraints for:

[0165] (30)

[0166] when The further away from the center of the safe interval, the higher the risk value; when it reaches or exceeds the interval boundary, the risk value approaches 1. This mapping is applicable to ODC state variables that need to be kept within a specific range.

[0167] (4) Discrete set boundary constraints

[0168] When the risk monitoring dimension is a discrete semantic set, let the safety set be... for:

[0169] (31)

[0170] in, This represents the nth discrete safety value in the safety set.

[0171] like If it belongs to the safe set, the risk is 0; otherwise, the risk is 1.

[0172] (32)

[0173] in, For discrete set boundary risks, the risk monitoring dimension is represented. Element-level risk under discrete set boundary constraints. This mapping is applicable to... , , Discrete semantic fields, etc. For example, for non-standard obstacle states, if the state field belongs to "no_obstacle, absent", the corresponding risk can be set to 0; if it belongs to "possible_obstacle, non_standard_obstacle, lane_blocking, uncertain", the corresponding risk can be set to 1.

[0174] After obtaining the element-level risks for each risk monitoring dimension, this invention uses a hierarchical weighted aggregation method to calculate the risk of state items. For state items... Its risk value Defined as:

[0175] (9)

[0176] in, For state items Inner To reduce subjective manual weighting, the weights for each risk monitoring dimension can be obtained statistically from samples labeled with ODC boundary states. Specifically, let... Indicates risk monitoring dimensions With boundary state labels The mutual information between them can be used to determine the weights using mutual information normalization:

[0177] (10)

[0178] in, and These are the state items. Inner One and One risk monitoring dimension.

[0179] Therefore, a higher weight for a risk monitoring dimension indicates a stronger contribution to distinguishing changes in the boundary state of the ODC. This weighting method is consistent with the risk correlation calculation in the ODC state item screening applicable to VLM mentioned above, which avoids relying entirely on manual experience for setting.

[0180] Step S402, Risk volatility coefficient estimation.

[0181] To describe the dynamic characteristics of state terms changing over time, this invention calculates the standard deviation of the risk change of state terms within a sliding window to obtain the risk volatility coefficient. :

[0182] (11)

[0183] in, This represents the number of samples within the sliding window. The larger the value, the more drastic the change in the status item, and the more frequently it needs to be called; conversely, the smaller the value, the more drastic the change. The smaller the value, the more stable the change in the status item, which can reduce the frequency of calls.

[0184] Step S403, rate distortion loss modeling.

[0185] This invention will increase the VLM call frequency. As a "rate," risk distortion is defined as the failure to detect out-of-bounds errors, delayed risk response, or failure to promptly capture state deterioration caused by insufficient sampling. The higher the sampling frequency and the shorter the monitoring interval, the lower the risk of distortion; conversely, the lower the sampling frequency and the shorter the monitoring interval, the higher the risk of distortion.

[0186] Let the safety boundary normalization threshold be 1, when When the value is less than 1, the state item has not yet crossed the boundary, resulting in risk distortion. The probability of exceeding the limit and not being detected before the next sampling can be expressed as:

[0187] ; <1(12)

[0188] in, This is the cumulative distribution function of the standard normal distribution; this formula shows that the closer the state is to the boundary, the larger the risk fluctuation coefficient, or the lower the sampling frequency, the higher the distortion risk.

[0189] when When the value is ≥1, the state item has reached or exceeded the boundary, resulting in risk distortion. This represents the expected cumulative out-of-bounds deviation within the sampling interval. Let... ,but It can be represented as:

[0190] ; ≥1 (13)

[0191] like If the value is too small or the target is distorted and unattainable, the status item can be directly set to the maximum allowed call frequency to ensure that high-risk statuses are responded to in a timely manner.

[0192] Step S404: Optimize dynamic call frequency under resource constraints.

[0193] When vehicle-side computing resources, communication bandwidth, or VLM inference budgets are limited, this invention solves for the frequency under resource constraints in each scheduling cycle and allocates it, constructing an objective function that minimizes the weighted total risk distortion, as shown in the following equation:

[0194] (7)

[0195] Its constraints are:

[0196] (8)

[0197] in, State item Importance weight, , ; For state items Inner Risk monitoring dimensions The weight, , ; For state items The number of risk monitoring dimensions; Indicates the status item to be called at a unit frequency. Required computational cost; This indicates the available computational budget at time t. and Representing state items respectively Minimum and maximum allowed call frequencies; State item In frequency of calls Risk distortion under the following conditions; M represents the number of state items to be identified; For state items The frequency of calls.

[0198] The above optimization can be solved by convex optimization, Lagrange multiplier method, priority sorting or greedy approximation strategy. This invention does not limit the specific solver.

[0199] In another embodiment, when quantifying risk, the present invention may employ manual risk lookup tables, risk level classification, TTC-type kinematic indicators, or risk scoring methods based on expert rules.

[0200] Reference Figure 5 The method shown here applies the risk quantification method, rate-distortion loss model and frequency calculation logic of ODC out-of-bounds monitoring in this invention to engineering implementation, and can realize the dynamic optimization allocation of the state call frequency of multi-source ODCs.

[0201] On the other hand, based on the same principle as the visual language model-based dynamic identification and monitoring method for ODC status described in the above embodiments, the present invention also provides a VLM-based dynamic identification and monitoring device for ODC status. This device includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the visual language model-based dynamic identification and monitoring method for ODC status described in the above embodiments. Specifically, the device can be an electronic computer or tablet computer, the processor can be a CPU, GPU, etc., and the memory can be RAM, ROM, EEPROM, CDROM, disk storage medium, or any other medium capable of carrying or storing the computer program and capable of being read by a computer; no limitation is made herein.

[0202] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the ODC status dynamic identification and monitoring method based on visual language model perception described in the above embodiments. Specifically, the computer-readable storage medium may be RAM, ROM, EEPROM, SSD, CDROM, DVD, USB flash drive, or any other medium capable of carrying or storing the computer program and capable of being read by a computer.

[0203] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the ODC status dynamic identification and monitoring method based on visual language model perception described in the above embodiments.

[0204] In summary, the innovation of this invention lies in the following: the ODC state dynamic identification and monitoring method based on visual language model perception does not directly use VLM for vehicle planning, control, or driving decisions, but instead uses it to identify semantic state items that are difficult for traditional perception models to cover stably, and transforms the identification results into structured state data that can be received, parsed, and continuously updated by the ODC monitoring system.

[0205] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamic identification and monitoring of ODC status based on visual language model perception, characterized in that, include: Establish a semantic ODC state item selection method applicable to the Visual Language Model (VLM) to select the most suitable state item for VLM to recognize from the candidate semantic ODC state items; establish a semantic ODC state item recognition and generation method to enable VLM to generate structured output fields that can be parsed by the ODC monitoring system for specific state items; and perform dynamic monitoring, scheduling and structured updates of VLM oriented towards resource constraints.

2. The method for dynamic identification and monitoring of ODC status based on visual language model perception according to claim 1, characterized in that, The MRMR feature selection algorithm is used to filter the state terms applicable to VLM, including: 1) Construct a candidate state item set, wherein the state item is a semantic ODC state item, including road construction state, non-standard obstacle state, abnormal road state, and accident scene state; 2) Obtain road scene samples with ODC boundary state labels, the samples including road scene images, observation results of candidate state items, and ODC boundary state labels; 3) Calculate the mutual information between candidate state items and ODC boundary state labels to obtain the risk correlation between them; 4) Initialize the selected state item set to an empty set. When the selected state item set is empty, select the candidate state item with the highest risk correlation as the first selected state item. 5) Calculate the average redundancy between the candidate state item and the set of selected state items, and calculate the MRMR gain of the candidate state item. Add the candidate state item with the largest MRMR gain to the set of selected state items. Candidate state item In the set of selected status items MRMR gain under certain conditions The calculation formula is as follows: （1） In the formula, Candidate state items The mutual information between the ODC boundary status label and the ODC boundary status label indicates the risk correlation between the two. Candidate state items With the set of selected state items Average redundancy between them; 6) Repeat step 5) until the number of selected state items reaches the preset number, or the maximum MRMR gain of the candidate state items is lower than the preset minimum gain threshold, to obtain the final set of selected state items applicable to VLM. .

3. The method for dynamic identification and monitoring of ODC status based on visual language model perception according to claim 1, characterized in that, A semantic ODC state recognition and generation method based on LoRA lightweight fine-tuning and structured Prompt constraints is established, enabling VLM to generate structured output fields for specific state items, including: 1) Establish a semantic ODC state recognition method based on ODC state similarity sample retrieval enhancement and element configuration table: An ODC state sample library is constructed to store labeled road scene images, state items, structured output fields, visual encoding vectors, and semantic descriptions. In this sample library: for any state item, the corresponding road scene image is obtained from the sensor cache or road scene data stream. The road scene image can be a single-frame image, a continuous-frame image, a multi-view image, or a multimodal visual input formed by any combination of single-frame images, continuous-frame images, and multi-view images. The structured output fields are obtained by manually annotating or rule-assisted annotating the state items, including the state item name. Is the current status item recognized as existing? Enumeration of state items The area where the status item appears or is affected VLM's confidence level in this structured output field Judgment basis The visual encoding vector is a sample vector obtained by a visual encoder or a VLM encoder; the semantic description is a textual or structured expression of a certain state item or a certain sample based on the key configuration items in the element configuration table, and the element configuration table includes state item definition, state item value range, judgment basis, risk evidence and area description; Based on the current road scene image and the state item to be identified, a similar example retrieval is performed in the example database: First, a query vector for the current road scene image is calculated. Then, based on the similarity between the query vector and the example vector, a similar example retrieval is performed in the example database to obtain the most similar example to the current road scene image. A reference example; After obtaining the set of similar samples, a search-enhanced structured Prompt is generated based on the element configuration table. This search-enhanced structured Prompt includes the status item names. Input type Applicable perspective Identify the target Judgment basis Structured output fields Static output items ; 2) Establish a semantic ODC state generation method based on LoRA lightweight fine-tuning and structured output constraints; The original parameters of the VLM are frozen, and a trainable low-rank increment matrix is ​​injected only into some linear layers or multimodal projection layers; an instruction fine-tuning dataset is constructed consisting of road scene images, state terms, retrieval-enhanced structured prompts, and structured output fields; a supervised generative loss is used to enable the VLM to learn the mapping from road scene images and retrieval-enhanced structured prompts to structured output fields.

4. The ODC status dynamic identification and monitoring method based on visual language model perception according to claim 3, characterized in that, A rate-distortion loss-based VLM dynamic monitoring and scheduling method is adopted to dynamically monitor, schedule and structure the VLM. Based on the state item risk value, risk fluctuation coefficient, target risk distortion and computing resource budget, the calling frequency of each VLM applicable state item is dynamically optimized so that high-risk and high-fluctuation state items get higher calling frequency, while low-risk or stable state items get lower calling frequency.

5. The method for dynamic identification and monitoring of ODC status based on visual language model perception according to claim 3, characterized in that, When establishing the ODC state sample library, sub-sample libraries are established for different ODC state items. Each sub-sample library contains at least positive samples, negative samples, and uncertain samples.

6. The ODC status dynamic identification and monitoring method based on visual language model perception according to claim 5, characterized in that, When searching for similar examples in the sample library based on the current road scene image and the state item to be identified: First, calculate the query vector of the current road scene image. The formula for calculating the query vector is as follows: （2） in, Represents road scene images Specify status item The query vector; Represents a visual language model encoder; Subsequently, in the status item Corresponding subsample library Perform a similar example retrieval in the middle to obtain images of the current road scene. Most similar Example: （3） （4） in, State item A set of similar examples; Indicates in the subsample library middle The first K values ​​of the sample; Indicates belonging to the subsample library The first in Sample; Represents the current query vector With the Sample vectors of bar samples Cosine similarity between them.

7. The method for dynamic identification and monitoring of ODC status based on visual language model perception according to claim 6, characterized in that, After injecting a trainable low-rank increment matrix into the linear layer of the VLM, its linear layer weights Weights obtained after adaptation Represented as: （5） in, and For trainable low-rank increment matrices, Let be the rank of the matrix. This is the scaling factor; The loss function of the VLM Represented as: （6） in, Indicates the first The length of the token in the structured output field; N represents the number of state items trained; State item The corresponding structured output fields The first in the sequence One token; State item The corresponding number A road scene image; State item Enhanced structured Prompt for retrieval; Indicates the first The structured output field content that was previously generated for each token; Let θ be the conditional generation probability model determined by the frozen basic VLM parameters θ and the trainable LoRA adaptation parameters ϕ, representing the probability that the model will generate the current target token given a road scene image, a structured Prompt, and historical output tokens.

8. The method for dynamic identification and monitoring of ODC status based on visual language model perception according to claim 7, characterized in that, The call frequency of each applicable state item of the VLM is dynamically optimized with the objective function of minimizing the weighted total risk distortion. The expression of this objective function is as follows: （7） The constraints of the objective function are: （8） in, State item Importance weights , ; For state items Inner Risk monitoring dimensions The weight, , ; For state items The number of risk monitoring dimensions; Indicates the status item to be called at a unit frequency. Required computational cost; This indicates the available computational budget at time t. and Representing state items respectively Minimum and maximum allowed call frequencies; State item In frequency of calls Risk distortion under the following conditions; M represents the number of state items to be identified; For state items The frequency of calls.

9. The method for dynamic identification and monitoring of ODC status based on visual language model perception according to claim 8, characterized in that, The status item State risk value The expression is as follows: （9） （10） in, Let be the element-level risk at time t, representing any risk monitoring dimension. The risk value obtained at time t based on its ODC boundary constraint type. ODC boundary constraints include binary boundary constraints, one-sided threshold boundary constraints, two-sided interval boundary constraints, and discrete set boundary constraints. Indicates risk monitoring dimensions With boundary state labels Mutual information between them; and These are the state items. Inner Individual and One risk monitoring dimension; The risk volatility coefficient The expression is as follows: （11） in, This represents the number of samples within the sliding window. The larger the value, the more likely it is to be a state term. The more drastic the change, the more frequently it needs to be invoked; conversely, the less drastic the change, the more necessary it is to increase its invocation frequency. The smaller the value, the more likely it is to be a state term. The changes are stable, which can reduce the frequency of its calls; State item The risk value of the state item in the k-th sliding window at time t; The risk distortion The expression is as follows: ； <1（12） ； ≥1（13） in, This is the cumulative distribution function of the standard normal distribution.

10. A VLM-based dynamic identification and monitoring device for ODC status, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: The processor executes the computer program to implement the ODC status dynamic identification and monitoring method based on visual language model perception as described in any one of claims 1-9.

Images