Method, device, medium and product for testing a visual-linguistic-action model

Abstract

The application relates to the technical field of artificial intelligence, in particular to a visual-language-action model testing method, device, medium and product. The scheme constructs an initial inference link containing multiple inference nodes based on the decision logic of a to-be-tested model, generates multiple target inference links according to a preset derivation strategy, and generates corresponding test scenes according to the mapping relationship between the inference links and scene parameters, and simulates and tests the model. The scheme enables the test scene to revolve around the actual perception, judgment and action decision process of the model, improves the consistency of the test scene and the decision logic, enhances the long-tail and complex traffic situation coverage capability, and thus more accurately evaluates the inference stability and robustness of the model under different levels of scenes.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a testing method, device, medium and product for a vision-language-action model. Background Technology

[0002] With the rapid development of autonomous driving technology, Vision-Language-Action (VLA) models, as core components of intelligent driving systems, are widely used in the integrated processing of perception, reasoning, and decision-making in complex traffic environments. Existing model testing methods mainly rely on static image perturbation or pre-set scenario libraries. These methods generate adversarial examples by embedding perturbations into images and text in a single modality or by combining visual and language perturbations to test the model's resistance to input noise. Alternatively, they rely on regulations, accident cases, and expert experience to build scenario libraries and use simulation platforms to generate operating conditions to complete model testing.

[0003] However, most of the above methods revolve around input layer noise or manually set scenarios. The test scenarios are not strongly related to the actual decision-making logic of the model, making it difficult to cover the key reasoning process in dynamic traffic, resulting in insufficient test coverage. Summary of the Invention

[0004] This application provides a testing method, device, medium, and product for a vision-language-action model to solve the aforementioned technical problems. The solution constructs a corresponding inference chain around the decision logic of the model under test, and forms a derivative chain with hierarchical expansion capabilities on this basis. This allows the testing process to more closely resemble the actual reasoning and decision-making process of the model in dynamic traffic scenarios, thereby improving the consistency between the testing scenario and the model's decision logic.

[0005] In a first aspect, embodiments of this application provide a testing method for a visual-language-action model, the method comprising:

[0006] An initial inference chain is generated based on the decision logic of the model to be tested. The initial inference chain includes multiple inference nodes.

[0007] A pre-defined derivation strategy is used to generate multiple target derivation inference links based on the initial inference link;

[0008] Based on the preset mapping relationship, each target-derived inference link is converted into a corresponding test scenario, so as to test the model under test using the test scenario. The test scenario includes the scenario parameters corresponding to the inference node.

[0009] In one possible embodiment, generating an initial inference chain includes:

[0010] Obtain multiple target decision nodes and multi-source historical driving information of the decision logic of the model under test;

[0011] Multiple target decision nodes are sorted based on multi-source historical driving information to generate an initial inference link, where different driving scenarios correspond to different initial inference links.

[0012] In one possible embodiment, the multiple inference nodes include an environmental perception node, a behavior prediction node, a safety assessment node, a behavior decision node, and a trajectory planning node;

[0013] The preset derivation strategies include at least one of the following: reasoning branch derivation, node sequence derivation, decision parameter derivation, object combination derivation, and environmental condition derivation.

[0014] In one possible embodiment, generating the target derivation inference chain includes at least one of the following steps:

[0015] Add a judgment and reasoning node to the initial reasoning chain to generate multiple reasoning branches;

[0016] Modify the decision parameters, boundary thresholds, or environmental conditions of the initial inference chain;

[0017] Adjust the execution order of the inference nodes in the initial inference chain;

[0018] Add or remove participant objects from the initial inference chain.

[0019] In one possible embodiment, the preset mapping relationship includes the mapping relationship between each inference node and scene variables, and also includes:

[0020] Determine the set of scenario variables corresponding to each inference node in the target derivation inference chain based on the preset mapping relationship;

[0021] The association rules between scenario variables are determined based on the logical dependencies between each inference node.

[0022] In one possible embodiment, association rules include activation conditions, numerical linkage relationships, and mutual exclusion constraints. Translated into a test scenario, this includes:

[0023] Based on association rules and semantic scene description text composed of scene variable sets, values ​​are assigned to the scene variables corresponding to each inference node to obtain the scene parameters corresponding to each inference node.

[0024] In one possible embodiment, it also includes:

[0025] Test scenarios are categorized into levels based on the complexity of the reasoning chain for each objective to generate tiered test scenarios. Tiered test scenarios include at least two of the following: basic scenarios, branch decision scenarios, boundary condition scenarios, multi-object interaction scenarios, and complex environment scenarios.

[0026] In one possible embodiment, it also includes:

[0027] Obtain the output of the model under test;

[0028] The robustness of model inference is evaluated based on the output results. The robustness evaluation of model inference includes inference chain consistency evaluation, consistency evaluation between inference results and behavioral decisions, and robustness decay evaluation under different complexity scenarios.

[0029] In one possible embodiment, it also includes:

[0030] The test scenario containing scenario parameters is input into a preset simulation tool for scenario modeling. The preset simulation tool is used to construct a virtual traffic environment and perform time-series simulation based on the scenario parameters, generating virtual sensor images and driving semantic commands as inputs to the model under test.

[0031] In one possible embodiment, it also includes:

[0032] Visual perturbations are added to the visual images output by the preset simulation tool, and text perturbations are added to the semantic instructions to obtain a multimodal adversarial scenario. The visual perturbations include image noise, lighting shift, and image blurring, while the text perturbations include semantic ambiguity and vague task descriptions.

[0033] In one possible embodiment, it also includes:

[0034] Obtain the inference output results of the model under test at the safety assessment node and trajectory planning node;

[0035] Adjust the scene parameters of the input preset simulation tool based on the inference output results to generate targeted adversarial sub-scenes.

[0036] In one possible embodiment, it also includes:

[0037] Logical verification is performed on the inference links derived from each target. Logical verification includes timing conflict verification, condition mutual exclusion verification, and semantic contradiction verification.

[0038] In one possible embodiment, it also includes:

[0039] A pre-trained reinforcement learning model is used to generate multiple target-derived inference links based on the initial inference link. The pre-trained reinforcement learning model is obtained by iterative training using the model inference robustness evaluation result as the reward signal.

[0040] In one possible embodiment, it also includes:

[0041] A pre-defined graph neural network is used to model the graph structure of each inference node and the candidate scenario variables in the target derivation inference link, so that the pre-defined graph neural network outputs the set of scenario variables corresponding to each inference node in the target derivation inference link.

[0042] In one possible embodiment, after generating the initial inference chain, the process further includes:

[0043] The initial reasoning chain is logically validated by pre-set expert tools to identify unreasonable reasoning chains that violate real driving logic, and unreasonable reasoning chains are corrected or eliminated. The pre-set expert tools include a traffic law library, a standard driving rule library, and a driving expert experience library.

[0044] Secondly, embodiments of this application provide an electronic device, the device including: a memory and a processor;

[0045] The memory stores instructions that the computer executes;

[0046] The processor executes computer execution instructions stored in memory, causing the processor to perform the methods described above.

[0047] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided above.

[0048] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0049] This application provides a testing method, device, medium, and product for a vision-language-action model. It generates an initial inference chain containing multiple inference nodes based on the decision logic of the model under test. The inference chain is built using the model's own decision logic, ensuring that all subsequent test scenarios accurately correspond to the model's actual inference path. A preset derivation strategy is used to generate multiple target derived inference chains based on the initial inference chain, directly modifying the inference chain itself, actively creating boundaries, conflicts, and complex decision paths, and uncovering inference weaknesses that input disturbances cannot expose. A standardized derivation strategy automatically generates a massive number of differentiated inference chains, multiplying the scale of test cases and improving test completeness. Derivation transformations can control scene complexity, enabling quantitative observation of the degree to which model inference performance decays with increasing scene complexity, thus achieving robustness quantitative evaluation. Based on a preset mapping relationship, each target derived inference chain is converted into a test scenario including scene parameters corresponding to the inference nodes. This ensures that the construction of test scenarios corresponds to the model's internal inference process, thereby more accurately covering key inference processes in dynamic traffic and achieving hierarchical quantitative evaluation of the robustness and inference stability of the vision-language-action model, as well as precise fault tracing. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0051] Figure 1 An application scenario diagram for the testing method of the vision-language-action model provided in this application;

[0052] Figure 2 A flowchart illustrating a testing method for a visual-language-action model provided in an embodiment of this application;

[0053] Figure 3 A flowchart illustrating a testing method for a visual-language-action model provided in another embodiment of this application;

[0054] Figure 4 A schematic diagram of the structure of a testing device for a visual-language-action model provided in an embodiment of this application;

[0055] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0056] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0058] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0059] To clearly understand the technical solution of this application, the solutions of the prior art will be described in detail first.

[0060] In dynamic traffic environments such as urban roads, intersections, pedestrian crossings, and severe weather, testing systems need to provide the model with visual information, task instructions, and scene states, and judge its reliability in complex driving tasks based on the model's output action results. Existing solutions typically employ input perturbation testing, multimodal joint perturbation testing, and simulation testing based on a pre-set scene library to verify the robustness of the visual-language-action model. These methods often introduce noise into the inputs (images, text, etc.) or pre-set several traffic scenarios, then input these scenarios into the model and observe changes in the model's recognition results, action output, or success rate. However, current testing content has a low degree of correlation with the model's internal decision-making logic. Facing the continuous reasoning process in dynamic traffic, it is difficult to map the model's key reasoning nodes in decision-making, resulting in test scenarios failing to cover weak points in the real decision-making chain. The test results cannot accurately reflect the model's reasoning stability in complex traffic interactions, and also increase the uncertainty of robustness assessment.

[0061] Therefore, when facing technical problems in existing technologies, to achieve targeted testing of the reasoning process, the native decision-making logic of the VLA model is first extracted and structurally modeled. The complete decision-making process from perception to action output is decomposed into standardized and ordered reasoning nodes, forming an initial reasoning link. Since relying on a single initial reasoning link can only cover basic routine conditions, standardized derivative transformation rules are set on the baseline initial reasoning link to automatically generate multiple differentiated target derivative reasoning links in batches, and to generate various extreme coupled conditions in a targeted manner, thus filling the gap in long-tail scenario coverage. To improve the accuracy of model testing, corresponding scenario parameters are automatically matched and assigned to each derivative reasoning link based on mapping relationships, and standardized test scenarios are assembled and generated. Scenario parameters are bound one-to-one with reasoning nodes, and when errors occur in model testing, the source can be traced back to the corresponding reasoning node.

[0062] Figure 1 This is an application scenario diagram illustrating the testing method for the vision-language-action model provided in this application, such as... Figure 1 As shown, the scene diagram corresponding to the testing method for the visual-language-action model provided in this application includes: a terminal device 101 and a server 102. It is understood that the server 102 integrates a model testing device, which may include an inference chain construction module, an inference chain derivation module, an inference chain scene conversion module, a scene modeling module, a VLA model testing module, and an inference robustness evaluation module.

[0063] For example, a user inputs a task to be tested into terminal device 101, such as a nighttime rainy day scenario at an intersection. The model under test needs to handle the following tasks simultaneously: pedestrian crossing, a pedestrian suddenly rushes into the lane from outside the zebra crossing, and the pedestrian is carrying a child; multi-vehicle interaction, a vehicle on the left prepares to change lanes and cuts in, and the vehicle behind does not slow down in time due to low visibility; environmental interference, rainy weather causes the camera image to be blurry, making lane line recognition difficult. This scenario requires the VLA model to generate safe actions, such as slowing down and stopping, under complex multimodal input through decision logic driven by inference chain.

[0064] Specifically, after receiving the task, the model testing device in server 102 constructs an initial inference chain based on the decision logic of the model under test, traffic regulations, real driving data, and expert experience. For the intersection scenario, the inference chain includes inference nodes such as pedestrian detection, trajectory prediction, collision risk assessment, and deceleration decision. Then, a preset derivation strategy is used to generate multiple target derived inference chains based on the initial inference chain. For example, a branch for judging whether the pedestrian is carrying a child is added to the trajectory prediction node; a node for approaching vehicles from behind is added to simulate multi-vehicle interaction scenarios. The derived inference chain may include inference nodes such as pedestrian detection, trajectory prediction, judging whether the pedestrian is carrying a child, risk assessment, and emergency braking decision. Finally, each target derived inference chain is converted into a corresponding test scenario according to a preset mapping relationship. For example, the pedestrian's position is 2m outside the zebra crossing, and the pedestrian's speed is 5m / s.

[0065] Optionally, the scenario parameters of the test scenario are input into the simulation platform to generate a dynamic traffic scenario, which is then input into the model under test for testing. By collecting the model's reasoning process and decision results, its consistency with the reasoning chain is analyzed, and its logical stability under complex scenarios is evaluated. The test data and evaluation results are sent to the terminal device 102 for user viewing.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0067] Figure 2 A flowchart illustrating a testing method for a visual-language-action model provided in an embodiment of this application is shown below. Figure 2As shown, the execution subject of this embodiment is a testing device for a vision-language-action model. This device can be implemented through a computer program, or through a medium storing the relevant computer program, such as a USB flash drive and / or optical disc, or through an electronic device integrating or installing the relevant computer program, such as a chip, server, or server cluster. The testing method for the vision-language-action model provided in this embodiment includes the following steps:

[0068] S201. Generate an initial inference link based on the decision logic of the model to be tested. The initial inference link includes multiple inference nodes.

[0069] The model under test refers to the VLA model, which is used to receive inputs such as visual images and natural language commands, and internally completes the entire process of environmental perception, risk reasoning, and driving decision-making, and finally outputs vehicle control actions.

[0070] Among them, decision logic refers to the inherent judgment, deduction, and selection rules within the VLA model from receiving multimodal inputs to outputting action commands. It covers the thinking process of identifying surrounding traffic elements, predicting participant behavior, assessing driving risks, determining driving operations, and planning driving trajectories, and is the underlying basis for the model to make driving actions.

[0071] The initial inference chain refers to a complete logical chain that is structured by integrating multi-source prior driving information based on the model's original decision logic. This chain has a fixed execution order.

[0072] In this context, a reasoning node refers to an independent and indivisible basic logical unit obtained by decomposing the complete decision-making logic. Each node undertakes only a single type of reasoning task. Nodes are sequentially connected in a fixed time sequence to form a complete initial reasoning chain, which is mainly divided into five categories: environmental perception, behavior prediction, risk assessment, behavior decision-making, and trajectory planning. Each node contains its own corresponding judgment conditions, judgment thresholds, and input / output constraints.

[0073] Specifically, this study analyzes the entire operational mechanism of the VLA model from multimodal input to action output, completely dissecting each inference and judgment rule within the model, including the sequential dependencies between judgment steps and the input-output transmission relationships. It retrieves relevant reference information on compliant driving, including written traffic regulations, standard driving data collected from real-vehicle road tests, archived undisputed normal accident cases, human driving experience from experienced drivers and experts in the field of autonomous driving, and standard compliant driving inference texts generated by a large language model. This information is then used to correct the extracted model decision logic, eliminating unreasonable judgment rules that violate traffic rules or do not conform to real driving habits.

[0074] Furthermore, the complete decision logic is segmented and divided into multiple independent reasoning nodes according to the type of reasoning task. Continuous judgment tasks of the same type are integrated into a single node, while tasks of different types are separated into independent nodes. All reasoning nodes are ordered according to the natural logic of driving reasoning to generate an initial reasoning chain.

[0075] For example, in a pedestrian crossing scenario, the following inference chain can be generated, including a pedestrian detection node, a pedestrian location determination node, a pedestrian approaching the lane determination node, a pedestrian crossing prediction node, a collision risk assessment node, and a deceleration decision node.

[0076] S202. A preset derivation strategy is used to generate multiple target derivation inference links based on the initial inference link.

[0077] The preset derivation strategy refers to a set of standardized logical transformation rules that are pre-defined offline and stored in a fixed manner. These rules are used to modify the logical structure, judgment conditions, constraint parameters, interaction objects, and environmental conditions of the initial inference chain. The entire strategy contains multiple non-conflicting transformation rules, which can be enabled individually or combined in any way.

[0078] Among them, the target-derived reasoning chain refers to multiple differentiated reasoning logic chains with boundary or complex working condition characteristics, which are obtained by using the initial reasoning chain as a base, completing logical transformation through at least one preset derivation strategy, and then undergoing legality verification and elimination of logically contradictory chains. Each derived chain corresponds to a traffic scenario that is different from the standard working condition and can be independently converted into a dedicated test scenario.

[0079] In practice, the initial inference chain is read, the chain is decomposed in a structured manner, and all internal components are extracted. This includes extracting all inference nodes, the fixed execution order of nodes, the boundary thresholds of decision parameters built into each node, the set of traffic participants involved in the chain, and the basic environmental constraints bound to the chain. Simultaneously, the logical dependencies between inference nodes and their branch decision conditions are recorded. Pre-configured offline derivation strategies are read, including branch derivation, parameter derivation, sequence derivation, object combination derivation, and environment derivation.

[0080] Optionally, depending on the testing requirements, you can choose to enable only a single type of transformation rule, or to superimpose and execute multiple transformation rules simultaneously to determine the combination of all transformation strategies used in generating the derivative link this time.

[0081] Furthermore, according to the selected derivation strategy, logical transformations are performed on the initial inference chain. For example, a new decision inference node may be inserted between any two adjacent inference nodes in the initial inference chain, adding multiple decision branches to the original linear main logic. Alternatively, the execution sequence of multiple inference nodes may be adjusted. These differentiated inference chains are then uniformly aggregated to form batch target derivation inference chains, which are used for subsequent conversion to generate test scenarios.

[0082] Optionally, after the derivation operation is completed, each generated temporary inference link is checked for logical conflicts. For example, if a link is found that violates common sense about driving, such as configuring a judgment node for an object when there is no corresponding traffic participant, or performing a deceleration decision before performing obstacle detection, it is directly marked as an invalid link and discarded, and will not proceed to the subsequent process.

[0083] S203. Based on the preset mapping relationship, convert each target-derived inference link into a corresponding test scenario, so as to test the model to be tested using the test scenario. The test scenario includes the scenario parameters corresponding to the inference node.

[0084] Among them, the preset mapping relationship refers to the standardized corresponding rule system that is pre-built and stored offline, which is used to establish the association between the inference logic layer and the simulation working condition layer, including the binding relationship between each inference node and the corresponding scenario variable.

[0085] Among them, scene parameters refer to entity configuration items obtained by assigning definite physical values ​​and simulation configuration values ​​to scene variables. They can be directly identified and loaded by autonomous driving simulation tools. Each set of scene parameters can be traced back to the original inference node that generated it.

[0086] Among them, the test scenario refers to a complete digital traffic condition formed by a combination of scenario parameters, including configuration information such as roads, traffic participants, environment, and vehicle movement. It can be imported into simulation tools to build a virtual traffic world and used as input material to supply the VLA model under test to complete robustness verification.

[0087] Specifically, each target-derived inference link is read one by one, all inference nodes within the link are analyzed, and the complete logical information carried by each node is extracted synchronously, including node type, built-in judgment conditions, derived modified parameter thresholds, traffic participants added or deleted in the link, environmental operating condition constraints bound to the link, etc., while recording the sequential logical dependencies between nodes in the entire link.

[0088] Furthermore, the pre-built offline mapping relationships are invoked. For each decomposed inference node, the set of scene variables specific to that node is retrieved to determine all the abstract variables required by the current inference logic. Combining the judgment conditions within the target-derived inference chain, the modified thresholds, newly added traffic objects, harsh environment settings, etc., each matched scene variable is assigned a corresponding physical value.

[0089] Optionally, the assignment process follows the variable association rules within the preset mapping relationship. If multiple inference nodes point to the same scene variable and give different values, the updated derived value within the derived inference link shall prevail.

[0090] Furthermore, the scenario parameters that have been assigned to all inference nodes under the entire derivation inference chain are summarized and integrated, and the basic fixed configurations necessary for simulation operation are supplemented to form a complete set of working condition configurations, which is the test scenario corresponding to the current chain.

[0091] Understandably, all the assembled test scenarios are sequentially imported into the autonomous driving simulation tool. The simulation tool builds a virtual traffic environment that evolves over time based on the complete set of scenario parameters, and continuously generates two types of multimodal input materials: virtual camera images and standardized driving language commands.

[0092] The image and language commands are synchronously sent to the VLA model under test. Test data such as the model's output reasoning text, driving action commands, and reasoning time are collected to complete the model verification process under a single test scenario. All test scenarios are then traversed sequentially to obtain full test results.

[0093] The testing method for the visual-language-action model provided in this application generates an initial inference link containing multiple inference nodes based on the decision logic of the model under test. The inference link is built using the model's own decision logic, ensuring that all subsequent test scenarios accurately correspond to the model's actual inference path. A preset derivation strategy is used to generate multiple target derived inference links based on the initial inference link, directly modifying the inference chain itself, actively creating boundaries, conflicts, and complex decision paths, and uncovering inference weaknesses that input disturbances cannot expose. A standardized derivation strategy automatically generates a massive number of differentiated inference links, multiplying the scale of test cases and improving test completeness. Derivation transformations can control scene complexity, enabling quantitative observation of the degree to which model inference performance decays with increasing scene complexity, thereby achieving a robustness quantitative evaluation. Based on a preset mapping relationship, each target derived inference link is converted into a test scenario including scene parameters corresponding to the inference nodes. This ensures that the construction of the test scenario corresponds to the model's internal inference process, thus more accurately covering key inference processes in dynamic traffic, achieving a hierarchical quantitative evaluation of the robustness and inference stability of the visual-language-action model and precise fault tracing.

[0094] As an optional implementation, based on the above embodiments, an initial inference chain is generated, including:

[0095] Obtain multiple target decision nodes and multi-source historical driving information of the decision logic of the model under test;

[0096] Multiple target decision nodes are sorted based on multi-source historical driving information to generate an initial inference link, where different driving scenarios correspond to different initial inference links.

[0097] Among them, the target decision node refers to the independent basic logic unit obtained after decomposing the model decision logic, and each node undertakes only a single type of reasoning task.

[0098] Among them, multi-source historical driving information refers to multi-dimensional prior materials in the driving field collected offline, including traffic regulations, real driving operation data collected from actual vehicles, various traffic accident cases, driving experience of experts in the field of autonomous driving, and standardized compliant driving reasoning text generated by large language models, which are used to correct and constrain node sorting logic.

[0099] Among them, driving scenarios refer to standard driving conditions that distinguish different roads, traffic participants, environments, and vehicle flow interaction characteristics, such as pedestrians crossing the street, adjacent vehicles changing lanes, and nighttime highways.

[0100] In practical implementation, the model's decision-making logic can be analyzed based on the output of the model under test, historical driving records, and rule base. Following the complete reasoning path from multimodal input to action output, all judgment steps within the model are broken down. Independent units are then segmented according to the boundaries of reasoning functions. Consecutive judgments with overlapping functions are integrated into the same unit, while independent judgment steps are separated into individual units. All indivisible units that perform a single reasoning task are selected as the entire target decision-making node.

[0101] Furthermore, multiple independent driving scenarios are categorized based on road type, environmental conditions, and core interactive traffic participants, such as pedestrian scenarios on urban roads, lane-changing scenarios on highways, and oncoming traffic scenarios in rainy weather. For any given driving scenario, traffic regulations, accident handling cases, and expert driving experience are retrieved as sorting criteria. All target decision nodes required for that scenario are matched, and the pre- and post-execution dependencies between nodes are determined, forming a scenario-specific node execution order. The target decision nodes, arranged in this order, are then sequentially linked together according to their dependencies, forming an initial inference chain corresponding to that driving scenario.

[0102] For example, in a pedestrian crossing scenario, the initial inference chain can sequentially connect pedestrian recognition, motion trend determination, safety assessment, and braking decision. In an intersection meeting scenario, the initial inference chain can sequentially connect oncoming vehicle recognition, traffic priority determination, lateral distance assessment, and yielding decision.

[0103] The testing method for the vision-language-action model provided in this application, on the one hand, anchors the model's internal real-world reasoning logic to ensure the test is targeted. On the other hand, it leverages multi-source real-world driving prior constraint link logic to balance traffic compliance and real-world authenticity; it generates benchmark links differentiated by scenario to achieve full coverage of multiple working conditions, thereby enabling subsequent tests to more accurately cover the model's key reasoning paths in different scenarios.

[0104] As an optional implementation, based on the above embodiments, the multiple inference nodes include an environmental perception node, a behavior prediction node, a safety assessment node, a behavior decision node, and a trajectory planning node;

[0105] The preset derivation strategies include at least one of the following: reasoning branch derivation, node sequence derivation, decision parameter derivation, object combination derivation, and environmental condition derivation.

[0106] Among them, the environmental perception node refers to the independent basic logical unit in the inference link responsible for collecting and identifying traffic environment elements. It mainly completes the identification and extraction of surrounding traffic targets and environmental information such as roads, vehicles, pedestrians, lanes, and weather, and outputs the position, size, and movement basic information of various traffic participants.

[0107] Among them, the behavior prediction node refers to the logical unit that receives traffic element information output by the environmental perception node, and predicts the potential behaviors of surrounding vehicles and pedestrians such as movement trends, lane changes, and crossings in the near future, and outputs the estimated movement trajectory and behavioral tendencies of each traffic participant.

[0108] Among them, the safety assessment node refers to a logical unit that quantifies the collision risk and safety margin between the vehicle and surrounding traffic participants based on perception results and behavior prediction results. It contains various distance, speed and visibility judgment thresholds and outputs high, medium and low driving risk conclusions.

[0109] Among them, the behavioral decision node refers to the logical unit that determines the macro driving operation intention of the vehicle based on the risk level output by the safety assessment. It includes basic driving behavior choices such as deceleration, acceleration, maintaining vehicle speed, changing lanes to the left or right, and parking, and outputs a determined vehicle control tendency.

[0110] Among them, the trajectory planning node refers to the logical unit that calculates the vehicle's driving path and speed change curve at each moment based on the operational intent determined by the behavioral decision, and outputs a continuous and executable temporal driving trajectory, which serves as the direct basis for the final action output of the VLA model.

[0111] Among them, reasoning branch derivation refers to the logical transformation rule that extends the linear main logic of the original reasoning link. It inserts a new judgment logic between any two reasoning nodes, adds multiple layers of judgment branches, and expands the single main line reasoning into a multi-branch parallel judgment structure.

[0112] Among them, node sequence derivation refers to the transformation rules that adjust the order of execution of nodes without modifying the internal judgment logic, parameters and output results of each inference node, thereby simulating unconventional inference processes such as inference timing misalignment, prediction before, and perception after.

[0113] Among them, decision parameter derivation refers to keeping the order of inference nodes, branch structure, traffic objects, and environmental conditions unchanged, and only modifying the transformation rules of boundary judgment thresholds such as speed, distance, and visibility used for risk judgment and behavior trade-off within the safety assessment and behavior decision nodes, in order to construct extreme boundary conditions.

[0114] Among them, object combination derivation refers to adjusting the transformation rules of the set of traffic participants involved in the interaction within the reasoning link. On the basis of the original, traffic objects such as pedestrians, following vehicles, non-motorized vehicles, and oncoming large vehicles can be added, or the original objects can be deleted, and corresponding perception and risk judgment logic for the objects can be added to form a multi-objective interaction scenario reasoning chain.

[0115] Among them, environmental condition derivation refers to replacing the basic environmental conditions bound to the inference link with the transformation rules. The initial link's default standard environment, such as sunny day, good lighting, and clear lane lines, is replaced with harsh environmental constraints such as rainy day, night, heavy fog, lane line wear, and backlighting / reflection, and the risk judgment logic of the safety assessment node is adapted simultaneously.

[0116] For example, in the inference branching implementation, the initial inference chain is as follows: detect the vehicle on the left, determine if the vehicle on the left is preparing to change lanes, determine a safe distance, and decide whether to slow down or maintain speed. The target-derived inference chain is as follows: after detecting the vehicle on the left, determine if it is approaching the lane line; otherwise, maintain the lane; if so, determine whether to change lanes. If it is changing lanes, determine a safe distance; otherwise, slow down.

[0117] Understandably, when actually generating a derivative link, one can choose any type of derivative strategy to perform the transformation, or two or more derivative strategies can be superimposed and executed synchronously to obtain a more complex multi-coupled derivative inference link.

[0118] The testing method for the visual-language-action model provided in this application constructs a regular and decomposable baseline reasoning link using five standardized reasoning nodes: perception, prediction, evaluation, decision-making, and planning. Then, through a derivative strategy covering all five dimensions, the baseline link is logically extended in multiple dimensions, forming a complete set of reasoning logic samples ranging from basic routine conditions to multi-coupling extreme conditions. The test scenarios maintain a correspondence with the actual decision-making link of the model, and continuously cover traffic situations of varying complexity.

[0119] As an optional implementation, based on the above embodiments, generating a target-derived inference chain includes at least one of the following steps:

[0120] Add a judgment and reasoning node to the initial reasoning chain to generate multiple reasoning branches;

[0121] Modify the decision parameters, boundary thresholds, or environmental conditions of the initial inference chain;

[0122] Adjust the execution order of the inference nodes in the initial inference chain;

[0123] Add or remove participant objects from the initial inference chain.

[0124] Among them, the judgment and reasoning node refers to an independent reasoning unit used to carry the condition judgment logic. It contains complete judgment conditions, judgment criteria, and subsequent execution paths corresponding to different judgment results.

[0125] Among them, reasoning branches refer to multiple independent logical paths formed by splitting newly added judgment and reasoning nodes, with different branches corresponding to the complete reasoning process under different judgment results.

[0126] Among them, the decision parameter boundary threshold refers to the critical value used for risk, vehicle speed, and vehicle distance determination within the safety assessment and behavioral decision-making nodes. It is the standard for distinguishing between safe working conditions and dangerous boundary working conditions.

[0127] Among them, environmental conditions refer to the external driving environment constraints bound to the inference link, including environmental constraint information such as illumination, rainfall, visibility, road surface adhesion coefficient, and lane line clarity.

[0128] Among them, the participants refer to all kinds of road traffic participants involved in the reasoning link, including the vehicle itself, the vehicle in front, the vehicle behind, non-motorized vehicles, pedestrians, and oncoming motorized vehicles.

[0129] Specifically, locate the midpoint between any two adjacent inference nodes in the initial inference chain and insert a completely new independent judgment inference node. Configure a complete set of binary or multi-factor judgment conditions for the new node, and split it into two or more independent logical paths based on whether the conditions are true or false.

[0130] For example, the inference node arrangement order, branch structure, and traffic participants are completely preserved, and only the decision parameter boundary thresholds are modified. For example, the safe following distance judgment threshold is increased or decreased, and the high-speed / low-speed boundary value is changed.

[0131] For example, the environmental condition-derived implementation replaces the basic environmental labels bound to the initial inference link, such as sunny day, high visibility, and dry road surface. It switches to adverse operating conditions such as rain, fog, nighttime, and backlighting, and simultaneously modifies the risk correction coefficient and visibility distance judgment threshold within the safety assessment node to adapt to this environment.

[0132] For example, a node sequence-derived implementation can be achieved by adjusting the behavior prediction node to be executed before the environment perception node, or by advancing the trajectory planning node to be executed before the behavior decision node.

[0133] For example, add a new participant object. Read the existing traffic participant list of the initial link and add interaction targets not yet included, such as pedestrians, following vehicles, non-motorized vehicles, and oncoming large vehicles. Simultaneously add environmental perception sub-logic, distance and risk judgment conditions adapted to this object. Remove participant objects. Remove some secondary traffic participants within the link, and simultaneously delete the perception and risk judgment reasoning logic associated with these participants.

[0134] The testing method for the visual-language-action model provided in this application comprehensively expands the basic inference link from four dimensions: decision branch structure, decision boundary, inference sequence, and traffic interaction object. The generated derivative link can make the local decision changes of the model under test explicit, so that scenarios with different risk levels and different interaction complexities can be expressed as executable links. This allows subsequent scenario transformation and model testing to revolve around the key decision path, improves the consistency between the inference link and the test scenario, and enhances the ability to represent the stability of model decision under complex traffic conditions.

[0135] As an optional implementation, based on the above embodiments, the preset mapping relationship includes the mapping relationship between each inference node and scene variables, and also includes:

[0136] Determine the set of scenario variables corresponding to each inference node in the target derivation inference chain based on the preset mapping relationship;

[0137] The association rules between scenario variables are determined based on the logical dependencies between each inference node.

[0138] Among them, scene variables refer to abstract descriptive quantities that only have semantic definitions and do not carry specific numerical values. They are used to determine the judgment elements in the corresponding inference nodes, such as pedestrian lateral offset, vehicle speed, and road surface adhesion coefficient.

[0139] Among them, the logical dependency relationship between inference nodes refers to the pre- and post-pre-precedence, branch triggering, and condition association relationships that naturally exist between nodes in the derived inference chain.

[0140] Among them, the association rules between scene variables refer to the variable constraint rules derived from the logical dependencies of nodes, including condition activation, numerical linkage, and mutual exclusion of values, which are used to constrain the assignment range and effective conditions of different scene variables.

[0141] Understandably, a mapping table between inference nodes and scene variables can be pre-established. This includes configuring variables such as pedestrian position, vehicle distance, occlusion level, and weather conditions for the environment perception node; configuring variables such as target movement direction, relative speed, and collision probability for the behavior prediction node; configuring variables such as safety assessment level, minimum safe distance, and braking threshold for the safety assessment node; configuring variables such as acceleration / deceleration commands, lane change intentions, and yielding status for the behavior decision node; and configuring variables such as path curvature, steering angle, and trajectory smoothness for the trajectory planning node.

[0142] Specifically, upon receiving the target derived inference chain, each chain is parsed sequentially, extracting all inference nodes within it. Simultaneously, the functional type, internal decision elements, newly added decision conditions resulting from the derived transformation, newly added traffic objects, and changed environmental constraints of each node are recorded. Pre-defined mapping relationships are retrieved, and for each parsed inference node, all pre-bound scene variables within the rules are searched. All retrieved scene variables are then uniformly aggregated to form a scene variable set specific to that inference node.

[0143] For example, when there is a dependency in the target-derived inference chain where pedestrian approach leads to an increase in safety assessment and triggers a deceleration decision, a linkage constraint can be established between the pedestrian position, vehicle distance, and safety assessment level. A mapping can also be established between the safety assessment level and the deceleration threshold, so that the deceleration command is activated when the safety assessment level reaches a preset range. When mutually exclusive relationships exist in the chain, such as lane-changing intention and emergency braking state not being able to occur simultaneously, mutually exclusive conditions can be written into the association rules to ensure that contradictory combinations are not formed during subsequent scenario generation. In practical applications, the mapping table, variable set, and association rules can be implemented by a rule engine, database retrieval module, or graph structure inference module; this application does not limit this.

[0144] The testing method for the visual-language-action model provided in this application first establishes a one-to-one correspondence between inference nodes and scene variables through static matching, and then constructs dynamic constraint rules for variables based on node logical dependencies. This achieves both precise binding of inference logic and simulation condition elements, ensuring traceability of faults, and eliminating logical conflicts in the conditions through variable constraints, thus guaranteeing that the scene closely matches real traffic patterns.

[0145] As an optional implementation, based on the above embodiments, the association rules include activation conditions, numerical linkage relationships, and mutual exclusion constraints, which are converted into test scenarios, including:

[0146] Based on association rules and semantic scene description text composed of scene variable sets, values ​​are assigned to the scene variables corresponding to each inference node to obtain the scene parameters corresponding to each inference node.

[0147] The "enabling condition" refers to the effectiveness constraints set based on the branches and temporal dependencies between inference nodes. It stipulates that a certain set of scenario variables are only allowed to participate in assignment and be written into the test scenario when the judgment condition of the preceding inference node is met. If the preceding judgment is not met, the corresponding variables are directly set to invalid and do not participate in the scenario construction.

[0148] Among them, numerical linkage relationship refers to defining the linkage constraint between the values ​​of variables in different scenarios. When the value of a certain environment or object scenario variable is changed, the value range of the corresponding scenario variables in the risk and distance categories is adjusted synchronously, replicating the objective law of parameter coordinated change in real traffic environment.

[0149] Mutual exclusion constraints refer to the stipulation that two or more sets of scenario variables cannot be assigned values ​​simultaneously or take effect simultaneously in the same test scenario, in order to avoid contradictory configurations that violate real-world traffic logic.

[0150] Among them, semantic scene description text refers to the textual scene description formed by integrating all scene variables matched by each target-derived inference link, including the semantic categories and logical relationships of scene variables.

[0151] Specifically, all association rules associated with the current derivation inference chain are retrieved, and the activation conditions, numerical linkage relationships between variables, and mutual exclusion constraints corresponding to each scenario variable are extracted. At the same time, the semantic scenario description text generated by integrating the scenario variable sets of each inference node is read, and all scenario variables covered by the entire chain, the inference nodes to which the variables belong, and the logical primary and secondary relationships between variables are extracted from the text.

[0152] Furthermore, the process iterates through all scenario variables in the semantic scenario description text, matching each one against its corresponding activation conditions for validity filtering. For any set of scenario variables, the logical judgment result corresponding to its preceding inference node is checked. If the preceding judgment condition is not met, the set of scenario variables is marked as invalid, and the assignment process is skipped. For all the filtered valid scenario variables, initial standard values ​​are assigned to the basic scenario variables based on the parameters and environmental derivation settings within the derived inference chain. The numerical linkage relationship is retrieved to identify variable groups with linkage binding. When the environmental variables are assigned values, the value range of the corresponding risk judgment variables is simultaneously corrected.

[0153] Optionally, after the initial assignment is completed, the mutual exclusion constraint relationship is retrieved for global verification. It checks whether there is a conflict where two sets of mutually exclusive scenario variables are simultaneously assigned valid values. If a conflict is detected, the core transformation logic of the derivation inference chain takes precedence, retaining the variable values ​​that are the focus of the derivation operation, and invalidating the other set of mutually exclusive variables.

[0154] After undergoing activation filtering, linked assignment, and mutual exclusion verification, all variables with definite compliant values ​​are organized into scenario parameters. These parameters are then grouped and aggregated according to inference nodes, with each inference node corresponding to a unique set of scenario parameters, establishing a one-to-one correspondence between scenario parameters and the original inference nodes.

[0155] In practical applications, semantic scene description text can be generated by a rule engine or manually edited and then converted into a structured intermediate representation by a parser. The parser can use a combination of lexical matching and semantic template matching to identify variables. For a given inference node, if the activation condition is not met, the scene variable corresponding to that node remains inactive. When multiple variables are constrained by numerical linkage relationships, linked values ​​are assigned according to preset proportions, threshold ranges, or function expressions. When a mutual exclusion constraint indicates that two variables cannot simultaneously take a certain state, the final value is determined through rollback, replacement, or recalculation.

[0156] The testing method for the visual-language-action model provided in this application embodiment can directly drive the assignment of scene variables by association rules, and the semantic scene description text can transform the inference link into an executable scene expression, thereby ensuring that the scene parameters corresponding to each inference node are consistent with the model decision logic, thereby improving the consistency and executability of test scene generation and enhancing the coverage of complex inference links.

[0157] As an optional implementation, based on the above embodiments, it further includes:

[0158] Test scenarios are categorized into levels based on the complexity of the reasoning chain for each objective to generate tiered test scenarios. Tiered test scenarios include at least two of the following: basic scenarios, branch decision scenarios, boundary condition scenarios, multi-object interaction scenarios, and complex environment scenarios.

[0159] Among them, the complexity of the target derivation inference link refers to the quantitative dimension that measures the difficulty of the reasoning logic and the degree of coupling of a single derivation inference link. It is determined by a combination of multiple dimensions, including the number of inference nodes in the link, whether new judgment branches are added, whether critical parameters are modified, the number of traffic participants, and the severity of the environment.

[0160] Among them, graded test scenarios refer to a set of test scenarios that are classified into layers according to their complexity from low to high, with the difficulty increasing sequentially between layers, forming a tiered test sample system.

[0161] The basic scenario (Level 1) refers to the test scenario obtained by transforming the initial inference link without any derivative transformations and retaining only the original standard logic. The inference link is a single linear, branchless structure, using standard parameter thresholds, a single traffic participant, and a good standard environment, resulting in the lowest complexity.

[0162] Among them, the branch decision scenario (Level 2) refers to the link transformation that only involves the derivation and transformation of reasoning branches. On the basis of the linear main logic, multiple layers of judgment branches are added, and there are multiple conditional selection logics. There are no limit parameters, multiple objects, and adverse environments superimposed, making the complexity higher than the basic scenario.

[0163] Among them, the boundary condition scenario (Level 3) refers to the link transformation that only involves the derivation and transformation of decision parameters. The judgment parameters such as risk, distance, and vehicle speed within the inference node are taken as critical limit values. Standard single linear logic, conventional traffic objects, and standard environments are used. The core difficulty lies in the judgment of critical parameters.

[0164] Among them, the multi-object interaction scenario (Level 4) refers to the link transformation that only involves object combination and derivation. Two or more new types of road traffic participants are added to form a mixed interaction logic of people and vehicles, and multiple vehicles. Conventional parameters, single linear branches, and standard environments are used to improve the interaction coupling.

[0165] Among them, the complex environment scenario (Level 5) refers to the link transformation derived from environmental conditions or superimposed with two or more types of derivative transformations; it includes harsh environments such as rain, fog, and backlight at night, and can simultaneously superimpose branches, boundary parameters, multi-object derivation, and multiple types of interference coupling superimposed, with the highest overall complexity.

[0166] In actual implementation, the complexity of each target-derived inference link is first quantified. The complexity can be calculated by weighting the number of inference nodes, the number of judgment branches, the number of boundary parameters, the number of participant objects, and the number of environmental interference factors. The calculation results are then mapped to low, medium, high, and ultra-high levels, and test scenario sets of different levels are generated accordingly.

[0167] For example, if the inference link has no derivative transformations and is only the original initial inference link, a basic scenario is generated. If the link only performs inference branch derivation without parameter, object, or environment derivation, a branch decision scenario is generated. If the link only performs decision parameter derivation without superimposing other derivation strategies, a boundary condition scenario is generated. If the link only performs object combination derivation without superimposing other derivation strategies, a multi-object interaction scenario is generated. If the link performs environment derivation, or simultaneously superimposes two or more types of derivation transformation strategies, including any two or more combinations of branches, boundary parameters, multiple objects, and harsh environments, it is uniformly classified as a complex environment scenario.

[0168] The testing method for the visual-language-action model provided in this application objectively classifies scene complexity from the underlying derived features of the inference chain, forming a five-level gradient test scenario from simple to complex and from single interference to multi-coupling interference. This enables quantitative evaluation of model inference performance across dimensions and difficulty levels. It can obtain the model's baseline performance under normal operating conditions and also expose inference vulnerabilities in branch decisions, critical boundaries, multi-vehicle interactions, and harsh coupling environments layer by layer. Simultaneously, layered testing optimizes simulation computing power allocation, reduces overall testing costs, and provides standardized and orderly test sample input for subsequent robustness degradation quantitative evaluation.

[0169] As an optional implementation, based on the above embodiments, it further includes:

[0170] Obtain the output of the model under test;

[0171] The robustness of model inference is evaluated based on the output results. The robustness evaluation of model inference includes inference chain consistency evaluation, consistency evaluation between inference results and behavioral decisions, and robustness decay evaluation under different complexity scenarios.

[0172] The output of the model under test refers to all the data that the model outputs after inputting multimodal materials from the simulation scene, including the thought chain reasoning text generated by the model, the final output driving behavior action instructions, and the time consumed by a single reasoning operation.

[0173] Among them, the reasoning chain consistency assessment refers to the evaluation dimension that compares the degree of matching between the thinking reasoning text output by the model itself and the logic of the original target-derived reasoning chain, and measures whether the internal thinking process of the model is consistent with the designed standard reasoning logic.

[0174] Among them, the consistency assessment between reasoning results and behavioral decisions refers to the evaluation dimension that verifies whether the reasoning logic generated by the model and the final output driving action are self-consistent and matches, and judges whether the model has a logical gap problem where the reasoning conclusion and the execution action contradict each other.

[0175] Among them, the robustness decay assessment under different complexity scenarios refers to the evaluation dimension that compares the changes in the inference accuracy of the model in low-complexity basic scenarios and high-complexity coupled scenarios based on gradient-based layered test scenarios, and quantifies the decline in model performance as the difficulty of the scenario increases.

[0176] Specifically, hierarchical gradient test scenarios are sequentially imported into the simulation tool. The simulation tool generates corresponding temporal visual images and standardized driving language commands, which are continuously fed into the VLA model. The complete thought chain reasoning text generated by the model for the current working condition is recorded, and the final driving behavior commands such as deceleration, lane changing, and stopping are extracted from the model's output. The response time consumed by the model to complete the entire set of inference operations in each scenario is recorded simultaneously. The reasoning text, behavior commands, and response times corresponding to the same gradient level and the same scenario are bound to the scenario number and stored uniformly to form a structured model output result dataset.

[0177] For example, retrieve the original target-derived reasoning link corresponding to the current test scenario, and disassemble all standard reasoning nodes and their logical sequence within the link. Semantically decompose the reasoning text of the thought chain output by the model, and extract the reasoning logic fragments related to perception, prediction, risk assessment, decision-making, and planning that appear in the text. Compare the matching situation of the text reasoning fragments with the standard logic of each node in the original reasoning link one by one, calculate the number of standard reasoning nodes that are completely covered in the model reasoning, and determine the degree of matching completeness by combining the total number of nodes in the link. For example, use Equation (1) for calculation.

[0178] (1)

[0179] in, This indicates the number of nodes in the thought chain reasoning text that match the original target and are derived from the reasoning chain. This represents the total number of nodes in the original target-derived inference link. Indicates the degree of matching.

[0180] For example, extract the reasoning conclusions and final behavioral instructions output by the model under the same test sample. Analyze the risk level, avoidance requirements, and safety judgment conclusions derived from the reasoning text, and then compare them with the actual driving actions output by the model. Check for logical contradictions, such as the reasoning text determining a high risk of collision, but the model outputting dangerous actions such as acceleration or maintaining the original speed. Count the number of samples with logical contradictions, calculate the percentage of samples where the reasoning and actions are consistent in the same level of scenario, and complete the consistency evaluation.

[0181] For example, the percentage of valid correct samples that meet the consistency standard of the model inference chain and have no contradiction between inference and action is statistically analyzed in each level of gradient scenario to obtain the basic accuracy corresponding to each complexity level. Using the accuracy of the lowest complexity basic scenario as the benchmark, the accuracy of higher complexity levels such as branch decision, boundary conditions, multi-object interaction, and complex environment is compared in turn. The decrease in accuracy from the basic scenario to the higher-level scenario is calculated to characterize the degree of decay of model inference performance as the scenario complexity increases. For example, Equation (2) is used for calculation.

[0182] (2)

[0183] in, For Level 1 scene accuracy, For the accuracy of Level n scenarios, This represents the robust decay rate.

[0184] The testing method for the visual-language-action (VLA) model provided in this application evaluates the model across three core dimensions: the completeness of its internal reasoning logic, the consistency between reasoning and actions, and the performance stability across complex scenarios, forming a comprehensive and multi-layered quantitative evaluation system. It can uncover deep and hidden logical flaws in reasoning and, through gradient scenarios, achieve quantitative comparative analysis of model robustness. The evaluation results can accurately pinpoint weak reasoning links, capability boundaries, and contradictory decision-making defects in the model, providing complete, objective, and reproducible data support for the iterative optimization and security verification of the VLA model.

[0185] As an optional implementation, based on the above embodiments, it further includes:

[0186] The test scenario containing scenario parameters is input into a preset simulation tool for scenario modeling. The preset simulation tool is used to construct a virtual traffic environment and perform time-series simulation based on the scenario parameters, generating virtual sensor images and driving semantic commands as inputs to the model under test.

[0187] Among them, the preset simulation tool refers to a simulation platform that is pre-selected and has the capability of virtual simulation for autonomous driving, supporting the construction of three-dimensional traffic environment, time-series dynamic simulation, and virtual sensor rendering output.

[0188] Scene modeling refers to the entire process by which simulation tools read a complete set of scene parameters, reconstruct the road network, various traffic participants, weather and lighting conditions, and vehicle movement trajectories, and build a complete virtual digital traffic world that combines static and dynamic elements.

[0189] Among them, the virtual traffic environment refers to the digital virtual driving scenario built by simulation tools based on scene parameters, which includes all traffic elements such as lanes, markings, pedestrians, motor vehicles, non-motor vehicles, rain and fog, day and night lighting, and can continuously evolve over time.

[0190] Among them, time-series simulation refers to the simulation tool advancing the virtual world frame by frame according to a fixed time step, continuously updating the position, speed, and attitude of the vehicle, surrounding vehicles, and pedestrians, simulating the continuous dynamic changes of the real world.

[0191] Virtual sensor images refer to the images rendered and output in real time by virtual vehicle cameras within the simulation tool. They simulate road visual images captured by real vehicle cameras, including all traffic targets and ambient lighting effects within the scene, and serve as the visual input for the VLA model.

[0192] Among them, driving semantic instructions refer to standardized fixed text-based language instructions that uniformly describe the basic requirements for safe driving. They serve as the fixed language input for the VLA model, and only the visual image is changed during the testing process to ensure single-variable control testing.

[0193] Specifically, the entire set of test scenarios, with hierarchical gradients and parameter assignments completed, is imported in batches into a preset simulation tool. The simulation tool then builds a 3D environment based on the parsed scene parameters. It generates a road network structure with corresponding lane numbers, directions, and markings according to road parameters. It generates traffic participant entities with corresponding numbers, initial positions, and initial speeds based on pedestrian and vehicle parameters. It configures global environmental effects such as rain, fog, nighttime backlighting, and road surface adhesion status based on environmental parameters. Simultaneously, it configures the installation position, shooting angle, and imaging parameters of the virtual camera, completely replicating all static and dynamic traffic elements corresponding to the current test scenario, thus completing the construction of the virtual traffic environment.

[0194] Furthermore, the simulation tool drives the virtual world to evolve sequentially frame by frame at fixed time intervals. Within each time step, it updates the positions, speeds, and directions of pedestrians and surrounding vehicles according to the preset motion logic of the scene parameters. If it subsequently receives action commands from the model under test, it synchronously updates the vehicle's trajectory and speed. Throughout the process, the virtual environment remains dynamically and continuously changing, replicating the interactive conditions of real roads changing over time.

[0195] Understandably, after each time-series simulation, a virtual camera is invoked to render the complete image of the current moment, outputting the virtual sensor image of the corresponding frame. Simultaneously, pre-defined standardized driving semantic commands are retrieved; these commands remain fixed throughout the process and do not change with scene switching. The virtual image generated at the same moment, along with the fixed semantic commands, is synchronously fed into the VLA model under test as input material for the model's inference calculation in this round. This process of time-series simulation, material generation, and model input is repeated until the entire test scenario simulation is completed.

[0196] The testing method for the visual-language-action model provided in this application embodiment can restore the test scene containing scene parameters into an executable virtual traffic environment within a preset simulation tool, and output visual and language input synchronized with the driving task, so that the model under test can be verified under a unified time sequence condition, thereby making the correspondence between the test scene and the model input clearer, and making the multimodal test results under complex traffic conditions easier for subsequent evaluation.

[0197] As an optional implementation, based on the above embodiments, it further includes:

[0198] Visual perturbations are added to the visual images output by the preset simulation tool, and text perturbations are added to the semantic instructions to obtain a multimodal adversarial scenario. The visual perturbations include image noise, lighting shift, and image blurring, while the text perturbations include semantic ambiguity and vague task descriptions.

[0199] Image noise refers to random noise interference superimposed on the pixel level of the original image, simulating image graininess caused by backlighting of the camera, hardware electromagnetic interference, and low light at night.

[0200] Among them, illumination shift refers to adjusting the overall brightness, contrast, color temperature and light and shadow distribution of the image to simulate real lighting change scenarios such as backlighting, sudden changes in light when entering and exiting tunnels, and strong light from streetlights at night.

[0201] Image blur refers to reducing the clarity of the image and weakening the edge contours of the target, simulating the blurring effect caused by fogging of car windows in rainy weather, dirty lenses, and motion blur of fast-moving images.

[0202] Semantic ambiguity refers to rewriting the original instruction statement structure, causing the same text to produce two or more completely different interpretations for driving, thus creating a conflict in instruction comprehension.

[0203] Among them, fuzzy task description refers to deleting or weakening the explicit driving constraints in the original instructions, removing clear operational boundaries, and generating driving text with vague descriptions and no clear execution standards.

[0204] Among them, the multimodal adversarial scenario refers to the entire test case after simultaneously superimposing visual perturbation and text perturbation processing, which includes both image distortion and semantic interference of language instructions, and is used to test the inference stability of the model under the coupling of two types of multimodal interference.

[0205] Optionally, for a single frame of original visual image, a single visual perturbation can be selected, or multiple visual perturbations can be superimposed and applied synchronously to the same image.

[0206] Specifically, when applying image noise, uniformly distributed noise interference is superimposed across the entire pixel area of ​​the image, replicating the grainy distortion effects of camera hardware interference and low-light imaging. When applying illumination shift, the global brightness of the image is uniformly adjusted, and the contrast of light and shadow in road and pedestrian areas is locally modified to simulate sudden changes in lighting conditions such as strong light, backlight, and alternating light and dark. When applying image blur, the edge details of traffic targets such as pedestrians, vehicles, and lane lines are weakened, reducing the overall resolution of the image and reproducing the blurring state caused by fogged car windows and dirty lenses. After perturbation processing, a perturbed visual image with image interference is generated.

[0207] Optionally, for the same set of original standard semantic instructions, one type of text perturbation can be used alone, or two types of text perturbations can be superimposed.

[0208] Specifically, when creating semantically ambiguous instructions, the word order of the original sentences is adjusted and key modifiers are replaced, allowing the text to simultaneously contain two completely opposite driving interpretation logics, resulting in ambiguous instructions with conflicting understanding. When creating fuzzy task description instructions, explicit speed, avoidance, and safety constraints in the original instructions are removed, simplifying the precise operation requirements and generating fuzzy driving text with broad expressions and no clear execution standards. After perturbation processing, perturbed text instructions with semantic interference are generated.

[0209] Understandably, images processed with visual perturbation are paired sequentially with semantic instructions that have undergone simultaneous text perturbation processing, forming a single-frame multimodal adversarial input. The entire set of sequentially perturbated images and corresponding perturbated texts together constitutes a complete multimodal adversarial scenario. This multimodal adversarial scenario can be superimposed onto any level of the basic test scenario within the gradient system, expanding the adversarial scenarios.

[0210] The testing method for the visual-language-action model provided in this application introduces common real-world interferences from both visual imaging and linguistic text dimensions, based on the original clear simulation materials. This expands the test input beyond single-modal variations to cover multimodal coupling mismatch scenarios, thereby more fully validating the model's recognition consistency, instruction understanding stability, and action output reliability in complex traffic environments. Since the types of disturbances directly correspond to the simulation output and semantic instructions, diverse adversarial examples can be constructed without altering the main structure of the simulation scene, and the test results more realistically reflect the model's robustness in the face of complex inputs.

[0211] As an optional implementation, based on the above embodiments, it further includes: obtaining the inference output results of the model to be tested at the safety assessment node and trajectory planning node; adjusting the scene parameters of the input preset simulation tool according to the inference output results to generate targeted adversarial sub-scenes.

[0212] Among them, the targeted adversarial sub-scenario refers to a locally adjusted simulation condition generated by dynamically modifying the original scenario parameters based on the erroneous reasoning conclusions of model security assessment and trajectory planning. It is specifically designed to amplify risk conflicts by targeting the reasoning weaknesses currently exposed by the model, and is used to target and uncover deep reasoning vulnerabilities.

[0213] Specifically, after the simulation tool generates multimodal input materials through time-series deduction and sends them to the VLA model under test, it captures the intermediate output content of the model's inference process. All intermediate conclusions of the safety assessment node are extracted separately, including the collision risk level determined by the model, the estimated safe following distance, and the environmental risk correction coefficient. Simultaneously, the output content of the trajectory planning node is extracted, including trajectory constraint information such as the model's planned driving speed, steering timing, following distance, and lane change path. The intermediate inference conclusions of the model's safety assessment and trajectory planning are compared with the actual working conditions of the current simulation environment to identify model inference deviations. The scene parameters input to the simulation tool are then modified according to the type of deviation.

[0214] Optionally, if the safety assessment node underestimates the collision risk and determines that the distance between the two vehicles is sufficient but the actual distance between the two vehicles is extremely small, the scenario parameters can be adjusted to reduce the lateral and longitudinal safety distance between the two vehicles, or the road visibility can be reduced and the speed of surrounding vehicles increased to further compress the safety margin.

[0215] Optionally, if the safety assessment node overestimates the risk or excessively judges the danger, the distance between traffic participants should be appropriately increased, the intensity of environmental interference should be reduced at the same time, and a low-risk benchmark control condition should be constructed.

[0216] Optionally, if the trajectory planning node outputs an unreasonable driving trajectory, such as planning close following in a high-speed scenario or rapid acceleration on a slippery road surface, then the scenario parameters are modified to increase the approaching speed of surrounding vehicles, shorten the following distance between the two vehicles, and reduce the road adhesion coefficient, thereby amplifying the degree of conflict between the trajectory planning scheme and the real environment.

[0217] Furthermore, the modified set of scenario parameters is re-input into the preset simulation tool. The simulation tool reconstructs the virtual traffic environment locally based on the updated parameters, updates the movement state of the traffic participants and environmental constraints, and generates a new time-series working condition designed only for the current model's inference defects, which is the targeted adversarial sub-scenario.

[0218] The testing method for the vision-language-action model provided in this application uses the output results of the safety assessment node and trajectory planning node as the basis for parameter reshaping. This allows the simulation scene to be reconstructed in a targeted manner around the deviation between the model's safety assessment judgment and trajectory generation, thereby ensuring that the generated sub-scenes correspond to the model's current decision-making weaknesses. Therefore, targeted adversarial sub-scenes can be directly used to verify the model's stability under high collision risk judgment, boundary trajectory planning, and complex interaction conditions, thereby improving the consistency between the test scene and the inference chain, and enhancing the coverage of long-tail driving scenarios.

[0219] As an optional implementation, based on the above embodiments, it further includes:

[0220] Logical verification is performed on the inference links derived from each target. Logical verification includes timing conflict verification, condition mutual exclusion verification, and semantic contradiction verification.

[0221] Among them, timing conflict verification refers to checking whether the execution order of reasoning nodes violates the objective driving process of perception, prediction, risk assessment, decision-making, and trajectory planning, and identifying timing disorder problems such as node reversal and pre-judgment followed by execution.

[0222] Among them, conditional mutual exclusion verification refers to checking whether the same link contains judgment conditions that cannot coexist in the same real traffic conditions. If two types of mutual exclusion conditions exist at the same time, the link is determined to be invalid.

[0223] Semantic contradiction verification refers to reviewing the complete reasoning semantic logic of the entire chain, judging whether the overall judgment process violates traffic regulations and common driving knowledge, and marking contradictions when there are self-contradictory judgment conclusions or non-compliant handling methods.

[0224] Understandably, the temporary inference chain after the derivation transformation is completed is read line by line, and all inference nodes, the preset execution order of nodes, the internal judgment conditions of each node, the traffic participants included in the chain, the bound environmental constraints, the judgment results corresponding to each branch are completely extracted, and the pre- and post-dependencies between nodes and the effective scope of various judgment conditions are recorded as review materials for the three types of logic verification.

[0225] Specifically, in accordance with the standard inference process specifications for real-world driving, the standard process is to first execute the environmental perception node, followed by the behavior prediction, safety assessment, behavior decision-making, and trajectory planning nodes in sequence. The order of all inference nodes in the current link is checked to determine if any subsequent inference nodes are executed prematurely or if any preliminary basic judgments are executed later. For example, if the order is reversed, such as making behavior decisions before environmental perception or completing trajectory planning before risk assessment, the link is directly determined to have a timing conflict and marked as invalid.

[0226] Furthermore, the entire decision condition and environmental / object constraints within the link are analyzed to identify combinations of conditions that are mutually exclusive. If two sets of incompatible constraints are used simultaneously in the link, then the conditions are considered mutually exclusive. For example, if the link simultaneously marks the road surface as dry and the environment as heavy rainfall, simultaneously determines that the scenario has no pedestrians but also includes the logic for pedestrians crossing the lane, and simultaneously sets two mutually exclusive states: the vehicle is stationary and the vehicle is traveling at high speed, all of these are considered mutually exclusive, and the link is marked as invalid.

[0227] Furthermore, the entire semantic reasoning process of the entire chain is thoroughly reviewed, and the overall reasoning logic is checked for consistency and compliance in conjunction with traffic regulations and conventional driving experience. On one hand, the judgment conclusions of preceding and following nodes are checked for contradictions; for example, if a preceding safety assessment determines a high collision risk, a subsequent behavioral decision node might directly output an acceleration operation. On the other hand, the entire handling logic is checked for violations of traffic regulations; for example, logic that determines no need to slow down or yield after recognizing a pedestrian violates common driving sense. If either of these situations occurs, a semantic contradiction is identified, and the chain is marked as invalid. Only reasoning chains that pass all three types of checks and have no logical conflicts are retained as the final usable target derived reasoning chains and sent to the subsequent mapping transformation and scene generation stages.

[0228] The testing method for the visual-language-action model provided in this application completes a full-dimensional screening of the derived links from three dimensions: execution order, working condition constraints, and overall reasoning semantics. The usability of the target derived reasoning links is improved, the links input in the scene generation stage are more consistent with real traffic logic, and the invalid test scenarios caused by logical errors are reduced, thereby improving the accuracy and stability of the model reasoning robustness assessment.

[0229] As an optional implementation, based on the above embodiments, it further includes:

[0230] A pre-trained reinforcement learning model is used to generate multiple target-derived inference links based on the initial inference link. The pre-trained reinforcement learning model is obtained by iterative training using the model inference robustness evaluation result as the reward signal.

[0231] In practical implementation, the initial inference link can be encoded as a graph structure or a sequence structure, and candidate derived inference links can be obtained by a pre-trained reinforcement learning model under the current policy. The candidate derived inference links are then input into the visual-language-action model to be tested for inference verification. After obtaining the corresponding output results, a robustness evaluation value is calculated. The evaluation value, after normalization, is used as a reward signal and fed back to the pre-trained reinforcement learning model to iteratively update the generation policy through policy gradient or other reinforcement learning optimization methods. This allows the model to gradually tend to generate target derived inference links that can cover weak inference nodes, complex interaction relationships, or changes in boundary conditions.

[0232] In one feasible approach, the pre-trained reinforcement learning model can be constructed using a combination of a graph neural network and a policy network. The graph neural network extracts node representations and topological relationships from the initial inference chain, while the policy network outputs the derivation method, node replacement method, or branch expansion method for the next inference node based on the representations. After multiple rounds of iterative training based on reward signals, the model can stably output multiple target-derived inference chains corresponding to the weak points in the decision-making of the model under test, thus providing a chain foundation for subsequent scene generation.

[0233] The testing method for the visual-language-action model provided in this application, through the aforementioned approach, eliminates the reliance on pre-defined human experience for the target-derived inference links. Instead, it adaptively generates links based on the robustness feedback of the model under test. This allows the derived inference links to more effectively cover unstable inference regions and increases the correlation between the test links and the model's actual defects. Simultaneously, the pre-trained reinforcement learning model obtained through iterative training continuously absorbs historical evaluation results, making the subsequently generated target-derived inference links more targeted and comprehensive, thereby improving the effectiveness and consistency of complex driving reasoning tests.

[0234] As an optional implementation, based on the above embodiments, it further includes:

[0235] A pre-defined graph neural network is used to model the graph structure of each inference node and the candidate scenario variables in the target derivation inference link, so that the pre-defined graph neural network outputs the set of scenario variables corresponding to each inference node in the target derivation inference link.

[0236] In practical implementation, the environmental perception nodes, behavior prediction nodes, safety assessment nodes, behavior decision nodes, and trajectory planning nodes in the target-derived inference chain can be encoded as graph nodes, and the candidate scene variables can be input as candidate attribute nodes into the graph neural network. Adjacency relationships are used to represent the logical dependencies between inference nodes and the adaptation relationships between inference nodes and scene variables. During message passing, the graph neural network aggregates node features and combines edge weights to complete relevance scoring, thereby outputting a set of scene variables corresponding to each inference node. This set can be used for subsequent scene parameter assignment and test scene generation. In practical applications, other graph neural network models can also be selected; this application does not limit this choice.

[0237] Understandably, the pre-defined graph neural network first structures the decision chain in the inference process, and then jointly represents the candidate scenario variables. By learning the node-variable associations under real driving semantics, it achieves automatic matching from inference nodes to scenario variables, thereby enabling the target-derived inference process to correspond to executable test scenario parameters.

[0238] The testing method for the visual-language-action model provided in this application uses graph neural networks to perform graph structure modeling on inference nodes and candidate scene variables, replacing the traditional static preset mapping relationship to achieve automatic matching of scene variable sets, and further improving the automation level and adaptability of the inference link to the scene variable conversion link.

[0239] As an optional implementation, after generating the initial inference chain based on the above embodiments, the method further includes:

[0240] The initial reasoning process is validated using pre-set expert tools to identify illogical reasoning links that contradict real driving logic. These illogical reasoning links are then corrected or eliminated. The pre-set expert tools include a traffic law database, a standard driving rules database, and a driving expert experience database. Specifically, these pre-set expert tools are standardized review tools that integrate multi-dimensional driving knowledge, serving as a pre-emptive quality control measure for the initial reasoning process.

[0241] In practical implementation, after generating the initial inference link, each inference node and the causal relationship between nodes are input into a preset expert tool for matching and verification. The traffic law library first performs constraint judgments on the yielding obligation, speed limit, following distance, and right-of-way relationship in the link. The standard driving rule library then judges the node timing, action continuity, and control switching rationality. The driving expert experience library supplements the judgment with empirical driving constraints in special weather, complex intersections, obscured targets, and multi-object interaction scenarios. If a certain inference link is judged to violate regulations, violate standard driving procedures, or conflict with empirical rules, it is marked as an unreasonable inference link, and some nodes in the link are replaced, rearranged, or have their parameters modified according to the conflict type. When the conflict cannot be restored to a reasonable link through local modification, the link is directly eliminated, and only the inference links that pass the verification continue to participate in subsequent derivation and scenario transformation.

[0242] The testing method for the visual-language-action model provided in this application, through the above processing, can make the reasoning link entering the subsequent scene generation stage closer to the real driving decision-making logic, reduce invalid tests caused by obvious erroneous reasoning, and improve the consistency between the test scenario and actual road behavior, thereby improving the credibility of the visual-language-action model reasoning stability assessment.

[0243] Figure 3 A flowchart illustrating a testing method for a visual-language-action model provided in another embodiment of this application is shown below. Figure 3 As shown, the testing method for the visual-language-action model provided in this embodiment includes the following steps:

[0244] S301. Obtain multiple target decision nodes and multi-source historical driving information of the decision logic of the model under test.

[0245] S302. Based on multi-source historical driving information, sort multiple target decision nodes to generate an initial inference link. The initial inference link includes multiple inference nodes, including environmental perception nodes, behavior prediction nodes, safety assessment nodes, behavior decision nodes, and trajectory planning nodes. Different initial inference links correspond to different driving scenarios.

[0246] S303. The initial reasoning chain is logically validated by a preset expert tool to identify unreasonable reasoning chains that violate real driving logic, and unreasonable reasoning chains are corrected or eliminated. The preset expert tools include a traffic law library, a standard driving rule library, and a driving expert experience library.

[0247] S304. A pre-trained reinforcement learning model is used to generate multiple target-derived inference links based on the initial inference link. The pre-trained reinforcement learning model is obtained by iterative training using the model inference robustness evaluation result as the reward signal.

[0248] S305. Using a preset derivation strategy, generate multiple target derivation inference links based on the initial inference link. The preset derivation strategy includes at least one of the following: adding judgment inference nodes to the initial inference link to generate multiple inference branches; modifying the decision parameter boundary thresholds or environmental conditions of the initial inference link; adjusting the execution order of the inference nodes of the initial inference link; adding or deleting participant objects of the initial inference link.

[0249] S306. Perform logical verification on each target-derived reasoning link. Logical verification includes timing conflict verification, condition mutual exclusion verification, and semantic contradiction verification.

[0250] S307. Determine the set of scene variables corresponding to each inference node in the target derivation inference chain according to the preset mapping relationship. The preset mapping relationship includes the mapping relationship between each inference node and the scene variables.

[0251] S308. A preset graph neural network is used to perform graph structure modeling on each inference node and candidate scene variable in the target derivation inference link, so that the preset graph neural network outputs the set of scene variables corresponding to each inference node in the target derivation inference link.

[0252] S309. Based on the logical dependencies between each inference node, determine the association rules between variables in each scenario. The association rules include activation conditions, numerical linkage relationships, and mutual exclusion constraints, and convert them into test scenarios.

[0253] S310. Based on association rules and semantic scene description text composed of scene variable sets, assign values ​​to the scene variables corresponding to each inference node to obtain the scene parameters corresponding to each inference node.

[0254] S311. Classify the test scenarios according to the complexity of the reasoning chain of each target to generate gradient test scenarios. Gradient test scenarios include at least two of the following: basic scenarios, branch decision scenarios, boundary condition scenarios, multi-object interaction scenarios, and complex environment scenarios.

[0255] S312. Input the test scenario containing scenario parameters into the preset simulation tool for scenario modeling. The preset simulation tool is used to construct a virtual traffic environment and perform time-series simulation based on the scenario parameters, and generate virtual sensor images and driving semantic commands.

[0256] S313. Add visual perturbations to the visual images output by the preset simulation tool and add text perturbations to the semantic instructions to obtain a multimodal adversarial scenario. The visual perturbations include image noise, lighting shift, and image blurring, while the text perturbations include semantic ambiguity and ambiguous task descriptions.

[0257] S314. Input the adversarial scenario into the model to be tested for testing.

[0258] S315. Obtain the output results of the model to be tested.

[0259] Execute S316 or execute S317:

[0260] S316. Based on the output results, evaluate the robustness of model inference. The robustness evaluation of model inference includes inference chain consistency evaluation, consistency evaluation between inference results and behavioral decisions, and robustness decay evaluation under different complexity scenarios.

[0261] S317. Adjust the scene parameters of the input preset simulation tool according to the inference output results to generate targeted adversarial sub-scenes, and execute S314.

[0262] It should be noted that the execution order of S304 and S305 is not important; the execution order of S307 and S308 is not important.

[0263] In this embodiment, the implementation method and technical effect of S301-S317 are similar to those of the corresponding solutions in the above embodiments, and will not be repeated here.

[0264] Figure 4 A schematic diagram of the structure of the testing device for the visual-language-action model provided in this application is shown below. Figure 5 As shown, the testing device 40 for the visual-language-action model provided in this embodiment includes a generation module 41 and a conversion module 42.

[0265] The generation module 41 is used to generate an initial inference link based on the decision logic of the model to be tested. The initial inference link includes multiple inference nodes. The generation module 41 is also used to generate multiple target derived inference links based on the initial inference link using a preset derivation strategy. The conversion module 42 is used to convert each target derived inference link into a corresponding test scenario according to a preset mapping relationship, so as to test the model to be tested using the test scenario. The test scenario includes scenario parameters corresponding to the inference nodes.

[0266] The testing device for the visual-language-action model provided in this embodiment can perform... Figure 2 and Figure 3 The implementation principles and technical effects of the methods shown are similar, and will not be repeated here.

[0267] Optionally, the generation module 41, when generating the initial inference link, is specifically used to: obtain multiple target decision nodes and multi-source historical driving information of the decision logic of the model to be tested; sort the multiple target decision nodes based on the multi-source historical driving information to generate the initial inference link, wherein different driving scenarios correspond to different initial inference links.

[0268] Optionally, the multiple inference nodes include environmental perception nodes, behavior prediction nodes, safety assessment nodes, behavior decision nodes, and trajectory planning nodes; the preset derivation strategies include at least one of inference branch derivation, node sequence derivation, decision parameter derivation, object combination derivation, and environmental condition derivation.

[0269] Optionally, the generation module 41, when generating the target derived inference link, is specifically used to: add judgment inference nodes to the initial inference link to generate multiple inference branches; modify the decision parameter boundary thresholds or environmental conditions of the initial inference link; adjust the execution order of the inference nodes of the initial inference link; and add or delete participant objects of the initial inference link.

[0270] Optionally, the testing device for the visual-language-action model provided in this embodiment further includes a determination module.

[0271] Correspondingly, the preset mapping relationship includes the mapping relationship between each inference node and the scene variable. The determination module is used to determine the set of scene variables corresponding to each inference node in the target derivation inference chain according to the preset mapping relationship; and to determine the association rules between each scene variable based on the logical dependency relationship between each inference node.

[0272] Optionally, the association rules include activation conditions, numerical linkage relationships, and mutual exclusion constraints. When converting to a test scenario, the conversion module 42 is specifically used to: assign values ​​to the scene variables corresponding to each inference node based on the association rules and the semantic scene description text composed of a set of scene variables, so as to obtain the scene parameters corresponding to each inference node.

[0273] Optionally, the testing device for the visual-language-action model provided in this embodiment further includes a segmentation module.

[0274] Correspondingly, a segmentation module is used to classify test scenarios according to the complexity of each target-derived inference link to generate gradient test scenarios. Gradient test scenarios include at least two of the following: basic scenarios, branch decision scenarios, boundary condition scenarios, multi-object interaction scenarios, and complex environment scenarios.

[0275] Optionally, the testing device for the visual-language-action model provided in this embodiment further includes an acquisition module and an evaluation module.

[0276] Correspondingly, the acquisition module is used to acquire the output results of the model to be tested; the evaluation module is used to evaluate the robustness of model inference based on the output results. The robustness evaluation of model inference includes inference chain consistency evaluation, consistency evaluation between inference results and behavioral decisions, and robustness decay evaluation under different complexity scenarios.

[0277] Optionally, the testing device for the visual-language-action model provided in this embodiment further includes an input module.

[0278] Correspondingly, the input module is used to input the test scenario containing scene parameters into the preset simulation tool for scene modeling. The preset simulation tool is used to construct a virtual traffic environment and perform time-series simulation based on the scene parameters, generating virtual sensor images and driving semantic commands as inputs to the model to be tested.

[0279] Optionally, the testing device for the visual-language-action model provided in this embodiment further includes a perturbation module.

[0280] Correspondingly, the perturbation module is used to add visual perturbations to the visual images output by the preset simulation tool and text perturbations to the semantic instructions in order to obtain a multimodal adversarial scenario. The visual perturbations include image noise, illumination shift, and image blurring, while the text perturbations include semantic ambiguity and ambiguous task descriptions.

[0281] Optionally, the testing device for the visual-language-action model provided in this embodiment further includes an adjustment module.

[0282] Correspondingly, the acquisition module is also used to acquire the inference output results of the model under test at the safety assessment node and trajectory planning node; the adjustment module is used to adjust the scene parameters of the input preset simulation tool according to the inference output results in order to generate targeted adversarial sub-scenes.

[0283] Optionally, the testing device for the visual-language-action model provided in this embodiment further includes a verification module.

[0284] Correspondingly, the verification module is used to perform logical verification on each target-derived inference link. The logical verification includes timing conflict verification, condition mutual exclusion verification, and semantic contradiction verification.

[0285] Optionally, the generation module 41 is also used to generate multiple target-derived inference links based on the initial inference link using a pre-trained reinforcement learning model, wherein the pre-trained reinforcement learning model is obtained by iteratively training with the model inference robustness evaluation result as the reward signal.

[0286] Optionally, the determining module is also used to perform graph structure modeling on each inference node and candidate scene variable in the target derivation inference link using a preset graph neural network, so that the preset graph neural network outputs the set of scene variables corresponding to each inference node in the target derivation inference link.

[0287] Optionally, the verification module is also used to verify the logical rationality of the initial reasoning link through preset expert tools, so as to identify unreasonable reasoning links that violate the logic of real driving, and to perform correction or elimination processing on unreasonable reasoning links. The preset expert tools include a traffic law library, a standard driving rule library, and a driving expert experience library.

[0288] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes a processor 51 and a memory 52. ​​The processor 51 and the memory 52 are connected via a bus and communicate with each other.

[0289] In the specific implementation process, the processor 51 executes the computer execution instructions stored in the memory 52, causing the processor 51 to perform the above-described method.

[0290] The specific implementation process of processor 51 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0291] In the above embodiments, it should be understood that the processor 51 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0292] The memory 52 may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0293] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0294] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0295] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0296] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0297] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0298] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0299] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0300] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0301] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0302] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0303] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A testing method for a visual-language-action model, characterized in that, The method includes: An initial inference chain is generated based on the decision logic of the model to be tested, and the initial inference chain includes multiple inference nodes; A pre-defined derivation strategy is used to generate multiple target derivation inference links based on the initial inference link; Each target-derived inference link is converted into a corresponding test scenario according to a preset mapping relationship, so as to test the model to be tested using the test scenario. The test scenario includes scenario parameters corresponding to the inference node.

2. The method according to claim 1, characterized in that, Generate the initial inference chain, including: Obtain multiple target decision nodes and multi-source historical driving information of the decision logic of the model under test; Multiple target decision nodes are sorted based on multi-source historical driving information to generate an initial inference link, where different driving scenarios correspond to different initial inference links.

3. The method according to claim 1, characterized in that, Multiple reasoning nodes include environmental perception nodes, behavior prediction nodes, safety assessment nodes, behavior decision-making nodes, and trajectory planning nodes; The preset derivation strategies include at least one of the following: reasoning branch derivation, node sequence derivation, decision parameter derivation, object combination derivation, and environmental condition derivation.

4. The method according to claim 3, characterized in that, Generate the target-derived inference chain, including at least one of the following steps: Add a judgment and reasoning node to the initial reasoning chain to generate multiple reasoning branches; Modify the decision parameters, boundary thresholds, or environmental conditions of the initial inference chain; Adjust the execution order of the inference nodes in the initial inference chain; Add or remove participant objects from the initial inference chain.

5. The method according to claim 1, characterized in that, The pre-defined mapping relationships include the mapping relationships between each inference node and scene variables, and also include: Determine the set of scenario variables corresponding to each inference node in the target derivation inference chain based on the preset mapping relationship; The association rules between scenario variables are determined based on the logical dependencies between each inference node.

6. The method according to claim 5, characterized in that, Association rules include activation conditions, numerical linkage relationships, and mutual exclusion constraints. These are converted into test scenarios, including: Based on the association rules and the semantic scene description text composed of a set of scene variables, values ​​are assigned to the scene variables corresponding to each inference node to obtain the scene parameters corresponding to each inference node.

7. The method according to claim 1, characterized in that, Also includes: Test scenarios are categorized into levels based on the complexity of the reasoning chain for each objective to generate gradient test scenarios. The gradient test scenarios include at least two of the following: basic scenarios, branch decision scenarios, boundary condition scenarios, multi-object interaction scenarios, and complex environment scenarios.

8. The method according to claim 1, characterized in that, Also includes: Obtain the output of the model under test; The robustness of model inference is evaluated based on the output results. The robustness evaluation of model inference includes inference chain consistency evaluation, consistency evaluation between inference results and behavioral decisions, and robustness decay evaluation under different complexity scenarios.

9. The method according to claim 1, characterized in that, Also includes: The test scenario containing scenario parameters is input into a preset simulation tool for scenario modeling. The preset simulation tool is used to construct a virtual traffic environment and perform time-series simulation based on the scenario parameters, and generate virtual sensor images and driving semantic commands as inputs to the model to be tested.

10. The method according to claim 9, characterized in that, Also includes: Visual perturbations are added to the visual images output by the preset simulation tool, and text perturbations are added to the semantic instructions to obtain a multimodal adversarial scenario. The visual perturbations include image noise, illumination shift, and image blurring, and the text perturbations include semantic ambiguity and ambiguous task descriptions.

11. The method according to claim 9, characterized in that, Also includes: Obtain the inference output results of the model under test at the safety assessment node and trajectory planning node; Adjust the scene parameters of the input preset simulation tool based on the inference output results to generate targeted adversarial sub-scenes.

12. The method according to claim 3, characterized in that, Also includes: Logical verification is performed on each target-derived reasoning link, including timing conflict verification, condition mutual exclusion verification, and semantic contradiction verification.

13. The method according to claim 8, characterized in that, Also includes: A pre-trained reinforcement learning model is used to generate multiple target-derived inference links based on the initial inference link. The pre-trained reinforcement learning model is obtained by iterative training using the model inference robustness evaluation result as the reward signal.

14. The method according to claim 5, characterized in that, Also includes: A pre-defined graph neural network is used to model the graph structure of each inference node and the candidate scenario variables in the target derivation inference link, so that the pre-defined graph neural network outputs the set of scenario variables corresponding to each inference node in the target derivation inference link.

15. The method according to claim 1, characterized in that, After generating the initial inference chain, the following is also included: The initial reasoning chain is logically validated by pre-set expert tools to identify unreasonable reasoning chains that violate real driving logic, and unreasonable reasoning chains are corrected or eliminated. The pre-set expert tools include a traffic law library, a standard driving rule library, and a driving expert experience library.

16. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-15.

18. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-15.

Images