Method, device and equipment for reasoning visual language model for automatic driving

Abstract

The invention belongs to the technical field of automatic driving, and particularly relates to a reasoning method, device and equipment of a visual language model for automatic driving. The method comprises the following steps: acquiring a token combination corresponding to a memory of a moment before a target moment and the probability of each token in the token combination in the visual language model; if the probabilities of all the tokens are greater than a preset threshold value, taking the memory of the previous moment as the memory of the target moment, if the tokens lower than the preset threshold value exist, segmenting the token combination into logic units, checking whether the tokens lower than the preset threshold value exist in each logic unit, and if the tokens lower than the preset threshold value exist in each logic unit, taking the memory of the previous moment as the memory of the target moment; if yes, refreshing the logic unit, and taking the memory of the previous moment after refreshing as the memory of the target moment; and based on the multi-view image flow and the memory of the target moment, utilizing the visual language model to generate an inference state and a meta-action for determining an automatic driving track.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of autonomous driving, and particularly relates to a reasoning method, apparatus, and device for a vision language model for autonomous driving. Background Art

[0002] Autonomous driving often adopts an end-to-end solution, that is, directly mapping sensor inputs to driving actions, integrating perception, decision-making, and control. Vision Language Models (VLMs) decompose complex scenes into interpretable intermediate reasoning steps and high-level meta-actions through the Chain-of-Thought (CoT) reasoning that combines visual perception and context knowledge, so as to guide conventional control models, which makes VLMs good at understanding complex or rare autonomous driving scenarios.

[0003] However, due to the high computational requirements of the CoT process, the single-frame inference time of VLM often exceeds several seconds, making it difficult to meet the real-time requirements of dynamic environments. Existing acceleration methods focus on reducing the spatial or temporal redundancy of input data (such as pruning image tokens or skipping frames based on pixel similarity), but ignore the key bottleneck - the redundant reasoning process. For example, repeating the calculation of CoT outputs for consecutive similar frames. Summary of the Invention

[0004] Embodiments of the present disclosure propose a reasoning scheme for a vision language model for an autonomous driving system to reduce redundant reasoning steps in the CoT reasoning of the vision language model.

[0005] The first aspect of the embodiments of the present disclosure provides a reasoning method for a vision language model for an autonomous driving system, including:

[0006] Obtaining a token combination corresponding to the memory at the previous moment of the target moment and the probability of each token therein in the vision language model, wherein the memory is not an empty set, and the memory at the target moment includes the reasoning state and meta-action at the previous moment of the target moment;

[0007] If the probabilities of all the tokens are greater than a preset threshold, then use the memory at the previous moment as the memory at the target moment. If there are tokens lower than the preset threshold, then divide the token combination into logical units, check whether there are tokens lower than the preset threshold for each logical unit. If there are, then refresh the logical unit, and use the refreshed memory at the previous moment as the memory at the target moment;

[0008] Based on the multi-view image stream and memory at the target moment, use the vision language model to generate a reasoning state and meta-action for determining an autonomous driving trajectory.

[0009] In some embodiments of the present disclosure, obtaining the token combination corresponding to the moment before the target moment and the probability of each token in the vision - language model includes:

[0010] Concatenate the multi - perspective image stream at the target moment and the memory at the previous moment as the context, input it into the vision - language model, and obtain the token combination for generating the inference state and meta - action and the probability of each token therein.

[0011] In some embodiments of the present disclosure, the method further includes:

[0012] Set different preset thresholds for different tokens based on attention weighting.

[0013] In some embodiments of the present disclosure, setting different preset thresholds for different tokens based on attention weighting includes:

[0014] Set different preset thresholds for different tokens based on the following formula

[0015] Θ' = Θ * t s / A a , where

[0016] Θ' is the preset threshold of the current token s, Θ is the standard preset threshold, t s is the cross - layer average attention score of the current token s; A a is the average attention score obtained from the offline test of all tokens in the NuScenes dataset.

[0017] In some embodiments of the present disclosure, refreshing the logical unit includes:

[0018] Identify the first invalid token in the logical unit, where the invalid token refers to a token with a probability lower than the preset threshold.

[0019] Retain the KV caches corresponding to all tokens before the invalid token, and regenerate KV for all tokens after the invalid token;

[0020] Connect the retained KV caches and the newly generated KV to refresh the logical unit.

[0021] In some embodiments of the present disclosure, regenerating KV for all tokens after the invalid token includes:

[0022] Use the retained KV cache as the initial context, and continue to generate new tokens and the KV corresponding to the new tokens using the vision - language model.

[0023] In some embodiments of the present disclosure, refreshing the logical unit includes:

[0024] If there are a consecutive preset number of invalid tokens in the logic unit, all tokens of the logic unit are regenerated.

[0025] In some embodiments of the present disclosure, the preset number is 3.

[0026] A second aspect of the embodiments of the present disclosure provides an inference device for a vision language model of an autonomous driving system, including:

[0027] An acquisition module, configured to acquire a token combination corresponding to the memory at the moment before the target moment and the probability of each token therein in the vision language model, wherein the memory is not an empty set, and the memory at the target moment includes the inference state and the meta-action at the moment before the target moment;

[0028] A refresh module, configured to, if the probabilities of all the tokens are greater than a preset threshold, use the memory at the previous moment as the memory at the target moment, and if there are tokens lower than the preset threshold, divide the token combination into logic units, check whether there are tokens lower than the preset threshold for each logic unit, and if so, refresh the logic unit and use the refreshed memory at the previous moment as the memory at the target moment;

[0029] An inference module, configured to generate an inference state and a meta-action for determining an autonomous driving trajectory based on the multi-view image stream and the memory at the target moment by using the vision language model.

[0030] A third aspect of the embodiments of the present disclosure provides an inference device for a vision language model of an autonomous driving system, including a memory and a processor,

[0031] The memory is configured to store a computer program;

[0032] The processor is configured to, when executing the computer program, implement the inference method for the vision language model of the autonomous driving system according to the first aspect of the present disclosure.

[0033] In summary, the inference method, device, and device for the vision language model of the autonomous driving system provided by the embodiments of the present disclosure adaptively reuse historical inference outputs through semantic consistency detection, reducing redundant inference steps; and by designing a hierarchical verification strategy from coarse-grained to fine-grained, combining global semantic checking and fine-grained token pruning, dynamically updating invalid inference segments, and avoiding full-scene regeneration; in particular, introducing an attention-aware threshold calibration method to dynamically adjust the verification criteria according to token importance, ensuring strict verification of key entities (such as "pedestrians" and "vehicles"), balancing safety and efficiency, thereby improving the inference speed of the vision language model. Description of the Drawings

[0034] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as imposing any limitation on the present disclosure. In the drawings:

[0035] Figure 1 is a schematic diagram of a memory-driven framework for accelerating visual language model inference in autonomous driving proposed by the present disclosure;

[0036] Figure 2 is a schematic diagram of a computer system applicable to the present disclosure;

[0037] Figure 3 is a flowchart of a method for inferring a visual language model for an autonomous driving system according to some embodiments of the present disclosure;

[0038] Figure 4 is a schematic diagram of an apparatus for inferring a visual language model for an autonomous driving system according to some embodiments of the present disclosure;

[0039] Figure 5 is a schematic diagram of a device for inferring a visual language model for an autonomous driving system according to some embodiments of the present disclosure. Detailed implementation manners

[0040] In the following detailed description, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those of ordinary skill in the art that the present disclosure can be practiced without these details. It should be understood that the terms "system", "apparatus", "unit" and / or "module" used in the present disclosure are a way to distinguish different components, elements, parts or assemblies at different levels in a sequential arrangement. However, if other expressions can achieve the same purpose, these terms can be replaced by other expressions.

[0041] It should be understood that when a device, unit or module is referred to as being "on", "connected to" or "coupled to" another device, unit or module, it can be directly on the other device, unit or module, connected or coupled to or communicating with the other device, unit or module, or there may be intermediate devices, units or modules, unless the context clearly indicates an exceptional situation. For example, the term "and / or" used in the present disclosure includes any and all combinations of one or more of the related listed items.

[0042] The terms used in this disclosure are for the purpose of describing particular embodiments only and are not intended to limit the scope of the disclosure. As used in the specification and claims of this disclosure, unless the context clearly dictates otherwise, the words "a," "an," "one," and / or "the" are not intended to refer to the singular and may include the plural. In general, the terms "comprising" and "including" are used to indicate the inclusion of the specifically identified features, integers, steps, operations, elements, and / or components, and such terms do not preclude the inclusion of other features, integers, steps, operations, elements, and / or components.

[0043] Referring to the following description and the accompanying drawings, these or other features and characteristics of the disclosure, the operating methods, the functions of the relevant elements of the structure, the combination of parts, and the economy of manufacture can be better understood, where the description and the drawings form a part of the specification. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of the disclosure. It is understood that the drawings are not drawn to scale.

[0044] A variety of structural diagrams are used in this disclosure to illustrate various variations according to the embodiments of the disclosure. It should be understood that the structures described above or below are not used to limit the disclosure. The scope of the disclosure is defined by the claims.

[0045] As a transformative technology, autonomous driving is expected to achieve safer and more efficient transportation. Modern autonomous driving often adopts an end-to-end approach, which directly maps sensor inputs to driving actions, integrating perception, decision-making, and control. Vision Language Models (VLMs) decompose complex scenarios into interpretable intermediate reasoning steps and high-level meta-actions by combining visual perception with Chain-of-Thought (CoT) reasoning of contextual knowledge to guide conventional control models, which makes VLMs good at understanding complex or rare driving scenarios. The intermediate reasoning generated by VLMs includes: descriptions of traffic, weather, and road conditions; assessments of the behavior of surrounding objects; and predictions of their impact on the host vehicle. Based on these outputs, VLMs generate high-level meta-actions (such as steering decisions "left / straight / right" and speed adjustments "fast / normal / slow / stop"). VLMs do not predict precise trajectories or signals to avoid increasing decision complexity and reducing interpretability. At the same time, the control model runs at a higher frequency, converting the meta-actions into specific control instructions (steering angle, throttle value) to manage the real-time trajectory of the vehicle. By decoupling the slow high-level reasoning from the fast low-level execution, the system achieves a balance between deep contextual understanding and real-time response capabilities.

[0046] However, due to the high computational requirements of the chain-of-thought process, the single-frame inference time of VLM often exceeds several seconds, making it difficult to meet the real-time requirements of dynamic environments. Existing acceleration methods focus on the spatial or temporal redundancy of input data (such as pruning image tokens or skipping frames based on pixel similarity), but ignore the key bottleneck - the redundant inference process. For example, repeatedly calculating the CoT output for consecutive similar frames is like repeatedly reading an unchanged paragraph.

[0047] In view of this, the present disclosure proposes an output-driven reuse framework without training, which realizes a transformation of the redundancy utilization paradigm. Different from existing methods (reducing calculations by pruning inputs or shallow features, such as discarding low-importance visual tokens or skipping frames based on pixel similarity), the present disclosure utilizes the semantic consistency of the visual language model's own inferences in consecutive frames to evaluate which parts of the previous output are still valid. This output-centered approach treats the model's chain of thought as a memory resource that can be selectively retained, rather than an object to be regenerated each time. The memory-driven framework for accelerating visual language model inference in autonomous driving proposed by the present disclosure is as Figure 1 shown.

[0048] Figure 2 is a schematic diagram of a computer system applicable to the present disclosure. Figure 2 The computer system shown includes an inference server that is data-connected to multiple image sensors and an autonomous driving vehicle. The inference server generates an inference state and a meta-action for generating the trajectory of the autonomous driving vehicle based on the image data of the autonomous driving vehicle acquired by the image sensors in combination with the historical inference chain of thought of the autonomous driving vehicle.

[0049] The image sensors can be various cameras and video cameras. The present disclosure acquires a multi-view image stream of the autonomous driving vehicle and its surrounding environment based on multiple image sensors. In particular, the image sensors can be multiple cameras deployed on the autonomous driving vehicle at a preset angle.

[0050] The inference server is deployed with a visual language model. The visual language model receives a multi-view image stream It simultaneously captured by multiple on-vehicle cameras, and through chain-of-thought processing, outputs {R t ,A t}, where R t is the inference state (textual scene description, such as traffic conditions, object behaviors, etc.), and A t is the meta-action (high-level decisions such as steering or speed adjustment) for the downstream model to generate the trajectory. The inference server can be any one of a single machine, a cluster, or a distributed server. In particular, the inference server can be a server deployed in the autonomous driving vehicle.

[0051] Figure 3The figure is a flowchart of an inference method for a vision - language model for an autonomous driving system as shown in some embodiments of the present disclosure. In some embodiments, the inference method for the vision - language model for the autonomous driving system is executed by Figure 2 the inference server shown below, and the method includes the following steps:

[0052] S310, obtaining, in the vision - language model, a token combination corresponding to the memory at the moment before the target moment and the probability of each token therein, wherein the memory is not an empty set, and the memory at the target moment includes the inference state and the meta - action at the moment before the target moment.

[0053] Define the memory M t as the thought - chain reasoning and meta - action verified in the previous time step. If the inference state R t-1 at time step t - 1 matches the multi - perspective image stream It at time step t, M t will include all the tokens in the inference state R t-1 and the meta - action A t -1 at time step t - 1, otherwise M t is an empty set.

[0054] Intuitively, M t , as a memory carrier, will retain the historical inference state that matches the current frame. Based on the characteristics of the driving scenario, the following conclusions can be drawn for the inference state R t :

[0055] (1) The change in the inference state mainly corresponds to significant semantic changes in the scene, rather than minor pixel differences;

[0056] (2) Scene changes usually only affect a subset of the inference state, that is, the difference between R t and R t-1 is sparse and local (for example, a new statement "pedestrian appears" is added, and the rest of the description remains unchanged). These characteristics reflect the way human perception works: we notice and describe differences, rather than re - describing the entire scene from scratch. By leveraging these characteristics, our memory mechanism retains the stable context and only updates the necessary parts.

[0057] Based on the above definition, the token combination corresponding to the memory at the moment before the target moment and the probability of each token therein can be obtained from the vision - language model. Specifically:

[0058] Concatenate the current image I t with the cached M t-1 to form the input context, and monitor the token probability ∏ in the pre - filling stage of the inference framework shown in Figure 1 . The size of the probability matrix is (|I t |+|M t-1|)×|V|, where V is the vocabulary of VLM.

[0059] S320. If the probabilities of all the tokens are greater than a preset threshold, the memory at the previous moment is taken as the memory at the target moment. If there are tokens lower than the preset threshold, the token combination is split into logical units, and each logical unit is checked to see if there are tokens lower than the preset threshold. If so, the logical unit is refreshed, and the refreshed memory at the previous moment is taken as the memory at the target moment.

[0060] First, a global check is performed on whether the memory can be reused. Specifically, if the probability of each token in the token combination corresponding to M t-1 is greater than the threshold Θ, then M t-1 is regarded as M t , achieving the reuse of the complete memory. This method that requires no additional training integrates the process of reusability verification into model inference, avoiding additional network overhead.

[0061] In the above judgment, Θ is an empirical threshold obtained from offline testing on the NuScenes dataset. Since different tokens have different importance, some embodiments of the present disclosure adopt an attention-weighted threshold mechanism. A more stringent standard is imposed on key entities (such as "vehicle" and "pedestrian"): Θ' = Θ * t s / A a , where Θ' is the preset threshold for the current token s, Θ is the standard preset threshold, t s is the cross-layer average attention score of the current token s; A a is the average attention score of all tokens obtained from offline testing on the NuScenes dataset. This means that high-attention tokens need to meet a higher threshold, while low-importance tokens allow small probability fluctuations to avoid over-constraining the grammar.

[0062] Then, it is checked whether local reuse is possible: If the global check fails, that is, there are tokens in M t-1 whose probabilities are less than the preset threshold Θ', then the memory is analyzed in finer granularity. The present disclosure does not discard the entire memory, but only trims the invalid part. Mt-1 is split into logical units (sentences or clauses), and the token probabilities are evaluated paragraph by paragraph. A logical unit with a token probability less than the preset threshold Θ is considered invalid. For example, if M t-1 contains "The pedestrian is moving", and the probability of the word "moving" decreases (maybe the pedestrian has stopped), then it is truncated to "The pedestrian is...". This method retains the valid part, avoids regenerating unchanged descriptions, and reduces the decoding computational complexity in direct proportion to the proportion of trimmed tokens.

[0063] After determining the valid part of the historical memory, the present disclosure integrates it into the model inference process as memory retention and memory update:

[0064] Memory retention: During VLM inference, each token generates its corresponding KV cache, which contains information about itself and all its prefix tokens. Due to the local prefix property of the Transformer, i.e., subsequent tokens absorb the information of prefix tokens, invalid tokens introduce incorrect information and affect the generation of subsequent tokens. Therefore, the present disclosure truncates the KV cache at the first invalid token t w , retains the KV cache of valid tokens, i.e., the KV cache of all prefix tokens of token t w , and discards the KV cache of all subsequent tokens of token t w . For example, if the content of M t-1 is "The pedestrian is moving and the bus is waiting", and the probability of the word "moving" decreases (possibly the pedestrian has stopped moving), then the KV cache corresponding to "The pedestrian is..." is truncated, and the KV cache corresponding to "...moving and the bus is waiting" is discarded. During the Figure 1 prefill stage shown, the KV caches corresponding to all historical memory tokens are obtained, so the present disclosure can directly obtain and truncate them without additional inference. The present disclosure retains the KV cache (K cache and V cache ) of valid tokens as the retention of valid historical memory for subsequent memory updates.

[0065] Memory update: Using the truncated KV cache as the initial context, the vision - language model continues to decode and generate new tokens for the current frame. During the decoding and generation process, the generated new tokens generate corresponding new keys K new and values V new . When performing attention calculation, the KV cache (K cache and V cache ) and the new keys K new and values V new are concatenated to integrate the context.

[0066] This method maintains spatio - temporal continuity: the persistent objects described in the prefix are still correctly referenced, and the newly added or changed objects are recalculated. After generating R t and A t , the memory state M t is updated to store the complete output of the current frame. Even if the truncated part during inference is output as usual, the update is still based on the newly generated inference, ensuring that the memory reflects the latest understanding of the scene.

[0067] Some embodiments of the present disclosure perform full memory refresh for extreme cases: if N consecutive tokens are below the threshold Θ', the entire historical memory is discarded and inference is generated from scratch.

[0068] Based on the empirical analysis of driving scenario mutations, some embodiments of the present disclosure set N = 3 to balance unnecessary resets and hazard detection delays.

[0069] S330. Based on the multi-view image stream and memory at the target moment, use the visual language model to generate an inference state and meta-actions for determining the autonomous driving trajectory.

[0070] One embodiment of the present disclosure Figure 3 The method described in S310 - S330 above (referred to as our method) was experimentally verified on two autonomous driving vision-language datasets. Among them, DriveLM-nuScenes is an artificially annotated dataset based on nuScenes, containing more than 5,000 perspective camera sequences, with an average of 91.4 question-answer pairs per frame; OmniDrive is a synthetic dataset generated using GPT-4, which extends nuScenes and provides diverse driving scenarios and language inference tasks. The evaluation was carried out based on the following VLMs: LLaVA-v1.6-vicuna-7B, Qwen2-VL-2B, Phi-3.5-Vision, InternVL2-4B, MobileVLM-V2-3B. These models cover different architectures and computational intensities, supporting comprehensive scalability analysis. The experiments were executed on the in-vehicle computing platform NVIDIA Jetson AGX Orin (64GB) and the end-side level graphics card NVIDIA RTX 3090. The experiments measured the end-to-end single-frame inference latency to evaluate the acceleration effect. The meta-action accuracy was evaluated by the F1 score. The F1 scores for steering (straight / left / right) and speed (fast / normal / slow / stop) were calculated independently, and the F1 score for both being satisfied simultaneously was comprehensively evaluated. The scene understanding quality was measured by the following language generation metrics: BLEU-4 measures the lexical alignment between the generated and the real scene description through 4-gram accuracy; CIDEr evaluates the semantic relevance through TF-IDF weighted n-gram consensus; METEOR combines synonyms and syntactic coherence to evaluate fluency.

[0071] A total of three baseline methods were compared: Vanilla is an unmodified VLM without a memory or redundancy utilization mechanism; SparseVLM is a spatial pruning method based on text-guided token sorting and recycling; VLA-Cach caches static tokens in continuous data to avoid recomputation of unchanged regions.

[0072] The experimental results are shown in Table 1:

[0073]

[0074] Table 1. Comparison of inference latency and accuracy of different VLM methods on NVIDIA AGX Orin and RTX 3090 platforms

[0075] As shown in Table 1, our method achieves 1.7x - 4.3x acceleration on edge devices with a precision degradation of <1%. For example, the latency of MobileVLM-3B decreases from 1.63 seconds / frame to 0.38 seconds / frame. Compared with SparseVLM and VLA-Cache, our method reduces redundant inference steps through semantic consistency checks (instead of image token redundancy). For example, in high-speed scenarios, the cached scene description (such as "the road ahead is clear") can remain valid for 5+ consecutive frames, skipping full recomputation.

[0076] Figure 4 is a schematic diagram of an inference device for a vision-language model of an autonomous driving system shown according to some embodiments of the present disclosure. As Figure 4 shown, the inference device 400 for the vision-language model of the autonomous driving system includes an acquisition module 410, a refresh module 420, and an inference module 430. In some embodiments of the present disclosure, the inference function of the vision-language model for the autonomous driving system is performed by Figure 2 the inference server shown. Wherein:

[0077] The acquisition module 410 is configured to acquire, in the vision-language model, a token combination corresponding to the memory at the previous moment of the target moment and the probability of each token therein, wherein the memory is not an empty set, and the memory at the target moment includes the inference state and meta-action at the previous moment of the target moment;

[0078] The refresh module 420 is configured to, if the probabilities of all the tokens are greater than a preset threshold, use the memory at the previous moment as the memory at the target moment; if there are tokens lower than the preset threshold, divide the token combination into logical units, check whether there are tokens lower than the preset threshold for each logical unit, and if so, refresh the logical unit and use the refreshed memory at the previous moment as the memory at the target moment;

[0079] The inference module 430 is configured to generate, based on the multi-view image stream and memory at the target moment, an inference state and meta-action for determining an autonomous driving trajectory by using the vision-language model.

[0080] Figure 5 is a schematic diagram of an inference device for a vision-language model of an autonomous driving system shown according to some embodiments of the present disclosure. As Figure 5 shown, the inference device 500 for the vision-language model of the autonomous driving system includes a memory 520 and a processor 510. The memory 520 is configured to store a computer program; the processor 510 is configured to, when executing the computer program, implement Figure 3The inference method of the vision - language model for the autonomous driving system described in S310 - S330 above.

[0081] In summary, for the inference method, apparatus, and device of the vision - language model for the autonomous driving system provided by the various embodiments of the present disclosure, by adaptively reusing historical inference outputs through semantic consistency detection, redundant inference steps are reduced; and by designing a hierarchical verification strategy from coarse - grained to fine - grained, combining global semantic checking and fine - grained token pruning, invalid inference segments are dynamically updated to avoid full - scene regeneration; in particular, an attention - aware threshold calibration method is introduced to dynamically adjust the verification criteria according to token importance, ensuring strict verification of key entities (such as "pedestrians" and "vehicles"), balancing safety and efficiency, thereby improving the inference speed of the vision - language model.

[0082] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above - described devices and modules can refer to the corresponding descriptions in the foregoing apparatus embodiments and will not be elaborated herein.

[0083] Although the subject matter described herein is provided in the general context of execution in conjunction with an operating system and application programs on a computer system, those skilled in the art will recognize that other implementations can also be performed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art can understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor - based or programmable consumer electronics, minicomputers, mainframe computers, etc., and can also be used in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

[0084] Those of ordinary skill in the art can realize that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0085] It should be understood that the above specific embodiments of the present disclosure are only for illustrative explanation or interpretation of the principles of the present disclosure, and do not constitute a limitation to the present disclosure. Therefore, any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present disclosure shall be included within the protection scope of the present disclosure. In addition, the appended claims of the present disclosure are intended to cover all variations and modifications that fall within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. An inference method for a vision-language model of an autonomous driving system, characterized in that, Including: Obtaining a token combination corresponding to the memory at the previous moment of the target moment and the probability of each token in the visual language model, where the memory is not an empty set, and the memory at the target moment includes the inference state and meta-action at the previous moment of the target moment; If the probabilities of all the tokens are greater than a preset threshold, using the memory at the previous moment as the memory at the target moment. If there are tokens lower than the preset threshold, splitting the token combination into logical units, checking each logical unit for the existence of tokens lower than the preset threshold. If so, refreshing the logical unit and using the refreshed memory at the previous moment as the memory at the target moment; Based on the multi-view image stream and memory at the target moment, using the visual language model to generate an inference state and meta-action for determining the autonomous driving trajectory.

2. The method according to claim 1, wherein The obtaining a token combination corresponding to the memory at the previous moment of the target moment and the probability of each token in the visual language model includes: Concatenating the multi-view image stream at the target moment and the memory at the previous moment into a context, inputting the context into the visual language model, and obtaining a token combination for generating an inference state and meta-action and the probability of each token therein.

3. The method according to claim 1, wherein Also including: Setting different preset thresholds for different tokens based on attention weighting.

4. The method according to claim 3, wherein The setting different preset thresholds for different tokens based on attention weighting includes: Setting different preset thresholds for different tokens based on the following formula Θ' = Θ * t s / A a , where Θ' is the preset threshold for the current token s, Θ is the standard preset threshold, and t s is the cross-layer average attention score for the current token s; A a is the average attention score obtained from the offline test of all tokens in the NuScenes dataset.

5. The method according to claim 4, wherein The refreshing the logical unit includes: Identifying the first invalid token in the logical unit, where the invalid token refers to a token with a probability lower than the preset threshold. Retaining the KV cache corresponding to all the tokens before the invalid token, and regenerating KV for all the tokens after the invalid token; Connecting the retained KV cache and the newly generated KV to refresh the logical unit.

6. The method according to claim 5, wherein The regenerating KV for all the tokens after the invalid token includes: Using the retained KV cache as the initial context and continuing to generate new tokens and the KV corresponding to the new tokens using the visual language model.

7. The method according to claim 1, wherein The refreshing the logical unit includes: If there are a continuous preset number of invalid tokens in the logical unit, regenerating all the tokens in the logical unit.

8. The method according to claim 7, wherein: The preset number is 3.

9. An inference device for a vision-language model of an autonomous driving system, characterized in that, Including: An obtaining module for obtaining a token combination corresponding to the memory at the previous moment of the target moment and the probability of each token in the visual language model, where the memory is not an empty set, and the memory at the target moment includes the inference state and meta-action at the previous moment of the target moment; A refreshing module for, if the probabilities of all the tokens are greater than a preset threshold, using the memory at the previous moment as the memory at the target moment. If there are tokens lower than the preset threshold, splitting the token combination into logical units, checking each logical unit for the existence of tokens lower than the preset threshold. If so, refreshing the logical unit and using the refreshed memory at the previous moment as the memory at the target moment; An inference module, configured to generate an inference state and a meta-action for determining an autonomous driving trajectory based on a multi-view image stream and memory at a target moment, by using the vision-language model.

10. An inference device for a vision-language model of an autonomous driving system, characterized in that: It includes a memory and a processor. The memory is configured to store a computer program. The processor is configured to, when executing the computer program, implement the inference method of the vision-language model for an autonomous driving system according to any one of claims 1-8.

Images