A control strategy generation method based on vehicle driving video and related equipment

Abstract

The application discloses a control strategy generation method based on vehicle driving video and related equipment, and the method comprises the following steps: acquiring visual features in a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generating a question and answer pair from the visual features and the driver privilege information; constructing a fine-tuning sample set based on the preset vehicle driving video and the question and answer pair, and performing supervised fine-tuning on a preset basic model with different architectures according to the constructed fine-tuning sample set to obtain a heterogeneous model pool; driving each model in the heterogeneous model pool to perform parallel inference on a real-time video stream through an asynchronous concurrent mechanism to obtain driving decision data and explanation data output by each model in the heterogeneous model pool; calculating semantic vector consistency of each model in the heterogeneous model pool based on the explanation data, and dividing each model in the heterogeneous model pool into an abnormal model and a healthy model through a single-point deviation detection method according to the semantic vector consistency; and generating a control strategy for the real-time video stream through a confidence weighted fusion method according to driving decision data output by the healthy model.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and autonomous driving safety technology, and in particular to a method and related equipment for generating control strategies based on vehicle driving videos. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large language models (LMMs) have demonstrated enormous potential in autonomous driving perception, prediction, and planning tasks due to their powerful text and image understanding and logical reasoning capabilities. Compared with traditional modular autonomous driving systems, end-to-end large models can better understand long-tail problems in complex traffic scenarios and provide interpretable driving suggestions. However, large models are inherently based on probability generation, and thus have inherent risks of "illusion" (i.e., generating seemingly reasonable but factual content) and reasoning uncertainty. In the field of autonomous driving, where safety requirements are extremely high, a single model, when faced with adversarial samples or extreme conditions, will directly threaten driving safety and lead to irreversible consequences if it makes a misjudgment or logical failure.

[0003] Existing autonomous driving safety redundancy solutions mostly focus on hardware redundancy at the sensor level or rule-based fallback at the control level. However, at the core algorithm level of perception and decision-making, effective intrinsic safety mechanisms are lacking. Traditional algorithm redundancy typically employs a "homogeneous redundancy" strategy, which simply backs up identical model instances. While this approach can handle hardware failures, it cannot solve the "common mode failure" problem—that is, if the model architecture itself has design flaws or blind spots for a certain type of sample, all backup models will fail simultaneously. Furthermore, while general-purpose multimodal base models (such as the original Qwen-VL or InternVL) possess extensive visual knowledge, they lack in-depth optimization for specific domains such as driving behavior norms and traffic rule interpretation. When directly applied to autonomous driving, they often exhibit poor command compliance and inaccurate recognition of key information (such as traffic light status and lane lines). Summary of the Invention

[0004] The purpose of this invention is to provide a control strategy generation method based on vehicle driving video, which aims to solve the reliability problem of a single model in autonomous driving scenarios by constructing a heterogeneous model pool and utilizing parallel inference and multi-dimensional adjudication mechanisms.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for generating control strategies based on vehicle driving videos includes the following steps: Obtain visual features from a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generate a question-and-answer pair from the visual features and driver privilege information; Based on the preset vehicle driving video and the question-and-answer pair, a fine-tuning sample set is constructed, and the preset base models of different architectures are supervisedly fine-tuned according to the constructed fine-tuning sample set to obtain a heterogeneous model pool. The asynchronous concurrency mechanism drives each model in the heterogeneous model pool to perform parallel inference on the real-time video stream, thereby obtaining the driving decision data and interpretation data output by each model in the heterogeneous model pool. Based on the explained data, the semantic vector consistency of each model in the heterogeneous model pool is calculated, and based on the semantic vector consistency, each model in the heterogeneous model pool is divided into abnormal models and healthy models by a single-point deviation detection method. Based on the driving decision data output by the health model, a control strategy for the real-time video stream is generated using a confidence-weighted fusion method.

[0007] Optionally, the step of obtaining visual features from a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generating a question-and-answer pair from the visual features and the driver privilege information, includes: A target detection algorithm based on YOLO is used to extract a set of bounding boxes of a preset number of key traffic elements in a preset vehicle driving video, and the set of bounding boxes is used as visual features. The visual features are serialized with privileged information, including at least speed and GPS trajectory, into structured text data; Based on the structured text data, question-answer pairs are generated using a large language model.

[0008] Optionally, the step of supervising the fine-tuning of preset base models with different architectures based on the constructed fine-tuning sample set to obtain a heterogeneous model pool includes: Based on the constructed fine-tuning sample set, supervised fine-tuning of preset base models with different architectures is performed with the goal of minimizing the autoregressive loss function, resulting in a heterogeneous model pool. The mathematical representation of the autoregressive loss function is as follows:

[0009] in, For the model The parameter set, The total length of the target answer sequence, Answer the target The first in One tag, For context labeling sequences, It is a conditional probability distribution.

[0010] Optionally, the step of driving each model in the heterogeneous model pool to perform parallel inference on the real-time video stream through an asynchronous concurrency mechanism to obtain the driving decision data and interpretation data output by each model in the heterogeneous model pool includes: The real-time video stream is distributed to each model in the heterogeneous model pool through the non-blocking message queue in the asynchronous concurrency mechanism, so that each model in the heterogeneous model pool can independently output a prediction vector, which includes explanatory data and driving decision data.

[0011] Optionally, the step of calculating the semantic vector consistency of each model in the heterogeneous model pool based on the explanatory data includes: The interpreted data is mapped into a high-dimensional semantic vector through a preset semantic embedding model; Calculate the cosine similarity matrix between different models in the heterogeneous model pool based on the high-dimensional semantic vector; The semantic vector consistency of each model in the heterogeneous model pool is determined based on the cosine similarity matrix.

[0012] Optionally, the step of classifying the models in the heterogeneous model pool into abnormal models and healthy models using a single-point deviation detection method includes: According to the single-point deviation detection method, models whose semantic vector consistency in the heterogeneous model pool is less than the preset threshold are identified as abnormal models; models whose semantic vector consistency in the heterogeneous model pool is greater than or equal to the preset threshold are identified as healthy models.

[0013] Optionally, the step of generating a control strategy for the real-time video stream based on the driving decision data output by the health model using a confidence-weighted fusion method includes: Based on the driving decision data output by the health model, a control strategy for the real-time video stream is generated using a weighted fusion method based on semantic consistency scores. The mathematical representation of the weighted fusion method based on semantic consistency scores is as follows:

[0014] in, For the first Numerical prediction output of an effective model, To control the temperature coefficient of the weights, For the control strategy of the real-time video stream, The set of driving decision data output by the health model.

[0015] On the other hand, this application provides a control strategy generation system based on vehicle driving video, including: The data acquisition module is used to acquire visual features from a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and to generate question-and-answer pairs from the visual features and the driver privilege information; The training module is used to construct a fine-tuning sample set based on the preset vehicle driving video and the question-answer pair, and to perform supervised fine-tuning of preset base models with different architectures according to the constructed fine-tuning sample set to obtain a heterogeneous model pool. The inference module is used to drive each model in the heterogeneous model pool to perform parallel inference on the real-time video stream through an asynchronous concurrency mechanism, so as to obtain the driving decision data and interpretation data output by each model in the heterogeneous model pool. The filtering module is used to calculate the semantic vector consistency of each model in the heterogeneous model pool based on the explained data, and to classify each model in the heterogeneous model pool into abnormal models and healthy models according to the semantic vector consistency by a single-point deviation detection method. The strategy generation module is used to generate a control strategy for the real-time video stream based on the driving decision data output by the health model and through a confidence-weighted fusion method.

[0016] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the control strategy generation method based on vehicle driving video as described above.

[0017] Fourthly, this application provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the control strategy generation method based on vehicle driving video as described above.

[0018] Beneficial effects: By constructing a heterogeneous model pool with varying architectures (such as the Qwen-VL and InternVL series) and parameter sizes (such as 8B and 32B), and leveraging the differences in feature extraction and inference logic among models, the attacker's attack chain is effectively disrupted. This exponentially reduces the probability of all models simultaneously generating the same "illusion" or error, thus endowing the system with intrinsic immunity against unknown risks at the algorithmic level without relying on external hardware stacking. By constructing a high-quality multimodal instruction fine-tuning dataset, the instruction compliance and scene understanding capabilities of the general-purpose large model in the autonomous driving vertical field are significantly improved. Addressing the shortcomings of general-purpose large models when directly applied to driving scenarios, such as a superficial understanding of traffic rules and insufficient accuracy in recognizing key targets (such as traffic lights and lane lines), the hard visual detection information of the YOLO algorithm and the soft logic inference chain generated by ChatGPT are integrated. This not only eliminates the redundant and redundant statements of the general-purpose model, enabling the output of structured instructions that conform to industrial control standards, but also achieves a qualitative leap in the accuracy of causal inference in complex long-tail scenarios, truly transforming the general-purpose large model into a "driving expert" with professional competence. By employing asynchronous parallel reasoning and a multi-dimensional semantic adjudication mechanism, the system ensures high reliability of decisions while also meeting real-time requirements. This ensures that the total system time depends solely on the slowest single model, rather than being a linear summation. Furthermore, the semantic adjudication mechanism based on vector space distance accurately identifies and eliminates anomalous outputs that deviate from consensus (i.e., model illusions). The final decision is generated through a confidence-weighted fusion method, providing an interpretable mathematical basis for every decision made in autonomous driving. This significantly enhances the safety of L4 autonomous driving systems in complex and open environments. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a control strategy generation method based on vehicle driving video provided in an embodiment of this application; Figure 2 This is a schematic diagram of the reasoning process in a control strategy generation method based on vehicle driving video provided in an embodiment of this application; Figure 3 This is a schematic diagram of a control strategy generation system based on vehicle driving video provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this invention, it should be noted that unless otherwise explicitly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] like Figure 1 As shown in the figure, this application provides a method for generating control strategies based on vehicle driving videos, including the following steps: S101. Obtain visual features from a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generate a question-and-answer pair from the visual features and the driver privilege information; S102. Based on the preset vehicle driving video and the question-and-answer pair, construct a fine-tuning sample set, and perform supervised fine-tuning of preset base models of different architectures according to the constructed fine-tuning sample set to obtain a heterogeneous model pool. S103. Drive each model in the heterogeneous model pool to perform parallel inference on the real-time video stream through an asynchronous concurrency mechanism to obtain the driving decision data and interpretation data output by each model in the heterogeneous model pool. S104. Calculate the semantic vector consistency of each model in the heterogeneous model pool based on the explained data, and classify each model in the heterogeneous model pool into abnormal models and healthy models according to the semantic vector consistency using a single-point deviation detection method. S105. Based on the driving decision data output by the health model, a control strategy for the real-time video stream is generated using a confidence-weighted fusion method.

[0023] For example, this embodiment provides an intrinsic safety method for autonomous driving video understanding based on a dynamic heterogeneous redundancy architecture. This method is deployed on an autonomous driving vehicle computing platform equipped with multiple cameras and high-performance computing units (such as NVIDIA Orin or server-grade GPUs). The implementation process is divided into an offline heterogeneous model construction stage and an online real-time inference and adjudication stage.

[0024] First, in the offline phase, this embodiment focuses on building a high-quality dataset for fine-tuning autonomous driving instructions and a heterogeneous model pool. Addressing the limitation of the original BDD-X dataset containing only single video and text labels, this invention introduces a multi-source information enhancement mechanism. The system utilizes the YOLOv8 object detection algorithm to extract video frames. middle A set of bounding boxes for key traffic elements (such as vehicles, pedestrians, and traffic lights). Simultaneously, extract privileged information recorded on the vehicle's CAN bus. (Including speed, acceleration, and GPS trajectory). Visual features. With privilege information Serialization into structured text description The data is then input into the ChatGPT-4 model, where prompting engineering generates multi-turn question-and-answer pairs covering perceptual description, behavioral decision-making, and causal explanation. The constructed fine-tuned sample S can be represented as... ,in This dataset consists of multiple consecutive frames of images. Based on this augmented dataset, K large-scale models with different architectures are selected. In this embodiment, we select... Using InternVL-Pro, Qwen-VL-Chat, and LLaVA-Next as bases, a heterogeneous model set is constructed. For each model Supervised Full Parameter Fine-Tuning (SFT) is performed, with the optimization objective of minimizing the autoregressive loss function. :

[0025] in, For model parameters, Target answer sequence The first in One tag, This serves as the preceding context. After fine-tuning, each heterogeneous model, while maintaining its architectural differences, possesses the ability to output standardized driving commands.

[0026] Secondly, during the online operation phase, the system executes a heterogeneous inference process based on asynchronous concurrency. The vehicle-mounted camera captures a real-time video stream at a fixed frame rate and divides it into sliding window segments of length $\Delta t$. The system uses a non-blocking message queue to... Simultaneously distributed to the heterogeneous model pool Each execution entity. Independently encode and infer from the input fragment, and output a natural language explanation. and numerical control commands (speed) Steering angle The prediction vector of ) Because the parameter scales and network layers of each model differ, the inference time varies. Therefore, the system sets a synchronization barrier to wait for all models to return results or until the maximum timeout threshold is reached. At this point, the system obtains a set of heterogeneous outputs. This provides redundant input for subsequent security decisions.

[0027] Finally, the system uses the intrinsic security adjudication module to evaluate the set. Multi-dimensional consistency verification and fusion are performed. For the natural language interpretation part, a pre-trained Sentence-BERT model is used to perform each interpretation... Mapped to high-dimensional semantic vectors Calculate the pairwise cosine similarity matrix between the models. , of which elements Defined as:

[0028] Based on this similarity matrix, a single-point deviation detection algorithm is applied to identify abnormal models. Model definition. Consistency score If a certain model Score Below the preset safety threshold If the model is deemed to be "hallucinating" or under attack, it is marked as an untrusted model and removed. For the remaining set of valid models... The system employs a semantically consistent weighted strategy to generate the final vehicle control commands, with the final speed as the determining factor. For example, the calculation formula is as follows:

[0029] in To control the temperature coefficient of weight sharpness. This fusion mechanism ensures that the final decision is biased towards the result with higher semantic consensus with most models, thereby achieving a qualitative change from "single-point unreliability" to "system high robustness" at the mathematical level, effectively defending against adversarial perturbations and logical failure risks against a single model.

[0030] In one possible implementation, the step of acquiring visual features from a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generating question-and-answer pairs from the visual features and the driver privilege information, includes: A target detection algorithm based on YOLO is used to extract a set of bounding boxes of a preset number of key traffic elements in a preset vehicle driving video, and the set of bounding boxes is used as visual features. The visual features are serialized with privileged information, including at least speed and GPS trajectory, into structured text data; Based on the structured text data, question-answer pairs are generated using a large language model.

[0031] For example, the original video (preset vehicle driving video) frame sequence is defined as follows: ,in Let be the length of the time window. The YOLOv8 object detection algorithm is used to extract the first... In the frame A set of bounding boxes for key target objects In addition, it incorporates driver privilege information specific to the dataset (such as GPS trajectory and speed). Using the above information as context The input is fed into a Large Language Model (LLM) to generate corresponding question-answer pairs. The final fine-tuned sample set was constructed. It can be represented as:

[0032] in, The total number of samples, This represents a mapping function that transforms visual features and privileged information into natural language descriptions, ensuring that the model can simultaneously learn visual perception and logical reasoning abilities during the fine-tuning phase.

[0033] In one possible implementation, the step of supervising the fine-tuning of preset base models with different architectures based on the constructed fine-tuning sample set to obtain a heterogeneous model pool includes: Based on the constructed fine-tuning sample set, supervised fine-tuning of preset base models with different architectures is performed with the goal of minimizing the autoregressive loss function, resulting in a heterogeneous model pool. The mathematical representation of the autoregressive loss function is as follows:

[0034] in, For the model The parameter set, The total length of the target answer sequence, Answer the target The first in One tag, For context labeling sequences, It is a conditional probability distribution.

[0035] For example, select A base model with different architectures (such as Qwen-VL, InternVL, etc.), for each model Set its parameter set as During the fine-tuning phase, an objective function based on autoregressive language modeling is used for optimization to maximize the given video context. and instructions Generate target answer The likelihood probability. Optimize the objective function. Defined as:

[0036] in, Is this the answer? The first in Tokens This represents the previous labeled sequence. By minimizing this loss function, each heterogeneous model... Transform into an expert model specific to the autonomous driving field .

[0037] In one possible implementation, the step of driving each model in the heterogeneous model pool to perform parallel inference on the real-time video stream through an asynchronous concurrency mechanism to obtain the driving decision data and interpretation data output by each model in the heterogeneous model pool includes: The real-time video stream is distributed to each model in the heterogeneous model pool through the non-blocking message queue in the asynchronous concurrency mechanism, so that each model in the heterogeneous model pool can independently output a prediction vector, which includes explanatory data and driving decision data.

[0038] For example, such as Figure 2 As shown, the system employs a non-blocking asynchronous inference pipeline for the input real-time video stream. Real-time video stream Divided into continuous segments Multiple heterogeneous execution entities The same segment is processed in parallel, with each actor outputting a prediction vector containing vehicle motion, interpretation, speed, and angle. This process can be formalized as follows:

[0039] in, These are preset prompt words. Classifying discrete actions, For natural language interpretation, and This is for continuous numerical prediction. Due to the different model architectures, the inference delay is... The total system response time varies and depends on the specific circumstances. Real-time performance is ensured through parallel design.

[0040] In one possible implementation, the step of calculating the semantic vector consistency of each model in the heterogeneous model pool based on the explained data includes: The interpreted data is mapped into a high-dimensional semantic vector through a preset semantic embedding model; Calculate the cosine similarity matrix between different models in the heterogeneous model pool based on the high-dimensional semantic vector; The semantic vector consistency of each model in the heterogeneous model pool is determined based on the cosine similarity matrix.

[0041] For example, regarding text interpretation (the interpretation data) The pre-trained semantic embedding model SentenceTransformer is used to interpret the text (the interpreted data). Mapped to a high-dimensional vector Computational model With model Cosine similarity matrix between : .

[0042] In one possible implementation, the step of classifying the models in the heterogeneous model pool into abnormal models and healthy models using a single-point deviation detection method includes: According to the single-point deviation detection method, models whose semantic vector consistency in the heterogeneous model pool is less than the preset threshold are identified as abnormal models; models whose semantic vector consistency in the heterogeneous model pool is greater than or equal to the preset threshold are identified as healthy models.

[0043] In one possible implementation, the step of generating a control strategy for the real-time video stream based on the driving decision data output by the health model using a confidence-weighted fusion method includes: Based on the driving decision data output by the health model, a control strategy for the real-time video stream is generated using a weighted fusion method based on semantic consistency scores. The mathematical representation of the weighted fusion method based on semantic consistency scores is as follows:

[0044] in, For the first Numerical prediction output of an effective model, To control the temperature coefficient of the weights, For the control strategy of the real-time video stream, The set of driving decision data output by the health model.

[0045] For example, setting a semantic consistency threshold ,like If model i is deemed abnormal (hallucination or being attacked), it is removed from the decision set. The remaining effective model set... The final numerical decision (For example, speed) employs a confidence-based weighted fusion:

[0046] Among them, weight The model k is dynamically adjusted based on its historical accuracy or semantic consistency score in the current scenario, thereby achieving intrinsic security protection.

[0047] like Figure 3 As shown, this application provides a control strategy generation system based on vehicle driving video, including: Data acquisition module 201 is used to acquire visual features in a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generate question-and-answer pairs from the visual features and the driver privilege information; Training module 202 is used to construct a fine-tuning sample set based on the preset vehicle driving video and the question-answer pair, and to perform supervised fine-tuning of preset base models with different architectures according to the constructed fine-tuning sample set to obtain a heterogeneous model pool. The inference module 203 is used to drive each model in the heterogeneous model pool to perform parallel inference on the real-time video stream through an asynchronous concurrency mechanism, so as to obtain the driving decision data and interpretation data output by each model in the heterogeneous model pool. The filtering module 204 is used to calculate the semantic vector consistency of each model in the heterogeneous model pool based on the explanation data, and to classify each model in the heterogeneous model pool into abnormal models and healthy models according to the semantic vector consistency by a single-point deviation detection method. The strategy generation module 205 is used to generate a control strategy for the real-time video stream based on the driving decision data output by the health model and through a confidence-weighted fusion method.

[0048] In one possible implementation, such as Figure 4 As shown, this application embodiment provides a terminal device 300, including: a memory 310, a processor 320, and a first computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the first computer program 311, it acquires visual features in a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generates a question-and-answer pair between the visual features and the driver privilege information. Based on the preset vehicle driving video and the question-and-answer pair, a fine-tuning sample set is constructed, and the preset base models of different architectures are supervisedly fine-tuned according to the constructed fine-tuning sample set to obtain a heterogeneous model pool. The asynchronous concurrency mechanism drives each model in the heterogeneous model pool to perform parallel inference on the real-time video stream, thereby obtaining the driving decision data and interpretation data output by each model in the heterogeneous model pool. Based on the explained data, the semantic vector consistency of each model in the heterogeneous model pool is calculated, and based on the semantic vector consistency, each model in the heterogeneous model pool is divided into abnormal models and healthy models by a single-point deviation detection method. Based on the driving decision data output by the health model, a control strategy for the real-time video stream is generated using a confidence-weighted fusion method.

[0049] In one possible implementation, such as Figure 5 As shown, this application embodiment provides a computer-readable storage medium 400, on which a second computer program 411 is stored. When the second computer program 411 is executed by a processor, it acquires visual features in a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generates a question-and-answer pair from the visual features and the driver privilege information. Based on the preset vehicle driving video and the question-and-answer pair, a fine-tuning sample set is constructed, and the preset base models of different architectures are supervisedly fine-tuned according to the constructed fine-tuning sample set to obtain a heterogeneous model pool. The asynchronous concurrency mechanism drives each model in the heterogeneous model pool to perform parallel inference on the real-time video stream, thereby obtaining the driving decision data and interpretation data output by each model in the heterogeneous model pool. Based on the explained data, the semantic vector consistency of each model in the heterogeneous model pool is calculated, and based on the semantic vector consistency, each model in the heterogeneous model pool is divided into abnormal models and healthy models by a single-point deviation detection method. Based on the driving decision data output by the health model, a control strategy for the real-time video stream is generated using a confidence-weighted fusion method.

[0050] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

[0051] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for generating control strategies based on vehicle driving video, characterized in that, Includes the following steps: Obtain visual features from a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generate a question-and-answer pair from the visual features and driver privilege information; Based on the preset vehicle driving video and the question-and-answer pair, a fine-tuning sample set is constructed, and the preset base models of different architectures are supervisedly fine-tuned according to the constructed fine-tuning sample set to obtain a heterogeneous model pool. The asynchronous concurrency mechanism drives each model in the heterogeneous model pool to perform parallel inference on the real-time video stream, thereby obtaining the driving decision data and interpretation data output by each model in the heterogeneous model pool. Based on the explained data, the semantic vector consistency of each model in the heterogeneous model pool is calculated, and based on the semantic vector consistency, each model in the heterogeneous model pool is divided into abnormal models and healthy models by a single-point deviation detection method. Based on the driving decision data output by the health model, a control strategy for the real-time video stream is generated using a confidence-weighted fusion method.

2. The method for generating control strategies based on vehicle driving video according to claim 1, characterized in that, The step of acquiring visual features from a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and generating question-and-answer pairs from the visual features and the driver privilege information, includes: A target detection algorithm based on YOLO is used to extract a set of bounding boxes of a preset number of key traffic elements in a preset vehicle driving video, and the set of bounding boxes is used as visual features. The visual features are serialized with privileged information, including at least speed and GPS trajectory, into structured text data; Based on the structured text data, question-answer pairs are generated using a large language model.

3. The method for generating control strategies based on vehicle driving video according to claim 1, characterized in that, The step of supervising the fine-tuning of preset base models with different architectures based on the constructed fine-tuning sample set to obtain a heterogeneous model pool includes: Based on the constructed fine-tuning sample set, supervised fine-tuning of preset base models with different architectures is performed with the goal of minimizing the autoregressive loss function, resulting in a heterogeneous model pool. The mathematical representation of the autoregressive loss function is as follows: in, For the model The parameter set, The total length of the target answer sequence, Answer the target The first in One tag, For context labeling sequences, It is a conditional probability distribution.

4. The method for generating control strategies based on vehicle driving video according to claim 1, characterized in that, The step of driving each model in the heterogeneous model pool to perform parallel inference on the real-time video stream through an asynchronous concurrency mechanism to obtain the driving decision data and interpretation data output by each model in the heterogeneous model pool includes: The real-time video stream is distributed to each model in the heterogeneous model pool through the non-blocking message queue in the asynchronous concurrency mechanism, so that each model in the heterogeneous model pool can independently output a prediction vector, which includes explanatory data and driving decision data.

5. The method for generating control strategies based on vehicle driving video according to claim 1, characterized in that, The step of calculating the semantic vector consistency of each model in the heterogeneous model pool based on the explained data includes: The interpreted data is mapped into a high-dimensional semantic vector through a preset semantic embedding model; Calculate the cosine similarity matrix between different models in the heterogeneous model pool based on the high-dimensional semantic vector; The semantic vector consistency of each model in the heterogeneous model pool is determined based on the cosine similarity matrix.

6. The method for generating control strategies based on vehicle driving video according to claim 1, characterized in that, The step of classifying the models in the heterogeneous model pool into abnormal models and healthy models using a single-point deviation detection method includes: According to the single-point deviation detection method, models whose semantic vector consistency in the heterogeneous model pool is less than the preset threshold are identified as abnormal models; models whose semantic vector consistency in the heterogeneous model pool is greater than or equal to the preset threshold are identified as healthy models.

7. The method for generating control strategies based on vehicle driving video according to claim 1, characterized in that, The step of generating a control strategy for the real-time video stream based on the driving decision data output by the health model using a confidence-weighted fusion method includes: Based on the driving decision data output by the health model, a control strategy for the real-time video stream is generated using a weighted fusion method based on semantic consistency scores. The mathematical representation of the weighted fusion method based on semantic consistency scores is as follows: in, For the first Numerical prediction output of an effective model, To control the temperature coefficient of the weights, For the control strategy of the real-time video stream, The set of driving decision data output by the health model.

8. A control strategy generation system based on vehicle driving video, characterized in that, include: The data acquisition module is used to acquire visual features from a preset vehicle driving video and driver privilege information corresponding to the preset vehicle driving video, and to generate question-and-answer pairs from the visual features and the driver privilege information; The training module is used to construct a fine-tuning sample set based on the preset vehicle driving video and the question-answer pair, and to perform supervised fine-tuning of preset base models with different architectures according to the constructed fine-tuning sample set to obtain a heterogeneous model pool. The inference module is used to drive each model in the heterogeneous model pool to perform parallel inference on the real-time video stream through an asynchronous concurrency mechanism, so as to obtain the driving decision data and interpretation data output by each model in the heterogeneous model pool. The filtering module is used to calculate the semantic vector consistency of each model in the heterogeneous model pool based on the explained data, and to classify each model in the heterogeneous model pool into abnormal models and healthy models according to the semantic vector consistency by a single-point deviation detection method. The strategy generation module is used to generate a control strategy for the real-time video stream based on the driving decision data output by the health model and through a confidence-weighted fusion method.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the control strategy generation method based on vehicle driving video as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the control strategy generation method based on vehicle driving video as described in any one of claims 1 to 7.

Images