A vehicle tracking and re-perception method and system based on a large language model

Abstract

The application discloses a kind of vehicle tracking and re-perception method and system based on large language model, through the intelligent analysis of pre-training large language model, quickly output control signal guides rear vehicle action, shorten the tracking interruption time after target vehicle loss, improve the continuity and overall efficiency of vehicle tracking;The present application is based on pre-training large language model, learns and understands various complex vehicle driving scenarios and target vehicle behavior patterns.Large language model makes reasonable judgment and prediction according to the input image information, generates control signals suitable for different situations, and has stronger adaptability and flexibility compared with traditional fixed rule-based method;Using large language model to analyze the images of the target vehicle for several seconds before disappearing can fully exploit useful information in the images, accurately infer the direction of the target vehicle, and improve the data utilization efficiency and the accuracy of re-perception.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, specifically relating to a vehicle tracking and re-sensing method and system based on a large language model. Background Technology

[0002] In vehicle tracking scenarios, accurately and continuously sensing the position and status of target vehicles is crucial. Current common vehicle tracking technologies primarily rely on sensors (such as cameras and radar) to acquire real-time information about target vehicles. However, in reality, due to complex road environments, target vehicles may become obstructed by other objects or enter sensor blind spots while turning, causing the tracking system to lose track of the target vehicle. Once the target vehicle is lost, existing systems often struggle to quickly and effectively re-establish tracking, impacting vehicle safety and traffic management efficiency, and even potentially leading to collisions. Furthermore, traditional rule-based or simple algorithm-based vehicle tracking and re-sensing methods have limited adaptability in complex and ever-changing real-world scenarios, making it difficult to handle various road conditions and target vehicle behaviors.

[0003] Existing vehicle tracking systems lack effective automatic recovery mechanisms when a target vehicle becomes undetectable for various reasons. This necessitates manual intervention or lengthy system re-search and relocation, resulting in inefficiency and potential dangers. Furthermore, tracking and re-sensing strategies based on fixed rules or simple algorithms cannot flexibly adapt to complex and changing road conditions and target vehicle behavior. Limited processing capabilities for the large amounts of data acquired by sensors make it difficult to accurately infer the possible destination and status of the target vehicle from limited information available before its loss. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a vehicle tracking and re-perception method and system based on a large language model. The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a vehicle tracking and re-perception method based on a large language model, the method comprising: S1: Acquire multimodal input data in real time through multiple vehicle-mounted sensors as the current input dataset, add a timestamp to each current input dataset, and construct a historical dataset from all acquired multimodal input data; S2, using a pre-trained visual encoder to extract the current visual feature vector and the historical visual feature vector from video frames in the current input dataset and the historical dataset respectively; using a pre-trained text encoder to convert the current text description and the historical text description corresponding to the video frame into the current text feature vector and the historical text feature vector. S3 uses an attention mechanism to dynamically fuse the current visual feature vector and the current text feature vector to generate the current multimodal representation, and dynamically fuses the historical visual feature vector and the historical text feature vector to generate the historical multimodal representation. S4 uses a deep learning-based target detection algorithm to analyze video frames in the current input dataset in real time. When a target vehicle is detected to be missing, the timestamp of the loss and the target vehicle information in the last video frame are recorded. S5 utilizes a pre-trained large language model to parse the historical state of the target vehicle before it was lost, based on the timestamp of the loss and the corresponding historical multimodal representation within the previous preset timestamp; it constructs a historical scene model based on the historical state, calculates the strength of environmental constraints, and generates vehicle control command parameters by filtering through consistency indicators and priorities. S6. After the vehicle executes the vehicle control command parameters, it obtains the current multimodal input data and generates a re-perception result. If the re-perception result is a successful perception, it continues to track the target vehicle. If the re-perception result is a failed perception, it merges the timestamp when the vehicle was lost, the corresponding historical multimodal representation within the previously preset timestamp, and the current multimodal representation into a deep-combined mode, updates the environmental constraint strength, dynamically adjusts the weights corresponding to each mode, and recalculates the consistency index and priority to generate new vehicle control command parameters. S7 controls the vehicle's movement based on the new vehicle control command parameters, determines whether the perception is successful, and if the perception is unsuccessful, repeats step S6 until the perception is successful, so as to achieve vehicle tracking and re-perception.

[0005] In one embodiment of the present invention, multimodal input data is acquired in real time through multiple vehicle-mounted sensors as the current input dataset, and a timestamp is added to each current input dataset, including: The vehicle-mounted camera captures video frames of the target vehicle and its surrounding environment from all directions, and provides corresponding text descriptions for each video frame. Vehicle dynamic information is obtained through vehicle-mounted radar, and the shape and position of the vehicle are obtained through vehicle-mounted lidar. The video frame, the corresponding text description of the video frame, the vehicle's dynamic information, the vehicle's shape and position are used as the current input dataset, and a timestamp is added to each current input dataset.

[0006] In one embodiment of the present invention, the current visual feature vector and the current text feature vector are dynamically fused through an attention mechanism to generate a current multimodal representation, including: An attention mechanism is introduced to obtain the query based on the visual feature vector and the key and value based on the current text feature vector. Based on the query, key, and value, the attention weights are calculated using a scaled dot product attention mechanism. The attention weights are fused with the current visual feature vector to output the corresponding current multimodal representation.

[0007] In one embodiment of the present invention, a deep learning-based target detection algorithm is used to analyze video frames in the current input dataset in real time. When a target vehicle is detected to be missing, the timestamp of the loss and the target vehicle information in the last video frame are recorded, including: Deep learning-based object detection algorithms construct feature vectors of target vehicles from video frames in the current input dataset, match the feature vectors of target vehicles with pre-trained target vehicle feature templates, and obtain matching scores. When the matching score of the target vehicle corresponding to a consecutive preset number of frames is lower than a preset threshold, it is determined that the target vehicle is lost; at the same time, the timestamp of the target vehicle being lost and the target vehicle information in the last video frame are recorded; wherein, the preset number of frames is 5, the preset threshold is 0.5, and the target vehicle information includes the vehicle's position and attitude.

[0008] In one embodiment of the present invention, a pre-trained large language model is used to parse the historical state of the target vehicle before it was lost, based on the timestamp at the time of loss and the corresponding historical multimodal representation within a preset timestamp. A historical scene model is constructed based on the historical state, the strength of environmental constraints is calculated, and vehicle control command parameters are generated through consistency indices and priority filtering, including: By using a pre-trained large language model, the timestamp at the time of loss and the corresponding historical multimodal representation within the previous preset timestamp are analyzed to confirm the historical state of the target vehicle before it was lost. A historical scenario model is constructed based on the historical state of the target vehicle before it was lost, and the strength of environmental constraints is calculated to generate multiple behavioral hypotheses for the target vehicle. Valid hypotheses that meet the requirements of the consistency index are selected from all behavioral hypotheses. Valid hypotheses are quantified and ranked based on a priority scoring formula to select the optimal hypothesis. Based on optimal assumptions and combined with the strength of environmental constraints, control commands corresponding to target speed, steering angle, and acceleration are derived and output as vehicle control command parameters.

[0009] In one embodiment of the present invention, the historical state of the target vehicle before it was lost includes: Trajectory, velocity, acceleration, vehicle attitude, and environmental constraints before disappearance.

[0010] In one embodiment of the present invention, the expression for the depth-bound modality is as follows: ; in, Indicates deep-association mode, Indicates the weight of historical data. Representing historical multimodal representation, Indicates the current data weight. This indicates the current multimodal representation.

[0011] Secondly, the present invention provides a vehicle tracking and re-perception system based on a large language model, the system comprising: The system comprises a vehicle perception module, a data transmission module, a large language model processing module, and a vehicle control module; among which... The vehicle perception module acquires multimodal input data in real time through multiple on-board sensors as the current input dataset; The data transmission module transmits the current input dataset to the large language model processing module; The large language model processing module adds a timestamp to each current input dataset, constructing a historical dataset from all acquired multimodal input data. It then uses a pre-trained visual encoder to extract current and historical visual feature vectors from video frames in both the current and historical datasets. A pre-trained text encoder converts the current and historical text descriptions corresponding to the video frames into current and historical text feature vectors. An attention mechanism dynamically fuses the current visual and text feature vectors to generate a current multimodal representation, and similarly, it dynamically fuses the historical visual and text feature vectors to generate a historical multimodal representation. A deep learning-based target detection algorithm analyzes the video frames in the current input dataset in real time. When a target vehicle is detected as missing, the module records the timestamp of the loss and the target vehicle information from the last video frame. Using the pre-trained large language model, it parses the historical state of the target vehicle before its loss based on the timestamp of the loss and the corresponding historical multimodal representation within the previously preset timestamps. A historical scene model is constructed based on the historical state, and the strength of environmental constraints is calculated. Consistency indicators and priority filtering are used to generate vehicle control command parameters. The vehicle control module executes the vehicle control command parameters, acquires the current multimodal input data, and generates a re-perception result. If the re-perception result is successful, it continues to track the target vehicle. If the re-perception result is unsuccessful, it uses the large language model processing module to fuse the timestamp at the time of loss, the corresponding historical multimodal representation within the previously preset timestamp, and the current multimodal representation into a deep-binding modality. It updates the environmental constraint strength, dynamically adjusts the weights corresponding to each modality, and recalculates the consistency index and priority to generate new vehicle control command parameters. Based on the new vehicle control command parameters, it controls the vehicle's movement and determines whether the perception is successful. If the perception is unsuccessful, it uses the large language model processing module to repeatedly execute the process of generating new vehicle control command parameters until successful perception is achieved, thus realizing vehicle tracking and re-perception.

[0012] Thirdly, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the vehicle tracking and re-perception method based on a large language model provided by the present invention.

[0013] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the vehicle tracking and re-perception method based on a large language model provided in the embodiments of the present invention.

[0014] The beneficial effects of this invention are: The solution provided by this invention can automatically and quickly initiate a re-sensing process after a target vehicle is lost, without manual intervention or lengthy system searches. Through intelligent analysis using a pre-trained large language model, control signals are rapidly output to guide following vehicles, significantly shortening the tracking interruption time after the target vehicle is lost and improving the continuity and overall efficiency of vehicle tracking. Based on a pre-trained large language model, this invention can learn and understand various complex vehicle driving scenarios and target vehicle behavior patterns. Whether it's a sudden turn of the target vehicle, varying degrees of occlusion, or multiple changes in complex road conditions, the large language model can make reasonable judgments and predictions based on the input image information, generating control signals adapted to different situations. Compared to traditional methods based on fixed rules, it has stronger adaptability and flexibility. Existing technologies often have limitations when processing limited information before the target vehicle is lost. This invention, however, utilizes a large language model to perform deep analysis of images from the seconds before the target vehicle disappears, fully extracting useful information from the images, such as changes in the target vehicle's posture and surrounding environmental features, thereby more accurately inferring the target vehicle's whereabouts and improving data utilization efficiency and the accuracy of re-sensing. Attached Figure Description

[0015] Figure 1 A schematic diagram illustrating the steps of a vehicle tracking and re-perception method based on a large language model provided in an embodiment of the present invention; Figure 2 A flowchart illustrating the workflow of a vehicle tracking and re-perception method based on a large language model, provided in an embodiment of the present invention. Figure 3 This diagram illustrates the reasoning, interpretation, and decision-making process of a large language model in a vehicle tracking and re-perception method based on a large language model, as provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0017] This invention provides a vehicle tracking and re-sensing method, system, electronic device, and storage medium based on a large language model.

[0018] It should be noted that the execution entity of the vehicle tracking and re-perception method based on a large language model provided in this embodiment of the invention can be a vehicle tracking and re-perception system based on a large language model, which can run in an electronic device. This electronic device can be a server or a terminal device, but is not limited to these.

[0019] Figure 1This is a schematic diagram illustrating the steps of a vehicle tracking and re-perception method based on a large language model provided in an embodiment of the present invention. Figure 2 Here is a flowchart of the vehicle tracking and re-sensing method. Below, we will first combine... Figure 1 and Figure 2 This invention introduces a vehicle tracking and re-perception method based on a large language model, as provided in an embodiment of the present invention.

[0020] The present invention provides a vehicle tracking and re-perception method based on a large language model, such as... Figure 1 and Figure 2 As shown, it may include the following steps: S1 acquires multimodal input data in real time through multiple vehicle-mounted sensors as the current input dataset, adds a timestamp to each current input dataset, and constructs a historical dataset from all acquired multimodal input data.

[0021] Multimodal input data is acquired in real time from multiple vehicle-mounted sensors as the current input dataset, and a timestamp is added to each current input dataset, which may include: The vehicle-mounted camera captures video frames of the target vehicle and its surrounding environment from all directions, and provides corresponding text descriptions for each video frame. Vehicle dynamic information is obtained through vehicle-mounted radar, and the shape and position of the vehicle are obtained through vehicle-mounted lidar. The video frame, the corresponding text description of the video frame, the vehicle's dynamic information, the vehicle's shape and position are used as the current input dataset, and a timestamp is added to each current input dataset.

[0022] Specifically, after entering the vehicle tracking task, the system acquires multimodal input data in real time through multiple onboard sensor devices: the onboard high-definition camera performs all-round image acquisition of the target vehicle and its surrounding environment at a frame rate of 30 frames per second to obtain video frames of the target vehicle and its surrounding environment. I Each video frame is accompanied by a corresponding text description. T The camera features autofocus, auto exposure, and wide dynamic range to ensure clear, accurate, and highly color-accurate images under various lighting conditions (such as direct sunlight, low light, and backlighting). The image resolution is set to at least 1920×1080 pixels to guarantee sufficient detail. Vehicle speed is obtained via onboard radar. V ,distance D Dynamic information, etc. Precise 3D point cloud data of the target vehicle is acquired through onboard LiDAR. P This provides more detailed environmental structure information, further identifying the shape and location of the target vehicle. All of the above data is used as the current input dataset. InputBy reading the system time and adding a timestamp to each current input dataset, effective registration of sensor data and image information is achieved. Simultaneously, this embodiment of the invention introduces a cyclical update mechanism. During subsequent tracking cycles, this mechanism continuously runs, collecting real-time environmental data after the vehicle's movement to form a real-time current input dataset, which, along with historical datasets, serves as input for subsequent decision-making.

[0023] S2 uses a pre-trained visual encoder to extract the current visual feature vector and the historical visual feature vector from video frames in the current input dataset and the historical dataset, respectively; and uses a pre-trained text encoder to convert the current text description and the historical text description corresponding to the video frame into the current text feature vector and the historical text feature vector.

[0024] Specifically, embodiments of the present invention use pre-trained visual encoders (such as ResNet, ViT, etc.) to extract high-dimensional current visual feature vectors from the current video frame and historical video frames, respectively. and historical visual feature vectors A caching mechanism is employed to store keyframe features, ensuring real-time processing efficiency. The number of encoder layers and feature dimensions are adjusted based on available computing resources. d For each video frame I i Extract visual feature vectors: z i =f vision ( I i )∈ R d , f vision It is a visual encoding function, and uses an efficient caching mechanism to store the latest N frames of visual features to achieve real-time processing.

[0025] This invention uses a pre-trained text encoder to convert the current text description and historical text description corresponding to video frames into high-dimensional current text feature vectors. and historical text feature vectors Adjust the text encoder parameters according to system requirements to ensure compatibility with the dimensions of visual features. For each text description... T i Extract text feature vectors: h i = f text ( T i )∈ R m , f text This represents a text encoding function.m This represents the dimension of the text feature vector. In this embodiment of the invention, timestamps are used to ensure the consistency of text features and corresponding video frame features along the timeline. Feature encoding is performed on radar and lidar data to obtain... , This ensures compatibility across all modal feature dimensions.

[0026] S3 dynamically fuses the current visual feature vector and the current text feature vector through an attention mechanism to generate the current multimodal representation, and dynamically fuses the historical visual feature vector and the historical text feature vector to generate the historical multimodal representation.

[0027] The current visual feature vector and the current text feature vector are dynamically fused through an attention mechanism to generate the current multimodal representation, which may include: An attention mechanism is introduced to obtain the query based on the visual feature vector and the key and value based on the current text feature vector. Based on the query, key, and value, the attention weights are calculated using a scaled dot product attention mechanism. The attention weights are fused with the current visual feature vector to output the corresponding current multimodal representation.

[0028] Specifically, embodiments of the present invention achieve visual feature recognition by introducing an attention mechanism. z i Text features h i Dynamic fusion to generate a unified multimodal representation Z multi This enhances the ability to identify and track target vehicles. Embodiments of this invention use visual features as a query criterion. Q Text features as keys K (Key) and value V (Value) enables information exchange and fusion between different modalities. It is defined as follows: ; ; ; in, This indicates a query for the corresponding learnable weight matrix. This represents the learnable weight matrix corresponding to the key. The learnable weight matrix corresponding to the value. This indicates the dimension corresponding to the value. This indicates the dimension corresponding to the value.

[0029] Next, the attention weights are calculated using the scaled dot product attention mechanism: ; Finally, the output of the attention mechanism is fused with the original visual features to generate the final multimodal representation. Z multi : .

[0030] Design an appropriate loss function to ensure that the fused multimodal representation can effectively support subsequent decision-making tasks. ;in, This represents the loss function of the decision-making module. This represents the actual control commands.

[0031] By introducing an attention mechanism, historical data (visual + text + radar + LiDAR) and current data (visual + text + radar + LiDAR) are internally fused separately to generate a unified historical multimodal representation. and current multimodal representation When entering the iterative decision-making stage, further integration is required. and ,generate This allows for a deep integration of historical experience with real-time conditions, enhancing the accuracy of decision-making.

[0032] S4, using a deep learning-based object detection algorithm to analyze video frames in the current input dataset in real time, when a target vehicle is detected to be missing, records the timestamp of the loss and the target vehicle information in the last video frame, which may include: Deep learning-based object detection algorithms construct feature vectors of target vehicles from video frames in the current input dataset, match the feature vectors of target vehicles with pre-trained target vehicle feature templates, and obtain matching scores. When the matching score of the target vehicle corresponding to the images of a consecutive preset number of frames is lower than the preset threshold, it is determined that the target vehicle is lost; at the same time, the timestamp of the target vehicle when it is lost and the target vehicle information in the last video frame are recorded; the preset number of frames is 5, the preset threshold is 0.5, and the target vehicle information includes the vehicle's position and attitude.

[0033] A deep learning-based target detection algorithm is used to analyze the acquired images in real time. This algorithm is pre-trained on a large dataset of vehicle images containing different types of vehicles and in different scenarios, enabling it to accurately identify the characteristics of target vehicles (such as vehicle model, color, and license plate). In each frame, the detection algorithm constructs a feature vector for the target vehicle and matches it with a pre-defined target vehicle feature template, calculating a matching score. When the matching score of the target vehicle in five consecutive frames is lower than a set threshold (e.g., 0.5), the target vehicle is considered lost. A closed-loop tracking process is immediately initiated, triggering historical data uploading and initial decision generation steps.

[0034] Once the target vehicle is detected as missing, the data transmission procedure is immediately initiated. Based on the timestamp of the target vehicle's disappearance, the multimodal representation of the last 5 seconds before its disappearance is transmitted to the large language model via the vehicle's internal high-speed data bus (such as Ethernet). A reliable data transmission protocol (such as TCP / IP) is used during transmission to ensure data integrity and accuracy, preventing data loss or corruption.

[0035] S5 utilizes a pre-trained large language model to parse the historical state of the target vehicle before its loss based on the timestamp at the time of loss and the corresponding historical multimodal representations within the previous preset timestamps; it constructs a historical scene model based on the historical state, calculates the strength of environmental constraints, and generates vehicle control command parameters through consistency indicators and priority filtering, which may include: By using a pre-trained large language model, the timestamp at the time of loss and the corresponding historical multimodal representation within the previous preset timestamp are analyzed to confirm the historical state of the target vehicle before it was lost. A historical scenario model is constructed based on the historical state of the target vehicle before it was lost, and the strength of environmental constraints is calculated to generate multiple behavioral hypotheses for the target vehicle. Valid hypotheses that meet the requirements of the consistency index are selected from all behavioral hypotheses. Valid hypotheses are quantified and ranked based on a priority scoring formula to select the optimal hypothesis. Based on optimal assumptions and combined with the strength of environmental constraints, control commands corresponding to target speed, steering angle, and acceleration are derived and output as vehicle control command parameters.

[0036] The pre-trained large language model takes "historical data - current data - re-perception results" as its core input and achieves closed-loop decision-making through the "hierarchical reasoning - cyclic optimization" mechanism. Each round of decision-making revolves around "data parsing - scenario modeling - hypothesis generation - verification and sorting - instruction generation".

[0037] In step S5, the pre-trained large language model only uses The input is used to parse the historical state of the target vehicle before it disappeared. The historical state of the target vehicle before it disappeared can include: trajectory, speed, acceleration, vehicle attitude and surrounding environmental constraints (such as traffic signs and road structure) before disappearance.

[0038] The input multimodal fusion data Z_multi is subjected to hierarchical analysis, and cross-modal feature associations are established through dynamic weight fusion formula. : The expression for cross-modal association features is as follows: ; in, Indicates the first Dynamic weights of visual features Indicates the first Dynamic weights of individual radar features Indicates the first Dynamic weights of environmental features Represents the first in the set of visual features A visual feature, Represents the first in the radar feature set One radar characteristic, Represents the first element in the set of environmental features. Environmental characteristics. Satisfying This provides a consistent data foundation for subsequent model inference.

[0039] Cross-modal feature association The set of visual modal features is It includes static features such as the target vehicle outline, trajectory before disappearance, vehicle attitude, and license plate; the radar modal feature set is... Includes the real-time speed of the target vehicle before it disappeared. acceleration Distance to surrounding objects; set of environmental features It includes the basic attributes of environmental factors within the scene, such as the total number of environmental factors. k Basic impact values ​​of traffic signs S p The straight-line distance between environmental factors and the target vehicle d p .

[0040] Based on generated cross-modal feature association According to the scenario p Each environmental factor is used to extract the set of environmental features. The basic impact value The straight-line distance between environmental factors and the target vehicle d p Construct a scene semantic model and calculate the real-time constraint strength of each environmental factor. C p Then the real-time constraint strength of this factor C p for: ; in, This represents the attenuation coefficient. The closer the distance and the higher the basic influence value, the stronger the constraint on the target vehicle's behavior.

[0041] Calculate the average constraint strength of all environmental factors within the scenario. Cavg As the environmental constraint strength, its formula is as follows: ; in, k This represents the total number of environmental factors.

[0042] By quantifying the "goal-environment interaction relationship", scenario boundary constraints are provided for subsequent behavioral assumptions.

[0043] Combined with the real-time speed of the target vehicle before it disappears from the cross-modal association features v t acceleration a t With the above-mentioned scenario constraint parameters C avg This generates behavioral assumptions for multiple target vehicles that conform to motion inertia and scene constraints.

[0044] Specifically, based on radar modal feature sets Includes the speed of the target vehicle before it disappeared. v t acceleration a t Based on common traffic behaviors, 5-8 behavioral hypotheses are initially generated, each including a predicted speed. Predicted acceleration , driving direction Three-dimensional parameters.

[0045] Introducing average constraint strength C avg As a scene constraint correction term, to ensure that the assumptions conform to motion inertia and scene rules, the formula is as follows: ; Here, represents the duration of the target vehicle's disappearance. Indicates the maximum speed allowed in the scene. Indicates the acceleration allowed by the scenario. This indicates the scene correction weight.

[0046] filter The assumptions were determined to conform to the motion inertia, and 3-5 were generated and included in the candidate set. Each candidate set contained predicted velocities. Predicted acceleration , driving direction For subsequent use.

[0047] In the process of selecting high-priority hypotheses, based on the valid hypotheses in the candidate set and the parameters in the above steps, a quantitative score is calculated from three dimensions: compliance, reasonableness, and scenario suitability. This score is then used to determine the priority of the hypotheses. The best hypothesis is selected to provide a unique basis for the generation of subsequent vehicle control commands.

[0048] After verifying the feasibility of candidate hypotheses, they are quantified and ranked using a priority scoring formula: compliance score. A Verify the compliance of the assumptions based on traffic rules; reasonableness score. B Directly mapping consistency metrics ,Right now B = Scene adaptability score C Average constraint strength of the mapping C avg That is, C= C avg Priority scoring The expression is as follows: ; Among them, the weighting coefficient , , ,satisfy .according to Sort values ​​in descending order and filter out The assumption is that... The highest optimal hypothesis is taken as the optimal hypothesis, and its corresponding prediction speed is determined. Predicted acceleration , driving direction It serves as the core input for vehicle control commands.

[0049] Based on the optimal assumptions selected, and combined with the dynamic parameters corresponding to the preceding steps, the three types of control commands—target speed, steering angle, and acceleration—are derived, generating vehicle control command parameters: Target speed command v cmd : Based on the assumed velocity change trend, combined with historical velocity decay factors Receive target speed command v cmd ; Target speed command v cmd The expression is as follows: ; in, Indicates the lead time for instruction execution.

[0050] Steering angle command : Driving direction based on optimal assumptions With visual modal feature set as F v Target vehicle heading angle Calculate directional deviation Steering angle command The formula is as follows: ; in, This represents the steering gain (k=1.2 for city roads, k=0.8 for highways). This indicates the vehicle's wheelbase.

[0051] Acceleration command a cmd : Predicted acceleration using optimal assumptions and combined with average constraint strength C avg Correct the acceleration calculation command a cmd : .

[0052] Ultimately, the target speed command will be executed. v cmd Steering angle command Acceleration command a cmd It is integrated into a structured set of control commands, which are then used as vehicle control command parameters.

[0053] S6. After the vehicle executes the vehicle control command parameters, it acquires the current multimodal input data and generates a re-perception result. If the re-perception result is a successful perception, it continues to track the target vehicle. If the re-perception result is a failed perception, it merges the timestamp when the loss occurred, the corresponding historical multimodal representation within the previously preset timestamp, and the current multimodal representation into a deep-combined mode, updates the environmental constraint strength, dynamically adjusts the weights corresponding to each mode, and recalculates the consistency index and priority to generate new vehicle control command parameters.

[0054] Specifically, after the vehicle executes the vehicle control command parameters, it acquires the current multimodal input data and generates the re-sensing result: If the perception result is "successful perception", the pre-trained large language model outputs a loop termination signal and continues to track the target vehicle. If the perception result is "unsuccessful perception", the iterative decision-making process is initiated, which integrates the current multimodal representation with the historical multimodal representation to obtain a deeply combined modality for iterative decision optimization.

[0055] The expression for the deep-binding modality is as follows: ; in, Indicates deep-association mode, Indicates the weight of historical data. Representing historical multimodal representation, Indicates the current data weight. This represents the current multimodal representation. Understandably, in the initial decision-making phase corresponding to step S5, , In the iterative decision-making stage corresponding to step S6, , ;satisfy The dynamic balance between the impact of historical and real-time data is achieved.

[0056] Iterative decision optimization: If the target vehicle is not successfully detected, the pre-trained large language model integrates historical and current dual-dimensional data to optimize the decision generation process. First, it analyzes the differences between the current environmental changes and historical scenarios, updates the environmental constraint strength, and dynamically adjusts the weights of visual, radar, and environmental features. Based on current environmental data, it corrects historical behavior assumptions, eliminating assumptions that contradict current road conditions (such as predicting the target vehicle to go straight when the current road is a dead end), supplementing with new assumptions that fit the current environment, recalculating consistency indices and priorities, and selecting the optimal assumption. Based on the corrected optimal assumption, combined with current environmental constraints, it optimizes and generates new vehicle control command parameters, making the commands more aligned with real-time road conditions, increasing the probability of successful detection, and ensuring that the target vehicle can be tracked again.

[0057] The inference, explanation, and decision-making process diagram of the large language model in the vehicle tracking and re-perception method based on the large language model provided in this embodiment of the invention is as follows: Figure 3 As shown, it can be seen that by engaging in a series of interactions with the large language model, the corresponding vehicle control command parameters are obtained.

[0058] The vehicle control command parameters generated by the large language model are encapsulated according to a predetermined data format to ensure signal integrity and readability. Low-latency, high-reliability communication methods are used to transmit control signals, and data verification is performed during transmission to ensure no errors or loss occur. Simultaneously, priority identifiers are added to control signals to ensure that vehicle tracking and re-sensing related control signals are processed first when the vehicle control system is busy.

[0059] Vehicle kinematics are described using a bicycle model: ; ; ; ; in, Indicates the vehicle's location. Indicates the heading angle. Indicates longitudinal velocity. Indicates the steering angle. Indicates longitudinal acceleration. Indicates the vehicle's wheelbase. Indicates a time interval.

[0060] High-level vehicle control commands generated from large language models Parameters are converted into specific vehicle control signals. To control the movement of the vehicle: Speed ​​control: ; Steering control: ; Acceleration control: ; in, Mapping This indicates that higher-level vehicle control commands are converted into lower-level vehicle control signals.

[0061] Since there is a deviation between the actual trajectory of the vehicle and the reference trajectory, this embodiment of the invention uses model predictive control for trajectory optimization: ; in, , , Indicates the weights of different control items. , This indicates the reference speed and heading angle.

[0062] After receiving the vehicle control signal, the system analyzes the parameters within it. Based on the steering angle parameter, it precisely controls the vehicle's steering through the electric power steering system, ensuring a steering angle error within ±0.2 degrees. Based on acceleration or deceleration values, it adjusts the vehicle's throttle or braking system to achieve smooth speed control, with a speed control error not exceeding ±0.2 m / s. While the vehicle is traveling according to the control signal, multimodal information is continuously collected. Once the target vehicle reappears within the image range, the system further fine-tunes the vehicle's driving state according to preset task requirements, restoring a stable vehicle tracking state. If the target vehicle is not detected, the system repeats the series of steps to generate vehicle control command parameters, regenerating the parameters to ensure continuous vehicle tracking and re-perception.

[0063] The vehicle tracking and re-sensing method based on a large language model provided in this invention can automatically and quickly initiate a re-sensing process after a target vehicle is lost, without manual intervention or lengthy system searches. Through intelligent analysis by the large language model, control signals are rapidly output to guide the actions of following vehicles, significantly shortening the tracking interruption time after the target vehicle is lost and improving the continuity and overall efficiency of vehicle tracking. The vehicle tracking and re-sensing method proposed in this invention is based on a large language model, capable of learning and understanding various complex vehicle driving scenarios and target vehicle behavior patterns. Whether it's a sudden turn by the target vehicle, varying degrees of occlusion, or multiple changes under complex road conditions, the large language model can make reasonable judgments and predictions based on the input image information, generating control signals adapted to different situations. Compared to traditional methods based on fixed rules, it has stronger adaptability and flexibility.

[0064] Existing technologies often have limitations when processing limited information about a target vehicle before it disappears. However, the vehicle tracking and re-sensing method proposed in this invention uses a large language model to perform in-depth analysis on images a few seconds before the target vehicle disappears. This allows for the full extraction of useful information from the images, such as changes in the target vehicle's posture and features of the surrounding environment. As a result, the direction of the target vehicle can be inferred more accurately, improving data utilization efficiency and the accuracy of re-sensing.

[0065] Secondly, corresponding to the above-described vehicle tracking and re-perception method embodiment based on a large language model, this embodiment of the invention also provides a vehicle tracking and re-perception system based on a large language model, which may include: The system comprises a vehicle perception module, a data transmission module, a large language model processing module, and a vehicle control module; among which... The vehicle perception module acquires multimodal input data in real time through multiple onboard sensors as the current input dataset; The data transmission module transmits the current input dataset to the large language model processing module; The large language model processing module adds timestamps to each current input dataset, constructing a historical dataset from all acquired multimodal input data. It uses a pre-trained visual encoder to extract current and historical visual feature vectors from video frames in both the current and historical datasets. A pre-trained text encoder converts the current and historical text descriptions corresponding to the video frames into current and historical text feature vectors. An attention mechanism dynamically fuses the current visual and text feature vectors to generate a current multimodal representation, and similarly, it dynamically fuses the historical visual and text feature vectors to generate a historical multimodal representation. A deep learning-based object detection algorithm analyzes the video frames in the current input dataset in real time. When a target vehicle is detected as missing, the module records the timestamp of the loss and the target vehicle information from the last video frame. Using the pre-trained large language model, it parses the historical state of the target vehicle before its loss based on the timestamp of the loss and the corresponding historical multimodal representation within the previously preset timestamps. A historical scene model is constructed based on the historical state, calculating the environmental constraint strength and using consistency indicators and priority filtering to generate vehicle control command parameters. The vehicle control module executes vehicle control command parameters, acquires the current multimodal input data, and generates a re-perception result. If the re-perception result is successful, it continues to track the target vehicle. If the re-perception result is unsuccessful, it uses the large language model processing module to fuse the timestamp at the time of loss, the corresponding historical multimodal representation within the previously preset timestamp, and the current multimodal representation into a deep-binding modality. It updates the environmental constraint strength, dynamically adjusts the weights corresponding to each modality, and recalculates the consistency index and priority to generate new vehicle control command parameters. Based on the new vehicle control command parameters, it controls the vehicle's movement and determines whether perception is successful. If perception is unsuccessful, it uses the large language model processing module to repeatedly execute the process of generating new vehicle control command parameters until perception is successful, thus achieving vehicle tracking and re-perception.

[0066] Specifically, the vehicle perception module is responsible for collecting image information of the target vehicle and its surrounding environment. High-definition cameras and other equipment are used to ensure image clarity and accuracy, providing a high-quality data source for subsequent processing. The data transmission module facilitates data transmission between the vehicle perception module and the large language model processing module. It rapidly and stably transmits image data from the few seconds before the target vehicle disappears to the large language model processing module, ensuring the timeliness and integrity of data transmission. The large language model processing module receives image data from the data transmission module using a built-in pre-trained large language model, analyzes and processes the images, and, combining its learned knowledge and patterns, outputs vehicle control command parameters to control the movement of following vehicles to re-perceive the target vehicle. The vehicle control module, based on the vehicle control command parameters output by the large language model processing module, controls the driving direction, speed, and other parameters of following vehicles, enabling them to drive according to a predetermined strategy to achieve the goal of re-perceiving the target vehicle.

[0067] Understandably, during the tracking process, the vehicle perception module continuously collects images of the target vehicle and its surrounding environment. When the target vehicle becomes undetectable due to turning or obstruction, the data transmission module immediately uploads images of the target vehicle from a few seconds prior to its disappearance to the large language model processing module. The large language model processing module analyzes the images, combines them with a pre-trained model, and outputs vehicle control command parameters to the vehicle control module. The vehicle control module generates corresponding vehicle control signals based on these parameters, adjusts the vehicle's driving state, and ensures that the following vehicle follows the path and method determined by the large language model, ultimately regaining detection of the target vehicle.

[0068] For example, in an urban road scenario: On a city road, mission vehicle A is tracking vehicle B. Suddenly, vehicle B disappears due to being obscured by a building ahead; The data acquisition module quickly collects images and data of vehicle B before it disappears and uploads them to the large language model; The large language model analyzes the behavior of vehicle B, infers that it may have changed lanes, and suggests that vehicle A slow down and keep going straight. Vehicle A followed the advice and drove to the location where Vehicle B had disappeared, then detected Vehicle B again and continued to track it.

[0069] For example, in a highway scenario: On the highway, vehicle A is tracking vehicle B. Vehicle B suddenly turns at an exit ramp and disappears from sight. The data acquisition module collects data and images of vehicle B before it disappears and uploads them to the large language model; The large language model analyzes the turning behavior of vehicle B and suggests that vehicle A adjust its speed and continue forward at the ramp entrance; Vehicle A drove to the ramp entrance as suggested, but failed to detect Vehicle B again, so it consulted the large language model again. Based on the current road environment, the large language model suggests that vehicle A continue forward and re-perceive the road at the next exit. Vehicle A followed the advice and eventually detected Vehicle B again at the next exit, and continued to track it.

[0070] The vehicle tracking and re-sensing system proposed in this invention uses image information from a few seconds before the target vehicle disappears as input to a large language model. Through the powerful analysis and reasoning capabilities of the model, it predicts the possible location and driving state of the target vehicle, thereby generating effective control signals. By receiving and analyzing images and vehicle data before the tracked vehicle disappears, it uses the large language model to infer the operational behavior of the tracked vehicle and provides tracking suggestions.

[0071] This invention enables automatic and rapid initiation of a re-sensing process after a target vehicle is lost, without manual intervention or lengthy system searches. Through intelligent analysis using a pre-trained large language model, control signals are quickly output to guide following vehicles, significantly reducing tracking interruption time after a target vehicle is lost and improving the continuity and overall efficiency of vehicle tracking. Based on a pre-trained large language model, this invention can learn and understand various complex vehicle driving scenarios and target vehicle behavior patterns. Whether it's a sudden turn, varying degrees of occlusion, or multiple changes in complex road conditions, the large language model can make reasonable judgments and predictions based on the input image information, generating control signals adapted to different situations. Compared to traditional methods based on fixed rules, it has stronger adaptability and flexibility. Existing technologies often have limitations when processing limited information before a target vehicle is lost. This invention, however, utilizes a large language model to perform deep analysis of images from the seconds before the target vehicle disappears, fully extracting useful information from the images, such as changes in the target vehicle's posture and surrounding environmental features, thereby more accurately inferring the target vehicle's whereabouts and improving data utilization efficiency and re-sensing accuracy.

[0072] Thirdly, embodiments of the present invention also provide an electronic device, such as... Figure 4 As shown, it includes a processor 001, a communication interface 002, a memory 003, and a communication bus 004, wherein the processor 001, the communication interface 002, and the memory 003 communicate with each other through the communication bus 004. The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of any of the vehicle tracking and re-perception methods based on a large language model provided in the first aspect of the present invention.

[0073] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0074] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0075] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0076] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0077] The vehicle tracking and re-perception method based on a large language model provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0078] Fourthly, corresponding to the vehicle tracking and re-perception method based on a large language model provided in the first aspect, this embodiment of the invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the vehicle tracking and re-perception methods based on a large language model provided in the first aspect of this invention.

[0079] For the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0080] It should be noted that the system, electronic device and storage medium in the embodiments of the present invention are respectively the system, electronic device and storage medium applying the above-mentioned vehicle tracking and re-sensing method based on large language model. Therefore, all embodiments of the above-mentioned vehicle tracking and re-sensing method based on large language model are applicable to the system, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0081] It should be noted that, in the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0082] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A vehicle tracking and re-perception method based on a large language model, characterized in that, include: S1: Acquire multimodal input data in real time through multiple vehicle-mounted sensors as the current input dataset, and add a timestamp to each current input dataset; All acquired multimodal input data are used to construct a historical dataset; S2, using a pre-trained visual encoder to extract the current visual feature vector and the historical visual feature vector from video frames in the current input dataset and the historical dataset respectively; using a pre-trained text encoder to convert the current text description and the historical text description corresponding to the video frame into the current text feature vector and the historical text feature vector. S3 uses an attention mechanism to dynamically fuse the current visual feature vector and the current text feature vector to generate the current multimodal representation, and dynamically fuses the historical visual feature vector and the historical text feature vector to generate the historical multimodal representation. S4 uses a deep learning-based target detection algorithm to analyze video frames in the current input dataset in real time. When a target vehicle is detected to be missing, the timestamp of the loss and the target vehicle information in the last video frame are recorded. S5 utilizes a pre-trained large language model to parse the historical state of the target vehicle before it was lost, based on the timestamp of the loss and the corresponding historical multimodal representation within the previous preset timestamp; it constructs a historical scene model based on the historical state, calculates the strength of environmental constraints, and generates vehicle control command parameters by filtering through consistency indicators and priorities. S6. After the vehicle executes the vehicle control command parameters, it obtains the current multimodal input data and generates a re-perception result. If the re-perception result is a successful perception, it continues to track the target vehicle. If the re-perception result is a failed perception, it merges the timestamp when the vehicle was lost, the corresponding historical multimodal representation within the previously preset timestamp, and the current multimodal representation into a deep-combined mode, updates the environmental constraint strength, dynamically adjusts the weights corresponding to each mode, and recalculates the consistency index and priority to generate new vehicle control command parameters. S7 controls the vehicle's movement based on the new vehicle control command parameters, determines whether the perception is successful, and if the perception is unsuccessful, repeats step S6 until the perception is successful, so as to achieve vehicle tracking and re-perception.

2. The vehicle tracking and re-perception method based on a large language model according to claim 1, characterized in that, The process of acquiring multimodal input data in real time through multiple vehicle-mounted sensors as the current input dataset, and adding a timestamp to each current input dataset, includes: The vehicle-mounted camera captures video frames of the target vehicle and its surrounding environment from all directions, and provides corresponding text descriptions for each video frame. Vehicle dynamic information is obtained through vehicle-mounted radar, and the shape and position of the vehicle are obtained through vehicle-mounted lidar. The video frame, the corresponding text description of the video frame, the vehicle's dynamic information, the vehicle's shape and position are used as the current input dataset, and a timestamp is added to each current input dataset.

3. The vehicle tracking and re-perception method based on a large language model according to claim 1, characterized in that, The process of dynamically fusing the current visual feature vector and the current text feature vector through an attention mechanism to generate the current multimodal representation includes: An attention mechanism is introduced to obtain the query based on the visual feature vector and the key and value based on the current text feature vector. Based on the query, key, and value, the attention weights are calculated using a scaled dot product attention mechanism. The attention weights are fused with the current visual feature vector to output the corresponding current multimodal representation.

4. The vehicle tracking and re-perception method based on a large language model according to claim 1, characterized in that, The method involves real-time analysis of video frames in the current input dataset using a deep learning-based target detection algorithm. When a target vehicle is detected as missing, the timestamp of the loss and the target vehicle information from the last video frame are recorded, including: Deep learning-based object detection algorithms construct feature vectors of target vehicles from video frames in the current input dataset, match the feature vectors of target vehicles with pre-trained target vehicle feature templates, and obtain matching scores. When the matching score of the target vehicle corresponding to a consecutive preset number of frames is lower than a preset threshold, it is determined that the target vehicle is lost; at the same time, the timestamp of the target vehicle being lost and the target vehicle information in the last video frame are recorded; wherein, the preset number of frames is 5, the preset threshold is 0.5, and the target vehicle information includes the vehicle's position and attitude.

5. The vehicle tracking and re-perception method based on a large language model according to claim 1, characterized in that, The pre-trained large language model is used to analyze the historical state of the target vehicle before it was lost, based on the timestamp at the time of loss and the corresponding historical multimodal representation within the previous preset timestamp. Based on historical states, a historical scenario model is constructed, the strength of environmental constraints is calculated, and vehicle control command parameters are generated through consistency indicators and priority filtering, including: By using a pre-trained large language model, the timestamp at the time of loss and the corresponding historical multimodal representation within the previous preset timestamp are analyzed to confirm the historical state of the target vehicle before it was lost. A historical scenario model is constructed based on the historical state of the target vehicle before it was lost, and the strength of environmental constraints is calculated to generate multiple behavioral hypotheses for the target vehicle. Valid hypotheses that meet the requirements of the consistency index are selected from all behavioral hypotheses. Valid hypotheses are quantified and ranked based on a priority scoring formula to select the optimal hypothesis. Based on optimal assumptions and combined with the strength of environmental constraints, control commands corresponding to target speed, steering angle, and acceleration are derived and output as vehicle control command parameters.

6. The vehicle tracking and re-perception method based on a large language model according to claim 5, characterized in that, The historical state of the target vehicle before it was lost includes: Trajectory, velocity, acceleration, vehicle attitude, and environmental constraints before disappearance.

7. The vehicle tracking and re-perception method based on a large language model according to claim 1, characterized in that, The expression for the depth-binding mode is as follows: ； in, Indicates deep-association mode, Indicates the weight of historical data. Representing historical multimodal representation, Indicates the current data weight. This indicates the current multimodal representation.

8. A vehicle tracking and re-sensing system based on a large language model, characterized in that, include: The system comprises a vehicle perception module, a data transmission module, a large language model processing module, and a vehicle control module; among which... The vehicle perception module acquires multimodal input data in real time through multiple on-board sensors as the current input dataset; The data transmission module transmits the current input dataset to the large language model processing module; The large language model processing module adds a timestamp to each current input dataset, constructing a historical dataset from all acquired multimodal input data. It then uses a pre-trained visual encoder to extract current and historical visual feature vectors from video frames in both the current and historical datasets. A pre-trained text encoder converts the current and historical text descriptions corresponding to the video frames into current and historical text feature vectors. An attention mechanism dynamically fuses the current visual and text feature vectors to generate a current multimodal representation, and similarly, it dynamically fuses the historical visual and text feature vectors to generate a historical multimodal representation. A deep learning-based target detection algorithm analyzes the video frames in the current input dataset in real time. When a target vehicle is detected as missing, the module records the timestamp of the loss and the target vehicle information from the last video frame. Using the pre-trained large language model, it parses the historical state of the target vehicle before its loss based on the timestamp of the loss and the corresponding historical multimodal representation within the previously preset timestamps. A historical scene model is constructed based on the historical state, and the strength of environmental constraints is calculated. Consistency indicators and priority filtering are used to generate vehicle control command parameters. The vehicle control module executes the vehicle control command parameters, acquires the current multimodal input data, and generates a re-perception result. If the re-perception result is successful, it continues to track the target vehicle. If the re-perception result is unsuccessful, it uses the large language model processing module to fuse the timestamp at the time of loss, the corresponding historical multimodal representation within the previously preset timestamp, and the current multimodal representation into a deep-binding modality. It updates the environmental constraint strength, dynamically adjusts the weights corresponding to each modality, and recalculates the consistency index and priority to generate new vehicle control command parameters. Based on the new vehicle control command parameters, it controls the vehicle's movement and determines whether the perception is successful. If the perception is unsuccessful, it uses the large language model processing module to repeatedly execute the process of generating new vehicle control command parameters until successful perception is achieved, thus realizing vehicle tracking and re-perception.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the vehicle tracking and re-perception method based on a large language model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the vehicle tracking and re-perception method based on a large language model as described in any one of claims 1-7.

Images