Traffic escape behavior semantic retrieval system and method based on special visual large model

Abstract

The application relates to the technical field of artificial intelligence, and discloses a traffic escape behavior semantic retrieval system and method based on a special visual large model, which comprises a multi-modal data analysis module that converts structured annotation data into a text sequence, activates a state bit marker and extracts a mask matrix and a key frame interval index set when a complete cause-effect logic chain is detected; a cross-modal encoding and fusion module that performs cross-modal fusion calculation on videos and texts, intercepts a cross-attention matrix and a deep visual feature vector sequence when the marker is activated; a joint supervision optimization module that constructs a three-dimensional joint loss function based on the intercepted features and performs back propagation to update model parameters; and a semantic retrieval module that calculates the similarity of a query instruction text and a to-be-retrieved video visual relationship feature vector and outputs a result. The application realizes alignment and mapping of bottom-layer physical features and upper-layer cause-effect logic, and improves the accuracy of complex traffic escape behavior retrieval.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a semantic retrieval system and method for traffic escape behavior based on a dedicated large visual model. Background Technology

[0002] In the field of intelligent transportation security, rapid retrieval of surveillance videos is a crucial step in handling traffic accidents and violations. Current traffic video analysis largely relies on basic computer vision algorithms such as object detection and trajectory tracking, combined with manually defined numerical rules to determine the occurrence of events. This approach, based on single physical attributes and hard-coded rules, has limitations when dealing with events like hit-and-run accidents, which possess complex behavioral characteristics. Hit-and-run accidents not only involve the instantaneous physical collision but also the subsequent departure action, exhibiting a causal logical connection.

[0003] While existing large-scale visual and linguistic multimodal models possess some cross-modal semantic extraction capabilities, their attention mechanisms are often scattered across the entire image when directly applied to complex traffic monitoring scenarios, lacking physical constraints on the actual spatial interaction areas of the collision. Furthermore, in processing temporal features, conventional models tend to smooth or pool features from consecutive frames, failing to capture the high-frequency temporal vibrations and abrupt energy changes generated during minor collisions. Due to the lack of dedicated supervision mechanisms for microscopic spatial interactions and high-frequency temporal jumps, existing systems are prone to misjudging easily confused scenarios such as close-range safe encounters and genuine collision escapes. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a semantic retrieval system and method for traffic escape behavior based on a dedicated large-scale visual model. This solves the problem that existing traffic video retrieval methods struggle to establish joint supervision constraints between microscopic physical feature jumps and macroscopic causal logic, resulting in limited retrieval accuracy for complex traffic escape behaviors.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] The first aspect of the present invention provides a semantic retrieval system for traffic escape behavior based on a dedicated large visual model. The system includes: a multimodal data parsing module, a cross-modal coding and fusion module, a joint supervised optimization module, and a semantic retrieval module.

[0007] The multimodal data parsing module is used to receive raw traffic video sequences and structured labeled data, convert the structured labeled data into natural language text sequences, and activate the status bit flag when the structured labeled data is detected to have the integrity of a causal logical chain.

[0008] The cross-modal coding and fusion module is used to convert the original traffic video sequence into a visual word sequence and map the natural language text sequence into a text embedding vector. It uses a cross-attention mechanism for fusion calculation and extracts the cross-modal cross-attention matrix and deep visual feature vector sequence when the state bit flag is detected to be activated.

[0009] The joint supervised optimization module is used to construct a three-dimensional joint loss function based on the truncated cross-modal attention matrix and the deep visual feature vector sequence, calculate the overall objective optimization function, and perform backpropagation to update the model parameters.

[0010] The semantic retrieval module is used to calculate the similarity between the text semantic vector of the natural language query command and the extracted visual relationship feature vectors pre-stored in the video library, and output the traffic escape video retrieval results.

[0011] Furthermore, the multimodal data parsing module parses the structured labeled data to obtain the subject relationships, spatial interaction distances, physical features, and causal logic chains representing traffic conflicts, and fills them into a preset prompt word template to reconstruct the target text sequence as the natural language text sequence. The multimodal data parsing module reads the causal logic chain field in the structured labeled data and performs character-level matching with a preset semantic tag library. When the causal logic chain field simultaneously contains a pre-collision tag representing physical contact and a post-escape tag representing subjective departure, it is determined that there is a complete causal logic chain from physical collision to subjective escape.

[0012] Furthermore, the cross-modal coding and fusion module maps the visual word sequence and the text embedding vector to the same hidden feature dimension through a linear projection layer; the cross-modal coding and fusion module uses the text embedding vector as the query feature of the cross-attention mechanism, and the visual word sequence as the key feature and value feature, calculates the inner product similarity between the query feature and the key feature, divides it by the dimension scaling factor, and then completes the cross-modal fusion calculation through a normalization function.

[0013] Furthermore, the multimodal data parsing module obtains the bounding box coordinate information of entities participating in traffic interaction. On a two-dimensional all-zero matrix aligned with the resolution pixels of a single frame image of the original traffic video sequence, the matrix element values ​​corresponding to the inner region of the bounding box are set to constant 1, and the matrix element values ​​corresponding to the outer region of the bounding box are set to constant 0 to generate the mask matrix. The multimodal data parsing module identifies the time start and time end of the instantaneous physical change of the traffic conflict, and converts them into a start frame index and a stop frame index by combining the frame rate parameters of the original traffic video sequence. The consecutive frame numbers between the start frame index and the stop frame index are constructed into the key frame interval index set.

[0014] Furthermore, when the status flag is activated, the cross-modal coding and fusion module extracts the cross-modal cross-attention matrix from the last network branch of the multimodal fusion layer, where the row dimension corresponds to the length of the text sequence and the column dimension corresponds to the number of visual lexical units. The cross-modal coding and fusion module obtains the keyframe interval index set, addresses it to the output of the projection layer of the visual network, and extracts the deep visual feature vector sequence corresponding to the time range. When the difference between the termination frame index and the starting frame index is less than 2, equal-width padding is performed on both sides of the time axis to set the length of the extracted continuous frame sequence to be no less than 3 frames.

[0015] Furthermore, the three-dimensional joint loss function includes basic semantic generation loss, spatial attention loss, and temporal feature fluctuation loss; the joint supervised optimization module uses an autoregressive cross-entropy loss function to quantify the conditional probability error of correctly predicting the next target word under the condition of current visual features and preceding text words, to obtain the basic semantic generation loss; the joint supervised optimization module calculates the overlap energy ratio of the cross-modal cross-attention matrix and the mask matrix in the feature map space, and adds a preset smoothing constant to the denominator of the division operation for smoothing, to obtain the spatial attention loss.

[0016] Further, the joint supervised optimization module calculates the squared second-order discrete difference norm of the feature vectors of adjacent frames in the deep visual feature vector sequence, quantizes it to obtain the energy magnitude fluctuation amplitude, and when the energy magnitude fluctuation amplitude is lower than the set abrupt change energy constant threshold, it generates a penalty gradient by maximizing the output positive difference of the function to obtain the temporal feature fluctuation loss; the joint supervised optimization module uses basic semantic weights, spatial supervision weights and temporal supervision weights to linearly combine the basic semantic generation loss, the spatial attention loss and the temporal feature fluctuation loss to obtain the overall objective optimization function.

[0017] Furthermore, the semantic retrieval module uses the inter-frame difference method to calculate the pixel change rate of continuous images in the traffic videos to be inspected in the video library. When the pixel change rate exceeds the set motion threshold, the core time window of the traffic conflict is located, and non-uniform frame extraction is performed to extract the key frame sequence. The semantic retrieval module converts the key frame sequence into a word matrix and assigns it a time position encoding, and outputs the visual relationship feature vector.

[0018] Furthermore, the semantic retrieval module extracts the text semantic vector of the natural language query instruction, calculates the inner product between the text semantic vector and the visual relationship feature vector, and divides it by the product of the moduli of their two norms. A preset smoothing constant is added during the denominator calculation to obtain the similarity value. The associated traffic video segments are sorted in descending order according to the similarity value, and video segments with similarity values ​​lower than the similarity filtering threshold are removed to output the traffic escape video retrieval result.

[0019] A second aspect of the present invention provides a semantic retrieval method for traffic escape behavior based on a dedicated large visual model, the method comprising:

[0020] Receive raw traffic video sequences and structured labeled data, convert the structured labeled data into natural language text sequences, and activate the status bit flag when the structured labeled data is detected to have the integrity of a causal logical chain;

[0021] The original traffic video sequence is converted into a visual word sequence and the natural language text sequence is mapped into a text embedding vector. A cross-attention mechanism is used for fusion calculation, and when the activation of the state bit flag is detected, the cross-modal cross-attention matrix and the deep visual feature vector sequence are extracted.

[0022] A three-dimensional joint loss function is constructed based on the extracted cross-modal cross-attention matrix and the deep visual feature vector sequence. The overall objective optimization function is calculated and backpropagation is performed to update the model parameters.

[0023] Calculate the similarity between the text semantic vector of the natural language query command and the extracted visual relationship feature vectors pre-stored in the video library, and output the traffic escape video retrieval results.

[0024] This invention provides a semantic retrieval system and method for traffic escape behavior based on a dedicated large-scale visual model. It has the following beneficial effects:

[0025] 1. This invention activates state flags by detecting the integrity of causal logic chains in structured data, and dynamically extracts cross-modal attention matrices and deep visual feature vector sequences during model forward propagation; it avoids introducing irrelevant traffic scene data into the specific parameter fine-tuning process, ensuring that the model optimization process is only performed on segments with real collision and escape logic, thus improving the targeting of feature extraction.

[0026] 2. This invention constructs a three-dimensional joint loss function that includes spatial attention loss and temporal feature fluctuation loss. It applies spatial physical constraints by the mask overlap ratio and uses second-order discrete difference quantization to quantize the high-frequency feature jumps at the moment of collision. It introduces the microscopic spatial interaction position and temporal energy fluctuation into the optimization objective of the model, which improves the model's ability to perceive instantaneous micro-movements such as traffic collisions and realizes the alignment of the underlying physical features with the upper-level escape motive.

[0027] 3. In the retrieval stage, this invention calculates the similarity between the text semantic vector of the natural language instruction and the extracted visual relationship feature vector, and outputs video segments based on the similarity filtering threshold. Unlike methods that rely on manually hard-coded rules, this matching mechanism is based on the alignment mapping of cross-modal feature space, which can directly process unstructured natural language queries, reduce misjudgments caused by complex background environments, and improve the accuracy of video retrieval for traffic escape incidents. Attached Figure Description

[0028] Figure 1 This is a system architecture diagram of the present invention;

[0029] Figure 2 This is a flowchart of the method of the present invention;

[0030] Figure 3 This is a flowchart of the cross-modal coding and fusion module processing of the present invention;

[0031] Figure 4 This is a flowchart of the joint supervision optimization module of the present invention;

[0032] Figure 5 This is a comparison chart showing the effectiveness of the retrieval method of the present invention. Detailed Implementation

[0033] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] See attached document Figure 1 This invention provides a semantic retrieval system for traffic escape behavior based on a dedicated large visual model. The retrieval system includes: a multimodal data parsing module, a cross-modal coding and fusion module, a joint supervised optimization module, and a semantic retrieval module.

[0035] In the system's specific workflow, the multimodal data parsing module receives raw traffic video sequences and corresponding structured labeled data. The module then parses the structured labeled data to obtain the subject relationships, spatial interaction distances, physical characteristics, and corresponding causal chains representing traffic conflicts.

[0036] Subsequently, the multimodal data parsing module transforms the data containing features and causal logic chains into natural language text sequences based on a preset template.

[0037] The multimodal data parsing module has built-in integrity verification logic. It detects the integrity of the causal logic chain from physical collision to subjective escape contained in the current input data and activates the status bit flag when the logic chain is complete.

[0038] In terms of spatial and temporal feature processing, the multimodal data parsing module extracts the spatial dimension of entity bounding box coordinates from structured labeled data and maps them into a mask matrix in pixel space. The multimodal data parsing module also extracts the temporal span of physical abrupt changes in traffic conflicts and generates a corresponding keyframe interval index set.

[0039] The cross-modal coding and fusion module is connected to the multimodal data parsing module, receiving raw traffic video sequences and natural language text sequences. The cross-modal coding and fusion module converts the raw traffic video sequences into visual word sequences and maps the natural language text sequences into text embedding vectors.

[0040] The cross-modal coding and fusion module uses a cross-attention mechanism to perform forward propagation fusion calculations on visual word sequences and text embedding vectors, establishing an alignment relationship between visual motion vectors and text semantic instructions.

[0041] Furthermore, the cross-modal coding and fusion module monitors state bit flags during multimodal fusion computation. When a state bit flag is activated, the cross-modal coding and fusion module triggers a feature interception mechanism to extract the cross-modal cross-attention matrix from the current network layer. Based on the keyframe interval index set, the cross-modal coding and fusion module extracts the deep visual feature vector sequence of the corresponding continuous image from the output of the visual projection layer.

[0042] The joint supervised optimization module is connected to the cross-modal coding and fusion module, receiving truncated feature data to construct a three-dimensional joint loss function. The joint supervised optimization module calculates the basic semantic generation loss and maximizes the conditional probability of predicting target lexical units through an autoregressive forward cross-entropy function.

[0043] The joint supervised optimization module obtains the cross-attention matrix and mask matrix, calculates the overlap energy ratio of the two in the feature map space, and generates spatial attention loss. The joint supervised optimization module receives the deep visual feature vector sequence, calculates the second-order discrete difference of the feature vector sequence on the time axis to quantify the energy magnitude of the feature space, and generates temporal feature fluctuation loss when the fluctuation amplitude is lower than a set constant threshold.

[0044] The joint supervised optimization module uses pre-defined hyperparameter weighting factors to weight and sum the basic semantic generation loss, spatial attention loss, and temporal feature fluctuation loss to construct an overall objective optimization function. Based on this overall objective optimization function, the joint supervised optimization module performs backpropagation to update the model parameters.

[0045] During the inference phase, the semantic retrieval module invokes the updated retrieval system model. The semantic retrieval module receives natural language query instructions and calculates the similarity between the semantic vector of the instruction text and the visual relationship feature vectors extracted from the video library. Based on the similarity calculation results, the semantic retrieval module sorts the video segments and outputs the traffic escape video retrieval results.

[0046] See attached document Figure 2 This invention provides a semantic retrieval method for traffic escape behavior based on a dedicated large visual model, comprising the following steps:

[0047] S10 receives the original traffic video sequence and structured labeled data, converts the structured labeled data into a natural language text sequence, and extracts the spatial dimension mask matrix and the temporal dimension key frame interval index set.

[0048] S20 converts the original traffic video sequence into a visual word sequence and maps the natural language text sequence into a text embedding vector. It uses a cross-attention mechanism for fusion calculation and extracts the cross-modal cross-attention matrix and deep visual feature vector sequence when the logical chain integrity status flag is activated.

[0049] S30: Construct a three-dimensional joint loss function based on the truncated cross-modal cross-attention matrix and deep visual feature vector sequence, calculate the overall objective optimization function and perform backpropagation to update the model parameters;

[0050] S40 receives natural language query instructions, calculates the similarity between the text semantic vector of the instruction text and the visual relationship feature vector extracted from the video library, sorts the video segments, and outputs the search results.

[0051] See attached document Figure 2The multimodal data parsing module transforms raw visual and textual data into a standard feature format that can be processed by downstream models by performing a series of data preprocessing and feature mapping steps. In this embodiment, the execution flow of this module includes the following steps:

[0052] In step S110, the multimodal data parsing module receives the original traffic video sequence and the corresponding structured labeled data, and converts the structured labeled data into a natural language text sequence based on a preset template. The original traffic video sequence contains complete video clips of traffic interaction scenes.

[0053] The structured labeled data is written in JSON format, which includes the subject relationships, spatial interaction distances, instantaneous physical features, and corresponding causal logic chains representing traffic conflicts. The subject relationships record the entity categories involved in traffic interactions and their topological associations; the spatial interaction distances record the relative position changes of different entities in the three-dimensional physical world; the instantaneous physical features record the velocity jumps and attitude deflections of entities during movement; and the causal logic chains record the temporal logical associations from the preceding interaction behavior to the subsequent response behavior.

[0054] The multimodal data parsing module parses the above fields, fills the extracted feature data into a preset prompt word template, and reconstructs a target text sequence that conforms to natural language grammar. To avoid computational anomalies in the preprocessing stage, if some non-core feature fields are detected as missing during parsing, the system uses preset semantically neutral placeholders to replace them, ensuring the structural stability of the generated text sequence. Redundancy removal: The specific algorithm implementations for structured data parsing and string concatenation to generate text sequences are well-known technologies in this field and will not be elaborated here.

[0055] In step S120, the multimodal data parsing module executes a logical integrity verification mechanism and controls the activation state of the status flag based on the verification result. The multimodal data parsing module reads the causal logic chain field from the structured labeled data and performs character-level matching with a preset semantic tag library.

[0056] In one implementation, the system sets the verification rule as follows: when the causal logic chain field simultaneously contains a preceding collision label representing physical contact or approach and a subsequent escape label representing subjective departure, the current input data is determined to have a complete causal logic chain from physical collision to subjective escape. Under the condition of a complete logic chain, the multimodal data parsing module activates its built-in state flag. This state flag is essentially a Boolean variable at the software level, serving as a global trigger signal for the subsequent cross-modal coding and fusion module to trigger the feature interception mechanism and for the joint supervised optimization module to initiate the calculation of the three-dimensional joint loss function. Under the condition of an incomplete logic chain, the state flag remains inactive to avoid irrelevant data interfering with the fine-tuning of specific model parameters.

[0057] In step S130, the multimodal data parsing module extracts spatial features from the structured labeled data and maps them into a mask matrix in a two-dimensional pixel space. The multimodal data parsing module also parses the spatial dimension field recording entity locations in the structured labeled data to obtain the bounding box coordinate information of entities participating in traffic interactions. The bounding box coordinate information is represented by vertex coordinate coefficient values ​​in the pixel coordinate system.

[0058] The multimodal data parsing module performs mask rendering on a two-dimensional all-zero matrix aligned with the pixel resolution of a single frame in the original traffic video sequence, based on the bounding box coordinate information. To prevent coordinate out-of-bounds errors that could cause matrix dimension anomalies, the system performs boundary limiting operations on the input vertex coordinates at this stage, confining them within the effective height and width dimensions of the image.

[0059] In practice, the matrix element values ​​corresponding to the regions inside the bounding box are set to a constant 1, while the matrix element values ​​corresponding to the regions outside the bounding box are kept as a constant 0, generating a binarized region mask matrix. This mask matrix is ​​used to identify the interactive pixel regions in the feature map space where the two parties involved in the incident made physical contact or approached each other in danger.

[0060] In step S140, the multimodal data parsing module extracts temporal features from the structured labeled data and generates a keyframe interval index set for locating physical abrupt changes. The multimodal data parsing module parses the temporal dimension field recording the time span in the structured labeled data to identify the start and end points of the instantaneous physical abrupt change in the traffic conflict.

[0061] The multimodal data parsing module combines the frame rate parameters of the original traffic video sequence to convert the start and end timestamps into absolute frame index numbers within the video sequence. During this conversion, a floor operation is used to process floating-point timestamps, and overflow protection logic for the total number of video frames is added. These are ultimately recorded as the start frame index and the end frame index, respectively. The multimodal data parsing module constructs a keyframe interval index set from the start frame index to the end frame index using consecutive frame numbers. This keyframe interval index set covers the instantaneous action time period from the collision to the moment the offending entity exhibits escape behavior, providing corresponding time axis positioning coordinates for subsequent extraction of high-frequency physical transition signals in the deep feature space.

[0062] See attached document Figure 3 The cross-modal coding and fusion module is mainly used to establish a deep representation mapping between low-level visual physical features and high-level textual semantics, and to execute a dynamic feature interception mechanism under specific logical judgment conditions to support subsequent fine-tuning of specific physical features. In this embodiment, the specific workflow of this module includes the following steps:

[0063] In step S210, the cross-modal coding and fusion module receives the original traffic video sequence and the natural language text sequence, and performs dual-stream independent coding of multimodal features. In the visual branch, the system employs a visual Transformer architecture with multi-layer self-attention mechanisms as the visual encoder. The original traffic video sequence is segmented into multiple 3D video blocks according to a fixed time step and spatial resolution, and after being superimposed with spatiotemporal position encoding, it is input into the visual encoder. After multi-layer linear projection and self-attention calculation, a visual word sequence with a unified feature dimension is extracted at the output of the visual encoder.

[0064] The system uses a pre-trained language model as a text encoder in the text branch, and performs word segmentation on the natural language text sequence output by the multimodal data parsing module, mapping it into text embedding vectors that contain syntactic and semantic logic.

[0065] Building upon this, the system uses a linear projection layer to uniformly map visual word sequences and text embedding vectors to the same hidden feature dimension, thus providing a dimension-aligned data foundation for subsequent cross-modal interactions. The visual word sequences contain the appearance and movement trajectory features of traffic participants, while the text embedding vectors contain interaction logic and action description information.

[0066] In step S220, the cross-modal coding and fusion module utilizes a multimodal fusion layer to perform cross-modal forward propagation fusion computation on the visual word sequence and text embedding vector. The multimodal fusion layer internally contains a multi-head cross-attention network structure. During the computation, the text embedding vector serves as the query feature for the cross-attention mechanism, while the visual word sequence serves as both the key and value features.

[0067] By calculating the inner product similarity between text query features and visual key features, and dividing by a dimensionality scaling factor to prevent gradient overflow, followed by normalization, the system can assign dynamic weights related to the current text semantics to each visual term. This computational mechanism establishes an alignment between visual motion vectors and text semantic instructions, enabling the model to focus its receptive field on physically relevant pixel regions in the video frame when understanding hit-and-run related text logic.

[0068] In step S230, during the forward propagation of the multimodal fusion computation, the cross-modal coding and fusion module monitors the status flag passed from the multimodal data parsing module in real time. When the status flag is detected to be active, the cross-modal coding and fusion module triggers a feature interception mechanism in the spatial dimension.

[0069] Since the activation of the state flag indicates the existence of a complete collision-to-escape physical logic chain in the currently processed data sample, the system directly extracts and copies the cross-modal attention matrix AAA from the last network branch of the current multimodal fusion layer. The row dimension of this cross-modal attention matrix AAA corresponds to the text sequence length, and the column dimension corresponds to the number of visual lexical units. The numerical value of each element in the matrix directly represents the attention weight allocation of the model to a specific spatial region of the image when generating the current logical lexical unit. Extracting this matrix aims to store it in GPU memory, providing spatial feature representations for subsequent guidance of the model's receptive field focusing on the interaction region.

[0070] In step S240, within the same computation cycle when the status flag is activated, the cross-modal coding and fusion module synchronously triggers a deep feature interception mechanism in the temporal dimension. The cross-modal coding and fusion module obtains the keyframe interval index set pre-extracted by the multimodal data parsing module. Combining the time range specified by the keyframe interval index set, the cross-modal coding and fusion module directly addresses to the output of the projection layer of the visual network and extracts the continuous image deep visual feature vector sequence corresponding to the time range.

[0071] The extracted deep visual feature vector sequence is defined as .in, and These represent the starting frame index and the ending frame index contained in the keyframe interval index set, respectively. Representing the The deep visual feature vector output after the frame image is encoded and dimensionality reduced.

[0072] Considering that time series fluctuation calculation requires a second-order difference operation, the system introduces sequence length verification logic here: when - When the length is less than 2, the system performs equal-width padding expansion along both sides of the time axis, setting the length of the extracted continuous frame sequence to be no less than 3 frames, thereby avoiding abnormal program interruptions caused by denominators of 0 or out-of-bounds errors in subsequent calculations. Extracting this deep visual feature vector sequence and independently truncating it can provide raw temporal feature support without pooling smoothing for capturing collision energy fluctuations.

[0073] Referring to Figure 4, the joint supervised optimization module establishes a supervised alignment mapping between the underlying microscopic physical features and the upper-level macroscopic causal logic by constructing and calculating a three-dimensional joint loss function, thereby enabling specific fine-tuning of model parameters. In this embodiment, the specific workflow of this module includes the following steps:

[0074] In step S310, the joint supervised optimization module calculates the basic semantic generation loss based on the natural language real label sequence and the model prediction output. In the autoregressive stage of text generation, the system uses the standard autoregressive cross-entropy loss function to quantify the conditional probability error of correctly predicting the next target word under the given visual features and preceding text words. To simplify the explanation and broaden the scope of protection, this basic semantic generation loss is denoted as... The basic semantic generation loss is used to ensure that the model has basic language organization capabilities, enabling it to output text descriptions that conform to language rules and basic traffic logic.

[0075] In step S320, the joint supervised optimization module obtains the cross-modal cross-attention matrix extracted by the cross-modal encoding and fusion module, and calculates the spatial attention loss by combining it with the mask matrix generated by the multimodal data parsing module. When processing traffic accident scenarios, the model needs to focus on the physical regions where spatial interactions occur. The joint supervised optimization module calculates the overlap energy ratio between the cross-modal cross-attention matrix and the mask matrix in the feature map space. Spatial attention loss. The calculation formula is as follows:

[0076] ;

[0077] In the formula, and These represent the width and height dimensions of the cross-modal attention matrix after it has been restored using a two-dimensional grid. Represents the cross-modal attention matrix in pixel coordinates The attention weight value at the location; This indicates the mask matrix at the corresponding pixel coordinates. The value of the binarized mask at the location; This represents the smoothing constant.

[0078] In specific engineering deployments, to avoid gradient anomalies caused by a denominator of 0 due to a lack of attention distribution in the background region, the system adds a preset smoothing constant to the denominator term of the division operation. Smoothing is performed. This formula imposes a physical constraint on the spatial location of the cross-attention layer by calculating the energy proportion of the attention weights within the effective mask region of the interactive entity and penalizing the degree to which they deviate from the ideal state.

[0079] In step S330, the joint supervised optimization module receives the extracted deep visual feature vector sequence and calculates the temporal feature fluctuation loss. Traffic collisions are typically accompanied by instantaneous high-frequency vibrations or abnormal posture changes in vehicles. The joint supervised optimization module utilizes the second-order discrete difference between feature vectors of adjacent frames in the sequence to quantify the amplitude of energy magnitude fluctuations in the deep feature space over time. Temporal feature fluctuation loss. The calculation formula is as follows:

[0080] ;

[0081] In the formula, and These represent the start and end frame indices in the keyframe interval index set, respectively. , , These correspond to the deep visual feature vectors of three consecutive adjacent frames on the time axis. This represents the squared L2 norm of the second-order difference vector, used to calibrate the energy modulus of characteristic transitions; The threshold of the mutation energy constant is statistically calibrated by the system based on historical accident sample data. It represents the lower limit of the minimum visual feature fluctuation energy required for an effective physical collision.

[0082] The judgment logic of this monitoring mechanism is based on physical correlation: when the second-order fluctuation amplitude of the feature vector in time series is lower than the set threshold of the mutation energy constant. When, maximize function The positive difference in the output will generate a penalty gradient to guide the model to learn high-frequency jump features; conversely, if the fluctuation amplitude reaches or exceeds the threshold, no penalty will be generated.

[0083] In step S340, the joint supervised optimization module uses a preset hyperparameter weighting factor to weight and sum the basic semantic generation loss, spatial attention loss, and temporal feature fluctuation loss to construct the overall objective optimization function. Specifically, the system sets the hyperparameter weighting factor to include the basic semantic weight. Spatial supervision weight and time-series supervision weights The overall objective function is calculated using a linear combination method. .

[0084] In this embodiment, , , The values ​​are all set to range from 0.1 to 1.0. The joint supervised optimization module performs automatic differentiation based on the overall objective optimization function, calculates the partial derivatives of the parameters of each network layer, and uses conventional optimization algorithms such as adaptive moment estimation to perform backpropagation to update the model parameters. Through iterative fine-tuning with batch sample data, the model gains the ability to perceive microscopic spatial interaction constraints and instantaneous temporal energy fluctuations.

[0085] See attached document Figure 2 The semantic retrieval module is used to deploy a jointly supervised optimized retrieval system model in real-world business scenarios, retrieving target video segments by parsing natural language query commands. In this embodiment, the specific workflow of this module includes the following steps:

[0086] In step S410, the semantic retrieval module receives a natural language query instruction input from the business terminal. The natural language query instruction contains an unstructured description representing three complete semantic logics: subject, location, and action. The semantic retrieval module invokes the updated text encoder network to perform word segmentation, word vector embedding, and deep self-attention encoding calculations on the query instruction, extracting the text semantic vector of the natural language query instruction at its output. This text semantic vector, in the feature space, represents the retrieval intent regarding the perpetrator, the spatial location of the collision, and the subsequent escape motive.

[0087] In step S420, during the processing of the traffic video to be inspected, the semantic retrieval module performs multi-scale segmentation of the traffic video through a dynamic sampling mechanism. The system uses the inter-frame difference method to calculate the pixel change rate of continuous images. When the change rate exceeds a set motion threshold, the core time window of the traffic conflict is located. Subsequently, the system performs non-uniform frame extraction within this core time window to extract key frame sequences that possess spatial interaction features such as object approach and physical features such as instantaneous vibration.

[0088] In step S430, the semantic retrieval module inputs the acquired keyframe sequence into the visual encoder, transforming it into a structured lexical matrix. To ensure that the model can recognize spatial distances within a single frame under a unified attention mechanism, the system assigns an independent temporal position encoding to each group of lexical units in the lexical matrix. This temporal position encoding is used to connect temporal behavior with causal motivation, enabling the model to output a visual relationship feature vector that effectively represents the underlying behavior of the video.

[0089] In step S440, the semantic retrieval module receives the text semantic vector and the visual relation feature vector, and performs high-dimensional similarity matching across the modal feature space. The system uses a cosine similarity algorithm to quantify the semantic correlation between the two. Specifically, by calculating the inner product between the text semantic vector and the visual relation feature vector and dividing it by the product of their L2 norm moduli, a similarity value representing the consistency of vector direction is obtained.

[0090] To prevent program errors caused by the feature vector magnitude approaching zero in specific boundary scenarios such as a completely static screen during division operations, the system adds a preset smoothing constant during denominator calculation. Smoothing is performed. This mechanism determines whether the video contains a subject, location, and complete action feature structure that matches the definition in the training set. It mainly relies on deep semantic alignment mapping to replace traditional single physical attribute recognition.

[0091] In step S450, the semantic retrieval module sorts the video segments based on the similarity calculation results and outputs the traffic escape video retrieval results. The system sorts the associated traffic video segments in descending order based on the cosine similarity value. To filter out background noise and low-relevance content, the system has a built-in similarity filtering threshold. The value of this similarity filtering threshold is set between 0.65 and 0.85, and the specific value is dynamically calibrated according to the accuracy requirements of the actual monitoring scenario. The system automatically removes video segments with similarity below the similarity filtering threshold and encapsulates the retained high-confidence candidate sequences into a result list for return.

[0092] Specific application examples:

[0093] Background setting:

[0094] In this embodiment, the application scenario is set as traffic surveillance video retrieval at a city intersection. The target query event is: a white sedan collided with an electric vehicle at the intersection and then accelerated away. The frame rate of the surveillance video is 30 frames / s, and the image resolution is 1920×1080. Structured labeled data recorded this interaction behavior: the physical collision occurred at 2.0s, and the offending vehicle showed obvious acceleration and evasive maneuvers at 4.0s. The similarity filtering threshold of the retrieval system is set to 0.75.

[0095] Implementation steps and numerical calculations:

[0096] S10, the multimodal data parsing module receives data and performs preprocessing. Based on the frame rate conversion, the starting frame index is determined. =2.0 × 30 = 60, Termination Frame Index =4.0 × 30 = 120. Extract the entity bounding box coordinates of this interval. On the 1920 × 1080 zero matrix, set the element values ​​of the overlapping area where the car and electric vehicle physically contacted each other to 1, and the rest to 0, to generate a binary mask matrix. .

[0097] S20, the cross-modal encoding and fusion module performs feature extraction. During the forward propagation of the multimodal fusion layer, a 16×16 cross-modal cross-attention matrix is ​​extracted. Simultaneously extract deep visual feature vector sequences from frames 60 to 120. .

[0098] S30, the joint supervised optimization module performs loss function calculation and parameter update.

[0099] (1) Spatial attention loss calculation: According to the formula disclosed in the specific implementation method:

[0100] ;

[0101] Assuming that the attention weights are in the binarized mask matrix after matrix dot product, Energy summation within the effective region The sum of global attention weights =1.0. A preset smoothing constant is introduced. 10 −5 Substituting into the formula, the spatial attention loss is calculated as follows:

[0102] ;

[0103] (2) Calculation of time series characteristic fluctuation loss: According to the formula disclosed in the specific implementation method:

[0104] ;

[0105] Set a threshold for the mutation energy constant =0.5. Within the interval from frame 60 to frame 120, calculate the squared second-order discrete difference norm of the visual feature vectors of three consecutive adjacent frames. During frames 60-65, when the collision occurs, the feature energy magnitude reaches 0.8, which is considered a valid transition, and the function output is maximized. (0, 0.5 − 0.8) = 0; During the smooth driving phase, the feature energy modulus is only 0.1, and the output penalty value is... (0, 0.5 − 0.1) = 0.4. Assuming the cumulative penalty on the time axis is 7.2, substituting this into the formula yields:

[0106] ;

[0107] (3) Construction of the overall objective function: According to the formula disclosed in the specific implementation method:

[0108] ;

[0109] Set basic semantic weights =0.5, Spatial Supervision Weight =0.3, Time-series supervision weight =0.2. The known basic semantic generation loss is... =0.25, then the total loss is:

[0110] =0.5×0.25+0.3×0.18+0.2×0.12=0.125+0.054+0.024=0.203;

[0111] The system performs backpropagation based on the loss value of 0.203 to fine-tune the model parameters.

[0112] S40, the semantic retrieval module performs matching. The user inputs that a white car hit an electric scooter and drove away. The system extracts the text semantic vector and the visual relationship feature vector extracted from the aforementioned video. The inner product of the two is calculated to be 0.88, and both are normalized to 1.0 using their L2 norm. A preset smoothing constant is then added. The cosine similarity was then calculated as follows:

[0113] ;

[0114] Since the calculated similarity of 0.88 is greater than the set similarity filtering threshold of 0.75, the system determines that the video clip matches the search intent, recalls it, and outputs it.

[0115] Experimental verification and effect comparison:

[0116] To verify the practical effectiveness of the technical solution of this invention, a test dataset containing 1000 traffic monitoring videos was constructed, including 300 real positive samples of traffic escape and 700 negative samples containing easily confused behaviors such as normal driving and regular parking. The method of this invention was compared with traditional comparison methods, and the evaluation metrics were retrieval accuracy and retrieval recall.

[0117] See attached document Figure 5 Combined with the appendix Figure 5 As shown in the data, the grouped bar chart on the left illustrates the difference in retrieval accuracy between the two methods, while the grouped bar chart on the right illustrates the difference in retrieval recall. In the figure, the dark gray bars represent the method of this invention, and the light gray bars represent the traditional comparison method.

[0118] The traditional comparison method achieved a retrieval precision of 76.0% and a retrieval recall of 68.5%, as shown in the attached figure. Figure 5 The values ​​are represented by the light gray bars. The errors often occur when a vehicle safely passes by at close range but is mistakenly identified as a hit-and-run, due to a lack of constraints based on microscopic physical characteristics. The method of this invention improves the retrieval accuracy to 92.5% and the retrieval recall rate to 89.0%, as shown in the attached figure. Figure 5 The values ​​are shown in the medium-dark gray bars.

[0119] Comparison conclusion:

[0120] Combining the above numerical calculations and appendices Figure 5 A direct comparison of the data shows that the solution of this invention, by introducing a formula... The spatial attention constraints and formulas represented The quantification of temporal characteristic fluctuations represents a way to compensate for the shortcomings of traditional large models in perceiving microscopic physical abrupt changes. The three-dimensional joint supervision mechanism reduces false negative samples caused by visual perspective illusions, improves the accuracy and recall of semantic retrieval of traffic escape behavior, and has practical engineering application value.

Claims

1. A semantic retrieval system for traffic escape behavior based on a dedicated large-scale visual model, characterized in that, It includes a multimodal data parsing module, which receives raw traffic video sequences and structured labeled data, converts the structured labeled data into natural language text sequences, and activates a status bit flag when the structured labeled data is detected to have a causal logical chain integrity. The cross-modal coding and fusion module is used to convert the original traffic video sequence into a visual word sequence and map the natural language text sequence into a text embedding vector. It uses a cross-attention mechanism for fusion calculation and extracts the cross-modal cross-attention matrix and deep visual feature vector sequence when the activation of the state bit flag is detected. The joint supervised optimization module is used to construct a three-dimensional joint loss function based on the truncated cross-modal attention matrix and the deep visual feature vector sequence, calculate the overall objective optimization function, and perform backpropagation to update the model parameters; The semantic retrieval module is used to calculate the similarity between the text semantic vector of the natural language query command and the extracted visual relationship feature vectors pre-stored in the video library, and output the traffic escape video retrieval results.

2. The semantic retrieval system for traffic escape behavior based on a dedicated large-scale visual model according to claim 1, characterized in that, The multimodal data parsing module parses the structured labeled data to obtain the subject relationships, spatial interaction distances, physical features and causal logic chains that characterize traffic conflicts, and fills them into a preset prompt word template to reconstruct the target text sequence as the natural language text sequence. The multimodal data parsing module reads the causal logic chain field in the structured labeled data and performs character-level matching with the preset semantic tag library. When the causal logic chain field contains both a pre-collision tag representing physical contact and a post-escape tag representing subjective departure, it is determined that there is a complete causal logic chain from physical collision to subjective escape.

3. The semantic retrieval system for traffic escape behavior based on a dedicated large visual model according to claim 1, characterized in that, The cross-modal encoding and fusion module maps the visual word sequence and the text embedding vector to the same hidden feature dimension through a linear projection layer. The cross-modal encoding and fusion module uses the text embedding vector as the query feature of the cross-attention mechanism, and the visual word sequence as the key feature and value feature. It calculates the inner product similarity between the query feature and the key feature, divides it by the dimension scaling factor, and then completes the cross-modal fusion calculation through a normalization function.

4. The semantic retrieval system for traffic escape behavior based on a dedicated large visual model according to claim 3, characterized in that, The multimodal data parsing module obtains the bounding box coordinate information of the entities participating in traffic interaction. On a two-dimensional all-zero matrix aligned with the resolution pixels of a single frame image of the original traffic video sequence, the matrix element values ​​corresponding to the inner region of the bounding box are set to constant 1, and the matrix element values ​​corresponding to the outer region of the bounding box are set to constant 0 to generate a mask matrix. The multimodal data parsing module identifies the start and end points of the instantaneous physical change in traffic conflict, and converts them into start and end frame indices by combining the frame rate parameters of the original traffic video sequence. The consecutive frame numbers between the start and end frame indices are then used to construct a key frame interval index set.

5. The traffic escape behavior semantic retrieval system based on a dedicated visual large model according to claim 4, characterized in that, When the status flag is activated, the cross-modal coding and fusion module extracts the cross-modal cross-attention matrix from the last network branch of the multimodal fusion layer, where the row dimension corresponds to the length of the text sequence and the column dimension corresponds to the number of visual lexical units. The cross-modal coding and fusion module obtains the key frame interval index set, addresses the output of the projection layer of the visual network, and extracts the deep visual feature vector sequence within the corresponding time range. When the difference between the termination frame index and the starting frame index is less than 2, it performs equal-width padding expansion along both sides of the time axis to ensure that the length of the extracted continuous frame sequence is not less than 3 frames.

6. The semantic retrieval system for traffic escape behavior based on a dedicated large visual model according to claim 5, characterized in that, The three-dimensional joint loss function includes basic semantic generation loss, spatial attention loss, and temporal feature fluctuation loss; The joint supervised optimization module uses an autoregressive cross-entropy loss function to quantify the conditional probability error of correctly predicting the next target word under the condition of current visual features and preceding text words, and obtains the basic semantic generation loss. The joint supervised optimization module calculates the overlap energy ratio between the cross-modal cross-attention matrix and the mask matrix in the feature map space, and adds a preset smoothing constant to the denominator of the division operation for smoothing to obtain the spatial attention loss.

7. The semantic retrieval system for traffic escape behavior based on a dedicated large visual model according to claim 6, characterized in that, The joint supervised optimization module calculates the squared second-order discrete difference L2 norm between adjacent frame feature vectors in the deep visual feature vector sequence, quantizes it to obtain the energy magnitude fluctuation amplitude, and when the energy magnitude fluctuation amplitude is lower than the set abrupt change energy constant threshold, the positive difference is output through the maximization function to generate a penalty gradient, thus obtaining the temporal feature fluctuation loss. The joint supervised optimization module uses basic semantic weights, spatial supervision weights, and temporal supervision weights to linearly combine the basic semantic generation loss, the spatial attention loss, and the temporal feature fluctuation loss to obtain the overall objective optimization function.

8. The semantic retrieval system for traffic escape behavior based on a dedicated large visual model according to claim 1, characterized in that, The semantic retrieval module uses the inter-frame difference method to calculate the pixel change rate of continuous images in the traffic videos to be inspected in the video library. When the pixel change rate exceeds the set motion threshold, the core time window of the traffic conflict is located, and non-uniform frame extraction is performed to extract the key frame sequence. The semantic retrieval module converts the keyframe sequence into a lexical matrix and assigns it a time position encoding, then outputs the visual relationship feature vector.

9. The semantic retrieval system for traffic escape behavior based on a dedicated large visual model according to claim 8, characterized in that, The semantic retrieval module extracts the text semantic vector of the natural language query instruction, calculates the inner product between the text semantic vector and the visual relationship feature vector, and divides it by the product of the moduli of their two norms. A preset smoothing constant is added during the denominator calculation to obtain the similarity value. The associated traffic video segments are sorted in descending order according to the similarity value, and video segments with similarity values ​​lower than the similarity filtering threshold are removed to output the traffic escape video retrieval result.

10. A semantic retrieval method for traffic escape behavior based on a dedicated large-scale visual model, characterized in that, The semantic retrieval system for traffic escape behavior based on a dedicated large visual model, as described in any one of claims 1-9, includes the following steps: Receive raw traffic video sequences and structured labeled data, convert the structured labeled data into natural language text sequences, and activate the status bit flag when the structured labeled data is detected to have the integrity of a causal logical chain; The original traffic video sequence is converted into a visual word sequence and the natural language text sequence is mapped into a text embedding vector. A cross-attention mechanism is used for fusion calculation, and when the activation of the state bit flag is detected, the cross-modal cross-attention matrix and the deep visual feature vector sequence are extracted. A three-dimensional joint loss function is constructed based on the extracted cross-modal cross-attention matrix and the deep visual feature vector sequence. The overall objective optimization function is calculated and backpropagation is performed to update the model parameters. Calculate the similarity between the text semantic vector of the natural language query command and the extracted visual relationship feature vectors pre-stored in the video library, and output the traffic escape video retrieval results.

Images