A question and answer method and system for traffic events based on natural language

Abstract

The application discloses a kind of based on natural language's traffic event's question and answer method and system, belong to intelligent question and answer field, first acquisition roadside image and user query, respectively by visual encoder and natural language encoder extraction multiscale visual feature set and text feature;Then utilize pre-training cross-modal intent alignment and focus module, with text feature as guide to spatial attention anchor and cross-modal intermingling to visual feature, output joint representation feature;Again, the feature is input light-weight decoder obtained by structured logic decoupling and migration strategy distillation, so that it executes step-by-step logical inference and coordinate regression based on spatial response graph in parallel, synchronously output natural language answer text and target bounding box coordinates;Finally generate traffic event question and answer result containing semantic answer and visual mark.The application effectively solves the problem that prior art cannot accurately and efficiently based on natural language to traffic event intelligent question and answer.

Description

Technical Field

[0001] This invention relates to the field of intelligent question answering, and more particularly to a question-and-answer method and system for traffic events based on natural language. Background Technology

[0002] Currently, roadside perception technologies in intelligent transportation systems mainly focus on instance-level perception tasks, such as object detection, multi-object tracking, and trajectory prediction. Although these methods have achieved some success in structured environment perception, their output is usually limited to low-semantic information such as bounding boxes, trajectory points, and category labels. They lack a deep understanding of the semantics of the scene and the ability to interact with natural language, making it difficult for users to query and make decisions about traffic participants and their behaviors in a flexible and intuitive way.

[0003] In recent years, while visual question answering technology has demonstrated its potential for multimodal joint understanding, its application in traffic scenarios, especially from a roadside perspective, still faces significant limitations. On the one hand, existing traffic visual question answering datasets are mostly constructed based on in-vehicle perspectives, with questions designed around explicit object attributes, lacking modeling of high-level semantics such as behavioral intent and traffic rule compliance. On the other hand, existing methods typically treat object detection and visual question answering as two separate tasks, failing to explicitly indicate the specific location of the traffic targets in the image while outputting natural language answers, leading to ambiguity in scenes with dense multi-object traffic. These shortcomings prevent existing technologies from accurately and efficiently providing intelligent question answering for traffic events based on natural language. Summary of the Invention

[0004] This invention provides a question-and-answer method and system for traffic events based on natural language, in order to solve the problem that existing technologies cannot accurately and efficiently perform intelligent question-and-answer based on natural language for traffic events.

[0005] Firstly, this application provides a natural language-based question-and-answer method for traffic events, including: Acquire real-time traffic scene images collected by roadside sensing devices and natural language queries input by users; The real-time traffic scene image is input into a preset visual encoder for multi-scale feature extraction to obtain a visual feature set; The natural language query is input into a preset natural language encoder for semantic encoding to obtain text features; The visual feature set and the text features are input into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module uses the text features as query guides, performs spatial attention anchoring and cross-modal feature fusion on the visual feature set, and outputs joint representation features; wherein, the cross-modal intent alignment and focusing module is trained based on historical traffic scene images, historical natural language queries, and corresponding target positions marked in historical natural language queries; The joint representation features are input into a preset lightweight decoder, which performs step-by-step logical reasoning based on a structured logic decoupling and transfer strategy, and performs coordinate regression based on the spatial response map in the joint representation features to output the bounding box coordinates of the natural language response text and the traffic target; wherein, the lightweight decoder is trained by distillation of the inference logic of a preset multimodal model. Based on the natural language response text and the bounding box coordinates, traffic event question and answer results are generated.

[0006] This application first acquires real-time traffic scene images and user natural language queries collected by roadside perception devices, and extracts visual feature sets and text features through visual encoders and natural language encoders, respectively. Then, using a pre-trained cross-modal intent alignment and focusing module, guided by text features, spatial attention anchoring and cross-modal feature fusion are performed on the visual feature set, enabling the model to dynamically focus on visual regions strongly related to the user's query intent, effectively suppressing interference from complex background noise, thereby generating joint representation features containing semantic alignment information. Based on this, the joint representation features are input into a lightweight decoder obtained through structured logic decoupling and transfer strategy distillation, which allows it to inherit the complex reasoning capabilities of large models while maintaining low computational overhead. During the decoding process, coordinate regression is performed synchronously based on the spatial response map in the joint representation features, and finally, the bounding box coordinates of the natural language response text and traffic targets are output in parallel. Therefore, based on the realization of high-level semantic reasoning and natural language interaction, this application further endows the system with the visual positioning capability of "the answer is the reference", that is, while outputting semantic answers, it accurately points out the location of traffic targets involved in the image. This application effectively solves the problem that the existing technology cannot accurately and efficiently perform intelligent question answering on traffic events based on natural language.

[0007] Furthermore, the step of inputting the visual feature set and the text features into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module uses the text features as query guidance, performs spatial attention anchoring and cross-modal feature fusion on the visual feature set, and outputs joint representation features, specifically: The visual feature set and the text features are input into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module maps the text features as a query vector after mapping by a linear projection matrix, and maps the visual feature set into a key matrix and a value matrix respectively after mapping by a linear projection matrix; The query vector and the key matrix are subjected to linear attention calculation based on a kernel function to obtain the attention weight distribution; The value matrix is ​​weighted and aggregated according to the attention weight distribution to obtain weighted visual features; The weighted visual features and the text features are fused across modalities to obtain the joint representation features.

[0008] This application first maps text features to query vectors via linear projection, and maps visual feature sets to key and value matrices respectively, thus unifying natural language queries and visual features into the same semantic space. Then, it performs kernel-based linear attention computation on the query vector and key matrix, utilizing the associative law of matrix multiplication to reduce computational complexity from the traditional quadratic level to the linear level, enabling real-time processing of high-resolution traffic images. Simultaneously, it accurately selects visual features strongly correlated with query intent through attention weight distribution. Based on this, it weights and aggregates the value matrix according to this weight distribution to obtain weighted visual features, and then performs cross-modal fusion with the original text features to generate a joint representation. Therefore, this application achieves deep semantic alignment between text and vision while significantly reducing the computational overhead of cross-modal fusion, enabling the model to dynamically focus on the traffic target area indicated by the user query, effectively avoiding interference from irrelevant background features. This provides a high-quality feature foundation for subsequent accurate localization and semantic inference, thereby achieving efficient and accurate real-time interactive response on roadside devices with limited edge computing resources.

[0009] Furthermore, the step of performing linear attention calculation based on a kernel function on the query vector and the key matrix to obtain the attention weight distribution is specifically as follows: Based on the feature mapping kernel function, nonlinear feature mapping is performed on the query vector and the key matrix respectively to obtain the mapped query vector and the mapped key matrix; According to the associative law of matrix multiplication, the mapped key matrix and the value matrix are first multiplied together, and then the result is multiplied together with the mapped query vector to obtain the attention weight distribution.

[0010] This application first performs nonlinear feature mapping on the query vector and key matrix using a feature mapping kernel function, transforming the original features to a high-dimensional space to enhance their expressive power. Then, based on the associative law of matrix multiplication, it changes the order of matrix operations in traditional attention calculation: first, multiplying the mapped key matrix with the value matrix, and then multiplying the result with the mapped query vector, thus obtaining the attention weight distribution. This adjustment of the calculation order reduces the quadratic complexity of calculating pairwise similarity between queries and keys to linear complexity, avoiding the memory explosion and computational latency problems caused by a large number of visual tokens in high-resolution traffic images. Therefore, this application significantly improves the efficiency of cross-modal attention calculation while achieving accurate focusing of visual features for text guidance and ensuring a high correlation between attention weights and query intent. This enables the model to process high-definition traffic images captured by the roadside at real-time frame rates on edge computing devices, providing a timely and accurate foundation for subsequent semantic reasoning and target localization.

[0011] Furthermore, the step of weighting and aggregating the value matrix according to the attention weight distribution to obtain weighted visual features specifically involves: Based on the attention weight distribution, the value matrix is ​​weighted and summed to obtain preliminary weighted features; Based on the attention weight distribution, residual weighted aggregation is performed on the multi-level visual feature maps in the visual feature set so that the high-level global topological structure features and the low-level texture detail features can be complementaryly fused. The residual weighted aggregated features are fused with the preliminary weighted features, and then concatenated with the text features to obtain the weighted visual features.

[0012] This application first performs a weighted summation of the value matrix based on the attention weight distribution to obtain preliminary weighted features, enabling text queries to initially filter single-layer visual features. Simultaneously, based on the same attention weight distribution, residual weighted aggregation is performed on multi-level visual feature maps within the visual feature set. This allows high-level global topological structure features and low-level texture detail features to be complementary and fused according to the query intent, thus avoiding the problem of semantic and detail information separation in traditional single-layer feature fusion. Finally, the multi-scale features obtained from residual weighted aggregation are fused with the preliminary weighted features and then concatenated with the text features to obtain the final weighted visual features. Therefore, this application, while maintaining consistency in text query guidance, fully utilizes multi-level information from visual features, enabling the model to perceive both the overall layout and spatial relationships of the target, as well as capture subtle attributes such as edges and textures. This significantly improves the richness and discriminative power of cross-modal feature expression, providing a solid feature foundation for accurate target localization and fine-grained semantic reasoning in complex traffic scenarios.

[0013] Furthermore, the step of inputting the joint representation features into a preset lightweight decoder, enabling the lightweight decoder to perform step-by-step logical reasoning based on a structured logic decoupling and transfer strategy, and to perform coordinate regression based on the spatial response map in the joint representation features, outputting the bounding box coordinates of the natural language response text and the traffic target, specifically: The joint representation features are input into a preset lightweight decoder, so that the lightweight decoder performs dual-stream parallel processing on the joint representation features and inputs them into the built-in text generation stream and positioning stream respectively; In the text generation stream, the joint representation features are subjected to step-by-step logical reasoning based on the structured logic decoupling and transfer strategy, and the natural language response text is generated by autoregression. In the localization stream, the spatial response map is extracted from the joint representation features, and coordinate regression is performed on the spatial response map through the location prediction branch to output the bounding box coordinates; The outputs of the text generation stream and the positioning stream are time-aligned, and the bounding box coordinates are output simultaneously while generating the natural language response text.

[0014] This application first inputs the joint representation features into a lightweight decoder and processes them in parallel with two streams, feeding them into a text generation stream and a localization stream, respectively. In the text generation stream, step-by-step logical reasoning is performed on the joint representation features based on structured logic decoupling and transfer strategies. By decoupling the complex reasoning task into sub-tasks such as target localization, attribute determination, and causal inference and decoding them sequentially, the lightweight model can inherit the reasoning capabilities of the large model and stably generate answers that conform to the semantics of traffic scenarios. At the same time, the spatial response map in the localization stream is directly extracted from the joint representation features, and coordinate regression is performed on the response map through the location prediction branch to output the bounding box coordinates of the traffic target in real time. Finally, the outputs of the two streams are time-aligned to achieve simultaneous output of bounding box coordinates while generating natural language response text. Therefore, this application, while ensuring that the lightweight model has complex reasoning capabilities, has for the first time achieved end-to-end collaborative output of semantic question answering and visual localization within the same framework, enabling the system to have the ability of "what is answered is what is referred to". In multi-object dense traffic scenarios, it accurately eliminates referential ambiguity, significantly improves the naturalness of human-computer interaction and decision interpretability, and the dual-stream parallel processing mechanism avoids the latency caused by serial computing, providing efficient and reliable technical support for roadside real-time interactive applications.

[0015] Furthermore, in the text generation stream, based on the structured logic decoupling and transfer strategy, step-by-step logical reasoning is performed on the joint representation features to generate the natural language response text through autoregression, specifically as follows: Obtain the reasoning logic parameters obtained through distillation of a preset teacher model; the reasoning logic parameters are used to simulate the step-by-step perception and reasoning process of the teacher model; Based on the inference logic parameters, the joint representation features are decoded step by step in the text generation stream. In each decoding step, the target localization subtask, attribute determination subtask, and causal inference subtask are executed sequentially, and the intermediate text of the corresponding subtask is generated. The intermediate texts from each subtask are concatenated in chronological order, and the natural language response text is generated through autoregression.

[0016] This application first obtains the inference logic parameters obtained through teacher model distillation. These parameters fully simulate the step-by-step perception and inference process of the teacher model, thereby transferring the complex logical capabilities of the large model to the lightweight decoder in the form of parameters. Based on this, the joint representation features are decoded step-by-step in the text generation stream based on these inference logic parameters. In each decoding step, the target localization subtask, attribute determination subtask, and causal inference subtask are executed sequentially, and the intermediate text of the corresponding subtask is generated. This decomposes the complex traffic scenario problem, which was originally difficult to solve directly, into multiple interpretable and traceable logical steps. Finally, the intermediate texts of each subtask are concatenated in chronological order, and a complete natural language answer text is generated autoregressively. Thus, this application enables the lightweight decoder to stably possess structured multi-hop inference capabilities without relying on a large-scale pre-trained model. It can generate answers according to the cognitive logic of "first locate the target, then determine the attribute, and finally infer the causality," effectively avoiding the "illusionary" answers that are prone to occur in complex compliance reasoning in traditional end-to-end models. This significantly improves the accuracy, logical rigor, and interpretability of semantic question answering in traffic scenarios, while reducing the computational cost and inference latency of edge deployment.

[0017] Furthermore, the acquisition of the reasoning logic parameters obtained through the pre-set teacher model distillation specifically involves: Based on a pre-set multimodal teacher model, combined with historical traffic scene images, historical natural language queries, and the corresponding target locations marked in the historical natural language queries, a perception and reasoning process is generated. The perception and reasoning process is spliced ​​with the historical traffic scene image, historical natural language query, and the corresponding target location marked in the historical natural language query to generate a training signal; Based on the training signal, knowledge distillation is performed on the lightweight decoder, and a multi-task joint loss function is used for optimization to obtain the inference logic parameters. The multi-task joint loss function includes target localization loss, inference chain consistency loss, and question-answer cross-entropy loss; the inference chain consistency loss is used to constrain the logical distribution difference between the lightweight decoder and the teacher model.

[0018] This application first utilizes a pre-defined multimodal teacher model, combined with historical traffic scene images, natural language queries, and the marked target locations, to generate a perception and reasoning process containing a complete cognitive process. Then, this reasoning process is concatenated with the original historical data to construct a training signal, ensuring that the training samples not only contain the input-output mapping relationship but also the intermediate logical steps from perception to reasoning. Based on this enhanced signal, knowledge distillation is performed on the lightweight decoder, and a multi-task joint loss function is used for optimization. The reasoning chain consistency loss is specifically used to constrain the differences in logical distribution between the lightweight decoder and the teacher model, forcing the lightweight decoder to mimic the step-by-step reasoning logic of the teacher model rather than simply fitting the final answer. Therefore, this application efficiently transfers the structured reasoning capabilities of large models to lightweight decoders in a parameterized form, enabling edge models with fewer than one billion parameters to stably output multi-hop, interpretable logical reasoning chains. This completely overcomes the pain point of performance degradation of lightweight models when handling complex cognitive tasks, while avoiding the shortcomings of traditional supervised fine-tuning that merely imitates the appearance of answers without grasping the true reasoning logic. This provides a practical and feasible technical path for roadside edge devices to achieve high accuracy and high interpretability in traffic scenario question answering.

[0019] Secondly, this application provides a natural language-based question-and-answer system for traffic events. The natural language-based question-and-answer system for traffic events includes: The acquisition module is used to acquire real-time traffic scene images collected by roadside sensing devices and natural language queries input by users. The extraction module is used to input the real-time traffic scene image into a preset visual encoder for multi-scale feature extraction to obtain a visual feature set; The encoding module is used to input the natural language query into a preset natural language encoder for semantic encoding to obtain text features; The fusion module is used to input the visual feature set and the text features into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module uses the text features as query guidance, performs spatial attention anchoring and cross-modal feature fusion on the visual feature set, and outputs joint representation features; wherein, the cross-modal intent alignment and focusing module is trained based on historical traffic scene images, historical natural language queries, and corresponding target positions marked in historical natural language queries; The decoding module is used to input the joint representation features into a preset lightweight decoder, so that the lightweight decoder performs step-by-step logical reasoning based on the structured logic decoupling and transfer strategy, and performs coordinate regression based on the spatial response map in the joint representation features, and outputs the bounding box coordinates of the natural language response text and the traffic target; wherein, the lightweight decoder is trained by distillation of the inference logic of a preset multimodal model; The generation module is used to generate traffic event question-and-answer results based on the natural language response text and the bounding box coordinates.

[0020] The acquisition module of this application first collects real-time traffic scene images from roadside perception devices and user natural language queries. The extraction and encoding modules transform the original heterogeneous data into visual feature sets and text features through visual encoders and natural language encoders, respectively, laying the foundation for subsequent cross-modal fusion. The fusion module utilizes a pre-trained cross-modal intent alignment and focusing module, guided by text features, to perform spatial attention anchoring and cross-modal feature fusion on the visual feature set, outputting joint representation features containing semantic alignment relationships, thereby dynamically focusing on the traffic target area pointed to by the user query and effectively suppressing background noise interference. The decoding module inputs the joint representation features into a lightweight decoder obtained through structured logic decoupling and transfer strategy distillation, which inherits the complex reasoning capabilities of large models while maintaining low computational overhead, and performs coordinate regression simultaneously based on the spatial response map in the joint representation features, outputting natural language answer text and bounding box coordinates in parallel. Finally, the generation module integrates the two into a complete traffic event question-and-answer result. Thus, the modules work together to achieve end-to-end closed-loop processing from multi-source perception, intent alignment, lightweight reasoning to "answer is the referent" visual positioning. This enables the system to output high-level reasoning answers that conform to the semantics of traffic scenarios on roadside devices with limited edge computing resources, and to simultaneously and accurately mark the locations of the targets involved. This completely eliminates the ambiguity of reference in multi-target dense scenarios and significantly improves the naturalness, interpretability and real-time response capability of intelligent transportation human-computer interaction. Attached Figure Description

[0021] Figure 1 : A schematic flowchart of an embodiment of the natural language-based question-and-answer method for traffic events provided in this application; Figure 2 : A schematic diagram of an embodiment comparing the prediction results provided in this application with the actual annotations; Figure 3 : A schematic diagram of an embodiment of the traffic incident question-and-answer process provided in this application; Figure 4 : A schematic diagram of an embodiment of the natural language-based traffic event question-and-answer system provided in this application. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 Please refer to Figure 1 In order to solve the problem that existing technologies cannot accurately and efficiently perform intelligent question answering on traffic events based on natural language, this invention provides a question answering method for traffic events based on natural language, including steps S01-S06.

[0024] S01: Acquire real-time traffic scene images collected by roadside sensing devices and natural language queries input by users; In a preferred embodiment of this invention, the acquisition of real-time traffic scene images collected by the roadside sensing device and natural language queries input by the user specifically includes: In one specific implementation, the data acquisition step is performed first. Specifically, a roadside sensing device is pre-deployed at typical urban intersections or along highways. This device includes at least one high-definition RGB camera with a resolution of no less than 1920×1080 pixels and a frame rate of 25 frames per second. It is mounted on a streetlight pole or a dedicated monitoring pole, covering the entire intersection or road segment. The camera captures a real-time video stream of the traffic scene. The system performs frame extraction processing on the video stream as needed, for example, extracting one frame every second as the real-time traffic scene image for the current moment. Simultaneously, the system receives natural language queries input by the user through an interactive interface. This interface can be a web client on the traffic management center operator's workstation, a mobile application, or a smart terminal that supports voice input. When the user inputs a query in text form, the system directly obtains the text string; when the user inputs in voice form, the system's front-end integrated speech recognition module converts the speech into a text string in real time before sending it to subsequent processing. To support multi-turn dialogue scenarios, the system also maintains a memory cache to store the current user's historical query records and corresponding interaction contexts. The acquired real-time traffic scene images and natural language query text together constitute the raw input data for the subsequent multimodal understanding module. At the same time, a unified timestamp is assigned to each set of inputs to ensure the temporal consistency in subsequent processing.

[0025] In practical deployment, the aforementioned roadside sensing device also includes an edge computing device, such as an industrial computer equipped with an NVIDIA Jetson AGX Orin module. This device is built into the roadside chassis and runs the multimodal understanding and reasoning model trained by this invention (e.g., a lightweight decoder with 2 billion parameters and accompanying visual encoder, natural language encoder, and cross-modal intent alignment and focusing module). Operators at the traffic management center can access the system services through a workstation web client or mobile application to achieve remote querying and real-time monitoring.

[0026] S02: Input the real-time traffic scene image into a preset visual encoder for multi-scale feature extraction to obtain a visual feature set; In a preferred embodiment of this invention, the step of inputting the real-time traffic scene image into a preset visual encoder for multi-scale feature extraction to obtain a visual feature set specifically involves: In one specific implementation, the system invokes a pre-configured visual encoder to perform multi-scale feature extraction on the acquired real-time traffic scene images. This visual encoder preferably employs a deep neural network based on the Vision Transformer architecture, whose network parameters have been pre-trained using a large number of roadside traffic scene images. The specific processing flow is as follows: First, the real-time traffic scene images are scaled to a uniform size, such as 448×448 pixels, and divided into fixed-size image blocks. Then, the visual encoder performs linear embedding and positional encoding on each image block, and performs feature transformation through a multi-layer Transformer encoder. During the encoding process, different layers of the visual encoder output feature maps with different semantic granularities. Shallow feature maps have higher resolution and contain rich low-level details such as edges and textures, while deep feature maps have lower resolution and contain high-level semantic information such as the global topological structure and spatial relationships of the target. These multi-level feature maps are organized hierarchically into a feature map set, denoted as the visual feature set. Where L is the number of layers in the encoder. Corresponding to the shallowest feature map, This corresponds to the deepest feature map. This visual feature set serves as one of the inputs to the subsequent cross-modal fusion module, providing the model with a complete visual representation pyramid from fine-grained to coarse-grained, enabling subsequent operations to adaptively select or fuse visual information from different levels according to the query intent.

[0027] S03: Input the natural language query into a preset natural language encoder for semantic encoding to obtain text features; In a preferred embodiment of this invention, the step of inputting the natural language query into a preset natural language encoder for semantic encoding to obtain text features specifically involves: In one specific implementation, the system invokes a pre-configured natural language encoder to semantically encode the natural language query input by the user. This natural language encoder preferably employs a pre-trained language model based on the Transformer architecture, such as BERT or RoBERTa, whose parameters have been pre-trained on a large-scale general corpus and can be fine-tuned according to the traffic scenario task. The specific processing flow is as follows: First, the natural language query text input by the user is segmented and tokenized. A special classification marker [CLS] is added to the beginning of each query sequence, and an end marker [SEP] is added to the end. Each word is mapped to a corresponding word embedding vector; simultaneously, positional encoding is superimposed to preserve word order information in the sequence. Then, the embedding sequence is input into a multi-layer Transformer encoder, which transforms layer by layer through a self-attention mechanism and a feedforward network to capture the contextual dependencies between words in the query. The encoder finally outputs the semantic representation of the entire query sequence. In this embodiment, the output vector corresponding to the [CLS] marker is taken as the aggregate semantic feature of the entire natural language query, denoted as text feature Ht. This feature vector is a high-dimensional continuous vector, and the dimension can be set to 768 or 1024 according to the model configuration. This text feature serves as the source for query guidance in the subsequent cross-modal fusion module. Its semantic content directly determines the direction of attention anchoring. For example, when the query contains keywords such as "pedestrian" and "red light," the text feature will encode the corresponding semantic information, which is used to locate the relevant region in the visual features.

[0028] S04: Input the visual feature set and the text features into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module uses the text features as query guidance, performs spatial attention anchoring and cross-modal feature fusion on the visual feature set, and outputs joint representation features; wherein, the cross-modal intent alignment and focusing module is trained based on historical traffic scene images, historical natural language queries, and corresponding target positions marked in historical natural language queries; In a preferred embodiment of this invention, the visual feature set and the text features are input into a preset cross-modal intent alignment and focusing module. This allows the cross-modal intent alignment and focusing module to use the text features as a query guide, perform spatial attention anchoring and cross-modal feature fusion on the visual feature set, and output joint representation features. Specifically: After acquiring the visual feature set and text features, the system inputs both into a pre-built and trained cross-modal intent alignment and focusing module. The core function of this module is to use text features as a query guide, perform spatial attention anchoring and cross-modal feature fusion on the visual feature set, and thus output a joint representation feature rich in semantic alignment information.

[0029] Specifically, the internal structure and forward computation process of this module will be described in detail first. Let the multi-level visual feature set output by the visual encoder be... ,in This represents the feature map of the Lth layer, where L is the layer number; the text features output by the text encoder are denoted as... The cross-modal intent alignment and focusing module first integrates text features. Through a linear projection matrix After mapping, it becomes a query vector. At the same time, visual features Mapped to a key matrix AND-value matrix To enable cross-modal interaction, this approach aims to overcome the excessive computational complexity of traditional self-attention mechanisms when processing high-resolution visual features. To address the shortcomings of traditional methods (where N is the number of visual tokens), this module employs a kernel-based linear attention mechanism. Specifically, a feature mapping kernel function is introduced. Nonlinear feature mapping is performed on the query vector and the key matrix respectively to obtain the mapped query vector. and the mapped key matrix Then, the associative law of matrix multiplication is used to change the order of calculation: first calculate... Then compare the result with Multiply and divide simultaneously and The summation of rows yields the final attention weight distribution. The above calculation process can be represented as: Where N is the number of visual tokens.

[0030] This linear attention mechanism reduces computational complexity from Significantly reduced to This enables real-time cross-modal fusion of high-resolution traffic images. After obtaining the attention weight distribution, the module adjusts the value matrix according to this weight distribution. Weighted aggregation is performed to obtain weighted visual features. Simultaneously, to fully utilize the multi-scale information of the visual features, the module also performs multi-level visual feature maps based on the same attention weight distribution. Residual weighted aggregation is performed to allow high-level global topological features and low-level texture detail features to be complementary and fused according to the query intent. Finally, the multi-scale features obtained from residual weighted aggregation are fused with the preliminary weighted features and then combined with the original text features. The features are concatenated to obtain the final joint representation feature. This joint representation feature not only contains visual information aligned with the deep semantics of the user query, but also retains the spatial response map of the text intent, providing a unified cross-modal feature foundation for subsequent text generation and target localization.

[0031] Regarding the training process of the cross-modal intent alignment and focusing module, this application employs offline supervised learning for parameter optimization. The training dataset consists of historical traffic scene images, historical natural language queries targeting these images, and the explicitly labeled target locations (boundary box coordinates) in each query. During the training phase, the historical images and queries are input into the module, and the joint representation features of the output and the bounding box coordinates predicted by the auxiliary localization head are calculated. Losses (such as GIoU loss or L1 loss) are then calculated using the labeled real target locations. Through backpropagation, the module learns how to accurately anchor the semantic information in the text query to the corresponding visual regions in the image. This enables the trained module to automatically focus on the relevant traffic target regions in the image based on any natural language query during actual inference, and output joint representation features with strong semantic correspondence.

[0032] S05: Input the joint representation features into a preset lightweight decoder, so that the lightweight decoder performs step-by-step logical reasoning based on the structured logic decoupling and transfer strategy, and performs coordinate regression based on the spatial response map in the joint representation features, and outputs the bounding box coordinates of the natural language response text and the traffic target; wherein, the lightweight decoder is trained by distillation of the inference logic of a preset multimodal model. In a preferred embodiment of this invention, the joint representation features are input into a preset lightweight decoder, enabling the lightweight decoder to perform step-by-step logical reasoning based on a structured logic decoupling and transfer strategy, and to perform coordinate regression based on the spatial response map in the joint representation features, outputting the bounding box coordinates of the natural language response text and the traffic target. Specifically: After obtaining the joint representation features, the system inputs them into a pre-built and trained lightweight decoder. This lightweight decoder is designed with dual-stream parallel processing capabilities, incorporating a text generation stream and a localization stream. The former performs step-by-step logical reasoning to generate natural language response text, while the latter performs coordinate regression based on the spatial response map from the joint representation features to output bounding box coordinates. To achieve these functionalities, the lightweight decoder is trained using a structured logic decoupling and transfer learning strategy, enabling it to inherit the complex reasoning capabilities of large models even on edge devices with fewer than one billion parameters.

[0033] Specifically, the training process of the structured logic decoupling and transfer strategy is as follows: First, using a pre-trained multimodal teacher model (such as GPT-4o or other large-scale multimodal language models), a structured perception and reasoning process is generated for each set of samples in the historical training data, including historical traffic scene images, corresponding natural language queries, and the target locations marked in the queries. This reasoning process is presented in the form of a natural language chain, decomposing complex questions that originally required a one-step answer into multiple interpretable sub-steps, such as "Step 1: Locate the pedestrian area in the image; Step 2: Identify the traffic light color; Step 3: Determine whether the pedestrian crossed the zebra crossing during the red light; Step 4: Draw a conclusion." Subsequently, the generated perception and reasoning process is concatenated with the original historical images, queries, and marked target locations to form a training signal. This enhanced signal not only contains the mapping relationship between input and output but also embodies the complete cognitive chain from perception to reasoning.

[0034] Then, based on the training signal, knowledge distillation is performed on the lightweight decoder. During distillation, a multi-task joint loss function is used for optimization. This loss function consists of three parts: target localization loss (such as GIoU loss or L1 loss, used to constrain the consistency between the predicted bounding box and the true target location), inference chain consistency loss (such as KL divergence, used to constrain the difference between the logistic distribution output by the lightweight decoder and the inference chain distribution output by the teacher model), and question-answer cross-entropy loss (used to constrain the semantic accuracy between the final generated answer text and the true answer), specifically: in Localize the loss for the target (such as GIoU or L1 Loss). To constrain the inference chain consistency loss (such as KL divergence) based on the difference in logical distribution between the small model and the teacher model. The cross-entropy loss is used to determine the final question-answer accuracy. Joint optimization ensures that the system not only provides the correct answer but also grasps rigorous reasoning logic.

[0035] By minimizing the aforementioned joint loss, the lightweight decoder is forced to learn the step-by-step reasoning logic of the teacher model, rather than simply mimicking the final answer. After sufficient training, the lightweight decoder internally acquires a set of usable reasoning logic parameters that fully simulate the perceptual and reasoning behavior of the teacher model.

[0036] During the inference phase (i.e., in practical application), the system first obtains the inference logic parameters obtained through teacher model distillation. Then, the joint representation features are fed into the text generation stream of the lightweight decoder. The text generation stream performs step-by-step decoding of the joint representation features based on these inference logic parameters. In this step-by-step decoding process, the lightweight decoder first dynamically generates a directed acyclic inference subtask sequence based on the semantic content of the user query and the joint representation features, denoted as... Where n is the number of subtasks, and each subtask corresponds to an explicit reasoning step. Specifically, in the subtask sequence... This is typically a target localization subtask, used to determine the approximate region of the target in an image to which the query is directed. This is the attribute determination subtask, used to identify attributes such as the target's color, type, and motion state; Subsequent subtasks are causal inference subtasks, used to determine whether the target behavior complies with traffic rules or to predict its intent. The subtasks are executed in a directed acyclic topological order, meaning the execution of each subsequent subtask depends on the output of the preceding subtask, and there are no circular dependencies in the execution path, thus avoiding inference deadlocks or infinite loops caused by logical loops. In each decoding step, the lightweight decoder executes the current subtask sequentially and generates the corresponding intermediate text, such as "A pedestrian has been located on the zebra crossing," "The pedestrian is moving during a red light," and "This behavior constitutes running a red light." Finally, the intermediate texts of each subtask are concatenated in chronological order, and an autoregressive algorithm generates a complete natural language response text.

[0037] Meanwhile, in the localization stream, the system directly extracts the spatial response map from the joint representation features. This response map is generated synchronously by the cross-modal intent alignment and focusing module when calculating attention weights, and the value at each position reflects the relevance of the visual region to the current query intent. The localization stream performs coordinate regression on this spatial response map through an independent multilayer perceptron detector head, directly outputting the bounding box coordinates of the traffic target strongly bound to the query intent, such as normalized coordinates [xmin, ymin, xmax, ymax]. The dual-stream parallel processing mechanism allows text generation and coordinate regression to occur simultaneously. Finally, through time alignment, it is ensured that when outputting each lexical unit of the natural language response text, the corresponding bounding box coordinates are output synchronously, achieving a real-time interactive effect of "the answer is what it refers to".

[0038] To more clearly illustrate the above reasoning and localization process, a specific query example is used below. Suppose the operator enters the text query on the client: "Please check if there is a pedestrian running a red light in the current frame?" The system sends this query along with the real-time traffic scene image of the current frame into the model. In the text generation stream, the lightweight decoder generates the following directed acyclic reasoning subtask sequence based on structured logic decoupling and transfer strategies: First, the target localization subtask is executed, outputting the intermediate text "A pedestrian has been located in the zebra crossing area of ​​the image, and the traffic light above them is red"; then, the attribute determination subtask is executed, outputting "The pedestrian is moving, and the traffic light is red"; finally, the causal inference subtask is executed, outputting "According to traffic rules, pedestrians are prohibited from entering the zebra crossing during a red light. This pedestrian is on the zebra crossing, therefore constituting a red light violation." After concatenating the intermediate text, the autoregressive generator produces the final natural language response: "Yes, there is a pedestrian crossing the zebra crossing in the center of the image, which is a red-light violation." Simultaneously, in the localization stream, the system extracts the spatial response map from the joint representation features and uses a multilayer perceptron detector head to regress the pedestrian's normalized bounding box coordinates, for example, [0.45, 0.33, 0.55, 0.68]. This dual-stream parallel processing ensures the synchronous output of the response text and the bounding box coordinates, achieving a real-time interactive effect where the response is exactly what is indicated.

[0039] S06: Based on the natural language response text and the bounding box coordinates, generate traffic event question and answer results.

[0040] In a preferred embodiment of this invention, the step of generating traffic event question-and-answer results based on the natural language response text and the bounding box coordinates specifically involves: After obtaining the bounding box coordinates of the natural language response text and the traffic target, the system enters the result generation and presentation stage. Specifically, the system uses the natural language response text as a semantic answer and pushes it directly to the user interface, such as the operator workstation of the traffic management center, mobile application, or vehicle terminal, to display it to the user in the form of text or speech synthesis. At the same time, the system resolves the bounding box coordinates into specific pixel positions in the image coordinate system and overlays and renders the corresponding bounding boxes on the original real-time traffic scene image. During rendering, a rectangle is drawn using a highlight color (such as red or yellow), and labels are added near the box, such as displaying the target type or violation judgment result. Finally, the image with bounding box labels and the natural language response text are encapsulated together into a visualized traffic event Q&A result, which is presented to the user in real time. If the user is in a multi-turn dialogue mode, the system also stores the joint representation features of the current frame, the dialogue record, and the Q&A result in a memory cache for subsequent reference resolution and contextual continuous interaction. Thus, the system not only provides a semantic answer but also intuitively indicates the traffic target involved through visual labels, realizing a complete interactive loop of "the answer is what it refers to".

[0041] After the above question-and-answer results are presented, the operator can ask further questions based on the visual bounding box information, such as: "From which direction did the pedestrian enter the intersection?" When the system detects that the current query contains referential words (such as 'the pedestrian') and there is a historical dialogue record, it automatically retrieves the joint representation features, dialogue record, and image features from the memory cache of the previous round of dialogue to eliminate the referential object "the pedestrian". Subsequently, the system repeats the above reasoning and localization process, generates new answer text (such as "the pedestrian entered from the zebra crossing entrance on the left side of the screen"), and outputs the corresponding trajectory or location marker. Throughout the multi-round dialogue process, the system maintains contextual consistency, eliminating the need for the user to repeatedly describe the target features, significantly improving the naturalness and efficiency of human-computer interaction.

[0042] like Figure 2 As shown, Figure 2 This paper demonstrates a comparison between the predicted results and the ground truth (true values) for queries in different traffic scenarios. The left side lists four natural language query questions, such as "Are there pedestrians in the picture?" and "Did the pedestrian walking from right to left violate traffic rules?", while the right side provides the system's predicted answers and the corresponding ground truth answers. The comparison shows that the system's predictions are highly consistent with the ground truth, verifying the accuracy and reliability of the proposed cross-modal intent alignment and structured reasoning method in question answering within complex traffic scenarios.

[0043] The overall process of this application is as follows: Figure 3 As shown, Figure 3The system takes a real-time traffic scene image as input on the left, which is then processed by a visual encoder and adaptive visual encoder to extract multi-scale visual features and serialize them into image patch markers. Simultaneously, the user's natural language query is processed by a word segmenter to form a language token, which, along with the visual features, is input into the cognitive anchoring fusion module (CAF). The text query serves as the attention guide to achieve cross-modal alignment, outputting multimodal token features. Subsequently, the features are fed into a large language model (such as Qianwen 2.5), combined with the auxiliary decoding thought chain context generated by GPT-4o, to finally generate a natural language answer and target localization result.

[0044] In summary, this application first acquires real-time traffic scene images and user natural language queries collected by roadside perception devices, and extracts visual feature sets and text features through visual encoders and natural language encoders, respectively. Then, using a pre-trained cross-modal intent alignment and focusing module, guided by text features, spatial attention anchoring and cross-modal feature fusion are performed on the visual feature set, enabling the model to dynamically focus on visual regions strongly related to the user's query intent and effectively suppress interference from complex background noise, thereby generating joint representation features containing semantic alignment information. Based on this, the joint representation features are input into a lightweight decoder obtained through structured logic decoupling and transfer strategy distillation, which allows it to inherit the complex reasoning capabilities of large models while maintaining low computational overhead. During the decoding process, coordinate regression is performed synchronously based on the spatial response map in the joint representation features, and finally, the bounding box coordinates of the natural language response text and traffic targets are output in parallel. Therefore, based on the realization of high-level semantic reasoning and natural language interaction, this application further endows the system with the visual positioning capability of "the answer is the reference", that is, while outputting semantic answers, it accurately points out the location of traffic targets involved in the image. This application effectively solves the problem that the existing technology cannot accurately and efficiently perform intelligent question answering on traffic events based on natural language.

[0045] Example 2 Please refer to Figure 4 This is a natural language-based question-and-answer system for traffic events provided in the embodiments of this application.

[0046] In this embodiment, the natural language-based traffic event question-and-answer system includes an acquisition module 10, an extraction module 20, an encoding module 30, a fusion module 40, a decoding module 50, and a generation module 60.

[0047] The acquisition module 10 is used to acquire real-time traffic scene images collected by roadside sensing devices and natural language queries input by users. The extraction module 20 is used to input the real-time traffic scene image into a preset visual encoder for multi-scale feature extraction to obtain a visual feature set; The encoding module 30 is used to input the natural language query into a preset natural language encoder for semantic encoding to obtain text features; The fusion module 40 is used to input the visual feature set and the text features into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module uses the text features as query guidance, performs spatial attention anchoring and cross-modal feature fusion on the visual feature set, and outputs joint representation features; wherein, the cross-modal intent alignment and focusing module is trained based on historical traffic scene images, historical natural language queries, and corresponding target positions marked in historical natural language queries; The decoding module 50 is used to input the joint representation features into a preset lightweight decoder, so that the lightweight decoder performs step-by-step logical reasoning based on the structured logic decoupling and transfer strategy, and performs coordinate regression based on the spatial response map in the joint representation features, and outputs the bounding box coordinates of the natural language response text and the traffic target; wherein, the lightweight decoder is trained by inference logic distillation based on a preset multimodal model; The generation module 60 is used to generate traffic event question and answer results based on the natural language answer text and the bounding box coordinates.

[0048] For ease of description and brevity, the system embodiments of the present invention include all the implementation methods described in the above embodiments of the question-and-answer method for traffic events based on natural language, and will not be repeated here.

[0049] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A question-and-answer method for traffic events based on natural language, characterized in that, include: Acquire real-time traffic scene images collected by roadside sensing devices and natural language queries input by users; The real-time traffic scene image is input into a preset visual encoder for multi-scale feature extraction to obtain a visual feature set; The natural language query is input into a preset natural language encoder for semantic encoding to obtain text features; The visual feature set and the text features are input into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module uses the text features as query guides, performs spatial attention anchoring and cross-modal feature fusion on the visual feature set, and outputs joint representation features; wherein, the cross-modal intent alignment and focusing module is trained based on historical traffic scene images, historical natural language queries, and corresponding target positions marked in historical natural language queries; The joint representation features are input into a preset lightweight decoder, which performs step-by-step logical reasoning based on a structured logic decoupling and transfer strategy, and performs coordinate regression based on the spatial response map in the joint representation features to output the bounding box coordinates of the natural language response text and the traffic target; wherein, the lightweight decoder is trained by distillation of the inference logic of a preset multimodal model. Based on the natural language response text and the bounding box coordinates, traffic event question and answer results are generated.

2. The question-and-answer method for traffic events based on natural language according to claim 1, characterized in that, The step involves inputting the visual feature set and the text features into a preset cross-modal intent alignment and focusing module. This module uses the text features as a query guide, performs spatial attention anchoring and cross-modal feature fusion on the visual feature set, and outputs joint representation features. Specifically: The visual feature set and the text features are input into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module maps the text features as a query vector after mapping by a linear projection matrix, and maps the visual feature set into a key matrix and a value matrix respectively after mapping by a linear projection matrix; The query vector and the key matrix are subjected to linear attention calculation based on a kernel function to obtain the attention weight distribution; The value matrix is ​​weighted and aggregated according to the attention weight distribution to obtain weighted visual features; The weighted visual features and the text features are fused across modalities to obtain the joint representation features.

3. The question-and-answer method for traffic events based on natural language according to claim 2, characterized in that, The step of performing a kernel-based linear attention calculation on the query vector and the key matrix to obtain the attention weight distribution is as follows: Based on the feature mapping kernel function, nonlinear feature mapping is performed on the query vector and the key matrix respectively to obtain the mapped query vector and the mapped key matrix; According to the associative law of matrix multiplication, the mapped key matrix and the value matrix are first multiplied together, and then the result is multiplied together with the mapped query vector to obtain the attention weight distribution.

4. The question-and-answer method for traffic events based on natural language according to claim 2, characterized in that, The step of weighted aggregation of the value matrix based on the attention weight distribution to obtain weighted visual features is specifically as follows: Based on the attention weight distribution, the value matrix is ​​weighted and summed to obtain preliminary weighted features; Based on the attention weight distribution, residual weighted aggregation is performed on the multi-level visual feature maps in the visual feature set so that the high-level global topological structure features and the low-level texture detail features can be complementaryly fused. The residual weighted aggregated features are fused with the preliminary weighted features, and then concatenated with the text features to obtain the weighted visual features.

5. The question-and-answer method for traffic events based on natural language according to claim 1, characterized in that, The step involves inputting the joint representation features into a preset lightweight decoder, enabling the lightweight decoder to perform step-by-step logical reasoning based on a structured logic decoupling and transfer strategy, and to perform coordinate regression based on the spatial response map in the joint representation features, outputting the bounding box coordinates of the natural language response text and the traffic target. Specifically: The joint representation features are input into a preset lightweight decoder, so that the lightweight decoder performs dual-stream parallel processing on the joint representation features and inputs them into the built-in text generation stream and positioning stream respectively; In the text generation stream, the joint representation features are subjected to step-by-step logical reasoning based on the structured logic decoupling and transfer strategy, and the natural language response text is generated by autoregression. In the localization stream, the spatial response map is extracted from the joint representation features, and coordinate regression is performed on the spatial response map through the location prediction branch to output the bounding box coordinates; The outputs of the text generation stream and the positioning stream are time-aligned, and the bounding box coordinates are output simultaneously while generating the natural language response text.

6. The question-and-answer method for traffic events based on natural language according to claim 5, characterized in that, In the text generation stream, based on a structured logic decoupling and transfer strategy, step-by-step logical reasoning is performed on the joint representation features to generate the natural language response text via autoregression, specifically as follows: Obtain the reasoning logic parameters obtained through distillation of a preset teacher model; the reasoning logic parameters are used to simulate the step-by-step perception and reasoning process of the teacher model; Based on the inference logic parameters, the joint representation features are decoded step by step in the text generation stream. In each decoding step, the target localization subtask, attribute determination subtask, and causal inference subtask are executed sequentially, and the intermediate text of the corresponding subtask is generated. The intermediate texts from each subtask are concatenated in chronological order, and the natural language response text is generated through autoregression.

7. The question-and-answer method for traffic events based on natural language according to claim 6, characterized in that, The acquisition of the reasoning logic parameters obtained through the pre-set teacher model distillation specifically involves: Based on a pre-set multimodal teacher model, combined with historical traffic scene images, historical natural language queries, and the corresponding target locations marked in the historical natural language queries, a perception and reasoning process is generated. The perception and reasoning process is spliced ​​with the historical traffic scene image, historical natural language query, and the corresponding target location marked in the historical natural language query to generate a training signal; Based on the training signal, knowledge distillation is performed on the lightweight decoder, and a multi-task joint loss function is used for optimization to obtain the inference logic parameters. The multi-task joint loss function includes target localization loss, inference chain consistency loss, and question-answer cross-entropy loss; the inference chain consistency loss is used to constrain the logical distribution difference between the lightweight decoder and the teacher model.

8. A question-and-answer system for traffic events based on natural language, characterized in that, include: The acquisition module is used to acquire real-time traffic scene images collected by roadside sensing devices and natural language queries input by users. The extraction module is used to input the real-time traffic scene image into a preset visual encoder for multi-scale feature extraction to obtain a visual feature set; The encoding module is used to input the natural language query into a preset natural language encoder for semantic encoding to obtain text features; The fusion module is used to input the visual feature set and the text features into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module uses the text features as query guidance, performs spatial attention anchoring and cross-modal feature fusion on the visual feature set, and outputs joint representation features; wherein, the cross-modal intent alignment and focusing module is trained based on historical traffic scene images, historical natural language queries, and corresponding target positions marked in historical natural language queries; The decoding module is used to input the joint representation features into a preset lightweight decoder, so that the lightweight decoder performs step-by-step logical reasoning based on the structured logic decoupling and transfer strategy, and performs coordinate regression based on the spatial response map in the joint representation features, and outputs the bounding box coordinates of the natural language response text and the traffic target; wherein, the lightweight decoder is trained by distillation of the inference logic of a preset multimodal model; The generation module is used to generate traffic event question-and-answer results based on the natural language response text and the bounding box coordinates.

9. The natural language-based question-and-answer system for traffic events according to claim 8, characterized in that, The fusion module includes: The first input unit is used to input the visual feature set and the text features into a preset cross-modal intent alignment and focusing module, so that the cross-modal intent alignment and focusing module maps the text features as a query vector after mapping by a linear projection matrix, and maps the visual feature set into a key matrix and a value matrix respectively after mapping by a linear projection matrix; The calculation unit is used to perform linear attention calculation based on kernel function on the query vector and the key matrix to obtain the attention weight distribution; An aggregation unit is used to perform weighted aggregation on the value matrix according to the attention weight distribution to obtain weighted visual features; The fusion unit is used to perform cross-modal fusion of the weighted visual features and the text features to obtain the joint representation features.

10. The natural language-based question-and-answer system for traffic events according to claim 8, characterized in that, The decoding module includes: The second input unit is used to input the joint representation features into a preset lightweight decoder, so that the lightweight decoder performs dual-stream parallel processing on the joint representation features and inputs them into the built-in text generation stream and positioning stream respectively; The text generation unit is used to perform step-by-step logical reasoning on the joint representation features based on the structured logic decoupling and transfer strategy in the text generation stream, and generate the natural language response text through autoregression. The localization unit is used to extract the spatial response map from the joint representation features in the localization stream, perform coordinate regression on the spatial response map through the position prediction branch, and output the bounding box coordinates. The alignment unit is used to time-align the output of the text generation stream with the output of the positioning stream, and to synchronously output the bounding box coordinates while generating the natural language response text.

Images