Highway event video annotation method and system based on structured multi-round question and answer

Abstract

This invention discloses a method and system for annotating highway event videos based on structured multi-turn question answering. The method includes quality assessment and video enhancement of the surveillance video stream, determination of the event observation time and segmentation and cropping of the surveillance video stream, identification of scene attributes for each surveillance video segment, construction of a question-answering mapping model to obtain annotation questions and generate JSON-formatted annotation templates, pre-annotation on the annotation templates using a pre-trained video understanding model, human-computer collaborative annotation to generate annotation text based on the pre-annotation results and quality assessment results, storage of the annotation text in a preset file directory, and writing the basic metadata and annotation text storage paths of the same surveillance video segment into a database table. The database table is used for association retrieval of surveillance video segments and annotation text. This method not only improves the efficiency and accuracy of highway event video annotation but also has good interpretability and can be directly applied to highway event video annotation systems.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation data processing and computer vision technology, and in particular to a method and system for annotating highway event videos based on structured multi-turn question answering. Background Technology

[0002] With the comprehensive coverage and high-definition upgrade of my country's highway network monitoring system, the generation of massive amounts of real-time monitoring video data has provided a valuable data source for training next-generation multimodal large language models and conducting in-depth analysis and prediction of traffic events. However, the raw video itself is an unstructured pixel stream that cannot be effectively understood and utilized by machines. How to efficiently, accurately, and consistently transform massive amounts of video into structured text-annotated data has become a key bottleneck restricting the development of intelligent transportation technology.

[0003] Currently, traditional methods of annotating highway videos mainly rely on manual viewing and writing of natural language descriptions, which have significant shortcomings: First, the annotation granularity is coarse, lacking the depiction of deep semantic information such as event details, vehicle interactions, and environmental causality; second, the consistency and standardization of annotation are poor, with different annotators having inconsistent descriptive styles and focuses for the same video, resulting in messy text formats; third, the unstructured data leads to low utilization, and plain text descriptions are difficult to retrieve and analyze in batches through programmatic methods; fourth, the connection between video and description is loose, with text annotations and video metadata (such as time, location, and weather) often managed separately, forming "data silos" that cannot support complex multi-condition joint queries and analyses. Therefore, this invention proposes a method and system for annotating highway event videos based on structured multi-turn question answering. By introducing multimodal features to automatically identify deep scene attributes of videos, and based on this, a question-answering mapping model is constructed that can intelligently recommend the most relevant annotation questions according to specific scenes and event types. This achieves a leap from "fixed templates" to "scene-adaptive templates." It incorporates a pre-trained video understanding model for pre-annotation and combines video quality assessment results for differentiated human-computer collaborative guidance. While ensuring annotation quality and consistency, it significantly improves annotation efficiency. This provides a feasible technical path for constructing large-scale, high-quality multimodal datasets in the transportation field and has important engineering practical significance for promoting the training and deployment of deep learning models in intelligent transportation systems. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for annotating highway event videos based on structured multi-turn question answering.

[0005] To achieve the above objectives, the present invention is implemented according to the following technical solution: This invention includes the following steps: Acquire the original highway surveillance video stream and basic metadata, perform quality assessment on the surveillance video stream, and perform video enhancement based on the quality assessment results; The event observation time is determined based on the basic metadata. The monitoring video stream is segmented and cut according to the event observation time. Multimodal features of each monitoring video segment are extracted to identify scene attributes. A question-and-answer mapping model is constructed based on the labeled questions, scene attributes, and event categories of historical monitoring video segments. The monitoring video segment to be labeled is input into the question-and-answer mapping model to obtain the labeling question. A JSON format labeling template is generated based on the labeling question, and the monitoring video segment to be labeled is associated with the labeling template. The monitoring video segment to be annotated is loaded into the annotation software and the corresponding annotation template is called. A pre-trained video understanding model is used to pre-annotate on the annotation template. Based on the pre-annotation results and quality assessment results, human-computer collaborative annotation is performed to generate annotation text. The labeled text is stored in a preset file directory. The basic metadata and labeled text storage paths of the same monitoring video segment are written into a database table, and the association retrieval of monitoring video segments and labeled text is performed through the database table. The basic metadata includes the event time, highway name, mileage marker, direction of travel, camera ID, lane location, and event category; The annotation questions cover event description, vehicle information, location information, environmental impact, and traffic flow status, and are assigned unique identifiers and required attributes.

[0006] Furthermore, the method for video enhancement based on quality assessment results includes: The original highway surveillance video stream is acquired, and the quality of continuous surveillance video segments is evaluated. The quality evaluation indicators include basic image quality, lighting conditions, motion blur, degree of occlusion, and scene complexity. Invalid surveillance video segments with any quality assessment index exceeding the invalid threshold are removed. All surveillance video segments with quality assessment indexes exceeding the corresponding enhancement threshold are directly included in the annotation queue. Conversely, the remaining surveillance video segments are enhanced and then included in the annotation queue. The video enhancement operations include image equalization, deblurring, local repair, and scene normalization. The surveillance video segments of the queue to be labeled are de-identified and then reassembled into a surveillance video stream according to the time sequence.

[0007] Furthermore, the method for segmenting and trimming the surveillance video stream includes: The event observation time is determined based on the basic metadata, and the expression is: ; in for Event observation time The time scale constant, For standard time intervals, The time of the event With peak traffic hours The absolute value of the time deviation The sensitivity coefficient for road grade. for The deviation between the road grade where the incident occurred and the standard road grade. The sensitivity coefficient for the number of lanes. for The deviation between the number of lanes where the incident occurred and the standard number of lanes. This is the camera resolution sensitivity coefficient. The deviation between the camera resolution and the standard camera resolution. for Event Categories The corresponding baseline observation time; The surveillance video stream was segmented and cut according to the event observation time.

[0008] Furthermore, the method for constructing the question-answering mapping model includes: The video images from the surveillance video segment are extracted and image features are obtained through image detection. Based on the image features, the main scene attributes and corresponding confidence scores are calculated. The main scene attributes include weather, time of day, and traffic density. The video and audio of the surveillance video segment are extracted for audio detection to determine the auxiliary scene attributes and corresponding confidence levels; the auxiliary scene attributes include weather and event clues. Extract video text from surveillance video segments, match and correct basic metadata; The scene attributes are obtained by weighted fusion of the main scene attributes and the auxiliary scene attributes; The historical surveillance video segments are labeled with question sequences, scene attributes, and event categories to form a comprehensive set. The comprehensive set is then randomly divided into a training set and a test set in a 6:4 ratio. The question-answering mapping model is trained using the training set and evaluated using the test set. The question-answering mapping model includes an input layer and an embedding layer, a feature fusion and interaction layer, a multi-label prediction layer, and an output layer. The input and embedding layers concatenate the input scene attributes and event categories into a long vector, and then embed and increase the dimensionality of this long vector using a fully connected layer. The feature fusion and interaction layer projects the long vector into a key vector and a value vector, uses the event category embedding vector as the query vector, calculates attention weights using a cross-modal attention module, and fuses to generate event-related scene features. The multi-label prediction layer performs non-linear transformations on the event-related scene features through three fully connected layers, and predicts the probability of all labeled questions being recommended using a sigmoid activation function, outputting a labeled question recommendation probability vector. The output layer connects to the multi-label prediction layer through a fully connected layer, extracts labeled questions from the labeled question recommendation probability vector that are greater than a probability threshold, and outputs a sequence of labeled questions. The question-answering mapping model uses the Adam optimizer to adjust the model's hyperparameters; the question-answering mapping model uses binary cross-entropy loss with class weights to improve the model's accuracy in predicting labeled questions, as expressed in: ; in For binary cross-entropy loss, For the sample size, for The weight of the labeling question, for Label the problem with the true label.

[0009] Furthermore, the method for generating labeled text through human-computer collaborative annotation includes: The process involves processing the surveillance video segments to be labeled to obtain scene attributes and event categories, inputting them into a question-and-answer mapping model to obtain a sequence of labeling questions, generating a JSON-formatted labeling template, and then associating the surveillance video segments to be labeled with the labeling template. The monitoring video segment to be annotated is loaded into the annotation software and the corresponding annotation template is called. A pre-trained video understanding model is used to pre-annotate the annotation template; the pre-annotation is used for the annotator's reference. When the scene complexity is less than the complexity threshold, the pre-annotated results and the monitoring video segments to be annotated are randomly assigned to one annotator for manual annotation to generate the annotation results; otherwise, the pre-annotated results and the monitoring video segments to be annotated are randomly assigned to two annotators for manual annotation. The two annotation results are compared. If there is no difference, the annotation result is generated directly. If there is a difference, the system automatically merges the difference items and submits them to experts for final review to generate the annotation result. The annotation results are encapsulated into a structured JSON object, generating annotation text that uniquely corresponds to the video file.

[0010] Secondly, a highway event video annotation system based on structured multi-turn question answering includes: Video processing module: used to perform quality assessment on the surveillance video stream, enhance the video based on the quality assessment results, determine the event observation time based on the basic metadata, and cut the surveillance video stream into segments according to the event observation time; Standard template generation module: used to extract multimodal features of each monitoring video segment to identify scene attributes, build a question-and-answer mapping model based on the annotation questions, scene attributes and event categories of historical monitoring video segments, input the monitoring video segment to be annotated into the question-and-answer mapping model to obtain the annotation questions, and generate a JSON format annotation template based on the annotation questions; Human-computer collaborative annotation module: Load the monitoring video segment to be annotated into the annotation software and call the corresponding annotation template. Use a pre-trained video understanding model to pre-annotate on the annotation template. Generate annotation text through human-computer collaborative annotation based on the pre-annotation results and quality assessment results. Data module: It is used to encapsulate the annotation results into a structured JSON object, generate annotation text that uniquely corresponds to the video file, store the annotation text in a preset file directory, write the basic metadata and annotation text storage path of the same monitoring video segment into a database table, and perform association retrieval of monitoring video segments and annotation text through the database table.

[0011] The beneficial effects of this invention are: This invention relates to a method and system for annotating highway event videos based on structured multi-turn question answering. Compared with existing technologies, this invention has the following technical advantages: This invention guides annotators by covering a multi-dimensional set of structured questions, forcibly directing their attention to key details, ensuring the depth, breadth, and standardization of the output annotations, and generating structured JSON data that can be directly used for training large models, thus changing the situation of coarse granularity and messy format of traditional natural language descriptions. This invention achieves scene-adaptive dynamic material selection and annotation by assessing video quality, dynamically cropping and annotating video segments based on metadata, automatically identifying the multimodal features of annotated videos, and intelligently generating a sequence of annotation questions that best match the current video scene and events using a trained question-answering mapping model. This invention uses a pre-trained video understanding model for automatic pre-annotation, providing human annotators with high-quality reference answers and starting points, effectively reducing repetitive work and basic errors. At the same time, based on the results of previous video quality assessments, it simplifies the annotation process for simple quality video segments, achieving complementary advantages between humans and machines and improving overall annotation efficiency and cost-effectiveness. The annotation data output by this invention adopts a standardized JSON format, which includes unique identifiers and mandatory attribute validation to ensure the standardization of the data structure and machine readability. The basic metadata and annotation text storage paths are written into the database table, establishing an efficient and fast association and retrieval mechanism between video files and annotation data, providing a ready-to-use data foundation for downstream applications. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the steps of the highway event video annotation method based on structured multi-turn question answering in this invention. Figure 2 This is a schematic diagram illustrating the anonymization process of surveillance video segments in an embodiment of the present invention; Figure 3 This is a schematic diagram of a traffic accident video annotation example in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the contents of a database table in an embodiment of the present invention. Detailed Implementation

[0013] The present invention will be further described below through specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0014] The present invention provides a method and system for annotating highway event videos based on structured multi-turn question answering, comprising the following steps: like Figure 1 As shown, this embodiment includes the following steps: Acquire the original highway surveillance video stream and basic metadata, perform quality assessment on the surveillance video stream, and perform video enhancement based on the quality assessment results; The event observation time is determined based on the basic metadata. The monitoring video stream is segmented and cut according to the event observation time. Multimodal features of each monitoring video segment are extracted to identify scene attributes. A question-and-answer mapping model is constructed based on the labeled questions, scene attributes, and event categories of historical monitoring video segments. The monitoring video segment to be labeled is input into the question-and-answer mapping model to obtain the labeling question. A JSON format labeling template is generated based on the labeling question, and the monitoring video segment to be labeled is associated with the labeling template. The monitoring video segment to be annotated is loaded into the annotation software and the corresponding annotation template is called. A pre-trained video understanding model is used to pre-annotate on the annotation template. Based on the pre-annotation results and quality assessment results, human-computer collaborative annotation is performed to generate annotation text. The labeled text is stored in a preset file directory. The basic metadata and labeled text storage paths of the same monitoring video segment are written into a database table, and the association retrieval of monitoring video segments and labeled text is performed through the database table. The basic metadata includes the event time, highway name, mileage marker, direction of travel, camera ID, lane location, and event category; The annotation questions cover event description, vehicle information, location information, environmental impact, and traffic flow status, and are assigned unique identifiers and required attributes.

[0015] In this embodiment, the method for video enhancement based on quality assessment results includes: The original highway surveillance video stream is acquired, and the quality of continuous surveillance video segments is evaluated. The quality evaluation indicators include basic image quality, lighting conditions, motion blur, degree of occlusion, and scene complexity. Invalid surveillance video segments with any quality assessment index exceeding the invalid threshold are removed. All surveillance video segments with quality assessment indexes exceeding the corresponding enhancement threshold are directly included in the annotation queue. Conversely, the remaining surveillance video segments are enhanced and then included in the annotation queue. The video enhancement operations include image equalization, deblurring, local repair, and scene normalization. After desensitizing the surveillance video segments of the queue to be labeled, they are reassembled into a surveillance video stream according to the time sequence. In the actual evaluation, the original highway surveillance video stream was acquired, and the surveillance video stream was divided into continuous surveillance video segments with a time continuity of 10 seconds. The basic image quality, lighting conditions, motion blur, degree of occlusion and scene complexity of each surveillance video segment were calculated. Among them, basic image quality is used to evaluate the overall image clarity and information richness, represented by the average grayscale image information entropy of the middle frames and uniformly sampled frames of the video clip (the lower the average grayscale image information entropy, the more concentrated the grayscale distribution and the more blurred the details); lighting conditions are used to determine whether the video is too dark or overexposed, affecting target recognition, represented by the average pixel brightness of all frames of the video clip; motion blur is used to detect image blur caused by camera shake or high-speed target movement, represented by the blur index (the higher the value, the more severe the motion blur), the blur index calculation steps include: using the optical flow method (Farneback algorithm) to calculate the pixel motion vector field between adjacent frames, analyzing the consistency of the amplitude and direction of the motion vector, and taking the average amplitude of the motion vector × (1 - the consistency coefficient of the motion vector direction) as the blur index; occlusion degree is used to evaluate the partial obstruction of the target vehicle. The proportion of occlusion affects the completeness of event analysis, represented by the average occlusion rate (the higher the value, the more severe the mutual occlusion). The average occlusion rate is calculated by: using a lightweight object detection model (YOLO variant) to detect vehicles on keyframes, calculating the overlap ratio of the bounding boxes of the detected vehicles, and taking the average of the maximum intersection-union ratios between all vehicle detection boxes as the average occlusion rate; Scene complexity is used to evaluate the complexity of the elements in the image, affecting the attention allocation of annotators, and is represented by scene entropy (a high value indicates a uniform distribution of scene categories, complex visual elements, and higher manpower / computing power costs for annotation). The scene complexity is calculated by: using a semantic segmentation model (DeepLabv3+) to segment keyframes into categories such as roads, vehicles, sky, and guardrails, and calculating the Shannon entropy of the pixel ratio of each category as the scene complexity. For surveillance video segments whose quality assessment indicators exceed the invalid threshold, the repair cost is too high or the information loss is serious, so they are directly removed. For surveillance video segments whose quality assessment indicators all exceed the corresponding enhancement threshold, no enhancement is required and they are directly included in the waiting list. For the remaining surveillance video segments whose one or more indicators do not exceed the corresponding enhancement threshold, the corresponding enhancement module is triggered. The image equalization operation employs Adaptive Histogram Equalization (CLAHE) and Low-Light Enhancement Network (Zero-DCE) to improve overall brightness and contrast; the deblurring operation uses a deep learning-based video deblurring method (EDVR) to restore the image using multi-frame information; the local restoration operation includes using temporal information to complete the video of intermittently appearing occlusions in consecutive frames and marking areas that are difficult to restore; the scene normalization operation includes resolution normalization (scaling to 1080p), frame rate unification (30fps), and color space normalization (sRGB). After desensitizing the surveillance video clips in the queue to be labeled, they were reassembled into a surveillance video stream according to time sequence. Due to the standard requirements for highway surveillance videos, most of the original videos have sensitive information marked in the upper left and lower right corners, including the shooting location, event, and road section. Since this information is sensitive and not convenient for external labelers to access directly, a mask was uniformly applied during preprocessing. Figure 2 As shown.

[0016] In this embodiment, the method for segmenting and trimming the surveillance video stream includes: The event observation time is determined based on the basic metadata, and the expression is: ; in for Event observation time The time scale constant, For standard time intervals, The time of the event With peak traffic hours The absolute value of the time deviation The sensitivity coefficient for road grade. for The deviation between the road grade where the incident occurred and the standard road grade. The sensitivity coefficient for the number of lanes. for The deviation between the number of lanes where the incident occurred and the standard number of lanes. This is the camera resolution sensitivity coefficient. The deviation between the camera resolution and the standard camera resolution. for Event Categories The corresponding baseline observation time; The surveillance video stream was segmented and cut according to the event observation time. In practical evaluations, the original videos are often excessively long, with many videos lasting for hours. Beyond key events, there are numerous useless video segments, significantly impacting efficiency during annotation and model training. Furthermore, the required video duration varies depending on the event. The timing of the event (the closer to peak traffic hours, the more information needs to be considered), lane specifications (road grade, number of lanes), and camera performance all affect the video observation duration. This method calculates the event observation time for each monitoring video segment based on the basic metadata at the time of the event (a complete video includes information before, during, and after the event). The event types in the basic metadata include accidents, lane stops, pedestrians, construction, spilled objects, congestion, obstacle removal, and no event.

[0017] In this embodiment, the method for constructing the question-answering mapping model includes: The video images from the surveillance video segment are extracted and image features are obtained through image detection. Based on the image features, the main scene attributes and corresponding confidence scores are calculated. The main scene attributes include weather, time of day, and traffic density. The video and audio of the surveillance video segment are extracted for audio detection to determine the auxiliary scene attributes and corresponding confidence levels; the auxiliary scene attributes include weather and event clues. Extract video text from surveillance video segments, match and correct basic metadata; The scene attributes are obtained by weighted fusion of the main scene attributes and the auxiliary scene attributes; The historical surveillance video segments are labeled with question sequences, scene attributes, and event categories to form a comprehensive set. The comprehensive set is then randomly divided into a training set and a test set in a 6:4 ratio. The question-answering mapping model is trained using the training set and evaluated using the test set. The question-answering mapping model includes an input layer and an embedding layer, a feature fusion and interaction layer, a multi-label prediction layer, and an output layer. The input and embedding layers concatenate the input scene attributes and event categories into a long vector, and then embed and increase the dimensionality of this long vector using a fully connected layer. The feature fusion and interaction layer projects the long vector into a key vector and a value vector, uses the event category embedding vector as the query vector, calculates attention weights using a cross-modal attention module, and fuses to generate event-related scene features. The multi-label prediction layer performs non-linear transformations on the event-related scene features through three fully connected layers, and predicts the probability of all labeled questions being recommended using a sigmoid activation function, outputting a labeled question recommendation probability vector. The output layer connects to the multi-label prediction layer through a fully connected layer, extracts labeled questions from the labeled question recommendation probability vector that are greater than a probability threshold, and outputs a sequence of labeled questions. The question-answering mapping model uses the Adam optimizer to adjust the model's hyperparameters; the question-answering mapping model uses binary cross-entropy loss with class weights to improve the model's accuracy in predicting labeled questions, as expressed in: ; in For binary cross-entropy loss, For the sample size, for The weight of the labeling question, for Label the problem with its true characteristics; In practical evaluation, deep feature vectors are extracted as global appearance features using a pre-trained Scene Recognition CNN. For the sky region obtained from semantic segmentation, its average brightness, color histogram (especially the proportion of blue and gray), and texture features are calculated. The global appearance features and sky region features are input into a multi-label classification network to output the probability distribution of each weather and time period label. Based on target detection and tracking, dynamic target features (vehicle count, average speed, density) are determined. The number of vehicle trajectories and average displacement within a fixed ROI (main lane) per unit time are counted, and traffic density is calculated as (number of vehicle trajectories / area of ​​main lane) * (1 - (average speed / speed limit)). Using the Mel-spectrogram of video audio as the basic feature in the time-frequency domain, a pre-trained audio event classification model (such as PANNs) is used to extract high-level embedding vectors from the Mel-spectrogram, which are then fed into a lightweight classifier to output the probability of a specific audio event (rain, thunder, horn). The PaddleOCR engine is used to detect and recognize text in fixed areas of keyframes. Through named entity recognition (NER) and rule matching, metadata fields such as "station number", "direction" and "time" are extracted from the OCR results in a structured manner. This information will directly supplement or correct the metadata extracted from the video file name and provide a more accurate "time period" judgment (such as "18:45" can be directly judged as evening). The scene attributes are obtained by weighting and fusing the primary scene attributes and the auxiliary scene attributes according to their confidence ratios. The question-answering mapping model mainly predicts the probability of each labeled question in the system's total question library being recommended based on the input scene attributes and event categories. When the probability of a labeled question being recommended is greater than the recommendation threshold, the code of the corresponding labeled question is output, and the codes of all the labeled questions that meet the requirements are combined to form a labeled question sequence. Among them, the physical function of feature fusion and interaction layer is to use "event category" as the query to retrieve the most relevant features (i.e. event-related scene features) in "scene attributes".

[0018] In this embodiment, the method for generating labeled text through human-computer collaborative annotation includes: The process involves processing the surveillance video segments to be labeled to obtain scene attributes and event categories, inputting them into a question-and-answer mapping model to obtain a sequence of labeling questions, generating a JSON-formatted labeling template, and then associating the surveillance video segments to be labeled with the labeling template. The monitoring video segment to be annotated is loaded into the annotation software and the corresponding annotation template is called. A pre-trained video understanding model is used to pre-annotate the annotation template; the pre-annotation is used for the annotator's reference. When the scene complexity is less than the complexity threshold, the pre-annotated results and the monitoring video segments to be annotated are randomly assigned to one annotator for manual annotation to generate the annotation results; otherwise, the pre-annotated results and the monitoring video segments to be annotated are randomly assigned to two annotators for manual annotation. The two annotation results are compared. If there is no difference, the annotation result is generated directly. If there is a difference, the system automatically merges the difference items and submits them to experts for final review to generate the annotation result. The annotation results are encapsulated into a structured JSON object, generating annotation text that uniquely corresponds to the video file; In actual evaluation, the monitoring video segments to be labeled are processed to obtain scene attributes and event categories. These are then input into a question-and-answer mapping model to obtain a sequence of annotation questions and generate a JSON-formatted annotation template. An example of the generated question-and-answer template design is shown below: Annotation Template 1: Q1: What traffic incident occurred in the video? Q2: Where did the traffic incident occur? Q3: Please describe the environmental conditions at the time of the incident. Q4: What is the traffic situation in the video? If there is a queue, does the queue extend beyond the camera's range? How many vehicles are in the queue? Annotation Template 2: Q1: Describe the traffic scene in the video; Q2: What is the traffic flow recorded in the video? Are there any congestion or abnormal traffic fluctuations? Q3: Please describe the environmental conditions when the event occurred; Q4: Are there any potential traffic safety hazards, such as vehicle malfunctions or road damage? The annotation software loads the surveillance video segments to be annotated and calls the corresponding annotation templates. A pre-trained video understanding model pre-annotates the templates. An annotator is assigned a number of tasks based on the scene complexity. Annotators manually answer questions using the annotation tool, watching the video and answering the template questions sequentially. The annotation software supports pause, slow motion, playback, and frame-by-frame extraction of the video. The annotator's answers are captured by the system in real time. After all questions are answered, the system automatically encapsulates all question-answer pairs into structured JSON objects, generating a unique annotation text file corresponding to each video file. Figure 3 This is a case study of a traffic accident. The generated labeled text files are stored in a preset file directory. A database table structure is established, and the video's metadata information and the storage path of the labeled text files are written into the database tables. The primary key field of the database enables fast association and retrieval of video files and labeled text files, forming a complete structured multimodal dataset. The database tables for this traffic accident are as follows: Figure 4 As shown.

[0019] Secondly, a highway event video annotation system based on structured multi-turn question answering includes: Standard template generation module: used to extract multimodal features of each monitoring video segment to identify scene attributes, build a question-and-answer mapping model based on the annotation questions, scene attributes and event categories of historical monitoring video segments, input the monitoring video segment to be annotated into the question-and-answer mapping model to obtain the annotation questions, and generate a JSON format annotation template based on the annotation questions; Human-computer collaborative annotation module: Load the monitoring video segment to be annotated into the annotation software and call the corresponding annotation template. Use a pre-trained video understanding model to pre-annotate on the annotation template. Generate annotation text through human-computer collaborative annotation based on the pre-annotation results and quality assessment results. Data module: It is used to encapsulate the annotation results into a structured JSON object, generate annotation text that uniquely corresponds to the video file, store the annotation text in a preset file directory, write the basic metadata and annotation text storage path of the same monitoring video segment into a database table, and perform association retrieval of monitoring video segments and annotation text through the database table.

[0020] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for annotating highway event videos based on structured multi-turn question answering, characterized in that, Includes the following steps: S1. Obtain the original highway surveillance video stream and basic metadata, perform quality assessment on the surveillance video stream, and perform video enhancement based on the quality assessment results; S2. Determine the event observation time based on the basic metadata, segment and cut the monitoring video stream according to the event observation time, extract the multimodal features of each monitoring video segment to identify scene attributes, and construct a question-answering mapping model based on the labeled questions, scene attributes, and event categories of historical monitoring video segments; S3. Input the monitoring video segment to be labeled into the question-and-answer mapping model to obtain the labeling question, generate a JSON format labeling template based on the labeling question, and associate the monitoring video segment to be labeled with the labeling template; S4. Load the monitoring video segment to be annotated into the annotation software and call the corresponding annotation template. Use a pre-trained video understanding model to pre-annotate on the annotation template. Generate annotation text through human-computer collaborative annotation based on the pre-annotation results and quality assessment results. S5. Store the annotation text in a preset file directory, write the basic metadata and annotation text storage path of the same monitoring video segment into a database table, and perform association retrieval of monitoring video segments and annotation text through the database table; The basic metadata includes the event time, highway name, mileage marker, direction of travel, camera ID, lane location, and event category; The annotation questions cover event description, vehicle information, location information, environmental impact, and traffic flow status, and are assigned unique identifiers and required attributes.

2. The method for annotating highway event videos based on structured multi-turn question answering as described in claim 1, characterized in that, The method for video enhancement based on quality assessment results includes: The original highway surveillance video stream is acquired, and the quality of continuous surveillance video segments is evaluated. The quality evaluation indicators include basic image quality, lighting conditions, motion blur, degree of occlusion, and scene complexity. Invalid surveillance video segments with any quality assessment index exceeding the invalid threshold are removed. All surveillance video segments with quality assessment indexes exceeding the corresponding enhancement threshold are directly included in the annotation queue. Conversely, the remaining surveillance video segments are enhanced and then included in the annotation queue. The video enhancement operations include image equalization, deblurring, local repair, and scene normalization. The surveillance video segments of the queue to be labeled are de-identified and then reassembled into a surveillance video stream according to the time sequence.

3. The method for annotating highway event videos based on structured multi-turn question answering according to claim 1, characterized in that, The method for segmenting and trimming the surveillance video stream includes: The event observation time is determined based on the basic metadata, and the expression is: ； in for Event observation time The time scale constant, For standard time intervals, The time of the event With peak traffic hours The absolute value of the time deviation The sensitivity coefficient for road grade. for The deviation between the road grade where the incident occurred and the standard road grade. The sensitivity coefficient for the number of lanes. for The deviation between the number of lanes where the incident occurred and the standard number of lanes. This is the camera resolution sensitivity coefficient. The deviation between the camera resolution and the standard camera resolution. for Event Categories The corresponding baseline observation time; The surveillance video stream was segmented and cut according to the event observation time.

4. The method for annotating highway event videos based on structured multi-turn question answering according to claim 1, characterized in that, The method for constructing the question-answering mapping model includes: The video images from the surveillance video segment are extracted and image features are obtained through image detection. Based on the image features, the main scene attributes and corresponding confidence scores are calculated. The main scene attributes include weather, time of day, and traffic density. The video and audio of the surveillance video segment are extracted for audio detection to determine the auxiliary scene attributes and corresponding confidence levels; the auxiliary scene attributes include weather and event clues. Extract video text from surveillance video segments, match and correct basic metadata; The scene attributes are obtained by weighted fusion of the main scene attributes and the auxiliary scene attributes; The historical surveillance video segments are labeled with question sequences, scene attributes, and event categories to form a comprehensive set. The comprehensive set is then randomly divided into a training set and a test set in a 6:4 ratio. The question-answering mapping model is trained using the training set and evaluated using the test set. The question-answering mapping model includes an input layer and an embedding layer, a feature fusion and interaction layer, a multi-label prediction layer, and an output layer. The input and embedding layers concatenate the input scene attributes and event categories into a long vector, and then embed and increase the dimensionality of this long vector using a fully connected layer. The feature fusion and interaction layer projects the long vector into a key vector and a value vector, uses the event category embedding vector as the query vector, calculates attention weights using a cross-modal attention module, and fuses to generate event-related scene features. The multi-label prediction layer performs non-linear transformations on the event-related scene features through three fully connected layers, and predicts the probability of all labeled questions being recommended using a sigmoid activation function, outputting a labeled question recommendation probability vector. The output layer connects to the multi-label prediction layer through a fully connected layer, extracts labeled questions from the labeled question recommendation probability vector that are greater than a probability threshold, and outputs a sequence of labeled questions. The question-answering mapping model uses the Adam optimizer to adjust the model's hyperparameters; the question-answering mapping model uses binary cross-entropy loss with class weights to improve the model's accuracy in predicting labeled questions, as expressed in: ； in For binary cross-entropy loss, For the sample size, for The weight of the labeling question, for Label the problem with the true label.

5. The method for annotating highway event videos based on structured multi-turn question answering according to claim 1, characterized in that, The method for generating labeled text through human-computer collaborative annotation includes: The process involves processing the surveillance video segments to be labeled to obtain scene attributes and event categories, inputting them into a question-and-answer mapping model to obtain a sequence of labeling questions, generating a JSON-formatted labeling template, and then associating the surveillance video segments to be labeled with the labeling template. The monitoring video segment to be annotated is loaded into the annotation software and the corresponding annotation template is called. A pre-trained video understanding model is used to pre-annotate the annotation template; the pre-annotation is used for the annotator's reference. When the scene complexity is less than the complexity threshold, the pre-annotated results and the monitoring video segments to be annotated are randomly assigned to one annotator for manual annotation to generate the annotation results; otherwise, the pre-annotated results and the monitoring video segments to be annotated are randomly assigned to two annotators for manual annotation. The two annotation results are compared. If there is no difference, the annotation result is generated directly. If there is a difference, the system automatically merges the difference items and submits them to experts for final review to generate the annotation result. The annotation results are encapsulated into a structured JSON object, generating annotation text that uniquely corresponds to the video file.

6. A highway event video annotation system based on structured multi-turn question answering, used to perform the method described in any one of claims 1-5, characterized in that, include: Video processing module: used to perform quality assessment on the surveillance video stream, enhance the video based on the quality assessment results, determine the event observation time based on the basic metadata, and cut the surveillance video stream into segments according to the event observation time; Standard template generation module: used to extract multimodal features of each monitoring video segment to identify scene attributes, build a question-and-answer mapping model based on the annotation questions, scene attributes and event categories of historical monitoring video segments, input the monitoring video segment to be annotated into the question-and-answer mapping model to obtain the annotation questions, and generate a JSON format annotation template based on the annotation questions; Human-computer collaborative annotation module: Load the monitoring video segment to be annotated into the annotation software and call the corresponding annotation template. Use a pre-trained video understanding model to pre-annotate on the annotation template. Generate annotation text through human-computer collaborative annotation based on the pre-annotation results and quality assessment results. Data module: It is used to encapsulate the annotation results into a structured JSON object, generate annotation text that uniquely corresponds to the video file, store the annotation text in a preset file directory, write the basic metadata and annotation text storage path of the same monitoring video segment into a database table, and perform association retrieval of monitoring video segments and annotation text through the database table.

Images