Traffic accident video analysis method based on multi-task learning network

Abstract

The invention discloses a traffic accident video analysis method based on a multi-task learning network. The method comprises the following steps: 1) acquiring and preprocessing traffic accident video data; 2) constructing a multi-task learning network comprising four core modules: extracting visual and text features from a video by a feature extraction module, and generating a distinguishing text prompt; the accident analysis module analyzes accident causes based on the extracted features and provides a responsibility determination basis; the context awareness module evaluates an accident environment including weather and road conditions; and the accident report generation module integrates the results of the modules and automatically generates a detailed accident report. By efficiently and automatically analyzing the video of the automobile data recorder, the accident related information can be quickly extracted, the accident reason can be accurately judged, an efficient solution is provided for accident evidence collection and analysis, and the accuracy and efficiency of traffic accident video scene analysis are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a traffic accident video analysis method based on a multi-task learning network. Background Art

[0002] Existing research mainly focuses on areas such as video analysis, behavior recognition and scene reconstruction, but these methods have limitations: they usually only focus on local factors of the accident, such as the accident background, driver behavior or scene elements, and fail to fully integrate multiple complex factors, resulting in the inability to accurately reflect the complete cause of the accident.

[0003] In order to solve this problem, the present invention proposes a traffic accident video analysis method based on a multi-task learning network. This method improves the accuracy and real-time performance of accident analysis by comprehensively considering multiple factors. Specifically, the method includes two parts: data collection and preprocessing, and multi-task learning network construction. The network contains four core modules: 1) The feature extraction module combines the CLIP network, cross-frame communication converter, multi-frame integration converter and prompt generator to extract multi-level features such as scenes, objects, and dynamics from the video to ensure that the accident-related information is effectively captured; 2) The accident analysis module conducts a comprehensive analysis through multi-level features (such as traffic signs, driving trajectories, etc.), which can accurately identify the time of the accident and analyze the cause of the accident to provide support for responsibility determination; 3) The situational awareness module automatically evaluates environmental variables such as weather and road conditions based on visual data, thereby enhancing the adaptability and accuracy of the model to accidents in complex environments based on the extracted features; 4) The accident report generation module automatically generates a detailed accident report based on the output of the first two modules. Through this comprehensive method, the dashcam video can be efficiently and automatically analyzed, the accident information can be accurately extracted, the cause of the accident can be identified, and a more efficient solution can be provided for accident evidence collection and analysis. Summary of the invention

[0004] In order to solve the above-mentioned shortcomings, the present invention proposes a traffic accident video analysis method based on a multi-task learning network, aiming to improve the efficiency and accuracy of traffic accident handling, thereby meeting the current urgent need for efficient traffic accident analysis.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0006] The traffic accident video analysis method based on a multi-task learning network of the present invention is characterized in that it comprises the following steps:

[0007] Step 1: Collection and preprocessing of traffic accident videos:

[0008] After collecting real traffic accident video data and filtering and cleaning it, the dimension is Traffic accident video ,in, and Respectively The number of frames, height, width and channels;

[0009] definition Text description of the incident ),in, Indicates scene information, Indicates road conditions, Indicates weather conditions, Indicates traffic signals, represents the trajectory of the vehicle, Indicates its vehicle trajectory, Indicate the cause of the accident. Indicates the collision type;

[0010] Step 2: Construct a multi-task learning network, including: feature extraction module, accident analysis module, situation awareness module and accident report generation module;

[0011] Step 2.1, the feature extraction module includes: video encoder , Text Encoder , Prompt Generator :

[0012] Step 2.1.1: Input to video encoder Processed in, the video feature vector representation is obtained ;

[0013] Step 2.1.2: Input to the text encoder Processed in, get the text feature vector representation ;

[0014] Step 2.1.3: The prompt generator right and Processing to obtain visual text representation ;

[0015] Step 2.2, the accident analysis module includes: a mean calculation layer, a feature splicing layer and an accident analysis classification head;

[0016] Step 2.2.1. Input to the mean calculation layer for processing to obtain the mean of visual text representation ;

[0017] Step 2.2.2: and After inputting into the feature concatenation layer and concatenating along the feature dimension, the concatenated feature vector is obtained. ;

[0018] Step 2.2.3: Input into the accident analysis classification head for processing to obtain the predicted vehicle trajectory , predict the trajectory of other vehicles , predict the cause of the accident and predict collision type ;

[0019] Step 2.3: The situational awareness module Perform classification processing to obtain predicted scene information , predict road conditions , forecast weather conditions and predictive traffic signals ;

[0020] Step 2.4, the accident report generation module includes: a prediction label mapping layer and an accident report template filling layer;

[0021] Step 2.4.1. and Input to the prediction label mapping layer for processing to obtain the accident text description prediction vector ;

[0022] Step 2.4.2: Input into the accident report template filling layer for processing to obtain the accident prediction report .

[0023] The traffic accident video analysis method based on multi-task learning network of the present invention is also characterized in that the video encoder in step 2.1.1 By CLIP network, It consists of a layer cross-frame communication converter and a multi-frame integration converter;

[0024] Step 2.1.1.1, Input into the CLIP network and convert the Frame image is divided into non-overlapping image patches ,in, Indicates The first image blocks, Indicates the length of the image block; ; ;

[0025] Step 2.1.1.2: The CLIP network uses a linear projection layer to Image blocks Process it and get High-dimensional projection vector ; Then, using formula (1), we can get Initial frame mark of frame image :

[0026] (1)

[0027] In formula (1), Indicates the category tag, Represents spatial position encoding;

[0028] Step 2.1.1.3, Layer cross-frame communication converter pair Process it and get Global spatiotemporal feature labeling of frames ;

[0029] Step 2.1.1.3.1. Initialization ;Will As the input of the 0th layer, we get Layer cross-frame communication converter output Frame markers ;

[0030] Step 2.1.1.3.2: Use linear layers to Process it and get Layer 3 Cross-Frame Communication Converter Message tag of the frame , and then use formula (2) to get the The updated first layer cross-frame communication converter outputs Message tag of the frame ;

[0031] (2)

[0032] In formula (2), represents the cross-frame fusion attention layer, Representation layer normalization layer;

[0033] Step 2.1.1.3.3: Use formula (3) to get The updated first layer cross-frame communication converter outputs Combination mark of frame ;

[0034] (3)

[0035] In formula (3), represents the intra-frame diffuse attention layer, Indicates Layer cross-frame communication converter output frame tag for the frame;

[0036] Step 2.1.1.3.4: Use formula (4) to get The layer cross-frame communication converter outputs the updated Frame markers :

[0037] (4)

[0038] In formula (4), represents the first feed-forward network;

[0039] Step 2.1.1.3.5. +1 assigned to , then return to step 2.1.1.3.1 and execute sequentially until > Stop, and get the The layer cross-frame communication converter outputs the updated Frame markers , recorded as Global spatiotemporal feature labeling of frames ; Then we get the global spatiotemporal feature marker sequence ;

[0040] Step 2.1.1.4: The multi-frame integrated converter uses equation (5) to obtain the video level representation :

[0041] (5)

[0042] In formula (5), represents a multi-frame integrated converter, represents the average pooling layer, Represents the time position code.

[0043] Furthermore, the text encoder in step 2.1.2 It consists of a preprocessing layer, an embedding layer, a position encoding layer, a Transformer encoder, a normalization layer, and a linear layer:

[0044] Step 2.1.2.1, the pre-processing layer Text segmentation and tokenization to obtain a text tag sequence ;

[0045] Step 2.1.2.2, the embedding layer will Each text token in is mapped to a high-dimensional vector space to obtain a token vector sequence ;

[0046] Step 2.1.2.3, the position encoding layer will Combined with its corresponding position information, a label vector sequence with position information is obtained ;

[0047] Step 2.1.2.4, the Transformer encoder Processing is performed to obtain the encoded tag vector sequence ;

[0048] Step 2.1.2.5, the normalization layer Perform normalization to obtain a normalized tag vector sequence ;

[0049] Step 2.1.2.6, the linear layer will Projected into the multimodal embedding space to obtain the text feature vector representation .

[0050] Furthermore, the prompt generator in step 2.1.3 It consists of an average calculation layer, a multi-head self-attention layer, and a second feed-forward network;

[0051] Step 2.1.3.1: The average calculation layer uses formula (6) to obtain the global spatiotemporal feature marker sequence The average :

[0052] (6)

[0053] Step 2.1.3.2: The multi-head self-attention layer uses formula (7) to obtain the enhanced text representation :

[0054] (7)

[0055] In formula (7), represents a multi-head self-attention layer;

[0056] Step 2.1.3.3: The second feed-forward network uses equation (8) to obtain a further refined and enhanced text representation. , and then use formula (9) to get the visual text representation :

[0057] (8)

[0058] (9)

[0059] In formula (8), represents the second feedforward network; in formula (9), is a parameter to be learned.

[0060] Furthermore, the accident analysis classification head in step 2.2.3 is a multi-label multi-classification head, and the predicted vehicle trajectory is obtained using formula (10): , predict the trajectory of other vehicles , predict the cause of the accident and predict collision type :

[0061] (10)

[0062] In formula (10), and are the weights of the own vehicle trajectory, other vehicle trajectories, accident causes and collision types, respectively. , , and They are the corresponding 4 bias terms, Represents the softmax activation function.

[0063] Furthermore, the context-aware classification head in step 2.3 is a multi-label multi-classification head, and the predicted scene information is obtained using formula (11): , predict road conditions , forecast weather conditions and predictive traffic signals ;

[0064] (11)

[0065] In formula (11), , , and are the weights of scene information, road conditions, weather conditions, and traffic signals, respectively. , , and They are the corresponding four bias terms respectively.

[0066] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the traffic accident video analysis method, and the processor is configured to execute the program stored in the memory.

[0067] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the traffic accident video analysis method when the computer program is executed by a processor.

[0068] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0069] 1. The present invention has enhanced feature extraction capabilities: by combining CLIP networks, cross-frame communication converters, multi-frame integration converters and prompt generator layers, the model's multi-level feature extraction capabilities in traffic accident videos are significantly improved. The method can also automatically generate highly discriminative text prompts based on the video content, thereby enhancing the model's generalization capabilities for different accident types in complex open environments.

[0070] 2. The present invention has a multi-label classification output function: the accident analysis module and the accident scenario perception module adopt a multi-head classification design, which can process multiple label tasks at the same time. This feature enables the system to accurately identify and classify accident causes, collision types and environmental factors, providing a reliable data basis for subsequent analysis.

[0071] 3. The present invention has a natural language report generation function: the accident report generation module integrates the output information from the analysis module and the situation awareness module into a natural language description to provide a detailed and clear text accident report for the traffic accident. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A multi-task learning network diagram of the present invention;

[0073] Figure 2 This is a structural diagram of the feature extraction module of the present invention;

[0074] Figure 3 This is the cross-frame attention structure diagram of the present invention. DETAILED DESCRIPTION

[0075] In this embodiment, a traffic accident video analysis method based on a multi-task learning network focuses on the automated analysis of the video recorded by the driving recorder. This method can quickly and accurately extract key accident-related information and clarify the cause and details of the accident. The application of this method aims to make full use of the traffic accident information contained in the video sequence and enhance the model's ability to analyze the accident scene. Specifically, the method includes the following steps:

[0076] Step 1. Collection and preprocessing of traffic accident videos: Collect traffic accident videos from the Internet using keywords such as "traffic accident", "vehicle collision", "car accident", and "rear-end collision". Screen and filter out blurry videos, videos with unclear accident process, videos with too short duration, and other vehicle accident videos not from the perspective of the dashcam. For a single video containing multiple accidents, divide it into multiple independent video clips. The collected data covers a variety of scenes with different times, vehicle models, road conditions, weather environments, and accident types. The video length is limited to 10-15 seconds, and the dimension is Traffic accident video ,in , , and Respectively represent the number of frames, height, width and number of channels of the video.

[0077] In order to support comprehensive traffic accident analysis, multiple text description labels are added to each accident video. The label information includes eight categories: scene information, road conditions, weather conditions, traffic signals, vehicle trajectories, accident causes, and collision types (see Table 1 for specific label definitions). The videos in the video are annotated with text, and the video content is abstracted into 8 corresponding tags to form a dataset Collection of tags ) ,in, Indicates scene information, Indicates road conditions, Indicates weather conditions, Indicates traffic signals, represents the trajectory of the vehicle, Indicates its vehicle trajectory, Indicate the cause of the accident. Indicates the collision type;

[0078] Table 1 Traffic accident dataset annotation labels

[0079]

[0080] Step 2: Build a multi-task learning network, such as Figure 1 As shown, it includes: a feature extraction module, an accident analysis module, a situation perception module and an accident report generation module;

[0081] Step 2.1, the feature extraction module includes, Figure 2 As shown, including: video encoder , Text Encoder , Prompt Generator :

[0082] Step 2.1.1. Video encoder By CLIP network, It consists of a layer cross-frame communication converter and a multi-frame integrated converter; Input to video encoder Processed in, the video feature vector representation is obtained .

[0083] Step 2.1.1.1. Input to the feature extraction network CLIP, The frame is divided into non-overlapping blocks , the size of each block is Pixels, where Represents the time index, ;

[0084] Step 2.1.1.2: CLIP network uses the linear projection layer to Image blocks Process and label the learnable categories Added to the initial frame sequence marker, called Mark, its state at the output of the encoder is used as the frame representation, and the output is represented as Frame image The high-dimensional vector obtained by linearly projecting the image blocks ; Then, using formula (1), we can get Initial frame mark of frame image :

[0085] (1)

[0086] In formula (1), Indicates the category tag, Represents spatial position encoding.

[0087] Step 2.1.1.3, if Figure 3 As shown, Input to The cross-frame communication converter layer is processed in the cross-frame communication converter. The cross-frame communication converter layer consists of cross-frame fusion attention, intra-frame diffusion attention and the first feed-forward network, and introduces a message tagging mechanism for each frame to abstract, send and receive information, thereby realizing the cross-frame exchange of visual information and finally obtaining the global spatiotemporal feature tag. ;

[0088] Step 2.1.1.3.1. Initialization ;Will As the The input of the layer is Layer cross-frame communication converter output Frame markers ;

[0089] Step 2.1.1.3.2: Use linear layers to Process it and get Layer 3 Cross-Frame Communication Converter Message tag of the frame , the cross-frame fusion attention uses all message tags to learn the global spatiotemporal dependencies of the input video, and then uses formula (2) to obtain The updated first layer cross-frame communication converter outputs Message tag of the frame ;

[0090] (2)

[0091] In formula (2), represents the cross-frame fusion attention layer, Representation layer normalization layer.

[0092] Step 2.1.1.3.3: Intra-frame diffuse attention combines the frame tag with the corresponding message tag to learn the visual representation. At the same time, the message tags involved can also diffuse the global spatiotemporal dependency to assist learning. Using formula (3), we get The updated first layer cross-frame communication converter outputs Combination mark of frame ;

[0093] (3)

[0094] In formula (3), represents the intra-frame diffuse attention layer, Indicates The updated first layer cross-frame communication converter outputs frame tag for the frame;

[0095] Step 2.1.1.3.4, the message tag is discarded before entering the first feedforward network layer and will not be passed to the next block, because it is generated online in each block and used for inter-frame communication. The first feedforward network performs operations on the frame tag and obtains the first The layer cross-frame communication converter outputs the updated Frame markers :

[0096] (4)

[0097] In formula (4), represents the first feed-forward network.

[0098] Step 2.1.1.3.5. +1 assigned to , then return to step 2.1.1.3.1 and execute sequentially until > Stop, and get the The layer cross-frame communication converter outputs the updated Frame markers , recorded as Global spatiotemporal feature labeling of frames ; Then we get the global spatiotemporal feature marker sequence .

[0099] Step 2.1.1.4, by The fusion and diffusion attention are alternately performed in the layer. The cross-frame communication transformer can encode the global spatial and temporal information of the video frame. The multi-frame integrated transformer uses formula (5) to obtain the video level representation :

[0100] (5)

[0101] In formula (5), represents a multi-frame integrated converter, represents the average pooling layer, Represents the time position code.

[0102] Step 2.1.2. Text Encoder It consists of a preprocessing layer, an embedding layer, a position encoding layer, a Transformer encoder, a normalization layer, and a linear layer. Input to the text encoder Processed in, get the text feature vector representation .

[0103] Step 2.1.2.1: Preprocessing layer pair Text segmentation and tokenization to obtain text tag sequences ;

[0104] Step 2.1.2.2, embedding layer Each text token in is mapped to a high-dimensional vector space to obtain a token vector sequence ;

[0105] Step 2.1.2.3, the position encoding layer will Combined with its corresponding position information, a label vector sequence with position information is obtained ;

[0106] Step 2.1.2.4, Transformer encoder pair Processing is performed to obtain the encoded tag vector sequence ;

[0107] Step 2.1.2.5, Normalize Layer Pairs Perform normalization to obtain a normalized tag vector sequence ;

[0108] Step 2.1.2.6, the linear layer will Projected into the multimodal embedding space to obtain the text feature vector representation .

[0109] Step 2.1.3. Prompt Generator It consists of an average calculation layer, a multi-head self-attention layer, and a second feed-forward network, which aims to enhance the text representation through video content and prompt the generator right and Processing to obtain visual text representation ;

[0110] Step 2.1.3.1: The average calculation layer uses formula (6) to obtain the global spatiotemporal feature label sequence The average :

[0111] (6)

[0112] Step 2.1.3.2: and Input to the multi-head self-attention layer, using text representation As a query, As keys and values, the text representation extracts relevant visual features from the vision, and the multi-head self-attention layer uses formula (7) to obtain the enhanced text representation :

[0113] (7)

[0114] In formula (7), represents a multi-head self-attention layer.

[0115] Step 2.1.3.3: The second feed-forward network uses equation (8) to obtain a further refined and enhanced text representation. , and then use formula (9) to get the visual text representation :

[0116] (8)

[0117]

[0118] In formula (8), represents the second feed-forward network;

[0119] In formula (9), is a parameter to be learned and is set to 0.1 during training.

[0120] Step 2.2, the accident analysis module includes: mean calculation layer, feature concatenation layer and accident analysis classification head;

[0121] Step 2.2.1. Input to the mean calculation layer for processing to obtain the mean of visual text representation ;

[0122] Step 2.2.2: and After inputting into the feature concatenation layer and concatenating along the feature dimension, the concatenated feature vector is obtained. .

[0123] Step 2.2.3: Input into the accident analysis classification head for processing, and use formula (10) to get the predicted vehicle trajectory , predict the trajectory of other vehicles , predict the cause of the accident and predict collision type ;

[0124] (10)

[0125] In formula (10), and They are the weights of the vehicle trajectory, other vehicle trajectories, accident causes and collision types, and the dimensions are ( ) ( )、( )and( ), , , and are the number of categories corresponding to the classification task, and they are all of dimension ( ) part of is the sum of the number of all categories, , , and are the corresponding bias terms, Represents the activation function softmax function.

[0126] Step 2.3: Context awareness module Perform classification processing and use formula (11) to obtain the predicted scene information , predict road conditions , forecast weather conditions and predictive traffic signals ;

[0127] (11)

[0128] In formula (11), , , and are the weights of scene information, road conditions, weather conditions and traffic signals, respectively, and the dimensions are ( ) ( )、( )and( ), , , and are the number of categories corresponding to the classification task, , , and They are the corresponding four bias terms respectively.

[0129] Step 2.4, the accident report generation module includes: a prediction label mapping layer and an accident report template filling layer;

[0130] Step 2.4.1. and Input into the prediction label mapping layer for processing, and use formula (12) to obtain the accident text description prediction vector ;

[0131] (12)

[0132] In formula (12), Represents a mapping function.

[0133] Step 2.4.2: Input into the accident report template filling layer for processing to obtain the accident prediction report :

[0134] This happened together Traffic accidents, when the accident occurred, the weather was , the traffic signal is , since the car , it's in According to the analysis, the preliminary judgment is that , resulting in ACCIDENT.

[0135] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0136] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. A traffic accident video analysis method based on a multi-task learning network, characterized in that: The steps include: Step 1: Collection and preprocessing of traffic accident videos: After collecting real traffic accident video data and screening and cleaning it, the dimension is Traffic accident video ,in, and Respectively The number of frames, height, width and channels; definition Text description of the incident ),in, Indicates scene information, Indicates road conditions, Indicates weather conditions, Indicates traffic signals, represents the trajectory of the vehicle, Indicates its vehicle trajectory, Indicate the cause of the accident. Indicates the collision type; Step 2: Construct a multi-task learning network, including: feature extraction module, accident analysis module, situation awareness module and accident report generation module; Step 2.1, the feature extraction module includes: video encoder , Text Encoder , Prompt Generator : Step 2.1.1: Input to video encoder Processed in, the video feature vector representation is obtained ; Step 2.1.2: Input to the text encoder Processed in, get the text feature vector representation ; Step 2.1.3: The prompt generator right and Processing to obtain visual text representation ; Step 2.2, the accident analysis module includes: a mean calculation layer, a feature splicing layer and an accident analysis classification head; Step 2.2.

1. Input to the mean calculation layer for processing to obtain the mean of visual text representation ; Step 2.2.2: and After inputting into the feature concatenation layer and concatenating along the feature dimension, the concatenated feature vector is obtained. ; Step 2.2.3: Input into the accident analysis classification head for processing to obtain the predicted vehicle trajectory , predict the trajectory of other vehicles , predict the cause of the accident and predict collision type ; Step 2.3: The situational awareness module Perform classification processing to obtain predicted scene information , predict road conditions , forecast weather conditions and predictive traffic signals ; Step 2.4, the accident report generation module includes: a prediction label mapping layer and an accident report template filling layer; Step 2.4.

1. and Input to the prediction label mapping layer for processing to obtain the accident text description prediction vector ; Step 2.4.2: Input into the accident report template filling layer for processing to obtain the accident prediction report .

2. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The video encoder in step 2.1.1 By CLIP network, It consists of a layer cross-frame communication converter and a multi-frame integration converter; Step 2.1.1.1, Input into the CLIP network and convert the Frame image is divided into non-overlapping image patches ,in, Indicates The first image blocks, Indicates the length of the image block; ; ; Step 2.1.1.2: The CLIP network uses a linear projection layer to Image blocks Process it and get High-dimensional projection vector ; Then, using formula (1), we can get Initial frame mark of frame image : (1) In formula (1), Indicates the category tag, Represents spatial position encoding; Step 2.1.1.3, Layer cross-frame communication converter pair Process it and get Global spatiotemporal feature labeling of frames ; Step 2.1.1.3.

1. Initialization ;Will As the input of the 0th layer, we get Layer cross-frame communication converter output Frame markers ; Step 2.1.1.3.2: Use linear layers to Process it and get Layer 3 Cross-Frame Communication Converter Message tag of the frame , and then use formula (2) to get the The updated first layer cross-frame communication converter outputs Message tag of the frame ; (2) In formula (2), represents the cross-frame fusion attention layer, Representation layer normalization layer; Step 2.1.1.3.3: Use formula (3) to get The updated first layer cross-frame communication converter outputs Combination mark of frame ; (3) In formula (3), represents the intra-frame diffuse attention layer, Indicates Layer cross-frame communication converter output frame tag for the frame; Step 2.1.1.3.4: Use formula (4) to get The layer cross-frame communication converter outputs the updated Frame markers : (4) In formula (4), represents the first feed-forward network; Step 2.1.1.3.

5. +1 assigned to , then return to step 2.1.1.3.1 and execute sequentially until > Stop, and get the The layer cross-frame communication converter outputs the updated Frame markers , recorded as Global spatiotemporal feature labeling of frames ; Then we get the global spatiotemporal feature marker sequence ; Step 2.1.1.4: The multi-frame integrated converter uses equation (5) to obtain the video level representation : (5) In formula (5), represents a multi-frame integrated converter, represents the average pooling layer, Represents the time position code.

3. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The text encoder in step 2.1.2 It consists of a preprocessing layer, an embedding layer, a position encoding layer, a Transformer encoder, a normalization layer, and a linear layer: Step 2.1.2.1, the pre-processing layer Text segmentation and tokenization to obtain a text tag sequence ; Step 2.1.2.2, the embedding layer will Each text token in is mapped to a high-dimensional vector space to obtain a token vector sequence ; Step 2.1.2.3, the position encoding layer will Combined with its corresponding position information, a label vector sequence with position information is obtained ; Step 2.1.2.4, the Transformer encoder Processing is performed to obtain the encoded tag vector sequence ; Step 2.1.2.5, the normalization layer Perform normalization to obtain a normalized tag vector sequence ; Step 2.1.2.6, the linear layer will Projected into the multimodal embedding space to obtain the text feature vector representation .

4. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The prompt generator in step 2.1.3 It consists of an average calculation layer, a multi-head self-attention layer, and a second feed-forward network; Step 2.1.3.1: The average calculation layer uses formula (6) to obtain the global spatiotemporal feature marker sequence The average : (6) Step 2.1.3.2: The multi-head self-attention layer uses formula (7) to obtain the enhanced text representation : (7) In formula (7), represents a multi-head self-attention layer; Step 2.1.3.3: The second feed-forward network uses equation (8) to obtain a further refined and enhanced text representation. , and then use formula (9) to get the visual text representation : (8) (9) In formula (8), represents the second feedforward network; in formula (9), is a parameter to be learned.

5. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The accident analysis classification head in step 2.2.3 is a multi-label multi-classification head, and the predicted vehicle trajectory is obtained using formula (10): , predict the trajectory of other vehicles , predict the cause of the accident and predict collision type : (10) In formula (10), and are the weights of the own vehicle trajectory, other vehicle trajectories, accident causes and collision types, respectively. , , and They are the corresponding 4 bias terms, Represents the softmax activation function.

6. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The context-aware classification head in step 2.3 is a multi-label multi-classification head, which uses formula (11) to obtain the predicted scene information , predict road conditions , forecast weather conditions and predictive traffic signals ; (11) In formula (11), , , and are the weights of scene information, road conditions, weather conditions, and traffic signals, respectively. , , and They are the corresponding 4 bias terms respectively.

7. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the traffic accident video analysis method described in any one of claims 1 to 6, and the processor is configured to execute the program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the traffic accident video analysis method according to any one of claims 1 to 6 are executed.

Images