A traffic accident video analysis method based on multi-task learning network

Abstract

The present invention discloses a traffic accident video analysis method based on a multi-task learning network, comprising: 1) acquiring and preprocessing traffic accident video data; 2) constructing a multi-task learning network comprising four core modules: a feature extraction module extracts visual and textual features from the video to generate discriminative text prompts; an accident analysis module analyzes the cause of the accident based on the extracted features and provides a basis for determining responsibility; a situational awareness module assesses the accident environment, including weather and road conditions; and an accident report generation module integrates the results of each module to automatically generate a detailed accident report. By efficiently and automatically analyzing dashcam videos, the present invention can quickly extract accident-related information and accurately determine the cause of the accident, providing a highly efficient solution for accident evidence collection and analysis, significantly improving the accuracy and efficiency of traffic accident video scene analysis.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a traffic accident video analysis method based on a multi-task learning network. Background Art

[0002] Existing research mainly focuses on areas such as video analysis, behavior recognition and scene reconstruction, but these methods have limitations: they usually only focus on local factors of the accident, such as the accident background, driver behavior or scene elements, and fail to fully integrate multiple complex factors, resulting in an inability to accurately reflect the complete cause of the accident.

[0003] To address this issue, this paper proposes a traffic accident video analysis method based on a multi-task learning network. This method comprehensively considers multiple factors to improve the accuracy and real-time performance of accident analysis. Specifically, the method consists of two major components: data collection and preprocessing, and the construction of a multi-task learning network. The network comprises four core modules: 1) A feature extraction module, combining a CLIP network, a cross-frame communication transformer, a multi-frame integration transformer, and a prompt generator, extracts multi-level features from the video, such as scene, object, and dynamics, ensuring that relevant accident information is effectively captured. 2) The accident analysis module, using multi-level features (such as traffic signs and driving trajectories) for comprehensive analysis, accurately identifies the moment of the accident and analyzes the cause, supporting liability determination. 3) The situational awareness module, based on visual data, automatically assesses environmental variables such as weather and road conditions, enhancing the model's adaptability and accuracy to accidents in complex environments based on the extracted features. 4) The accident report generation module, based on the outputs of the first two modules, automatically generates detailed accident reports. This integrated approach enables efficient and automatic analysis of dashcam videos, accurately extracting accident information and identifying the cause, providing a more effective solution for accident evidence collection and analysis. Summary of the Invention

[0004] To address the above shortcomings, the present invention proposes a traffic accident video analysis method based on a multi-task learning network, aiming to improve the efficiency and accuracy of traffic accident handling, thereby meeting the current urgent need for efficient traffic accident analysis.

[0005] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0006] The traffic accident video analysis method based on a multi-task learning network of the present invention is characterized in that it includes the following steps:

[0007] Step 1: Collection and preprocessing of traffic accident videos:

[0008] After collecting real traffic accident video data and screening and cleaning it, the dimension is Traffic accident video ,in, and Respectively The number of frames, height, width and channels;

[0009] definition Text description of the incident ),in, Indicates scene information, Indicates road conditions, Indicates weather conditions, Indicates traffic signals, represents the vehicle trajectory, Indicates its vehicle trajectory, Indicate the cause of the accident, Indicates the collision type;

[0010] Step 2: Build a multi-task learning network, including: feature extraction module, accident analysis module, situational awareness module and accident report generation module;

[0011] Step 2.1: The feature extraction module includes: video encoder , text encoder , prompt generator :

[0012] Step 2.1.1, Input to video encoder Processed in, the video feature vector representation is obtained ;

[0013] Step 2.1.2, Input to the text encoder Processed in, get the text feature vector representation ;

[0014] Step 2.1.3: The prompt generator right and Processing to obtain visual text representation ;

[0015] Step 2.2, the accident analysis module includes: a mean calculation layer, a feature splicing layer and an accident analysis classification head;

[0016] Step 2.2.1, Input into the mean calculation layer for processing to obtain the mean of visual text representation ;

[0017] Step 2.2.2, and After inputting into the feature concatenation layer and concatenating along the feature dimension, the concatenated feature vector is obtained. ;

[0018] Step 2.2.3, Input into the accident analysis classification head for processing to obtain the predicted vehicle trajectory , predict the trajectory of other vehicles , predict the cause of the accident and predicted collision type ;

[0019] Step 2.3, the situational awareness module Perform classification processing to obtain predicted scene information , predict road conditions , forecast weather conditions and predictive traffic signals ;

[0020] Step 2.4, the accident report generation module includes: a prediction label mapping layer and an accident report template filling layer;

[0021] Step 2.4.1, and Input into the prediction label mapping layer for processing to obtain the accident text description prediction vector ;

[0022] Step 2.4.2, Input into the accident report template filling layer for processing to obtain the accident prediction report .

[0023] The traffic accident video analysis method based on multi-task learning network of the present invention is also characterized in that the video encoder in step 2.1.1 By CLIP Network, It consists of a layer cross-frame communication converter and a multi-frame integration converter;

[0024] Step 2.1.1.1, Input into the CLIP network and Frame image is divided into non-overlapping image patches ,in, Indicates the The first frame in the image image blocks, Indicates the length of the image block; ; ;

[0025] Step 2.1.1.2, the CLIP network uses the linear projection layer to Image blocks Process it and get High-dimensional projection vector ; Thus, using formula (1) we can get Initial frame mark of the frame image :

[0026] (1)

[0027] In formula (1), Indicates the category label, Represents spatial position encoding;

[0028] Step 2.1.1.3, Layer cross-frame communication converter pair Process it and get Global spatiotemporal feature labeling of frames ;

[0029] Step 2.1.1.3.1. Initialization ;Will As the input of the 0th layer, we get Layer cross-frame communication converter output Frame markers for frames ;

[0030] Step 2.1.1.3.2, using linear layer pair Process it and get Layer 3 cross-frame communication converter Message tag of the frame , and thus use formula (2) to get the The updated first layer cross-frame communication converter output Message tag of the frame ;

[0031] (2)

[0032] In formula (2), represents the cross-frame fusion attention layer, Representation layer normalization layer;

[0033] Step 2.1.1.3.3, use formula (3) to get The updated first layer cross-frame communication converter output Combination mark of frame ;

[0034] (3)

[0035] In formula (3), represents the intra-frame diffuse attention layer, Indicates the Layer cross-frame communication converter output Frame markers for frames;

[0036] Step 2.1.1.3.4: Use formula (4) to get The cross-frame communication converter outputs the updated Frame markers for frames :

[0037] (4)

[0038] In formula (4), represents the first feedforward network;

[0039] Step 2.1.1.3.5, +1 assigned to , then return to step 2.1.1.3.1 and execute sequentially until > Stop, and get the The cross-frame communication converter outputs the updated Frame markers for frames , recorded as Global spatiotemporal feature labeling of frames ; Then obtain the global spatiotemporal feature marker sequence ;

[0040] Step 2.1.1.4: The multi-frame integrated converter uses formula (5) to obtain the video level representation :

[0041] (5)

[0042] In formula (5), represents a multi-frame integrated converter, represents the average pooling layer, Represents time position coding.

[0043] Furthermore, the text encoder in step 2.1.2 It consists of a preprocessing layer, an embedding layer, a positional encoding layer, a Transformer encoder, a normalization layer, and a linear layer:

[0044] Step 2.1.2.1, the pre-processing layer Text segmentation and tokenization to obtain a text token sequence ;

[0045] Step 2.1.2.2, the embedding layer will Each text token in is mapped to a high-dimensional vector space to obtain a token vector sequence ;

[0046] Step 2.1.2.3, the position encoding layer will Combined with its corresponding position information, a tag vector sequence with position information is obtained ;

[0047] Step 2.1.2.4, the Transformer encoder pair Processing is performed to obtain the encoded tag vector sequence ;

[0048] Step 2.1.2.5, the normalization layer Perform normalization to obtain a normalized tag vector sequence ;

[0049] Step 2.1.2.6, the linear layer will Projected into the multimodal embedding space to obtain the text feature vector representation .

[0050] Furthermore, the prompt generator in step 2.1.3 It consists of an average calculation layer, a multi-head self-attention layer, and a second feedforward network;

[0051] Step 2.1.3.1: The average calculation layer uses formula (6) to obtain the global spatiotemporal feature marker sequence Average value :

[0052] (6)

[0053] Step 2.1.3.2: The multi-head self-attention layer uses formula (7) to obtain the enhanced text representation :

[0054] (7)

[0055] In formula (7), represents a multi-head self-attention layer;

[0056] Step 2.1.3.3: The second feedforward network uses formula (8) to obtain a further refined and enhanced text representation. , and then use formula (9) to get the visual text representation :

[0057] (8)

[0058] (9)

[0059] In formula (8), represents the second feedforward network; in formula (9), is a parameter to be learned.

[0060] Furthermore, the accident analysis classification head in step 2.2.3 is a multi-label multi-classification head, and the predicted vehicle trajectory is obtained using formula (10): , predict the trajectory of other vehicles , predict the cause of the accident and predicted collision type :

[0061] (10)

[0062] In formula (10), and are the weights of the vehicle trajectory, other vehicle trajectories, accident causes and collision types, respectively. 、 、 and They are the corresponding 4 bias terms, Represents the softmax activation function.

[0063] Furthermore, the context-aware classification head in step 2.3 is a multi-label multi-classification head, which uses formula (11) to obtain the predicted scene information , predict road conditions , forecast weather conditions and predictive traffic signals ;

[0064] (11)

[0065] In formula (11), 、 、 and are the weights of scene information, road conditions, weather conditions and traffic signals respectively, 、 、 and They are the corresponding 4 bias terms respectively.

[0066] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the traffic accident video analysis method, and the processor is configured to execute the program stored in the memory.

[0067] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the traffic accident video analysis method when the computer program is executed by a processor.

[0068] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0069] 1. Enhanced feature extraction: By combining a CLIP network, a cross-frame communication transformer, a multi-frame integration transformer, and a prompt generator layer, the model significantly improves its multi-level feature extraction capabilities in traffic accident videos. This method also automatically generates discriminative textual prompts based on the video content, thereby enhancing the model's generalization to different accident types in complex and open environments.

[0070] 2. This invention features multi-label classification output: The accident analysis module and accident scenario awareness module utilize a multi-head classification design, capable of processing multiple labeling tasks simultaneously. This feature enables the system to accurately identify and classify accident causes, collision types, and environmental factors, providing a reliable data foundation for subsequent analysis.

[0071] 3. The present invention has a natural language report generation function: the accident report generation module integrates the output information from the analysis module and the situational awareness module into a natural language description, providing a detailed and clear text accident report for the traffic accident. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a multi-task learning network diagram of the present invention;

[0073] Figure 2 This is a structural diagram of the feature extraction module of the present invention;

[0074] Figure 3 This is the cross-frame attention structure diagram of the present invention. DETAILED DESCRIPTION

[0075] In this embodiment, a traffic accident video analysis method based on a multi-task learning network focuses on automated analysis of videos recorded by a dashcam. This method can quickly and accurately extract key accident-related information, clarifying the cause and details of the accident. This method aims to fully utilize the traffic accident information contained in video sequences and enhance the model's ability to analyze accident scenes. Specifically, the method includes the following steps:

[0076] Step 1. Collection and preprocessing of traffic accident videos: Collect traffic accident videos from the Internet using keywords such as "traffic accident", "vehicle collision", "car accident", and "rear-end collision". Screen and filter out blurry videos, videos with unclear accident process, videos that are too short, and videos of other vehicle accidents that are not from the perspective of the dashcam. For a single video containing multiple accidents, split it into multiple independent video clips. The collected data covers a variety of scenes with different times, vehicle models, road conditions, weather environments, and accident types. The video length is limited to 10-15 seconds, and the dimension is Traffic accident video ,in 、 、 and Represents the number of frames, height, width and channels of the video respectively.

[0077] In order to support comprehensive traffic accident analysis, multiple text description labels are added to each accident video. The label information includes eight categories, including scene information, road conditions, weather conditions, traffic signals, vehicle trajectory, accident cause, and collision type (see Table 1 for specific label definitions). The video content is abstracted into 8 corresponding tags to form a dataset. Tag collection ) ,in, Indicates scene information, Indicates road conditions, Indicates weather conditions, Indicates traffic signals, represents the vehicle trajectory, Indicates its vehicle trajectory, Indicate the cause of the accident, Indicates the collision type;

[0078] Table 1 Traffic accident dataset annotation labels

[0079]

[0080] Step 2: Build a multi-task learning network, such as Figure 1 As shown, it includes: feature extraction module, accident analysis module, situation perception module and accident report generation module;

[0081] Step 2.1, the feature extraction module includes, Figure 2 As shown, including: video encoder , text encoder , prompt generator :

[0082] Step 2.1.1. Video Encoder By CLIP Network, Layer cross-frame communication converter and multi-frame integrated converter; Input to video encoder Processed in, the video feature vector representation is obtained .

[0083] Step 2.1.1.1, Input into the feature extraction network CLIP, The frame is divided into non-overlapping blocks , the size of each block is pixels, where Represents the time index, ;

[0084] Step 2.1.1.2, CLIP network uses linear projection layer to Image blocks Process and label the learnable categories Added to the initial frame sequence marker, called Mark, its state at the output of the encoder is used as the frame representation, and the output is represented as Frame image The high-dimensional vector obtained by linear projection of the image block ; Thus, using formula (1) we can get Initial frame mark of the frame image :

[0085] (1)

[0086] In formula (1), Indicates the category label, Represents spatial position encoding.

[0087] Step 2.1.1.3, as Figure 3 As shown, Input to The cross-frame communication converter layer is processed, which consists of cross-frame fusion attention, intra-frame diffusion attention and the first feedforward network. It also introduces a message tagging mechanism for each frame to abstract, send and receive information, thereby realizing cross-frame exchange of visual information and finally obtaining a global spatiotemporal feature tag. ;

[0088] Step 2.1.1.3.1. Initialization ;Will As the first The input of the layer is Layer cross-frame communication converter output Frame markers for frames ;

[0089] Step 2.1.1.3.2, using linear layer pair Process it and get Layer 3 cross-frame communication converter Message tag of the frame , cross-frame fusion attention utilizes all message tags to learn the global spatiotemporal dependencies of the input video, and thus uses formula (2) to obtain the first The updated first layer cross-frame communication converter output Message tag of the frame ;

[0090] (2)

[0091] In formula (2), represents the cross-frame fusion attention layer, Representation layer normalization layer.

[0092] Step 2.1.1.3.3: Intra-frame diffuse attention combines the frame tag with the corresponding message tag to learn the visual representation. At the same time, the message tags involved can also diffuse the global spatiotemporal dependency to assist learning. Using formula (3), we can get The updated first layer cross-frame communication converter output Combination mark of frame ;

[0093] (3)

[0094] In formula (3), represents the intra-frame diffuse attention layer, Indicates the The updated first layer cross-frame communication converter output Frame markers for frames;

[0095] Step 2.1.1.3.4, the message tag is discarded before entering the first feedforward network layer and will not be passed to the next block, because it is generated online in each block and used for inter-frame communication. The first feedforward network performs operations on the frame tag and uses formula (4) to obtain the first The cross-frame communication converter outputs the updated Frame markers for frames :

[0096] (4)

[0097] In formula (4), represents the first feed-forward network.

[0098] Step 2.1.1.3.5, +1 assigned to , then return to step 2.1.1.3.1 and execute sequentially until > Stop, and get the The cross-frame communication converter outputs the updated Frame markers for frames , recorded as Global spatiotemporal feature labeling of frames ; Then obtain the global spatiotemporal feature marker sequence .

[0099] Step 2.1.1.4, by The fusion and diffusion attention are alternately performed in the layer, the cross-frame communication transformer can encode the global spatial and temporal information of the video frame, and the multi-frame integrated transformer uses formula (5) to obtain the video level representation :

[0100] (5)

[0101] In formula (5), represents a multi-frame integrated converter, represents the average pooling layer, Represents time position coding.

[0102] Step 2.1.2, Text Encoder It consists of a preprocessing layer, an embedding layer, a position encoding layer, a Transformer encoder, a normalization layer, and a linear layer. Input to the text encoder Processed in, get the text feature vector representation .

[0103] Step 2.1.2.1, preprocessing layer pair Text segmentation and tokenization to obtain text tag sequences ;

[0104] Step 2.1.2.2, embedding layer will Each text token in is mapped to a high-dimensional vector space to obtain a token vector sequence ;

[0105] Step 2.1.2.3, the position encoding layer will Combined with its corresponding position information, a tag vector sequence with position information is obtained ;

[0106] Step 2.1.2.4, Transformer encoder pair Processing is performed to obtain the encoded tag vector sequence ;

[0107] Step 2.1.2.5, Normalize Layer Pairs Perform normalization to obtain a normalized tag vector sequence ;

[0108] Step 2.1.2.6, the linear layer will Projected into the multimodal embedding space to obtain the text feature vector representation .

[0109] Step 2.1.3. Prompt Generator It consists of an average calculation layer, a multi-head self-attention layer, and a second feed-forward network, which aims to enhance the text representation through video content and prompt the generator. right and Processing to obtain visual text representation ;

[0110] Step 2.1.3.1: The average calculation layer uses formula (6) to obtain the global spatiotemporal feature label sequence Average value :

[0111] (6)

[0112] Step 2.1.3.2, and Input to the multi-head self-attention layer, using text representation As a query, As keys and values, the text representation extracts relevant visual features from the vision, and the multi-head self-attention layer uses formula (7) to obtain the enhanced text representation :

[0113] (7)

[0114] In formula (7), represents a multi-head self-attention layer.

[0115] Step 2.1.3.3: The second feedforward network uses formula (8) to obtain a further refined and enhanced text representation. , and then use formula (9) to get the visual text representation :

[0116] (8)

[0117]

[0118] In formula (8), represents the second feedforward network;

[0119] In formula (9), is a parameter to be learned and is set to 0.1 during training.

[0120] Step 2.2, the accident analysis module includes: mean calculation layer, feature splicing layer and accident analysis classification head;

[0121] Step 2.2.1, Input into the mean calculation layer for processing to obtain the mean of visual text representation ;

[0122] Step 2.2.2, and After inputting into the feature concatenation layer and concatenating along the feature dimension, the concatenated feature vector is obtained. .

[0123] Step 2.2.3, Input into the accident analysis classification head for processing, and use formula (10) to obtain the predicted vehicle trajectory , predict the trajectory of other vehicles , predict the cause of the accident and predicted collision type ;

[0124] (10)

[0125] In formula (10), and They are the weights of the vehicle trajectory, other vehicle trajectories, accident causes and collision types, and the dimensions are ( ) ( )、( )and( ), 、 、 and are the number of categories corresponding to the classification task, and they are all of dimension ( ) part of is the sum of the number of all categories, 、 、 and are the corresponding bias terms, Represents the activation function softmax function.

[0126] Step 2.3: Context awareness module Perform classification processing and use formula (11) to obtain the predicted scene information , predict road conditions , forecast weather conditions and predictive traffic signals ;

[0127] (11)

[0128] In formula (11), 、 、 and are the weights of scene information, road conditions, weather conditions and traffic signals, respectively, and the dimensions are ( ) ( )、( )and( ), 、 、 and are the number of categories corresponding to the classification task, 、 、 and They are the corresponding 4 bias terms respectively.

[0129] Step 2.4, the accident report generation module includes: a prediction label mapping layer and an accident report template filling layer;

[0130] Step 2.4.1, and Input into the prediction label mapping layer for processing, and use formula (12) to obtain the accident text description prediction vector ;

[0131] (12)

[0132] In formula (12), Represents a mapping function.

[0133] Step 2.4.2, Input into the accident report template filling layer for processing to obtain the accident prediction report :

[0134] This happened together Traffic accidents, when the accident happened, the weather was , the traffic signal is , since the car , it's in According to the analysis, the preliminary judgment is that , resulting in ACCIDENT.

[0135] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0136] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

Claims

1. A traffic accident video analysis method based on a multi-task learning network, characterized in that: The steps include: Step 1: Collection and preprocessing of traffic accident videos: After collecting real traffic accident video data and screening and cleaning it, the dimension is Traffic accident video ,in, and Respectively The number of frames, height, width and channels; definition Text description of the incident ),in, Indicates scene information, Indicates road conditions, Indicates weather conditions, Indicates traffic signals, represents the vehicle trajectory, Indicates its vehicle trajectory, Indicate the cause of the accident, Indicates the collision type; Step 2: Build a multi-task learning network, including: feature extraction module, accident analysis module, situational awareness module and accident report generation module; Step 2.1: The feature extraction module includes: video encoder , text encoder , prompt generator : Step 2.1.1, Input to video encoder Processed in, the video feature vector representation is obtained ; Step 2.1.2, Input to the text encoder Processed in, get the text feature vector representation ; Step 2.1.3: The prompt generator right and Processing to obtain visual text representation ; Step 2.2, the accident analysis module includes: a mean calculation layer, a feature splicing layer and an accident analysis classification head; Step 2.2.1, Input into the mean calculation layer for processing to obtain the mean of visual text representation ; Step 2.2.2, and After inputting into the feature concatenation layer and concatenating along the feature dimension, the concatenated feature vector is obtained. ; Step 2.2.3, Input into the accident analysis classification head for processing to obtain the predicted vehicle trajectory , predict the trajectory of other vehicles , predict the cause of the accident and predicted collision type ; Step 2.3, the situational awareness module Perform classification processing to obtain predicted scene information , predict road conditions , forecast weather conditions and predictive traffic signals ; Step 2.4, the accident report generation module includes: a prediction label mapping layer and an accident report template filling layer; Step 2.4.1, and Input into the prediction label mapping layer for processing to obtain the accident text description prediction vector ; Step 2.4.2, Input into the accident report template filling layer for processing to obtain the accident prediction report .

2. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The video encoder in step 2.1.1 By CLIP Network, It consists of a layer cross-frame communication converter and a multi-frame integration converter; Step 2.1.1.1, Input into the CLIP network and Frame image is divided into non-overlapping image patches ,in, Indicates the The first frame in the image image blocks, Indicates the length of the image block; ; ; Step 2.1.1.2, the CLIP network uses the linear projection layer to Image blocks Process it and get High-dimensional projection vector ; Thus, using formula (1) we can get Initial frame mark of the frame image : (1) In formula (1), Indicates the category label, Represents spatial position encoding; Step 2.1.1.3, Layer cross-frame communication converter pair Process it and get Global spatiotemporal feature labeling of frames ; Step 2.1.1.3.

1. Initialization ;Will As the input of the 0th layer, we get Layer cross-frame communication converter output Frame markers for frames ; Step 2.1.1.3.2, using linear layer pair Process it and get Layer 3 cross-frame communication converter Message tag of the frame , and thus use formula (2) to get the The updated first layer cross-frame communication converter output Message tag of the frame ; (2) In formula (2), represents the cross-frame fusion attention layer, Representation layer normalization layer; Step 2.1.1.3.3, use formula (3) to get The updated first layer cross-frame communication converter output Combination mark of frame ; (3) In formula (3), represents the intra-frame diffuse attention layer, Indicates the Layer cross-frame communication converter output Frame markers for frames; Step 2.1.1.3.4: Use formula (4) to get The cross-frame communication converter outputs the updated Frame markers for frames : (4) In formula (4), represents the first feedforward network; Step 2.1.1.3.5, +1 assigned to , then return to step 2.1.1.3.1 and execute sequentially until > Stop, and get the The cross-frame communication converter outputs the updated Frame markers for frames , recorded as Global spatiotemporal feature labeling of frames ; Then obtain the global spatiotemporal feature marker sequence ; Step 2.1.1.4: The multi-frame integrated converter uses formula (5) to obtain the video level representation : (5) In formula (5), represents a multi-frame integrated converter, represents the average pooling layer, Represents time position coding.

3. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The text encoder in step 2.1.2 It consists of a preprocessing layer, an embedding layer, a positional encoding layer, a Transformer encoder, a normalization layer, and a linear layer: Step 2.1.2.1, the pre-processing layer Text segmentation and tokenization to obtain a text token sequence ; Step 2.1.2.2, the embedding layer will Each text token in is mapped to a high-dimensional vector space to obtain a token vector sequence ; Step 2.1.2.3, the position encoding layer will Combined with its corresponding position information, a tag vector sequence with position information is obtained ; Step 2.1.2.4, the Transformer encoder pair Processing is performed to obtain the encoded tag vector sequence ; Step 2.1.2.5, the normalization layer Perform normalization to obtain a normalized tag vector sequence ; Step 2.1.2.6, the linear layer will Projecting into the multimodal embedding space to obtain the text feature vector representation .

4. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The prompt generator in step 2.1.3 It consists of an average calculation layer, a multi-head self-attention layer, and a second feedforward network; Step 2.1.3.1: The average calculation layer uses formula (6) to obtain the global spatiotemporal feature marker sequence Average value : (6) Step 2.1.3.2: The multi-head self-attention layer uses formula (7) to obtain the enhanced text representation : (7) In formula (7), represents a multi-head self-attention layer; Step 2.1.3.3: The second feedforward network uses formula (8) to obtain a further refined and enhanced text representation. , and then use formula (9) to get the visual text representation : (8) (9) In formula (8), represents the second feedforward network; in formula (9), is a parameter to be learned.

5. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The accident analysis classification head in step 2.2.3 is a multi-label multi-classification head, and the predicted vehicle trajectory is obtained using formula (10) , predict the trajectory of other vehicles , predict the cause of the accident and predicted collision type : (10) In formula (10), and are the weights of the vehicle trajectory, other vehicle trajectories, accident causes and collision types, respectively. 、 、 and They are the corresponding 4 bias terms, Represents the softmax activation function.

6. The traffic accident video analysis method based on multi-task learning network according to claim 1 is characterized in that: The context-aware classification head in step 2.3 is a multi-label multi-classification head, which uses formula (11) to obtain the predicted scene information , predict road conditions , forecast weather conditions and predictive traffic signals ; (11) In formula (11), 、 、 and are the weights of scene information, road conditions, weather conditions and traffic signals respectively, 、 、 and They are the corresponding 4 bias terms respectively.

7. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the traffic accident video analysis method according to any one of claims 1 to 6, and the processor is configured to execute the program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the traffic accident video analysis method according to any one of claims 1 to 6 are executed.

Images