Traffic accident key frame extraction method and system for reinforcement learning guided by multi-stream feature fusion

Abstract

The invention discloses a traffic accident key frame extraction method and system for reinforcement learning guided by multi-stream feature fusion, and belongs to the field of traffic video processing. The system comprises a feature extraction and enhancement module, a feature fusion module and a multi-stream feature fusion guided reinforcement learning module. The method comprises the following steps: in a feature extraction and enhancement module, carrying out feature extraction and enhancement on a traffic accident video; taking the output of the feature extraction and enhancement module as the input of a feature fusion module to obtain the importance score of each frame of the video; constructing a reinforcement learning module guided by multi-stream feature fusion, calculating a reward value through the output of the feature fusion module, and extracting a key frame of the accident video according to the maximum reward value to obtain a final result; according to the invention, the accident key frame can be rapidly extracted from the traffic accident video, the manual intervention is reduced, and the working efficiency is improved; and a solid foundation is provided for subsequent traffic management and safety service.

Description

Technical Field

[0001] The present invention relates to the field of traffic video processing, and in particular to a traffic accident key frame extraction method and a system thereof using multi-stream feature fusion guided reinforcement learning. Background Art

[0002] With the acceleration of urbanization, the frequency of traffic accidents has increased year by year, causing serious loss of life and economic losses. Therefore, it is particularly important to scientifically and effectively analyze the causes of accidents and classify the importance of frames in accident videos. Key frame extraction technology can help researchers quickly locate the key moments of accidents, so as to better understand the causes and development process of accidents.

[0003] Existing key frame extraction methods can be divided into shot-based, comparison-based and clustering-based methods. Shot-based methods implement shot segmentation by calculating the similarity matrix between frames and select the initial frame of each shot as the key frame of the video. However, when processing traffic accident videos, the scene in the video does not change much, and the high-speed movement of the vehicle causes motion frame blur, and the shot segmentation may be inaccurate, with low precision and low robustness. Comparison-based methods require detecting sudden changes in the distance between consecutive frames, and key frames can only be identified when the difference between frames exceeds a certain threshold. However, it is difficult to determine an appropriate threshold under different conditions. Clustering-based key frame extraction algorithms cluster video frames by analyzing the global or local features of each frame, aiming to obtain representative frames for each category as key frames. However, clustering-based algorithms ignore the temporal information of frames, which is a major defect in the field of key frame extraction. Reinforcement learning has the potential to improve the performance of key frame extraction by optimizing selection strategies, adapting to different scenarios, and automatically learning video features, but the applicability of reinforcement learning in the field of key frame extraction remains to be explored. Summary of the invention

[0004] In order to solve the problems of low accuracy, poor robustness and poor adaptability in the above-mentioned prior art for extracting key frames from traffic videos, the present invention provides a traffic accident key frame extraction method and system based on multi-stream feature fusion guided reinforcement learning.

[0005] To implement the above technical solution, the details are as follows:

[0006] A method for extracting key frames from traffic accident videos using multi-stream feature fusion-guided reinforcement learning, comprising the following steps:

[0007] S 1. In the feature extraction and enhancement module, feature extraction and enhancement are performed on the traffic accident video;

[0008] The feature extraction part includes: multimodal pre-training model (Contrastive Language-ImagePre-Training, CLIP), optical flow estimation model (Recurrent Al1-Pairs Field Transforms, RAFT);

[0009] The enhancement part includes: the feature enhancement model GRU-Transformer, which consists of a gated recurrent unit (GRU) and a Transformer;

[0010] Here are the steps:

[0011] S1.1, Extract static visual features of input traffic accident video based on CLIP model;

[0012] Since the content of adjacent frames in a video changes little, after uniformly sampling all frames of the video, the sampled frames are input into the pre-trained CLIP model to extract the visual features of the video frames. To represent the static content of the video, where i∈I represents the video number and I represents the total number of videos;

[0013] S1.2, extracting video dynamic features of input traffic accident videos based on CLIP model and RAFT model;

[0014] Input continuous video frames into the pre-trained model RAFT to extract optical flow features At the same time, continuous video frames are fed into the pre-trained CLIP model to extract RGB features The extracted three-stream features are

[0015] S1.3. Add the extracted features to obtain the three-stream features to characterize the state of reinforcement learning; the expression is as follows:

[0016]

[0017] Where, X i Represents the state of reinforcement learning; It means extracting visual features of video frames, i.e., the static content of the video; Represents optical flow features; Represents RGB features;

[0018] S1.4. Input the three-stream features into the GRU-Transformer model for feature enhancement. The steps are as follows:

[0019] S1.4.1. Input the three-stream features into the GRU network to obtain the hidden state features

[0020] S1.4.2. The hidden state feature X′ of each time step i Input into Transformer, use the self-attention mechanism to enhance the long-term dependency of features and obtain enhanced features

[0021] S1.5, jump link the input three-stream features with the enhanced features to enhance the information flow and capture capabilities of the model and reduce the difficulty of training;

[0022] Use skip connections to add the enhanced features to the input features to get the latent representation:

[0023] S2. The output of the feature extraction and enhancement module is used as the input of the feature fusion module to obtain the importance score of each frame of the video. The feature fusion steps are as follows:

[0024] S2.1. All potential representation features obtained through the GRU-Transformer module are normalized and dropped out to obtain the vector representation of the input frame.

[0025] S2.2, using the self-attention mechanism to obtain fusion features;

[0026] Specifically, the three-stream enhanced features are used as the query, key and value of the self-attention mechanism (corresponding to and ) input, the three feature vectors and Fusion; the three feature vectors correspond to the input features The potential representation of (visual content, RGB, optical flow); M is the fusion feature of the video, d is the dimension of the feature vector 1024, and the expression is as follows:

[0027]

[0028] S2.3, input the fused features into the frame importance classifier to calculate the importance score of each frame;

[0029] After the feature vector M is input to the frame importance classifier, it passes through the linear layer (L4) to generate a hidden state for each frame (T is the number of video frames), these hidden states encapsulate time dependency and context information; finally, the hidden state is input into the Sigmoid activation function to obtain the importance score y of each frame i，t , which represents the possibility of each frame being extracted as a key frame; the action of reinforcement learning is generated through the importance score;

[0030] S3, construct a reinforcement learning module guided by multi-stream feature fusion, calculate the reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extract the key frames of the accident video according to the maximum reward value to obtain the final result;

[0031] The reward value is calculated through the reinforcement learning module guided by multi-stream feature fusion. The reward value includes: supervised reward and unsupervised reward;

[0032] The construction steps are as follows:

[0033] S3.1. Calculate the action of reinforcement learning and obtain a set of key frames;

[0034] That is, through the Bernoulli distribution i,t Sampling is performed to obtain the action a of each frame i，t , the expression is as follows:

[0035] a i，t ~Bernoulli i，t )

[0036] In the formula, a i，t ∈{0, 1} indicates whether the tth frame is extracted, 0 indicates no extraction, 1 indicates extraction, representing the action of reinforcement learning; traverse each frame in the video and obtain the key frame set Y = {t|a i，t =1, t=1, ..., T i}, T i Indicates the number of video frames numbered i. The specific value is determined by the length of the video;

[0037] S3.2, calculate the reward value through the key frame set; the steps are as follows:

[0038] S3.2.1. Calculate unsupervised rewards;

[0039] During the training of the reinforcement learning framework, the model will receive an unsupervised reward to evaluate the information content of the extracted accident keyframes; the goal of the model is to maximize the reward over time by extracting the accident keyframes with the highest information content;

[0040] This paper proposes a sparse reward R F and an intensive reward R A To generate accurate and interpretable decisions, the total unsupervised reward R at each time step A +R F Maximum; as follows:

[0041] S3.2.1.1. Calculate the sparsity reward; evaluate the sparsity of the extracted accident key frames by measuring the difference between the selected frames in the feature space. Calculate R FAs the average of the pairwise differences between the selected frames:

[0042]

[0043] In the formula, Obtained from the feature extraction part of S1, it is used to characterize the state of reinforcement learning at the tth frame;

[0044] S3.2.1.2. Computational Intensity Reward; Computational Intensity Reward R A Defined as:

[0045]

[0046] The dense reward can guide the agent to select frames close to the center point in the feature space; R A and R F They complement each other and jointly guide the reinforcement learning model to extract key frames;

[0047] S3.2.2, calculate supervision rewards;

[0048] This paper proposes a new semantic reward R sem To measure the degree of similarity between the actual key frame sequence C and the selected frame sequence; the supervision reward uses the actual key frame C as the supervision information to calculate the cosine similarity between the two sets; the expression is as follows:

[0049] R sem =cosine(C,Y)

[0050] Where, Y represents the selected frame sequence;

[0051] S3.2.3. Calculate the total reward;

[0052] The supervision reward R sem Combined with unsupervised reward R A +R F The reinforcement learning model is trained together. The total reward R(S) of reinforcement learning can be expressed as:

[0053] R(S)=R F +R A +R sem ;

[0054] S3.3, training and strategy optimization;

[0055] That is, the goal of the model is to learn a policy function π with parameter θ θ , which is defined as follows:

[0056]

[0057] In the formula, pθ(ai，1：T ) represents the probability distribution over possible action sequences, and the expectation operator is used Indicates that J(θ) is the objective function, and the goal is to find the parameter value that maximizes the objective function; the episodic REINFORCE algorithm is used to calculate the derivative of the objective function J(θ) with respect to θ, and the expression is as follows:

[0058]

[0059] In the formula, R(S) (n) represents the reward of the nth round, N represents the total number of rounds; one round refers to sampling y i，t , create a selection frame sequence and calculate its reward for each iteration; in order to make the model converge and reduce the variance, the countermeasure adopted is to subtract a constant baseline b from the reward (n) ; Among them, b (n) is the moving average of the rewards of the training rounds.

[0060] A traffic accident video key frame extraction system guided by multi-stream feature fusion and reinforcement learning, comprising: a feature extraction and enhancement module, a feature fusion module and a reinforcement learning module guided by multi-stream feature fusion;

[0061] The feature extraction and enhancement module is used to perform: S1, in the feature extraction and enhancement module, feature extraction and enhancement of the traffic accident video;

[0062] The feature extraction and enhancement module is used to perform: S2, using the output of the feature extraction and enhancement module as the input of the feature fusion module to obtain the importance score of each frame of the video;

[0063] The multi-stream feature fusion guided reinforcement learning module is used to execute: S3, construct a multi-stream feature fusion guided reinforcement learning module, calculate the reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extract the key frames of the accident video according to the maximum reward value to obtain the final result.

[0064] Beneficial effects of the present invention

[0065] The present invention has high precision and robustness; the rich color information in the RGB features can enhance recognition capabilities, and by more accurately distinguishing different types of traffic objects (such as vehicles and pedestrians), the accuracy of key frame extraction is improved. By effectively capturing motion information through optical flow, dynamic changes and abnormal behaviors can be identified at critical moments, further improving the accuracy of event detection. Combined with static image features, it helps to understand the scene background more systematically. The fusion of RGB, optical flow, and static features enables the model to comprehensively consider various information when analyzing complex scenes, thereby improving the understanding and recognition capabilities of complex events.

[0066] The present invention is highly adaptable; it uses a reinforcement learning algorithm for dynamic decision-making, and the model can adaptively learn the best key frame selection strategy. This adaptive feature enables the algorithm to respond quickly to different scenarios and conditions. The introduction of reinforcement learning allows the model to continuously learn and optimize. By interacting with the environment, the model can continuously update and adjust its extraction strategy, adapt to new data situations, and improve performance in long-term key frame extraction applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flow chart of the present invention;

[0068] Figure 2 It is a schematic diagram of a module of the present invention;

[0069] Figure 3 A schematic diagram of a traffic accident key frame extraction method based on feature fusion guided reinforcement learning according to the present invention;

[0070] Figure 4 A diagram showing the construction of a frame importance classifier of the present invention;

[0071] Figure 5 Schematic diagrams of the results of the present invention, wherein (a) is a schematic diagram of the prediction results of a collision process between a white car and a red tractor; (b) is a schematic diagram of the prediction results of a collision process between a black car and a dark blue pickup truck. DETAILED DESCRIPTION

[0072] The present invention is further described in detail below in conjunction with embodiments.

[0073] like Figure 1 , Figure 3 As shown, a traffic accident video key frame extraction method guided by multi-stream feature fusion reinforcement learning includes the following steps:

[0074] S1. In the feature extraction and enhancement module, feature extraction and enhancement are performed on the traffic accident video;

[0075] Traffic accident videos are obtained from open source datasets;

[0076] The feature extraction part includes: multimodal pre-training model (Contrastive Language-Image Pre-Training, CLIP), optical flow estimation model (Recurrent All-Pairs Field Transforms, RAFT);

[0077] The enhancement part includes: the feature enhancement model GRU-Transformer, which consists of a gated recurrent unit (GRU) and a Transformer;

[0078] Specifically, the CLIP model is used to extract the static visual features and dynamic RGB features of the video respectively; at the same time, the RAFT (Recurrent All-Pairs Field Transforms) model is used to extract the optical flow features; then, the GRU-Transformer model is used to enhance the extracted three-stream features respectively;

[0079] Here are the steps:

[0080] S1.1, Extract static visual features of input traffic accident video based on CLIP model;

[0081] Since the content of adjacent frames in a video changes little, after uniformly sampling all frames of the video, the sampled frames are input into the pre-trained CLIP model to extract the visual features of the video frames. To represent the static content of the video, where i∈I represents the video number and I represents the total number of videos;

[0082] In this embodiment, the uniform sampling method is a sampling method of selecting one frame every 5 frames;

[0083] S1.2, extracting video dynamic features of input traffic accident videos based on CLIP model and RAFT model;

[0084] Input continuous video frames into the pre-trained model RAFT to extract optical flow features At the same time, continuous video frames are fed into the pre-trained CLIP model to extract RGB features The extracted three-stream features are

[0085] S1.3. Add the extracted features to obtain the three-stream features to characterize the state of reinforcement learning; the expression is as follows:

[0086]

[0087] Where, X i Represents the state of reinforcement learning; It means extracting visual features of video frames, i.e., the static content of the video; Represents optical flow features; Represents RGB features;

[0088] S1.4. Input the three-stream features into the GRU-Transformer model for feature enhancement. The steps are as follows:

[0089] S1.4.1. Input the three-stream features into the GRU network to obtain the hidden state features

[0090] S1.4.2. The hidden state feature X′ of each time step i Input into Transformer, use the self-attention mechanism to enhance the long-term dependency of features and obtain enhanced features

[0091] S1.5, jump link the input three-stream features with the enhanced features to enhance the information flow and capture capabilities of the model and reduce the difficulty of training;

[0092] Use skip connections to add the enhanced features to the input features to get the latent representation:

[0093] S2. The output of the feature extraction and enhancement module is used as the input of the feature fusion module to obtain the importance score of each frame of the video. The feature fusion steps are as follows:

[0094] S2.1. All potential representation features obtained through the GRU-Transformer module are normalized and dropped out to obtain the vector representation of the input frame. The expression is as follows:

[0095]

[0096] S2.2, using the self-attention mechanism to obtain fusion features;

[0097] Specifically, the three-stream enhanced features are used as the query, key and value of the self-attention mechanism (corresponding to and ) input, the three feature vectors and Fusion; the three feature vectors correspond to the input features The potential representation of (visual content, RGB, optical flow); M is the fusion feature of the video, d is the dimension of the feature vector 1024, and the expression is as follows:

[0098]

[0099] S2.3, such as Figure 4 As shown, the fused features are input into the frame importance classifier to calculate the importance score of each frame;

[0100] The feature vector M is input to the frame importance classifier: it passes through two linear layers (L1 and L2), a normalization layer, a linear layer (L3), a ReLU activation function, a Dropout layer, and another normalization layer, and finally a linear layer (L4) to generate a hidden state for each frame. (T is the number of video frames), these hidden states encapsulate time dependency and context information; finally, the hidden state is input into the Sigmoid activation function to obtain the importance score y of each frame i,t , which represents the possibility of each frame being extracted as a key frame; the action of reinforcement learning is generated by the importance score; the expression is as follows:

[0101] h t =L4(Norm(Dropout(ReLU(L3(Norm(L2(L1(M)))))))))

[0102] y i，t =Sigmoid(h t )

[0103] S3, construct a reinforcement learning module guided by multi-stream feature fusion, calculate the reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extract the key frames of the accident video according to the maximum reward value to obtain the final result;

[0104] The reward value is calculated through the reinforcement learning module guided by multi-stream feature fusion. The reward value includes: supervised reward and unsupervised reward;

[0105] Specifically, in the key frame extraction framework based on reinforcement learning, the state (State) of reinforcement learning is represented by the three-stream features extracted in S1 and the action (Action) of reinforcement learning is generated by the importance score obtained in S2, and the reward value of reinforcement learning is calculated, and then the key frame selection process is optimized; according to the maximum reward value, the accident key frame is extracted.

[0106] The construction steps are as follows:

[0107] S3.1. Calculate the action of reinforcement learning and obtain a set of key frames;

[0108] That is, through the Bernoulli distribution i，t Sampling is performed to obtain the action a of each frame i，t , the expression is as follows:

[0109] a i，t ~Bernoulli i，t )

[0110] In the formula, a i，t ∈{0, 1} indicates whether the tth frame is extracted, 0 indicates no extraction, 1 indicates extraction, representing the action of reinforcement learning; traverse each frame in the video and obtain the key frame set Y = {t|a i，t =1, t=1, ..., T i}, T iIndicates the number of video frames numbered i. The specific value is determined by the length of the video;

[0111] S3.2, calculate the reward value through the key frame set; the steps are as follows:

[0112] S3.2.1. Calculate unsupervised rewards;

[0113] During the training of the reinforcement learning framework, the model will receive an unsupervised reward to evaluate the information content of the extracted accident keyframes; the goal of the model is to maximize the reward over time by extracting the accident keyframes with the highest information content;

[0114] Therefore, the present invention proposes a sparse reward R F and an intensive reward R A To generate accurate and interpretable decisions, the total unsupervised reward R at each time step A +R F Maximum; as follows:

[0115] S3.2.1.1. Calculate the sparsity reward; evaluate the sparsity of the extracted accident key frames by measuring the difference between the selected frames in the feature space. Calculate R F As the average of the pairwise differences between the selected frames:

[0116]

[0117] In the formula, The feature extraction part of S1 is used to characterize the state of reinforcement learning of the tth frame. The equation is a combinatorial function that calculates the cosine similarity between the features of other frames in the selected frame sequence and the features of the current frame. Intuitively, the more dissimilar the selected frames are, the higher the sparsity reward is obtained.

[0118] In this embodiment, the value of is 96;

[0119] S3.2.1.2. Calculate the density reward; this reward measures the degree to which the extracted accident key frame represents the original accident video. Then, the density of an accident key frame is expressed as a k-medoids problem, where the agent is expected to select a set of center points and calculate the mean square error between the video frame and its nearest center point; therefore, the density reward R A Defined as:

[0120]

[0121] The dense reward can guide the agent to select frames close to the center point in the feature space; R A and R F They complement each other and jointly guide the reinforcement learning model to extract key frames;

[0122] S3.2.2, calculate supervision rewards;

[0123] This paper proposes a new semantic reward R sem To measure the degree of similarity between the actual key frame sequence C and the selected frame sequence; the supervision reward uses the actual key frame sequence C as the supervision information to calculate the cosine similarity between the two sets; the expression is as follows:

[0124] R sem =cosine(C,Y)

[0125] Where, Y represents the selected frame sequence;

[0126] S3.2.3. Calculate the total reward;

[0127] Combine the supervised reward Rsem with the unsupervised reward R A +R F The reinforcement learning model is trained together. The total reward R(S) of reinforcement learning can be expressed as:

[0128] R(S)=R F +R A +R sem ;

[0129] S3.3, training and strategy optimization;

[0130] That is, the goal of the model is to learn a policy function π with parameter θ θ , which is defined as follows:

[0131]

[0132] In the formula, p θ (a i，1：T ) represents the probability distribution over possible action sequences, and the expectation operator is used Indicates that J(θ) is the objective function, and the goal is to find the parameter value that maximizes the objective function; the episodic REINFORCE algorithm is used to calculate the derivative of the objective function J(θ) with respect to θ, and the expression is as follows:

[0133]

[0134] In the formula, R(S) (n) represents the reward of the nth round, N represents the total number of rounds, which is 300 in this embodiment; one round refers to sampling y i，t , create a selection frame sequence and calculate its reward for each iteration; in order to make the model converge and reduce the variance, the countermeasure adopted is to subtract a constant baseline b from the reward (n) ; Among them, b (n)is the moving average of the rewards of the training rounds, where the sliding window length is 2;

[0135] like Figure 2 As shown, a traffic accident video key frame extraction system guided by multi-stream feature fusion and reinforcement learning includes: a feature extraction and enhancement module, a feature extraction and enhancement module and a reinforcement learning module guided by multi-stream feature fusion;

[0136] The feature extraction and enhancement module is used to perform: S1, in the feature extraction and enhancement module, feature extraction and enhancement of the traffic accident video;

[0137] The feature fusion module is used to perform: S2, using the output of the feature extraction and enhancement module as the input of the feature fusion module to obtain the importance score of each frame of the video;

[0138] The multi-stream feature fusion guided reinforcement learning module is used to perform: S3, construct a multi-stream feature fusion guided reinforcement learning module, calculate the reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extract the key frame of the accident video according to the maximum reward value to obtain the final result;

[0139] The results of the present invention are as follows Figure 5 As shown:

[0140] (1) Figure 5 Part (a) shows the collision between a white car and a red tractor.

[0141] #78 is the starting point of the vehicle collision. At this time, the two vehicles begin to collide, and their overlap in the video frame is very high, making the collision almost imperceptible to the naked eye.

[0142] #88 is the middle process of the vehicle collision. At this time, the trajectories of the two vehicles change significantly, and it can be clearly observed that the white car collides with the rear end of the red tractor.

[0143] #105 is the end of the vehicle collision. At this time, the collision between the two vehicles ended. The left front light of the white car was obviously damaged, while the red tractor was shrouded in smoke, and the current status of the red tractor could not be seen clearly.

[0144] (2) Figure 5 Part (b) shows the collision between a black car and a dark blue truck.

[0145] In #95, a black car collided with a pickup truck. The black car hit the rear of the pickup truck, causing the two vehicles to change direction. The front of the pickup truck hit the green belt, while the black car lay horizontally in the middle of the road, maintaining a collision posture with the rear of the pickup truck.

[0146] #105 is the middle process of a collision between a black car and a small truck. The two cars move, and the front of the small truck hits the green belt more and more, and the small truck gradually turns to a horizontal position. The black car continues to lie horizontally in the middle of the road and moves forward, but still maintains the posture of colliding with the rear of the small truck.

[0147] The collision between the black car #122 and the pickup truck has ended. At this time, the positions of the two cars have been significantly shifted compared to #95, and both cars have become horizontal, still maintaining the rear-end collision posture.

[0148] Depend on Figure 5 It can be seen that the results from the actual key frames are similar to those from the extracted key frames, and the results match well.

Claims

1. A traffic accident video key frame extraction method guided by multi-stream feature fusion reinforcement learning, characterized in that: The following steps are involved: S1. In the feature extraction and enhancement module, feature extraction and enhancement are performed on the traffic accident video; The feature extraction part includes: CLIP model and RAFT model; The enhancement part includes: a feature enhancement model GRU-Transformer composed of a gated recurrent unit GRU and a Transformer; S2, using the output of the feature extraction and enhancement module as the input of the feature fusion module to obtain the importance score of each frame of the video; S3, construct a reinforcement learning module guided by multi-stream feature fusion, calculate the reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extract the key frames of the accident video according to the maximum reward value to obtain the final result; The reinforcement learning module guided by multi-stream feature fusion calculates the reward value, and the reward value includes: supervised reward and unsupervised reward.

2. According to claim 1, a method for extracting key frames from traffic accident videos by multi-stream feature fusion guided reinforcement learning is characterized in that: In the feature extraction and enhancement module, the steps of extracting and enhancing features of traffic accident videos are as follows: S1.1 Extracting static visual features from input traffic accident videos based on the CLIP model S1.2, extracting dynamic features of input traffic accident videos based on CLIP model and RAFT model; Input continuous video frames into the pre-trained model RAFT to extract optical flow features Where i is the video number; at the same time, the continuous video frames are sent to the pre-trained CLIP model to extract RGB features The three-stream extraction features are S1.

3. Add the extracted features to obtain the three-stream features to characterize the state of reinforcement learning; the expression is as follows: Where, X i Represents the state of reinforcement learning; It means extracting visual features of video frames, i.e., static visual feature extraction of traffic accident videos; Represents optical flow features; Represents RGB features; S1.

4. Input the three-stream features into the GRU-Transformer model for feature enhancement. The steps are as follows: S1.4.

1. Input the three-stream features into the GRU network to obtain the hidden state features S1.4.

2. The hidden state feature X′ of each time step i Input into Transformer, use the self-attention mechanism to enhance the long-term dependency of features and obtain enhanced features S1.5, skip-connect the input three-stream features and the enhanced features; Use skip connections to add the enhanced features to the input features to get the latent representation:

3. According to claim 1, a method for extracting key frames from traffic accident videos by multi-stream feature fusion guided reinforcement learning is characterized in that: The steps of using the output of the feature extraction and enhancement module as the input of the feature fusion module to obtain the importance score of each frame of the video are as follows: S2.

1. All potential representation features obtained through the GRU-Transformer module are normalized and dropped out to obtain the vector representation of the input frame. S2.2, using the self-attention mechanism to obtain fusion features; The details are as follows: The three-stream enhanced features are used as the query, key and value of the self-attention mechanism (corresponding to and ) input, the three feature vectors and Fusion; The three eigenvectors correspond to the input features The potential representation of; M is the fusion feature of the video, d is the dimension of the feature vector 1024, the expression is as follows: S2.

3. Input the fused features into the frame importance classifier to calculate the importance score of each frame; the details are as follows: After the feature vector M is input into the frame importance classifier, it passes through the linear layer L4 to generate hidden states for each frame. These hidden states encapsulate time dependency and context information. The hidden states are input into the Sigmoid activation function to obtain the importance score y of each frame. i,t , which represents the possibility of each frame being extracted as a key frame; Generating actions for reinforcement learning via importance scores.

4. According to claim 1, a method for extracting key frames from traffic accident videos by multi-stream feature fusion guided reinforcement learning is characterized in that: The steps of constructing a multi-stream feature fusion-guided reinforcement learning module, calculating the reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extracting the key frames of the accident video according to the maximum reward value to obtain the final result are as follows: S3.1, calculate the reinforcement learning action and obtain the key frame set; That is, through the Bernoulli distribution i,t Sampling is performed to obtain the action a of each frame i,t , the expression is as follows: a i,t ~Bernoulli(y i,t ) In the formula, a i,t ∈{0,1} indicates whether the tth frame is extracted, 0 means no extraction, 1 means extraction, representing the action of reinforcement learning; traverse each frame in the video to obtain the key frame set Y={t|a i,t =1,t=1,...,T i }, T i Indicates the number of video frames numbered i. The specific value is determined by the length of the video; S3.2, calculate the reward value through the key frame set; the steps are as follows: S3.2.

1. Calculate unsupervised rewards; S3.2.1.

1. Calculate the sparsity reward; the expression is as follows: In the formula, Obtained from the feature extraction part of S1, it is used to characterize the state of reinforcement learning at the tth frame; S3.2.1.

2. Computational Intensity Reward; Computational Intensity Reward R A Defined as: The dense reward can guide the agent to select frames close to the center point in the feature space; R A and R F They complement each other and jointly guide the reinforcement learning model to extract key frames; S3.2.2, calculate supervision rewards; The expression is as follows: R sem =cosine(C,Y) In the formula, C represents the actual key frame sequence, and Y represents the selected frame sequence; S3.2.

3. Calculate the total reward; The expression is as follows: R(S)=R F +R A +R sem ; S3.3, training and strategy optimization; That is, the goal of the model is defined as learning a policy function π with parameter θ θ , which is defined as follows: In the formula, p θ (a i,1:T ) represents the probability distribution of possible action sequences, and the expectation operator is used Indicates that J(θ) is the objective function, and the goal is to find the parameter value that maximizes the objective function; the episodic REINFORCE algorithm is used to calculate the derivative of the objective function J(θ) with respect to θ, and the expression is as follows: In the formula, R(S) (n) represents the reward of the nth round, N represents the total number of rounds; one round refers to sampling y i,t , create a selection frame sequence and calculate its reward for each iteration; in order to make the model converge and reduce the variance, the countermeasure adopted is to subtract a constant baseline b from the reward (n) ; Among them, b (n) is the moving average of the rewards of the training rounds.

5. A traffic accident video key frame extraction system guided by multi-stream feature fusion and reinforcement learning, characterized in that: The system is used to execute the steps in the method according to any one of claims 1 to 4.

Images