A multi-flow feature fusion guided reinforcement learning traffic accident key frame extraction method and system thereof

Abstract

The application discloses a traffic accident key frame extraction method and system based on multi-flow feature fusion guided reinforcement learning, and belongs to the field of traffic video processing.The system comprises a feature extraction and enhancement module, a feature fusion module and a multi-flow feature fusion guided reinforcement learning module.The method comprises the following steps: in the feature extraction and enhancement module, traffic accident video is subjected to feature extraction and enhancement; the output of the feature extraction and enhancement module is taken as the input of the feature fusion module to obtain the importance score of each frame of the video; the multi-flow feature fusion guided reinforcement learning module is constructed, the reward value is calculated according to the output of the feature fusion module and the output of the feature fusion module, and according to the maximum reward value, the key frame of the accident video is extracted to obtain the final result.The application can quickly extract the accident key frame from the traffic accident video, reduces manual intervention, improves work efficiency, and provides a solid foundation for subsequent traffic management and safety services.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of traffic video processing, in particular to a traffic accident key frame extraction method based on multi-stream feature fusion guided reinforcement learning and a system thereof. BACKGROUND

[0002] With the acceleration of urbanization, the frequency of traffic accidents is increasing year by year, causing serious loss of life and economy. Therefore, it is particularly important to scientifically and effectively analyze the causes of accidents and classify the frames of accident videos. Key frame extraction technology can help researchers quickly locate the key moments of accidents, so as to better understand the causes and development process of accidents.

[0003] Existing key frame extraction methods can be divided into three categories: shot-based, comparison-based and clustering-based methods. Shot-based methods achieve shot segmentation by calculating the similarity matrix between frames, and select the initial frame of each shot as the key frame of the video. However, when processing traffic accident videos, the scene in the video changes little, and the high-speed movement of vehicles causes motion blur, so the shot segmentation may not be accurate, with low precision and weak robustness. Comparison-based methods need to detect sudden changes in distance between consecutive frames, and only when the difference between frames exceeds a certain threshold can the key frame be identified. However, it is difficult to determine an appropriate threshold under different conditions. Clustering-based key frame extraction algorithms analyze the global or local features of each frame to cluster video frames, aiming to obtain representative frames of each class as key frames. However, clustering-based algorithms ignore the temporal information of frames, which is a major drawback in the field of key frame extraction. Reinforcement learning has the potential to improve key frame extraction performance by optimizing selection strategies, adapting to different scenarios and automatically learning video features, but the applicability of reinforcement learning in the field of key frame extraction still needs to be explored. SUMMARY

[0004] In order to solve the problems of low precision, poor robustness and poor adaptability of existing key frame extraction methods for traffic videos, the present application provides a traffic accident key frame extraction method based on multi-stream feature fusion guided reinforcement learning and a system thereof.

[0005] To achieve the above technical solutions, the following is specific:

[0006] A traffic accident video key frame extraction method based on multi-stream feature fusion guided reinforcement learning, comprising the following steps:

[0007] S 1, in the feature extraction and enhancement module, the features of the traffic accident video are extracted and enhanced;

[0008] The feature extraction part includes a multi-modal pre-training model (Contrastive Language-Image Pre-Training, CLIP) and an optical flow estimation model (Recurrent Al1-Pairs Field Transforms, RAFT);

[0009] The enhancement part includes a feature enhancement model GRU-Transformer composed of a Gated Recurrent Unit (GRU) and a Transformer;

[0010] The steps are as follows:

[0011] S1.1, based on the CLIP model, the static visual features of the input traffic accident video are extracted;

[0012] Since the content of adjacent frames in the video changes little, after uniform sampling of all frames of the video, the sampled frames are input into the pre-trained CLIP model to extract the visual features of the video frames to represent the static content of the video, where i∈I represents the video number, and I represents the total number of videos;

[0013] S1.2, based on the CLIP model and the RAFT model, the video dynamic features of the input traffic accident video are extracted;

[0014] The continuous video frames are input into the pre-trained model RAFT to extract the optical flow features At the same time, the continuous video frames are input into the pre-trained CLIP model to extract the RGB features The three flow features extracted are

[0015] S1.3, the three flow features extracted are added to obtain the three flow features, which represent the state of the reinforcement learning; the expression is as follows:

[0016]

[0017] In the formula, X i represents the state of the reinforcement learning; represents the visual features of the video frames, i.e., the static content of the video; represents the optical flow features; represents the RGB features;

[0018] S1.4, the three flow features are input into the GRU-Transformer model for feature enhancement; the steps are as follows:

[0019] S1.4.1, the three flow features are input into the GRU network to obtain the hidden state features

[0020] S1.4.2, the hidden state feature X' of each time step is input into the Transformer to enhance the long-term dependence of the feature by using the self-attention mechanism and to obtain an enhanced feature i

[0021] S1.5, the input three-flow feature and the enhanced feature are linked by a skip connection to enhance the information flow and capture ability of the model and reduce the difficulty of training;

[0022] The enhanced feature is added to the input feature using a skip connection to obtain a latent representation:

[0023] S2, the output of the feature extraction and enhancement module is input into the feature fusion module to obtain the importance score of each frame of the video; the feature fusion step is as follows:

[0024] S2.1, all latent representation features obtained by the GRU-Transformer module are used to obtain the vector representation of the input frame using a normalization layer and a Dropout layer

[0025] S2.2, the self-attention mechanism is used to obtain the fused feature;

[0026] Specifically, the three-flow enhanced features are respectively input into the Query, Key and Value (corresponding to and ) of the self-attention mechanism, and the three feature vectors and are fused; the three feature vectors correspond to the latent representations of the input features (visual content, RGB, optical flow); M is the fused feature of the video, d is the dimension of the feature vector 1024, and the expression is as follows:

[0027]

[0028] S2.3, the fused feature is input into the frame importance classifier to calculate the importance score of each frame;

[0029] After the feature vector M is input into the frame importance classifier, it passes through a linear layer (L4) to generate a hidden state (T is the number of video frames) for each frame, which encapsulates the temporal dependence and context information; finally, the hidden state is input into a Sigmoid activation function to obtain the importance score y of each frame i，t , which represents the possibility of each frame being extracted as a key frame; the importance score generates the action of reinforcement learning;

[0030] ​S3, construct a multi-flow feature fusion guided reinforcement learning module, calculate a reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extract key frames of the accident video according to the maximum reward value to obtain a final result;

[0031] The multi-flow feature fusion guided reinforcement learning module calculates a reward value, and the reward value includes a supervised reward and an unsupervised reward.

[0032] The construction steps are as follows:

[0033] S3.1, calculate an action (Action) of reinforcement learning and obtain a key frame set;

[0034] That is, a y i,t is sampled through a Bernoulli distribution to obtain an action a i，t of each frame, and the expression is as follows:

[0035] a i，t ~ Bernoulli (y i，t )

[0036] In the formula, a i，t ∈ {0, 1} represents whether the t-th frame is extracted, 0 represents not extracted, 1 represents extracted, and a i，t characterizes an action (Action) of reinforcement learning; each frame in the video is traversed to obtain a key frame set Y = {t | a i = 1, t = 1,..., T i , T F represents the number of video frames numbered i, and the specific value is determined by the video length.

[0037] S3.2, calculate a reward value through the key frame set; the steps are as follows:

[0038] S3.2.1, calculate an unsupervised reward;

[0039] During the training of the reinforcement learning framework, the model will obtain an unsupervised reward to evaluate the information amount of the extracted accident key frame; the goal of the model is to extract the accident key frame with the highest information amount and maximize the reward over time;

[0040] The present application proposes a sparsity reward R F and a density reward R A to generate accurate and interpretable decisions, so that the total unsupervised reward R A + R F of each time step is maximized; the specific process is as follows:

[0041] S3.2.1.1, calculate a sparsity reward; the sparsity degree of the extracted accident key frame is evaluated by measuring the difference between the selected frames in the feature space. R FAs the average value of the pair difference between the selected frames:

[0042]

[0043] In the formula, Obtained by the feature extraction part of S1, used to represent the state of the t-th frame reinforcement learning;

[0044] S3.2.1.2, calculation of intensive reward; intensive reward R A Defined as:

[0045]

[0046] Intensive reward can guide the agent to select frames close to the center point in the feature space; R A And R F Complement each other, and guide the reinforcement learning model to extract key frames together;

[0047] S3.2.2, calculate the supervision reward;

[0048] The present application proposes a new semantic reward R sem To measure the degree of similarity between the actual key frame sequence C and the selected frame sequence; the supervision reward takes the actual key frame C as the supervision information, and calculates the cosine similarity between the two sets; the expression is as follows:

[0049] R sem =cosine(C, Y)

[0050] In the formula, Y represents the selected frame sequence;

[0051] S3.2.3, calculate the total reward;

[0052] Combine the supervision reward R sem With the unsupervised reward R A +R F To train the reinforcement learning model together, and the total reward R(S) of reinforcement learning can be expressed as:

[0053] R(S) = R F +R A +R sem ;

[0054] S3.3, training and policy optimization;

[0055] That is, the goal of the model is to learn a policy function π θ , which is defined as follows:

[0056]

[0057] In the formula, pθ(ai，1：T ) represents the probability distribution over possible action sequences, and the expectation operator is denoted by , J(θ) is the objective function, and the goal is to find the parameter values that maximize the objective function; the derivative of the objective function J(θ) with respect to θ is calculated using the episodic REINFORCE algorithm, which is expressed as follows:

[0058]

[0059] where R(S (n) ) represents the reward of the nth round, and N represents the total number of rounds; one round refers to each iteration of sampling y i，t , creating a selection frame sequence and calculating its reward; in order to make the model converge and reduce the variance, the countermeasure adopted is to subtract a constant baseline b (n) from the reward; where b (n) is the moving average value of the reward of the trained rounds.

[0060] A traffic accident video key frame extraction system based on multi-stream feature fusion guided reinforcement learning, comprising: a feature extraction and enhancement module, a feature fusion module and a multi-stream feature fusion guided reinforcement learning module;

[0061] The feature extraction and enhancement module is used to perform: S1, in the feature extraction and enhancement module, performing feature extraction and enhancement on the traffic accident video;

[0062] The feature extraction and enhancement module is used to perform: S2, taking the output of the feature extraction and enhancement module as the input of the feature fusion module to obtain the importance score of each frame of the video;

[0063] The multi-stream feature fusion guided reinforcement learning module is used to perform: S3, constructing the multi-stream feature fusion guided reinforcement learning module, calculating the reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extracting the key frame of the accident video according to the maximum reward value to obtain the final result.

[0064] The beneficial effects of the present application

[0065] The present application has high precision and robustness; the rich color information in the RGB feature can enhance the recognition ability, and by more accurately distinguishing different types of traffic objects (such as vehicles and pedestrians), the accuracy of key frame extraction is improved. By effectively capturing motion information through optical flow, dynamic changes and abnormal behaviors can be identified at critical moments, further improving the accuracy of event detection. Combined with static image features, it helps to more systematically understand the scene background. The fusion of RGB, optical flow and static features enables the model to consider multiple aspects of information when analyzing complex scenes, thereby improving the understanding and recognition ability of complex events.

[0066] This invention exhibits strong adaptability; it employs reinforcement learning algorithms for dynamic decision-making, enabling the model to adaptively learn the optimal keyframe selection strategy. This adaptive characteristic allows the algorithm to respond quickly under different scenarios and conditions. The introduction of reinforcement learning allows the model to continuously learn and optimize. Through interaction with the environment, the model can continuously update and adjust its extraction strategy, adapting to new data conditions and improving performance in long-term keyframe extraction applications. Attached Figure Description

[0067] Figure 1 This is a flowchart of the present invention;

[0068] Figure 2 This is a schematic diagram of the modules of the present invention;

[0069] Figure 3 This is a schematic diagram of the traffic accident keyframe extraction method based on feature fusion-guided reinforcement learning according to the present invention.

[0070] Figure 4 This is a diagram illustrating the construction of the frame importance classifier of the present invention.

[0071] Figure 5 The diagrams are schematic diagrams of the results of the present invention, wherein (a) is a schematic diagram of the predicted results of a collision between a white car and a red tractor; and (b) is a schematic diagram of the predicted results of a collision between a black car and a dark blue van. Detailed Implementation

[0072] The present invention will be further described in detail below with reference to the embodiments.

[0073] like Figure 1 , Figure 3 As shown, a method for extracting keyframes from traffic accident videos using multi-stream feature fusion-guided reinforcement learning includes the following steps:

[0074] S1. In the feature extraction and enhancement module, features are extracted and enhanced from traffic accident videos;

[0075] Traffic accident videos were obtained from an open-source dataset;

[0076] The feature extraction part includes: a multimodal pre-trained model (Contrastive Language-Image Pre-Training, CLIP) and an optical flow estimation model (RecurrentAll-Pairs Field Transforms, RAFT);

[0077] The enhancements include: the feature enhancement model GRU-Transformer, which consists of a gated recurrent unit (GRU) and a Transformer;

[0078] Specifically, the static visual features and dynamic RGB features of the video are extracted using the CLIP model respectively; at the same time, the optical flow features are extracted using the RAFT (Recurrent All-Pairs Field Transforms) model; then, the three flow features extracted are enhanced respectively using the GRU-Transformer model;

[0079] The steps are as follows:

[0080] S1.1, based on the CLIP model, the static visual features of the input traffic accident video are extracted;

[0081] Since the content of adjacent frames in the video changes little, after uniform sampling of all frames of the video, the sampled frames are input into the pre-trained CLIP model to extract the visual features of the video frames to represent the static content of the video, wherein i∈I represents the video number, and I represents the total number of videos;

[0082] In this embodiment, the uniform sampling method is to select one frame every 5 frames;

[0083] S1.2, based on the CLIP model and the RAFT model, the video dynamic features of the input traffic accident video are extracted;

[0084] The continuous video frames are input into the pre-trained model RAFT to extract the optical flow features At the same time, the continuous video frames are input into the pre-trained CLIP model to extract the RGB features The three flow features extracted are

[0085] S1.3, the extracted features are added to obtain the three flow features, which represent the state of the reinforcement learning; the expression is as follows:

[0086]

[0087] In the formula, X i represents the state of the reinforcement learning; represents the visual features of the video frames, i.e., the static content of the video; represents the optical flow features; represents the RGB features;

[0088] S1.4, the three flow features are input into the GRU-Transformer model respectively for feature enhancement; the steps are as follows:

[0089] S1.4.1, the three flow features are input into the GRU network to obtain the hidden state features

[0090] S1.4.2, the hidden state feature X' of each time step is obtained by inputting the enhanced feature into the GRU i

[0091] S1.5, the input three-flow feature and the enhanced feature are linked by jump, the information flow and capture ability of the model are enhanced, and the difficulty of training is reduced;

[0092] The enhanced feature and the input feature are added by using the jump connection to obtain the latent representation:

[0093] S2, the output of the feature extraction and enhancement module is taken as the input of the feature fusion module to obtain the importance score of each frame of the video; the feature fusion step is as follows:

[0094] S2.1, all the latent representation features obtained by the GRU-Transformer module are used to obtain the vector representation of the input frame by using the normalization layer and the Dropout layer The expression is as follows:

[0095]

[0096] S2.2, the self-attention mechanism is used to obtain the fusion feature;

[0097] Specifically, the three-flow enhanced features are respectively taken as the inputs of Query, Key and Value (corresponding to and ) of the self-attention mechanism, and the three feature vectors and are fused; the three feature vectors correspond to the latent representations of the input features (visual content, RGB, optical flow); M is the fusion feature of the video, d is the dimension of the feature vector 1024, and the expression is as follows:

[0098]

[0099] S2.3, as shown in Figure 4 , the fused feature is input into the frame importance classifier to calculate the importance score of each frame;

[0100] The feature vector M is input into the frame importance classifier: after sequentially passing through two linear layers (L1 and L2), a normalization layer, a linear layer (L3), a ReLU activation function, a Dropout layer, and another normalization layer, the final linear layer (L4) is used to generate the hidden state for each frame ​(T is the number of video frames), which encapsulates temporal dependency and context information; finally, the hidden state is input into a sigmoid activation function to obtain the importance score y of each frame i,t , which represents the possibility of each frame being extracted as a key frame; the importance score generates the action of reinforcement learning; the expression is as follows:

[0101] h t =L4(Norm(Dropout(ReLU(L3(Norm(L2(L1(M))))))))

[0102] y i，t =Sigmoid(h t )

[0103] S3, a multi-stream feature fusion guided reinforcement learning module is constructed, the reward value is calculated through the output of the feature fusion module and the output of the feature extraction and enhancement module, and the key frame of the accident video is extracted according to the maximum reward value, and the final result is obtained;

[0104] The reward value is calculated by the multi-stream feature fusion guided reinforcement learning module, and the reward value includes supervised reward and unsupervised reward.

[0105] Specifically, in the key frame extraction framework based on reinforcement learning, the three-stream features extracted in S1 represent the state of reinforcement learning, and the importance score obtained in S2 generates the action of reinforcement learning, the reward value of reinforcement learning is calculated, and then the selection process of key frame is optimized; according to the maximum reward value, the key frame of the accident is extracted.

[0106] The construction steps are as follows:

[0107] S3.1, calculate the action of reinforcement learning and obtain the key frame set;

[0108] That is, y i，t is sampled through Bernoulli distribution to obtain the action a i，t of each frame, and the expression is as follows:

[0109] a i，t ~Bernoulli(y i，t )

[0110] In the formula, a i，t ∈{0,1} represents whether the t-th frame is extracted, 0 represents not extracted, and 1 represents extracted, which represents the action of reinforcement learning; each frame in the video is traversed to obtain the key frame set Y = {t|a i，t = 1, t = 1,..., T i}, T idenotes the number of video frames, the specific value is determined by the length of the video;

[0111] S3.2, calculating the reward value through the key frame set; the steps are as follows:

[0112] S3.2.1, calculating the unsupervised reward;

[0113] During the training of the reinforcement learning framework, the model will obtain an unsupervised reward to evaluate the information amount of the extracted accident key frame; the goal of the model is to extract the accident key frame with the highest information amount and maximize the reward over time;

[0114] Therefore, the present application proposes a sparsity reward R F and a density reward R A to generate accurate and interpretable decisions, so that the unsupervised total reward R A +R F of each time step is maximized; specifically as follows:

[0115] S3.2.1.1, calculating the sparsity reward; the sparsity degree of the extracted accident key frame is evaluated by measuring the difference between the selected frames in the feature space. R F is calculated as the average of the pairwise differences between the selected frames:

[0116]

[0117] In the formula, obtained by the feature extraction part of S1, used to represent the state of the t-th frame of reinforcement learning; the equation is a combination function that calculates the cosine similarity between the features of other frames in the selected frame sequence and the current frame features; intuitively, the less similar the selected frames are, the higher the sparsity reward obtained will be;

[0118] In this embodiment, the value of is 96;

[0119] S3.2.1.2, calculating the density reward; this reward measures the degree to which the extracted accident key frame represents the original accident video. Then, the density degree of an accident key frame is expressed as a k-medoids problem, expecting the agent to select a set of center points to calculate the mean square error between the video frames and the nearest center points; therefore, the density reward R A is defined as:

[0120]

[0121] The density reward can guide the agent to select frames close to the center points in the feature space; R A and R F complement each other and jointly guide the reinforcement learning model to extract key frames;

[0122] S3.2.2, calculate the supervision reward;

[0123] The present application proposes a new semantic reward R sem to measure the degree of identity between the actual key frame sequence C and the selected frame sequence; the supervision reward takes the actual key frame sequence C as the supervision information, and calculates the cosine similarity between the two sets; the expression is as follows:

[0124] R sem = cosine(C, Y)

[0125] In the formula, Y represents the selected frame sequence;

[0126] S3.2.3, calculate the total reward;

[0127] Combine the supervision reward Rsemwith the unsupervised reward R A + R F to jointly train the reinforcement learning model; the total reward R(S) of the reinforcement learning can be expressed as:

[0128] R(S) = R F + R A + R sem ;

[0129] S3.3, training and policy optimization;

[0130] That is, the goal of the model is to learn a policy function π θ with parameter θ, which is defined as follows:

[0131]

[0132] In the formula, p θ (a i，1：T ) represents the probability distribution on the possible action sequence, the expectation operator is represented by , J(θ) is the objective function, and the goal is to find the parameter value that maximizes the objective function; the derivative of the objective function J(θ) with respect to θ is calculated using the episodic REINFORCE algorithm, and the expression is as follows:

[0133]

[0134] In the formula, R(S) (n) represents the reward of the nth round, and N represents the total number of rounds, which is 300 in this embodiment; one round refers to each iteration of sampling y i，t , creating a selected frame sequence and calculating its reward; in order to make the model converge and reduce the variance, the countermeasure adopted is to subtract a constant baseline b (n) from the reward; wherein b (n)is a moving average of the trained episode rewards with a sliding window length of 2;

[0135] As shown in Figure 2 A traffic accident video key frame extraction system based on multi-stream feature fusion guided reinforcement learning, comprising: a feature extraction and enhancement module, a feature extraction and enhancement module and a multi-stream feature fusion guided reinforcement learning module;

[0136] The feature extraction and enhancement module is used to perform: S1, in the feature extraction and enhancement module, feature extraction and enhancement are performed on the traffic accident video;

[0137] The feature fusion module is used to perform: S2, taking the output of the feature extraction and enhancement module as the input of the feature fusion module, to obtain the importance score of each frame of the video;

[0138] The multi-stream feature fusion guided reinforcement learning module is used to perform: S3, constructing a multi-stream feature fusion guided reinforcement learning module, calculating a reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extracting key frames of the accident video according to the maximum reward value to obtain a final result;

[0139] The results of the present application are shown in Figure 5 As shown in

[0140] (1) Figure 5 Part (a) shows the process of a white car colliding with a red tractor.

[0141] #78 is the starting point of the vehicle collision, at which time the two vehicles begin to collide, and their overlap in the video frame is very high, and the naked eye can hardly detect the occurrence of the collision.

[0142] #88 is the middle process of the vehicle collision, at which time the two vehicles have obvious trajectory changes, and the collision between the white car and the red tractor tail can be clearly observed.

[0143] #105 is the end of the vehicle collision, at which time the collision of the two vehicles ends, the left front light of the white car is obviously damaged, and the red tractor is covered with smoke, making it impossible to see the status of the red tractor.

[0144] (2) Figure 5 Part (b) shows the process of a black car colliding with a dark blue van.

[0145] In #95, a black car collides with a van. The black car hits the tail of the van, causing the driving direction of the two vehicles to change. The head of the van hits the green belt, while the black car is placed horizontally in the center of the road, maintaining the posture of colliding with the tail of the van.

[0146] #105 is a black car in the process of colliding with a small van. The two vehicles are displaced, and the degree of impact of the small van's head on the green belt deepens, and the small van gradually turns into a transverse state. The black car continues to be transversely placed in the center of the road and is displaced forward, but still maintains the posture of colliding with the tail of the small van.

[0147] #122 The collision of the black car and the small van has ended. At this time, the positions of the two vehicles have obviously changed compared to #95, and both vehicles have become transversely placed and still maintain the posture of colliding with the tail.

[0148] By Figure 5 It can be seen that the actual key frame results and the extracted key frame results are similar, and the matching degree of the results is good.

Claims

1. A traffic accident video key frame extraction method based on multi-flow feature fusion guided reinforcement learning, characterized in that, The method comprises the following steps: S1, in the feature extraction and enhancement module, the traffic accident video is extracted and enhanced; The feature extraction part comprises a CLIP model and a RAFT model; The enhancement part comprises a feature enhancement model GRU-Transformer composed of a gated recurrent unit (GRU) and a transformer; The step of extracting and enhancing the traffic accident video in the feature extraction and enhancement module is as follows: S1.1, extracting static visual features based on the CLIP model on the input traffic accident video ; S1.2, based on the CLIP model and the RAFT model, the dynamic features of the input traffic accident video are extracted; Input the continuous video frame into the pre-trained model RAFT to extract the optical flow feature , wherein i is the video number; at the same time, the continuous video frame is input into the pre-trained CLIP model to extract the RGB feature ; then the three-flow extracted features are ; S1.3, the extracted features are added to obtain three-flow features, which represent the state of the reinforcement learning; the expression is as follows: ； In the formula, represents the state of reinforcement learning; represents the extraction of visual features of video frames, i.e. static visual feature extraction of traffic accident videos; represents the optical flow feature; represents the RGB feature; S1.4, the three-flow features are input into the GRU-Transformer model for feature enhancement; the steps are as follows: S1.4.1, input the three-flow features into the GRU network respectively to obtain hidden state features ; S1.4.2, the hidden state feature of each time step is input into the Transformer to enhance the long-term dependency of the feature by using a self-attention mechanism and obtain an enhanced feature ; S1.5, the input three-flow features are connected with the enhanced features; The enhanced features are added to the input features using skip connections to obtain the latent representation: ; S2, the output of the feature extraction and enhancement module is taken as the input of the feature fusion module, and the importance score of each frame of the video is obtained; S3, a multi-flow feature fusion guided reinforcement learning module is constructed, the action of the reinforcement learning is generated through the importance score, the reward value is calculated through the output of the feature fusion module and the output of the feature extraction and enhancement module, and the key frame of the accident video is extracted according to the maximum reward value, and the final result is obtained; The reward value comprises a supervised reward and an unsupervised reward.

2. The traffic accident video key frame extraction method of claim 1, wherein The step of taking the output of the feature extraction and enhancement module as the input of the feature fusion module to obtain the importance score of each frame of the video is as follows: S2.

1. All latent representation features obtained by the GRU-Transformer module use a normalization layer and a Dropout layer to obtain the vector representation of the input frame , , ; S2.2, the fusion features are obtained by using a self-attention mechanism; Specifically as follows: Triplet enhanced features , and are fused as Query, Key and Value of self-attention mechanism respectively , and ​ The three eigenvectors correspond to the latent representation of the input features , , respectively; is the fusion feature of the video, d is the dimension of the eigenvector 1024, and the expression is as follows: ； S2.3, the fused features are input into a frame importance classifier to calculate the importance score of each frame; specifically as follows: After the feature vector M is input into the frame importance classifier, it passes through a linear layer L4 to generate hidden states for each frame, which encapsulate time dependence and context information; the hidden states are input into a Sigmoid activation function to obtain the importance score of each frame, i.e. the possibility of each frame being extracted as a key frame . The action of the reinforcement learning is generated through the importance score.

3. The traffic accident video key frame extraction method of claim 2, wherein, The step of constructing the multi-flow feature fusion guided reinforcement learning module, calculating the reward value through the output of the feature fusion module and the output of the feature extraction and enhancement module, and extracting the key frame of the accident video according to the maximum reward value to obtain the final result is as follows: S3.1, the action of the reinforcement learning is calculated and the key frame set is obtained; That is, by Bernoulli distribution to Sampling, get each frame of action Expression as follows: ； In the formula, indicates the first frame is extracted, 0 indicates not extracted, 1 indicates extracted, and represents the action of reinforcement learning; each frame in the video is traversed to obtain a key frame set , indicates the number of video frames numbered , and the specific value is determined by the video length; S3.2, the reward value is calculated through the key frame set; the steps are as follows: S3.2.1, the unsupervised reward is calculated; S3.2.1.1, the sparsity reward is calculated; the expression is as follows: ； In the formula, obtained by the feature extraction section of S1 for characterizing the first frame of the state of the reinforcement learning; S3.2.1.

2. Computationally intensive rewards; intensive rewards R A defined as: ； The dense reward can guide the agent to select frames close to the center point in the feature space; R A and R F complement each other, and jointly guide the reinforcement learning model to extract key frames; S3.2.2, the supervised reward is calculated; The expression is as follows: ； In the formula, denotes the actual key frame sequence, denotes the selected frame sequence; S3.2.3, the total reward is calculated; The expression is as follows: ； S3.3, training and policy optimization; Defining the objective of the model as learning a policy function with a parameter θ defined as follows: ； wherein denotes the probability distribution over possible action sequences, the expectation operator is denoted by denotes, is the objective function, the goal being to find the parameter values that maximize the objective function; Calculating the objective function using the episodic REINFORCE algorithm The derivative with respect to θ is given by the expression ； wherein Rnrepresents the reward of the nth episode, N represents the total number of episodes; an episode refers to a sampling , creating a sequence of selected frames and calculating its reward; in order to make the model converge and reduce the variance, the countermeasure adopted is to subtract a constant baseline from the reward ; wherein, is the moving average of the reward of the trained episode.

4. A traffic accident video key frame extraction system based on multi-flow feature fusion guided reinforcement learning, characterized in that, The system is used to execute the steps in the method of any one of claims 1 to 3.

Images