Video-based traffic accident analysis method, device, equipment and medium

Abstract

The invention relates to the technical field of computers, and provides a video-based traffic accident analysis method, device and equipment and a medium, and the method comprises the steps: extracting the video features of a traffic video, and extracting the text features of a marked traffic scene problem; training a projector model to align the video features with the text features and splice the video features and the text features to obtain video-problem text features; on the basis of the video-question text features, a prediction answer corresponding to the marked traffic scene question is generated through a large language model, a projector model is updated on the basis of the prediction answer and the corresponding mark, and the step of extracting the video features and the text features is returned until a preset condition is met; and analyzing the cause of the traffic accident based on the trained projector model and the big language model. According to the scheme, traffic accident reasons are analyzed as accurately as possible, the interpretability of traffic accident analysis is improved, and the accuracy and efficiency of traffic accident analysis are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a video-based traffic accident analysis method, device, equipment and medium. Background Art

[0002] Roadside cameras can collect road traffic conditions in real time and capture abnormal situations in a timely manner. By analyzing the traffic videos collected by roadside cameras, potential traffic accident risks can be identified in a timely manner, and early warnings can be issued in a timely manner to reduce the possibility of traffic accidents.

[0003] However, the existing video-based traffic accident analysis methods generally identify traffic accidents based on preset rules and further predict the cause of the accident. For example, if the queue time at a certain location is too long, a traffic accident may occur and the cause of the accident may be further predicted. This method has a low accuracy rate in traffic accident judgment and is easily affected by other traffic participants on the road, making it difficult to accurately analyze the cause of the accident. Furthermore, identifying traffic accidents based on target detection methods improves the accuracy rate of traffic accident judgment, but it is impossible to analyze the cause of the traffic accident, and the entire process requires massive traffic data to train the model, which increases the consumption of computing resources.

[0004] The existing published patent (publication number CN115391511A) provides a video question-answering method, which extracts video and text features and splices them into a spliced ​​feature vector. The first pre-trained model is first input to learn the modal information to obtain the encoded second spliced ​​feature vector, which is then split into the encoded second video features and second text features, and then input into the modal fusion model for feature fusion to obtain a fused feature vector, and finally predicts the candidate answer through the decoding layer. This process requires splicing and splitting the video features and text feature vectors and then splicing and fusing them. The processing process of video features and text features is relatively complicated, and the common sense of the large language model is not fully utilized, so the model lacks interpretability and accuracy.

[0005] Based on this, there is an urgent need for a method to analyze the causes of traffic accidents as accurately as possible. Summary of the invention

[0006] In view of the problem that the method in the prior art for identifying traffic accidents based on preset rules and further predicting the cause of the accident is difficult to accurately analyze the cause of the accident and requires a large amount of computing resources, the present invention proposes a video-based traffic accident analysis method, device, equipment and medium.

[0007] Based on the above objectives, one aspect of the present invention provides a video-based traffic accident analysis method, comprising: Extract video features of traffic videos and extract text features of marked traffic scene problems; Training the projector model to align the video features with the text features and concatenate the two to obtain video-question text features; Based on the video-question text features, a predicted answer corresponding to the marked traffic scene question is generated through a large language model, the projector model is updated based on the predicted answer and the corresponding mark, and the step of extracting video features and text features is returned until a preset condition is met; The cause of the traffic accident is analyzed based on the trained projector model and the large language model.

[0008] In some embodiments, the step of training the projector model to align the video feature with the text feature and concatenate the two to obtain the video-question text feature includes: Based on the dimension of the text feature, training a projector model to expand the video feature into a feature with the same dimension as the text feature, so as to align the video feature with the text feature; The aligned video features and text features are concatenated to obtain video-question text features.

[0009] In some embodiments, the step of updating the projector model based on the predicted answer and the corresponding tag, and returning to the step of extracting video features and text features until a preset condition is met, includes: Establishing a loss function based on the predicted answer and the true answer corresponding to the marked traffic scenario question; Calculate the loss value corresponding to the current round of training of the projector model based on the loss function; Based on the loss value, the parameters of the projector model are updated through a back propagation algorithm, and the step of extracting video features and text features is returned until the loss value is less than a preset threshold or the maximum number of training times is reached, thereby obtaining a trained projector model.

[0010] In some embodiments, the step of analyzing the cause of the traffic accident based on the trained projector model and the large language model includes: Based on traffic accident-related questions and traffic videos collected in the target area, answers corresponding to the traffic accident-related questions are obtained through the trained projector model and the large language model to analyze whether a traffic accident has occurred in the target area and the cause of the accident.

[0011] In some embodiments, the marked traffic scene question is selected from a traffic scene question and answer set, and the process includes: A traffic scenario question-answer set is constructed based on historical traffic accident videos and historical normal traffic videos collected by roadside equipment, and each traffic scenario question and its corresponding true answer are marked; Select a marked traffic scenario question from the traffic scenario question and answer set.

[0012] In some embodiments, the step of extracting video features of traffic video includes: Extract image features corresponding to the frame sequences of traffic videos; The features of each image are concatenated to obtain the video features.

[0013] In some embodiments, the step of combining the image features to obtain the video features includes: The features of each image are spliced ​​according to the time sequence and spatial position to obtain the video features.

[0014] In some embodiments, the step of performing feature splicing of each image feature according to time sequence and spatial position to obtain video features includes: Concatenate the first features of the image features that do not contain position information in series according to the time sequence to obtain the time sequence features corresponding to the frame sequence of the traffic video; Performing spatial pooling operations on the remaining second features containing position information in each of the image features to obtain spatial features corresponding to the frame sequence of the traffic video; The temporal features and the spatial features are concatenated to obtain video features.

[0015] In some embodiments, the step of performing spatial pooling operations on the remaining second features containing position information in each of the image features to obtain spatial features corresponding to the frame sequence of the traffic video includes: The remaining second features containing position information in each of the image features are respectively subjected to an average pooling operation or a maximum pooling operation to obtain spatial features corresponding to the frame sequence of the traffic video.

[0016] In some embodiments, the step of extracting image features corresponding to the frame sequences of the traffic video includes: The image features corresponding to the frame sequences of the traffic video are extracted through the image encoder in the graph pre-training model or the image-text image pre-training model.

[0017] In some embodiments, the step of extracting image features corresponding to the frame sequences of the traffic video includes: The traffic video frame sequences are extracted from traffic participants through the target detection model and spliced ​​into foreground frame sequences; Based on the foreground frame sequence, image features corresponding to the frame sequences of the traffic video are extracted.

[0018] In some embodiments, the step of generating a predicted answer corresponding to the marked traffic scenario question by a large language model includes: Generate a predicted answer unit sequence corresponding to the labeled traffic scenario question through a large language model; The predicted answer unit sequence is combined and concatenated through a text de-tokenizer to obtain the corresponding predicted answer.

[0019] Another aspect of the embodiment of the present invention further provides a video-based traffic accident analysis device, comprising: The first module is used to extract video features of traffic videos and text features of marked traffic scene problems; The second module is used to train the projector model to align the video features with the text features and concatenate the two to obtain video-question text features; The third module is used to generate a predicted answer corresponding to the marked traffic scene question through a large language model based on the video-question text features, update the projector model based on the predicted answer and the corresponding mark, and return to the step of extracting video features and text features until a preset condition is met; The fourth module is used to analyze the cause of the traffic accident based on the trained projector model and the large language model.

[0020] Another aspect of an embodiment of the present invention provides an electronic device, comprising at least one processor; and a memory, wherein the memory stores computer instructions that can be executed on the processor, and when the instructions are executed by the processor, the steps of the above-mentioned video-based traffic accident analysis method are implemented.

[0021] In another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video-based traffic accident analysis method are implemented.

[0022] The present invention has at least the following beneficial effects: a video-based traffic accident analysis method of the present invention extracts the video features of the traffic video and the text features of the marked traffic scene questions respectively, trains the projector model to align and splice the two, obtains the video-question text features, generates the predicted answers corresponding to the marked traffic scene questions through the large language model based on the video-question text features, updates the projector model based on the predicted answers and the corresponding tags, and returns to the step of extracting the video features and the text features until the preset conditions are met, and analyzes the cause of the traffic accident based on the trained projector model and the large language model. During the entire traffic accident analysis process, the projector model is trained to continuously learn the relationship between the given traffic video and the traffic scene question-answer pair, and the powerful context understanding ability of the large language model is used to make the predicted answer outputted as close to the real answer as possible, so that the traffic accident can be accurately identified and the cause of the traffic accident can be obtained, thereby improving the accuracy of the traffic accident analysis. At the same time, the entire process only needs to train the projector model, which reduces the computational consumption required for training, and is conducive to improving the accuracy and efficiency of traffic accident analysis.

[0023] Compared with the existing video question-answering methods, in the entire traffic accident analysis process of the present application, video features and text features only need to be spliced ​​once, eliminating the processing of multiple splicing and splitting, and only the projector model is trained separately, without the need to coordinate multiple models for complex training, thereby reducing the complexity of model training in the entire traffic accident analysis process and reducing the amount of model calculation. At the same time, the projector model is trained specifically to learn the relationship between a given traffic video and a traffic scene question-answer pair, so that the final output predicted answer is as close to the real answer as possible. The learning objectives and processes are more focused and clear, reducing the risk of overfitting to local features of the training data, effectively reducing the overfitting problem, and focusing more on optimizing the projector model, so that the interpretability of the entire model system is enhanced, which is conducive to improving the accuracy and efficiency of traffic accident analysis.

[0024] In addition, the video-based traffic accident analysis device, electronic device and medium of the present invention also have at least the above beneficial effects, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A flowchart of a video-based traffic accident analysis method provided for the first embodiment of the present invention; Figure 2 A schematic diagram of a model corresponding to the video-based traffic accident analysis method provided in the second embodiment of the present invention; Figure 3 A schematic diagram of the network architecture of an image and text encoder of a CLIP model provided in a second embodiment of the present invention; Figure 4 A schematic structural diagram of a video-based traffic accident analysis device provided in a third embodiment of the present invention; Figure 5 A schematic structural diagram of an electronic device provided in accordance with a fourth embodiment of the present invention; Figure 6 A schematic structural diagram of a computer-readable storage medium provided for the fifth embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present invention. However, it should be understood that the disclosed embodiments are merely examples and other embodiments can take various alternative forms.

[0028] Furthermore, it should be noted that the terms "includes," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but may also include elements not explicitly listed or inherent to these processes, methods, articles, or apparatuses.

[0029] One or more embodiments of the present application will be described below with reference to the accompanying drawings.

[0030] Based on the above objectives, a first aspect of an embodiment of the present invention provides an embodiment of a video-based traffic accident analysis method. Figure 1 FIG. 1 is a flow chart of a video-based traffic accident analysis method provided by an embodiment of the present invention, such as Figure 1 As shown, including: S1, extracting video features of traffic videos, and extracting text features of marked traffic scene problems; Specifically, the video features of a traffic video refer to the features that characterize the traffic video, and the text features of a marked traffic scene question refer to the features that characterize the traffic scene question. A marked traffic scene question refers to a traffic scene question that has a true answer marked to it. For example, if the traffic scene question is "Are all vehicles driving normally?", the true answer marked to it is "All vehicles are driving normally and no traffic accidents have occurred."

[0031] Specifically, please refer to Figure 2, extract the video features of the traffic video, and collect the traffic video V=[I1, I2, …, I N ], N represents the number of video frames, N is an integer greater than 0. During the training process, the collected traffic video is divided into multiple training batches, corresponding to multiple rounds of training for the projector model. Each batch includes multiple continuous frame sequences (continuous image sequences) for one round of training for the projector model. The video feature extraction process before each round of training for the projector model is as follows: extract the image features of the corresponding batch of continuous frame sequences respectively, such as extracting through an image pre-training model, or extracting through an image encoder in an image text pre-training model, such as the image encoder of the CLIP (Contrastive Language-Image Pre-Training) model. It is worth noting that if the image text pre-training model is used to extract image features, only the image encoder therein is needed, and the text encoder therein is not needed. Among them, the extracted image features include object category features, object position features, scene layout features, visual appearance features (shape, color, etc.), dynamic features (motion trajectory, speed acceleration, etc.), etc. After that, the extracted image features are spliced ​​to obtain the corresponding video features, so as to input the video features into the projector model and perform one round of training for the projector model.

[0032] Specifically, please refer to Figure 2, extract the text features of the marked traffic scene problems, for example, extract the text features of the question "Are all vehicles driving normally?" It can be understood that the method of extracting text features can be based on the existing pre-trained deep learning network extraction, can be based on word vector extraction, or can be based on the overall text feature extraction of the deep learning model, etc. This application does not limit the method of extracting text features. The text features of the extracted traffic scene problems are also used for the training of the projector model. It can be understood that when using the projector model for training, there are many feasible ways to arrange the training data related to the traffic scene. For example, a specific traffic scene problem can be selected, and multiple rounds of training can be carried out on the projector model based on the problem, and the model's processing ability for this specific problem can be strengthened by continuously repeating the training process. In addition, it is also possible to prepare multiple different traffic scene problems, and then use all of these problems in a round of training of the projector model, so that the model can be exposed to a variety of traffic scene situations in one round of training, and broaden its cognitive scope of different traffic scene problems. Another way is to replace a new traffic scenario problem with the projector model in each round of training. In this way, the model will face different traffic scenario challenges in each round of training, which will help improve its generalization ability to deal with various traffic scenario problems.

[0033] S2, train the projector model to align the video features with the text features and concatenate the two to obtain the video-question text features; Specifically, please refer to Figure 2 The projector model is a neural network model, and its architecture is based on the existing neural network model architecture, such as the transform architecture, which is not shown in detail here. According to the dimension of the text feature, the projector model is trained to expand the video feature into a feature with the same dimension as the text feature to align the video feature with the text feature, and then the aligned video feature and text feature are concatenated to obtain the video-question text feature.

[0034] S3, based on the video-question text features, generate the predicted answer corresponding to the marked traffic scene question through the large language model, update the projector model based on the predicted answer and the corresponding tag, and return to the step of extracting video features and text features until the preset conditions are met; Specifically, the Large Language Model (LLM) includes Shusheng Puyu, Xingchen, etc. Please refer to Figure 2,In this step, the large language model is pre-trained, and the video-question text features are input into the trained large language model.,Based on the common sense knowledge and powerful context understanding ability of the large language model,,it analyzes whether a traffic accident occurred in a given traffic video and the main cause of the accident,,and generates the predicted answer corresponding to the labeled traffic scene question through the large language model.

[0035] Specifically, the projector model is updated by predicting the answer and the real answer corresponding to the mark of the traffic scene question, and the step of extracting video features and text features is returned until the preset conditions are met. For example, the traffic scene question is "Are all vehicles driving normally?", and the real answer corresponding to the mark is "All vehicles are driving normally and no traffic accidents have occurred", while the predicted answer output by the large language model is "The vehicle ahead is driving normally". Obviously, there is a difference between the predicted answer and the real answer. The parameters of the projector model are updated according to the difference between the predicted answer and the real answer corresponding to the mark until the preset conditions are met to obtain the trained projector model. By training the projector model, the predicted answer output by the large language model is made as close as possible to the corresponding real answer.

[0036] S4. Analyze the causes of traffic accidents based on the trained projector model and large language model.

[0037] Specifically, for a target area that needs to be monitored and analyzed for the cause of a traffic accident, text features of traffic scene problems related to the target area are extracted, and at the same time, traffic videos of the target area are collected, such as collecting multiple continuous traffic video clips. The traffic video clip is first input into an image pre-training model or an image text pre-training model (such as a CLIP model), and the image features corresponding to each frame of the traffic video clip are first preliminarily extracted through the image pre-training model or the image text pre-training model, and then input into a trained projector model, and the video features of the traffic video clip aligned with the text features are obtained through the trained projector model to obtain aligned video-question text features, and then input into a trained large language model, and the large language model is used to obtain the predicted answers corresponding to the traffic accident-related questions in the target area, and based on the obtained predicted answers, it is analyzed whether a traffic accident has occurred in the target area and the cause of the traffic accident.

[0038] In the above-mentioned video-based traffic accident analysis method, during the entire traffic accident analysis process, the video features and text features are only spliced ​​once, avoiding the process of multiple splicing, splitting and splicing of the video features and text features, reducing the complexity of processing the video features and text features, training the projector model to continuously learn the relationship between the given traffic video and the traffic scene question-answer pair, and using the powerful context understanding ability of the large language model, so that the predicted answer output in the end is as close to the real answer as possible, realizing the most accurate analysis of the cause of the traffic accident, and improving the accuracy of the traffic accident analysis. The entire process only needs to train the projector model, and there is no need to coordinate multiple models for complex training, which reduces the complexity of model training in the entire traffic accident analysis process and reduces the model calculation amount. At the same time, the projector model is trained specifically for the relationship between the given traffic video and the traffic scene question-answer pair, so that the predicted answer output in the end is as close to the real answer as possible, and the learning goal and process are more focused and clear, reducing the risk of overfitting to the local features of the training data, effectively reducing the overfitting problem, and focusing more on optimizing the projector model, so that the interpretability of the entire model system is enhanced, which is conducive to improving the accuracy and efficiency of traffic accident analysis.

[0039] According to several embodiments of the present invention, the step of training a projector model to align video features with text features and concatenate the two to obtain video-question text features includes: based on the dimensions of the text features, training the projector model to expand the video features into features with the same dimensions as the text features to align the video features with the text features; and concatenating the aligned video features and text features in series to obtain video-question text features.

[0040] Specifically, please refer to Figure 2 In each round of training of the projector model, based on the dimension of the extracted text features, the projector model is trained to expand the extracted video features into features with the same dimension as the text features, so that the video features and text features can be operated in the same dimension. For example, in the above, the text features of the question "Are all vehicles driving normally?" are extracted. Based on the dimension of the text features, the projector model is trained to expand the extracted video features into video features with the same dimension as the text features in this round of training. Only when the dimensions of the video features and text features are the same can the two be spliced ​​in series to obtain the video-question text features. In this way, the video and question text features input into the large language model are aligned, so that the large language model can more accurately capture the intrinsic semantic connection between the video content and the question asked, and obtain a more accurate predicted answer.

[0041] According to some embodiments of the present invention, the projector model is updated based on the predicted answer and the corresponding mark, and the step of extracting video features and text features is returned until a preset condition is met, including: establishing a loss function based on the predicted answer and the real answer corresponding to the marked traffic scene question; calculating the loss value corresponding to the current round of training of the projector model based on the loss function; based on the loss value, updating the parameters of the projector model through the back propagation algorithm, and returning to the step of extracting video features and text features until the loss value is less than a preset threshold or the maximum number of training times is reached, thereby obtaining a trained projector model.

[0042] Specifically, please refer to Figure 2 , a loss function is established based on the corresponding predicted answers and the corresponding marked true answers of each round of the projector model, and the loss value corresponding to the current round of training of the projector model is calculated based on the loss function. Based on the loss value, the parameters of the projector model are updated through the back propagation algorithm until the preset conditions are met, that is, until the loss value is less than the preset threshold or the maximum number of training times is reached. For example, the true answer to the question "Are all vehicles driving normally?" is "All vehicles are driving normally and no traffic accidents have occurred". After multiple rounds of training of the projector model, the corresponding predicted answer output by the large language model is as close as possible to the true answer "All vehicles are driving normally and no traffic accidents have occurred", so as to obtain a trained projector model. During the entire traffic accident analysis process, only the projector model is trained, and the parameters of the projector model are updated by the value corresponding to the loss function constructed by the predicted answer and the true answer, so as to accurately control the training of the projector model with the expected performance and avoid the problem of overtraining or undertraining.

[0043] According to some embodiments of the present invention, the step of analyzing the cause of a traffic accident based on a trained projector model and a large language model includes: based on traffic accident-related questions and traffic videos collected in a target area, obtaining answers corresponding to traffic accident-related questions through a trained projector model and a large language model to analyze whether a traffic accident has occurred in the target area and the cause of the accident.

[0044] Specifically, obtain traffic videos of the target area where traffic accident analysis is required, such as traffic videos collected continuously over a period of time, such as videos recorded continuously during the morning rush hour (such as 7 to 9 a.m.) of a certain road section. In order to improve the efficiency of outputting analysis results, divide it into multiple continuous video clips. For example, if the original video is one hour long, we can evenly divide it into six continuous video clips according to the standard of one clip every ten minutes. Obtain some traffic scene problems related to traffic accidents in the target area, such as constructing traffic scene problems based on historical traffic accident video sets and historical normal traffic video sets collected by roadside equipment. Based on the aforementioned trained projector model, the large language model is a pre-trained large language model. Specifically, input each frame of the traffic video clip into an image text pre-training model or an image pre-training model, such as the image encoder of the CLIP model, extract the image features corresponding to each frame and splice them to obtain video features. Then, the video features are input into the trained projector model, and the video features aligned with the text features are obtained through the trained projector model. The aligned video features and text features are concatenated in series to obtain the video-question text features and output. Then, the video-question text features output from the trained projector model are input into the large language model, and the predicted answer corresponding to the traffic scene question is output through the large language model. The specific process of outputting the predicted answer corresponding to the traffic scene question through the large language model is: the predicted answer unit token sequence is generated through the large language model, and then the corresponding predicted answer is obtained by combining and splicing through the text de-tokenizer. Finally, according to the traffic scene question and its corresponding predicted answer, it is analyzed whether a traffic accident occurs in the target area. If so, the cause of the traffic accident is analyzed according to the predicted answer. Through the trained projector model and the large language model, it is possible to judge whether a traffic accident occurs in the target area, and analyze the cause of the traffic accident in a timely manner, which is conducive to assisting the traffic accident handling process.

[0045] According to some embodiments of the present invention, the step of extracting video features of a traffic video includes: extracting image features corresponding to frame sequences of the traffic video; and splicing the image features to obtain video features.

[0046] Specifically, as mentioned above, before a round of training of the projector model, it is necessary to extract the video features of the corresponding batch of continuous frame sequences. The specific process is to input the frame sequence of the corresponding batch of traffic videos into the image pre-training model, or into the image encoder in the image text pre-training model (such as the image encoder of the CLIP model), respectively extract the image features corresponding to each frame, and then splice the extracted image features in chronological order to obtain video features. The present application extracts video features from a continuous frame sequence, extracts the corresponding image features from each frame, and then splices them in chronological order to obtain video features. It can retain all the detailed information in the video to the greatest extent, which is conducive to the subsequent analysis of more accurate causes of traffic accidents, and is conducive to improving the accuracy and efficiency of traffic accident analysis. At the same time, when performing subsequent traffic accident cause analysis, the video can be flexibly analyzed at any time slice as needed, without being restricted to fixed time points.

[0047] Specifically, please refer to Figure 2 The image pre-training model or the image text pre-training model is a pre-trained model. For example, the image encoder of the trained CLIP model is used to extract the image features of the corresponding batch of continuous frame sequences, and then the corresponding video features are obtained by splicing. In other words, the Image Encoder in the CLIP model is used as the Visual Encoder for extracting video features in this application. Each frame of the image can extract a corresponding feature through the image encoder of the CLIP model. The process is as follows: F i =Encoder(I i )(Formula 1) The extracted F i It consists of two parts, one of which is a global feature that does not contain location information. , the rest is the image block feature containing location information The corresponding image features are spliced ​​to obtain video features. By extracting image features and splicing them to obtain video features, traffic accidents can be analyzed based on videos rather than single images, which is conducive to improving the accuracy of traffic accident analysis.

[0048] According to some embodiments of the present invention, the step of splicing the image features to obtain the video features includes: splicing the image features according to the time sequence and the spatial position to obtain the video features.

[0049] Specifically, as mentioned above, please refer to Figure 2 , because each extracted image feature consists of two parts, one of which is a global feature that does not contain location information , and the other part is the remaining image block features containing location information . Global Features Used to characterize the temporal order of image features, image block features Used to characterize the spatial location of image features through global features and image patch features , which can fully characterize the temporal and spatial position relationship of image features. By splicing the image features corresponding to each image according to the temporal sequence and spatial position to obtain video features, the video features can contain both temporal information and spatial position information, so that the video features can more accurately reflect the corresponding traffic video.

[0050] According to several embodiments of the present invention, the step of splicing the features of each image according to the time sequence and spatial position to obtain the video feature includes: splicing the first features of each image feature that do not contain position information in series according to the time sequence to obtain the time sequence features corresponding to the frame sequence of the traffic video; performing spatial pooling operations on the remaining second features containing position information in each image feature to obtain the spatial features corresponding to the frame sequence of the traffic video; splicing the time sequence features and the spatial features to obtain the video feature.

[0051] Specifically, as mentioned above, please refer to Figure 2 , the global features of each image feature are transformed in time sequence (i.e. the first feature) is concatenated in series to obtain the temporal features T corresponding to a batch of continuous frame sequences, and the formula is as follows: T= … (Formula 2) in, Indicates feature splicing processing.

[0052] The remaining image block features (i.e., the second features) of each image feature containing position information are spatially pooled to obtain a batch of spatial features S corresponding to a series of continuous frame sequences, and the formula is as follows: S=Pooling( , ,…, ) (Formula 3) Among them, Pooling represents the spatial pooling operation, and Pooling is a pooling layer.

[0053] Finally, the obtained temporal features T and spatial features S are concatenated to obtain video features. The projector model is trained to obtain video features T corresponding to the text features. v , so that T v and the text features T of the labeled traffic scene problem q Alignment, the formula is as follows: T v=Projector([T S]) (Formula 4) The complete process of training the projector model, that is, according to the dimension of the text feature, the projector model is trained to expand the video feature into features with the same dimension as the text feature, so as to align the video feature with the text feature, and then the aligned video feature and text feature are concatenated in series to obtain the video-question text feature [T v T q ]. For the video features obtained after splicing, the time information and spatial position information of the original frame sequence are retained. Among them, the second feature space pooling operation can make the video features retain the key information including the position information in the original frame sequence.

[0054] According to some embodiments of the present invention, the steps of performing spatial pooling operations on the remaining second features containing position information in each image feature to obtain spatial features corresponding to the frame sequence of the traffic video include: performing average pooling operations or maximum pooling operations on the remaining second features containing position information in each image feature to obtain spatial features corresponding to the frame sequence of the traffic video.

[0055] Specifically, as in the aforementioned formula 3, the pooling operation includes an average pooling operation or a maximum pooling operation. As two frequently used pooling operations, the average pooling operation or the maximum pooling operation reduces the data dimension by operating on the data in the local area, thereby reducing the subsequent calculation amount and improving the subsequent processing efficiency.

[0056] According to several embodiments of the present invention, labeled traffic scenario questions are selected from a traffic scenario question and answer set, and the process is: constructing a traffic scenario question and answer set based on historical traffic accident videos and historical normal traffic videos collected by roadside equipment, and marking each traffic scenario question and its corresponding true answer; selecting labeled traffic scenario questions from the traffic scenario question and answer set.

[0057] Specifically, a traffic scene question-and-answer set is constructed based on the historical traffic accident video set and the historical normal traffic video set collected by the roadside equipment. For example, the question is "Are all vehicles driving normally?" The answer is "All vehicles are driving normally and no traffic accidents have occurred." For another example, "Why did the black car collide with the pedestrian?" The answer is "The black car did not give way to the pedestrian", etc. It can be understood that the present application does not restrict the construction method of constructing the traffic scene question-and-answer set, and a variety of existing methods of constructing training sets can be used for construction. For example, artificial intelligence question-and-answer tools (such as large pre-trained language models, etc.) can be used to obtain a traffic scene question-and-answer set with stronger generalization ability, and generate a traffic scene question-and-answer set containing more similar content. Each traffic scene question and its corresponding true answer are marked, and the marked traffic scene questions are selected as a batch of training text data. It can be understood that when a batch is trained, generally one marked traffic scene question is selected for training, but multiple marked traffic scene questions can also be selected for training at the same time. Traffic scenario question and answer sets are constructed in a variety of different ways and used as training sets for the entire process. They include questions and answers about the causes of traffic accidents, as well as questions and answers related to normal traffic, covering as many different traffic scenarios as possible and expanding the applicability of traffic accident analysis.

[0058] According to some embodiments of the present invention, the step of extracting image features corresponding to the frame sequences of the traffic video includes: extracting image features corresponding to the frame sequences of the traffic video through an image encoder in an image pre-training model or an image-text pre-training model.

[0059] Specifically, the image pre-training model or the image text pre-training model are pre-trained models. Image text features such as CLIP (Contrastive Language-Image Pre-Training) model, please Figure 2 Based on this, further reference Figure 3 , Figure 3 The network architecture of the image and text encoder of the CLIP model is given. During the training process, the training set D = {I i , T i}, i=1...N, represents N <image, text> pairs in a training batch. By training the CLIP model, the features of paired text-image samples are made close, and the features of unpaired text-image samples are made far away. The formula is as follows: (I i ) (T i ) (I i ) (T j ) (Formula 5) Among them I i represents the i-th image, T i Represents the i-th text, for example, Figure 3 A text "This is Mild dementia" is shown in and They represent the image encoder and text encoder respectively, which are used to map the image and text to a common subspace respectively.

[0060] This embodiment only uses the image encoder of the trained CLIP model to extract image features corresponding to the corresponding batch of video frame sequences, so as to extract features that are closer to the text description of the video. The extracted image features are first preliminarily aligned with the text description of the video, and then spliced ​​to obtain the corresponding video features, which is conducive to improving the overall efficiency.

[0061] According to some embodiments of the present invention, the step of extracting image features corresponding to the frame sequences of the traffic video includes: extracting traffic participants from the frame sequences of the traffic video through a target detection model, and splicing them into a foreground frame sequence; based on the foreground frame sequence, extracting image features corresponding to the frame sequences of the traffic video.

[0062] Specifically, the traffic video V corresponding to the Region Of Interest (ROI) collected by the roadside equipment can be selected according to the actual situation. roi =ROI(v), for example, find areas unrelated to the road in advance, such as the sky, buildings, etc., use semantic segmentation models (such as Mask2Former, etc.) to remove areas unrelated to the road, and only keep the parts related to traffic participants. Use the target detection model to detect traffic participants (such as motor vehicles, pedestrians, bicycles, etc.) and generate the foreground video V in a jigsaw way. obj =Detection([I1,I2,…,I N ]), and then extract the corresponding image features from the foreground frame sequence of the foreground video. By extracting the image features corresponding to the frame sequence of the traffic video by only including the foreground frame sequence of the traffic participants, it is beneficial to reduce the computational complexity, reduce the amount of data, and also help improve the efficiency of traffic accident analysis.

[0063] According to several embodiments of the present invention, the step of generating a predicted answer corresponding to a labeled traffic scene question through a large language model includes: generating a predicted answer unit sequence corresponding to the labeled traffic scene question through a large language model; combining and splicing the predicted answer unit sequence through a text de-tagmer to obtain a corresponding predicted answer.

[0064] Specifically, please refer to Figure 2 , the video-question text features are input into the large language model, and the predicted answer unit token sequence corresponding to the marked traffic scene question is generated by the large language model, and then combined and spliced ​​through the text de-tokenizer to obtain the corresponding predicted answer. This application directly reuses the existing large language model to reduce the training process of the large language model. Only the projector model needs to be trained, which reduces the training computing power requirements.

[0065] For a further understanding of the video-based traffic accident analysis method of the present invention, please continue to refer to Figure 2-3 , which is described in detail in a specific embodiment below.

[0066] Obtain traffic videos within the perception range collected by roadside equipment, divide the traffic videos into multiple continuous training batches in chronological order, and divide each batch into a continuous video frame sequence (continuous image sequence) to perform multiple rounds of training on the projector model in batches.

[0067] At the same time, a traffic scene question-answering set is constructed based on the historical traffic accident video set and the historical normal traffic video set collected by the roadside equipment. For example, the question is "Are all vehicles driving normally?" The answer is "All vehicles are driving normally and no traffic accidents occur." Another example is "Why did the black car collide with the pedestrian?" The answer is "The black car did not give way to the pedestrian." In this embodiment, a labeled traffic scene question is selected to train the projector model as an example.

[0068] In each round of training, the video features corresponding to a batch of continuous frame sequences are extracted, and the text features corresponding to the selected marked traffic scene problems are extracted. The process of extracting video features and extracting text features does not affect each other. Taking one round of training of the projector model as an example, it can be understood that in practical applications, the projector model needs to be trained for multiple rounds to make the final predicted answer close to the real answer. The specific process is as follows, (1) Extracting video features. In a round of training, a batch of continuous frame sequences (image sequences) are respectively input into the image encoder of the CLIP model, and the corresponding image features are extracted through the image encoder. The pre-trained CLIP model makes the paired text-image sample features close and the unpaired text-image sample features far away. Its principle formula is as shown in the above formula 5, which will not be repeated here. In this embodiment, the image encoder (Image Encoder) in the CLIP model is used as the video encoder (Visual Encoder) for extracting video features in this application. Each frame of the image is passed through the image encoder of the CLIP model to extract a corresponding feature F.i , F i The extraction formula of F is as above formula 1, which will not be repeated here. i It consists of two parts, one of which is a global feature that does not contain location information. , the rest is the image block feature containing location information The corresponding image features are concatenated to obtain video features.

[0069] Specifically, the global features of each image feature are transformed into The temporal features T corresponding to a batch of continuous frame sequences are obtained by concatenating them in series. The formula is as described in Formula 2 above and will not be repeated here. Then, the remaining image block features containing position information of each image feature are spatially pooled to obtain a batch of spatial features S corresponding to a batch of continuous frame sequences. The formula is as described in Formula 3 above and will not be repeated here. The temporal features and spatial features are concatenated T S, the video features are obtained, and the video features obtained at this time have not been aligned with the text features. For example, the dimension of the spatial feature S is 5*5*256, and the dimension of the temporal feature T is 5*256. The spatial feature S is expanded from the plane matrix by row or column to obtain a 25*256 feature vector, which is the same dimension as the temporal feature T. Then, the feature vectors are concatenated to obtain the video features (30*256).

[0070] (2) Extracting text features of the marked traffic scene problems. In this embodiment, the text features of the marked traffic scene problems are extracted by a text tokenizer.

[0071] Furthermore, based on the aforementioned extracted video features, the projector model is trained to obtain the video features T corresponding to the text features. v , T v and the text features T of the labeled traffic scene problem q Alignment, the formula is as above formula 4, no longer repeated. According to the dimension of text features, the projector model is trained to expand the video features into features with the same dimension as the text features, so as to align the video features with the text features, and then the aligned video features and text features are concatenated in series to obtain the video-question text features [T v T q ]. The video-question text features [T v T q ] is input into the large language model, which generates a token sequence of predicted answer units corresponding to the marked traffic scenario questions, and then combined and spliced ​​through the text de-tokenizer to obtain the corresponding predicted answer.

[0072] In a round of training of the projector model, a loss function is established based on the predicted answer and the corresponding marked true answer, and the loss value corresponding to the current round of training of the projector model is calculated based on it. Based on the loss value, the parameters of the projector model are updated through the back propagation algorithm. Until the loss value is less than the preset threshold or the maximum number of training times is reached, a trained projector model is obtained. Through the training of the projector model, the corresponding predicted answer output by the large language model is made as close to the true answer as possible, and finally a trained projector model is obtained.

[0073] In the scenario of using the trained projector model and the large language model to analyze traffic accidents in the target area, based on the traffic videos collected in the target area and the questions related to the traffic accidents, the trained projector model and the large language model can be used to obtain the predicted answers corresponding to the questions related to the traffic accidents in the collected traffic videos. The predicted answers can be used to analyze whether a traffic accident has occurred in the target area and the cause of the traffic accident.

[0074] A second aspect of the embodiments of the present invention provides a video-based traffic accident analysis device. Figure 4 FIG. 1 is a schematic diagram of a video-based traffic accident analysis device provided by an embodiment of the present invention, such as Figure 4 As shown, it includes: a first module 101, which is used to extract video features of traffic videos and text features of marked traffic scene problems; a second module 102, which is used to train a projector model to align video features with text features and splice the two to obtain video-question text features; a third module 103, which is used to generate predicted answers corresponding to marked traffic scene problems through a large language model based on the video-question text features, update the projector model based on the predicted answers and corresponding tags, and return to the step of extracting video features and text features until a preset condition is met; a fourth module 104, which is used to analyze the cause of the traffic accident based on the trained projector model and the large language model.

[0075] According to another aspect of the present invention, an electronic device is provided. The electronic device may be a server. Figure 5As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the video-based traffic accident analysis method described above is implemented.

[0076] Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium, and its internal structure diagram is shown in FIG. Figure 6 The computer readable storage medium stores a computer program that, when executed by a processor, performs the above-mentioned video-based traffic accident analysis method.

[0077] Finally, it should be noted that a person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program of the method for setting system parameters can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program can be a disk, an optical disk, a read-only storage memory (ROM) or a random access memory (RAM), etc. The above-mentioned computer program embodiment can achieve the same or similar effect as any of the above-mentioned method embodiments corresponding thereto.

[0078] In addition, the method disclosed in the embodiment of the present invention can also be implemented as a computer program executed by a processor, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the above functions defined in the method disclosed in the embodiment of the present invention are performed.

[0079] In addition, the above method steps and system units may also be implemented using a controller and a computer-readable storage medium for storing a computer program that enables the controller to implement the above steps or unit functions.

[0080] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0081] In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted through a computer-readable medium as one or more instructions or codes. Computer-readable media include computer storage media and communication media, which include any media that facilitate the transfer of computer programs from one location to another. The storage medium may be any available medium that can be accessed by a general or special-purpose computer. As an example and not by way of limitation, the computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the required program code in the form of instructions or data structures and can be accessed by a general or special-purpose computer or a general or special-purpose processor. In addition, any connection may be appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves are used to send software from a website, server, or other remote source, the above-mentioned coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are all included in the definition of the medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, Blu-ray disc, etc., where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0082] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0083] It should be understood that, as used herein, the singular forms "a", "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations including one or more of the associated listed items.

[0084] The serial numbers of the embodiments disclosed in the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0085] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0086] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A video-based traffic accident analysis method, characterized in that: include: Extract video features of traffic videos and extract text features of marked traffic scene problems; Training the projector model to align the video features with the text features and concatenate the two to obtain video-question text features; Based on the video-question text features, a predicted answer corresponding to the marked traffic scene question is generated through a large language model, the projector model is updated based on the predicted answer and the corresponding mark, and the step of extracting video features and text features is returned until a preset condition is met; The cause of the traffic accident is analyzed based on the trained projector model and the large language model.

2. The video-based traffic accident analysis method according to claim 1, characterized in that: The step of training the projector model to align the video features with the text features and concatenate the two to obtain video-question text features includes: Based on the dimension of the text feature, training a projector model to expand the video feature into a feature with the same dimension as the text feature, so as to align the video feature with the text feature; The aligned video features and text features are concatenated to obtain video-question text features.

3. The video-based traffic accident analysis method according to claim 1, characterized in that: The step of updating the projector model based on the predicted answer and the corresponding mark, and returning to the step of extracting video features and text features until a preset condition is met, comprises: Establishing a loss function based on the predicted answer and the true answer corresponding to the marked traffic scenario question; Calculate the loss value corresponding to the current round of training of the projector model based on the loss function; Based on the loss value, the parameters of the projector model are updated through a back propagation algorithm, and the step of extracting video features and text features is returned until the loss value is less than a preset threshold or the maximum number of training times is reached, thereby obtaining a trained projector model.

4. The video-based traffic accident analysis method according to claim 3, characterized in that: The step of analyzing the cause of the traffic accident based on the trained projector model and the large language model includes: Based on traffic accident-related questions and traffic videos collected in the target area, answers corresponding to the traffic accident-related questions are obtained through the trained projector model and the large language model to analyze whether a traffic accident has occurred in the target area and the cause of the accident.

5. The video-based traffic accident analysis method according to claim 1, characterized in that: The marked traffic scenario questions are selected from the traffic scenario question-answer set, and the process includes: A traffic scenario question-answer set is constructed based on historical traffic accident videos and historical normal traffic videos collected by roadside equipment, and each traffic scenario question and its corresponding true answer are marked; The marked traffic scenario question is selected from the traffic scenario question and answer set.

6. The video-based traffic accident analysis method according to claim 1, characterized in that: The step of extracting video features of the traffic video comprises: Extract image features corresponding to the frame sequences of traffic videos; The features of each image are concatenated to obtain the video features.

7. The video-based traffic accident analysis method according to claim 6, characterized in that: The step of splicing the image features to obtain the video features includes: The features of each image are spliced ​​according to the time sequence and spatial position to obtain the video features.

8. The video-based traffic accident analysis method according to claim 7, characterized in that: The step of performing feature splicing of each image feature according to the time sequence and spatial position to obtain the video feature includes: Concatenate the first features of the image features that do not contain position information in series according to the time sequence to obtain the time sequence features corresponding to the frame sequence of the traffic video; Performing spatial pooling operations on the remaining second features containing position information in each of the image features to obtain spatial features corresponding to the frame sequence of the traffic video; The temporal features and the spatial features are concatenated to obtain video features.

9. The video-based traffic accident analysis method according to claim 8, characterized in that: The step of performing spatial pooling operations on the remaining second features containing position information in each of the image features to obtain spatial features corresponding to the frame sequence of the traffic video includes: The remaining second features containing position information in each of the image features are respectively subjected to an average pooling operation or a maximum pooling operation to obtain spatial features corresponding to the frame sequence of the traffic video.

10. The video-based traffic accident analysis method according to claim 6, characterized in that: The step of extracting image features corresponding to the frame sequences of the traffic video includes: The image features corresponding to the frame sequences of the traffic video are extracted through the image encoder in the image pre-training model or the image-text pre-training model.

11. The video-based traffic accident analysis method according to claim 6, characterized in that: The step of extracting image features corresponding to the frame sequences of the traffic video includes: The traffic video frame sequences are extracted from traffic participants through the target detection model and spliced ​​into foreground frame sequences; Based on the foreground frame sequence, image features corresponding to the frame sequences of the traffic video are extracted.

12. The video-based traffic accident analysis method according to claim 1, characterized in that: The step of generating a predicted answer corresponding to the marked traffic scenario question by using a large language model comprises: Generate a predicted answer unit sequence corresponding to the labeled traffic scenario question through a large language model; The predicted answer unit sequence is combined and concatenated through a text de-tokenizer to obtain the corresponding predicted answer.

13. A video-based traffic accident analysis device, characterized in that: The device comprises: The first module is used to extract video features of traffic videos and text features of marked traffic scene problems; The second module is used to train the projector model to align the video features with the text features and concatenate the two to obtain video-question text features; The third module is used to generate a predicted answer corresponding to the marked traffic scene question through a large language model based on the video-question text features, update the projector model based on the predicted answer and the corresponding mark, and return to the step of extracting video features and text features until a preset condition is met; The fourth module is used to analyze the cause of the traffic accident based on the trained projector model and the large language model.

14. An electronic device, characterized in that: include: at least one processor; and a memory storing computer instructions executable on the processor, wherein the instructions, when executed by the processor, implement the steps of the video-based traffic accident analysis method according to any one of claims 1 to 12.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the video-based traffic accident analysis method as described in any one of claims 1-12 are performed.

Images