A method and system for traffic incident recognition based on large models in video.

Abstract

A method and system for traffic event recognition based on a large model using video is disclosed. The method includes: training a pre-defined attention-based neural network model using historical traffic videos as input data and corresponding classification information as output data; training a pre-defined image-text encoder using keyframe data from historical traffic videos as input data and corresponding encoded data as output data; fusing and encoding the classification information and encoded data in a projection layer to obtain fused data; training a pre-defined large language model using the fused data as input data; and inputting real-time traffic videos into the combined model to obtain fine-grained classification information corresponding to the real-time traffic videos. This method achieves finer-grained traffic event recognition, better meeting current traffic management needs.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a method and system for traffic event recognition based on large models of videos. Background Technology

[0002] Effective and timely identification of traffic incidents has become a crucial aspect of urban traffic management. With the development of video surveillance and computer vision technologies, video-based traffic incident identification technology has gradually become an important tool for traffic management.

[0003] Current traffic incident recognition technologies mainly rely on traditional computer vision algorithms, such as image segmentation, feature extraction, and pattern recognition. While these methods can detect traffic incidents to some extent, they are limited to basic events, such as vehicles stopping or pedestrians trespassing. For complex traffic scenarios, such as road maintenance and rescue operations, these methods often struggle to identify them accurately because they require a deep understanding of the video, which exceeds the capabilities of traditional computer vision algorithms. Therefore, current traffic incident recognition technologies have a low degree of granularity in classifying traffic incidents and are insufficient to meet current traffic management needs. Summary of the Invention

[0004] This application provides a method and system for traffic event recognition based on a large model in video, which enables more granular traffic event recognition and better meets the current needs of traffic management.

[0005] In a first aspect, this application provides a method for traffic event recognition based on a large model for video. The method includes: training a preset attention-based neural network model using historical traffic videos as input data and classification information corresponding to the historical traffic videos as output data, so that the preset attention-based neural network model has the ability to classify traffic videos; training a preset image-text encoder using keyframe data of historical traffic videos as input data and encoded data corresponding to the keyframe data of historical traffic videos as output data, so that the preset image-text encoder has the ability to encode based on the keyframes of traffic videos; inputting the classification information and encoded data into a projection layer for fusion and encoding to obtain fused data; using the fused data as input data to train a preset large language model; and inputting real-time traffic videos into a combined model to obtain fine classification information corresponding to the real-time traffic videos. The combined model is composed of a preset large language model connected in series with a preset attention-based neural network model and a preset image-text encoder based on a projection layer.

[0006] In the above embodiments, by using a pre-trained attention-based neural network model and an image-text encoder, richer and more representative features can be extracted from the video, as well as a deep understanding of key frames. This will enhance the model's ability to understand complex traffic scenarios and improve the accuracy of traffic event recognition. Furthermore, by training a large language model, the model can translate complex traffic scenarios into corresponding event classifications, thereby achieving more granular traffic event recognition and better meeting the needs of traffic management.

[0007] In conjunction with some embodiments of the first aspect, in some embodiments, after inputting real-time traffic video into a combination model to obtain fine classification information corresponding to the real-time traffic video, the combination model is composed of a preset large language model based on a projection layer, a preset attention-based neural network model, and a preset image-text encoder respectively. The method further includes: taking the real-time traffic video as historical traffic video and the fine classification information as classification information, and jumping to the step of training the preset attention-based neural network model with historical traffic video as input data and the classification information corresponding to the historical traffic video as output data.

[0008] In the above embodiments, when a new real-time traffic video is received, it is treated as a historical traffic video, and its corresponding fine classification information is used as classification information for further training. This enables the combined model to have better adaptability and recognition ability for new traffic events, achieving the effect of real-time updating and optimization of the combined model.

[0009] In conjunction with some embodiments of the first aspect, in some embodiments, real-time traffic video is used as historical traffic video and fine classification information is used as classification information. The process then jumps to a step of training a preset attention-based neural network model using historical traffic video as input data and the classification information corresponding to the historical traffic video as output data. Specifically, this includes: if it is determined that the real-time traffic video is an abnormal event video, saving the abnormal event video; after a preset time interval, using the abnormal event video as historical traffic video and the fine classification information corresponding to the abnormal event video as classification information, the process then jumps to a step of training a preset attention-based neural network model using historical traffic video as input data and the classification information corresponding to the historical traffic video as output data.

[0010] In the above embodiments, when a real-time traffic video is identified as an abnormal event video, the abnormal event video will be saved and used as historical traffic video for further training after a preset time interval, so as to make the updating and optimization of the combined model more targeted.

[0011] In conjunction with some embodiments of the first aspect, in some embodiments, real-time traffic video is input into a combination model to obtain fine classification information corresponding to the real-time traffic video. The combination model consists of a preset large language model based on a projection layer, a preset neural network model based on an attention mechanism, and a preset image-text encoder, respectively. Specifically, it includes: determining whether there is an anomaly in the road in the real-time traffic video; if an anomaly is found, the real-time traffic video is input into the combination model to obtain fine classification information corresponding to the real-time traffic video. The combination model consists of a preset large language model based on a projection layer, a preset neural network model based on an attention mechanism, and a preset image-text encoder, respectively.

[0012] In the above embodiment, it is necessary to first determine whether there is any abnormality in the road in the real-time traffic video. If there is an abnormality, further judgment is made on the combined model to reduce the number of calculations of the combined model and save on usage costs.

[0013] In conjunction with some embodiments of the first aspect, in some embodiments, the model architecture of the preset attention-based neural network model includes:

[0014] The input layer receives the complete location of each frame of the historical traffic video, which is obtained by segmenting each frame of the historical traffic video.

[0015] The intermediate layer set is connected to the output of the input layer and is used to perform attention calculations on all positions to obtain the transformation sequence. The intermediate layer set includes one or more intermediate layers. When the intermediate layer set consists of multiple intermediate layers, the intermediate layers are connected end to end in sequence to form the intermediate layer set.

[0016] A fully connected layer, connected to the output of the intermediate layer set, is used to convert the transformed sequence into scores for each category;

[0017] The normalized exponential function layer, connected to the output of the fully connected layer, is used to convert the scores of each category into the probabilities of each category.

[0018] The output layer, connected to the output of the normalized exponential function layer, is used to output the probabilities of each category.

[0019] In the above embodiments, the preset attention-based neural network model can better understand and process historical traffic videos, perform deep learning on them, thereby improving the ability to identify traffic incidents and improving the accuracy and efficiency of traffic management.

[0020] In conjunction with some embodiments of the first aspect, in some embodiments, the intermediate layer includes:

[0021] The temporal attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer. It is used to perform attention calculations on the same position of multiple frames of images to obtain the first sequence.

[0022] The spatial attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer. It is used to perform attention calculations on all positions of the same frame image to obtain the second sequence.

[0023] A feedforward neural network, connected to the output of a temporal attention layer or a spatial attention layer, is used to perform a nonlinear transformation on a first sequence and a second sequence to obtain a transformed sequence.

[0024] In the above embodiments, the intermediate layer includes a temporal attention layer and a spatial attention layer. A neural network model based on an attention mechanism can be preset so that the model performs attention calculations on the same position in multiple frames of images and all positions in the same frame of images, thereby achieving a deep understanding of the images and improving the ability to identify complex traffic events.

[0025] In conjunction with some embodiments of the first aspect, in some embodiments, the projection layer is a linear transformation layer.

[0026] In the above embodiments, the projection layer enables the combined model to perform simple and efficient data transformation, thereby improving the computational efficiency of the combined model.

[0027] Secondly, this application provides a traffic event recognition system based on a large model for video, which includes a server, and the server includes:

[0028] The first training module is used to train a preset attention-based neural network model with historical traffic videos as input data and the corresponding classification information as output data, so that the preset attention-based neural network model has the ability to classify traffic videos.

[0029] The second training module is used to train a preset image text encoder by using keyframe data from historical traffic videos as input data and the encoded data corresponding to the keyframe data from historical traffic videos as output data, so that the preset image text encoder has the ability to encode based on the keyframes of traffic videos.

[0030] The fusion encoding module is used to input classification information and encoded data into the projection layer for fusion and encoding to obtain fused data;

[0031] The third training module is used to train the preset large language model by using the fused data as input data;

[0032] The module is used to input real-time traffic video into the combined model to obtain fine classification information corresponding to the real-time traffic video. The combined model consists of a preset large language model based on a projection layer, a preset attention-based neural network model, and a preset image text encoder.

[0033] In conjunction with some embodiments of the second aspect, in some embodiments, the server further includes:

[0034] The update module is used to treat real-time traffic video as historical traffic video and fine classification information as classification information, and then jumps to the first training module.

[0035] In conjunction with some embodiments of the second aspect, in some embodiments, the updating module specifically includes:

[0036] The determination submodule is used to save the abnormal event video when the real-time traffic video is determined to be an abnormal event video;

[0037] The update submodule is used to, after a preset time interval, treat the abnormal event video as historical traffic video and the fine classification information corresponding to the abnormal event video as classification information, and then jump to the first training module.

[0038] In conjunction with some embodiments of the second aspect, in some embodiments, the module specifically includes:

[0039] The judgment submodule is used to determine whether there are any abnormalities in the road in the real-time traffic video;

[0040] The submodule is used to input real-time traffic video into the combined model to obtain fine classification information corresponding to the real-time traffic video if an anomaly occurs. The combined model consists of a preset large language model based on a projection layer, a preset neural network model based on an attention mechanism, and a preset image text encoder.

[0041] In conjunction with some embodiments of the second aspect, in some embodiments, the model architecture of the preset attention-based neural network model includes:

[0042] The input layer receives the complete location of each frame of the historical traffic video, which is obtained by segmenting each frame of the historical traffic video.

[0043] The intermediate layer set is connected to the output of the input layer and is used to perform attention calculations on all positions to obtain the transformation sequence. The intermediate layer set includes one or more intermediate layers. When the intermediate layer set consists of multiple intermediate layers, the intermediate layers are connected end to end in sequence to form the intermediate layer set.

[0044] A fully connected layer, connected to the output of the intermediate layer set, is used to convert the transformed sequence into scores for each category;

[0045] The normalized exponential function layer, connected to the output of the fully connected layer, is used to convert the scores of each category into the probabilities of each category.

[0046] The output layer, connected to the output of the normalized exponential function layer, is used to output the probabilities of each category.

[0047] In conjunction with some embodiments of the second aspect, in some embodiments, the intermediate layer includes:

[0048] The temporal attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer. It is used to perform attention calculations on the same position of multiple frames of images to obtain the first sequence.

[0049] The spatial attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer. It is used to perform attention calculations on all positions of the same frame image to obtain the second sequence.

[0050] A feedforward neural network, connected to the output of a temporal attention layer or a spatial attention layer, is used to perform a nonlinear transformation on a first sequence and a second sequence to obtain a transformed sequence.

[0051] In conjunction with some embodiments of the second aspect, in some embodiments, the projection layer is a linear transformation layer.

[0052] Thirdly, embodiments of this application provide a traffic event recognition system based on a large model for video, the system comprising: one or more processors and a memory;

[0053] The memory is coupled to the one or more processors and is used to store computer program code, which includes computer instructions that the one or more processors invoke to cause the large model-based video traffic incident recognition system to perform the methods described in the first aspect and any possible implementation thereof.

[0054] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a server, cause the server to execute the method described in the first aspect and any possible implementation thereof.

[0055] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a server, cause the server to perform the method described in the first aspect and any possible implementation thereof.

[0056] Understandably, the traffic incident recognition system based on a large model for video provided in the second aspect, the traffic incident recognition system based on a large model for video provided in the third aspect, the computer program product provided in the fourth aspect, and the computer storage medium provided in the fifth aspect are all used to execute the traffic incident recognition method based on a large model for video provided in the embodiments of this application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0057] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0058] 1. The traffic event recognition method based on a large model provided in this application can extract richer and more representative features from videos and gain a deep understanding of key frames through a pre-trained neural network model based on an attention mechanism and an image text encoder. This will enhance the model's ability to understand complex traffic scenes and improve the accuracy of traffic event recognition. Furthermore, by training a large language model, the model can translate complex traffic scenes into corresponding event classifications, thereby achieving more granular traffic event recognition and better meeting the needs of traffic management.

[0059] 2. The traffic event recognition method based on a large model provided in this application treats new real-time traffic videos as historical traffic videos and uses their corresponding fine classification information as classification information for further training. This enables the combined model to have better adaptability and recognition ability for new traffic events, achieving the effect of real-time updating and optimization of the combined model.

[0060] 3. The traffic event recognition method based on a large model provided in this application requires first determining whether there are any abnormalities in the road in the real-time traffic video. If there are abnormalities, further judgment is made by combining the model to reduce the number of calculations of the combined model and save on usage costs. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of an information interaction scenario for the traffic incident recognition system based on a large model for video provided in this application.

[0062] Figure 2 This is a flowchart illustrating the traffic incident recognition method based on a large model for video provided in this application.

[0063] Figure 3 Another flowchart illustrating the traffic event recognition method based on a large model for video provided in this application.

[0064] Figure 4A schematic diagram of the modular virtual device for a traffic incident recognition system based on a large model provided in this application.

[0065] Figure 5 A schematic diagram of the physical device for the traffic incident recognition system based on a large model for video provided in this application. Detailed Implementation

[0066] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all of the listed items that may be combined, including one or more of them.

[0067] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0068] refer to Figure 1 , Figure 1 This is a schematic diagram of an information interaction scenario for a traffic incident recognition system based on a large model for video, as provided in this application. The system includes a monitoring system, an alarm platform, and a server.

[0069] It can be seen that the application scenario of this method for traffic event recognition based on large models is based on the recognition and classification of traffic videos. Therefore, in this embodiment, a monitoring system is required to provide traffic videos. The monitoring system is a system that includes multiple video cameras and other related hardware devices. These cameras are designed to capture traffic videos in specific areas (such as roads, traffic intersections, etc.), i.e., real-time traffic videos.

[0070] It is important to know that an alerting platform is an application or service that receives information from a server and issues warnings or notifications when anomalies or specific traffic incidents are detected. These warnings can then be further relayed to relevant personnel or departments so that they can take appropriate action immediately.

[0071] The server can be one or more computers configured to perform traffic event recognition methods based on large models of video.

[0072] The following describes the traffic event recognition method based on a large model for video in this embodiment:

[0073] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the traffic incident recognition method based on a large model for video provided in this application.

[0074] S201. Train a preset attention-based neural network model using historical traffic videos as input data and the corresponding classification information as output data, so that the preset attention-based neural network model has the ability to classify traffic videos.

[0075] It's easy to imagine that a video library would be pre-set before this step. This video library stores a large number of historical traffic videos and corresponding classification information. These historical traffic videos include video clips of various traffic events, such as vehicles stopping on highways, pedestrians running across roads, traffic accidents, maintenance events, etc., without being limited to any specific event. These videos are usually obtained from surveillance systems and undergo preprocessing, such as frame segmentation, compression, and standardization, to meet the input requirements of the subsequent attention-based neural network model.

[0076] More specifically, the video library stores videos of traffic incidents, each video having a tag. The tags are mainly categorized into incidents of vehicles stopping on highways, pedestrians running across, traffic accidents, maintenance incidents, and rescues, etc., without further limitation. This classification information is usually provided by professional data labelers who watch each video clip and label it with one or more category tags based on the video content, such as "incidents of vehicles stopping on highways," "pedestrians running across," "traffic accidents," "maintenance incidents," and "rescue," etc., without further limitation.

[0077] In some embodiments, the neural network model based on the attention mechanism is a Transformer model. Continuing the previous example, a Transformer model was trained using these videos and their corresponding labels, enabling it to convert video event information into text information. In this process, the Transformer mainly performs Time Attention and Space Attention calculations on the multi-frame images of the video (composed of multiple frames of images), and adds a simple fully connected layer and a Softmax layer at the end of the model, giving the model the ability to classify videos.

[0078] Time Attention performs attention calculations on the same location (patch) across multiple frames in a video. Space Attention performs attention calculations on different locations (patch) within the same image in a video.

[0079] Therefore, the pre-defined model architecture for attention-based neural network models includes:

[0080] The input layer receives the complete location of each frame of the historical traffic video, which is obtained by segmenting each frame of the historical traffic video.

[0081] The intermediate layer set is connected to the output of the input layer and is used to perform attention calculations on all positions to obtain the transformation sequence. The intermediate layer set includes one or more intermediate layers. When the intermediate layer set consists of multiple intermediate layers, the intermediate layers are connected end to end in sequence to form the intermediate layer set.

[0082] A fully connected layer, connected to the output of the intermediate layer set, is used to convert the transformed sequence into scores for each category;

[0083] The normalized exponential function layer, connected to the output of the fully connected layer, is used to convert the scores of each category into the probabilities of each category.

[0084] The output layer, connected to the output of the normalized exponential function layer, is used to output the probabilities of each category.

[0085] It is evident that a neural network model based on an attention mechanism can better understand and process historical traffic videos, enabling deep learning and thus improving the ability to identify traffic incidents and enhance the accuracy and efficiency of traffic management.

[0086] In some embodiments, the intermediate layer includes:

[0087] The temporal attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer. It is used to perform attention calculations on the same position of multiple frames of images to obtain the first sequence.

[0088] The spatial attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer. It is used to perform attention calculations on all positions of the same frame image to obtain the second sequence.

[0089] A feedforward neural network, connected to the output of a temporal attention layer or a spatial attention layer, is used to perform a nonlinear transformation on a first sequence and a second sequence to obtain a transformed sequence.

[0090] As can be seen, the intermediate layer includes a temporal attention layer and a spatial attention layer. The preset neural network model based on the attention mechanism enables the model to perform attention calculations on the same position in multiple frames of images and all positions in the same frame of images, thereby achieving a deep understanding of the images and improving the ability to identify complex traffic events.

[0091] S202. Train a preset image-text encoder using keyframe data from historical traffic videos as input data and the encoded data corresponding to the keyframe data from historical traffic videos as output data, so that the preset image-text encoder has the ability to encode based on the keyframes of traffic videos.

[0092] In some embodiments, the preset image text encoder VT is an encoder.

[0093] Keyframe data is typically important frames extracted from historical traffic videos. They contain the main visual information in the video, while the encoded data is usually a description of these keyframe data. For example, they can be textual descriptions of traffic events or tags or metadata about the keyframe content.

[0094] S203. Input the classification information and coding data into the projection layer for fusion and coding to obtain fused data.

[0095] The attention-based neural network model outputs an encoding of temporal information from parsed traffic event videos, while the image-to-text encoder encodes information from individual images obtained by extracting frames from the video. These two modalities (video encoding and image encoding) cannot be directly fused; they must be mapped to a feature representation space for fusion. The projection layer is a simple linear transformation layer that multiplies each feature of the input encoded information by a weight, resulting in a linear transformation. This maps the outputs of the attention-based neural network model and the image-to-text encoder to the same dimension as a large language model.

[0096] As can be seen, the projection layer enables the combined model to perform simple and efficient data transformation, thereby improving the computational efficiency of the combined model.

[0097] S204. Use the fused data as input data to train the preset large language model.

[0098] Training a large language model can yield more refined classification information. For example, pedestrians can be classified as rescue workers or construction workers, vehicles can be classified as rescue vehicles or engineering vehicles, and scenes can be classified as rescue scenes or congestion scenes. No specific classification is required here.

[0099] The fused data is used as input to train a pre-defined Large Language Model (LLM). This Large Language Model can be a Transformer-based model, such as GPT or BERT, designed to process and understand large-scale text data.

[0100] Large Language Models (LLMs) are deep learning models whose primary task is to understand and generate human language. LLMs are trained to predict the next word given a text context. This task is known as language modeling. In this process, LLMs need to learn how to understand and represent semantic features such as word meaning, sentence structure, context, and intonation.

[0101] LLMs can be used for many tasks. For example, LLMs can be used in text classification, sentiment analysis, named entity recognition, and other tasks. In these tasks, LLMs transform the input text into a high-dimensional vector (called an "embedding") that captures the semantic information of the input text. This embedding can then be used for various tasks, such as predicting the classification label and sentiment polarity of the text, without further limitation.

[0102] In other embodiments, after training is completed, the trained large language model, attention-based neural network model, and image text encoder are frozen to prevent changes in the weights of these models, thus avoiding the loss of their original knowledge and the resulting impact on model performance.

[0103] S205. Input the real-time traffic video into the combined model to obtain the fine classification information corresponding to the real-time traffic video. The combined model consists of a preset large language model based on the projection layer, a preset neural network model based on the attention mechanism, and a preset image text encoder.

[0104] The combined model works as follows: First, a preset attention-based neural network model performs preliminary classification of real-time traffic videos and generates preliminary classification information. Then, a preset image-text encoder encodes keyframes of the real-time traffic videos and generates encoded data. Finally, this classification information and encoded data are fused by a projection layer and input into a preset large language model to obtain refined classification information corresponding to the real-time traffic videos.

[0105] It is evident that by using a pre-trained attention-based neural network model and image-text encoder, richer and more representative features can be extracted from videos, along with a deeper understanding of keyframes. This will enhance the model's ability to understand complex traffic scenarios and improve the accuracy of traffic event recognition. Furthermore, by training a large language model, the model can translate complex traffic scenarios into corresponding event classifications, thereby achieving more granular traffic event recognition and better meeting the needs of traffic management.

[0106] The above embodiments achieve more granular traffic event recognition, better meeting current traffic management needs. However, in actual use, traffic events are updated, and older models may no longer meet actual requirements. Therefore, this paper takes one approach to solving the above problem as an example, combining... Figure 3 The illustrated embodiments provide a more detailed description of the embodiments of this application:

[0107] refer to Figure 3 , Figure 3 Another flowchart illustrating the traffic event recognition method based on a large model for video provided in this application.

[0108] S301. Train a preset attention-based neural network model using historical traffic videos as input data and the corresponding classification information as output data, so that the preset attention-based neural network model has the ability to classify traffic videos.

[0109] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in step S201.

[0110] S302. Train a preset image-text encoder using keyframe data from historical traffic videos as input data and the encoded data corresponding to the keyframe data from historical traffic videos as output data, so that the preset image-text encoder has the ability to encode based on the keyframes of traffic videos.

[0111] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in step S202.

[0112] S303. Input the classification information and coding data into the projection layer for fusion and coding to obtain fused data.

[0113] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in step S203.

[0114] S304. Use the fused data as input data to train the preset large language model.

[0115] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in step S204.

[0116] S305. Determine if there are any abnormalities in the road in the real-time traffic video.

[0117] It should be noted that if every frame of real-time traffic video needs to be input into the combined model for judgment, it will consume a lot of computing resources and time. Therefore, the combined model is only needed for fine classification when an anomaly is initially judged, thereby improving efficiency.

[0118] In one embodiment, motion detection is used to determine whether an anomaly has occurred. For each frame of real-time traffic video, the motion difference between the current frame and the previous frame can be calculated. If the motion difference exceeds a preset threshold, it can be determined that an anomaly may have occurred. For example, if a vehicle suddenly stops, its motion difference will increase significantly, and it can be determined that an anomaly has occurred.

[0119] S306. If an anomaly occurs, the real-time traffic video is input into the combined model to obtain the fine classification information corresponding to the real-time traffic video. The combined model consists of a preset large language model based on a projection layer, a preset neural network model based on an attention mechanism, and a preset image text encoder.

[0120] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in step S205.

[0121] It is evident that it is necessary to first determine whether there are any anomalies in the road in the real-time traffic video. If anomalies are found, further judgment is made by the combined model to reduce the number of calculations in the combined model and save on usage costs.

[0122] S307. If it is determined that the real-time traffic video is an abnormal event video, save the abnormal event video.

[0123] The purpose of this is to facilitate subsequent analysis and research. For example, these anomalous events can be studied in greater depth to identify their causes and take measures to prevent similar events from recurring. Simultaneously, these videos of anomalous events can be used to further train and optimize models, improving their accuracy and robustness.

[0124] S308. After a preset time interval, the abnormal event video is used as historical traffic video, and the fine classification information corresponding to the abnormal event video is used as classification information. Then, the process jumps to the step of training a preset neural network model based on the attention mechanism with historical traffic video as input data and classification information corresponding to historical traffic video as output data.

[0125] It is evident that when new real-time traffic videos are received, they are treated as historical traffic videos, and their corresponding fine-grained classification information is used as classification information for further training. This enables the combined model to have better adaptability and recognition capabilities for new traffic events, achieving the effect of real-time updating and optimization of the combined model.

[0126] It is evident that when real-time traffic video is identified as an abnormal event video, the abnormal event video will be saved and used as historical traffic video for further training after a preset time interval, thereby making the updating and optimization of the combined model more targeted.

[0127] The following are device embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the device embodiments of this application, please refer to the method embodiments of this application.

[0128] refer to Figure 4 This application provides a traffic event recognition system based on a large model for video. The system includes a server, which comprises:

[0129] The first training module 401 is used to train a preset attention-based neural network model with historical traffic videos as input data and classification information corresponding to the historical traffic videos as output data, so that the preset attention-based neural network model has the ability to classify traffic videos.

[0130] The second training module 402 is used to train a preset image text encoder with key frame data of historical traffic videos as input data and encoded data corresponding to the key frame data of historical traffic videos as output data, so that the preset image text encoder has the ability to encode according to the key frames of traffic videos.

[0131] The fusion encoding module 403 is used to input classification information and encoding data into the projection layer for fusion and encoding to obtain fused data;

[0132] The third training module 404 is used to train the preset large language model by using the fused data as input data;

[0133] Module 405 is used to input real-time traffic video into the combined model to obtain fine classification information corresponding to the real-time traffic video. The combined model consists of a preset large language model based on a projection layer, a preset attention-based neural network model, and a preset image-text encoder.

[0134] In some embodiments, the server further includes:

[0135] The update module is used to treat real-time traffic video as historical traffic video and fine classification information as classification information, and then jumps to the first training module.

[0136] In some embodiments, the update module specifically includes:

[0137] The determination submodule is used to save the abnormal event video when the real-time traffic video is determined to be an abnormal event video;

[0138] The update submodule is used to, after a preset time interval, treat the abnormal event video as historical traffic video and the fine classification information corresponding to the abnormal event video as classification information, and then jump to the first training module.

[0139] In some embodiments, the module used specifically includes:

[0140] The judgment submodule is used to determine whether there are any abnormalities in the road in the real-time traffic video;

[0141] The submodule is used to input real-time traffic video into the combined model to obtain fine classification information corresponding to the real-time traffic video if an anomaly occurs. The combined model consists of a preset large language model based on a projection layer, a preset neural network model based on an attention mechanism, and a preset image text encoder.

[0142] In some embodiments, the model architecture of the preset attention-based neural network model includes:

[0143] The input layer receives the complete location of each frame of the historical traffic video, which is obtained by segmenting each frame of the historical traffic video.

[0144] The intermediate layer set is connected to the output of the input layer and is used to perform attention calculations on all positions to obtain the transformation sequence. The intermediate layer set includes one or more intermediate layers. When the intermediate layer set consists of multiple intermediate layers, the intermediate layers are connected end to end in sequence to form the intermediate layer set.

[0145] A fully connected layer, connected to the output of the intermediate layer set, is used to convert the transformed sequence into scores for each category;

[0146] The normalized exponential function layer, connected to the output of the fully connected layer, is used to convert the scores of each category into the probabilities of each category.

[0147] The output layer, connected to the output of the normalized exponential function layer, is used to output the probabilities of each category.

[0148] In some embodiments, the intermediate layer includes:

[0149] The temporal attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer. It is used to perform attention calculations on the same position of multiple frames of images to obtain the first sequence.

[0150] The spatial attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer. It is used to perform attention calculations on all positions of the same frame image to obtain the second sequence.

[0151] A feedforward neural network, connected to the output of a temporal attention layer or a spatial attention layer, is used to perform a nonlinear transformation on a first sequence and a second sequence to obtain a transformed sequence.

[0152] In some embodiments, the projection layer is a linear transformation layer.

[0153] This application also discloses a traffic incident recognition system based on a large model for video. (See reference...) Figure 5 This is a schematic diagram of the physical device of the traffic incident recognition system based on a large model for video provided in this application. The server 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0154] The communication bus 502 is used to enable communication between these components.

[0155] The user interface 503 may include a display screen and a camera. Optionally, the user interface 503 may also include a standard wired interface and a wireless interface.

[0156] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0157] The processor 501 may include one or more processing cores. The processor 501 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 505, and by calling data stored in memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 501 and may be implemented as a separate chip.

[0158] The memory 505 may include random access memory (RAM) or read-only memory. Optionally, the memory 505 may include a non-transitory computer-readable storage medium. The memory 505 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. (Refer to...) Figure 5 The memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application for traffic event recognition based on a large model of video.

[0159] exist Figure 5 In the server 500 shown, the user interface 503 is mainly used to provide an input interface for users and obtain user input data; while the processor 501 can be used to call the application for traffic event recognition based on a large model of video stored in the memory 505. When executed by one or more processors 501, the server 500 performs one or more of the methods described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for this application.

[0160] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0161] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0165] The above description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0166] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for traffic event recognition from video based on a large model, characterized in that, The method includes: A preset attention-based neural network model is trained using historical traffic videos as input data and the classification information corresponding to the historical traffic videos as output data, so that the preset attention-based neural network model has the ability to classify traffic videos. A preset image-text encoder is trained using keyframe data from historical traffic videos as input data and encoded data corresponding to the keyframe data from the historical traffic videos as output data, so that the preset image-text encoder has the ability to encode based on the keyframes of traffic videos. The classification information and the encoded data are input into the projection layer for fusion and encoding to obtain fused data; The fused data is used as input data to train a preset large language model; Real-time traffic video is input into a combined model to obtain fine classification information corresponding to the real-time traffic video. The combined model is composed of the preset large language model based on the projection layer, the preset attention-based neural network model, and the preset image text encoder.

2. The method for traffic event recognition based on a large model in video according to claim 1, characterized in that, The method further includes inputting real-time traffic video into a combined model to obtain fine-grained classification information corresponding to the real-time traffic video. The combined model is composed of the preset large language model, which is constructed by concatenating the preset attention-based neural network model and the preset image-text encoder based on the projection layer. Using the real-time traffic video as historical traffic video and the fine classification information as classification information, the process jumps to the step of training a preset attention-based neural network model using historical traffic video as input data and the classification information corresponding to the historical traffic video as output data.

3. The method for traffic event recognition based on a large model in video according to claim 2, characterized in that, The step of using real-time traffic video as historical traffic video and fine-grained classification information as classification information, and then transitioning to the step of training a preset attention-based neural network model using historical traffic video as input data and the classification information corresponding to the historical traffic video as output data, specifically includes: If the real-time traffic video is determined to be an abnormal event video, the abnormal event video is saved; After a preset time interval, the abnormal event video is used as historical traffic video, and the fine classification information corresponding to the abnormal event video is used as classification information. Then, the process jumps to the step of training a preset neural network model based on an attention mechanism with historical traffic video as input data and classification information corresponding to the historical traffic video as output data.

4. The method for traffic event recognition based on a large model in video according to claim 1, characterized in that, The process involves inputting real-time traffic video into a combined model to obtain refined classification information corresponding to the real-time traffic video. The combined model is composed of a preset large language model, based on the projection layer, concatenated with a preset attention-based neural network model and a preset image-text encoder. Specifically, it includes: Determine if there are any anomalies in the road in real-time traffic video; If an anomaly occurs, the real-time traffic video is input into the combined model to obtain fine classification information corresponding to the real-time traffic video. The combined model is composed of the preset large language model based on the projection layer, which is connected in series with the preset attention-based neural network model and the preset image text encoder.

5. The method for traffic event recognition based on a large model in video according to claim 1, characterized in that, The model architecture of the preset attention-based neural network model includes: The input layer is used to receive the complete position of each frame of the historical traffic video, the position being obtained by segmenting each frame of the historical traffic video; An intermediate layer set, connected to the output of the input layer, is used to perform attention calculations on all positions to obtain a transformation sequence. The intermediate layer set includes one or more intermediate layers. When the intermediate layer set consists of multiple intermediate layers, the intermediate layers are connected end to end in sequence to form the intermediate layer set. A fully connected layer, connected to the output of the intermediate layer set, is used to convert the transformed sequence into scores for each category; A normalized exponential function layer, connected to the output of the fully connected layer, is used to convert the scores of each category into probabilities of each category. The output layer, connected to the output of the normalized exponential function layer, is used to output the probabilities of each category.

6. The method for traffic event recognition based on a large model in video according to claim 5, characterized in that, The intermediate layer includes: A temporal attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer, and is used to perform attention calculations on the same position of multiple frames of images to obtain a first sequence. A spatial attention layer is connected to the output of the input layer or the output of the normalization layer of the previous intermediate layer, and is used to perform attention calculations on all positions of the same frame image to obtain a second sequence. A feedforward neural network, connected to the output of the temporal attention layer and the output of the spatial attention layer, is used to perform a nonlinear transformation on the first sequence and the second sequence to obtain a transformed sequence.

7. The method for traffic event recognition based on a large model in video according to claim 1, characterized in that, The projection layer is a linear transformation layer.

8. A traffic incident recognition system based on a large model for video, the system comprising a server, characterized in that, The server includes: The first training module is used to train a preset attention-based neural network model with historical traffic videos as input data and classification information corresponding to the historical traffic videos as output data, so that the preset attention-based neural network model has the ability to classify traffic videos. The second training module is used to train a preset image text encoder by using keyframe data from historical traffic videos as input data and the encoded data corresponding to the keyframe data from the historical traffic videos as output data, so that the preset image text encoder has the ability to encode based on the keyframes of traffic videos. The fusion encoding module is used to input the classification information and the encoded data into the projection layer for fusion and encoding to obtain fused data; The third training module is used to train the preset large language model by using the fused data as input data; The module is used to input real-time traffic video into a combined model to obtain fine classification information corresponding to the real-time traffic video. The combined model is composed of the preset large language model based on the projection layer, the preset attention-based neural network model, and the preset image text encoder.

9. A traffic incident recognition system based on a large model for video, characterized in that, include: One or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the traffic event recognition system based on a large model for video to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is run on a traffic incident recognition system based on a large model of video, the system performs the method as described in any one of claims 1-7.

Images