Traffic accident recognition method and device based on multi-modal large language model

Abstract

The present application provides a traffic accident recognition method and device based on a multi-modal large language model. The method comprises: acquiring multi-modal data in a driving state of a target vehicle, wherein the multi-modal data comprises video data, audio data, and sensor data; inputting the multi-modal data into an accident recognition model obtained by training a multi-modal large language model for multi-modal information mining, and determining, on the basis of the mined multi-modal information, whether a traffic accident has occurred to the target vehicle; and if the traffic accident has occurred to the target vehicle, sending alarm information. According to the method, effective capturing of global information in the driving process of a target vehicle can be realized by means of an accident recognition model obtained by training a multi-modal large language model, so that recognition of traffic accidents in different scenes is enhanced, and the recognition accuracy of traffic accidents in different scenes is improved.

Description

Traffic accident identification method and device based on multimodal large language model Technical Field

[0001] The present application relates to the technical field of traffic accident identification, and in particular to a traffic accident identification method and device based on a multimodal large language model, and also to an electronic device and storage medium for executing the traffic accident identification method based on a multimodal large language model. Background Art

[0002] As people's living standards improve, urban traffic environments become increasingly complex, and the incidence of traffic accidents remains high. Traffic accidents not only cause traffic congestion and personal and public property damage, but can even endanger life and cause severe physical and psychological trauma. Therefore, accurate identification of traffic accidents is increasingly important for fleet management and social security. Timely and accurate identification of traffic accidents helps minimize fleet losses. Currently, traditional accident recognition algorithms are limited by factors such as device-side computing power. Models have small parameter counts, simple structures, and limited multimodal feature alignment. In addition, the learned representation information lacks generalization capabilities, cannot capture complex data patterns, has low relationship accuracy, and is prone to excessive false positives and omissions. Technical issues

[0003] In view of this, the embodiments of the present application provide a traffic accident identification method and device based on a multimodal large language model, as well as an electronic device and storage medium for executing the traffic accident identification method based on a multimodal large language model, aiming to better mine multimodal information and improve the accuracy of traffic accident identification in different scenarios. Technical Solutions

[0004] A first aspect of an embodiment of the present application provides a traffic accident identification method based on a multimodal large language model, comprising: obtaining multimodal data of a target vehicle in a driving state, wherein the multimodal data includes video data, audio data, and sensor data; inputting the multimodal data into an accident identification model trained based on the multimodal large language model to perform multimodal information mining, and determining whether a traffic accident occurs with the target vehicle based on the mined multimodal information; and issuing an alarm message if a traffic accident occurs with the target vehicle.

[0005] In one possible implementation, before the step of inputting the multimodal data into an accident recognition model obtained by training a multimodal large language model to perform multimodal information mining, and determining whether the target vehicle has been involved in a traffic accident based on the mined multimodal information, the step also includes: building a multimodal large language model framework; and training the multimodal large language model framework using training data sets in a variety of different description forms to obtain the accident recognition model, wherein the training data sets in a variety of different description forms include training data sets in a dialogue form, training data sets in a detailed description form, and training data sets in an inference form.

[0006] In a possible implementation, the multimodal large language model framework includes: a visual encoding module, a video adapter module, an audio encoding module, an audio adapter module, a Gsensor encoding module, a transformer module and a large language model module, wherein: the visual encoding module is used to extract features from all video frames in the video data to obtain a two-dimensional embedding vector corresponding to each video frame; the video adapter module is used to generate a video encoding vector corresponding to the video data based on the two-dimensional embedding vector corresponding to each video frame and the position encoding information corresponding to each video frame; the audio encoding module is used to extract features from the audio data to obtain dense vectors corresponding to several audio clips; the audio adapter module, Used to generate an audio coding vector corresponding to the audio data based on the dense vector corresponding to each audio segment and the position coding information corresponding to each audio segment; the Gsensor coding module is used to sample the sensor data to obtain a number of sensor reading values, perform one-dimensional convolution projection on each sensor reading value, and obtain a reading sequence; the transformer module is used to perform transformer coding processing on the reading sequence to obtain a transformer coding vector corresponding to the sensor data; the large language model module is used to generate text description information that restores the vehicle driving process based on the video coding vector, the audio coding vector, and the transformer coding vector.

[0007] In one possible implementation, a first linear projection layer is provided in the video adapter module, and a second linear projection layer is provided in the audio adapter module, wherein: the first linear projection layer is used to convert the video encoding vector into a video query vector so as to align the video features with the language model space; and the second linear projection layer is used to convert the audio encoding vector into an audio query vector so as to align the audio features with the language model space.

[0008] In a possible implementation, the multimodal large language model framework further includes a Prompts module, and the Prompts module is used to guide the large language model module to generate text description information.

[0009] In one possible implementation, the step of training the multimodal large language model framework using a training data set in a variety of different description forms to obtain the accident recognition model includes: freezing the visual encoding module, the audio encoding module and the large language model module, and training the first linear projection layer in the video adapter module and the second linear projection layer in the audio adapter module.

[0010] In one possible implementation, after the steps of freezing the visual encoding module, the audio encoding module and the large language model module, and training the first linear projection layer in the video adapter module and the second linear projection layer in the audio adapter module, the method further includes: performing end-to-end adjustment on the parameters of the first linear projection layer, the second linear projection layer and the large language model module in the multimodal large language model framework based on downstream tasks, wherein the downstream tasks include multimodal chat tasks.

[0011] The second aspect of an embodiment of the present application provides a traffic accident identification device based on a multimodal large language model, comprising: an acquisition unit for acquiring multimodal data of a target vehicle in a driving state, wherein the multimodal data includes video data, audio data, and sensor data; an identification unit for inputting the multimodal data into an accident identification model trained based on the multimodal large language model to perform multimodal information mining, and determining whether a traffic accident occurs with the target vehicle based on the mined multimodal information; and an alarm unit for issuing an alarm message if a traffic accident occurs with the target vehicle.

[0012] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the electronic device. When the processor executes the computer program, the processor implements the steps of the traffic accident identification method based on a multimodal large language model provided in the first aspect.

[0013] The fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the traffic accident identification method based on a multimodal large language model provided in the first aspect. Beneficial effects

[0014] Compared with the prior art, the embodiments of the present application have the following advantages: by acquiring multimodal data of the target vehicle's driving state, wherein the multimodal data includes video data, audio data, and sensor data; inputting the multimodal data into an accident recognition model trained based on a multimodal large language model to perform multimodal information mining, and determining whether the target vehicle has been involved in a traffic accident based on the mined multimodal information; and if the target vehicle has been involved in a traffic accident, issuing an alarm message. The above method, through the accident recognition model trained based on the multimodal large language model, can effectively capture global information of the target vehicle during its driving process, enhance traffic accident recognition in different scenarios, and improve the accuracy of traffic accident recognition in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] FIG1 is a flowchart of an implementation method of a traffic accident identification method based on a multimodal large language model provided in an embodiment of the present application;

[0017] FIG2 is a flowchart illustrating a method for implementing a training accident recognition model in a traffic accident recognition method based on a multimodal large language model according to an embodiment of the present application;

[0018] FIG3 is a schematic diagram of a framework of a multimodal large language model in a traffic accident identification method based on a multimodal large language model according to an embodiment of the present application;

[0019] FIG4 is a basic structural block diagram of a traffic accident identification device based on a multimodal large language model provided in an embodiment of the present application;

[0020] FIG5 is a basic structural block diagram of an electronic device provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0021] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0022] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0023] Please refer to Figure 1, which is a flowchart of a traffic accident identification method based on a multimodal large language model provided by an embodiment of the present application. Specifically, it may include the following steps S11 to S12.

[0024] S11: Acquire multimodal data of a target vehicle in a driving state, wherein the multimodal data includes video data, audio data, and sensor data.

[0025] In this embodiment, the multimodal data on the target vehicle's driving state includes video data, audio data, and sensor data. Video data includes, but is not limited to, video data of the target vehicle's front and surrounding environment, and video data from within the target vehicle reflecting the driver's behavior and status. Audio data includes voice data from the driver within the target vehicle. Sensor data is data measured by various sensors configured on the target vehicle for detecting the vehicle's motion state, such as acceleration data.

[0026] Video data can be acquired through cameras and sensors installed on the vehicle. In one specific example, the vehicle's built-in Advanced Driving Assistance System (ADAS) can capture images of the vehicle's front and surrounding environment using an ADAS camera, thereby acquiring video data of the target vehicle's front and surrounding environment. It is understood that based on the information displayed in the video data of the target vehicle's front and surrounding environment, driving scene data such as the vehicle's lane position, the positions of surrounding vehicles, and the positions of surrounding pedestrians can be calculated. In another specific example, the vehicle's built-in Digital Indoor System (DIS) can capture images of the driver's face using a DIS camera, thereby acquiring video data reflecting the driver's behavior and status within the target vehicle. It is understood that based on the information displayed in the video data reflecting the driver's behavior and status within the target vehicle, mental state data such as the driver's facial position, facial key points, driver's expression, and the state of the driver's eye opening can be calculated. In another specific example, vehicle motion can be detected using sensors such as a six-axis sensor and a GPS sensor, thereby acquiring sensor data reflecting the driver's driving behavior, such as sudden acceleration, sudden deceleration, sudden turns, and vehicle speed.

[0027] S12: Inputting the multimodal data into an accident recognition model trained based on a multimodal large language model to perform multimodal information mining, and determining whether the target vehicle has been involved in a traffic accident based on the mined multimodal information.

[0028] In this embodiment, to address the issue of low traffic accident identification accuracy, a multimodal large language model (LLM) is pre-designed. This large language model is used to comprehensively analyze the multimodal data generated during the actual driving of the target vehicle, thereby better mining multimodal information, effectively capturing global information during the target vehicle's driving, and enhancing traffic accident identification in different scenarios. Specifically, the large language model is trained using a large number of multimodal training datasets until convergence is achieved, resulting in an accident identification model. The accident identification model is trained to mine multimodal information based on the multimodal data of the target vehicle, and then determines whether the target vehicle has been involved in a traffic accident based on the mined multimodal information. Traffic accident identification using the accident identification model trained with the large language model can enhance robustness and promote road traffic safety. Specifically, the accident recognition model obtained by training the multimodal large language model can be deployed on the service platform side. Data communication is carried out between the target vehicle and the service platform side, and the multimodal data of the target vehicle is uploaded to the service platform side. The service platform side then performs identification and reasoning to determine whether the target vehicle has been involved in a traffic accident, which can enhance computing power and speed up reasoning.

[0029] S13: If a traffic accident occurs to the target vehicle, an alarm message is issued.

[0030] In this embodiment, when the accident recognition model trained based on a multimodal large language model identifies a target vehicle involved in a traffic accident, an alarm process is automatically executed and an alarm message is issued. Specifically, the alarm process includes but is not limited to sending the target vehicle's alarm information to a designated device or platform. The alarm information can be sent in the form of text messages or voice broadcasts, and includes but is not limited to the target vehicle's location information, the accident risk level, casualties, and other accident information.

[0031] From the above, it can be seen that the traffic accident identification method based on the multimodal large language model provided in the embodiment of the present application can perform a comprehensive analysis of the multimodal data generated during the actual driving process of the target vehicle through the accident identification model obtained by training based on the multimodal large language model, better mine the multimodal information, improve the effective capture of the global information during the driving process of the target vehicle, enhance the recognition of traffic accidents in different scenarios, and improve the accuracy of traffic accident recognition in different scenarios.

[0032] In some embodiments of the present application, please refer to Figure 2, which is a flowchart of a method for training an accident recognition model in a traffic accident recognition method based on a multimodal large language model provided in an embodiment of the present application. As shown in Figure 2, the method may specifically include the following steps S21 to S22.

[0033] S21: Build a multimodal large language model framework;

[0034] S22: Using training data sets in a variety of different description forms to train the multimodal large language model framework to obtain the accident recognition model, wherein the training data sets in a variety of different description forms include training data sets in a dialogue form, training data sets in a detailed description form, and training data sets in an inference form.

[0035] In this embodiment, the network framework of the accident recognition model is built as a large language model framework. It is understandable that a language model is a statistical model used to predict the probability of a series of words in a text sequence. As an important artificial intelligence technology based on artificial neural networks, the Language Model can understand language and predict the next word in a sequence by training on large-scale text data. A large language model (LLM) is a neural network with a large number of adjustable parameters that enables it to learn complex patterns and structures in language. By training a large language model, it is possible to learn the contextual relationships between words, grammatical rules, common phrases and sentence structures, so that coherent and natural text can be generated according to the specified context.

[0036] In a specific example, building a multimodal large language model framework includes, but is not limited to, building a visual encoding module, an audio encoding module, a Gsensor encoding module, a large language model module, and the like. Multiple encoding modules are used to mine and analyze data from multiple modalities, such as video data, audio data, and sensor data, to mine multimodal information. The large language model module then generates a text description based on the mined multimodal information, resulting in a comprehensive text description of the target vehicle's driving process. This text description of the target vehicle's driving process can be used to determine whether the target vehicle has been involved in a traffic accident. For example, the text description can be expressed as follows: Based on the video data, it can be concluded that there are xx vehicles in front of the target vehicle, the time is daytime (or nighttime), the weather is clear (or overcast, cloudy, rainy, etc.), and the driver is driving at a fast (or medium, slow, etc.) speed. The driver of the target vehicle has his eyes closed and is yawning (describe the driver's behavior and the movements he made), and he drove straight through the traffic light (or made a U-turn at an intersection, or turned at a curve, or changed lanes to overtake, or slowed down on a smooth road, etc.). Due to the target vehicle's eyes being closed and yawning, the driver of the target vehicle rear-ended the vehicle in front (or hit or scratched other vehicles or pedestrians), and a dangerous situation appeared in the video, and the screen shook. In the end, the driver stopped the car (or did not stop), and the final conclusion was: the video data describes a minor (or serious) traffic accident, the target vehicle was (or was not) damaged, and the main responsibility for the accident lies with the target vehicle. The description of the above text information is only an example, and this embodiment does not impose any limitation on this.

[0037] In a specific example, when training an accident recognition model based on a multimodal large language model, a plurality of training data sets in different description forms are used to train the multimodal large language model framework to a convergence state to obtain an accident recognition model. Among them, the training data sets in different description forms include training data sets in dialogue form, training data sets in detailed description form, and training data sets in reasoning form. It should be noted that the training data set in dialogue form is generated based on the instruction data set of QA, the training data set in detailed description form is generated based on the instruction data set of Captions, and the training data set in reasoning form is generated based on the instruction data set of Boxes.

[0038] In some embodiments of the present application, please refer to Figure 3, which is a schematic diagram of a multimodal large language model framework in a traffic accident identification method based on a multimodal large language model provided in an embodiment of the present application. As shown in Figure 3, when building the multimodal large language model framework, the multimodal large language model framework can specifically include a visual encoding module, a video adapter module, an audio encoding module, an audio adapter module, a Gsensor encoding module, a transformer module, and a large language model module.

[0039] The Visual Encoder Module (VEM) is used to extract features from all video frames (Frame N) in the video data, obtaining a two-dimensional embedding vector corresponding to each frame. Specifically, while a driver is driving, their vehicle is equipped with devices that support video recording, such as ADAS cameras and DMS cameras. The VEM leverages the contextual relationships between previous and subsequent frames in the video sequence to fully perceive the driver's driving state and the vehicle's driving scene, effectively utilizing temporal and contextual information to improve the performance and efficiency of accident recognition. Specifically, the VEM can utilize the pre-trained vision module from BLIP (Bootstrapping Language-Image Pretraining)-2, consisting of a Vision Transformer (VIT). BLIP-2 leverages pre-trained vision and language models to enhance multimodal performance and reduce training costs. The pre-trained vision model provides high-quality visual representations, while the pre-trained language model provides powerful language generation capabilities. Assuming the video data input to the VEM contains N frames, N two-dimensional embedding vectors can be obtained after processing by the VEM.

[0040] The Video Adapter Module (VAM) is used to generate a video encoding vector corresponding to the video data based on the 2D embedding vector corresponding to each video frame and the position encoding information corresponding to each video frame. The position encoding information is obtained by the position layer in the VAM. Specifically, the VAM is connected to the visual encoding module. Since the 2D embedding vector of each video frame obtained by the visual encoding module does not take into account temporal information, the VAM can incorporate the position encoding information of each video frame as temporal information into the 2D embedding vector of each video frame. The 2D embedding vector of each video frame with position encoding information is input into a video-specific Q-former (Querying Transformer). This video-specific Q-former uses a learnable query vector set to extract visual features from a frozen visual model to generate a video encoding vector corresponding to the video data. It should be noted that the Querying Transformer is a lightweight transformer that can perform visual-language alignment between the frozen visual model and a large language model. Specifically, in order to adapt the video to the input of the LLM model, a first linear projection layer is added to the video adapter module. The first linear projection layer is used to convert the video encoding vector into a video query vector to achieve alignment of video features with the language model space.

[0041] The Audio Encoder Module (AEM) is used to extract features from audio data and obtain dense vectors corresponding to several audio segments (Seg M). Specifically, a pre-trained Imagebind is used as the audio encoder. Imagebind is a method that learns a shared representation space by pairing data from multiple modalities (such as text, audio, depth, IMU, etc.) with images. In this embodiment, the audio data can be uniformly sampled in advance to obtain several audio segments. Then, for each audio segment, a Mel-spectrogram transform is used to convert each audio segment into a spectrogram. The obtained spectrogram is then input into the audio encoding module for feature extraction processing to obtain a dense vector corresponding to each audio segment.

[0042] The Audio Adapter Module (AAM) is used to generate an audio encoding vector corresponding to the audio data based on the dense vector corresponding to each audio segment and the position encoding information corresponding to each audio segment. The position encoding information is obtained through the position layer in the audio adapter module. Specifically, the audio adapter module is connected to the audio encoding module. Similar to the video adapter module, the audio adapter module can incorporate the position encoding information representing the time of each audio segment into the dense vector corresponding to each audio segment, and input the dense vector of the audio segment with the position encoding information into an audio-specific Q-former to obtain the audio encoding vector corresponding to the audio data. Similarly, in order to make the audio adapt to the input of the LLM model, a second linear projection layer is added to the audio adapter module, which converts the audio encoding vector into an audio query vector through the second linear projection layer to achieve alignment of the audio features with the large language model space.

[0043] The Gsensor encoding module is used to sample the sensor data to obtain a number of sensor readings, and then perform a one-dimensional convolution projection on each sensor reading to obtain a reading sequence. Specifically, the sensor data may be an IMU signal (inertial signal) composed of accelerometer and gyroscope measurements on the vehicle's X, Y, and Z axes. In this embodiment, the sensor data can be sampled using a pre-set step size to obtain a number of sensor readings, and then a one-dimensional convolution projection is performed on each sensor reading to obtain a reading sequence, wherein the convolution kernel size used in the one-dimensional convolution projection is 8. It can be understood that this reading sequence serves as a feature of the sensor data and is used to characterize the sensor data.

[0044] The transformer module performs transformer encoding on the reading sequence to obtain the transformer encoding vector corresponding to the sensor data. Transformer encoding allows each reading to be represented secondary to its position in the reading sequence. This incorporates the positional information representing time into the sensor data, resulting in the transformer encoding vector corresponding to the sensor data.

[0045] The Large Language Model Module (LLM Module) is used to generate text description information that restores the vehicle's driving process based on the video coding vector, audio coding vector, and transformer coding vector. Specifically, the Large Language Model Module is connected to the video adapter module, audio adapter module, and transformer module respectively. The Large Language Model Module is the brain of the accident recognition model and is used to perform multimodal tasks, such as obtaining the vectors output by the video adapter module, audio adapter module, and transformer module, and then integrating all the obtained vectors for text description, realizing global information capture by integrating multimodal data such as video data, audio data, and sensor data to obtain a text description information that restores the vehicle's driving process.

[0046] In a specific example, as shown in Figure 3, when building a multimodal large language model framework, the multimodal large language model framework can also include a Prompts module. Specifically, the Prompts module is connected to the large language model module. Prompts is an input form or template that can help the large language model module recall what it has learned during pre-training. In this embodiment, the Prompts module can be used to guide the large language model module to generate text description information.

[0047] In some embodiments of the present application, in the process of training the multimodal large language model framework to obtain an accident recognition model, in order to improve the training efficiency and enable the model to have a certain understanding ability, the visual encoding module, audio encoding module and large language model module in the multimodal large language model framework can be frozen, and only the first linear projection layer in the video adapter module and the second linear projection layer in the audio adapter module can be trained to achieve spatial alignment of video features, audio features and word embedding in the large language model. It is understandable that the video part and the audio part in the multimodal large language model framework are trained separately. Exemplarily, for the video part, the training method can be as follows: freeze the parameters of the visual encoding module and the large language model module, and only change the parameters of the first linear projection layer. The training is to align the visual features extracted by the visual encoding module with the text embedding obtained by the embedding layer of the large language model module. Specifically, the video frame can be input into the visual encoding module, and the text visual information extracted by the linear projection layer can be input into the large language model module. The large language model module is allowed to make the next word prediction and generate the text title corresponding to the video frame. Alternatively, pre-construct some instructions that allow the large language model to simply describe video frame information. During training, sample these instructions and visual features to construct a command prompt and input it into the large language model, allowing it to generate the corresponding text captions for the video frames. The training principles for audio are similar to those for video and are not detailed here.

[0048] In some embodiments of the present application, after training the first linear projection layer in the video adapter module and the second linear projection layer in the audio adapter module by freezing the visual encoding module, the audio encoding module and the large language model module, a specific downstream task is performed on the trained multimodal large language model framework, such as a multimodal chat task. Based on the multimodal chat task, the parameters of the first linear projection layer, the second linear projection layer and the large language model module in the multimodal large language model framework are adjusted end-to-end so that the trained multimodal large language model framework obtains better recognition and description effects. It can be understood that the end-to-end adjustment is a supervised adjustment. In this embodiment, the task execution form of the multimodal chat task includes a dialogue form, a detailed description form, an inference form, etc.

[0049] It can be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0050] In some embodiments of the present application, please refer to Figure 4, which is a basic structural block diagram of a traffic accident identification device based on a multimodal large language model provided in an embodiment of the present application. The various units included in the device in this embodiment are used to execute the various steps in the above-mentioned method embodiment. Please refer to the relevant description in the above-mentioned method embodiment for details. For the convenience of explanation, only the parts related to this embodiment are shown. As shown in Figure 4, the traffic accident identification device based on a multimodal large language model includes: an acquisition unit 41, an identification unit 42 and an alarm unit 43. Among them: the acquisition unit 41 is used to obtain multimodal data of the target vehicle under the driving state, wherein the multimodal data includes video data, audio data and sensor data. The identification unit 42 is used to input the multimodal data into the accident identification model obtained by training based on the multimodal large language model to perform multimodal information mining, and determine whether the target vehicle has a traffic accident based on the mined multimodal information. The alarm unit 43 is used to issue an alarm message if the target vehicle has a traffic accident.

[0051] It should be understood that the above-mentioned traffic accident identification device based on a multimodal large language model corresponds one-to-one to the above-mentioned traffic accident identification method based on a multimodal large language model, and will not be repeated here.

[0052] In some embodiments of the present application, please refer to Figure 5, which is a basic structural block diagram of an electronic device provided in an embodiment of the present application. As shown in Figure 5, the electronic device 5 of this embodiment includes: a processor 51, a memory 52, and a computer program 53 stored in the memory 52 and executable on the processor 51, such as a program for a traffic accident identification method based on a multimodal large language model. When the processor 51 executes the computer program 53, the steps in each embodiment of the above-mentioned traffic accident identification method based on a multimodal large language model are implemented. Alternatively, when the processor 51 executes the computer program 53, the functions of each module in the embodiments corresponding to the above-mentioned traffic accident identification device based on a multimodal large language model are implemented. Please refer to the relevant description in the embodiments for details, which will not be repeated here.

[0053] Exemplarily, the computer program 53 may be divided into one or more modules (units), which are stored in the memory 52 and executed by the processor 51 to implement the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 53 in the electronic device 5. For example, the computer program 53 may be divided into an acquisition unit, an identification unit, and an alarm unit, with the specific functions of each module being as described above.

[0054] The electronic device may include, but is not limited to, a processor 51 and a memory 52. ​​Those skilled in the art will appreciate that FIG5 is merely an example of the electronic device 5 and does not limit the electronic device 5. The electronic device 5 may include more or fewer components than shown in the figure, or may combine certain components or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0055] The processor 51 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0056] The memory 52 can be an internal storage unit of the electronic device 5, such as a hard drive or memory of the electronic device 5. The memory 52 can also be an external storage device of the electronic device 5, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped with the electronic device 5. Furthermore, the memory 52 can include both an internal storage unit of the electronic device 5 and an external storage device. The memory 52 is used to store the computer program and other programs and data required by the electronic device. The memory 52 can also be used to temporarily store data that has been output or is about to be output.

[0057] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0058] The present application also provides a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor, the computer-readable storage medium can implement the steps of the above-mentioned method embodiments. In this embodiment, the computer-readable storage medium can be non-volatile or volatile.

[0059] An embodiment of the present application provides a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal can implement the steps of the above-mentioned various method embodiments when executing the computer program product.

[0060] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0061] If the integrated module / unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0062] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0063] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A traffic accident identification method based on a multimodal large language model, characterized in that: include: Acquiring multimodal data of a target vehicle in a driving state, wherein the multimodal data includes video data, audio data, and sensor data; Inputting the multimodal data into an accident recognition model trained based on a multimodal large language model to perform multimodal information mining, and determining whether the target vehicle has been involved in a traffic accident based on the mined multimodal information; If a traffic accident occurs to the target vehicle, an alarm message is issued.

2. The traffic accident identification method based on a multimodal large language model according to claim 1 is characterized in that: Before the step of inputting the multimodal data into an accident recognition model trained based on a multimodal large language model to perform multimodal information mining, and determining whether the target vehicle has been involved in a traffic accident based on the mined multimodal information, the method further includes: Build a multimodal large language model framework; The multimodal large language model framework is trained using training data sets in a variety of different description forms to obtain the accident recognition model, wherein the training data sets in a variety of different description forms include training data sets in a dialogue form, training data sets in a detailed description form, and training data sets in an inference form.

3. The traffic accident identification method based on a multimodal large language model according to claim 2 is characterized in that: The multimodal large language model framework includes: a visual encoding module, a video adapter module, an audio encoding module, an audio adapter module, a Gsensor encoding module, a transformer module and a large language model module, wherein: The visual encoding module is used to extract features from all video frames in the video data to obtain a two-dimensional embedding vector corresponding to each video frame; The video adapter module is configured to generate a video coding vector corresponding to the video data based on the two-dimensional embedding vector corresponding to each video frame and the position coding information corresponding to each video frame; The audio encoding module is used to extract features from the audio data to obtain dense vectors corresponding to a plurality of audio segments; The audio adapter module is configured to generate an audio coding vector corresponding to the audio data based on the dense vector corresponding to each audio segment and the position coding information corresponding to each audio segment; The Gsensor encoding module is used to sample the sensor data to obtain a number of sensor reading values, and perform a one-dimensional convolution projection on each sensor reading value to obtain a reading sequence; The transformer module is used to perform transformer encoding processing on the reading sequence to obtain a transformer encoding vector corresponding to the sensor data; The large language model module is used to generate text description information that restores the vehicle driving process based on the video coding vector, the audio coding vector and the transformer coding vector.

4. The traffic accident identification method based on a multimodal large language model according to claim 3 is characterized in that: The video adapter module is provided with a first linear projection layer, and the audio adapter module is provided with a second linear projection layer, wherein: The first linear projection layer is used to convert the video encoding vector into a video query vector so as to align the video features with the language model space; The second linear projection layer is used to convert the audio encoding vector into an audio query vector so as to align the audio features with the language model space.

5. The traffic accident identification method based on a multimodal large language model according to claim 3 is characterized in that: The multimodal large language model framework also includes a Prompts module, which is used to guide the large language model module to generate text description information.

6. The traffic accident identification method based on a multimodal large language model according to any one of claims 3 to 5, characterized in that: The step of using a plurality of training data sets with different description forms to train the multimodal large language model framework to obtain the accident recognition model includes: Freeze the visual encoding module, the audio encoding module, and the large language model module, and train the first linear projection layer in the video adapter module and the second linear projection layer in the audio adapter module.

7. The traffic accident identification method based on a multimodal large language model according to claim 6 is characterized in that: After the steps of freezing the visual encoding module, the audio encoding module, and the large language model module, and training the first linear projection layer in the video adapter module and the second linear projection layer in the audio adapter module, the method further includes: Based on downstream tasks, parameters of the first linear projection layer, the second linear projection layer, and the large language model module in the multimodal large language model framework are adjusted end-to-end, wherein the downstream tasks include multimodal chat tasks.

8. A traffic accident identification device based on a multimodal large language model, characterized in that: include: an acquisition unit, configured to acquire multimodal data of a target vehicle under a driving state, wherein the multimodal data includes video data, audio data, and sensor data; an identification unit, configured to input the multimodal data into an accident identification model trained based on a multimodal large language model to perform multimodal information mining, and determine whether the target vehicle has been involved in a traffic accident based on the mined multimodal information; The alarm unit is used to send out an alarm message if the target vehicle has a traffic accident.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images