Large language model-based vehicle theft / hijacking identification method, apparatus, and system, and storage medium

Abstract

The present application relates to the field of safe driving, and in particular to a large language model-based vehicle theft / hijacking identification method, apparatus, and system, and a storage medium. The method comprises: acquiring multi-modal data of a vehicle; extracting an image feature in video data by means of a video encoding module, and extracting an audio feature in audio data by means of an audio encoding module; by means of a video adaptation module, converting the image feature into a video query vector adapted to a large language model; by means of an audio adaptation module, converting the audio feature into an audio query vector adapted to the large language model; and inputting the audio query vector and the video query vector into the large language model, and outputting text description related to vehicle theft / hijacking and corresponding to the video data and the audio data. Due to the large language model used in the method, richer representation information can be learned on the basis of video data and audio data, and the generalization capability is stronger; thus, complex data pattens can be effectively captured, and the identification accuracy can be improved.

Description

Traffic theft and robbery identification method, device, system and storage medium based on large model Technical Field

[0001] The present application relates to the field of safe driving, and in particular to a method, device, system and storage medium for identifying traffic theft and robbery based on a large model. Background Art

[0002] With the development of transportation and logistics, logistics networks span the globe. However, due to inadequate public security in some regions, the property of logistics workers in these areas may be vulnerable. Therefore, timely identification of stolen logistics vehicles and reporting them to the police are becoming increasingly important for fleet management and social security. This is especially true on sparsely populated roads. Accurate and timely identification of vehicle thefts helps minimize losses to fleets.

[0003] Traditional device-based theft and robbery identification algorithms have a small number of model parameters, insufficient learned representation information and generalization capabilities, are unable to capture complex data patterns, have low recognition accuracy, and suffer from excessive false positives or omissions. Technical issues

[0004] In view of this, the embodiments of the present application provide a traffic theft identification method, device, system and storage medium based on a large model to solve the problems of the device-based theft identification algorithm in the existing technology, such as small model parameters, insufficient learned representation information and generalization ability, inability to capture complex data patterns, low recognition accuracy, and excessive false alarms or missed alarms. Technical Solutions

[0005] A first aspect of an embodiment of the present application provides a method for identifying traffic robbery based on a large model, wherein the large model includes a video encoding module, a video adaptation module, an audio encoding module, an audio adaptation module, and a large language model. The method includes:

[0006] Acquiring multimodal data of the vehicle, the multimodal data including video data and audio data;

[0007] Extracting image features from the video data through a video encoding module, and extracting audio features from the audio data through an audio encoding module;

[0008] Converting the image features into a video query vector adapted to a large language model through a video adaptation module;

[0009] Converting the audio features into an audio query vector adapted to a large language model through an audio adaptation module;

[0010] The audio query vector and the video query vector are input into the large language model, and a text description about traffic robbery corresponding to the video data and the audio data is output.

[0011] In combination with the first aspect, in a first possible implementation of the first aspect, the multimodal data includes sensor data;

[0012] Before inputting the audio query vector and the video query vector into the large language model and outputting a text description of traffic robbery corresponding to the video data and the audio data, the method further includes:

[0013] Acquiring sensor data of the vehicle during driving, and performing convolution projection processing on the sensor data to generate projection data;

[0014] The projection data is transformed and coded to obtain coded data of the sensing data.

[0015] Inputting the audio query vector and the video query vector into the large language model, and outputting a text description of traffic robbery corresponding to the video data and the audio data, including:

[0016] The encoded data of the sensor data, the audio query vector, and the video query vector are input into the large language model, and a text description corresponding to the multimodal data is output.

[0017] In conjunction with the first possible implementation of the first aspect, in a second possible implementation of the first aspect, obtaining sensor data of the vehicle during driving includes:

[0018] Collect IMU signals detected by accelerometers and / or gyroscopes;

[0019] Clipping the IMU signal according to a preset time length to obtain an IMU signal segment;

[0020] The IMU signal segments are sampled according to a preset time step to obtain the sensing data.

[0021] In combination with the first aspect, in a third possible implementation of the first aspect, converting the image features into a video query vector adapted by a large language model by a video adaptation module includes:

[0022] embedding position coding information representing time information into the image features;

[0023] transforming the image features into a video coding vector by a query transformer;

[0024] The video encoding vector is converted into a video query vector by adding a linear layer.

[0025] In combination with the first aspect, in a fourth possible implementation manner of the first aspect, before extracting image features from the video data by the video encoding module and extracting audio features from the audio data by the audio encoding module, the method further includes:

[0026] Parameters in the video encoding module and the large language model are frozen, and parameters in the video adaptation module and the audio adaptation module are trained according to audio and video training samples.

[0027] In conjunction with the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, freezing the parameters in the video encoding module and the large language model, and training the parameters in the video adaptation module and the audio adaptation module based on the audio and video training samples, includes:

[0028] Inputting the video data in the audio and video training sample into a video encoding module with frozen parameters to extract image features, and converting the image features into a video query vector through a video adaptation module;

[0029] Inputting audio data in the audio and video training sample into a video encoding module with frozen parameters to extract image features, and converting the image features into an audio query vector through a video adaptation module;

[0030] Input the video query vector and the audio query vector into a large language model and output a text description;

[0031] The output text description is compared with the standard file description in the audio and video training samples, and the parameters in the audio adaptation module and the video adaptation module are adjusted according to the deviation between the two until the deviation meets the preset requirements.

[0032] In combination with the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, the video encoding module includes a first video encoding module corresponding to the first video captured by the advanced driving assistance system, and a second video encoding module corresponding to the second video captured by the camera of the driver monitoring system, and the video adaptation module includes a first video adaptation module corresponding to the first video encoding module, and a second video adaptation module corresponding to the second video encoding module.

[0033] A second aspect of the embodiments of the present application provides a traffic theft and robbery identification device based on a large model, wherein the large model includes a video encoding module, a video adaptation module, an audio encoding module, an audio adaptation module, and a large language model, and the device includes:

[0034] a data acquisition unit, configured to acquire multimodal data of the vehicle, wherein the multimodal data includes video data and audio data;

[0035] an encoding unit, configured to extract image features from the video data through a video encoding module, and extract audio features from the audio data through an audio encoding module;

[0036] A first adaptation unit, configured to convert the image features into a video query vector adapted by a large language model through a video adaptation module;

[0037] A second adaptation unit, configured to convert the audio features into an audio query vector adapted to a large language model through an audio adaptation module;

[0038] An output unit is used to input the audio query vector and the video query vector into the large language model, and output a text description about traffic robbery corresponding to the video data and the audio data.

[0039] In conjunction with the second aspect, in a first possible implementation of the second aspect, the multimodal data includes sensor data;

[0040] The device further comprises:

[0041] A projection unit, configured to obtain sensor data of the vehicle during driving and perform convolution projection processing on the sensor data to generate projection data;

[0042] The transform coding unit is used to perform transform coding on the projection data to obtain coded data of the sensing data.

[0043] The output unit is used to input the encoded data of the sensor data, the audio query vector and the video query vector into the large language model, and output a text description corresponding to the multimodal data.

[0044] With reference to the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the projection unit includes:

[0045] A signal acquisition subunit, used to collect IMU signals detected by the accelerometer and / or gyroscope;

[0046] A clipping subunit, configured to clip the IMU signal according to a preset duration to obtain an IMU signal segment;

[0047] The sampling subunit is used to sample the IMU signal segments according to a preset time step to obtain the sensor data.

[0048] With reference to the second aspect, in a third possible implementation of the second aspect, the first adaptation unit includes:

[0049] an embedding subunit, configured to embed position coding information representing time information into the image feature;

[0050] a transcoding subunit, configured to transform the image features into a video coding vector by querying a transformer;

[0051] The vector conversion subunit is used to convert the video encoding vector into a video query vector by adding a linear layer.

[0052] In combination with the second aspect, in a fourth possible implementation of the second aspect, the apparatus further includes:

[0053] A parameter training unit is used to freeze the parameters in the video encoding module and the large language model, and train the parameters in the video adaptation module and the audio adaptation module according to audio and video training samples.

[0054] In conjunction with the fourth possible implementation manner of the second aspect, in a fifth possible implementation manner of the second aspect, the parameter training unit includes:

[0055] A feature processing subunit, configured to input the video data in the audio and video training sample into a video encoding module with frozen parameters to extract image features, and convert the image features into a video query vector through a video adaptation module;

[0056] An audio processing subunit, configured to input the audio data in the audio and video training sample into a video encoding module with frozen parameters to extract image features, and convert the image features into an audio query vector through a video adaptation module;

[0057] a large language model output subunit, configured to input the video query vector and the audio query vector into a large language model and output a text description;

[0058] The parameter adjustment subunit is used to compare the output text description with the standard file description in the audio and video training samples, and adjust the parameters in the audio adaptation module and the video adaptation module according to the deviation between the two until the deviation meets the preset requirements.

[0059] In combination with the fifth possible implementation of the second aspect, in the sixth possible implementation of the second aspect, the video encoding module includes a first video encoding module corresponding to the first video captured by the advanced driving assistance system, and a second video encoding module corresponding to the second video captured by the camera of the driver monitoring system, and the video adaptation module includes a first video adaptation module corresponding to the first video encoding module, and a second video adaptation module corresponding to the second video encoding module.

[0060] The third aspect of an embodiment of the present application provides a large-model-based traffic theft identification system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0061] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented. Beneficial effects

[0062] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: the embodiments of the present application extract image features of video data by setting a video encoding model, extract audio features of audio data by an audio encoding module, convert image features into video query vectors by a video adaptation module, convert audio features into audio query vectors by an audio adaptation module, input the video query vectors and audio query vectors into a large language model, and output a text description for representing information related to traffic theft, thereby obtaining a traffic theft recognition result based on a large model. Since the large language model used by this method can learn richer representation information based on video data and audio data, it has stronger generalization ability, can effectively capture complex data patterns, and is conducive to improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0064] FIG1 is a schematic diagram of an implementation scenario of a large-scale model-based traffic theft and robbery identification method provided in an embodiment of the present application;

[0065] FIG2 is a schematic diagram of a flow chart of a method for identifying traffic robbery based on a large model according to an embodiment of the present application;

[0066] FIG3 is a schematic diagram of a traffic theft and robbery identification model based on a large model provided in an embodiment of the present application;

[0067] FIG4 is a schematic diagram of a traffic theft and robbery identification device based on a large model provided in an embodiment of the present application;

[0068] FIG5 is a schematic diagram of a traffic theft and robbery identification system based on a large model provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0069] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0070] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0071] With the development of transportation and logistics, logistics networks span the globe. However, due to inadequate public security in some regions, the property of logistics workers in these areas may be vulnerable. Therefore, timely identification of stolen logistics vehicles and reporting them to the police are becoming increasingly important for fleet management and social security. This is especially true on sparsely populated roads. Accurate and timely identification of vehicle thefts helps minimize losses to fleets.

[0072] Traditional device-based theft and robbery identification algorithms are limited by factors such as device-side computing power. The model parameters are small, the learned representation information and generalization capabilities are insufficient, and they are unable to capture complex data patterns. The recognition accuracy is low and there are too many false positives or missed positives.

[0073] Based on this, an embodiment of the present application provides a traffic theft identification method based on a large model to improve the accuracy of identifying theft risks during driving, which is described in detail below.

[0074] Figure 1 is a schematic diagram of an implementation scenario for a large-model-based traffic theft and robbery identification method provided in an embodiment of the present application. As shown in Figure 1 , the implementation scenario includes a cloud and a vehicle terminal. The vehicle terminal includes, among other things, multimodal sensors and an alarm. Multimodal sensors are used to collect multimodal sensor data. Multimodal sensors include one or more of an ADAS (Advanced Driving Assistance System) camera, a DMS (Driver Monitoring System) camera, a microphone, an IMU (Inertial Measurement Unit), and a gravity sensor. Multimodal sensor data includes one or more of video, audio, gravity sensor data, inertial sensor data, and other data. The multimodal sensor data collected by the vehicle terminal can be combined with calibrated text descriptions as training sample data (including audio and video training sample data) and uploaded to the cloud. The cloud uses training sample data uploaded from multiple vehicle terminals, fixes the parameters of the video encoding module, audio encoding module, and large language model, and trains the video adaptation module and audio adaptation module in the large-model-based traffic robbery recognition model. After training, the large-model-based traffic robbery recognition model, which includes the video encoding module, audio encoding module, video adaptation module, audio adaptation module, and large language model, can be used to perform large-model-based traffic robbery recognition, outputting a textual description of the robbery risk identification result. Because the large language model used in this method can learn richer representation information and has stronger generalization capabilities, it can effectively capture complex data patterns, which helps improve recognition accuracy.

[0075] FIG2 is a schematic diagram of a flow chart of a traffic robbery identification method based on a large model provided in an embodiment of the present application, which is described in detail as follows:

[0076] In S201 , multimodal data of a vehicle is acquired, where the multimodal data includes video data and audio data.

[0077] The multimodal data in the embodiments of the present application may include data collected by the vehicle's multimodal sensors, including data collected by one or more of an ADAS (Advanced Driving Assistance System) camera, a DMS (Driver Monitoring System) camera, a driving recorder camera, a microphone, an IMU (Inertial measurement unit) and a gravity sensor.

[0078] In a possible implementation, the multimodal data in the embodiments of the present application may include video data and audio data. The video data may include video data collected by the ADAS camera, video data collected by the DMS camera, and video data collected by the dashcam camera. Since the image data collected by the DMS camera is driver image data inside the vehicle, and the image data collected by the ADAS camera is image data related to the exterior of the vehicle, to improve the accuracy of image recognition, the driver image data and the exterior image data can be collected separately, and encoding and adaptation can be performed separately for the different image data to improve the recognition accuracy of the model.

[0079] In S202, image features in the video data are extracted by a video encoding module, and audio features in the audio data are extracted by an audio encoding module.

[0080] Based on the collected video and audio data, a large-scale model-based traffic theft and robbery identification model can be used to calculate risk identification. Figure 3 is a schematic diagram of the structure of a large-scale model-based traffic theft and robbery identification model provided in an embodiment of the present application. As shown in Figure 3, the large-scale model-based traffic theft and robbery identification model includes a video encoding module, an audio encoding module, a video adaptation module, an audio adaptation module, and a large language model. The audio encoding module, video encoding module, and large language model are neural network models that have been pre-trained using large datasets.

[0081] In real-world application scenarios, traditional image encoders can only focus on the information in a single image and are unable to fully perceive the driver's state by utilizing the contextual relationship between the previous and next frames in the video sequence. While the driver is driving, there are many devices that support video recording, including ADAS cameras and DMS cameras. Therefore, the use of a video encoding module can better learn the driver's state changes, identify and intervene in risks, and utilize contextual information to solve some complex driving actions and scenarios. Therefore, in the driver risk identification method based on video understanding in the embodiment of the present application, for video-related tasks, the introduction of a video encoding module can better utilize time and contextual information, thereby improving the performance and efficiency of the model.

[0082] The video encoding module in the embodiments of this application can utilize a pre-trained vision module from BLIP-2. For example, the video encoding module can include a VIT (Vision Transformer). When training a large-scale traffic robbery recognition model, the parameters in the video encoding module are frozen. A video containing N frames of images, after passing through the video encoding module, can generate N two-dimensional embedding vectors, i.e., the image features of the N frames of images. For a single image, this can be considered as a single-frame video input.

[0083] To better encode different types of videos and improve model recognition accuracy, as shown in Figure 3, the first video captured by the ADAS camera and the second video captured by the DMS camera can be encoded separately. For example, the first video captured by the ADAS camera can be encoded by the first video encoding module to obtain the first image feature; the second video captured by the DMS camera can be encoded by the second video encoding module to obtain the second image feature.

[0084] The audio encoding module in the embodiment of the present application is used to encode the audio data of the vehicle driver.

[0085] When encoding audio data, K audio clips of predetermined length can be collected. A spectral transformation method, such as the Mel-spectrogram transform, is then used to transform the audio clips into spectrograms. The spectrograms are then input into the audio encoding module, which, pre-trained using big data, extracts features and outputs audio features.

[0086] In S203, the image features are converted into video query vectors adapted to a large language model through a video adaptation module.

[0087] The video encoding module extracts image features from video data, and the video adaptation module converts image features into image query vectors to adapt to the input requirements of the large language model.

[0088] Specifically, since the image features from the video encoding module do not yet consider temporal information, positional encoding information can be further embedded and applied to representations from different frames. The frame representations with positional encoding information are input into a video-specific Q-former to obtain video encoding vectors. Furthermore, to adapt the video representation to the input of the Large Language Model (LLM), a linear layer can be added to convert the video encoding vector into a video query vector, aligning the video features with the LLM space.

[0089] In S204, the audio features are converted into audio query vectors adapted to a large language model through an audio adaptation module.

[0090] Similar to the video adaptation module, after the audio encoding module extracts audio features from the audio data, it first embeds the positional encoding information representing the temporal information into the audio clip. The audio features with the positional encoding information are then fed into an audio-specific Q-former to obtain an audio encoding vector. A linear layer is used to convert the audio encoding vector into an audio query vector, which is then mapped into the embedding space of the Large Language Model (LLM) to align the audio features with the LLM space.

[0091] In S205, the audio query vector and the video query vector are input into the large language model, and a text description of traffic robbery corresponding to the video data and the audio data is output.

[0092] The video query vector output by the video adaptation module is aligned with the large language model space, and the audio query vector output by the audio adaptation module is aligned with the large language model space. Therefore, the large language model can be used to effectively process the video data and audio data, and output the descriptive text of traffic theft and robbery identification based on the large model corresponding to the video data and audio data.

[0093] For example, the model of the output description text can be:

[0094] First, we can see that this is a driver of (vehicle type) traveling on (road type). The video shows (number of vehicles) ahead, the time is (daytime / nighttime), the weather is (clear, overcast, partly cloudy, rainy, etc.), and the driver is traveling at a (slow, medium, fast, etc.) speed. Next, we see the driver (describe the driver's behavior, including actions such as driving straight through a traffic light, making a U-turn at an intersection, turning around a curve, changing lanes to overtake, or slowing down on a smooth road). Then, due to (cause type) (the driver rear-ends the vehicle in front, is hit or scratched, hits a pedestrian, or the vehicle rolls over), the video becomes dangerous and the screen shakes. Ultimately, the driver (stops or does not stop). Overall, the video depicts a (minor, major) vehicle theft, based on (based on content).

[0095] Through the large language model, the features in the video data and audio data can be combined to output template-based description text.

[0096] The large language model, video encoding module, and audio encoding module in the embodiments of the present application can be pre-trained using large-scale data. After pre-training, they can be fine-tuned using high-quality instruction data.

[0097] In an embodiment of the present application, training sample data (including video training sample data and audio training sample data) can also be used to train and optimize the parameters in the video adaptation module and the audio adaptation module in the traffic theft and robbery recognition model based on the large model, that is, the parameters in the large language model, the video encoding module and the audio encoding module are fixed or frozen, and the training sample data is calculated through the traffic theft and robbery recognition model based on the large model, and the video data in the audio and video training samples are input into the video encoding module with frozen parameters to extract image features, and the image features are converted into video query vectors through the video adaptation module; the audio data in the audio and video training samples are input into the video encoding module with frozen parameters to extract image features, and the image features are converted into audio query vectors through the video adaptation module; the video query vector and the audio query vector are input into the large language model, and a text description is output. The output text description is compared with the standard text description in the training sample data, and the parameters in the video adaptation module and the audio adaptation module are adjusted according to the deviation between the two until the deviation becomes small to meet the preset requirements, thereby completing the training of the video adaptation module and the audio adaptation module.

[0098] In addition, in a possible implementation, as shown in FIG3 , embodiments of the present application can also be combined with other sensor data, including data collected by sensors such as IMUs, accelerometers, and gravity sensors. The sensor data can be cropped according to a preset time length to obtain sensor data segments, which are then convolved and projected using a convolution kernel of a predetermined size to obtain projection data. The projection data can then be transform-encoded to obtain encoded sensor data.

[0099] When calculating the text description of traffic theft, the encoded data of the sensor data, the audio query vector and the video query vector can be input into the large language model, so that the text description corresponding to the multimodal data can be output more accurately.

[0100] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0101] FIG4 is a schematic diagram of a large-scale model-based traffic theft and robbery identification device provided in an embodiment of the present application. As shown in FIG4 , the device includes:

[0102] A data acquisition unit 401 is configured to acquire multimodal data of a vehicle, wherein the multimodal data includes video data and audio data;

[0103] An encoding unit 402 is configured to extract image features from the video data through a video encoding module and extract audio features from the audio data through an audio encoding module;

[0104] A first adaptation unit 403 is configured to convert the image features into a video query vector adapted by a large language model through a video adaptation module;

[0105] A second adaptation unit 404 is configured to convert the audio features into an audio query vector adapted to a large language model through an audio adaptation module;

[0106] The output unit 405 is configured to input the audio query vector and the video query vector into the large language model, and output a text description of traffic robbery corresponding to the video data and the audio data.

[0107] The large-model-based traffic theft identification device shown in FIG4 corresponds to the large-model-based traffic theft identification method shown in FIG2 .

[0108] Figure 5 is a schematic diagram of a large-scale model-based traffic theft identification system provided in an embodiment of the present application. As shown in Figure 5 , the large-scale model-based traffic theft identification system 5 of this embodiment includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50, such as a large-scale model-based traffic theft identification program. When the processor 50 executes the computer program 52, it implements the steps of each of the aforementioned large-scale model-based traffic theft identification method embodiments. Alternatively, when the processor 50 executes the computer program 52, it implements the functions of each module / unit in the aforementioned device embodiments.

[0109] For example, the computer program 52 may be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 52 in the large-scale model-based traffic theft and robbery identification system 5.

[0110] The large-scale model-based traffic theft identification system 5 can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The large-scale model-based traffic theft identification system can include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5 is merely an example of a large-scale model-based traffic theft identification system 5 and does not limit the large-scale model-based traffic theft identification system 5. The large-scale model-based traffic theft identification system 5 can include more or fewer components than shown, or combinations of certain components, or different components. For example, the large-scale model-based traffic theft identification system can also include input and output devices, network access devices, buses, and the like.

[0111] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0112] The memory 51 can be an internal storage unit of the large-scale model-based traffic theft identification system 5, such as a hard drive or memory of the large-scale model-based traffic theft identification system 5. The memory 51 can also be an external storage device of the large-scale model-based traffic theft identification system 5, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 51 can include both the internal storage unit of the large-scale model-based traffic theft identification system 5 and an external storage device. The memory 51 is used to store the computer program and other programs and data required by the large-scale model-based traffic theft identification system. The memory 51 can also be used to temporarily store data that has been output or is about to be output.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0114] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0115] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0117] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0119] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments through hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0120] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A traffic robbery identification method based on a large model, characterized in that: The large model includes a video encoding module, a video adaptation module, an audio encoding module, an audio adaptation module and a large language model, and the method includes: Acquiring multimodal data of the vehicle, the multimodal data including video data and audio data; Extracting image features from the video data through a video encoding module, and extracting audio features from the audio data through an audio encoding module; Converting the image features into a video query vector adapted to a large language model through a video adaptation module; Converting the audio features into an audio query vector adapted to a large language model through an audio adaptation module; The audio query vector and the video query vector are input into the large language model, and a text description about traffic robbery corresponding to the video data and the audio data is output.

2. The method according to claim 1, characterized in that The multimodal data includes sensory data; Before inputting the audio query vector and the video query vector into the large language model and outputting a text description of traffic robbery corresponding to the video data and the audio data, the method further includes: Acquiring sensor data of the vehicle during driving, and performing convolution projection processing on the sensor data to generate projection data; performing transform coding on the projection data to obtain coded data of the sensing data; Inputting the audio query vector and the video query vector into the large language model, and outputting a text description of traffic robbery corresponding to the video data and the audio data, including: The encoded data of the sensor data, the audio query vector, and the video query vector are input into the large language model, and a text description corresponding to the multimodal data is output.

3. The method according to claim 2, characterized in that Acquire sensor data of the vehicle during driving, including: Collect IMU signals detected by accelerometers and / or gyroscopes; Clipping the IMU signal according to a preset time length to obtain an IMU signal segment; The IMU signal segments are sampled according to a preset time step to obtain the sensing data.

4. The method according to claim 1, wherein The image features are converted into a video query vector adapted to a large language model by a video adaptation module, including: embedding position coding information representing time information into the image features; transforming the image features into a video coding vector by a query transformer; The video encoding vector is converted into a video query vector by adding a linear layer.

5. The method according to claim 1, wherein Before extracting image features from the video data by the video encoding module and extracting audio features from the audio data by the audio encoding module, the method further includes: Parameters in the video encoding module and the large language model are frozen, and parameters in the video adaptation module and the audio adaptation module are trained according to audio and video training samples.

6. The method according to claim 5, characterized in that Freezing parameters in the video encoding module and the large language model, and training parameters in the video adaptation module and the audio adaptation module according to audio and video training samples, including: Inputting the video data in the audio and video training sample into a video encoding module with frozen parameters to extract image features, and converting the image features into a video query vector through a video adaptation module; Inputting audio data in the audio and video training sample into a video encoding module with frozen parameters to extract image features, and converting the image features into an audio query vector through a video adaptation module; Input the video query vector and the audio query vector into a large language model and output a text description; The output text description is compared with the standard file description in the audio and video training samples, and the parameters in the audio adaptation module and the video adaptation module are adjusted according to the deviation between the two until the deviation meets the preset requirements.

7. The method according to claim 6, characterized in that The video encoding module includes a first video encoding module corresponding to the first video captured by the advanced driving assistance system, and a second video encoding module corresponding to the second video captured by the camera of the driver monitoring system. The video adaptation module includes a first video adaptation module corresponding to the first video encoding module, and a second video adaptation module corresponding to the second video encoding module.

8. A traffic theft identification device based on a large model, characterized in that: The large model includes a video encoding module, a video adaptation module, an audio encoding module, an audio adaptation module and a large language model, and the device includes: a data acquisition unit, configured to acquire multimodal data of the vehicle, wherein the multimodal data includes video data and audio data; an encoding unit, configured to extract image features from the video data through a video encoding module, and extract audio features from the audio data through an audio encoding module; A first adaptation unit, configured to convert the image features into a video query vector adapted by a large language model through a video adaptation module; A second adaptation unit, configured to convert the audio features into an audio query vector adapted to a large language model through an audio adaptation module; An output unit is used to input the audio query vector and the video query vector into the large language model, and output a text description about traffic robbery corresponding to the video data and the audio data.

9. A traffic theft and robbery identification system based on a large model, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images