Large model-based driving risk identification method and apparatus, device, and storage medium

Abstract

The present application relates to the field of assisted driving, and in particular to a large model-based driving risk identification method and apparatus, a device, and a storage medium. The method comprises: acquiring individual sample data; and inputting individual multi-modal sensing data into a trained universal risk identification model with fixed parameters, and inputting same into an individual feature optimization channel, performing calculation by means of the universal risk identification model and the individual feature optimization channel to obtain a first risk identification result, comparing the first risk identification result with a first risk calibration result, adjusting parameters of the individual feature optimization channel on the basis of the difference between the first risk identification result and the first risk calibration result until the difference meets a predetermined requirement to obtain a large model comprising the universal risk identification model and the individual feature optimization channel, and preforming risk identification on an individual on the basis of the large model. Driving risk identification is performed on the basis of the large model, individual features can be effectively combined for identification, thereby improving the accuracy of risk identification.

Description

Driving risk identification method, device, equipment and storage medium based on large model Technical Field

[0001] The present application relates to the field of assisted driving, and in particular to a large-scale model-based driving risk identification method, device, equipment and storage medium. Background Art

[0002] With increasing traffic volume and the growing importance of road safety, driving risk identification has become a crucial component of ensuring road safety. Drivers must accurately identify and address various potential risk factors, such as traffic flow, road conditions, driver distraction, and fatigue. Accurate risk identification can help drivers take timely and appropriate measures, reducing the likelihood of accidents and improving road safety.

[0003] When identifying driving risks, a universal risk identification method is often used. However, due to industry differences and diversity, drivers define risks differently in different industries and working conditions. Using a universal risk identification method is not conducive to improving risk identification accuracy. Technical issues

[0004] In view of this, the embodiments of the present application provide a driving risk identification method, device, equipment and storage medium based on a large model to solve the problem in the prior art that is not conducive to improving the accuracy of risk identification. Technical Solutions

[0005] A first aspect of an embodiment of the present application provides a driving risk identification method based on a large model, the method comprising:

[0006] Acquiring individual sample data, the individual sample data including individual multimodal sensing data and a first risk calibration result determined according to individual characteristics;

[0007] The individual multimodal sensing data is input into a trained universal risk identification model with fixed parameters, as well as an individual feature optimization channel. A first risk identification result is calculated based on the universal risk identification model and the individual feature optimization channel. The first risk identification result is compared with the first risk calibration result. The parameters of the individual feature optimization channel are adjusted according to the difference between the two until the difference meets the predetermined requirements. A large model including the universal risk identification model and the individual feature optimization channel is obtained, and risk identification is performed for the individual based on the large model.

[0008] In conjunction with the first aspect, in a first possible implementation of the first aspect, before inputting the multimodal sensing data into a trained universal risk identification model with fixed parameters, the method further includes:

[0009] Acquire general sample data, where the general sample data includes general multimodal sensing data and a second risk calibration result;

[0010] The universal multimodal sensing data is input into the universal risk identification model, a second risk identification result is calculated, the second risk identification result is compared with the second risk calibration result, and the parameters of the universal risk identification model are adjusted according to the difference between the two until the difference meets the predetermined requirements.

[0011] In combination with the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, inputting the universal multimodal sensing data into the universal risk identification model includes:

[0012] Extracting an image feature vector of an input video through a video encoder, and encoding the image feature vector into an image coding vector;

[0013] Position coding information is inserted into the image coding vector through a video adaptation module to generate a video coding vector including time information, the video coding vector is converted into a video query vector through a linear layer, and the video query vector is input into the large language model in the universal risk identification model.

[0014] In combination with the first possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, inputting the universal multimodal sensing data into the universal risk identification model includes:

[0015] The input audio is sampled and encoded by the audio encoder to obtain an audio segment encoding vector;

[0016] The audio adaptation module inserts position coding information into the audio segment coding vector to generate an audio coding vector including time information, and the audio coding vector is mapped to the large language model in the universal risk identification model through a linear layer.

[0017] In combination with the second possible implementation manner or the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the method further includes:

[0018] Gravity sensor data, map positioning data, weather sensor data, and calibrated text embedding information are input into the large language model in the universal risk identification model.

[0019] In combination with the first aspect, in a fifth possible implementation of the first aspect, calculating and obtaining a first risk identification result based on the general risk identification model and the individual feature optimization channel includes:

[0020] The first risk identification result is calculated based on the first weight matrix of the general risk identification model and the sum of the second weight matrices of the individual feature optimization channel as the weight matrix of the large language model.

[0021] In combination with the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, the second weight matrix includes a first matrix and a second matrix, the first matrix is ​​used to perform dimensionality reduction processing on the training layer of the individual feature optimization channel, and the second matrix is ​​used to perform dimensionality increase processing on the training layer of the individual feature optimization channel.

[0022] A second aspect of an embodiment of the present application provides a large model, which includes the individual feature optimization channel and the universal risk identification model described in any one of the first aspects.

[0023] In a third aspect, an embodiment of the present application provides a driving risk identification device based on a large model, the device comprising:

[0024] A data acquisition unit, configured to acquire individual sample data, wherein the individual sample data includes individual multimodal sensing data and a first risk calibration result determined according to individual characteristics;

[0025] A training unit is used to input the individual multimodal sensing data into a trained universal risk identification model with fixed parameters, and an individual feature optimization channel, calculate a first risk identification result based on the universal risk identification model and the individual feature optimization channel, compare the first risk identification result with the first risk calibration result, and adjust the parameters of the individual feature optimization channel according to the difference between the two until the difference meets predetermined requirements.

[0026] In conjunction with the third aspect, in a first possible implementation of the third aspect, the apparatus further includes:

[0027] A general sample data acquisition unit, configured to acquire general sample data, wherein the general sample data includes general multimodal sensing data and a second risk calibration result;

[0028] A universal risk identification model training unit is used to input the universal multimodal sensing data into the universal risk identification model, calculate a second risk identification result, compare the second risk identification result with the second risk calibration result, and adjust the parameters of the universal risk identification model according to the difference between the two until the difference meets predetermined requirements.

[0029] In combination with the first possible implementation manner of the third aspect, in a second possible implementation manner of the third aspect, the general risk identification model training unit includes:

[0030] A first encoding subunit, configured to extract an image feature vector of an input video through a video encoder, and encode the image feature vector into an image coding vector;

[0031] The first adaptation subunit is used to insert position coding information into the image coding vector through a video adaptation module to generate a video coding vector including time information, convert the video coding vector into a video query vector through a linear layer, and input the video query vector into the large language model in the universal risk identification model.

[0032] In conjunction with the first possible implementation of the third aspect, in a third possible implementation of the third aspect, the general risk identification model training unit includes:

[0033] A second encoding subunit, configured to sample and encode the input audio through an audio encoder to obtain an audio segment encoding vector;

[0034] The second adaptation subunit is used to insert position coding information into the audio segment coding vector through the audio adaptation module, generate an audio coding vector including time information, and map the audio coding vector to the large language model in the general risk identification model through a linear layer.

[0035] In combination with the second possible implementation manner or the third possible implementation manner of the third aspect, in a fourth possible implementation manner of the third aspect, the apparatus further includes:

[0036] An input unit is used to input gravity sensor data, map positioning data, weather sensor data and calibrated text embedding information into the large language model in the universal risk identification model.

[0037] In combination with the third aspect, in a fifth possible implementation of the third aspect, the training unit is used to calculate a first risk identification result based on the first weight matrix of the general risk identification model and the sum of the second weight matrices of the individual feature optimization channel as the weight matrix of the large language model.

[0038] In combination with the fifth possible implementation of the third aspect, in the sixth possible implementation of the third aspect, the second weight matrix includes a first matrix and a second matrix, the first matrix is ​​used to perform dimensionality reduction processing on the training layer of the individual feature optimization channel, and the second matrix is ​​used to perform dimensionality increase processing on the training layer of the individual feature optimization channel.

[0039] The fourth aspect of an embodiment of the present application provides a driving risk identification device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in any one of the first aspects are implemented.

[0040] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented. Beneficial effects

[0041] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: the embodiments of the present application obtain individual sample data, input the multimodal sensor data in the individual sample data into a pre-trained, parameter-fixed general risk identification model, and calculate the general risk identification result corresponding to the individual sample data. The risk identification result of the individual characteristics can be calculated according to the individual feature optimization channel. The first risk identification result is obtained according to the general risk identification result and the risk identification result of the individual characteristics. According to the difference between the first risk identification result and the first risk calibration result, the matrix parameters of the individual feature optimization channel are adjusted, and the parameters of the general risk identification model are kept unchanged until the difference is adjusted to meet the preset requirements. Then, a large model including the general risk identification model and the individual feature optimization channel is obtained. Driving risk identification is performed according to the model, and individual characteristics can be effectively combined for identification, which is conducive to improving the accuracy of risk identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] FIG1 is a schematic diagram of an implementation scenario of a driving risk identification method based on a large model provided in an embodiment of the present application;

[0044] FIG2 is a schematic diagram of an implementation flow of a driving risk identification method based on a large model provided in an embodiment of the present application;

[0045] FIG3 is a schematic diagram of a large model provided in an embodiment of the present application;

[0046] FIG4 is a schematic diagram of an implementation flow of inputting video data into a large language model according to an embodiment of the present application;

[0047] FIG5 is a schematic diagram of an implementation flow of inputting audio data into a large language model according to an embodiment of the present application;

[0048] FIG6 is a schematic diagram of an implementation flow of the method provided in an embodiment of the present application;

[0049] FIG7 is a schematic diagram of a driving risk identification device based on a large model provided in an embodiment of the present application;

[0050] FIG8 is a schematic diagram of a driving risk identification device provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0051] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0052] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0053] With increasing traffic volume and the growing importance of road safety, driving risk identification has become a crucial component of ensuring road safety. Drivers must accurately identify and address various potential risk factors, such as traffic flow, road conditions, driver distraction, and fatigue. Accurate risk identification can help drivers take timely and appropriate measures, reducing the likelihood of accidents and improving road safety.

[0054] However, when detecting driving risks, risk identification is usually performed based on pre-defined risk characteristics. However, due to individual differences, risk identification based on pre-defined risk characteristics may lead to misjudgment, which is not conducive to improving the driving experience.

[0055] Based on this, an embodiment of the present application provides a driving risk identification method based on a large model, and a method for risk identification based on a large model trained by this method, so as to improve the accuracy of driving risk identification, which is described in detail below.

[0056] Figure 1 shows a schematic diagram of an implementation scenario for a large-scale model-based driving risk identification method and a hazard identification method provided in an embodiment of the present application. As shown in Figure 1 , the implementation scenario includes a cloud and a vehicle terminal. The vehicle terminal includes multimodal sensors and alarms, among other things. Multimodal sensors are used to collect multimodal sensor data. Multimodal sensors include one or more of an ADAS (Advanced Driving Assistance System) camera, a DMS (Driver Monitoring System) camera, a microphone, an IMU (Inertial Measurement Unit), and a gravity sensor. Multimodal sensor data includes one or more of video, audio, gravity sensor data, and inertial sensor data. The multimodal sensor data collected by the vehicle terminal can be combined with calibration information as sample data and uploaded to the cloud. The cloud uses the sample data uploaded by multiple vehicle terminals to train the general risk identification model in the large-scale model. After training, the parameters of the general risk identification model are fixed. Uploaded individual sample data is used to optimize the parameters of the individual feature optimization channel in the large model to complete the training of the large model. Based on the trained large model, it can more accurately identify risks based on individual differences, improve recognition accuracy, and send alarm alerts to vehicle terminals when risks are identified.

[0057] FIG2 is a schematic diagram of an implementation flow of a driving risk identification method based on a large model provided in an embodiment of the present application, which is described in detail as follows:

[0058] In S201 , individual sample data is acquired, where the individual sample data includes individual multimodal sensing data and a first risk calibration result determined according to individual characteristics.

[0059] The individual sample data in the embodiments of the present application is calibrated data collected by the multimodal sensors of a single vehicle. The collected data includes data collected by one or more of an ADAS (Advanced Driving Assistance System) camera, a DMS (Driver Monitoring System) camera, a microphone, an IMU (Inertial Measurement Unit), and a gravity sensor. The first risk calibration result corresponding to the multimodal sensor data may include a risk result calibrated by the vehicle driver.

[0060] The individual sample data in the embodiment of the present application can be individual sample data determined based on the customer's standards, that is, individual characteristics, after the large model including the risk identification model and the individual characteristic optimization channel is deployed.

[0061] For example, individual sample data can be identified as matching the customer's standards through the customer's labeling operation. Alternatively, the customer's standards can be learned through the customer's click operation. For example, the driving actions taken by the customer in the same multimodal data can be learned, and the customer's personalized sample data can be determined based on the number of driving actions and driving results. For example, in the driving data of a taxi driver, if the detected multimodal data includes actions such as looking at the navigation and talking to the customer, the learned personalized label is the label for normal driving. For the driving data of a truck driver, if the detected multimodal data includes excessive looking around, the learned personalized label is the label for abnormal driving.

[0062] Since individual sample data can be gradually learned by individuals (customers) during use after the large model is deployed, the large model can obtain individual sample data through online learning during customer use, and update and improve the large model through the individual sample data obtained through online learning to obtain a large model that better matches individual characteristics, thereby being able to more accurately generate the first risk identification result that is adapted to individual characteristics.

[0063] This application conducts comprehensive analysis of multimodal sensor data to provide more comprehensive and accurate basic data.

[0064] In S202, the individual multimodal sensing data is input into a trained universal risk identification model with fixed parameters, and an individual feature optimization channel is input, and a first risk identification result is calculated based on the universal risk identification model and the individual feature optimization channel.

[0065] The large model in the embodiment of the present application includes a general risk identification model and an individual feature optimization channel. As shown in Figure 3, both the general risk identification model and the individual feature optimization channel can receive multimodal sensor data to perform risk identification calculations.

[0066] To effectively reduce the number of parameters to be calculated, the individual feature optimization channel includes a first matrix and a second matrix. Among them, the first matrix is used to reduce the dimension of the training layer of the individual feature optimization channel, and the second matrix is used to increase the dimension of the training layer of the individual feature optimization channel. As shown in Figure 3, for example, the individual feature optimization channel includes a first matrix A and a second matrix B. The dimensions of the trainable layer and the pre-trained layer are d. The first matrix A reduces the dimension from d to r through a fully connected layer, and the second matrix B maps the dimension r back to dimension d through a fully connected layer. Among them, r << d, and r is the rank of the matrix. The number of parameters for matrix calculation changes from d×d to d×r + r×d, thus effectively reducing the number of parameters to be calculated.

[0067] When calculating the first risk recognition result, the product BA of the first matrix A and the second matrix B can be calculated, and the new weight matrix W1 obtained by summing the product and the original weight matrix W is used to replace the original weight matrix W, that is, to replace the weight matrix W of the risk recognition model when the individual feature optimization channel is not added. The new weight matrix W1 is used as the weight matrix of a large model, such as a large language model, for calculation.

[0068] That is, for the input data x, the original output result h1 = W×x. After replacement, the output result h1 = W1×x = Wx + BA×x, and the first risk recognition result is calculated according to the replaced weight matrix.

[0069] In S203, the first risk recognition result is compared with the first risk calibration result to obtain the difference between the two.

[0070] The individual sample data input into the large model includes a pre-calibrated first risk calibration result. This first risk calibration result corresponds to the individual multimodal sensing data. That is, for the multimodal sensing data in the individual sample data, for this individual, it corresponds to the first risk calibration result, and for other individuals, it may correspond to a second risk calibration result different from the first risk calibration result.

[0071] For example, for a taxi driver, his working environment is mainly in the city, and looking at the navigation and talking to customers are normal driving behaviors. However, for a driver transporting dangerous goods, excessive looking around or excessive checking of the navigation is an abnormal driving behavior. Therefore, the same behavior actions will result in different risk recognition results calibrated for different drivers.

[0072] By comparing the first risk identification result calculated by the large model with the calibrated first risk calibration result, the difference between the two identification results can be determined. This difference can be represented by the difference in risk scores. For example, if the risk score of the first risk identification result is 80 points and the risk score of the second risk identification result is 75 points, the difference between the two is 5 points. Alternatively, the first risk identification result and the first risk calibration result determined based on individual characteristics can also be the result of whether the behavior is dangerous. In this case, the difference between the first risk identification result and the first risk calibration result can be the same or different.

[0073] In S204 , it is detected whether the difference meets the preset requirement.

[0074] If the difference between the first risk identification result and the calibrated first risk calibration result is a risk score difference, the preset requirement may be a score difference threshold. If the score difference is less than or equal to the predetermined score difference threshold, the preset requirement is met; if the score difference is greater than the score difference threshold, the preset requirement is not met.

[0075] If the difference between the first risk identification result and the calibrated first risk calibration result is no difference or different, when the difference is no difference, it meets the preset requirements; when the difference is different, it does not meet the preset requirements.

[0076] In S205 , when the difference does not meet the preset requirement, the parameters of the individual feature optimization channel are adjusted according to the difference, and the process returns to S202 .

[0077] If the difference does not meet the preset requirements, it means that the parameters of the current individual feature optimization channel have not been optimized and need to be further adjusted and optimized so that the output result is closer to the calibration result.

[0078] When adjusting the parameters of the individual feature optimization process based on the differences, the parameters in the weight matrix of the individual feature optimization process can be adjusted based on the differences, thereby reducing the difference between the first risk identification result and the first risk calibration result. After adjusting the parameters, the process returns to S202 to calculate and compare the driving risk identification of the individual sample data.

[0079] In S206 , when the difference meets predetermined requirements, a large model including the universal risk identification model and the individual feature optimization channel is obtained, and risk identification is performed for the individual based on the large model.

[0080] By continuously adjusting and optimizing the parameters of the individual feature optimization channel, the difference between the first risk identification result calculated by the large model and the first risk calibration result meets the predetermined requirements, thereby completing the optimization training of the large model.

[0081] Because the trained large model includes individual feature optimization channels optimized using individual sample data, the calculation results can better adapt to individual risk assessment differences, resulting in more accurate risk identification results. Furthermore, during training, only the parameters of the individual feature channels need to be optimized and updated. Optimization is performed based on the optimized general risk identification model, effectively reducing the amount of optimization calculations and improving model training efficiency.

[0082] Before training the large model, the embodiment of the present application may also train the general risk identification model. The training process may include: obtaining general sample data, wherein the general sample data includes general multimodal sensing data and a second risk calibration result; inputting the general multimodal sensing data into the general risk identification model, calculating the second risk identification result, comparing the second risk identification result with the second risk calibration result, and adjusting the parameters of the general risk identification model according to the difference between the two until the difference meets the predetermined requirements.

[0083] The difference between this and large-scale model training is that the individual sample data used as input for the large-scale model is the multimodal data of a single individual and their first risk calibration results. In contrast, the general risk identification model uses general sample data for training, including multimodal data from a large number of different individuals and their corresponding second risk calibration results.

[0084] Since the training data of the general risk identification model includes multimodal data of a large number of vehicles and the corresponding second risk calibration results, the general risk identification model can adapt to the general risk identification requirements of different individuals, but the accuracy of risk identification with individual characteristics is not high. To this end, an individual feature optimization channel is added to the general risk identification model. Through the method shown in Figure 2, individual sample data, that is, individual multimodal sensor data, is input to train the large model including the general risk identification model and the individual feature optimization channel. When the parameters of the general risk identification model are fixed, the parameters in the individual feature optimization channel are optimized, so that the training of the large model can be completed more efficiently, and the trained large model can effectively identify risks based on individual characteristics, thereby improving the accuracy of risk identification.

[0085] The general risk identification model in the embodiments of the present application may include a large language model. The large language model can receive individual sample data, calculate multimodal data in the individual sample data, and output language prompt content of the risk identification result. The process of the large language model receiving individual sample data and calculating multimodal data in the individual sample data can be shown in Figure 4, including:

[0086] In S401 , an image feature vector of an input video is extracted by a video encoder, and the image feature vector is encoded into an image coding vector.

[0087] For video-related tasks, video encoders can better utilize time and context information to improve model performance and efficiency.

[0088] The video encoder can use the pretrained vision model from BLIP-2 (Bootstrapping Language-Image Pretraining-2), including the Vision Transformer (VIT) and the Q-former (a lightweight transformer that uses a learnable queue vector set to extract visual features from a frozen vision model). During training, the parameters are frozen. The VIT extracts the image feature vector of the input video, which is then encoded into an image code vector using the Q-former.

[0089] For a video including N (N is a natural number) frames, N two-dimensional image coding vectors may be generated by a video encoder.

[0090] In S402, position coding information is inserted into the image coding vector through a video adaptation module to generate a video coding vector including time information, the video coding vector is converted into a video query vector through a linear layer, and the video query vector is input into the large language model in the universal risk identification model.

[0091] Since the image coding vector from the video encoding module does not take into account time information, it is necessary to further embed the position coding information as time information. The frame representation with position coding information is input into the video-specific Q-former to obtain the video coding vector. In order to make the video coding vector adapt to the input of the large language model, the video coding vector can be converted into a video query vector through a linear layer, so that the video coding features are spatially aligned with the large language model. Among them, the video query vector is a technology based on the vector space model, which is used to represent video data as a vector and can determine the similarity between data by calculating the similarity between vectors.

[0092] Corresponding to the video data processing method, for audio data, an audio encoding module and an audio adaptation module may also be included. Inputting audio data into the general risk identification model may be as shown in FIG5 , including:

[0093] In S501, the input audio is sampled and encoded by an audio encoder to obtain an audio segment encoding vector.

[0094] To process given audio content, embodiments of the present application can use a pre-trained Imagebind model (used to bind data from different modalities, such as images, text, audio, depth, thermal imaging, and IMU data, to enable retrieval, combination, detection, and generation between different modalities) as an audio encoder. The audio can be sampled at a predetermined duration to obtain multiple audio segments. A spectral transformation method, such as the Mel-spectrogram transform, is then used to convert the audio into a spectrogram. The spectrogram is then input into the audio encoder to obtain an audio segment encoding vector.

[0095] In S502, the audio adaptation module inserts position coding information into the audio segment coding vector to generate an audio coding vector including time information, and the audio coding vector is mapped to the large language model in the general risk identification model through a linear layer.

[0096] Similar to the video adaptation module, the audio adaptation module first embeds positional encoding information representing temporal information into the audio clip. The audio clip with positional encoding information is then fed into the corresponding Q-former to obtain an audio encoding vector. This audio encoding vector can be mapped to a common risk identification model, such as a large language model, through a linear layer, achieving spatial alignment between the audio encoding vector and the large language model.

[0097] As shown in Figure 6, in addition to receiving the video encoding information of the video encoding model and the video adaptation module, and the audio encoding information of the audio encoding module and the audio adaptation module, in an embodiment of the present application, gravity sensor data, map positioning data, weather sensor data, and calibrated text embedding information can also be input into the large language model in the general risk identification model. The risk identification results can be obtained through learning and calculation of the large language model, and the risk identification results can also be output in text.

[0098] In addition, after the driving risk identification based on the large model is completed, the embodiment of the present application can collect individual multi-modal data in real time, input the collected individual multi-modal data into the trained large model, and calculate in real time to obtain more accurate risk identification results.

[0099] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0100] FIG7 is a schematic diagram of a driving risk identification device based on a large model provided in an embodiment of the present application, the device comprising:

[0101] The data acquisition unit 701 is configured to acquire individual sample data, where the individual sample data includes individual multimodal sensing data and a first risk calibration result determined according to individual characteristics.

[0102] The training unit 702 is used to input the individual multimodal sensing data into a trained universal risk identification model with fixed parameters, and an individual feature optimization channel, calculate a first risk identification result based on the universal risk identification model and the individual feature optimization channel, compare the first risk identification result with the first risk calibration result, and adjust the parameters of the individual feature optimization channel according to the difference between the two until the difference meets the predetermined requirements.

[0103] The large-scale model-based driving risk identification device shown in FIG7 corresponds to the large-scale model-based driving risk identification method shown in FIG2 .

[0104] Figure 8 is a schematic diagram of a driving risk identification device provided in an embodiment of the present application. As shown in Figure 8 , the driving risk identification device 8 of this embodiment includes a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as a large-scale model-based driving risk identification or driving risk identification program. When the processor 80 executes the computer program 82, it implements the steps described in the various embodiments of the large-scale model-based driving risk identification or driving risk identification method. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of the various modules / units described in the various device embodiments described above.

[0105] For example, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 82 in the driving risk identification device 8.

[0106] The driving risk identification device 8 can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The driving risk identification device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will appreciate that FIG8 is merely an example of a driving risk identification device 8 and does not limit the driving risk identification device 8 . The driving risk identification device 8 may include more or fewer components than shown, or may combine certain components or have different components. For example, the driving risk identification device may also include input / output devices, network access devices, buses, and the like.

[0107] The processor 80 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0108] The memory 81 can be an internal storage unit of the driving risk identification device 8, such as a hard drive or memory of the driving risk identification device 8. Alternatively, the memory 81 can be an external storage device of the driving risk identification device 8, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 81 can include both the internal storage unit of the driving risk identification device 8 and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the driving risk identification device. The memory 81 can also be used to temporarily store data that has been output or is about to be output.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0110] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0111] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0114] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0115] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments through hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0116] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A driving risk identification method based on a large model, characterized in that: The method comprises: Acquiring individual sample data, the individual sample data including individual multimodal sensing data and a first risk calibration result determined according to individual characteristics; The individual multimodal sensing data is input into a trained universal risk identification model with fixed parameters, as well as an individual feature optimization channel. A first risk identification result is calculated based on the universal risk identification model and the individual feature optimization channel. The first risk identification result is compared with the first risk calibration result. The parameters of the individual feature optimization channel are adjusted according to the difference between the two until the difference meets the predetermined requirements. A large model including the universal risk identification model and the individual feature optimization channel is obtained, and risk identification is performed for the individual based on the large model.

2. The method according to claim 1, characterized in that Before inputting the multimodal sensing data into a trained universal risk identification model with fixed parameters, the method further includes: Acquire general sample data, where the general sample data includes general multimodal sensing data and a second risk calibration result; The universal multimodal sensing data is input into the universal risk identification model, a second risk identification result is calculated, the second risk identification result is compared with the second risk calibration result, and the parameters of the universal risk identification model are adjusted according to the difference between the two until the difference meets the predetermined requirements.

3. The method according to claim 2, characterized in that Inputting the universal multimodal sensor data into the universal risk identification model comprises: Extracting an image feature vector of an input video through a video encoder, and encoding the image feature vector into an image coding vector; Position coding information is inserted into the image coding vector through a video adaptation module to generate a video coding vector including time information, the video coding vector is converted into a video query vector through a linear layer, and the video query vector is input into the large language model in the universal risk identification model.

4. The method according to claim 2, characterized in that Inputting the universal multimodal sensor data into the universal risk identification model comprises: The input audio is sampled and encoded by the audio encoder to obtain an audio segment encoding vector; The audio adaptation module inserts position coding information into the audio segment coding vector to generate an audio coding vector including time information, and the audio coding vector is mapped to the large language model in the universal risk identification model through a linear layer.

5. The method according to claim 3 or 4, characterized in that The method further comprises: Gravity sensor data, map positioning data, weather sensor data, and calibrated text embedding information are input into the large language model in the universal risk identification model.

6. The method according to claim 1, characterized in that The first risk identification result is obtained by calculating the general risk identification model and the individual feature optimization channel, including: The first risk identification result is calculated based on the first weight matrix of the general risk identification model and the sum of the second weight matrices of the individual feature optimization channel as the weight matrix of the large language model.

7. The method according to claim 6, characterized in that The second weight matrix includes a first matrix and a second matrix, the first matrix is ​​used to perform dimensionality reduction processing on the training layer of the individual feature optimization channel, and the second matrix is ​​used to perform dimensionality increase processing on the training layer of the individual feature optimization channel.

8. A driving risk identification device based on a large model, characterized in that: The device comprises: A data acquisition unit, configured to acquire individual sample data, wherein the individual sample data includes individual multimodal sensing data and a first risk calibration result determined according to individual characteristics; A training unit is used to input the individual multimodal sensing data into a trained universal risk identification model with fixed parameters, and an individual feature optimization channel, calculate a first risk identification result based on the universal risk identification model and the individual feature optimization channel, compare the first risk identification result with the first risk calibration result, and adjust the parameters of the individual feature optimization channel according to the difference between the two until the difference meets predetermined requirements.

9. A driving risk identification device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images