Accident identification method and vehicle

Abstract

The invention discloses an accident identification method and a vehicle, and belongs to the technical field of vehicle accident analysis. According to the technical scheme provided by the embodiment of the invention, the vehicle state feature and the vehicle visual feature of the target vehicle are fused to obtain the target fusion feature of the target vehicle, and whether the target vehicle has an accident or not is judged by using the target fusion feature. And equivalently, fault judgment is performed by combining the vehicle state information and the visual information of two dimensions, so that the judgment accuracy is relatively high. Besides, under the condition that an accident occurs, the target fusion features are utilized to identify the accident type and / or the accident severity, so that the corresponding accident condition is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle accident analysis, and more particularly, to an accident identification method and a vehicle in the technical field of vehicles. BACKGROUND

[0002] With the development of vehicle technology, the functions of vehicles are becoming more and more rich, for example, some vehicles provide the ability of accident identification, that is, using a collision sensor to determine whether an accident occurs to the vehicle.

[0003] However, the accuracy of using a collision sensor to determine whether an accident occurs to the vehicle is low, and the above-mentioned method can only determine whether an accident occurs, and cannot obtain the specific situation of the accident, resulting in poor usability of the accident identification function.

[0004] Therefore, how to realize more accurate accident identification and accident situation determination is a research hotspot. SUMMARY

[0005] The embodiments of the present application provide an accident identification method and a vehicle, which can realize more accurate accident identification and accident situation determination, and the technical solutions are as follows:

[0006] In one aspect, an accident identification method is provided, and the method comprises:

[0007] Fusing vehicle state features and vehicle visual features of a target vehicle to obtain target fusion features of the target vehicle, wherein the vehicle state features are obtained by performing feature extraction on vehicle state information of the target vehicle, and the vehicle visual features are obtained by performing feature extraction on visual information collected by the target vehicle;

[0008] Determining whether an accident occurs to the target vehicle based on the target fusion features;

[0009] In the case that the target vehicle has an accident, determining an accident type of the accident based on the target fusion features, and / or determining an accident severity of the accident based on the target fusion features and the vehicle state features.

[0010] In one possible implementation, the fusing of the vehicle state features and the vehicle visual features of the target vehicle to obtain the target fusion features of the target vehicle comprises:

[0011] Splicing the vehicle state features and the vehicle visual features to obtain first spliced features;

[0012] Fusing the vehicle state features and the vehicle visual features based on a cross-attention mechanism to obtain initial fusion features;

[0013] fuse the first splicing feature and the initial fusion feature to obtain the target fusion feature.

[0014] In a possible implementation, the cross-attention mechanism is used to fuse the vehicle state feature and the vehicle visual feature to obtain an initial fusion feature, including:

[0015] performing first linear transformation on the vehicle state feature to obtain a first query matrix;

[0016] performing second linear transformation on the vehicle visual feature to obtain a first key matrix, and performing third linear transformation on the vehicle visual feature to obtain a first value matrix;

[0017] determining a first attention weight based on the first query matrix and the first key matrix;

[0018] multiplying the first attention weight and the first value matrix to obtain the initial fusion feature.

[0019] In a possible implementation, the target vehicle is determined to have an accident based on the target fusion feature, including:

[0020] determining a first classification sub-feature from the target fusion feature, the dimension of the first classification sub-feature being lower than that of the target fusion feature;

[0021] determining an accident classification value based on the first classification sub-feature;

[0022] in a case where the accident classification value is greater than or equal to a classification value threshold, determining that the target vehicle has an accident, the classification value threshold being determined based on the speed of the target vehicle;

[0023] in a case where the accident classification value is less than the classification value threshold, determining that the target vehicle does not have an accident.

[0024] In a possible implementation, the first classification sub-feature is determined from the target fusion feature, including:

[0025] obtaining the first classification sub-feature from the first N dimensions of the target fusion feature;

[0026] or performing principal component analysis on the target fusion feature to obtain the first classification sub-feature;

[0027] or performing linear transformation on the target fusion feature to obtain the first classification sub-feature.

[0028] In a possible implementation, the accident type is determined based on the target fusion feature, including:

[0029] encoding the target fusion feature based on an attention mechanism to obtain a fusion encoding feature of the target fusion feature;

[0030] fully connecting and normalizing the fusion encoding feature to obtain a probability set, the probability set including a plurality of probabilities, one probability corresponding to one candidate accident type;

[0031] determining the candidate accident type corresponding to the highest probability in the probability set as the accident type of the accident.

[0032] In a possible implementation, the determining the accident severity of the accident based on the target fusion feature and the vehicle state feature includes:

[0033] splicing the vehicle state feature and the target fusion feature to obtain a second spliced feature;

[0034] fully connecting and normalizing the second spliced feature to obtain an accident level of the accident, the accident level being used to represent the accident severity.

[0035] In a possible implementation, before the fusing the vehicle state feature of the target vehicle and the vehicle visual feature to obtain the target fusion feature of the target vehicle, the method further includes:

[0036] obtaining the vehicle state information and the visual information;

[0037] synchronizing the vehicle state information and the visual information by a hardware clock;

[0038] sampling the synchronized vehicle state information by a sliding window to obtain initial sampling information;

[0039] extracting key frames from the synchronized visual information to obtain a plurality of key frames;

[0040] extracting features from the initial sampling information to obtain the vehicle state feature;

[0041] extracting features from the plurality of key frames to obtain the vehicle visual feature.

[0042] In a possible implementation, the method further includes:

[0043] in a case where the target vehicle has an accident, enabling a safety island mechanism to store the vehicle state information and the visual information;

[0044] uploading the vehicle state information and the visual information to a cloud platform, so that the cloud platform performs accident reconstruction and / or accident liability determination based on the vehicle state information and the visual information.

[0045] In an aspect, an accident identification apparatus is provided, the apparatus comprising:

[0046] a feature fusion module configured to fuse a vehicle state feature of a target vehicle and a vehicle visual feature of the target vehicle to obtain a target fusion feature of the target vehicle, the vehicle state feature being extracted from vehicle state information of the target vehicle, and the vehicle visual feature being extracted from visual information collected by the target vehicle;

[0047] an accident determination module configured to determine whether the target vehicle has an accident based on the target fusion feature;

[0048] an accident analysis module configured to, in a case where the target vehicle has an accident, determine an accident type of the accident based on the target fusion feature, and / or determine an accident severity of the accident based on the target fusion feature and the vehicle state feature.

[0049] In a possible implementation, the feature fusion module is configured to splice the vehicle state feature and the vehicle visual feature to obtain a first spliced feature; fuse the vehicle state feature and the vehicle visual feature based on a cross-attention mechanism to obtain an initial fusion feature; and fuse the first spliced feature and the initial fusion feature to obtain the target fusion feature.

[0050] In a possible implementation, the feature fusion module is configured to perform a first linear transformation on the vehicle state feature to obtain a first query matrix; perform a second linear transformation on the vehicle visual feature to obtain a first key matrix, and perform a third linear transformation on the vehicle visual feature to obtain a first value matrix; determine a first attention weight based on the first query matrix and the first key matrix; and multiply the first attention weight and the first value matrix to obtain the initial fusion feature.

[0051] In a possible implementation, the accident determination module is configured to determine a first classification sub-feature from the target fusion feature, the first classification sub-feature having a dimension lower than the target fusion feature; determine an accident classification value based on the first classification sub-feature; in a case where the accident classification value is greater than or equal to a classification value threshold, determine that the target vehicle has an accident, the classification value threshold being determined based on a vehicle speed of the target vehicle; and in a case where the accident classification value is less than the classification value threshold, determine that the target vehicle does not have an accident.

[0052] In a possible implementation, the accident determination module is configured to: obtain the first classification sub-feature from the first N dimensions of the target fusion feature; or perform principal component analysis on the target fusion feature to obtain the first classification sub-feature; or perform linear transformation on the target fusion feature to obtain the first classification sub-feature.

[0053] In a possible implementation, the accident analysis module is configured to: encode the target fusion feature based on an attention mechanism to obtain a fusion encoding feature of the target fusion feature; perform full connection and normalization on the fusion encoding feature to obtain a probability set, the probability set including a plurality of probabilities, one probability corresponding to one candidate accident type; and determine the candidate accident type corresponding to the highest probability in the probability set as the accident type of the accident.

[0054] In a possible implementation, the accident analysis module is configured to: splice the vehicle state feature and the target fusion feature to obtain a second spliced feature; and perform full connection and normalization on the second spliced feature to obtain an accident level of the accident, the accident level being used to represent the severity of the accident.

[0055] In a possible implementation, the apparatus further includes a feature extraction module configured to: obtain the vehicle state information and the visual information; synchronize the vehicle state information and the visual information by using a hardware clock; sample the synchronized vehicle state information by using a sliding window to obtain initial sampling information; extract key frames from the synchronized visual information to obtain a plurality of key frames; extract features from the initial sampling information to obtain the vehicle state feature; and extract features from the plurality of key frames to obtain the vehicle visual feature.

[0056] In a possible implementation, the apparatus further includes an information processing module configured to: in a case where the target vehicle has an accident, enable a safety island mechanism to store the vehicle state information and the visual information; and upload the vehicle state information and the visual information to a cloud platform, so that the cloud platform performs accident reconstruction and / or accident liability determination based on the vehicle state information and the visual information.

[0057] In an aspect, a vehicle is provided, which includes one or more processors and one or more memories, the one or more memories storing at least one program code, the program code being loaded and executed by the one or more processors to implement operations performed by the accident identification method.

[0058] In an aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing at least one program code, the program code being loaded and executed by a processor to implement operations performed by the accident identification method.

[0059] By the technical solution provided in the embodiments of the present application, the vehicle state features and the vehicle visual features of the target vehicle are fused to obtain target fusion features of the target vehicle, and the target fusion features are used to determine whether the target vehicle has an accident, which is equivalent to combining vehicle state information and visual information in two dimensions to determine the fault, and the determination accuracy is high. In addition, in the case of an accident, the target fusion features are used to identify the accident type and / or the accident severity, so as to obtain the corresponding accident situation. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a schematic diagram of an implementation environment of an accident identification method provided by the embodiments of the present application;

[0061] Figure 2 is a flowchart of an accident identification method provided by the embodiments of the present application;

[0062] Figure 3 is a flowchart of another accident identification method provided by the embodiments of the present application;

[0063] Figure 4 is a structural schematic diagram of an MLP provided by the embodiments of the present application;

[0064] Figure 5 is a flowchart of still another accident identification method provided by the embodiments of the present application;

[0065] Figure 6 is a structural schematic diagram of an accident identification device provided by the embodiments of the present application;

[0066] Figure 7 is a structural schematic diagram of a vehicle provided by the embodiments of the present application. DETAILED DESCRIPTION

[0067] The technical solutions in the present application will be described in detail below with reference to the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B: "and / or" in the text only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0068] Hereinafter, the terms "first", "second" are only used for descriptive purposes and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of technical features reflected. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features.

[0069] In order to explain the technical solutions provided by the embodiments of the present application, first, some terms related to the embodiments of the present application are introduced.

[0070] Artificial intelligence (AI) is to use digital computers or digital computer controlled machine simulation, extension and expansion of human intelligence, perception of environment, acquisition of knowledge and use of knowledge to obtain better results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence.

[0071] Machine learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a branch of computer science that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0072] Attention mechanism: Attention mechanism is a technique that simulates the selective focus on important information in human cognitive process, widely used in deep learning and natural language processing. Its core idea is to dynamically allocate weights, so that the model can focus on the most relevant part of the input data for the current task, while ignoring irrelevant information.

[0073] Cross-attention mechanism: Cross-attention is a variant of attention mechanism, specifically designed to handle the dependency between two different input sequences.

[0074] Attention weight: can represent the importance of some data in the training or prediction process, the importance represents the size of the influence of the input data on the output data. The data with high importance has a higher value of the corresponding attention weight, and the data with low importance has a lower value of the corresponding attention weight. In different scenarios, the importance of data is not the same, and the process of training attention weight of the model is also the process of determining the importance of data.

[0075] Mask: A mask is a string of binary codes that is multiplied by the target field to mask or display certain characters in the target field. For example, if the target field is (1, 1, 0, 1) and the mask is (1, 0, 1, 0), the product of the target field and the mask is (1, 0, 0, 0). This means that the first and third characters in the target field are retained, while the second and fourth characters are "masked" to 0. The mask can be used to determine which characters in the target field are retained and which are "masked."

[0076] Normalization: Mapping sequences of numbers with different value ranges to the interval (0, 1) facilitates data processing. In some cases, the normalized values ​​can be directly implemented as probabilities.

[0077] Learning Rate: This controls the model's learning progress. It guides how the model uses the gradient of the loss function to adjust network weights during gradient descent. If the learning rate is too high, the loss function may overshoot the global minimum, resulting in excessive loss. If the learning rate is too low, the loss function changes very slowly, significantly increasing the network's convergence complexity and making it easy for the network to become trapped in a local minimum or saddle point.

[0078] Embedded Coding: Embedded coding mathematically represents a correspondence, where data in the X space is mapped to the Y space through a function F, where the function F is an injective function. The mapping result is structural preservation. The injective function indicates that the data after mapping uniquely corresponds to the data before mapping. Structural preservation means that the size relationship of the data before mapping is the same as the size relationship of the data after mapping. For example, before mapping, there are data X1 and X2, and after mapping, the data corresponding to X1 is Y1 and the data corresponding to X2 is Y2. If the data before mapping X1 is greater than X2, then the data after mapping Y1 is greater than Y2. For words, this means mapping the words to another space to facilitate subsequent machine learning and processing.

[0079] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0080] The following describes the implementation environment of the embodiment of the present application. Figure 1 The implementation environment of the accident identification method provided in the embodiment of the present application includes a vehicle-mounted terminal 101 and a cloud platform 103.

[0081] The vehicle terminal 101 is a terminal arranged on a vehicle, configured to acquire vehicle state information and visual information, and process the vehicle state information and the visual information. The vehicle terminal 101 is connected to the cloud platform 103 through a wireless network, and can interact with the cloud platform 103.

[0082] The cloud platform 103 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. In some embodiments, the cloud platform 103 is also referred to as a TSP (Telematics Service Provider) platform.

[0083] After introducing the implementation environment of the embodiments of the present application, the application scenarios of the technical solutions provided by the embodiments of the present application are introduced. The technical solutions provided by the embodiments of the present application can be applied to various vehicles with vehicle terminals. By using the technical solutions provided by the embodiments of the present application, the vehicle state features and the vehicle visual features can be used for accident identification and accident situation identification, so as to determine whether the target vehicle has an accident, and determine the accident type and / or the accident severity in the case of an accident, thereby realizing the identification of the accident situation.

[0084] After introducing the implementation environment and the application scenarios of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are introduced, which are described with reference to Figure 2 Taking the vehicle terminal as an example, the method includes the following steps.

[0085] 201. The vehicle terminal fuses the vehicle state features and the vehicle visual features of the target vehicle to obtain target fusion features of the target vehicle. The vehicle state features are obtained by feature extraction on the vehicle state information of the target vehicle, and the vehicle visual features are obtained by feature extraction on the visual information collected by the target vehicle.

[0086] The vehicle state information is used for describing a vehicle state of the target vehicle. For example, the vehicle state information includes CAN (Controller Area Network) bus data and navigation data. The visual information is collected by a visual information collection component of the target vehicle, and is used for reflecting a situation around the target vehicle. For example, the visual information collection component includes an image collection component and / or a radar, and correspondingly, the visual information includes an image and / or a point cloud. The feature extraction on the vehicle state information is performed to encode the vehicle state information into vehicle state features with stronger expression capability, so as to facilitate subsequent processing of the vehicle state features by the vehicle terminal. The feature extraction on the visual information is performed in the same way. In addition, since the vehicle state information and the visual information are information of different modalities, the feature extraction on the vehicle state information and the visual information is performed respectively, which is helpful for fusion between the information of different modalities. The target fusion feature is obtained by fusing the vehicle state features and the vehicle visual features.

[0087] 202. The vehicle terminal determines, based on the target fusion feature, whether the target vehicle has an accident.

[0088] The target fusion feature fuses the vehicle state information and the visual information, and can reflect actual situations of the target vehicle from two dimensions. The target fusion feature is used to determine whether the target vehicle has an accident, and an accurate result can be obtained.

[0089] 203. In a case where the target vehicle has an accident, the vehicle terminal determines, based on the target fusion feature, an accident type of the accident, and / or the vehicle terminal determines, based on the target fusion feature and the vehicle state feature, an accident severity of the accident.

[0090] The accident type is a classification result of the accident. For example, the accident type includes rear-end collision, being rear-ended, colliding with an obstacle, and overturning, and the like, which are set by a technician according to actual situations. The accident severity can indirectly represent a damage degree of the target vehicle, and thus represents a degree of repair required by the target vehicle. Generally, the accident severity is represented by an accident level.

[0091] By the technical solution provided in the embodiments of the present application, the vehicle state features and the vehicle visual features of the target vehicle are fused to obtain the target fusion feature of the target vehicle. The target fusion feature is used to determine whether the target vehicle has an accident, which is equivalent to determining the accident by combining the vehicle state information and the visual information of two dimensions, and the accuracy of the determination is high. In addition, in a case where the target vehicle has an accident, the target fusion feature is used to identify the accident type and / or the accident severity, so that the corresponding accident situation is obtained.

[0092] It should be noted that the above steps 201-203 are a simple description of the accident identification method provided by the embodiments of the present application. In the following, the accident identification method provided by the embodiments of the present application will be described in more detail in combination with some examples. See Figure 3 Taking the vehicle-mounted terminal as an example, the method comprises the following steps.

[0093] 301. The vehicle-mounted terminal obtains vehicle state information and visual information of the target vehicle.

[0094] The vehicle-mounted terminal is the vehicle-mounted terminal of the target vehicle, and the target vehicle is a vehicle to be identified for an accident. The vehicle state information is used to describe the vehicle state of the target vehicle, for example, the vehicle state information comprises CAN (Controller Area Network) bus data and navigation data. The visual information is collected by a visual information collection component of the target vehicle, and is used to reflect the situation around the target vehicle, for example, the visual information collection component comprises an image collection component and / or a radar, and correspondingly, the visual information comprises an image and / or a point cloud.

[0095] In a possible implementation, the vehicle-mounted terminal obtains the CAN bus data and the navigation data of the target vehicle. The vehicle-mounted terminal obtains the visual information of the target vehicle through the visual information collection component.

[0096] The CAN bus data comprises power system parameters, vehicle body control parameters, chassis system parameters, and new energy vehicle specific parameters, etc. The power system parameters comprise vehicle speed, engine speed, accelerator pedal opening degree, brake pressure, gearbox gear position, and torque demand, etc. The vehicle body control parameters comprise door state, window position, light state, air conditioning setting, and seat position, etc. The chassis system parameters comprise ABS state, ESP intervention, tire pressure data, and steering angle, etc. The new energy vehicle specific parameters comprise battery state, charging current, motor temperature, and energy recovery level, etc. In some embodiments, the sampling frequency of the CAN bus data is 100 Hz. The navigation data comprises absolute position navigation data and inertial navigation data. The absolute position navigation data comprises GPS navigation data, Beidou navigation data, etc. The inertial navigation data comprises three-axis acceleration and angular velocity, etc. In some embodiments, the sampling frequency of the navigation data is 50 Hz.

[0097] For example, the vehicle-mounted terminal obtains the CAN bus data from the CAN bus of the target vehicle, and obtains the navigation data from the navigation system of the target vehicle. The vehicle-mounted terminal collects images around the target vehicle through the image collection component, and / or collects point clouds around the target vehicle through the radar. In the embodiments of the present application, the image refers to a video stream, that is, a collection of video frames.

[0098] In some embodiments, after obtaining the vehicle state information and the visual information, the vehicle terminal synchronizes the vehicle state information and the visual information by a hardware clock to obtain synchronized vehicle state information and synchronized visual information.

[0099] The hardware clock synchronization is characterized by relying on a physical layer signal rather than a software algorithm to keep time consistency among multiple devices. In the embodiments of the present application, the time consistency of the CAN bus data and the visual information is kept, so that more accurate identification results can be obtained in subsequent accident identification. The hardware clock synchronization can be realized by GPS clock synchronization or precision time protocol (PTP). Of course, with the development of science and technology, the hardware clock synchronization can also be realized by other ways, which are not limited in the embodiments of the present application.

[0100] It should be noted that after the above step 301, the vehicle terminal can synchronously execute the following steps 302 and 303 to realize the synchronous extraction of the vehicle state features and the vehicle visual features. Of course, the following step 302 can be executed first, and then the following step 303 can be executed, or the following step 303 can be executed first, and then the following step 302 can be executed, which are not limited in the embodiments of the present application.

[0101] 302, the vehicle terminal extracts features from the vehicle state information to obtain vehicle state features.

[0102] The feature extraction from the vehicle state information is to encode the vehicle state information into vehicle state features with stronger expression ability, so as to facilitate the subsequent processing of the vehicle terminal.

[0103] In a possible implementation, the vehicle terminal samples the synchronized vehicle state information by a sliding window to obtain initial sampling information. The vehicle terminal extracts features from the initial sampling information to obtain the vehicle state features.

[0104] The length of the sliding window is a preset length, and the step length is a preset step length. Generally, the length of the sliding window determines how much information is covered by the sliding window at a time, and the step length of the sliding window determines the degree of sampling precision. In the experiment, the preset length is determined as 3s, and the preset step length is determined as 0.5s, so that 90% of the accident period can be covered, and the amount of information collected will not be too much. Of course, for different vehicle terminals and requirements, the preset length and the preset step length can be set by the technical personnel according to the actual situation and requirements, which are not limited in the embodiments of the present application. In the case that the vehicle state information includes 28 kinds of information, the preset length is 3s, and the sampling frequency is 100Hz, the vehicle state information is a time sequence matrix of 300x28, and the subsequent feature extraction is performed on this time sequence matrix.

[0105] In this implementation, the synchronized vehicle state information is sampled by using a sliding window to obtain initial sampling information, thereby realizing local sampling of the vehicle state information, reducing the amount of data for subsequent processing, and realizing acquisition of the vehicle state feature in the case of limited computing power of the vehicle terminal. Feature extraction is performed on the initial sampling information to obtain the corresponding vehicle state feature.

[0106] In order to more clearly illustrate the above implementation, the above implementation is described below through several examples.

[0107] Example 1: The vehicle terminal samples the synchronized vehicle state information by using a sliding window to obtain initial sampling information. The vehicle terminal performs multiple full connections on the initial sampling information to obtain the vehicle state feature.

[0108] Among them, the vehicle terminal will use an activation function to add a nonlinear factor after each full connection to improve the expression ability of the processing result. In addition, in order to prevent gradient disappearance, the vehicle terminal will add a residual connection after each full connection, that is, the result obtained by full connection is fused with the initial sampling information, and the next full connection is performed on the result after the residual connection. This can make the initial sampling information maintain integrity during the full connection process, avoid information loss, and thus prevent gradient disappearance. In addition, the vehicle terminal will also add regularization after the last full connection, which can provide more stable vehicle state features.

[0109] For example, referring to Figure 4 The above multiple full connections are realized by 5-layer MLP (Multilayer Perceptron, Multilayer Perceptron) (that is, five full connections), and the parameters of the 5-layer MLP are 128, 256, 512, 256, and 128 in turn. The activation function used by the MLP is LeakyReLU (negative slope coefficient 0.01), and the regularization method used during output is Batch Normalization (Batch Normalization). The final vehicle state feature can be used to represent the dynamic behavior pattern of the target vehicle.

[0110] Example 2: The vehicle terminal samples the synchronized vehicle state information by using a sliding window to obtain initial sampling information. The vehicle terminal performs multiple convolutions on the initial sampling information to obtain the vehicle state feature.

[0111] Among them, similarly to the above full connection process, the vehicle terminal will also use an activation function and an activation function after each convolution, and will also add regularization after the last convolution.

[0112] In example 3, the vehicle terminal samples the synchronized vehicle state information by using a sliding window to obtain initial sampling information. The vehicle terminal encodes the initial sampling information in time sequence to obtain the vehicle state feature.

[0113] In this example, since the vehicle state information is actually a time sequence information, the time sequence encoding can be implemented by using a LSTM (Long Short Time Memory) network.

[0114] 303. The vehicle terminal extracts features from the visual information to obtain a vehicle visual feature.

[0115] In this example, the feature extraction from the visual information is performed to encode the visual information into a vehicle visual feature with stronger expression capability, so as to facilitate the subsequent processing of the vehicle terminal.

[0116] In one possible implementation, the vehicle terminal extracts key frames from the synchronized visual information to obtain a plurality of key frames. The vehicle terminal extracts features from the plurality of key frames to obtain the vehicle visual feature.

[0117] In this example, when the visual information includes images, the key frames are image frames; and when the visual information includes point clouds, the key frames are point cloud frames.

[0118] In this implementation, the key frames are extracted from the visual information to obtain a plurality of key frames, and then features are extracted from the plurality of key frames to obtain the vehicle visual feature, so as to achieve the purpose of obtaining the vehicle visual feature with reduced computation amount.

[0119] To make the above-mentioned implementation clearer, the following describes the above-mentioned implementation in several parts.

[0120] First part: The vehicle terminal extracts key frames from the synchronized visual information to obtain a plurality of key frames.

[0121] In one possible implementation, the vehicle terminal extracts frames from the synchronized visual information at a preset time interval to obtain a plurality of key frames.

[0122] In this example, the preset time interval is set by a technician according to actual conditions, for example, set to 0.1 s. In this case, under the premise that the size of the sliding window in the above-mentioned step 302 is 3 s, one sliding window corresponds to 30 key frames.

[0123] The following describes another implementation of the above-mentioned first part.

[0124] In a possible implementation, the vehicle terminal determines information amounts of a plurality of frames in the synchronized visual information. The vehicle terminal determines, as key frames, frames with information amounts greater than or equal to a preset information amount in the plurality of frames, to obtain the plurality of key frames.

[0125] In the above implementation, the frame is a video frame or a point cloud frame, and the information amount can be determined in a manner in the related art, which is not limited in the embodiments of the present application.

[0126] In addition, it should be noted that, in addition to the above two manners, the vehicle terminal can also use other key frame extraction manners in the related art, which is not limited in the embodiments of the present application.

[0127] In the second part, the vehicle terminal extracts features from the plurality of key frames to obtain the vehicle visual feature.

[0128] In a possible implementation, for any key frame in the plurality of key frames, the vehicle terminal encodes the key frame based on an attention mechanism to obtain a key frame feature of the key frame. The vehicle terminal fuses the key frame features of the plurality of key frames to obtain the vehicle visual feature.

[0129] For example, taking a video frame as the key frame, for any key frame in the plurality of key frames, the vehicle terminal divides the key frame into a plurality of image blocks. The vehicle terminal embeds and encodes each image block to obtain an image block embedding feature of each image block. The vehicle terminal splices the image block embedding feature of each image block and a position embedding feature to obtain an image block feature of each image block, where the position embedding feature is used to represent the position of the image block in the key frame. The vehicle terminal linearly transforms the image block feature of each image block to obtain a second query matrix, a second key matrix, and a second value matrix of each image block. The vehicle terminal determines an attention weight between each two image blocks in the plurality of image blocks based on the second query matrix and the second key matrix of each image block. The vehicle terminal fuses the second value matrix of each image block based on the attention weight to obtain the key frame feature of the key frame. The vehicle terminal fully connects the key frame features of each key frame to obtain a target key frame feature of each key frame. The vehicle terminal adds the target key frame features of each key frame and divides the sum by the number of the plurality of key frames to obtain the vehicle visual feature.

[0130] In the above implementation, the full connection of the key frame is to adjust the size of the key frame feature, so as to facilitate the subsequent processing process. The process of mapping the image block feature into the query matrix, the key matrix, and the value matrix is the process of multiplying the image block feature with three linear transformation matrices, and the three linear transformation matrices are determined through a training process.

[0131] In some embodiments, the manner of encoding the key frame based on the attention mechanism to obtain the key frame feature of the key frame can be implemented by a feature extractor, for example, the feature extractor is an attention encoder Transformer Encoder, which can include multiple encoding layers (for example, 12 layers), and each encoding layer includes multiple attention heads (for example, 12 heads). The process described in the above example is the encoding process of one attention head of one encoding layer. For multiple attention heads, the encoding process of each head belongs to the same inventive concept as described above, and the difference is that the outputs of multiple attention heads will be fused to obtain the output of the encoding layer.

[0132] It should be noted that the above is an example of visual information as an image. In the case of point cloud as visual information, the processing manner belongs to the same inventive concept as described above, and will not be repeated here.

[0133] The following describes another implementation of the second part.

[0134] In a possible implementation, for any key frame in the plurality of key frames, the vehicle terminal performs multiple convolutions on the key frame to obtain a key frame feature of the key frame. The vehicle terminal fuses the key frame features of the plurality of key frames to obtain the vehicle visual feature.

[0135] For example, taking a key frame as a video frame, for any key frame in the plurality of key frames, the vehicle terminal performs convolution on the key frame using multiple convolution kernels to obtain multiple convolution features of the key frame. The vehicle terminal fuses the multiple convolution features to obtain a key frame feature of the key frame. The vehicle terminal fuses the key frame features of the plurality of key frames to obtain the vehicle visual feature.

[0136] Wherein, the parameters of different convolution kernels are different but the sizes are the same. It should be noted that the above is an example of visual information as an image. In the case of point cloud as visual information, the processing manner belongs to the same inventive concept as described above, and will not be repeated here.

[0137] In some embodiments, after obtaining the plurality of key frames, the vehicle terminal can also preprocess each key frame to eliminate irrelevant information in the key frame and improve the accuracy of the key frame feature obtained by subsequent feature extraction. For example, in the case of a key frame being a video frame, the vehicle terminal performs adaptive histogram equalization (CLAHE) on the video frame to enhance the low-light scene, or the vehicle terminal crops the video frame to retain the central region of the video frame. In this way, in the case of the video frame recording the road conditions in front of the target vehicle, the key information of the road conditions in front can be retained, and the influence of irrelevant information can be eliminated.

[0138] 304、The vehicle terminal fuses the vehicle state feature and the vehicle visual feature of the target vehicle to obtain a target fusion feature of the target vehicle.

[0139] In the above embodiment, the vehicle state feature and the vehicle visual feature are fused by the vehicle terminal. In one possible implementation, the vehicle state feature and the vehicle visual feature are fused by the vehicle terminal based on a cross-attention mechanism.

[0140] In one possible implementation, the vehicle terminal splices the vehicle state feature and the vehicle visual feature to obtain a first spliced feature. The vehicle terminal fuses the vehicle state feature and the vehicle visual feature based on a cross-attention mechanism to obtain an initial fusion feature. The vehicle terminal fuses the first spliced feature and the initial fusion feature to obtain the target fusion feature.

[0141] In the above embodiment, the vehicle state feature and the vehicle visual feature are fused by the vehicle terminal. In one possible implementation, the vehicle state feature and the vehicle visual feature are fused by the vehicle terminal based on a cross-attention mechanism.

[0142] In order to more clearly illustrate the above embodiment, the manner of fusing the vehicle state feature and the vehicle visual feature in the above embodiment is described below.

[0143] In one possible implementation, the vehicle terminal performs first linear transformation on the vehicle state feature to obtain a first query matrix. The vehicle terminal performs second linear transformation on the vehicle visual feature to obtain a first key matrix, and performs third linear transformation on the vehicle visual feature to obtain a first value matrix. The vehicle terminal determines a first attention weight based on the first query matrix and the first key matrix. The vehicle terminal multiplies the first attention weight and the first value matrix to obtain the initial fusion feature.

[0144] The linear transformation is implemented by full connection, for example, multiplying the vehicle state feature with a first linear transformation matrix to obtain a first query matrix, which is the process of the first linear transformation in the above embodiment; multiplying the vehicle visual feature with a second linear transformation matrix to obtain a first key matrix, which is the process of the second linear transformation in the above embodiment; and multiplying the vehicle visual feature with a third linear transformation matrix to obtain a first value matrix, which is the process of the third linear transformation in the above embodiment. In this case, the first linear transformation matrix is also referred to as a query transformation matrix, the second linear transformation matrix is also referred to as a key transformation matrix, and the third linear transformation matrix is also referred to as a value transformation matrix, and the first linear transformation matrix, the second linear transformation matrix and the third linear transformation matrix are determined through training.

[0145] For example, the vehicle terminal multiplies the vehicle state feature with the first linear transformation matrix to obtain the first query matrix. The vehicle terminal multiplies the vehicle visual feature with the second linear transformation matrix to obtain the first key matrix. The vehicle terminal multiplies the vehicle visual feature with the third linear transformation matrix to obtain the first value matrix, and the first query matrix, the first key matrix and the first value matrix have the same size. The vehicle terminal multiplies the first query matrix with the transpose of the first key matrix and divides the result by a preset value to obtain the first attention weight. The vehicle terminal multiplies the first attention weight with the first value matrix to obtain the initial fusion feature.

[0146] The preset value is determined based on the dimensions of the first query matrix, the first key matrix and the first value matrix, for example, the preset value can be obtained by taking the square root of the dimensions. The above process actually uses one-way attention weight from the vehicle state feature to the vehicle visual feature, so that the vehicle state feature can focus on the key area of the vehicle visual feature, such as focusing on the front vehicle when braking. In the experiment, compared with the traditional fusion method, the above fusion method improves the CloU index by 19.2%. CloU is a loss function for target detection, and the full name is "Completed IoU", which aims to improve the traditional IoU (intersection over union) loss function.

[0147] The above example describes the process through the following formula (1).

[0148]

[0149] wherein Attention(Q, K, V) represents the initial fusion feature, Softmax() represents a normalization function, Q represents the first query matrix, K represents the first key matrix, V represents the first value matrix, and d represents the preset value, The first attention weight is represented. In some embodiments, the dimensions of Q, K and V are all 256, and d is 256 accordingly.

[0150] In the embodiments of the present application, the above-mentioned various features are shared by the vehicle terminal and the cloud platform, ensuring consistency in processing by the vehicle terminal and the cloud platform.

[0151] 305、The vehicle terminal determines whether the target vehicle has an accident based on the target fusion feature.

[0152] The target fusion feature fuses vehicle state information and visual information, and can reflect the actual situation of the target vehicle from two dimensions. Using the target fusion feature to determine whether the target vehicle has an accident can obtain a more accurate result.

[0153] In a possible implementation, the vehicle terminal determines a first classification sub-feature from the target fusion feature, and the dimension of the first classification sub-feature is lower than that of the target fusion feature. The vehicle terminal determines an accident classification value based on the first classification sub-feature. In a case where the accident classification value is greater than or equal to a classification value threshold, the vehicle terminal determines that the target vehicle has an accident. The classification value threshold is determined based on the speed of the target vehicle. In a case where the accident classification value is less than the classification value threshold, the vehicle terminal determines that the target vehicle has no accident.

[0154] The first classification sub-feature is determined from the target fusion feature to achieve more rapid accident identification. The dimension of the first classification sub-feature is lower than that of the target fusion feature, so that the efficiency of using the first classification sub-feature for accident identification is higher. The classification value threshold is determined based on the speed. Generally, the classification value threshold is negatively correlated with the speed, that is, the greater the speed, the lower the classification value threshold; the smaller the speed, the higher the classification value threshold. The principle of this design is that the higher the speed, the greater the possibility of the target vehicle having an accident, so the classification value threshold can be appropriately reduced to make the target vehicle more easily identified as having an accident. The lower the speed, the smaller the possibility of the target vehicle having an accident, so the classification value threshold can be appropriately increased to make the target vehicle less easily identified as having an accident. For example, the value range of the classification value threshold is designed to be (0.85-0.92), and the mapping relationship between the speed and the classification value threshold is set by the technician according to the actual situation, which is not limited in the embodiments of the present application.

[0155] In this implementation, the first classification sub-feature is determined from the target fusion feature, the accident classification value is determined using the first classification sub-feature, and the accident classification value and the classification value threshold related to the speed are compared to determine whether an accident occurs. The accuracy of accident determination is high.

[0156] In order to make the above-mentioned embodiments clearer, the following will be described in several parts.

[0157] The first part, the vehicle terminal determines a first classification sub-feature from the target fusion feature, and the dimension of the first classification sub-feature is lower than that of the target fusion feature.

[0158] In a possible implementation, the vehicle terminal obtains the first classification sub-feature by taking the first N-dimensional sub-feature in the target fusion feature.

[0159] Wherein, N is a positive integer, which is set by the technician according to the demand.

[0160] In this embodiment, the first N-dimensional sub-feature in the target fusion feature is directly taken as the first classification sub-feature for accident identification, and the efficiency of obtaining the first classification sub-feature is higher.

[0161] The following will describe another embodiment of the first part.

[0162] In a possible implementation, the vehicle terminal performs principal component analysis on the target fusion feature to obtain the first classification sub-feature.

[0163] Wherein, principal component analysis (PCA) is also known as principal component analysis, which aims to transform multiple indexes into a few comprehensive indexes by dimension reduction. Principal component analysis is a linear transformation. This transformation transforms the data (multiple sub-features in the target fusion feature) into a new coordinate system, so that the first largest variance of any data projection is on the first coordinate (called the first principal component), the second largest variance is on the second coordinate (the second principal component), and so on. The goal of principal component analysis is to find r (r < n) new sub-features that reflect the main features of the target fusion feature, compress the size of the original target fusion feature, and reduce the dimension of the target fusion feature, that is, to select the least dimension to summarize the most important features, and n is the number of sub-features in the target fusion feature. Each new sub-feature is a linear combination of the original sub-features, reflecting the comprehensive effect of the original sub-features and having certain practical meaning. The r new sub-features are called “principal components”, which can reflect the influence of the original n sub-features to a great extent, and these new sub-features are mutually independent and orthogonal.

[0164] In this embodiment, the target fusion feature is processed by using principal component analysis to obtain the first classification sub-feature, and the first classification sub-feature not only has a lower dimension than the target fusion feature, but also has a similar expression ability to the target fusion feature, and the use of the first classification sub-feature can obtain a more accurate classification result.

[0165] Another embodiment of the first part is described below.

[0166] In a possible implementation, the vehicle terminal linearly transforms the target fusion feature to obtain the first classification sub-feature.

[0167] In the above embodiment, the linear transformation is for dimension reduction, that is, to convert the high-dimensional target fusion feature into a low-dimensional first classification sub-feature. The linear transformation matrix used in the linear transformation is designed by the technician according to the actual situation, and the embodiment of the present application is not limited thereto.

[0168] The second part, the vehicle terminal determines an accident classification value based on the first classification sub-feature.

[0169] In a possible implementation, the vehicle terminal performs full connection and normalization on the first classification sub-feature through an accident detection model to obtain the accident classification value.

[0170] The accident detection model is a binary classification model, and the accident detection model is trained using a plurality of positive sample data and a plurality of negative sample data. The positive sample data is a sample classification sub-feature corresponding to an accident, and the negative sample data is a sample classification sub-feature corresponding to no accident. Using a plurality of positive sample data and a plurality of negative sample data can enable the accident detection model to learn the potential rules of the classification sub-feature of the accident and the classification sub-feature of no accident, so as to subsequently classify using the first classification sub-feature. In addition, in the actual training process of the accident detection model, the number of positive samples and negative samples is unbalanced. Generally, the number of negative samples is much larger than that of positive samples, because negative samples are easier to obtain. At this time, in order to eliminate the influence of this number imbalance, FocalLoss can be used to train the accident detection model, for example, α in Focal Loss is set to 0.8 and γ is set to 2. Of course, the technician can adjust the hyperparameters in the model training process according to the needs, and the embodiment of the present application is not limited thereto. The accident classification value can be regarded as the probability of the target vehicle having an accident, so its value range can be (0, 1). In some embodiments, the accident detection model is a LightGBM model, and of course it can also be other binary classification models. In the embodiment of the present application, the LightGBM model is optimized, and the feature bin number is reduced from 256 to 64, which significantly reduces the model size and helps to deploy on the vehicle terminal. In addition, the embodiment of the present application also provides the feature importance ranking of LightGBM, which improves the explainability of accident judgment. For example, in the judgment of the accident, the weight proportion of the brake feature is 32%.

[0171] Optionally, after step 305, the vehicle terminal can perform at least one of steps 306-309.

[0172] 306、In the case of an accident of the target vehicle, the vehicle terminal determines an accident type of the accident based on the target fusion feature.

[0173] The accident type is a classification result of the accident, for example, the accident type includes rear-end collision, being rear-ended, colliding with an obstacle, and rollover, etc., which is set by a technician according to actual conditions.

[0174] In a possible implementation, the vehicle terminal encodes the target fusion feature based on an attention mechanism to obtain a fusion encoding feature of the target fusion feature. The vehicle terminal fully connects and normalizes the fusion encoding feature to obtain a probability set, the probability set including a plurality of probabilities, one probability corresponding to one candidate accident type. The vehicle terminal determines a candidate accident type corresponding to the highest probability in the probability set as the accident type of the accident.

[0175] In this implementation, the target fusion feature is encoded based on the attention mechanism, which can mine the correlation between the vehicle state feature and the vehicle visual feature to obtain the fusion encoding feature, and the accuracy of classifying the accident by using the fusion encoding feature is high.

[0176] For example, the vehicle terminal inputs the target fusion feature into an accident classification model, embeds and encodes the target fusion feature by using the accident classification model to obtain a fusion embedding feature of the target fusion feature. The vehicle terminal encodes the fusion embedding feature based on the attention mechanism by using the accident classification model to obtain a fusion encoding feature of the target fusion feature. The vehicle terminal fully connects and normalizes the fusion encoding feature to obtain a probability set. The vehicle terminal determines a candidate accident type corresponding to the highest probability in the probability set as the accident type of the accident.

[0177] The accident classification model is a multi-classification model based on an attention mechanism. The accident classification model can be used to classify accidents and obtain the type of the accident. When training the accident classification model, a plurality of sample fusion features and the labeled accident types corresponding to the plurality of sample fusion features are used, and the training process includes a plurality of iterations. The normalization can be implemented by using a Softmax function. In addition, to alleviate the overfitting of the accident classification model, label smoothing can be introduced, and a learnable position encoding can be added to the accident classification model, so that the above-mentioned encoding process based on the attention mechanism can better capture the time sequence relationship of the multi-modal features, i.e., the target fusion features. In some embodiments, the accident classification model is a Transformer classifier, which includes a Transformer encoder and a classification head. The Transformer encoder is used to encode the target fusion features based on the attention mechanism, and the classification head is used for full connection and normalization to obtain a probability set. In the embodiments of the present application, a class activation heat map of the accident classification model is provided to highlight the collision-related area, facilitating the backtracking of the accident classification.

[0178] 307、In the case where the target vehicle has an accident, the vehicle terminal determines the accident severity of the accident based on the target fusion feature and the vehicle state feature.

[0179] The accident severity can indirectly represent the damage degree of the target vehicle, thereby representing the degree of repair required by the target vehicle. Generally, the accident severity is represented by an accident level. In addition, the accident severity is usually directly related to the impact strength of the accident, so the accident severity can be replaced by the impact strength. Generally, the greater the impact strength, the higher the accident severity; the smaller the impact strength, the lower the accident severity.

[0180] In one possible implementation, the vehicle terminal splices the vehicle state feature and the target fusion feature to obtain a second spliced feature. The vehicle terminal performs full connection and normalization on the second spliced feature to obtain the accident level of the accident, and the accident level is used to represent the accident severity.

[0181] In this implementation, the vehicle state feature and the target fusion feature are spliced again to obtain a second spliced feature. The second spliced feature is used to determine the accident level, and the accuracy of the accident level is higher.

[0182] For example, the vehicle terminal concatenates the vehicle state feature and the target fusion feature to obtain a second concatenated feature. The vehicle terminal then inputs the second concatenated feature into a severity recognition model, which then fully connects and normalizes the second concatenated feature to obtain a predicted accident level. Based on the predicted accident level, the vehicle terminal determines the accident level.

[0183] Among them, the severity recognition model is a regression model, which can map the input second splicing feature to the accident level prediction value using the severity recognition model. During training, the severity recognition model will use multiple sample splicing features and the labeled accident level prediction value corresponding to each sample splicing feature, so that the severity recognition model learns the relationship between the splicing feature and the accident level prediction value, thereby completing the regression. In some embodiments, the severity recognition model is an XGBoost model, or other regression model, which is not limited in the embodiments of the present application. Normalization in the above process can be achieved using the Sigmoid function. In the embodiments of the present application, an XGBoost SHAP (SHapley Additive exPlanations) value analysis function is provided to show the sharing degree of different features for the severity of the accident.

[0184] The following describes the method of determining the accident level based on the accident level prediction value in the above example.

[0185] In some embodiments, the vehicle terminal determines the accident level by using the following formula (2) to calculate the accident level prediction value.

[0186] Level=N×Sigmoid(y pred ) (2)

[0187] Where Level represents the accident level, and N represents the number of accident levels. If there are 5 accident levels, then N = 5, y pred Indicates the predicted value of accident level.

[0188] In the above steps 305-307, the accident detection model, the accident classification model and the severity recognition model are used respectively. When training the accident detection model, the accident classification model and the severity recognition model, the accident detection model, the accident classification model and the severity recognition model can be trained independently first, and then jointly fine-tuned. In this way, the accident detection model, the accident classification model and the severity recognition model can maintain consistency. The loss function of the joint fine-tuning is shown in the following formula (3).

[0189] L=λ1·L ce +λ2·L mse +λ3·Lkl (3)

[0190] Wherein, L is a joint loss function, λ1 is a first weight, λ2 is a second weight, and λ3 is a third weight. The three weights are set by the technician according to the actual situation, and the embodiments of the present application do not limit this. ce L is a loss function of the accident detection model. mse L is a loss function of the accident classification model. kl L is a loss function of the severity identification model.

[0191] In addition, when constructing the training samples of the above-mentioned accident detection model, accident classification model and severity identification model, the diversity of the training samples can be improved by means of Gaussian noise injection and random occlusion, so that the model obtained by training has stronger generalization ability.

[0192] 308、In the case that the target vehicle has an accident, the vehicle terminal enables the safety island mechanism to store the vehicle state information and the visual information.

[0193] Wherein, the safety island (Safety Island) is an isolated protection zone specially designed to improve the safety of vehicle electronic systems. The safety island mechanism is enabled to store vehicle state information and visual information, which can ensure the safety of vehicle state information and visual information, prevent power loss, and facilitate subsequent accident analysis.

[0194] In addition, in addition to caching the vehicle state information and the visual information, the vehicle terminal can also cache the original data (such as IMU original waveform) within a preset time period before and after the accident. The subsequent can be uploaded to the cloud platform, or local analysis. The preset time period is set by the technician according to the actual situation, such as 30s.

[0195] 309、In the case that the target vehicle has an accident, the vehicle terminal uploads the vehicle state information and the visual information to the cloud platform, so that the cloud platform performs accident reconstruction and / or accident liability determination based on the vehicle state information and the visual information.

[0196] The cloud platform is a remote service provider of the target vehicle, the accident reconstruction is to simulate the driving state of the target vehicle by using the vehicle state information, and simulate the driving scene of the target vehicle by using the visual information, for example, the attitude change of the target vehicle is obtained by using the vehicle state information, the driving scene is matched by using the visual information, the accident animation is obtained by using the rendering engine to combine the attitude change of the target vehicle and the driving scene, and the accident animation is the accident reconstruction result. In some embodiments, the rendering engine is UE5, and of course, it can also be other rendering engines, and the embodiments of the present application are not limited thereto. The accident liability determination can be obtained by analyzing the vehicle state information to obtain the turn signal state and the driving trajectory of the target vehicle, comparing the driving trajectory with the lane line data of the high-precision map, and determining whether the target vehicle has violated the lane change.

[0197] In a possible implementation, in the case that the target vehicle has an accident, the vehicle terminal compresses the vehicle state information and the visual information to obtain metadata. The vehicle terminal uploads the metadata to the cloud platform.

[0198] The compression can reduce the occupation of network bandwidth, so as to transmit the vehicle state information and the visual information to the cloud platform faster. In some embodiments, the compression method can use the LZ4 algorithm (compression ratio 1:15), and of course, other compression algorithms can also be used in other possible implementations, and the embodiments of the present application are not limited thereto.

[0199] In addition, the cloud platform has the ability of automatic capacity expansion, for example, one processing node is expanded for each preset number of accidents, the preset number is set by a technician according to the actual situation, for example, 100, and the processing node is a stage with the above-mentioned accident reconstruction and accident liability determination capability, for example, a GPU (Graphics Processing Unit, image processor) node.

[0200] In addition, the cloud platform has a priority judgment function, and for secondary accidents, priority processing is performed.

[0201] In order to more clearly illustrate the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application will be introduced below in conjunction with Figure 5 Figure 5 .

[0202] ​The vehicle terminal's data acquisition layer acquires the target vehicle's state and visual information. Hardware clock synchronization is performed on the vehicle state and visual information, and the vehicle state information is sampled using a sliding window and keyframe extraction. The vehicle terminal's feature extraction layer processes the sampled vehicle state information using dynamic data branching, which can be implemented using the 5-layer residual structure of an MLP network, ultimately generating 128-dimensional vehicle state features. The vehicle terminal's feature extraction layer processes the sampled visual information using the visual data branching, which can be implemented using ViT-Base's 12-layer Transformer, ultimately generating 768-dimensional vehicle visual features. The vehicle terminal's multimodal fusion layer concatenates the 128-dimensional vehicle state features and the 768-dimensional vehicle visual features to generate a first concatenated feature. This first concatenated feature is processed using a cross-attention mechanism to generate a target fused feature. The vehicle terminal's multi-task recognition layer includes an accident detection model, an accident classification model, and a severity recognition model, each used to perform the corresponding accident recognition task. A lightweight deployment approach was adopted when deploying accident detection, classification, and severity identification models to reduce processing pressure on the vehicle terminal. The vehicle terminal uploads data to the cloud platform via the edge-cloud collaboration layer, enabling 3D accident reconstruction and / or accident responsibility determination.

[0203] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0204] Through the technical solution provided in the embodiment of the present application, in view of the uniqueness of the field of vehicle accident analysis, the vehicle state characteristics of the target vehicle (such as speed, acceleration, etc., which are extracted based on vehicle state information) and the vehicle visual characteristics (such as features extracted from visual information collected by the camera) are deeply fused, and the target fusion features are generated specifically through splicing and cross-attention mechanisms. This fusion method can provide information complementarity in vehicle accident scenes (for example, state characteristics capture physical dynamics, and visual characteristics provide environmental context), and is also significantly different from other image recognition fields (such as security or medical image recognition), which usually rely on a single modality or static images for processing. In the embodiment of the present application, based on the target fusion feature, a high-accuracy accident judgment is achieved. More importantly, when an accident occurs, the fusion feature is further used to identify the type of accident and / or the severity of the accident. This overcomes the defects of low accuracy and single function of existing collision sensor technology, provides a comprehensive accident identification solution dedicated to the field of vehicle accident analysis, and improves the availability and reliability of accident identification.

[0205] Figure 6 This is a schematic diagram of the structure of an accident identification device provided in an embodiment of the present application, see Figure 6The device comprises a feature fusion module 601, an accident judgment module 602, and an accident analysis module 603.

[0206] The feature fusion module 601 is configured to fuse a vehicle state feature of a target vehicle and a vehicle visual feature to obtain a target fusion feature of the target vehicle, the vehicle state feature being obtained by feature extraction on vehicle state information of the target vehicle, and the vehicle visual feature being obtained by feature extraction on visual information collected from the target vehicle.

[0207] The accident judgment module 602 is configured to determine whether the target vehicle has an accident based on the target fusion feature.

[0208] The accident analysis module 603 is configured to, in the case that the target vehicle has an accident, determine an accident type of the accident based on the target fusion feature, and / or determine an accident severity of the accident based on the target fusion feature and the vehicle state feature.

[0209] In a possible implementation, the feature fusion module 601 is configured to splice the vehicle state feature and the vehicle visual feature to obtain a first spliced feature. The vehicle state feature and the vehicle visual feature are fused based on a cross-attention mechanism to obtain an initial fusion feature. The first spliced feature and the initial fusion feature are fused to obtain the target fusion feature.

[0210] In a possible implementation, the feature fusion module 601 is configured to perform a first linear transformation on the vehicle state feature to obtain a first query matrix. A second linear transformation is performed on the vehicle visual feature to obtain a first key matrix, and a third linear transformation is performed on the vehicle visual feature to obtain a first value matrix. A first attention weight is determined based on the first query matrix and the first key matrix. The first attention weight is multiplied by the first value matrix to obtain the initial fusion feature.

[0211] In a possible implementation, the accident judgment module 602 is configured to determine a first classification sub-feature from the target fusion feature, the first classification sub-feature having a dimension lower than that of the target fusion feature. An accident classification value is determined based on the first classification sub-feature. In the case that the accident classification value is greater than or equal to a classification value threshold, it is determined that the target vehicle has an accident, the classification value threshold being determined based on a vehicle speed of the target vehicle. In the case that the accident classification value is less than the classification value threshold, it is determined that the target vehicle does not have an accident.

[0212] In a possible implementation, the accident judging module 602 is configured to obtain the first classification sub-feature by fusing the first N sub-features in the target fusion feature. Alternatively, the first classification sub-feature is obtained by performing principal component analysis on the target fusion feature. Alternatively, the first classification sub-feature is obtained by performing linear transformation on the target fusion feature.

[0213] In a possible implementation, the accident analyzing module 603 is configured to encode the target fusion feature based on an attention mechanism to obtain a fusion encoded feature of the target fusion feature. The fusion encoded feature is fully connected and normalized to obtain a probability set, where the probability set includes a plurality of probabilities, and one probability corresponds to one candidate accident type. The candidate accident type corresponding to the highest probability in the probability set is determined as the accident type of the accident.

[0214] In a possible implementation, the accident analyzing module 603 is configured to splice the vehicle state feature and the target fusion feature to obtain a second spliced feature. The second spliced feature is fully connected and normalized to obtain an accident level of the accident, where the accident level is used to represent the severity of the accident.

[0215] In a possible implementation, the apparatus further includes a feature extraction module configured to obtain the vehicle state information and the visual information. The vehicle state information and the visual information are synchronized by a hardware clock. The synchronized vehicle state information is sampled by using a sliding window to obtain initial sampling information. The synchronized visual information is key frame extracted to obtain a plurality of key frames. The initial sampling information is feature extracted to obtain the vehicle state feature. The plurality of key frames are feature extracted to obtain the vehicle visual feature.

[0216] In a possible implementation, the apparatus further includes an information processing module configured to, in a case where the target vehicle has an accident, enable a safety island mechanism to store the vehicle state information and the visual information. The vehicle state information and the visual information are uploaded to a cloud platform, so that the cloud platform performs accident reconstruction and / or accident liability determination based on the vehicle state information and the visual information.

[0217] It should be noted that the accident identification apparatus provided in the above examples identifies the accident by using the division of the above functional modules as an example, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the accident identification apparatus and the accident identification method provided in the above examples belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0218] By means of the technical solutions provided in the embodiments of the present application, the vehicle state features and the vehicle visual features of the target vehicle are fused to obtain target fusion features of the target vehicle, and the target fusion features are used to determine whether the target vehicle has an accident, which is equivalent to combining vehicle state information and visual information in two dimensions to determine the fault, and the determination accuracy is high. In addition, in the case of an accident, the target fusion features are used to identify the accident type and / or the accident severity, so as to obtain the corresponding accident situation.

[0219] The embodiments of the present application also provide a vehicle, Figure 7 FIG. 1 is a structural schematic diagram of a vehicle provided by the embodiments of the present application.

[0220] Generally, the vehicle 700 includes one or more processors 701 and one or more memories 702.

[0221] The processor 701 can include one or more processing cores, such as a 4-core processor, a 7-core processor, etc. The processor 701 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 701 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 701 can be integrated with a GPU, which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 701 can also include an AI (Artificial Intelligence) processor, which is used to process machine learning-related computing operations.

[0222] The memory 702 can include one or more computer-readable storage media, which can be non-transitory. The memory 702 also includes a high-speed random access memory and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one computer program for being executed by the processor 701 to implement the accident identification method provided by the method embodiments of the present application.

[0223] Those skilled in the art can understand that,Figure 7 The structure shown in the figure does not constitute a limitation on the vehicle 700, and can include more or fewer components than shown, or combine some components, or adopt a different component arrangement.

[0224] In addition, the apparatus provided by the embodiments of the present application can be a chip, a component, or a module, and the chip can include a processor and a memory connected to each other. The memory is used to store instructions, and when the processor invokes and executes the instructions, the chip can execute the method for accident identification provided by the above embodiments.

[0225] The embodiment further provides a computer-readable storage medium, which stores computer program codes. When the computer program codes run on a computer, the computer is caused to execute the above related method steps to implement the method for accident identification provided by the above embodiments.

[0226] The embodiment further provides a computer program product, which causes a computer to execute the above related steps when the computer program product runs on the computer, so as to implement the method for accident identification provided by the above embodiments.

[0227] The apparatus, the computer-readable storage medium, the computer program product or the chip provided by the embodiments can be used to execute the corresponding method provided above, and thus the beneficial effects achieved by the apparatus, the computer-readable storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided above, which will not be described here.

[0228] From the above description of the embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above.

[0229] In the embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the above-described apparatus embodiments are only schematic, and the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between units can be indirect coupling or communication connection through some interfaces, and can be electrical, mechanical or other forms.

[0230] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An accident identification method, characterized in that: The method comprises: Fusing the vehicle state feature and the vehicle visual feature of the target vehicle to obtain a target fusion feature of the target vehicle, wherein the vehicle state feature is obtained by extracting the vehicle state information of the target vehicle, and the vehicle visual feature is obtained by extracting the visual information collected from the target vehicle; Determining whether the target vehicle has an accident based on the target fusion feature; In the event that an accident occurs to the target vehicle, the accident type of the accident is determined based on the target fusion feature, and / or the accident severity of the accident is determined based on the target fusion feature and the vehicle state feature.

2. The method according to claim 1, characterized in that The step of fusing the vehicle state features and the vehicle visual features of the target vehicle to obtain the target fusion features of the target vehicle includes: Splicing the vehicle state feature and the vehicle visual feature to obtain a first splicing feature; The vehicle state features and the vehicle visual features are fused based on a cross attention mechanism to obtain an initial fused feature; The first splicing feature and the initial fusion feature are fused to obtain the target fusion feature.

3. The method according to claim 2, characterized in that The vehicle state features and vehicle visual features are fused based on the cross attention mechanism to obtain initial fused features, including: Performing a first linear transformation on the vehicle state feature to obtain a first query matrix; Performing a second linear transformation on the vehicle visual features to obtain a first key matrix, and performing a third linear transformation on the vehicle visual features to obtain a first value matrix; determining a first attention weight based on the first query matrix and the first key matrix; Multiplying the first attention weight by the first value matrix to obtain the initial fusion feature.

4. The method according to claim 1, wherein The determining whether the target vehicle has an accident based on the target fusion feature includes: determining a first classification sub-feature from the target fusion feature, where the dimension of the first classification sub-feature is lower than that of the target fusion feature; determining an accident classification value based on the first classification sub-feature; determining that an accident has occurred with the target vehicle if the accident classification value is greater than or equal to a classification value threshold, the classification value threshold being determined based on a speed of the target vehicle; When the accident classification value is less than the classification value threshold, it is determined that no accident has occurred with the target vehicle.

5. The method according to claim 4, characterized in that The determining of the first classification sub-feature from the target fusion feature includes: The sub-features of the first N dimensions in the target fusion feature are combined to obtain the first classification sub-feature; Alternatively, performing principal component analysis on the target fusion feature to obtain the first classification sub-feature; Alternatively, a linear transformation is performed on the target fusion feature to obtain the first classification sub-feature.

6. The method according to claim 1, wherein The determining the accident type of the accident based on the target fusion feature includes: Encoding the target fusion feature based on the attention mechanism to obtain a fusion encoding feature of the target fusion feature; Fully connecting and normalizing the fused coding features to obtain a probability set, where the probability set includes multiple probabilities, and one probability corresponds to one candidate accident type; The candidate accident type corresponding to the highest probability in the probability set is determined as the accident type of the accident.

7. The method according to claim 1, characterized in that The determining the severity of the accident based on the target fusion feature and the vehicle state feature includes: Splicing the vehicle state feature and the target fusion feature to obtain a second splicing feature; The second concatenated features are fully connected and normalized to obtain an accident level of the accident, where the accident level is used to indicate the severity of the accident.

8. The method according to claim 1, characterized in that Before fusing the vehicle state features and the vehicle visual features of the target vehicle to obtain the target fusion features of the target vehicle, the method further includes: acquiring the vehicle status information and the visual information; Synchronizing the vehicle status information and the visual information through a hardware clock; Sampling the synchronized vehicle status information using a sliding window to obtain initial sampling information; Extracting key frames from the synchronized visual information to obtain multiple key frames; Performing feature extraction on the initial sampling information to obtain the vehicle state feature; Feature extraction is performed on the multiple key frames to obtain the vehicle visual features.

9. The method according to claim 1, characterized in that The method further comprises: In the event of an accident involving the target vehicle, enabling a safety island mechanism to store the vehicle status information and the visual information; The vehicle status information and the visual information are uploaded to a cloud platform, so that the cloud platform can perform accident reconstruction and / or accident responsibility determination based on the vehicle status information and the visual information.

10. A vehicle, characterized in that: The vehicle comprises: a memory for storing executable program code; A processor is used to call and run the executable program code from the memory, so that the vehicle executes the accident identification method as described in any one of claims 1 to 9.

Images