Vehicle collision detection method based on multi-modal fusion visual terminal

Abstract

The invention discloses a vehicle collision detection method based on a multi-modal fusion visual terminal, and the method comprises the steps: S1, analyzing a track data packet uploaded by a device end in real time, and obtaining vehicle motion parameters of a predetermined time length and a predetermined dimension; s2, collecting audio and video data of a preset time window of the trigger track data packet, and decoding audio and video information of a set time length; s3, performing data preprocessing on each analyzed modal parameter, constructing a feature set of a single-modal collision detection dimension, and performing feature screening; s4, fusing the sensor gyroscope and the audio and video multi-modal features; and S5, constructing a multi-modal subdivision scene based on the multi-modal features, and constructing a push rule of vehicle collision detection by adopting a decision tree algorithm. According to the invention, modeling is carried out by fusing the three-axis acceleration and audio and video data acquired by the sensor, so that the collaborative and complementary detection of vehicle collision based on multi-modal data is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle collision detection, and in particular to a vehicle collision detection method based on a multi-modal fusion visual terminal. Background Art

[0002] Currently, in vehicle collision detection, most of the vehicle collision detection is based on independent single-mode data. There are few technologies that fuse multi-modal data from vehicle-mounted driving recorders for collision detection. In addition, single-mode vehicle collision detection technology has the following disadvantages:

[0003] 1. When the current vehicle collision detection is based on single-mode data, the independence is extremely strong, resulting in the vehicle collision detection system being triggered as long as one of the multiple modes is abnormal, thus ignoring the synergy between the modes to restrict each other;

[0004] 2. The single-mode detection capability is not sensitive in some scenarios. For example, the sensor detection is not sensitive to minor scratches, and the video detection technology is not sensitive to out-of-field collisions (chassis scratches). Summary of the invention

[0005] To solve the above problems, the present invention provides a vehicle collision detection method based on a multimodal fusion visual terminal. When a vehicle collides, modeling is performed by fusing the three-axis acceleration, audio and video data collected by the sensor, thereby realizing detection of whether the vehicle has a scratch collision based on the synergy and complementarity of multimodal data.

[0006] The present invention provides a vehicle collision detection method based on a multimodal fusion visual terminal, and the specific technical solution is as follows:

[0007] S1: Analyze the trajectory data packets uploaded by the device in real time to obtain the vehicle motion parameters of a predetermined duration and dimension;

[0008] S2: Collect audio and video data of the predetermined time window of the trigger trajectory data packet and decode the audio and video information of the predetermined time length;

[0009] S3: Perform data preprocessing on the parsed modal parameters, construct a feature set of single-modal collision detection dimension, and perform feature screening;

[0010] S4: Fusion of sensor gyroscope and audio and video multimodal features;

[0011] S5: Construct multimodal segmentation scenarios based on multimodal features, and use decision tree algorithm to build push rules for vehicle collision detection.

[0012] Furthermore, in step S1, the data obtained by parsing the trajectory data packet also includes the data upload time;

[0013] The vehicle motion parameters of a given duration and a predetermined dimension include: positioning state, speed and three-axis acceleration.

[0014] Furthermore, in step S2, taking the time of the triggering trajectory data packet as the center point, the audio and video data are intercepted according to a time window of a set length, invalid video data is eliminated, and audio and video data of a predetermined time length are obtained.

[0015] Furthermore, in step S3, the data preprocessing includes eliminating invalid values ​​that exceed the predetermined acquisition range of the sensor, eliminating dimensions with higher missing rates, and filling dimensions with lower missing rates with the median of previous and next data packets.

[0016] Furthermore, in step S3, the feature set of the unimodal collision detection dimension includes vehicle collision detection features constructed based on a sensor gyroscope, vehicle collision detection features constructed based on audio data, vehicle collision detection features constructed based on video data, and driving scene features.

[0017] Furthermore, the vehicle collision detection features constructed based on the sensor gyroscope include the maximum sudden fluctuation amount of three axes, the slope of the three-axis with larger changes, the duration of the maximum sudden fluctuation of three axes and the collision level. The collision level is a collision level that is supervised and trained based on the maximum sudden fluctuation amount of three axes, the slope of the three-axis with larger changes and the duration of the maximum sudden fluctuation of three axes.

[0018] Furthermore, the vehicle collision detection features constructed based on the audio data include: a human voice abnormality alarm score, a vehicle distance alarm sound score, a vehicle collision alarm score, and a sound abnormality degree level.

[0019] Furthermore, vehicle collision detection features and driving scene features are constructed based on video data, as follows:

[0020] Based on the target detection algorithm, detect and count the category and location information of the target;

[0021] Calculate the distance between the vehicle camera and the detection target based on the detected target position information and the camera internal reference focal length of the driving recorder;

[0022] The driving scene is constructed by detecting the number of frames and distances of the target objects.

[0023] Further, in step S4, the sensor gyroscope and audio and video multimodal features are fused, including constructing multimodal cooperative derived features and multimodal complementary derived features;

[0024] Constructing multimodal collaborative derived features, the specific process is as follows:

[0025] According to the independent single modal dimensional features, a three-dimensional space is constructed with the sensor modality as the x-axis, the audio modality as the y-axis, and the video modality as the z-axis;

[0026] Calculate the normal vector of each plane in the three-dimensional space, and calculate the angle between any two planes;

[0027] According to the normal vector of each plane, the projection of the independent mode on the plane formed by the other two modes is obtained, and the projection mode length is calculated;

[0028] Constructing multimodal complementary derived features, the specific process is as follows:

[0029] Calculate the information gain of one modality feature based on another modality.

[0030] Furthermore, in step S5, the construction of the multimodal segmentation scenario is specifically as follows:

[0031] The modal fusion subdivision scenes are divided based on the sensor collision level, audio abnormality level and construction of video driving scenes.

[0032] The beneficial effects of the present invention are as follows:

[0033] The present invention integrates the modal features of the gyroscope sensor, the audio modal features and the video modal features, and constructs the synergy and complementarity of multiple modalities to perform collision detection, thereby compensating for the insensitivity of vehicle collision detection in single-modal blind spot scenarios and improving the recall rate of vehicle collision detection. At the same time, based on the synergy between the sensor gyroscope and the audio and video multi-modalities, mutual constraints are performed to improve the precision rate of vehicle collision detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0035] The following description clearly and completely describes the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0037] In the description of the embodiments of the present invention, it should be noted that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art, or the orientation or positional relationship in which the invention product is usually placed when in use, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first" and "second" are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0038] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] Example 1

[0040] Embodiment 1 of the present invention discloses a vehicle collision detection method based on a multi-modal fusion visual terminal, such as Figure 1 As shown, the specific process is as follows:

[0041] S1: Analyze the trajectory data packets uploaded by the device in real time to obtain the vehicle motion parameters of a predetermined duration and dimension;

[0042] The relevant data information fields of the data packet analysis include: upload time, positioning status, speed and three-axis acceleration.

[0043] S2: Collect audio and video data of the predetermined time window of the trigger trajectory data packet and decode the audio and video information of the predetermined time length;

[0044] As a preferred embodiment, the time of the trigger trajectory data packet is taken as the center point, the audio and video data are intercepted according to a time window of a set length, invalid video data is eliminated, and audio and video data of a predetermined length are obtained.

[0045] S3: Perform data preprocessing on the parsed modal parameters, construct a feature set of single-modal collision detection dimension, and perform feature screening;

[0046] As a preferred embodiment, the data preprocessing includes eliminating invalid values ​​beyond the predetermined acquisition range of the sensor, eliminating dimensions with higher missing rates, and filling dimensions with lower missing rates with the median of previous and next data packets.

[0047] As a preferred embodiment, the feature set of the unimodal collision detection dimension includes vehicle collision detection features constructed based on a sensor gyroscope, vehicle collision detection features constructed based on audio data, vehicle collision detection features constructed based on video data, and driving scene features.

[0048] Specifically, the vehicle collision detection features based on the sensor gyroscope construction include the maximum sudden fluctuation amount of three axes, the slope of the three-axis with larger changes, the maximum sudden fluctuation duration of three axes, and the collision level. The collision level is a collision level that is supervised and trained based on the maximum sudden fluctuation amount of three axes, the slope of the three-axis with larger changes, and the maximum sudden fluctuation duration of three axes.

[0049] The vehicle collision detection features constructed based on audio data include: a human voice abnormality alarm score, a vehicle distance alarm sound score, a vehicle collision alarm score, and a sound abnormality degree level.

[0050] Construct vehicle collision detection features and driving scene features based on video data, as follows:

[0051] Based on the target detection algorithm, the category and location information of the target of interest are detected and counted; the category includes, for example, barriers, speed bumps, zebra crossings, traffic lights, etc.

[0052] According to the detected target position information combined with the intrinsic focal length of the camera of the driving recorder, the distance between the vehicle camera and the detected target is calculated based on monocular ranging, as follows:

[0053]

[0054] Among them, D represents the distance between the vehicle camera and the detection target (people and cars), F represents the focal length of the vehicle camera, and P represents the pixel height or width of the detection target (people and cars) in the picture.

[0055] By detecting the number of frames and distance of the target object, the driving scene (including barriers, speed bumps, zebra crossings, traffic lights, intersections, etc.) is constructed.

[0056] S4: Fusion of sensor gyroscope and audio and video multimodal features;

[0057] As a preferred embodiment, the fusion of sensor gyroscope and audio and video multimodal features includes constructing multimodal cooperative derivative features and multimodal complementary derivative features;

[0058] Constructing multimodal collaborative derived features, the specific process is as follows:

[0059] According to the independent single modal dimensional features calculated in step S3, a three-dimensional space is constructed with the sensor modality as the x-axis, the audio modality as the y-axis, and the video modality as the z-axis;

[0060] Calculate the normal vectors of each plane in the three-dimensional space;

[0061] Specifically, the normal vector of the plane where the sensor gyroscope and the audio mode are located is N sa =(x sa ,y sa ,z sa ), the normal vector of the plane where the audio and video modes are located is N av =(x av ,y av ,z av ), the normal vector between the sensor gyroscope and the plane of the video modality is N sv =(x sv ,y sv ,z sv );

[0062] Based on the above-calculated normal vectors of each plane, calculate the angles between each two planes respectively;

[0063] The angle between the plane where the sensor and audio mode are located and the plane where the audio and video mode are located is used as an example for calculation. The angle calculation is expressed as follows:

[0064]

[0065] Among them, θ∈[0°,90°], if θ is greater than 90°, the angle correction is θ=180°-θ;

[0066] According to the normal vectors of each plane, the projection of the independent mode on the plane formed by the other two modes is obtained, and the projection modulus length of the independent mode on the plane formed by the other two modes is calculated respectively;

[0067] Taking the projection of the sensor modality on the plane formed by audio and video as an example, the projection calculation is expressed as follows:

[0068]

[0069] in, and They represent the projection vector of the sensor gyroscope on the plane where the audio and video are located, the projection vector of the video modality on the plane where the sensor gyroscope and the audio modality are located, and the projection vector of the audio modality on the plane where the sensor gyroscope and the video modality are located. and They represent the vehicle collision detection feature vector constructed by the sensor gyroscope, the vehicle collision detection feature vector constructed based on audio data, and the vehicle collision detection feature constructed based on video data, respectively. and They respectively represent the modulus of the normal vector of the plane where the audio and video modalities are located, the modulus of the normal vector of the plane where the sensor gyroscope and audio modalities are located, and the modulus of the normal vector of the plane where the sensor gyroscope and video modalities are located;

[0070] The projection modulus length calculation is expressed as follows:

[0071]

[0072] in, and They respectively represent the projection vector modulus of the sensor gyroscope on the plane where the audio and video are located, the projection vector modulus of the video modality on the plane where the sensor gyroscope and the audio modality are located, and the projection vector modulus of the audio modality on the plane where the sensor gyroscope and the video modality are located;

[0073] Constructing multimodal complementary derived features, the specific process is as follows:

[0074] Calculate and obtain the information gain of one modal feature based on another modality; take the information gain of sensor feature S in the audio modality as an example to illustrate the calculation, the calculation is as follows:

[0075]

[0076] Among them, g sa represents the information gain of sensor feature S on the audio mode, Ent(s) represents the information entropy of sensor feature s, i represents the i-th value of the audio feature, n represents the total number of audio feature values, |a i | represents the number of audio features taking the i-th value, |a| represents the number of samples in the statistical data set, Ent(a) audio information entropy representation p k represents the probability of category k.

[0077] S5: Construct multimodal segmentation scenarios based on multimodal features, and use decision tree algorithm to build push rules for vehicle collision detection.

[0078] The construction of the multimodal segmentation scenario is as follows:

[0079] The modal fusion subdivision scenes are divided based on the sensor collision level, audio abnormality level and construction of video driving scenes.

[0080] According to each segmented scenario, combined with the synergistic and complementary derivative characteristics of multimodal fusion, the decision tree algorithm is used to establish the push rules for vehicle collision detection to achieve vehicle collision detection.

[0081] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A vehicle collision detection method based on a multimodal fusion visual terminal, characterized in that: include: S1: Analyze the trajectory data packets uploaded by the device in real time to obtain the vehicle motion parameters of a predetermined duration and dimension; S2: Collect audio and video data of the predetermined time window of the trigger trajectory data packet and decode the audio and video information of the predetermined time length; S3: Perform data preprocessing on the parsed modal parameters, construct a feature set of single-modal collision detection dimension, and perform feature screening; S4: Fusion of sensor gyroscope and audio and video multimodal features; S5: Construct multimodal segmentation scenarios based on multimodal features, and use decision tree algorithm to build push rules for vehicle collision detection.

2. The vehicle collision detection method based on a multimodal fusion visual terminal according to claim 1 is characterized in that: In step S1, the data obtained by parsing the trajectory data packet also includes the data upload time; The vehicle motion parameters of a given duration and a predetermined dimension include: positioning state, speed and three-axis acceleration.

3. The vehicle collision detection method based on a multimodal fusion visual terminal according to claim 1 is characterized in that: In step S2, the time of the trigger trajectory data packet is taken as the center point, the audio and video data are intercepted according to the time window of the set length, the invalid video data is eliminated, and the audio and video data of the predetermined time length are obtained.

4. The vehicle collision detection method based on a multimodal fusion visual terminal according to claim 1 is characterized in that: In step S3, the data preprocessing includes eliminating invalid values ​​that exceed the predetermined acquisition range of the sensor, eliminating dimensions with higher missing rates, and filling dimensions with lower missing rates using the median of previous and next data packets.

5. The vehicle collision detection method based on a multimodal fusion visual terminal according to claim 1 is characterized in that: In step S3, the feature set of the unimodal collision detection dimension includes vehicle collision detection features constructed based on a sensor gyroscope, vehicle collision detection features constructed based on audio data, vehicle collision detection features constructed based on video data, and driving scene features.

6. The vehicle collision detection method based on a multimodal fusion visual terminal according to claim 5 is characterized in that: The vehicle collision detection features based on the sensor gyroscope structure include the maximum sudden fluctuation amount of three axes, the slope of the three-axis with larger changes, the maximum sudden fluctuation duration of three axes and the collision level. The collision level is a collision level that is supervised and trained based on the maximum sudden fluctuation amount of three axes, the slope of the three-axis with larger changes and the maximum sudden fluctuation duration of three axes.

7. The vehicle collision detection method based on a multimodal fusion visual terminal according to claim 5 is characterized in that: The vehicle collision detection features constructed based on audio data include: a human voice abnormality alarm score, a vehicle distance alarm sound score, a vehicle collision alarm score, and a sound abnormality degree level.

8. The vehicle collision detection method based on a multimodal fusion visual terminal according to claim 5 is characterized in that: Construct vehicle collision detection features and driving scene features based on video data, as follows: Based on the target detection algorithm, detect and count the category and location information of the target; Calculate the distance between the vehicle camera and the detection target based on the detected target position information and the camera internal reference focal length of the driving recorder; The driving scene is constructed by detecting the number of frames and distances of the target object.

9. The vehicle collision detection method based on a multimodal fusion visual terminal according to claim 1 is characterized in that: In step S4, the sensor gyroscope and audio and video multimodal features are integrated, including constructing multimodal cooperative derived features and multimodal complementary derived features; Constructing multimodal collaborative derived features, the specific process is as follows: According to the independent single modal dimensional features, a three-dimensional space is constructed with the sensor modality as the x-axis, the audio modality as the y-axis, and the video modality as the z-axis; Calculate the normal vector of each plane in the three-dimensional space, and calculate the angle between any two planes; According to the normal vector of each plane, the projection of the independent mode on the plane formed by the other two modes is obtained, and the projection mode length is calculated; Constructing multimodal complementary derived features, the specific process is as follows: Calculate the information gain of one modality feature based on another modality.

10. According to the vehicle collision detection method based on a multimodal fusion visual terminal according to claim 1, in step S5, the construction of the multimodal subdivision scene is specifically as follows: The modal fusion subdivision scenes are divided based on the sensor collision level, audio abnormality level and construction of video driving scenes.

Images