Multi-source data fusion vehicle machine screen detection system and method

Abstract

The invention provides a multi-source data fusion vehicle machine screen detection system and method. The method comprises the following steps: collecting source data of multiple data types; performing feature extraction on the source data of each data type, and constructing feature data of each data type; performing fusion processing on the feature data of each data type to obtain fusion features; calling a first damage detection model corresponding to each data type to process the feature data of the same data type to obtain a respective first damage confidence score about the screen; fusing the first damage confidence score obtained by each damage detection model to obtain a second damage confidence score; calling a pre-trained second damage detection model to process the fusion features to obtain a third damage confidence score about the screen; and fusing the second damage confidence score and the third damage confidence score to obtain the final damage confidence of the screen. The accuracy and real-time performance of detection are improved, and effective guarantee is provided for vehicle safety.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic product defect recognition, and in particular to a vehicle screen detection system and method for multi-source data fusion. Background Art

[0002] With the increase in the market share of new energy vehicles and the popularization of L2 and L3 autonomous driving functions, the large screen of the car, as an important carrier of human-vehicle interaction, is becoming increasingly important to investigate and verify whether it is broken and the specific cause in an accident.

[0003] In actual cases, when a car screen is broken due to a non-road traffic accident, in the absence of on-site inspection by traffic police or car insurance surveyors, the main means of detecting the broken screen is currently based on the insured's oral statement, photo or video. Due to the different understandings of the insured on the case claim standards, there are deviations in the actual inspection results of the damaged object.

[0004] This not only increases the complexity and difficulty of claims for the insured, but also affects the accuracy of accident judgment, which is not in line with the goal of customer service-centered, intelligent and convenient compensation. Summary of the invention

[0005] In order to solve the deficiencies of the prior art, the present invention provides a vehicle screen detection system and method with multi-source data fusion to address the problem of vehicle large screen breakage that is not easily detected and is caused by non-road traffic accidents.

[0006] A multi-source data fusion vehicle screen detection system, comprising:

[0007] Multi-source data acquisition module, used to collect source data of various data types;

[0008] A preprocessing module, connected to the multi-source data acquisition module, is used to preprocess various types of source data;

[0009] A feature construction module, connected to the preprocessing module, is used to extract features from the preprocessed source data of each data type and construct feature data of each data type;

[0010] A feature fusion module is connected to the feature construction module and is used to fuse the feature data of each data type to obtain a fused feature;

[0011] A first decision module, connected to the feature construction module, is used to call the first damage detection model corresponding to each data type to process the feature data of the same data type to obtain respective first damage confidence scores for the screen;

[0012] a first fusion module, connected to the first decision module, for fusing the first damage confidence scores obtained by the damage detection models to obtain a second damage confidence score;

[0013] The second decision module is connected to the feature fusion module and is used to call the pre-trained second damage detection model to process the fused features and obtain a third damage confidence score for the screen;

[0014] The second fusion module is connected to the first fusion module and the second decision module respectively, and is used to fuse the second damage confidence score and the third damage confidence score to obtain a final damage confidence of the screen.

[0015] Furthermore, the vehicle-mounted screen is a touch screen, and the multi-source data acquisition module includes:

[0016] An image acquisition unit, used for acquiring image data of the vehicle-mounted screen;

[0017] The touch acquisition unit is used to collect touch data from the vehicle screen.

[0018] A sound collection unit, used for collecting sound data inside the vehicle;

[0019] A vibration collection unit, used to collect vibration impact data of the vehicle;

[0020] A driving behavior collection unit, used to collect driving behavior data of the vehicle;

[0021] The feature construction module is used to extract image feature data from image data, extract touch feature data from touch data, extract sound feature data from sound data, extract vibration feature data from vibration impact data, and extract driving behavior feature data from driving behavior data;

[0022] The feature fusion module fuses the image feature data, the touch feature data, the vibration feature data, the sound feature data and the driving behavior feature data to obtain a fusion feature.

[0023] Furthermore, the feature fusion module forms fusion features by vector concatenation, weighted averaging, or principal component analysis of the image feature data, touch feature data, vibration feature data, sound feature data, and driving behavior feature data.

[0024] Furthermore, the driving behavior characteristic data includes vehicle speed characteristics, acceleration characteristics, steering characteristics, braking characteristics and driving mode characteristics.

[0025] Furthermore, the first fusion module fuses the first damage confidence scores obtained by each damage detection model based on weighted average or Bayesian method to obtain a second damage confidence score.

[0026] A vehicle screen detection method based on multi-source data fusion, characterized in that the vehicle screen detection system based on multi-source data fusion is used, comprising:

[0027] Step A1, collecting source data of various data types;

[0028] Step A2, preprocessing various types of source data;

[0029] Step A3, extracting features from the preprocessed source data of each data type, and constructing feature data of each data type;

[0030] Step A4, fusing the feature data of each data type to obtain a fused feature;

[0031] Step A5, calling the first damage detection model corresponding to each data type to process the feature data of the same data type to obtain respective first damage confidence scores for the screen;

[0032] Step A6, fusing the first damage confidence scores obtained by the damage detection models to obtain a second damage confidence score;

[0033] Step A7, calling a pre-trained second damage detection model to process the fused features, and obtaining a third damage confidence score for the screen;

[0034] Step A8: The second damage confidence score and the third damage confidence score are merged to obtain a final damage confidence of the screen.

[0035] Further, the source data of the multiple data types include image data, touch data, vibration impact data, sound data, and driving behavior data;

[0036] In step A3, image feature data is extracted from the image data, touch feature data is extracted from the touch data, sound feature data is extracted from the sound data, vibration feature data is extracted from the vibration impact data, and driving behavior feature data is extracted from the driving behavior data;

[0037] In step A4, the image feature data, the touch feature data, the vibration feature data, the sound feature data and the driving behavior feature data are fused to obtain fused features.

[0038] Furthermore, in step A4, the image feature data, touch feature data, vibration feature data, sound feature data and driving behavior feature data are combined to form fusion features through vector concatenation, weighted averaging or principal component analysis.

[0039] Furthermore, the driving behavior characteristic data includes vehicle speed characteristics, acceleration characteristics, steering characteristics, braking characteristics and driving mode characteristics.

[0040] Furthermore, in step A6, the first damage confidence scores obtained by the damage detection models are fused based on weighted average or Bayesian method to obtain a second damage confidence score.

[0041] The beneficial technical effect of the present invention is that the collected multi-source data is integrated and analyzed using a deep learning algorithm model to accurately detect the breakage of the large screen of the vehicle computer, thereby improving the accuracy and real-time performance of the detection and providing effective protection for vehicle safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a module of a vehicle screen detection system with multi-source data fusion according to the present invention;

[0043] Figure 2 The present invention is a flowchart of the steps of a vehicle screen detection method using multi-source data fusion. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0046] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0047] See also Figure 1 The present invention provides a multi-source data fusion vehicle screen detection system, comprising:

[0048] A multi-source data acquisition module (1), used for acquiring source data of various data types;

[0049] A preprocessing module (2), connected to the multi-source data acquisition module (1), is used to preprocess various types of source data;

[0050] A feature construction module (3), connected to the preprocessing module (2), is used to extract features from the preprocessed source data of each data type and construct feature data of each data type;

[0051] A feature fusion module (4), connected to the feature construction module (3), is used to fuse the feature data of each data type to obtain a fusion feature;

[0052] A first decision module (5), connected to the feature construction module (3), is used to call the first damage detection model corresponding to each data type to process the feature data of the same data type to obtain respective first damage confidence scores for the screen;

[0053] A first fusion module (6), connected to the first decision module (5), is used to fuse the first damage confidence scores obtained by each damage detection model to obtain a second damage confidence score;

[0054] A second decision module (7), connected to the feature fusion module (4), is used to call a pre-trained second damage detection model to process the fused features and obtain a third damage confidence score for the screen;

[0055] The second fusion module (8) is connected to the first fusion module (6) and the second decision module (7) respectively, and is used to fuse the second damage confidence score and the third damage confidence score to obtain a final damage confidence of the screen.

[0056] The car screen is a large screen.

[0057] Furthermore, the multi-source data acquisition module (1) comprises:

[0058] The image acquisition unit (11) is used to acquire image data of the vehicle-mounted screen.

[0059] Image data of the vehicle screen taken by a built-in or external camera periodically or after a specific event. The data type is image data type.

[0060] The image feature data for constructing the image data includes color features, texture features, shape features, and frequency domain features.

[0061] Color features include color histogram and color moment. Color histogram mainly counts the pixel value distribution of each color channel (such as RGB or HSV) in the image. Color moment mainly calculates the first-order, second-order and third-order moments of the image color channel, reflecting the mean, variance and skewness of the color.

[0062] Texture features include gray-level co-occurrence matrix (GLCM) and local binary pattern (LBP). Gray-level co-occurrence matrix (GLCM) calculates the spatial relationship between gray levels in an image and extracts features such as contrast, correlation, energy, and homogeneity. Local binary pattern (LBP) describes the local texture features of an image and is robust to illumination changes and noise.

[0063] Shape features include edge detection and contour features. Edge detection uses edge detection algorithms such as Canny and Sobel to extract image edge information. Contour features extract the contours of objects in the image and calculate the perimeter, area, convex hull, etc. of the contour.

[0064] Frequency domain features include Fourier transform and wavelet transform. Fourier transform converts the image from the spatial domain to the frequency domain and extracts frequency domain features such as spectrum, phase, etc. Wavelet transform uses wavelet transform to extract multi-scale features of the image and reflect the detailed information of the image at different scales.

[0065] The first damage detection model corresponding to the image data type includes a convolutional neural network algorithm (CNN) and an xgboost algorithm.

[0066] The trained convolutional neural network algorithm (CNN) processes the image feature data to obtain a first damage confidence score regarding the probability of screen damage.

[0067] The xgboost algorithm is used to train the xgboost algorithm. The image feature data (including pixel value distribution of color channels, first-order, second-order and third-order moments, image contrast, image contour perimeter, area, convex hull, spectrum of frequency domain transformation, phase, etc.) constructed from the image data used as training samples are taken as X variables, and the user feedback data (marked as 1 if the screen is abnormal, otherwise marked as 0) is taken as Y variable. The trained xgboost algorithm is then used to process the image feature data to obtain the probability of screen damage.

[0068] Finally, the maximum value of the screen damage probability obtained by the convolutional neural network algorithm (CNN) and the screen damage probability obtained by the xgboost algorithm is used as the first damage confidence score output by the first damage detection model corresponding to the image data type.

[0069] Furthermore, the vehicle-mounted screen is a touch screen, and the multi-source data acquisition module (1) includes:

[0070] The touch acquisition unit (12) is used to acquire touch data of the vehicle-mounted screen.

[0071] The touch acquisition unit (12) collects feedback data from the screen touch sensor in real time, and records touch abnormalities and unresponsive areas. The data type is the touch data type.

[0072] The touch feature data constructed by touch data includes:

[0073] Touch point density, that is, the number of touch points per unit area;

[0074] Touch pressure distribution, i.e., statistics such as the mean and variance of touch pressure;

[0075] Touch mode, such as the length and complexity of the touch path;

[0076] Time series features, such as the time interval and frequency of touch events.

[0077] The first damage detection model corresponding to the touch data type includes an xgboost algorithm.

[0078] The xgboost algorithm is used to train the xgboost algorithm, and the touch feature data (including touch point density, mean and variance of touch pressure, length and complexity of touch path, time interval and frequency of touch events) constructed by the touch data as training samples are used as X variables, and the user feedback data (the screen is abnormal and marked as 1, otherwise marked as 0) is used as the Y variable. The touch feature data is then processed using the trained xgboost algorithm to obtain the first damage confidence score.

[0079] Furthermore, the multi-source data acquisition module (1) comprises:

[0080] The vibration collection unit (13) is used to collect vibration impact data of the vehicle.

[0081] The vibration collection unit (13) is, for example, an acceleration sensor and a gyroscope, which monitors the vibration data and impact data of the vehicle. The vibration collection unit can be provided by the system and is usually installed on the chassis or body frame of the vehicle to measure the vibration and impact and other motion parameters of the vehicle. The data type is a vibration data type.

[0082] The vibration feature data constructed from vibration impact data includes time domain features, frequency domain features, time-frequency domain features, statistical features, morphological features, etc.

[0083] Time domain features include:

[0084] Mean, the average value of the vibration signal;

[0085] Variance, the degree of fluctuation of the vibration signal;

[0086] Standard deviation, the square root of the variance, reflects the stability of the signal;

[0087] Peak, the maximum value of the signal;

[0088] Peak-to-peak value, the difference between the maximum and minimum values ​​of the signal;

[0089] skewness, the asymmetry of the signal distribution;

[0090] Kurtosis, the sharpness of the signal distribution.

[0091] Frequency domain features include:

[0092] Power spectral density (PSD), the distribution of the signal’s energy at different frequencies;

[0093] Spectrum, the Fourier transform result of a signal, reflects the frequency components of the signal;

[0094] Center frequency, the frequency at which the signal energy is concentrated;

[0095] bandwidth, the frequency range over which the signal energy is distributed;

[0096] Time-frequency domain features include:

[0097] Short-time Fourier transform (STFT), which divides the signal into multiple time periods and performs Fourier transform on each of them to obtain a time-frequency domain representation;

[0098] Wavelet transform, use wavelet transform to extract the multi-scale features of the signal and reflect the detailed information of the signal at different scales.

[0099] Statistical features include:

[0100] Autocorrelation function, the correlation of a signal with itself at different time delays;

[0101] Cross-correlation function, the correlation between two signals.

[0102] Morphological features include:

[0103] Waveform length, the total length of the signal;

[0104] Waveform area, the area enclosed by the signal and the time axis.

[0105] The first damage detection model corresponding to the vibration data type includes an LSTM algorithm.

[0106] The LSTM algorithm is used to train the LSTM algorithm, with the vibration feature data constructed from the vibration impact data as training samples as the X variable, and the user feedback data (marked as 1 if the screen is abnormal, otherwise marked as 0) as the Y variable. The vibration feature data is then processed using the trained LSTM algorithm to obtain the first damage confidence score.

[0107] Furthermore, the multi-source data acquisition module (1) comprises:

[0108] The sound collection unit (14) is used to collect sound data inside the vehicle.

[0109] The sound collection unit, such as a microphone, uses the microphone to collect sound data in real time, especially the impact sound in the car. The microphone can be built-in to the vehicle, such as some vehicles with voice recognition function, mainly located near the central control screen. The data type is the sound data type.

[0110] The sound feature data constructed from sound data include time domain features, frequency domain features, time-frequency domain features, statistical features, morphological features, etc., which are the same as the features extracted from vibration impact data.

[0111] The first damage detection model corresponding to the sound data type includes an LSTM algorithm, that is, the same algorithm as that used for the vibration data type.

[0112] The LSTM algorithm is used to train the LSTM algorithm, with the sound feature data constructed from the sound data used as training samples as the X variable and the user feedback data (marked as 1 if the screen is abnormal, otherwise marked as 0) as the Y variable. The sound feature data is then processed using the trained LSTM algorithm to obtain the first damage confidence score.

[0113] Furthermore, the multi-source data acquisition module (1) comprises:

[0114] The driving behavior collection unit (15) is used to collect driving behavior data of the vehicle.

[0115] The data type of driving behavior data is driving data type. Driving behavior data recorded by vehicle sensors, such as sudden braking, collision, acceleration, steering angle, etc.

[0116] The driving behavior characteristic data constructed from the driving behavior data include vehicle speed characteristics, acceleration characteristics, steering characteristics, braking characteristics and driving mode characteristics.

[0117] Vehicle speed characteristics include:

[0118] Average speed, the average speed during driving;

[0119] Speed ​​variance, the degree of fluctuation of vehicle speed;

[0120] Maximum speed, the maximum speed during driving;

[0121] Minimum vehicle speed: the minimum vehicle speed during driving.

[0122] Acceleration characteristics include:

[0123] Average acceleration, the average acceleration during driving;

[0124] Acceleration variance, the degree of fluctuation of acceleration;

[0125] Maximum acceleration, the maximum acceleration during driving;

[0126] Minimum acceleration, the minimum acceleration during driving.

[0127] Steering features include:

[0128] Average steering angle, the average steering angle during driving;

[0129] Steering angle variance, the degree of fluctuation of the steering angle;

[0130] Maximum steering angle, the maximum steering angle during driving;

[0131] Minimum steering angle: the minimum steering angle during driving.

[0132] Braking features include:

[0133] Average braking force, the average braking force during driving;

[0134] Braking force variance, the degree of fluctuation in braking force;

[0135] Maximum braking force, the maximum braking force during driving;

[0136] Minimum braking force: the minimum braking force during driving.

[0137] Driving mode features include:

[0138] Driving mode change frequency, the frequency of driving mode changes (such as smooth, aggressive);

[0139] Driving mode duration, the duration of different driving modes.

[0140] The first damage detection model corresponding to the driving behavior data type includes an XGBoost algorithm and an entropy weight method, that is, the XGBoost algorithm is used to establish a relationship model between driving behavior and screen damage.

[0141] The XGBoost algorithm analyzes the impact of various driving behavior characteristics on screen damage and determines the key influencing factors.

[0142] Based on the key influencing factors determined above, the entropy weight method is used to determine the weight of each influencing factor, and then the weighted sum is taken to obtain the first damage confidence score.

[0143] The entropy weight method is an objective weighting method that calculates the information entropy of each indicator, determines the weight of each indicator, and then scores the driving behavior. The lower the score, the worse the driving behavior and the greater the potential impact on the vehicle screen damage.

[0144] The preprocessing module (2) preprocesses the collected image data, sound data, touch data, vibration impact data and driving behavior data. Different data types have different preprocessing methods, including image denoising, sound signal filtering, touch signal calibration and driving behavior data analysis, etc. When used as training data, it prepares for subsequent deep learning model training. When used as inference to be evaluated, it prepares for subsequent inference.

[0145] When used as training data, user feedback data is also added and used as auxiliary data for model training, correction and optimization. A user feedback interface is provided to collect screen damage reports from car owners as user feedback data.

[0146] The first fusion module is decision-level fusion, which usually involves the combination of decision results obtained from multiple independent decision modules (the first damage detection models of the first decision module). Assume that we have m independent decision modules, and the decision result of each module is d j , where j represents the jth decision module. Decision-level fusion can be expressed as:

[0147]

[0148] in:

[0149] D is the fused decision result, i.e., the second damage confidence score;

[0150] It is a fusion function, which can be voting, weighted average, Bayesian fusion, etc.

[0151] The feature fusion module (4) fuses the image feature data, touch feature data, vibration feature data, sound feature data and driving behavior feature data to obtain a fusion feature. The feature fusion module is a feature-level fusion module that forms a comprehensive feature vector by using a feature fusion mechanism to fuse the feature data such as image, sound, vibration, touch and driving behavior constructed by each source data. A multimodal fusion algorithm is used to improve the accuracy of detection. Afterwards, the decision layer performs classification to determine whether the screen is broken and outputs a second damage confidence score.

[0152] Feature-level fusion usually involves the combination of feature vectors extracted from multiple sensors or data sources. Assume that we have n data acquisition units, and the feature data (in the form of feature vectors) extracted by each data acquisition unit is f i , where i represents the i-th data acquisition unit. Feature-level fusion can be expressed as:

[0153] F=φ(f1,f2,...,f n )

[0154] in:

[0155] F is the fused feature vector, that is, the final fused feature vector after dimensionality reduction.

[0156] φ is a fusion function, which can be simple vector concatenation, weighted average, principal component analysis (PCA), etc. In other words, dimensionality reduction can be performed or not.

[0157] Then the initial fused vector is reduced in dimension to reduce the computational difficulty.

[0158] The decision results after feature fusion and decision-level fusion are fused to obtain the final damage confidence. The formula is:

[0159] P=χ(F,D)

[0160] in,

[0161] P is the final damage confidence;

[0162] χ is a comprehensive fusion function, which can be weighted average, Bayesian network, etc.

[0163] Description of the application example of feature data: Touch feature data, extracted feature vector [0.2, 0.5, 0.1] (after standardization, the same below).

[0164] Image feature data, extracted feature vector [0.4, 0.3, 0.3].

[0165] Vibration feature data, extracted feature vector [0.3, 0.2, 0.4].

[0166] Sound feature data, extracted feature vector [0.1, 0.3, 0.2].

[0167] Driving behavior feature data, extracted feature vector [0.1, 0.2, 0.1].

[0168] Feature level fusion, the normalized feature vectors are concatenated to obtain the fused feature vector [0.2, 0.5, 0.1, 0.4, 0.3, 0.3, 0.3, 0.2, 0.4, 0.1, 0.3, 0.2, 0.1, 0.2, 0.1].

[0169] Feature processing, use PCA to reduce the dimension of the fused feature vector, assuming that the new feature vector [0.4, 0.3, 0.2] is obtained.

[0170] Scoring, use the new feature vector to input the second damage detection model, and the second damage detection model outputs a third damage confidence score of 0.8.

[0171] The decision results of each first damage detection model are:

[0172] Touch feature data, first damage confidence score 0.8.

[0173] Image feature data, first damage confidence score 0.7.

[0174] Vibration characteristic data, first damage confidence score 0.5.

[0175] Sound feature data, first damage confidence score 0.6.

[0176] Driving behavior characteristic data, first damage confidence score 0.7.

[0177] The decision-level fusion performs a weighted average fusion algorithm on the first damage confidence scores output by each first damage detection model. Settings: touch feature data weight 0.3, image feature data weight 0.3, vibration feature data weight 0.15, sound feature data weight 0.15, driving behavior feature data weight 0.1.

[0178] Calculate the weighted average: (0.8*0.3)+(0.7*0.3)+(0.5*0.15)+(0.6*0.15)+(0.7*0.1)=0.685.

[0179] The decision results of feature level fusion and decision fusion are further weighted and fused.

[0180] Set the weight of the decision result of feature-level fusion to 0.5, and the weight of the decision result of decision-level fusion to 0.5.

[0181] Finally, the damage confidence of the large screen of the car is: (0.8*0.5)+(0.685*0.5)=0.7425.

[0182] The final damage confidence level is used to determine whether the screen is broken. If the screen is broken or suspected to be broken, the owner is notified through the vehicle interface, voice prompts, or mobile phone applications, and further inspection or repair is recommended. User feedback information is recorded to continuously optimize the performance of each deep learning model.

[0183] See also Figure 2 The present invention also provides a vehicle screen detection method based on multi-source data fusion, using the aforementioned vehicle screen detection system based on multi-source data fusion, comprising:

[0184] Step A1, collecting source data of various data types;

[0185] Step A2, preprocessing various types of source data;

[0186] Step A3, extracting features from the preprocessed source data of each data type, and constructing feature data of each data type;

[0187] Step A4, fusing the feature data of each data type to obtain a fused feature;

[0188] Step A5, calling the first damage detection model corresponding to each data type to process the feature data of the same data type to obtain respective first damage confidence scores for the screen;

[0189] Step A6, fusing the first damage confidence scores obtained by the damage detection models to obtain a second damage confidence score;

[0190] Step A7, calling a pre-trained second damage detection model to process the fused features, and obtaining a third damage confidence score for the screen;

[0191] Step A8: The second damage confidence score and the third damage confidence score are merged to obtain a final damage confidence of the screen.

[0192] Common scenarios of vehicle screen breakage caused by non-road traffic accidents are as follows:

[0193] 1. Temperature changes

[0194] Thermal expansion and contraction: Extreme temperature changes may cause the screen material to expand and contract, resulting in stress and causing the screen to break.

[0195] Thermal shock: For example, on a hot summer day, the temperature inside the car is extremely high. If you suddenly rinse the vehicle with cold water, the screen may shatter due to the drastic temperature change.

[0196] 2. Physical shock

[0197] Daily use: In daily use, if a hard object accidentally hits the screen, it may cause the screen to break.

[0198] Parking collision: When parking, if the vehicle is hit by other objects, especially side impact, the screen may be shattered.

[0199] 3. Manufacturing Defects

[0200] Material issues: If there are defects in the screen material, such as internal stress that has not been fully released, it may break during normal use.

[0201] Installation issues: Improper screen installation, such as unstable fixation or uneven pressure, can also cause breakage.

[0202] 4. Vibration and shock

[0203] Road conditions: Driving on uneven roads subjects the vehicle to constant vibration and shock, which may cause damage to the screen over time.

[0204] Sound system: Vibrations from high-powered sound systems may also affect the screen.

[0205] 5. Chemical substances

[0206] Cleaners: Using improper cleaners or chemicals to clean the screen may corrode the screen material and cause it to break.

[0207] Corrosive substances: Contact with corrosive substances inside or outside the vehicle may also cause damage to the screen.

[0208] 6. Design issues

[0209] Structural strength: If the screen’s design is not structurally strong enough to withstand the expected stress, it may break.

[0210] Protective measures: Lack of adequate protective measures, such as shock-proof pads, anti-collision frames, etc., may also increase the risk of screen breakage.

[0211] 7. Improper maintenance

[0212] Improper cleaning: Using sharp objects or rough cloth to clean the screen may scratch the screen surface and increase the risk of breakage.

[0213] Lack of maintenance: Long-term lack of proper maintenance and inspection may result in potential screen problems not being discovered and repaired in time.

[0214] 8. Other factors

[0215] Battery swelling: A swelling vehicle battery can put pressure on the screen, causing it to shatter.

[0216] Accidents: such as objects falling from the car and hitting the screen, or children accidentally hitting the screen while playing in the car.

[0217] The present invention integrates high-definition cameras, sound sensors, touch sensors, etc., and uses deep learning algorithms to fuse and analyze the collected multi-source data to achieve timely and accurate detection of vehicle screen breakage. In particular, the present invention takes into account the impact of driving behavior on screen breakage, and by analyzing driving behavior data, improves the accuracy and real-time nature of detection, providing effective protection for vehicle safety.

[0218] In the technical solution of the present invention, through the collection and model analysis of vehicle and environmental data, it can effectively help insurance companies simplify the claims process, improve claims efficiency, reduce human errors and processing time; help insurance companies understand the specific damage to the vehicle, so as to more reasonably control the repair cost and compensation amount; help improve service quality, reduce operating costs, provide car owners with a more convenient, efficient and personalized service experience, and create greater value for insurance companies and car owners.

[0219] The above description is only a preferred embodiment of the present invention, and does not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-source data fusion vehicle screen detection system, characterized in that: include: Multi-source data acquisition module, used to collect source data of various data types; A preprocessing module, connected to the multi-source data acquisition module, for preprocessing the source data of each type; A feature construction module, connected to the preprocessing module, for extracting features from the preprocessed source data of each data type and constructing feature data of each data type; A feature fusion module, connected to the feature construction module, for fusing the feature data of each data type to obtain a fused feature; A first decision module, connected to the feature construction module, is used to call the first damage detection model corresponding to each data type to process the feature data of the same data type to obtain respective first damage confidence scores for the screen; a first fusion module, connected to the first decision module, and configured to fuse the first damage confidence scores obtained by the damage detection models to obtain a second damage confidence score; A second decision module, connected to the feature fusion module, is used to call a pre-trained second damage detection model to process the fusion feature to obtain a third damage confidence score for the screen; The second fusion module is connected to the first fusion module and the second decision module respectively, and is used to fuse the second damage confidence score and the third damage confidence score to obtain a final damage confidence of the screen.

2. The multi-source data fusion vehicle screen detection system according to claim 1, characterized in that: The vehicle screen is a touch screen, and the multi-source data acquisition module includes: An image acquisition unit, used for acquiring image data of the vehicle-mounted screen; A touch acquisition unit, used to acquire touch data of the vehicle-mounted screen; A sound collection unit, used for collecting sound data inside the vehicle; A vibration collection unit, used to collect vibration impact data of the vehicle; A driving behavior collection unit, used to collect driving behavior data of the vehicle; The feature construction module is used to extract image feature data from the image data, extract touch feature data from the touch data, extract sound feature data from the sound data, extract vibration feature data from the vibration impact data, and extract driving behavior feature data from the driving behavior data; The feature fusion module fuses the image feature data, the touch feature data, the vibration feature data, the sound feature data and the driving behavior feature data to obtain the fused feature.

3. The multi-source data fusion vehicle screen detection system as claimed in claim 2, characterized in that: The feature fusion module forms the fusion feature by vector concatenation, weighted averaging, or principal component analysis of the image feature data, the touch feature data, the vibration feature data, the sound feature data, and the driving behavior feature data.

4. The multi-source data fusion vehicle screen detection system as claimed in claim 2, characterized in that: The driving behavior characteristic data includes vehicle speed characteristics, acceleration characteristics, steering characteristics, braking characteristics and driving mode characteristics.

5. The multi-source data fusion vehicle screen detection system according to claim 1, characterized in that: The first fusion module fuses the first damage confidence scores obtained by each of the damage detection models based on weighted average or Bayesian method to obtain the second damage confidence score.

6. A vehicle screen detection method based on multi-source data fusion, characterized in that: A vehicle screen detection system using a multi-source data fusion as described in any one of claims 1 to 5, comprising: Step A1, collecting source data of various data types; Step A2, preprocessing the source data of each data type; Step A3, extracting features from the preprocessed source data of each data type, and constructing feature data of each data type; Step A4, fusing the feature data of each data type to obtain a fused feature; Step A5, calling the first damage detection model corresponding to each data type to process the feature data of the same data type to obtain respective first damage confidence scores for the screen; Step A6, fusing the first damage confidence scores obtained by the damage detection models to obtain a second damage confidence score; Step A7, calling a pre-trained second damage detection model to process the fused features to obtain a third damage confidence score for the screen; Step A8: Merge the second damage confidence score and the third damage confidence score to obtain a final damage confidence of the screen.

7. The vehicle screen detection method of multi-source data fusion as claimed in claim 6, characterized in that: Source data of various data types include image data, touch data, vibration and impact data, sound data, and driving behavior data; In the step A3, image feature data is extracted from the image data, touch feature data is extracted from the touch data, sound feature data is extracted from the sound data, vibration feature data is extracted from the vibration impact data, and driving behavior feature data is extracted from the driving behavior data; In the step A4, the image feature data, the touch feature data, the vibration feature data, the sound feature data and the driving behavior feature data are fused to obtain the fused feature.

8. The vehicle screen detection method of multi-source data fusion as claimed in claim 7, characterized in that: In step A4, the image feature data, the touch feature data, the vibration feature data, the sound feature data and the driving behavior feature data are formed into the fusion feature by vector concatenation, weighted averaging or principal component analysis.

9. The vehicle screen detection method of multi-source data fusion as claimed in claim 7, characterized in that: The driving behavior characteristic data includes vehicle speed characteristics, acceleration characteristics, steering characteristics, braking characteristics and driving mode characteristics.

10. The vehicle screen detection method of multi-source data fusion as claimed in claim 6, characterized in that: In step A6, the first damage confidence scores obtained by the damage detection models are fused based on weighted average or Bayesian method to obtain the second damage confidence score.

Images