Method for realizing glass smashing event detection by using 4G recorder

Abstract

The invention belongs to the technical field of vehicle safety control, and particularly relates to a method for realizing glass smashing event detection by using a 4G recorder, after parking, a millimeter wave radar detects objects around a vehicle through periodic scanning, and determines whether to wake up other systems to enter a working state; the automobile data recorder detects whether a person in the video frame has a glass smashing behavior or not through a target detection algorithm; meanwhile, sound analysis is used for assisting judgment; the change of a three-dimensional space is measured through an acceleration sensor to capture physical impact or vibration; through data fusion, a comprehensive score or confidence is calculated to determine whether to send an alarm signal. According to the technical scheme, after the glass smashing event is accurately detected, real-time local alarm is achieved through the built-in alarm system, meanwhile, a 4G network is used for notifying a message of the glass smashing event to a vehicle owner app, a video is automatically uploaded to a cloud end for storage, loss of the evidence obtaining video is prevented, and the property loss of a vehicle owner is reduced to the maximum extent.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle safety control, and in particular relates to a method for realizing glass-smashing event detection by utilizing a 4G recorder. Background Art

[0002] At present, there are common incidents of vehicles being smashed and stolen after parking in foreign countries, which has caused great trouble to the lives of local people. The current common practice is to use the parking monitoring function of the vehicle-mounted driving recorder to detect in real time whether there is a glass smashing incident. The parking monitoring function of the traditional driving recorder is mainly to achieve a time-lapse recording state when parking, record the parking image video for a long time, and use the 3-axis acceleration sensor to sense whether the vehicle is vibrating. Once a vibration occurs, the current video will be locked, and an alarm will be triggered when the owner gets in the car next time: the owner will be reminded by voice that an abnormal event occurred when parking and the owner will be asked to check the parking video.

[0003] The main problems of existing technologies are:

[0004] 1. The sensing source is single. It only uses the acceleration sensor to sense whether the vehicle is vibrating. It cannot accurately identify whether a glass-breaking incident has occurred. There will be interference between the vibration signal and the environmental noise, low vibration detection accuracy, and inability to determine the source of the vibration. For example, the normal opening and closing of the door / trunk after parking will cause the vehicle to vibrate and trigger an alarm. When a large truck passes by the vehicle, or when there is thunder or rain, the vehicle will vibrate and trigger an alarm.

[0005] 2. There is no multi-sensor data fusion and an effective fusion algorithm. It is impossible to solve the problems of false positives (due to the limitations of single perception, for example, a simple vibration detection algorithm cannot accurately distinguish between normal vehicle door opening and closing vibrations and glass smashing events, which may lead to false alarms) and false negatives (due to the limitations of single perception, if the vibration amplitude of a glass smashing event is very small, such as using special tools to cut the glass, the recorder cannot capture the event in time, which may lead to missed alarms). There are problems of high false alarm rate and high missed detection rate.

[0006] 3. High power consumption. The existing technology requires continuous power supply to ensure 24-hour parking video recording. When parking, turning on the camera and acceleration sensor all the time may cause power supply problems for the car battery. Especially when parking for a long time, the car battery may be over-discharged, affecting battery life and vehicle starting.

[0007] 4. Lack of real-time feedback and remote monitoring capabilities. Current dashcams do not have 4G networks and cannot achieve real-time communication and remote monitoring with external devices or cloud systems. Therefore, it is impossible to transmit the detected glass-breaking incident to the car owner in a timely manner. There is no remote alarm, and no function of uploading data to the cloud for storage (when the dashcam is taken away by the perpetrator, products with data upload function can upload the video before and after the incident to the cloud for storage in real time).

[0008] In summary, the parking monitoring technology currently used in vehicle-mounted dashcams has a single sensing source, low detection accuracy, and a large number of false detections and missed detections. It is impossible to adopt a localized real-time alarm strategy (such as flashing lights, loudspeaker sound, and other alarm methods) for each detection result to drive away the perpetrator. In addition, due to the lack of 4G network, it is impossible to remotely alarm the car owner and upload the event video in real time as evidence. The current technology cannot solve the problem of frequent glass smashing and car theft in foreign countries. Summary of the invention

[0009] The purpose of the present invention is to provide a method for detecting glass-breaking events using a 4G recorder, so as to solve the problem that the existing technology cannot accurately identify glass-breaking events.

[0010] Another purpose of the present invention is to solve the problem that existing products and technologies have very high power consumption when performing parking monitoring (the camera and various sensors need to be turned on and in working state all the time), which may cause the vehicle battery to be fed.

[0011] Another purpose of the present invention is to further solve the problem that when a glass-breaking incident occurs, the car owner cannot be notified immediately, and the video before and after the glass-breaking incident cannot be uploaded to the cloud storage in time after the glass-breaking incident occurs, and the perpetrator takes away the driving recorder, resulting in the inability to obtain evidence.

[0012] To achieve the above objectives, this application is implemented through the following technical solutions:

[0013] A method for detecting a glass-breaking event using a 4G recorder comprises the following steps:

[0014] S1. After the vehicle is turned off, the millimeter-wave radar detects objects around the vehicle through periodic scanning. If an object is detected approaching the set range, other functional modules and the core controller are immediately awakened to enter the working state of glass-breaking detection;

[0015] S2. The dashcam generates a continuous video stream and uses OpenCV tools to capture and process video frames. The target detection algorithm is used to detect the position and movement of people in the video frames to determine whether there is glass breaking behavior.

[0016] S3, recording the sound of the surrounding environment through a sound collection module, and assisting in determining the frame in step S2 through sound analysis;

[0017] S4, capturing the physical impact or vibration of the vehicle by measuring the change of acceleration in three-dimensional space through an acceleration sensor;

[0018] S5, fusing the data from step S2 to step S4, calculating the posterior probability to obtain a comprehensive score or confidence;

[0019] S6. If the score or confidence is greater than a set threshold T, an alarm signal is sent.

[0020] Furthermore, the functional module includes a camera, a 3-axis acceleration sensor, a microphone and a display screen.

[0021] Furthermore, in step S2, a background subtraction algorithm is used to separate the external environment and the background, identify possible intruders, and separate the foreground and the background by calculating the difference between the current frame and the background model. The specific steps are:

[0022] S21, initializing the background model;

[0023] S22, for each frame, calculating the difference between the current frame and the background model, and the part where the difference value is greater than the set value is the foreground;

[0024] S23. Update the background model to ensure that the background model can adapt to environmental changes over time.

[0025] Furthermore, in step S22, the algorithm for calculating the difference between the current frame and the background model is a Gaussian mixture model, and the formula is:

[0026] For each pixel x t The value at time t uses a Gaussian mixture model to describe its background. Each pixel x t The value of is expressed as a weighted sum of multiple Gaussian distributions:

[0027] ,in, is the weight of the kth Gaussian component, So and is a Gaussian distribution with mean and variance, is the number of mixed Gaussians, usually 3 to 5;

[0028] When the new pixel value x t When it arrives, the GMM parameters are updated according to the current pixel value:

[0029] ,

[0030] ,

[0031] , where α is the learning rate, usually between 0.01 and 0.1; is an indicator function, indicating that x t Whether it belongs to the kth Gaussian component.

[0032] Furthermore, in step S2, the position and action of the person in the video frame are detected by the target detection algorithm to determine whether there is glass breaking behavior. The 3D convolutional network is used for calculation. The formula of the 3D convolutional network is:

[0033] Represents time t and spatial position The pixel value of

[0034] ,in, is the output feature after the convolution operation, It is a convolution kernel, usually a three-dimensional filter, where T, H, and W are the time dimension, image height, and width, respectively.

[0035] Furthermore, in step S3, the sound features are extracted by the MFCC method. The MFCC extraction steps and formula are as follows:

[0036] S301, pre-emphasis, the formula is: ,in, is the original audio signal, is the pre-emphasized signal, α is the emphasis coefficient, usually between 0.95 and 0.99;

[0037] S302, framing, dividing the pre-emphasized audio signal into a series of small frames, each frame being between 20ms and 40ms;

[0038] S303, windowing, applying a Hamming window on each frame to reduce the channel leakage effect, the formula is:

[0039] , where L is the frame length;

[0040] S304, Fourier transform, convert the signal from the time domain to the frequency domain, generate a spectrum, the formula is:

[0041] ,in is the Kth frequency component, L is the length of Fourier transform;

[0042] S305, Mel filter, maps the spectrum of step S304 to Mel frequency, the formula is:

[0043] ,in is the linear frequency in Hz, is the corresponding Mel frequency;

[0044] S306, logarithmic power spectrum, calculate the logarithmic energy of each Mel filter output, the formula is:

[0045] ,in is the frequency response of the mth Mel filter, N is the number of filters;

[0046] S307, discrete cosine transform, perform discrete cosine transform on the Mel frequency logarithmic power spectrum to obtain MFCC coefficients, the formula is:

[0047] , where M is the number of filters and n is the index of the cepstral coefficient.

[0048] Furthermore, in step S3, the steps of constructing the CNN sound classifier are:

[0049] S311, data preprocessing, converting MFCC features into a two-dimensional matrix with a matrix size of (T, N), where T is the time step and N is the number of MFCC coefficients per frame, and using the processed MFCC matrix as the input of CNN;

[0050] S312, CNN architecture design:

[0051] 1) The shape of the input layer is (T, N, 1), where the two-dimensional one is the MFCC matrix and the third dimension is the number of channels;

[0052] 2) The convolution layer extracts local features in the input image through the convolution kernel. The convolution formula is:

[0053] , where X is the input feature map, W is the convolution kernel, and b is the bias term;

[0054] 3) Activation layer, which performs nonlinear transformation on the output of the convolutional layer. The activation function is: ;

[0055] 4) Pooling layer: To reduce the size of the convolutional layer output, a maximum pooling layer is used to retain the main features;

[0056] S313, fully connected layer, the outputs of the convolutional layer and the pooling layer are expanded and input into the fully connected layer;

[0057] S314, output layer, the last layer is the softmax layer, which is used to output the probability of each category. The softmax function is: , where z c is the score of category c;

[0058] S315. Use the cross entropy loss function for training. The formula of the cross entropy loss function is:

[0059] , where C is the number of categories and y c is the true label, is the predicted probability;

[0060] S316, model evaluation and tuning, after training the model on the training set, use the validation set and test set to evaluate the model performance, and optimize the accuracy and generalization ability of the model by adjusting the CNN architecture and training parameters.

[0061] Furthermore, in step S5, Bayesian inference is used to fuse multi-sensor data. Assuming that there are multiple sensors S1, S2, ..., S n , each sensor has different measurement values ​​z1, z2, ..., z n , the formula for Bayesian inference is:

[0062] ,in:

[0063] is the posterior probability given the sensor observation data;

[0064] is the likelihood function, i.e., the probability of the target state given the sensor observation;

[0065] is the prior probability, which represents the prior knowledge of the target state;

[0066] is the evidence, representing the probability of all observed data.

[0067] Furthermore, the 4G recorder includes two high-definition cameras, and has two built-in millimeter-wave radars, a microphone, a 3-axis acceleration sensor, a 4G communication module, a GPS positioning module display screen, and also includes a speaker and an alarm light, and the speaker and the alarm light are both connected to the core controller signal.

[0068] Furthermore, the alarm signal includes a local alarm mechanism and uses the 4G network to notify the car owner's app of the glass-breaking incident, and uploads videos before and after the glass-breaking incident to the cloud for storage.

[0069] The beneficial effects of the present invention are:

[0070] This technical solution can accurately detect glass-breaking incidents in ultra-low power parking monitoring mode when parking. After accurately detecting the glass-breaking incident, the built-in alarm system can be used to realize real-time local alarm, which can deter the perpetrator on the spot at the first time, scare off the perpetrator, and reduce the property loss of the car owner. At the same time, the 4G network communication capability is used to notify the car owner's app of the glass-breaking incident, and automatically upload the video before and after the glass-breaking incident to the cloud storage, so that the car owner can view the entire process of the glass-breaking incident and prevent the perpetrator from unplugging the recorder and taking it away, thereby losing the evidence video. After the car owner receives the glass-breaking incident on the app, he can also conduct real-time voice intercom between the mobile app and the recorder in the car in real time, which can further deter the perpetrator and minimize the property loss of the car owner. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 The present invention is a logic diagram of a method for realizing glass-breaking event detection and alarm using a 4G recorder. DETAILED DESCRIPTION

[0072] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. The following embodiments are merely exemplary and can only be used to explain and illustrate the technical solution of the present invention, but cannot be construed as limiting the technical solution of the present invention.

[0073] The present application provides a method for detecting and alarming glass-breaking events using a 4G recorder. The 4G recorder involved is a 4G driving recorder with two horizontally rotatable cameras, which can be obtained by a person skilled in the art through commercial channels.

[0074] When the vehicle is moving, one camera faces outside the car, and the other faces inside the car to record the driving. When the vehicle is turned off and parked, the camera facing outside the car is turned horizontally to the inside of the car. At this time, both cameras are facing inside the car, and the perspectives of the two cameras are combined to cover the entire interior of the car and the area outside the windows on both sides.

[0075] The 4G driving recorder involved in this application has two built-in millimeter-wave radars. The detection angles of the two millimeter-wave radars are facing the left and right glass, which can cover the glass on both sides of the entire front and rear rows. According to data statistics, the glass is mainly broken on the left and right sides. Breaking the front and rear glass does not make it convenient to enter the car and take away the items in the car. The 4G driving recorder also has a built-in microphone, a 3-axis acceleration sensor, a 4G communication module and a GPS module. The 4G driving recorder has a display screen, a large speaker and flashing red and blue warning lights.

[0076] S1. After the vehicle is turned off, except for the millimeter-wave radar, other functional modules are turned off, and the core controller enters a low-power sleep mode.

[0077] When the vehicle is in the shutdown parking monitoring mode, the millimeter wave radar will detect surrounding objects (such as moving objects, approaching people, etc.) through periodic scanning. The millimeter wave radar only needs to transmit signals and receive reflected waves at specific time intervals, and will not continuously consume a large amount of power. This detection method is very energy-saving. The millimeter wave radar can accurately determine the distance, speed and angle of the target object by analyzing the time delay and Doppler frequency shift of the reflected wave, which makes it very effective in detecting moving targets; the millimeter wave radar can work under various lighting conditions, especially in low light or completely dark environments, the radar performance is not affected, and it can work all-weather; at the same time, the high penetration ability of the millimeter wave radar enables it to detect dynamic changes in the environment outside the vehicle, even glass windows can be penetrated, and it is not affected by rain, fog, snow, etc. in the environment. This enables the millimeter wave radar to maintain efficient and stable detection capabilities.

[0078] The millimeter-wave radar detects objects around the vehicle through periodic scanning. If an object is detected approaching the set range, other functional modules and the core controller are immediately awakened to enter the working state of glass breaking detection. The period of periodic scanning here is set according to needs. Specifically, the scanning period can be adjusted according to the specific situation of the millimeter-wave radar scanning around the vehicle. When there are frequent people passing around the vehicle, the scanning period is increased. If there are fewer people around the vehicle and the environment is quiet, it is a high-incidence period of glass breaking. The scanning period can be appropriately shortened. For specific situations, technical personnel in this field can make corresponding adjustments according to actual needs.

[0079] Millimeter-wave radar can accurately detect vehicles, pedestrians or objects, and distinguish these targets from the surrounding background noise (such as wind blowing trees, raindrops, etc.). Through the radar data, you can accurately know whether the object is approaching. When the millimeter-wave radar detects that an object is approaching the set range, it will immediately wake up the entire system and other sensors: such as cameras, 3-axis accelerometers, microphones and displays. The system begins to record the images of the two cameras in real time. At this time, the display screen will emit white light, acting as a flashlight to illuminate the entire environment, allowing the camera to work better in dark environments. At this time, the entire system is awakened by the millimeter-wave radar (the entire system wakes up within 1s) and begins to enter the formal working state of glass breaking detection.

[0080] In this technical solution, when the millimeter radar wakes up the entire system, including the light detection module, it detects the current light intensity. If it is detected that the light intensity is sufficient, such as during the day or at night, but the light around the vehicle is relatively sufficient, there is no need to wake up the display screen, that is, the display screen does not need to emit white light. Only when insufficient light intensity is detected, the display screen is woken up, and at this time the display screen emits white light.

[0081] In this application, when entering the working state of glass breaking detection, the entire glass breaking event detection process is: real-time video analysis (computer vision) --- sound analysis --- 3-axis acceleration sensor fusion perception --- edge computing and low-latency processing --- multimodal data fusion --- decision logic --- local alarm and event notification upload. The following is a separate explanation of the working content and principle of each processing module.

[0082] S2. The dashcam generates a continuous video stream and uses OpenCV tools to capture and process video frames. The target detection algorithm is used to detect the position and movement of people in the video frames to determine whether there is any glass-breaking behavior.

[0083] Video stream processing:

[0084] Background modeling: Since 4G dashcams are usually installed in fixed locations, the background is relatively fixed when the owner leaves the car. Background subtraction algorithms (such as background modeling and foreground segmentation) can be used to separate the external environment and background to identify possible intruders.

[0085] Background Subtraction: Background subtraction is one of the most commonly used background modeling methods and is applicable to most video surveillance scenarios. This method separates the foreground and background by calculating the difference between the current frame and the background model. The specific steps are as follows:

[0086] Initialize the background model (can be initialized by averaging multiple frames).

[0087] For each frame, the difference between the current frame and the background model is calculated, and the part with a larger difference value is the foreground.

[0088] When updating the background model, the most recent frames are usually considered to ensure that the background model can adapt to environmental changes over time. The commonly used algorithm is the Gaussian Mixture Model (GMM): GMM is a very popular background modeling method suitable for dynamic environments. The Gaussian Mixture Model (GMM) models the background of each pixel through multiple Gaussian distributions. Assuming that each pixel value can be represented by a mixture model composed of multiple Gaussian distributions, the formula is:

[0089] For each pixel x t The value at time t uses a Gaussian mixture model to describe its background. Each pixel x t The value of is expressed as a weighted sum of multiple Gaussian distributions:

[0090] ,in, is the weight of the kth Gaussian component, So and is a Gaussian distribution with mean and variance, is the number of mixed Gaussians, usually 3 to 5;

[0091] When the new pixel value x t When it arrives, the GMM parameters are updated according to the current pixel value:

[0092] ,

[0093] ,

[0094] , where α is the learning rate, usually between 0.01 and 0.1; is an indicator function, indicating that x t Whether it belongs to the kth Gaussian component.

[0095] Foreground detection: Determine whether it is a foreground based on the difference between the current pixel and the background model. Specifically, if the pixel x t The Gaussian distribution in the background model is quite different, for example, if it exceeds 3σ, the pixel point is regarded as the foreground. This is only an example. Those skilled in the art may determine the pixel x according to the actual situation or in different external environments. t The difference from the Gaussian distribution in the background model may also be different, so the value of the difference is determined according to the specific situation.

[0096] In this application, the key to background modeling is to be able to stably identify the separation of background and foreground. In vehicle-mounted video surveillance, background modeling is often used for: monitoring changes in the external environment (for example, identifying people or objects approaching the vehicle); detecting background changes and filtering out irrelevant dynamic objects (such as swaying trees, rain and snow, and other weather changes).

[0097] Motion Detection:

[0098] The position and movement of the person in the video frame are detected by the target detection algorithm. Glass breaking is usually accompanied by rapid hand movements, and the presence of glass breaking behavior can be determined by analyzing the hand trajectory. In other embodiments of the present application, a comprehensive judgment can also be made based on the hand movement combined with whether an object or instrument is held. Usually, the action of breaking glass needs to be combined with the corresponding instrument to achieve the actual situation of breaking glass.

[0099] In this application, a 3D convolutional network (3D CNN) is used: this method directly performs convolution operations on consecutive frames in the video to extract spatial-temporal features, which is suitable for action recognition tasks. Each frame in the video is used as input, and the network captures the timing information by convolution of consecutive frames.

[0100] Formula of 3D CNN: 3D convolutional network performs convolution operations on the three-dimensional voxels (space and time) of the image. The core operation is to apply the convolution kernel to time-continuous image blocks.

[0101] Represents time t and spatial position The pixel value of

[0102] ,in, is the output feature after the convolution operation, It is a convolution kernel, usually a three-dimensional filter, where T, H, and W are the time dimension, image height, and width, respectively.

[0103] This process allows the network to capture dynamic changes in time and spatial structure, so that complex actions in the video can be recognized. The action detection of this application can recognize actions in video streams, especially for fast actions with obvious trajectories (such as breaking glass). The combination of these two can greatly improve the accuracy of event recognition and can capture and judge whether there is glass breaking in real time.

[0104] S3. Record the sound of the surrounding environment through a sound collection module, and use sound analysis to assist in determining the frame in step S2.

[0105] Sound Analysis:

[0106] In this application, in addition to the aforementioned video analysis, a microphone is also used to record the sounds of the surrounding environment. When the glass of a vehicle is smashed, a specific cracking sound will be emitted, and sound analysis is used to assist in judging the frame.

[0107] This application uses the MFCC (Mel Frequency Cepstral Coefficients) method to extract sound features, and then uses a neural network machine learning algorithm for classification and recognition. MFCC is an audio feature extraction method that is very suitable for extracting and analyzing the frequency features of human speech and complex sounds (such as glass breaking). Its main principle is to map the audio signal to the Mel frequency that the human ear is sensitive to, and extract the main spectral features. The MFCC extraction steps and formula are as follows:

[0108] S301, pre-emphasis, before frequency analysis, first pre-emphasize the input audio signal to enhance the high-frequency signal to compensate for the attenuation of the low-frequency signal. The formula is:

[0109] ,in, is the original audio signal, is the pre-emphasized signal, α is the emphasis coefficient, which is usually between 0.95 and 0.99.

[0110] S302, Framing: dividing the pre-emphasized audio signal into a series of small frames, each frame being between 20ms and 40ms, so as to be able to process instantaneous changes in the frequency domain.

[0111] S303, windowing, applying a Hamming window on each frame to reduce the channel leakage effect, the formula is:

[0112] , where L is the frame length.

[0113] S304, Fourier transform (FFT), convert the signal from the time domain to the frequency domain and generate a spectrum. The formula is:

[0114] ,in is the Kth frequency component, and L is the length of the Fourier transform.

[0115] S305, Mel Filter Bank, maps the spectrum of step S44 to Mel frequency, simulating the difference in human ear's perception of different frequencies. The formula is:

[0116] ,in is the linear frequency in Hz, is the corresponding Mel frequency; in this application, a set of triangular filters are used to weight the spectrum to obtain the Mel frequency.

[0117] S306, Logarithmic Power Spectrum, calculates the logarithmic energy of each Mel filter output to improve the robustness of the feature. The formula is:

[0118] ,in is the frequency response of the mth Mel filter, and N is the number of filters.

[0119] S307, discrete cosine transform (DCT), perform discrete cosine transform on the Mel frequency logarithmic power spectrum to obtain MFCC coefficients. Usually, the first 13 coefficients are taken as the main features. The formula is:

[0120] , where M is the number of filters and n is the index of the cepstral coefficient.

[0121] Through the above steps, the MFCC features of each frame of audio can be obtained, which are then converted into a time-frequency graph (similar to an image) and classified using a convolutional neural network (CNN). CNN performs well in processing image data and is also applicable to time-frequency graphs.

[0122] In this application, the construction steps of the CNN sound classifier are:

[0123] S311, data preprocessing, converting the MFCC features into a two-dimensional matrix with a matrix size of (T, N), where T is the time step and N is the number of MFCC coefficients per frame (e.g. 13), and using the processed MFCC matrix as the input of the CNN.

[0124] S312, CNN architecture design:

[0125] 1) The shape of the input layer is (T, N, 1), where the two-dimensional one is the MFCC matrix and the third dimension is the number of channels.

[0126] 2) The convolution layer extracts local features from the input image through the convolution kernel, usually using a 3×3 or 5×5 convolution kernel. The convolution formula is:

[0127] , where X is the input feature map, W is the convolution kernel, and b is the bias term.

[0128] 3) Activation layer (ReLU), which performs nonlinear transformation on the output of the convolutional layer. The ReLU activation function is: .

[0129] 4) Pooling: To reduce the size of the convolutional layer output, a maximum pooling layer is used to retain the main features. Through the combination of multiple convolutional and pooling layers, high-order features in time and frequency can be gradually extracted.

[0130] S313, Fully Connected Layer: The outputs of the convolutional layer and the pooling layer are flattened and then input into the fully connected layer. The fully connected layer further combines the high-order features for classification.

[0131] S314, output layer, the last layer is the softmax layer, which is used to output the probability of each category. The softmax function is: , where z c is the score of category c. This probability facet is used for the final classification, and the category with the highest probability is selected as the output.

[0132] S315. Use the cross entropy loss function for training. The formula of the cross entropy loss function is:

[0133] , where C is the number of categories and y c is the true label, is the predicted probability.

[0134] In this application, back propagation and optimization are also included. The gradient is calculated by the back propagation algorithm, and the network parameters are updated using optimizers such as gradient descent or Adam to minimize the loss function.

[0135] S316, model evaluation and tuning. After training the model on the training set, use the validation set and test set to evaluate the model performance. By adjusting the CNN architecture (such as the number of convolutional layers, convolution kernel size, learning rate, etc.) and training parameters, the accuracy and generalization ability of the model can be optimized.

[0136] This application also includes model training:

[0137] By optimizing CNN through cross entropy loss function and back propagation, we can finally achieve accurate classification and recognition of cracking sound. By recording with a microphone and combining MFCC and CNN, we can effectively detect and classify the cracking sound events of glass breaking.

[0138] S4. The acceleration sensor measures the change in acceleration in three-dimensional space to capture physical impact or vibration. When an impact event such as breaking glass occurs, the sensor detects the instantaneous change in acceleration. If this matches the action detection result in the video stream, the system can improve the recognition accuracy of the event through fusion perception.

[0139] 1. Reduce False Positives:

[0140] False positives usually refer to an algorithm mistakenly misinterpreting a normal scene as an abnormal event. In video analysis, this can happen when there is slight movement in the environment or when the vehicle shakes. With the addition of an accelerometer, the sensor's vibration threshold can be used to reduce such false positives.

[0141] Vibration Threshold: Set a threshold. When the vibration detected by the accelerometer exceeds this value, it is considered that an external impact event (such as breaking glass) may have occurred. For example, a slight wind or raindrop is unlikely to cause vibration greater than a certain threshold.

[0142] Speed ​​and motion correlation: By analyzing the characteristics of the acceleration signal (such as amplitude, duration, etc.), it is possible to effectively distinguish irrelevant vibrations (such as opening and closing car doors) from vibrations related to glass smashing.

[0143] In this way, if only the video analysis detects an unusual movement (such as someone approaching and waving their arms quickly), but the acceleration sensor does not detect obvious vibration, the system can assume that the event is unlikely to be glass breaking, thereby reducing false alarms.

[0144] 2. Reduce False Negatives:

[0145] A false negative occurs when the system fails to correctly identify a real event. In the case of a broken glass scene, a false negative can occur when the video analysis fails to accurately detect the rapid motion, or when there is too much background noise for the algorithm to distinguish the glass breaking. Accelerometers can compensate for this by detecting vibrations:

[0146] Vibration sensing: When a high-energy impact such as glass breaking occurs, the sensor detects rapid vibration changes. Even if the video analysis does not fully capture the details of the person's movements, the vibration signal can be used as an additional trigger signal to improve the reliability of detection.

[0147] Vibration pattern matching: By training the model to identify normal vibration patterns (such as opening and closing car doors or trunks) and abnormal vibration patterns (such as severe vibrations that cause glass to break), the signal from the acceleration sensor can be used as a confirmation signal for the occurrence of an event, reducing missed reports.

[0148] This application also includes edge computing and low-latency processing:

[0149] Edge computing: 4G dashcams usually do not have strong computing power, so the algorithm needs to be lightweight so that it can run efficiently on embedded platforms. The TensorFlow Lite framework is used to optimize the model to adapt it to low-power hardware platforms.

[0150] Low-latency processing: To ensure that glass-breaking incidents can be detected in real time, the model and algorithm need to be optimized to ensure that the latency of data processing is as low as possible. Efficiency can be improved through model compression, quantization, and edge inference.

[0151] Embedded system: 4G driving recorder uses embedded Linux as the underlying operating system and is developed using languages ​​such as C and C++.

[0152] S5. Fuse the data from step S2 to step S4, and calculate the posterior probability to obtain a comprehensive score or confidence. Integrating data from different sensors (such as cameras, microphones, and 3-axis accelerometers) into a unified framework can make more accurate event judgments. Use Bayesian Inference to fuse multi-sensor data. Integrate the signals of different sensors by calculating the posterior probability. Assuming that the observations of each sensor are independent, the state of the target can be updated according to the Bayesian theorem.

[0153] Using Bayesian inference to fuse multi-sensor data, assuming there are multiple sensors S1, S2, …, S n, each sensor has different measurement values ​​z1, z2, ..., z n , the formula for Bayesian inference is:

[0154] ,in:

[0155] is the posterior probability given the sensor observation data;

[0156] is the likelihood function, i.e., the probability of the target state given the sensor observation;

[0157] is the prior probability, which represents the prior knowledge of the target state;

[0158] is the evidence, representing the probability of all observed data.

[0159] For each sensor S i , the sensor observation value z can be calculated i likelihood (i.e., the probability of observing this sensor data in the current state).

[0160] For camera data, the likelihood function can predict whether the target appears in a certain area of ​​the video based on the target's position and speed. For microphones, the sound feature recognition can be used to determine whether it is a glass-breaking event. For accelerometers, the likelihood function can determine whether it meets the requirements of external force impact (such as glass-breaking) based on the amplitude and frequency of the vibration. Through Bayesian inference, the information of all sensors can be combined to calculate the posterior probability, thereby obtaining a more reliable target state estimate.

[0161] S6. If the posterior probability score or confidence Sfinal is greater than the set threshold T, an alarm signal is sent, Sfinal=f(Svideo, Smic, Sacc), where f is a weighted average fusion function.

[0162] After the glass-breaking incident is detected through the above detection process, since the detection result is reliable, the local alarm mechanism can be triggered immediately: use a loud speaker to simulate the alarm sound + red and blue lights to simulate the flashing of the police light + light up the display screen to emit white light to simulate the illumination of a flashlight. The perpetrator is deterred at the first time on the spot, scaring off the perpetrator and reducing the property loss of the car owner. At the same time, the 4G network is used to notify the car owner's app of the glass-breaking incident, and a 1-minute video before and after the glass-breaking incident is automatically uploaded to the cloud storage, so that the car owner can view the entire process of the glass-breaking incident and prevent the perpetrator from unplugging the recorder and taking it away, thereby losing the evidence video.

[0163] The present application also involves transmitting an alarm signal when a glass-breaking situation is detected, and at the same time detecting whether the current network signal situation meets the signal transmission requirements. If it is detected that the network signal at the current vehicle position is weak and cannot meet the transmission of the alarm signal, the data of the glass-breaking event (must include video data) is stored in the BCU while maintaining data transmission to increase the possibility of preserving evidence.

[0164] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will be appreciated by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for detecting glass-breaking events using a 4G recorder, characterized in that: The following steps are involved: S1. After the vehicle is turned off, the millimeter-wave radar detects objects around the vehicle through periodic scanning. If an object is detected approaching the set range, other functional modules and the core controller are immediately awakened to enter the working state of glass-breaking detection; S2. The dashcam generates a continuous video stream and uses OpenCV tools to capture and process video frames. The target detection algorithm is used to detect the position and movement of people in the video frames to determine whether there is glass breaking behavior. S3, recording the sound of the surrounding environment through a sound collection module, and assisting in determining the frame in step S2 through sound analysis; S4, capturing the physical impact or vibration of the vehicle by measuring the change of acceleration in three-dimensional space through an acceleration sensor; S5, fusing the data from step S2 to step S4, calculating the posterior probability to obtain a comprehensive score or confidence; S6. If the score or confidence is greater than a set threshold T, an alarm signal is sent.

2. The method for detecting glass-breaking events using a 4G recorder according to claim 1, characterized in that: The functional modules include a camera, a 3-axis acceleration sensor, a microphone and a display screen.

3. The method for detecting glass-breaking events using a 4G recorder according to claim 1, characterized in that: In step S2, a background subtraction algorithm is used to separate the external environment and the background, identify possible intruders, and separate the foreground and background by calculating the difference between the current frame and the background model. The specific steps are: S21, initializing the background model; S22, for each frame, calculating the difference between the current frame and the background model, and the part where the difference value is greater than the set value is the foreground; S23. Update the background model to ensure that the background model can adapt to environmental changes over time.

4. The method for detecting glass-breaking events using a 4G recorder according to claim 3, characterized in that: In step S22, the algorithm for calculating the difference between the current frame and the background model is a Gaussian mixture model, and the formula is: For each pixel x t The value at time t uses a Gaussian mixture model to describe its background. Each pixel x t The value of is expressed as a weighted sum of multiple Gaussian distributions: ,in, is the weight of the kth Gaussian component, So and is a Gaussian distribution with mean and variance, is the number of mixed Gaussians, usually 3 to 5; When the new pixel value x t When it arrives, the GMM parameters are updated according to the current pixel value: ， ， , where α is the learning rate, usually between 0.01 and 0.1; is an indicator function, indicating that x t Whether it belongs to the kth Gaussian component.

5. The method for detecting glass-breaking events using a 4G recorder according to claim 4, characterized in that: In step S2, the position and action of the person in the video frame are detected by the target detection algorithm to determine whether there is glass breaking behavior. The 3D convolutional network is used for calculation. The formula of the 3D convolutional network is: Represents time t and spatial position The pixel value of ,in, is the output feature after the convolution operation, It is a convolution kernel, usually a three-dimensional filter, where T, H, and W are the time dimension, image height, and width, respectively.

6. The method for detecting glass-breaking events using a 4G recorder according to claim 1, characterized in that: In step S3, the sound features are extracted by the MFCC method. The MFCC extraction steps and formula are as follows: S301, pre-emphasis, the formula is: ,in, is the original audio signal, is the pre-emphasized signal, α is the emphasis coefficient, usually between 0.95 and 0.99; S302, framing, dividing the pre-emphasized audio signal into a series of small frames, each frame being between 20ms and 40ms; S303, windowing, applying a Hamming window on each frame to reduce the channel leakage effect, the formula is: , where L is the frame length; S304, Fourier transform, convert the signal from the time domain to the frequency domain, generate a spectrum, the formula is: ,in is the Kth frequency component, L is the length of Fourier transform; S305, Mel filter, maps the spectrum of step S304 to Mel frequency, the formula is: ,in is the linear frequency in Hz, is the corresponding Mel frequency; S306, logarithmic power spectrum, calculate the logarithmic energy of each Mel filter output, the formula is: ,in is the frequency response of the mth Mel filter, N is the number of filters; S307, discrete cosine transform, perform discrete cosine transform on the Mel frequency logarithmic power spectrum to obtain MFCC coefficients, the formula is: , where M is the number of filters and n is the index of the cepstral coefficient.

7. The method for detecting glass-breaking events using a 4G recorder according to claim 6, characterized in that: In step S3, the steps for constructing the CNN sound classifier are: S311, data preprocessing, converting MFCC features into a two-dimensional matrix with a matrix size of (T, N), where T is the time step and N is the number of MFCC coefficients per frame, and using the processed MFCC matrix as the input of CNN; S312, CNN architecture design: 1) The shape of the input layer is (T, N, 1), where the two-dimensional one is the MFCC matrix and the third dimension is the number of channels; 2) The convolution layer extracts local features in the input image through the convolution kernel. The convolution formula is: , where X is the input feature map, W is the convolution kernel, and b is the bias term; 3) Activation layer, which performs nonlinear transformation on the output of the convolutional layer. The activation function is: ; 4) Pooling layer: To reduce the size of the convolutional layer output, a maximum pooling layer is used to retain the main features; S313, fully connected layer, the outputs of the convolutional layer and the pooling layer are expanded and input into the fully connected layer; S314, output layer, the last layer is the softmax layer, which is used to output the probability of each category. The softmax function is: , where z c is the score of category c; S315. Use the cross entropy loss function for training. The formula of the cross entropy loss function is: , where C is the number of categories and y c is the true label, is the predicted probability; S316, model evaluation and tuning, after training the model on the training set, use the validation set and test set to evaluate the model performance, and optimize the accuracy and generalization ability of the model by adjusting the CNN architecture and training parameters.

8. The method for detecting glass-breaking events using a 4G recorder according to claim 1, characterized in that: In step S5, Bayesian inference is used to fuse multi-sensor data. Assume that there are multiple sensors S1, S2, ..., S n , each sensor has different measurement values ​​z1, z2, ..., z n , the formula for Bayesian inference is: ,in: is the posterior probability given the sensor observation data; is the likelihood function, i.e., the probability of the target state given the sensor observation; is the prior probability, which represents the prior knowledge of the target state; is the evidence, representing the probability of all observed data.

9. The method for detecting glass-breaking events using a 4G recorder according to claim 1, characterized in that: The 4G recorder includes two high-definition cameras, and has two built-in millimeter-wave radars, a microphone, a 3-axis acceleration sensor, a 4G communication module, a GPS positioning module display screen, a speaker and an alarm light, and the speaker and the alarm light are both connected to the core controller signal.

10. The method for detecting glass-breaking events using a 4G recorder according to claim 1, characterized in that: The alarm signal includes a local alarm mechanism and uses the 4G network to notify the car owner's app of the glass-breaking incident, and uploads the video before and after the glass-breaking incident to the cloud for storage.

Images