Vehicle monitoring method and system and vehicle-mounted terminal

Abstract

The invention provides a vehicle monitoring method and system and a vehicle-mounted terminal. The vehicle monitoring method comprises the following steps: respectively acquiring a sensor signal and a positioning signal based on an inertial sensor array and a positioning unit, and extracting a wavelet packet energy entropy and a kurtosis index according to the sensor signal as initial features; performing fusion processing on the initial feature and the positioning signal to obtain a fusion feature, obtaining a vibration feature according to the fusion feature, and obtaining a voiceprint spectrum of a microphone array at the same time; and obtaining a current threat event in combination with the vibration characteristics and the voiceprint spectrum, so as to wake up a camera according to the current threat event, and inverting a camera field angle change model, thereby obtaining a threat event focus in combination with the current threat event. According to the method and the device, the energy consumption of the camera can be effectively reduced, the recognition accuracy of threat events in various environments can be improved, and a more reliable and efficient solution is provided for safety monitoring of the vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of camera wake-up, and in particular to a vehicle monitoring method and system and a vehicle terminal. BACKGROUND

[0002] In today's society, the safety and monitoring of vehicles are increasingly concerned. In order to timely discover and respond to various threats that vehicles may face, such as collisions, vandalism, theft, etc., many vehicles are equipped with visual monitoring systems, i.e. cameras. However, the existing visual systems have many serious defects, which greatly affect their actual application effect and performance.

[0003] At present, the cameras installed on vehicles have a very high miss rate of more than 60% under low illumination (light intensity <0.5 lux) and in rain and fog weather, etc. special environmental conditions, which leads to a significant decline in the ability to identify threat events; this makes the vehicle lack effective monitoring and protection in these special environments, and the vehicle owner cannot know the safety status of the vehicle in time, increasing the risk of vehicle damage and property loss. At the same time, the existing cameras are usually in a continuous working state, which leads to a very high energy consumption, with a power of more than 200W, and long-term high energy consumption increases the electrical burden of the vehicle and affects the vehicle's endurance; or only rely on vibration sensors to trigger the camera, which may frequently start the camera due to misjudgment, waste a lot of power and storage space, and further affect the reliability of vehicle safety monitoring. SUMMARY

[0004] The present application provides a vehicle monitoring method, system and vehicle terminal to solve the above technical problems.

[0005] Specifically, the present application provides a vehicle monitoring method, which pre-sets an inertial sensor array, a positioning unit and a microphone array in a set area of a target vehicle; the vehicle monitoring method comprises the following steps: acquiring sensor signals and positioning signals based on the inertial sensor array and the positioning unit respectively, and extracting wavelet packet energy entropy and kurtosis index as initial features according to the sensor signals; fusing the initial features and the positioning signals to obtain fusion features, and obtaining vibration features according to the fusion features, and simultaneously acquiring the voiceprint spectrum of the microphone array; and combining the vibration features and the voiceprint spectrum to obtain a current threat event, awakening a camera according to the current threat event, and inverting a camera field of view angle change model, and then obtaining a threat event focus in combination with the current threat event.

[0006] The above technical solution can not only effectively reduce the energy consumption of cameras and avoid unnecessary continuous operation, but also improve the accuracy of threatening event identification in various environments. It solves the problems of high false negative rate and high energy consumption of existing vision systems, and provides a more reliable and efficient solution for vehicle security monitoring.

[0007] Furthermore, obtaining initial features includes: synchronizing the sensor signals obtained by each sensor in the inertial sensor array in time, constructing a vibration wave direction-of-arrival difference (DOAD) localization model based on the time-synchronized sensor signals, and extracting wavelet packet energy entropy and kurtosis index based on the vibration wave DOAD localization model as initial features.

[0008] The above technical solution ensures the consistency and comparability of data collected by different sensors at the same time point, avoiding data deviations caused by time asynchrony. This makes the subsequently constructed vibration wave direction-of-arrival difference (DOA) localization model more accurate, thereby improving the extraction accuracy of initial features. By constructing the ODA localization model, the location of the vibration source can be determined more accurately by utilizing the time difference and spatial relationship of sensor signals. This helps to more accurately determine the location of the threat event and provides strong support for further processing. In addition, wavelet packet energy can describe the energy distribution of the vibration signal in different frequency bands, while kurtosis index can highlight the impact component in the vibration signal.

[0009] Furthermore, the acquisition of the current threat event includes: inputting the vibration features and acoustic spectrum into a set convolutional model to output the current threat event.

[0010] In the above technical solutions, the convolutional model has good adaptability and generalization ability, and can learn the feature patterns corresponding to different threat events through a large amount of training data; in practical applications, it can effectively classify and identify various complex threat situations and has high reliability.

[0011] Furthermore, obtaining the focus of the threat event includes: based on the initial feature inversion of the camera field of view change model, and combined with the target detection results and the current threat event, blurring the non-threat area to obtain the focus of the threat event; it also includes: generating a threat analysis report based on the focus of the threat event and outputting it to the display module and the cloud respectively, while performing early warning work based on the focus of the threat event.

[0012] The above technical solutions can focus attention on the area where the threat event occurs, highlight key information, help users observe and analyze threat events more clearly, and improve processing efficiency; through fuzzing, unnecessary data volume can be reduced, reducing the pressure on data storage and transmission, while effectively avoiding infringement on the privacy of others; outputting threat analysis reports and conducting early warning work, to a certain extent, protects the safety of vehicles and personnel, and facilitates users and managers to understand the threat situation in a timely manner.

[0013] Furthermore, after obtaining the focus of the threat event, the method also includes: encrypting the initial features and the focus of the threat event to obtain an encrypted data packet and storing it in a local database.

[0014] In the above technical solutions, encryption can effectively protect the security and privacy of data, prevent data from being stolen or tampered with during storage and transmission, and ensure that sensitive information is not leaked.

[0015] Furthermore, the set convolutional model further includes: acquiring user-labeled false positive events, generating encrypted feature gradients based on the user-labeled false positive events; uploading the encrypted feature gradients to the cloud for aggregation with historical events stored in the cloud, and updating the set convolutional model based on the aggregation result.

[0016] The above technical solution enables the convolutional model to continuously learn and improve, reducing the false alarm rate and improving the accuracy of recognition.

[0017] Furthermore, updating the set convolutional model further includes: obtaining a scene adaptation parameter package from the cloud based on a set period, so as to update the set convolutional model according to the scene adaptation parameter package.

[0018] The above technical solutions enable the convolutional model to adapt to different environments and scene changes in a timely manner.

[0019] Furthermore, it also includes: using a first power supply module to power the multimodal sensor module, and after outputting the current threat event, using a second power supply module to power the multimodal sensor module; wherein the multimodal sensor module includes at least an inertial sensor array, a positioning unit, a microphone array, and a camera.

[0020] In the above technical solution, different power supply modules can provide appropriate power support according to different working states, ensuring that the multimodal sensor module can operate stably under various conditions and avoiding system failures caused by insufficient or unstable power. Specifically, before outputting a threat event, a low-power power supply method is adopted to reduce the energy consumption of daily operation, while after outputting a threat event, a power supply module that can provide higher power can be used to ensure that the system can work stably.

[0021] Based on the same concept, this application also provides a vehicle monitoring system, the system comprising: a first acquisition module, configured to acquire sensor signals and positioning signals based on the inertial sensor array and the positioning unit respectively, and extract wavelet packet energy entropy and kurtosis index as initial features based on the sensor signals; a fusion module, configured to fuse the initial features and the positioning signals to obtain fused features; a second acquisition module, configured to acquire vibration features based on the fused features, and simultaneously acquire the acoustic spectrum of the microphone array; a wake-up module, configured to acquire a current threat event by combining the vibration features and the acoustic spectrum, so as to wake up the camera based on the current threat event; and a monitoring module, configured to invert the camera's field of view change model and acquire the focus of the threat event by combining the current threat event.

[0022] In the above technical solution, the combination of vibration characteristics and acoustic spectrum can effectively improve the accuracy of threat event identification, enabling vehicles to be effectively monitored and protected in special environments; at the same time, the camera will only be activated when a threat event is detected, reducing energy consumption and further ensuring the reliability of vehicle safety monitoring.

[0023] Based on the same concept, this application also provides an in-vehicle terminal, which is equipped with at least a vehicle monitoring system, a multimodal sensor module, a first power supply module, a second power supply module, and a display module, so as to implement the vehicle monitoring method through the vehicle monitoring system.

[0024] Compared with the prior art, the beneficial effects of this application are as follows: This application can effectively reduce the energy consumption of cameras and improve the accuracy of threat event identification in various environments. It solves the problems of high false negative rate and high energy consumption in existing vision systems, thus providing a more reliable and efficient solution for vehicle security monitoring. Attached Figure Description

[0025] Figure 1 This is a flowchart of the vehicle monitoring method described in this application.

[0026] Figure 2 This is a framework diagram of the vehicle monitoring system described in this application.

[0027] Figure 3 This is a framework diagram of the vehicle-mounted terminal described in this application. Detailed Implementation

[0028] The following describes in further detail a vehicle monitoring method, system, and vehicle terminal of this application with reference to specific embodiments and accompanying drawings.

[0029] Please see Figure 1This application provides a vehicle monitoring method, which pre-sets an inertial sensor array, a positioning unit, and a microphone array in a designated area of ​​the target vehicle; the vehicle monitoring method includes the following steps S100-S300.

[0030] In one feasible implementation, IMU signals and GNSS (Global Navigation Satellite System) positioning signals are acquired by an inertial sensor array and a dual-frequency GNSS positioning unit, respectively. Wavelet packet energy entropy and kurtosis index are extracted from the IMU signals as initial features. The initial features and the positioning signals are then fused to obtain fused features. Vibration features are extracted from the fused features, and acoustic information is acquired by a microphone array to obtain the acoustic spectrum. The vibration features and acoustic spectrum are output to a lightweight convolutional model to output threat classification events, such as environmental interference and minor scratches. When the vibration features are within a specific frequency band and the acoustic spectrum matches a set rule (such as the acoustic spectrum of metal friction or glass shards), the camera is activated. The camera's field of view change model is then inverted using the IMU signals, and the non-threat areas are automatically Gaussian blurred based on the target detection results, retaining only the clear event focus.

[0031] The following describes steps S100-S300 in detail.

[0032] S100: Based on the inertial sensor array and the positioning unit, sensor signals and positioning signals are acquired respectively, and wavelet packet energy entropy and kurtosis index are extracted from the sensor signals as initial features.

[0033] In some embodiments, high-precision MEMS-IMUs (i.e., the inertial sensor arrays) are pre-deployed in eight collision-sensitive areas of the vehicle body, such as the four doors and four wheel arches. These include at least a 6-axis accelerometer and a gyroscope, with a zero-bias stability of less than 0.5° / h.

[0034] The initial feature acquisition process includes: synchronizing the sensor signals obtained by each sensor in the inertial sensor array in time, constructing a vibration wave direction-of-arrival difference (DOAD) localization model based on the time-synchronized sensor signals, and extracting wavelet packet energy entropy and kurtosis index based on the vibration wave DOAD localization model as initial features.

[0035] In some embodiments, microsecond-level time synchronization of the inertial sensor array is achieved through the CAN-FD bus, and a Time Difference of Arrival (TDOA) positioning model is constructed with an error of <0.2m.

[0036] Based on the time-synchronized sensor signals, a vibration wave direction-of-arrival (TDOA) positioning model is constructed using the TDOA principle. TDOA positioning primarily utilizes the time difference between the arrival times of signals at different sensors to determine the location of the signal source (i.e., the vibration source). Specifically, based on the signals collected by each inertial sensor and their corresponding time differences, combined with the installation location information of the inertial sensors, a mathematical model is established to describe the propagation of vibration waves within the vehicle structure and the time relationship between their arrival times at different sensors. This model allows the calculation of the vibration source's position relative to the sensor array. Wavelet packet decomposition is performed on the sensor signals involved in the vibration wave TDOA positioning model to decompose the signals into different frequency sub-bands. Then, based on the wavelet packet decomposition, the energy of each frequency sub-band is calculated, and the wavelet packet energy entropy is calculated according to the energy distribution. Simultaneously, the kurtosis index of the sensor signals is calculated. Finally, the wavelet packet energy entropy and kurtosis index are used as initial features.

[0037] The above technical solution ensures the consistency and comparability of data collected by different sensors at the same time point, avoiding data deviations caused by time asynchrony. This makes the subsequently constructed vibration wave direction-of-arrival difference (DOA) localization model more accurate, thereby improving the extraction accuracy of initial features. By constructing the ODA localization model, the location of the vibration source can be determined more accurately by utilizing the time difference and spatial relationship of sensor signals. This helps to more accurately determine the location of the threat event and provides strong support for further processing. In addition, wavelet packet energy can describe the energy distribution of the vibration signal in different frequency bands, while kurtosis index can highlight the impact component in the vibration signal.

[0038] S200: The initial features and the positioning signal are fused to obtain fused features, and vibration features are obtained based on the fused features. At the same time, the acoustic spectrum of the microphone array is obtained.

[0039] In some embodiments, the positioning unit is such as a dual-frequency GNSS, and the initial features are fused with the dual-frequency GNSS signal (i.e., the positioning signal) using the RTS smoothing algorithm (Rauch-Tung-Striebel). When the satellite fails, the altitude error is less than 0.3m / 60 seconds.

[0040] The dual-frequency GNSS positioning unit receives satellite signals in real time and calculates the vehicle's position, including latitude, longitude, altitude, and other positioning signals. The dual-frequency GNSS uses signals from two frequencies for positioning calculation to effectively reduce the impact of ionospheric errors. The layout of the microphone array is set by the technicians themselves to collect the vehicle's own operating sounds, external environmental sounds, and sounds generated by possible threatening events.

[0041] Furthermore, based on the fusion features, vibration features are further extracted. The fusion features contain comprehensive information from the initial features and the positioning signal. By analyzing and processing the fusion features, features that better reflect the essence of vehicle vibration can be extracted. For example, signal processing methods can be used to perform filtering, spectrum analysis, and other operations on the fusion features to extract characteristic parameters such as vibration frequency and amplitude.

[0042] Furthermore, the voiceprint signal acquired by the microphone array is processed. First, a time-domain to frequency-domain conversion is performed. Methods such as Fast Fourier Transform (FFT) are typically used to convert the voiceprint signal from the time domain to the frequency domain. This allows us to obtain the distribution of the voiceprint signal at different frequencies, i.e., the voiceprint spectrum. Feature parameters such as peak frequency, bandwidth, and energy distribution are extracted from the voiceprint spectrum. These features can reflect the characteristic patterns of the voiceprint signal.

[0043] S300: Combine the vibration characteristics and acoustic spectrum to obtain the current threat event, wake up the camera according to the current threat event, and invert the camera's field of view change model, and then combine the current threat event to obtain the focus of the threat event.

[0044] The step of obtaining the current threat event includes: inputting the vibration features and acoustic spectrum into a set convolutional model to output the current threat event.

[0045] In some embodiments, the five-level threat classification includes environmental disturbances (falling leaves), minor scrapes (delivery trucks), severe collisions (door-opening attacks), vandalism (smashing windows), and attempted theft.

[0046] A large amount of data related to the above five threat classifications needs to be collected in advance, including vibration characteristics and acoustic signature spectrum data under various scenarios. Each scenario's data should be labeled with a corresponding threat level tag, followed by further preprocessing such as normalization. A lightweight convolutional model is then constructed, employing a multi-layered structure of convolutional, pooling, and fully connected layers to automatically extract feature patterns from the data. Training and optimization are performed using training and testing sets respectively to obtain the final convolutional model. Specifically, after receiving vibration characteristics and acoustic signature spectra, the convolutional model outputs the corresponding threat level classification based on the learned feature patterns and classification rules, serving as the current threat event.

[0047] Further, based on the current threat event, determine whether the vibration frequency band is within the range of 3-15Hz, and whether the acoustic signature matches specific spectral rules (such as the acoustic signature spectrum of metal friction or glass shards). If both conditions are met, activate the camera to start recording images or video information around the vehicle in order to further confirm the threat event.

[0048] It should be noted that the vibration frequency band and spectrum rules are set by those skilled in the art and are not restricted here.

[0049] In the above technical solutions, the convolutional model has good adaptability and generalization ability, and can learn the feature patterns corresponding to different threat events through a large amount of training data; in practical applications, it can effectively classify and identify various complex threat situations and has high reliability.

[0050] Furthermore, the set convolutional model further includes: acquiring user-labeled false positive events, generating encrypted feature gradients based on the user-labeled false positive events; uploading the encrypted feature gradients to the cloud for aggregation with historical events stored in the cloud, and updating the set convolutional model based on the aggregation result.

[0051] In some embodiments, after outputting threat events, they are typically displayed to the user. If the user finds that the model does not match the output threat events, they will mark them as false alarms. Based on the information of the recorded false alarm events, the feature gradient of the false alarm event is calculated using a backpropagation algorithm similar to the model training process. The feature gradient mainly reflects the direction and magnitude of the parameter changes when the model processes the false alarm event, and is used to guide the model update.

[0052] In order to protect the security of data, feature gradients are usually encrypted, such as by using symmetric encryption algorithms or asymmetric encryption algorithms to encrypt the feature gradients into ciphertext.

[0053] The historical events, such as previously collected false alarm events and correct classification events, are aggregated with the newly uploaded encrypted gradient features. The aggregation method can be selected according to the specific algorithm and requirements, such as using a weighted average, merging the new feature gradient with the historical feature gradient according to a certain weight, and then updating the convolution model based on the aggregated feature gradient.

[0054] The above technical solution enables the convolutional model to continuously learn and improve, reducing the false alarm rate and improving the accuracy of recognition.

[0055] Furthermore, updating the set convolutional model further includes: obtaining a scene adaptation parameter package from the cloud based on a set period, so as to update the set convolutional model according to the scene adaptation parameter package.

[0056] In some embodiments, the set period is preferably 24 hours, and the scenario adaptation parameter package includes, for example, the noise reduction coefficient for rainy environments in the south and the zero bias compensation for low-temperature IMUs in the north.

[0057] The above technical solutions enable the convolutional model to adapt to different environments and scene changes in a timely manner.

[0058] The process of obtaining the focus of a threat event includes: retrieving the camera field of view change model based on the initial features, and blurring non-threat areas by combining the target detection results and the current threat event to obtain the focus of the threat event; it also includes: generating a threat analysis report based on the focus of the threat event and outputting it to the display module and the cloud respectively, while performing early warning work based on the focus of the threat event.

[0059] In some embodiments, the correlation between the previously acquired initial features and the changes in the camera's field of view is analyzed to establish a mathematical model that can predict changes in the camera's field of view based on the initial features.

[0060] The YOLOv7 algorithm is used to perform target detection on images or videos captured by a camera. YOLOv7 can identify various targets in the image, such as people, vehicles, and objects, and provide their location and category information. The target detection results are then correlated with the current threat events output by the previous convolutional model. For example, if the convolutional model determines that there is a threat event of "vandalism (breaking windows)," the YOLOv7 detection results are combined to determine the specific location of the targets related to this threat event (such as the person breaking the window, the broken window, etc.) in the image.

[0061] Based on the target detection results and threat event information, non-threat areas are further identified. For example, an area 3 meters away in a non-threat direction is defined as a non-threat area. The non-threat area can be accurately delineated by the camera's position, field of view, and target distance information.

[0062] The system automatically Gaussian blurs the identified non-threat areas, retaining only the clear event focus, resulting in a desensitization coverage of more than 90% for non-threat areas. These focuses include information most relevant to the current threat event, such as the subject of the threat behavior and the affected targets.

[0063] Furthermore, based on the focus of the threat incident, organize the content of the threat analysis report; the report should include the type of threat incident (such as "minor scratch (delivery vehicle)", etc.), the time and location of the incident (which can be determined by the vehicle's location information), and a detailed description of the threatening behavior (such as the actions of the person, the damage to the target, etc.); further, generate a threat analysis report according to a certain format, and the report can be presented in various forms such as text and charts.

[0064] Furthermore, the generated threat analysis reports are output to a display module (such as an in-vehicle display screen, a mobile app, etc.) and a cloud server. The display module allows users to understand the current threat situation in a timely manner, while the cloud server can be used for subsequent data analysis and storage.

[0065] Furthermore, early warning work is carried out based on the focus of threat events; different early warning methods are adopted according to the severity of the threat events; for example, for serious threat events such as "severe collision (door opening kill)", users can be notified in time through sound and light alarms (such as vehicle alarms, mobile phone vibration and notification sounds); for relatively minor threat events such as "minor scratches (delivery vehicles)", reminders can be given through push messages from mobile apps.

[0066] The above technical solutions can focus attention on the area where the threat event occurs, highlight key information, help users observe and analyze threat events more clearly, and improve processing efficiency; through fuzzing, unnecessary data volume can be reduced, reducing the pressure on data storage and transmission, while effectively avoiding infringement on the privacy of others; outputting threat analysis reports and conducting early warning work, to a certain extent, protects the safety of vehicles and personnel, and facilitates users and managers to understand the threat situation in a timely manner.

[0067] Furthermore, after obtaining the focus of the threat event, the method also includes: encrypting the initial features and the focus of the threat event to obtain an encrypted data packet and storing it in a local database.

[0068] In some embodiments, symmetric or asymmetric encryption algorithms can also be used to encrypt the initial features and the focus of the threat event.

[0069] In the above technical solutions, encryption can effectively protect the security and privacy of data, prevent data from being stolen or tampered with during storage and transmission, and ensure that sensitive information is not leaked.

[0070] Furthermore, it also includes: using a first power supply module to power the multimodal sensor module, and after outputting the current threat event, using a second power supply module to power the multimodal sensor module; wherein the multimodal sensor module includes at least an inertial sensor array, a positioning unit, a microphone array, and a camera.

[0071] In some embodiments, the first power supply module, such as a flexible solar film on the roof (with a conversion rate greater than 22%), prioritizes powering the multimodal sensor module. After the voiceprint confirms the threat (i.e., the current threat event is obtained), the power battery (i.e., the second power supply battery) is used to supply power, so that the energy consumption in 10 hours is less than or equal to 0.1 kWh.

[0072] In the above technical solution, different power supply modules can provide appropriate power support according to different working states, ensuring that the multimodal sensor module can operate stably under various conditions and avoiding system failures caused by insufficient or unstable power. Specifically, before outputting a threat event, a low-power power supply method is adopted to reduce the energy consumption of daily operation, while after outputting a threat event, a power supply module that can provide higher power can be used to ensure that the system can work stably.

[0073] In summary, the vehicle monitoring method described in this application can not only effectively reduce the energy consumption of cameras and avoid unnecessary continuous operation, but also improve the accuracy of identifying threat events in various environments. It solves the problems of high false negative rate and high energy consumption in existing vision systems, and provides a more reliable and efficient solution for vehicle security monitoring.

[0074] Based on the same concept, please refer to Figure 2 This application also provides a vehicle monitoring system, the system comprising: a first acquisition module, configured to acquire sensor signals and positioning signals based on the inertial sensor array and positioning unit respectively, and extract wavelet packet energy entropy and kurtosis index as initial features based on the sensor signals; a fusion module, configured to fuse the initial features and the positioning signals to obtain fused features; a second acquisition module, configured to acquire vibration features based on the fused features, and simultaneously acquire the acoustic spectrum of the microphone array; a wake-up module, configured to acquire a current threat event by combining the vibration features and the acoustic spectrum, so as to wake up the camera based on the current threat event; and a monitoring module, configured to invert the camera's field of view change model and acquire the focus of the threat event by combining the current threat event.

[0075] In the above technical solution, the combination of vibration characteristics and acoustic spectrum can effectively improve the accuracy of threat event identification, enabling vehicles to be effectively monitored and protected in special environments; at the same time, the camera will only be activated when a threat event is detected, reducing energy consumption and further ensuring the reliability of vehicle safety monitoring.

[0076] Based on the same concept, please refer to Figure 3 This application also provides an in-vehicle terminal, which is equipped with at least a vehicle monitoring system, a multimodal sensor module, a first power supply module, a second power supply module, and a display module, so as to implement the vehicle monitoring method through the vehicle monitoring system.

[0077] It should be noted that the vehicle monitoring system and vehicle terminal described above are all based on the same concept as the vehicle monitoring method, and their specific implementation methods are also the same, so they will not be described in detail here.

[0078] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0081] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0082] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] Although the description of this application has been made in conjunction with the specific embodiments described above, it will be apparent to those skilled in the art that many substitutions, modifications, and variations can be made based on the foregoing. Therefore, all such substitutions, modifications, and variations are included within the spirit and scope of the appended claims.

Claims

1. A vehicle monitoring method characterized by, The inertial sensor array, the positioning unit and the microphone array are arranged in a set region of a target vehicle in advance; the vehicle monitoring method comprises the following steps: Sensor signals and positioning signals are acquired based on the inertial sensor array and the positioning unit respectively, and wavelet packet energy entropy and kurtosis indexes are extracted from the sensor signals as initial features; The initial features and the positioning signals are fused to obtain fused features, and vibration features are obtained based on the fused features, and a voiceprint spectrum of the microphone array is acquired; The current threat event is acquired based on the vibration features and the voiceprint spectrum, the camera is woken up based on the current threat event, a camera field of view angle change model is inverted, and a threat event focus is obtained based on the current threat event.

2. The vehicle monitoring method according to claim 1, characterized by, The initial features are acquired, comprising: The sensor signals acquired by each sensor in the inertial sensor array are time-synchronized, a vibration wave direction difference positioning model is constructed based on the time-synchronized sensor signals, and wavelet packet energy entropy and kurtosis indexes are extracted based on the vibration wave direction difference positioning model as initial features.

3. The vehicle monitoring method according to claim 1, characterized by, The current threat event is acquired, comprising: The vibration features and the voiceprint spectrum are input into a set convolution model to output the current threat event.

4. The vehicle monitoring method according to claim 3, characterized by, The threat event focus is obtained, comprising: The camera field of view angle change model is inverted based on the initial features, non-threat regions are blurred based on target detection results and the current threat event, and the threat event focus is obtained; The threat analysis report is generated based on the threat event focus and is output to a display module and a cloud, and the pre-warning work is performed based on the threat event focus.

5. The vehicle monitoring method according to claim 4, characterized by, After the threat event focus is obtained, the initial features and the threat event focus are encrypted to obtain encrypted data packets and store them in a local database. The set convolution model further comprises:

6. The vehicle monitoring method according to claim 3, characterized by, The user-labeled false alarm events are acquired, and encrypted feature gradients are generated based on the user-labeled false alarm events; The encrypted feature gradients are uploaded to the cloud to aggregate with historical events stored in the cloud, and the set convolution model is updated based on the aggregation result. The set convolution model is updated, further comprising:

7. The vehicle monitoring method according to claim 6, characterized by, The scene adaptation parameter package is acquired from the cloud based on a set period, and the set convolution model is updated based on the scene adaptation parameter package. Further comprising:

8. The vehicle monitoring method according to claim 3, characterized by, The first power supply module is used to supply power to the multi-modal sensor module, and the second power supply module is used to supply power to the multi-modal sensor module after the current threat event is output; wherein the multi-modal sensor module at least comprises the inertial sensor array, the positioning unit, the microphone array and the camera. The system comprises:

9. A system employing the vehicle monitoring method according to any one of claims 1 to 8, characterized by The first acquisition module is used to acquire sensor signals and positioning signals based on the inertial sensor array and the positioning unit respectively, and wavelet packet energy entropy and kurtosis indexes are extracted from the sensor signals as initial features; The fusion module is used to fuse the initial features and the positioning signals to obtain fused features; The second acquisition module is used to obtain vibration features based on the fused features, and a voiceprint spectrum of the microphone array is acquired; ​ An awakening module is configured to acquire a current threat event in combination with the vibration feature and the voiceprint spectrum, and to awaken a camera according to the current threat event; A monitoring module is configured to invert a camera field of view angle change model, and to acquire a threat event focus in combination with the current threat event.

10. A vehicle terminal, characterized by comprising: The vehicle-mounted terminal is at least configured with a vehicle monitoring system, a multi-modal sensor module, a first power supply module, a second power supply module, and a display module, so as to realize the vehicle monitoring method as claimed in any one of claims 1-8 through the vehicle monitoring system.

Images