Low-power vehicle-mounted monitoring method and system based on audio and image frame synthesis

Abstract

This invention relates to the field of intelligent connected vehicle technology, specifically to a low-power vehicle monitoring method and system based on audio and image frame synthesis, including an audiovisual acquisition module, a coordinate mapping module, a heatmap generation module, and a video synthesis module. This invention employs continuous audio monitoring and amplitude triggering mechanisms to sparsely acquire discrete visual keyframes without inter-frame predictive coding, and utilizes a microphone array to calculate the azimuth angle of the sound source. Cross-modal audiovisual coordinate mapping is performed in the cloud, projecting the physical sound source location as an image pixel trajectory, and generating a continuous dynamic saliency heatmap within the missing segments of the visual frames, moving along the trajectory and with its shape controlled by amplitude. By superimposing the heatmap sequence frame by frame onto static keyframes, a monitoring video stream with spatiotemporal continuity is synthesized. This invention effectively solves the problems of temporal discontinuities and missing spatial information of the sound source in monitoring blind spots with extremely low power consumption and bandwidth, significantly improving scene fidelity.

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicle technology, specifically to a low-power vehicle monitoring method and system based on audio and image frame synthesis. Background Technology

[0002] With the widespread adoption of vehicle-to-everything (V2X) technology and 4G / 5G networks, in-vehicle video surveillance has become a crucial means of vehicle security, especially with the increasing demand for parking monitoring after the vehicle is turned off. Existing in-vehicle monitoring solutions typically use public network cameras for continuous video capture, encoding, and uploading, or utilize in-vehicle intelligent cockpit domain controllers in conjunction with surround-view cameras for environmental perception. To address the high power consumption and high bandwidth issues associated with continuous recording, some methods have proposed a tiered acquisition strategy: reducing the frame rate when no valid target is detected, and only initiating high-specification recording when a threat is detected.

[0003] However, in practical applications, there are still insurmountable technical bottlenecks, mainly in the following three aspects: First, existing low-power solutions still rely heavily on high computing power on the edge, resulting in high basic static power consumption. Although existing technologies have reduced the frame rate of image acquisition, in order to achieve on-demand triggering or maintain the basic format of the video stream, the vehicle's main control chip still needs to continuously run visual perception algorithms to detect moving targets and must complete complex video encoding locally. This power-saving mechanism, which consumes power for the sake of saving power, makes it easy to deplete the vehicle's battery even when the device is parked in an airport or underground parking garage for several weeks, failing to achieve true microwatt-level ultra-long standby. Second, transmission reliability is poor and traffic costs are high in weak network environments such as underground parking garages. Traditional monitoring logic relies on uploading continuous video streams or large video segments. In areas with weak or unstable signals, such as underground parking garages, tunnels, or remote outdoor areas, bandwidth is often insufficient to support real-time video stream transmission, which can easily lead to packet loss or upload failure of critical monitoring data. In addition, even low-bitrate continuous video streams contain temporal redundancy information, which leads to unnecessary data traffic consumption and increases the user's operating costs. Finally, simply reducing the frame rate during data acquisition not only disrupts the temporal continuity of monitoring but also leads to a severe fragmentation of audiovisual spatial information. Existing technologies typically acquire images at extremely low frame rates (e.g., one frame every few seconds) in low-power mode. These discrete images appear as a "slideshow" during playback, failing to reconstruct the dynamic process of an event. Although some solutions attempt to incorporate audio recording, the lack of means to map sound signals to visual space means that when users hear a scraping or impact sound while viewing a still photograph of a vehicle, they cannot determine the specific location (e.g., left or right) and movement trajectory of the sound source. This misalignment of audiovisual modalities results in the monitoring data losing its ability to reconstruct the spatiotemporal trajectory of the scene during the "blind spot" period lacking continuous visual frames, significantly reducing the credibility and intuitiveness of the evidence.

[0004] In summary, how to solve the problems of discontinuous time dimension and inability to visualize and perceive spatial location information of sound sources in monitoring images due to missing visual frames under the premise of low power consumption and low data flow is a technical problem that urgently needs to be solved in this field.

[0005] To address this, a low-power vehicle monitoring method and system based on audio and image frame synthesis is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a low-power vehicle monitoring method and system based on audio and image frame synthesis. This invention employs continuous audio monitoring and amplitude triggering mechanisms to sparsely acquire discrete visual keyframes without inter-frame predictive coding, and utilizes a microphone array to calculate the azimuth angle of the sound source. Cross-modal audiovisual coordinate mapping is performed in the cloud, projecting the physical sound source location into an image pixel trajectory, and generating a continuous dynamic saliency heatmap within the missing segments of the visual frame, which moves along the trajectory and whose shape is controlled by amplitude. By superimposing the heatmap sequence frame by frame onto static keyframes, a monitoring video stream with spatiotemporal continuity is synthesized. This invention effectively solves the problems of temporal discontinuities and missing spatial information of the sound source in monitoring blind spots with extremely low power consumption and bandwidth, significantly improving scene reproduction accuracy.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a low-power vehicle monitoring method based on audio and image frame synthesis, comprising: continuously acquiring ambient sound slices to generate audio stream segments and monitoring sound wave signals; when the amplitude of the sound wave signal exceeds a preset trigger threshold, acquiring discrete visual keyframes without inter-frame predictive coding; calculating the spatial azimuth angle of the sound source of the sound wave signal using a microphone array, and associating and packaging the audio stream segments, the spatial azimuth angle of the sound source, and the discrete visual keyframes and uploading them to the cloud.

[0008] For the time gap between adjacent discrete visual keyframes, the cloud performs cross-modal audiovisual coordinate mapping; based on the camera's intrinsic parameter matrix and installation position, the spatial azimuth angle of the sound source is projected onto the pixel coordinate system of the discrete visual keyframe to generate the sound source trajectory coordinates.

[0009] Based on the coordinates of the sound source trajectory and the amplitude characteristics of the sound wave signal, a continuous dynamic saliency heatmap is generated within the time gap. The position of the highlighted area in the dynamic saliency heatmap moves with the coordinates of the sound source trajectory.

[0010] The dynamic saliency heatmap sequence is overlaid and fused frame by frame with the discrete visual keyframes collected at the trigger time as the static background map to generate a transition video frame sequence that fills the time gaps. This sequence is then concatenated with the discrete visual keyframes to synthesize a monitoring video stream.

[0011] Preferably, the continuous acquisition of environmental sound slices to generate audio stream segments and monitoring of sound wave signals specifically includes: the vehicle monitoring terminal keeping the microphone array always on in low-power standby mode, converting the acquired analog sound wave signals into digital signals and writing them into a circular audio buffer, and calculating the short-time energy value of the sound wave signals in real time; when the short-time energy value exceeds a preset silence threshold, it is determined that an audio stream segment needs to be generated; locking the current moment as a time anchor point, extracting data from the circular audio buffer before and after a preset time period before and after the time anchor point and splicing them together to generate the audio stream segment, and marking it with a unified Coordinated Universal Time (UTC) timestamp.

[0012] Preferably, when the amplitude of the acoustic signal exceeds a preset trigger threshold, discrete visual keyframes without inter-frame predictive coding are acquired, including: when the amplitude of the acoustic signal exceeds the preset trigger threshold, a wake-up command is sent to the image signal processor; the image signal processor directly generates single-frame RAW format raw data as discrete visual keyframes; after completing one acquisition, a forced cooling timer is started, and before the forced cooling timer expires, subsequent wake-up commands are blocked, and only audio data and sound source location information are recorded.

[0013] Preferably, the method of calculating the spatial azimuth angle of the sound source using a microphone array includes: performing frame segmentation and windowing processing on the microphone array signal; calculating the time difference between microphone units using a generalized cross-correlation algorithm; constructing a geometric trigonometric relationship model based on the time difference, the physical spacing between each microphone unit in the microphone array, and the sound speed constant in the air; calculating the incident angle of the sound source relative to the normal of the microphone array based on the geometric trigonometric relationship model, which is used as the spatial azimuth angle of the sound source; if the calculated incident angle has multiple solutions and ambiguity, introducing signal data from redundant microphone units for data verification.

[0014] Preferably, projecting the spatial azimuth angle of the sound source onto the pixel coordinate system of the discrete visual keyframe includes: retrieving a pre-calibrated extrinsic parameter matrix, which represents the rigid body rotation and translation transformation relationship of the microphone array coordinate system relative to the camera optical center coordinate system; constructing a three-dimensional unit direction vector in the microphone array coordinate system based on the spatial azimuth angle of the sound source; performing a coordinate space transformation on the three-dimensional unit direction vector using the extrinsic parameter matrix to generate a line-of-sight vector in the camera optical center coordinate system; retrieving the camera's intrinsic parameter matrix, which includes focal length parameters, principal point coordinate parameters, and radial distortion coefficients; constructing a three-dimensional sound source coordinate system by combining a preset sound source depth reference surface; projecting the three-dimensional sound source coordinate system onto the camera's two-dimensional imaging plane using the intrinsic parameter matrix; calculating the corresponding two-dimensional pixel coordinate points; determining whether the two-dimensional pixel coordinate points are within the resolution boundary range of the discrete visual keyframe; if within the range, marking the original pixel coordinate points as valid projection anchor points; if outside the range, calculating the geometric intersection of the three-dimensional line-of-sight vector and the image boundary line, and marking them as edge projection anchor points.

[0015] Preferably, generating sound source trajectory coordinates that change continuously over time specifically includes: performing time-frequency analysis on audio stream segments in the cloud, extracting instantaneous sound source location data at multiple discrete time points using a preset sampling rate, and generating multiple corresponding discrete projection anchor points; using a Kalman filter algorithm, performing time-series fitting with multiple discrete projection anchor points as control points to generate a smooth, continuous two-dimensional trajectory; for each transition video frame to be generated within the time gap, linearly indexing the corresponding coordinate position on the continuous two-dimensional trajectory according to time, and defining it as the sound source trajectory coordinates; when the sound source trajectory coordinates originate from the fitting path of the edge projection anchor points, they are marked as edge adsorption state.

[0016] Preferably, based on the sound source trajectory coordinates and the amplitude characteristics of the sound wave signal, a continuous dynamic saliency heatmap is generated within a time-deficient segment. Specifically, this includes: determining the duration of the time-deficient segment and calculating the total number of transition video frames to be generated within the time-deficient segment according to a preset video output frame rate; establishing a time index sequence synchronized with the transition video frame sequence; for each time point in the time index sequence, extracting the instantaneous amplitude characteristics of the corresponding moment from the audio stream segment and indexing the sound source trajectory coordinates of the corresponding moment from the continuous two-dimensional trajectory; using the extracted instantaneous amplitude characteristics and the sound source trajectory coordinates as input parameters, rendering the corresponding heatmap image frame by frame, and combining them into a continuous dynamic saliency heatmap sequence in time sequence.

[0017] Preferably, the position of the highlighted area in the dynamic saliency heatmap moves with the coordinates of the sound source trajectory, including: constructing a two-dimensional Gaussian distribution rendering model and anchoring the geometric center to the coordinates of the sound source trajectory; if the coordinates of the sound source trajectory are in an edge-attached state, locking the center of the two-dimensional Gaussian distribution pattern on the image boundary, and only rendering the breathing halo of the semi-circular area cut by the image boundary; if the coordinates of the sound source trajectory are not in an edge-attached state, rendering the complete two-dimensional Gaussian distribution pattern; establishing a first mapping relationship between the amplitude of the sound wave signal and the standard deviation of the Gaussian distribution, and a second mapping relationship between the amplitude of the sound wave signal and the peak opacity of the Gaussian distribution; and adjusting the parameters of the two-dimensional Gaussian distribution in real time according to the first and second mapping relationships to generate a dynamic saliency heatmap.

[0018] Preferably, the dynamic saliency heatmap sequence is overlaid and fused frame by frame using discrete visual keyframes acquired at the trigger time as a static background base map, including: obtaining the discrete visual keyframe preceding the start of the time-missing segment as the static base map; using an Alpha blending algorithm to overlay the dynamic saliency heatmap onto the static base map; setting a global attenuation coefficient for the heatmap layer, wherein the global attenuation coefficient is inversely proportional to the duration of the time-missing segment; and reducing the overall opacity of the heatmap layer according to the global attenuation coefficient as the transition video frame sequence progresses, until the heatmap layer is completely transparent at the end of the time-missing segment.

[0019] The low-power vehicle monitoring system based on audio and image frame synthesis includes: an audio-visual acquisition module: continuously acquiring ambient sound slices to generate audio stream segments and monitoring sound wave signals; when the amplitude of the sound wave signal exceeds a preset trigger threshold, acquiring discrete visual keyframes without inter-frame predictive coding; using a microphone array to calculate the spatial azimuth angle of the sound wave signal source, and associating and packaging the audio stream segments, the spatial azimuth angle of the sound source, and the discrete visual keyframes for uploading to the cloud.

[0020] Coordinate mapping module: The cloud performs cross-modal audiovisual coordinate mapping for the time gap between adjacent discrete visual keyframes; based on the camera's intrinsic parameter matrix and installation position, the spatial azimuth angle of the sound source is projected onto the pixel coordinate system of the discrete visual keyframe to generate the sound source trajectory coordinates.

[0021] Heatmap generation module: Based on the coordinates of the sound source trajectory and the amplitude characteristics of the sound wave signal, it generates a continuous dynamic saliency heatmap within the time gap. The position of the highlighted area in the dynamic saliency heatmap moves with the coordinates of the sound source trajectory.

[0022] Video synthesis module: The dynamic saliency heatmap sequence is overlaid and fused frame by frame with discrete visual keyframes collected at the trigger time as static background base map to generate a transition video frame sequence that fills the time missing segment, and is concatenated with discrete visual keyframes to synthesize a monitoring video stream.

[0023] Compared with existing technologies, the beneficial effects of this invention are as follows: 1. This invention, by employing an asymmetric acquisition strategy that combines continuous audio stream acquisition with sparse image capture triggered by sound wave amplitude, eliminates the terminal device's reliance on high-performance video encoding engines and continuous visual perception algorithms. This mechanism allows the vehicle terminal to maintain only microampere-level audio monitoring circuitry operation for the vast majority of the time, with the image sensor only momentarily awakened when a valid sound event occurs. This reduces the overall static power consumption to less than an order of magnitude of traditional visual monitoring solutions, significantly extending the monitoring endurance of the vehicle when the engine is off. Simultaneously, it significantly reduces the network traffic cost of uploading to the cloud, making it particularly suitable for data transmission in weak network environments such as underground parking garages.

[0024] 2. This invention effectively solves the problems of blind spots and missing spatial information caused by low-frequency image acquisition, significantly improving the integrity of the evidence chain. Utilizing microphone array sound source localization technology and cross-modal audiovisual coordinate mapping algorithms, it can accurately project auditory spatial orientation information onto a two-dimensional visual plane. Even in dim lighting, with visual obstruction, or during the visual vacuum period between two image acquisition frames, the system can still accurately pinpoint the specific location and movement trajectory of the sound source relative to the vehicle. For sound sources outside the field of view, it can also provide directional indication through an edge-attaching mechanism, thus endowing static images with environmental spatial perception capabilities not possessed by static images, avoiding the shortcomings of traditional low-power solutions that "only hear the sound but cannot see its location."

[0025] 3. This invention, through audiovisual fusion rendering technology, significantly optimizes the playback experience and information intuitiveness of low frame rate surveillance videos, reconstructing the spatiotemporal continuity of the surveillance scene. By generating continuous dynamic saliency heatmaps between adjacent discrete visual keyframes, this invention utilizes the energy characteristics of audio to drive changes in visual elements, visually filling the gap in the time dimension. This dynamic heatmap not only intuitively displays the occurrence process, intensity changes, and movement trends of sound events, but also effectively eliminates the "slideshow" stuttering sensation of traditional low frame rate playback, allowing users to obtain a smooth perception and a sense of on-site restoration, similar to continuous video, when viewing surveillance playback, greatly improving the readability of surveillance data and user experience. Attached Figure Description

[0026] Figure 1 A flowchart of a low-power vehicle monitoring method based on audio and image frame synthesis provided in an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the structure of a low-power vehicle monitoring system based on audio and image frame synthesis, provided in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram illustrating the principle of cross-modal audiovisual coordinate mapping and heatmap generation provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Please see Figures 1 to 3This invention provides a low-power vehicle monitoring method and system based on audio and image frame synthesis. The technical solution is as follows: The low-power vehicle monitoring method based on audio and image frame synthesis includes: continuously acquiring environmental sound slices to generate audio stream segments and monitoring sound wave signals; when the amplitude of the sound wave signal exceeds a preset trigger threshold, acquiring discrete visual keyframes without inter-frame predictive coding; calculating the spatial azimuth angle of the sound source of the sound wave signal using a microphone array, and associating and packaging the audio stream segments, the spatial azimuth angle of the sound source, and the discrete visual keyframes and uploading them to the cloud.

[0031] For the time gap between adjacent discrete visual keyframes, the cloud performs cross-modal audiovisual coordinate mapping; based on the camera's intrinsic parameter matrix and installation position, the spatial azimuth angle of the sound source is projected onto the pixel coordinate system of the discrete visual keyframe to generate the sound source trajectory coordinates.

[0032] Based on the coordinates of the sound source trajectory and the amplitude characteristics of the sound wave signal, a continuous dynamic saliency heatmap is generated within the time gap. The position of the highlighted area in the dynamic saliency heatmap moves with the coordinates of the sound source trajectory.

[0033] The dynamic saliency heatmap sequence is overlaid and fused frame by frame with the discrete visual keyframes collected at the trigger time as the static background map to generate a transition video frame sequence that fills the time gaps. This sequence is then concatenated with the discrete visual keyframes to synthesize a monitoring video stream. Example 1

[0034] The application scenario of this embodiment is as follows: A vehicle is parked next to a pillar in an underground parking garage. The area is dimly lit, and the 4G network signal strength is very weak (only about -110dBm). The vehicle has been locked and turned off for 15 days. During this period, a neighboring vehicle attempting to reverse into the garage accidentally and slightly scratches the left front bumper of this vehicle before driving away. Due to insufficient light and the slight movement, traditional visual trigger monitoring may not be triggered due to insufficient illumination or excessively high algorithm thresholds, or even if triggered, the video stream may not be uploaded due to poor network conditions. However, this invention accurately captures the "scratching sound" through acoustic features and generates a synthetic video containing the sound source trajectory, perfectly recreating the accident process.

[0035] As one embodiment of the present invention, refer to Figure 1 The flowchart of a low-power vehicle monitoring method based on audio and image frame synthesis is shown below. Figure 2 A schematic diagram of a low-power vehicle monitoring system based on audio and image frame synthesis, referring to... Figure 3 A schematic diagram illustrating the principle of cross-modal audiovisual coordinate mapping and heatmap generation.

[0036] Furthermore, the system continuously collects ambient sound slices to generate audio stream segments and monitors sound wave signals. Specifically, this includes: in low-power standby mode, the vehicle-mounted monitoring terminal keeps the microphone array constantly on, converts the collected analog sound wave signals into digital signals and writes them into a circular audio buffer, and calculates the short-time energy value of the sound wave signals in real time; when the short-time energy value exceeds a preset silence threshold, it determines that an audio stream segment needs to be generated; the current moment is locked as a time anchor point, and data from the circular audio buffer before and after the time anchor point for a preset time period is spliced ​​together to generate the audio stream segment, and a unified Coordinated Universal Time (UTC) timestamp is added. The preset time period is designed to ensure complete recording of the start and end of sound events and ambient background noise, and its duration is set to 3 to 15 seconds to cover the duration of most short-term scratch or impact events.

[0037] The audio stream segments employ pulse code modulation (PCM) in its original format, with a sampling rate of 16kHz and a quantization precision of 16 bits, stored as mono or multi-channel stereo data. Each audio stream segment contains continuous audio data from T_pre seconds before the trigger to T_post seconds after the trigger, where T_pre and T_post are preset parameters, typically set to 3 to 5 seconds, to ensure the initial acoustic characteristics of the event are captured before triggering and the decay process is captured after triggering. The trigger moment is defined as the sampling moment in the circular audio buffer when the short-time energy value first exceeds a preset silence threshold; this moment is marked with a uniform UTC timestamp (Coordinated Universal Time) with millisecond precision. When multiple triggers occur during the cooling timer masking period, each trigger generates only a new, independent audio stream segment, and each audio stream segment is time-aligned using its UTC timestamp.

[0038] Specifically, the vehicle monitoring terminal is equipped with a low-power microcontroller unit and a microelectromechanical system (MEMS) microphone array. After the vehicle is turned off and enters parking security mode, the main control chip and image signal processor enter deep sleep mode, with only the audio analog-to-digital converter module of the microcontroller unit continuously operating at a 16kHz sampling rate. The microcontroller unit internally allocates a 128KB random access memory area as a ring-shaped audio buffer, configured to store the most recent 8 seconds of raw PCM audio data. The digital signal processor core within the microcontroller unit calculates the short-time energy value of the written data every 20 milliseconds. Assuming a preset silence threshold of -40dBFS, when a neighboring vehicle scrapes against another vehicle, a sharp friction sound lasting approximately 1.5 seconds with a peak intensity of -10dBFS is generated. The preset silence threshold represents the upper limit of ambient background noise energy, used to distinguish between a silent state and a potential active state; its value range is typically set from -60dBFS to -35dBFS, depending on the acoustic substrate of the vehicle's environment (such as a quiet parking garage or a noisy roadside). The microcontroller detects that the energy value far exceeds the threshold and determines that the triggering event is valid. At this point, the microcontroller immediately locks the trigger time point T0 and uses a pointer offset algorithm to extract a total of 10 seconds of audio data from the circular buffer, from 5 seconds before T0 to 5 seconds after T0. This mechanism ensures that even if the scraping sound is sudden, the ambient noise before the sound occurs and the echo after the scraping ends can be completely recorded, forming a complete closed-loop audio stream segment of the evidence chain, and stamped with a precise UTC timestamp.

[0039] This invention solves the technical problem of "trigger-as-delay" in low-power monitoring through the aforementioned circular buffer and backtracking mechanism. In traditional solutions, device wake-up takes time, often resulting in key sounds ending before recording begins. However, this invention achieves pre-recording effects through a constantly open low-power circular buffer, ensuring the integrity of acoustic evidence for sudden sound events without significantly increasing power consumption, and providing a precise temporal reference for subsequent audiovisual synthesis.

[0040] Furthermore, when the amplitude of the acoustic signal exceeds a preset trigger threshold, discrete visual keyframes without inter-frame predictive coding are acquired, including: when the amplitude of the acoustic signal exceeds the preset trigger threshold, a wake-up command is sent to the image signal processor; the image signal processor directly generates single-frame RAW format raw data as discrete visual keyframes; after completing one acquisition, a forced cooling timer is started, and before the forced cooling timer expires, subsequent wake-up commands are blocked, and only audio data and sound source location information are recorded.

[0041] Specifically, in the aforementioned scratch event, when the microcontroller detects that the acoustic wave amplitude exceeds the trigger threshold (set to a wake-up threshold of -20dBFS), the microcontroller sends a high-level hard interrupt signal to the image sensor and image signal processor via a GPIO pin. The preset trigger threshold is used to determine the occurrence of a valid threat event (such as scratching or impact), and its value is strictly higher than the preset silence threshold, typically set between -30dBFS and -5dBFS to filter out common environmental noise interference. The image signal processor cold-starts from sleep mode, with the startup time controlled within 200ms. Unlike traditional monitoring systems that encode H.264 video streams after startup, in this embodiment, the image signal processor is configured in "single-frame capture mode." It directly reads the sensor's RAW data, performs de-mosaicing and automatic white balance, and outputs a 1920x1080 resolution JPEG image with a file size of approximately 300KB. After this capture is completed, a forced cooling timer with a duration of 5 seconds is immediately started. Assuming the sound fluctuates continuously for 10 seconds during the scratch, and the amplitude exceeds the threshold multiple times, the image signal processor rejects all new wake-up requests during the cooldown period due to the existence of a cooling timer. This means that only one or two key still images are captured during the entire 10-second event, rather than a continuous video stream. Statistics show that this strategy reduces the duty cycle of the image acquisition module to below 1%, and the power consumption per event is only 1 / 20th of that of traditional 10-second video recording.

[0042] This invention fundamentally eliminates the two largest power sources in vehicle monitoring—high-frequency image signal processor computation and video encoder power consumption—by skipping complex inter-frame predictive coding and forced cooling strategies. This design not only significantly extends battery life, enabling vehicles to maintain monitoring capabilities even when parked for weeks, but also generates extremely small amounts of discrete JPEG image data, creating a prerequisite for successful transmission in weak network environments such as underground parking garages, thus solving the industry pain point of "capturing images but being unable to transmit them."

[0043] Furthermore, the spatial azimuth angle of the sound source signal is calculated using a microphone array, including: framing and windowing the microphone array signal; calculating the time difference between microphone units using a generalized cross-correlation algorithm; constructing a geometric trigonometric relationship model based on the time difference, the physical spacing between microphone units in the microphone array, and the sound speed constant in air; calculating the incident angle of the sound source relative to the normal of the microphone array based on the geometric trigonometric relationship model, which is used as the spatial azimuth angle of the sound source; if the calculated incident angle has multiple solutions and ambiguity, signal data from redundant microphone units are introduced for data verification.

[0044] If the calculated incident angle exhibits multiple solutions (specifically, when the maximum observable frequency of the microphone pair satisfies d > λ_max / 2, the angle corresponding to the time difference has multiple solutions within the range of 0° to 180°), then signal data from redundant microphone units is introduced to verify each candidate solution. The candidate solution with the smallest global matching error is selected as the true azimuth angle of the sound source.

[0045] The geometric trigonometric relationship model is based on the plane wave acoustic assumption, which states that when the sound source is sufficiently far from the microphone array, the wavefronts of the sound waves reaching each microphone unit can be approximated as parallel straight lines. Assume a linear microphone array contains two microphone units, and the sound wave emitted by the source is incident at a certain angle relative to the array normal. Due to the existence of the incident angle, there is a path difference in the path of the sound wave reaching the two microphone units. This path difference is equal to the product of the physical distance between the microphones and the cosine of the incident angle; the corresponding time difference is equal to the path difference divided by the speed of sound constant.

[0046] Based on the above geometric relationship, a mapping relationship between the incident angle of the sound source and the time difference can be constructed: the cosine value of the incident angle of the sound source is equal to the product of the sound speed constant and the time difference, divided by the physical distance between the microphones; by taking the inverse cosine function of this cosine value, the incident angle of the sound source relative to the normal of the microphone array can be obtained.

[0047] Taking this embodiment as an example, the specific calculation process is as follows: Obtain the time difference between microphone units output by the generalized cross-correlation algorithm, which is 0.085ms in this example; Determine the known hardware parameters: there are three gaps between microphone one and microphone four, with a total physical gap of 12cm; the sound speed constant in air is taken as 340m / s; Calculate the product of the sound speed constant and the time difference to obtain a sound wave path difference of approximately 2.89cm; Calculate the ratio of the path difference to the physical gap between the microphones to obtain a cosine value of the incident angle of approximately 0.24; Apply the inverse cosine function to this cosine value to obtain an incident angle of the sound source relative to the array normal of approximately 76°; Convert the incident angle to the azimuth angle relative to the front of the vehicle: since the microphone array normal is perpendicular to the front of the vehicle, the azimuth angle of the sound source is equal to 90° minus the incident angle, which is approximately 14°; Combine the positive and negative signs of the time difference to determine that the sound source is located on the left side of the vehicle, and finally determine the azimuth of the sound source to be approximately 35° to the left front.

[0048] The above-mentioned geometric trigonometric relationship model is a deterministic analytical model. It does not require a machine learning training process and can directly calculate the azimuth angle of the sound source based solely on the laws of acoustic physics and the calibrated hardware parameters.

[0049] Specifically, the vehicle terminal is equipped with a linear array of four microphones spaced 4 centimeters apart. When a scratch sound is collected, the microcontroller processes the four digital audio signals in frames. For each frame, the algorithm first applies a Hanning window to reduce spectral leakage, then uses a generalized cross-correlation-phase transform algorithm to calculate the cross-correlation function between the microphone pairs. The core of the generalized cross-correlation-phase transform algorithm lies in its whitening filter, which effectively suppresses strong wall echoes and reverberation interference in underground parking garages. The peak position found in the cross-correlation function is the time difference. For example, the arrival time difference between microphone 1 and microphone 4 is calculated to be 0.000085 seconds. The algorithm uses the inverse cosine function to calculate the ratio of the product of the sound speed and the time difference to the physical distance, thus determining that the incident angle of the sound source relative to the front of the vehicle is 35° to the left front. To prevent "front-rear mirror blur" caused by the linear array, a third redundant microphone signal installed at the rearview mirror is used for verification to confirm that the sound wave energy mainly comes from the front of the vehicle, thereby locking the unique azimuth angle of the sound source.

[0050] The associated packaging specifically refers to binding the following data elements into a single data packet and uploading it to the cloud: (1) the start UTC timestamp T_audio_start (millisecond precision) and end timestamp T_audio_end of the audio stream segment; (2) the discrete sound source azimuth data sequence extracted every Δt_feature time (the time interval corresponding to 10Hz-50Hz) within the audio time period and its corresponding time marker; (3) the UTC timestamp T_image of the acquisition time of the discrete visual keyframe, with millisecond precision; (4) the original pixel data of the discrete visual keyframe; (5) the vehicle identification, camera serial number and microphone array serial number; After receiving the data packet, the cloud first retrieves the camera intrinsic parameter matrix, extrinsic parameter matrix, and calibration timestamp of the vehicle model from the local database based on the vehicle model identifier. If the current time is more than the preset parameter validity period (usually 12 months) from the calibration timestamp, the system sends a calibration update request to the vehicle terminal.

[0051] This invention employs a generalized cross-correlation-phase transform algorithm combined with microphone array geometric calculations to effectively solve the problem of inaccurate sound source localization caused by multipath effects and reverberation in vehicle environments. Compared to simple energy detection, this solution extracts the "spatial attributes" of sound, enabling the monitoring system to not only know "what happened" but also "where it happened," providing precise physical evidence for subsequently drawing dynamic trajectories on static images and significantly enhancing the evidentiary value of monitoring data.

[0052] Further, projecting the spatial azimuth angle of the sound source onto the pixel coordinate system of the discrete visual keyframe includes: retrieving a pre-calibrated extrinsic parameter matrix, which represents the rigid body rotation and translation transformation relationship of the microphone array coordinate system relative to the camera optical center coordinate system; constructing a three-dimensional unit direction vector in the microphone array coordinate system based on the spatial azimuth angle of the sound source; performing a coordinate space transformation on the three-dimensional unit direction vector using the extrinsic parameter matrix to generate a line-of-sight vector in the camera optical center coordinate system; retrieving the camera's intrinsic parameter matrix, which includes focal length parameters, principal point coordinate parameters, and radial distortion coefficients; constructing three-dimensional sound source coordinates by combining a preset sound source depth reference surface; projecting the three-dimensional sound source coordinates onto the camera's two-dimensional imaging plane using the intrinsic parameter matrix, and calculating the corresponding two-dimensional pixel coordinate points; determining whether the two-dimensional pixel coordinate points are within the resolution boundary range of the discrete visual keyframe; if within the range, marking the original pixel coordinate points as valid projection anchor points; if outside the range, calculating the geometric intersection of the three-dimensional line-of-sight vector and the image boundary line, and marking them as edge projection anchor points.

[0053] The microphone array is arranged linearly. The time difference calculated using a generalized cross-correlation algorithm uniquely determines the azimuth angle φ of the sound source relative to the microphone array normal in the horizontal plane. Due to the characteristics of a linear array, this invention employs the following approach: it assumes all sound sources are located within the horizontal plane of the microphone array (elevation angle θ = 0°), an assumption that holds true in monitoring scenarios around vehicles. For more accurate elevation angle estimation, a dual-axis microphone array or other sensors can be used.

[0054] The sound source depth reference surface is set as a preset reference plane, and the distance between this plane and the depth of the microphone array is defined as the reference depth distance. For forward-looking monitoring scenarios in standard passenger vehicles, this reference depth distance is typically set to 1.5 meters to 4.0 meters. When the sound source distance cannot be dynamically estimated, 2.5 meters is used as the default conservative reference distance.

[0055] Furthermore, the present invention includes a dynamic distance estimation mechanism based on the energy ratio of direct sound to reverberant sound. The specific steps of performing cepstral analysis on audio stream segments in the cloud include: (1) performing a short-time Fourier transform on each audio frame to obtain an amplitude spectrum; (2) taking the natural logarithm of the amplitude spectrum to obtain a logarithmic amplitude spectrum; (3) performing an inverse Fourier transform on the logarithmic amplitude spectrum to obtain a cepstral sequence, wherein the order of the cepstral coefficients is set to thirteenth order; (4) analyzing the peak position of the cepstral sequence, defining the sum of low-order cepstral coefficients with index values ​​less than half the order of the cepstral as the direct sound component energy, and defining the sum of high-order cepstral coefficients with index values ​​greater than half the order of the cepstral and less than or equal to the full order as the reverberant sound component energy; (5) calculating the energy ratio, specifically calculating the base-10 logarithm of the ratio of the direct sound component energy to the reverberant sound component energy, and then multiplying the logarithmic value by 10 to obtain the energy ratio in decibels.

[0056] This distance estimation mechanism is effective for the following application scenarios: reverberation time less than 1.5 seconds; sound source frequency concentrated in the mid-to-high frequency range of 2 kHz to 8 kHz; and the energy ratio of direct sound to reverberant sound ranging from -15 dB to +10 dB. When any of the above conditions are not met, the distance estimation logic automatically degrades and directly adopts the preset 2.5 meters as the default reference distance.

[0057] The external parameter matrix is ​​obtained through joint calibration at the factory. During the calibration phase, a synchronous calibration source that emits both sound and light is used to record the sound source angle data calculated by the microphone array and the pixel coordinate data of the light point captured by the camera, respectively. The rotation matrix and translation vector that minimize the projection error of the sound source angle are calculated using the least squares method iterative solution. This rotation matrix and translation vector constitute the external parameter matrix, which is used to characterize the rigid body transformation relationship between the microphone array coordinate system and the camera optical center coordinate system.

[0058] Specifically, the cloud server stores the calibration data of the vehicle's camera. The extrinsic parameter matrix describes the rotation and translation relationship between the microphone array center and the optical center of the forward-facing camera. The intrinsic parameter matrix includes the focal length (e.g., 1200 pixels), principal point coordinates, and distortion coefficients. A three-dimensional gaze vector is constructed for the previously calculated "35° left front" sound source azimuth angle. This is transformed to the camera coordinate system by multiplying the rotation matrix by the microphone coordinate vector and adding the translation vector. Then, using the principle of perspective projection—specifically, the ratio of focal length to depth distance—the corresponding pixel coordinates (240, 600) on a 1080P image are calculated. Simultaneously, boundary checks are performed: assuming another impact sound comes from the left side of the vehicle, the projection calculation result is a negative coordinate, outside the image resolution range. At this point, the algorithm calculates the geometric intersection of the gaze vector and the left boundary line of the image, marks this intersection as an "edge projection anchor point," and records its direction attribute as "left side."

[0059] This invention achieves precise alignment between auditory and visual spaces through rigorous computer vision geometric projection technology. This step serves as a bridge connecting "invisible sound" and "visible image." In particular, the calculation and marking of edge intersections for sound sources outside the field of view endows the monitoring system with "beyond-line-of-sight perception" capabilities—even if the user does not see the offending vehicle in the image, they can intuitively determine which side of the image the threat originates from through edge markings, greatly expanding the information dimension of static images.

[0060] The cloud performs time-frequency analysis on the audio stream segment. First, the audio stream segment is preprocessed: (1) A first-order pre-emphasis filter (pre-emphasis coefficient 0.97) is applied to enhance the high-frequency components and improve the signal-to-noise ratio of the high-frequency capture of the microphone array; (2) The preprocessed audio stream segment is framed with 50% overlap, and each frame has 512 sampling points (about 32ms, corresponding to a 16kHz sampling rate). The Hanning window function is applied to reduce spectral leakage; (3) A 1024-point fast Fourier transform is performed on each frame to obtain a time-frequency spectrum. The number of time-frequency units is 512 (the positive frequency range corresponding to the 512 sampling points), and the frequency resolution is 31.25Hz (16000Hz÷512). The cloud traverses the time-frequency spectrum with a feature extraction frequency of 10Hz to 50Hz (different from the original sampling rate of 16kHz of the audio signal), and extracts instantaneous sound source location data once every 32ms to 100ms to generate corresponding discrete projection anchor points.

[0061] Furthermore, generating sound source trajectory coordinates that change continuously over time specifically includes: performing time-frequency analysis on audio stream segments in the cloud, extracting instantaneous sound source location data at multiple discrete time points using a preset sampling rate (i.e., feature extraction frequency, which is different from the original sampling rate of the audio signal, usually set to 10Hz to 50Hz to reduce computation while ensuring the smoothness of the sound source trajectory), and generating multiple corresponding discrete projection anchor points; using the Kalman filter algorithm, performing time-series fitting with multiple discrete projection anchor points as control points to generate a smooth continuous two-dimensional trajectory; for each transition video frame to be generated within the time gap, linearly indexing the corresponding coordinate position on the continuous two-dimensional trajectory according to time, and defining it as the sound source trajectory coordinates; when the sound source trajectory coordinates originate from the fitting path of the edge projection anchor points, it is marked as an edge adsorption state.

[0062] When using the Kalman filter algorithm for time series fitting, the state vector is defined as a four-dimensional vector containing the horizontal coordinate position, the vertical coordinate position, the horizontal velocity component, and the vertical velocity component. The process model is based on the assumption that the sound source moves at a constant velocity within the missing segment, meaning that the position state at the next moment is equal to the position state at the current moment plus the product of the velocity state at the current moment and the time interval. It is also assumed that the velocity state at the next moment is consistent with the current moment. The time interval corresponds to the sampling duration between adjacent discrete projection anchor points, typically ranging from 20 milliseconds to 100 milliseconds.

[0063] The observation model only observes position coordinates; the observation vector includes the projected positions of the horizontal and vertical coordinates, but does not include velocity components. The process noise covariance matrix is ​​set as a diagonal matrix, with the position noise variance component set to 4 square pixels and the velocity noise variance component set to 1 square pixel per square millisecond to simulate the minute random jitter of the sound source motion. The observation noise covariance matrix is ​​also set as a diagonal matrix, with its position observation noise variance set to 9 square pixels to accommodate angle estimation errors.

[0064] For each transitional video frame to be generated within the time-gap segment, the relative position of the frame's time point in the time-frequency feature sequence is first calculated. If the time point lies between two adjacent anchor points, cubic spline interpolation is used to interpolate the output trajectory of the Kalman filter to calculate the precise coordinate position corresponding to that time point. Furthermore, when the sound source trajectory coordinates are determined to be in an edge-attached state, the coordinate values ​​are forcibly restricted to the image boundary line in subsequent Kalman filter prediction steps to prevent the trajectory from extrapolating beyond the field of view.

[0065] Specifically, when processing the aforementioned 10-second audio stream segment on the cloud server, time-frequency analysis is performed at a high frequency of 20 times per second to extract a total of 200 discrete instantaneous sound source location data. After projection as described in claim 5, these 200 data points are converted into 200 discrete pixels on the image plane. Due to the jitter in the sound wave signal, these original points may appear as jagged jumps on the image. To address this, the cloud uses a Kalman filter algorithm to perform state estimation and smoothing processing on these points. Assume that during the 2nd to 4th second after the scrape, the offending vehicle scrapes the bumper from left to right. The original data points jump from the left pixel coordinates to the right pixel coordinates. The Kalman filter, based on a physical motion model, predicts and corrects noise, generating a smooth Bézier curve trajectory. For the transition video frames to be generated (assuming an output of 15fps), this smooth curve is linearly indexed on the time axis. For example, for frame 30, the algorithm finds the corresponding coordinate point (450, 620) on the curve and defines it as the sound source trajectory coordinates for that frame. If a certain trajectory originates from an edge projection anchor point, then all coordinate points on that trajectory are marked as edge snapping, and the coordinate values ​​are fixed on the image boundary.

[0066] This invention transforms discrete, jittery sound source localization data into a continuous, smooth visual trajectory through temporal fitting and interpolation techniques. This process effectively solves the temporal gap problem caused by low-frequency acquisition, enabling the description of an event's development using a virtual trajectory that conforms to the laws of physical motion, even without real video frames. This "virtual-to-real" approach provides a continuous and stable spatiotemporal reference for subsequent heatmap rendering, avoiding visual abruptness.

[0067] Furthermore, generating sound source trajectory coordinates that change continuously over time also includes a depth coordinate correction step based on distance estimation: the cloud server performs cepstral analysis or power spectral density comparison on the audio stream segment, calculates the energy ratio of direct sound to reverberant sound, and maps the energy ratio to a sound source distance value according to a preset environmental acoustic model; combining the sound source spatial azimuth and the sound source distance value, the polar coordinate position of the sound source in three-dimensional space is constructed, and the three-dimensional position is projected onto the pixel coordinate system of discrete visual keyframes to obtain the sound source trajectory coordinates containing depth information; the diffusion range of the dynamic saliency heatmap no longer depends solely on the amplitude, but is determined by both the sound source distance value and the amplitude: under the same amplitude, the smaller the sound source distance value, the more convergent the diffusion range of the highlighted area; the larger the sound source distance value, the more divergent the diffusion range of the highlighted area, simulating the perspective effect of near objects appearing larger and far objects appearing smaller.

[0068] Specifically, when processing the scraping sound segment, the cloud server's digital signal processor (DSP) calculates the cepstral distance of the audio signal to determine that the direct sound to reverberant energy ratio is -3dB. Based on a pre-set acoustic attenuation model of an underground parking garage, the distance between the sound source and the microphone array is estimated to be approximately 2.5 meters. When rendering the heatmap of frame 15, although the amplitude of the scraping sound is relatively large (-10dBFS), due to the close calculated distance, a large diffuse reflection spot is not rendered. Instead, a Gaussian spot with a radius of 50 pixels but extremely high center brightness is generated. Conversely, if an impact sound of the same amplitude is detected at a distance (e.g., 10 meters away), the reverberant energy ratio is -15dB, generating a pale spot with a radius of 300 pixels but extremely blurred edges. By introducing a distance variable, the simplistic logic that loudness equals area is corrected, realistically restoring the physical depth of the sound source.

[0069] Cepstral analysis is reliable when the following conditions are met: (a) reverberation time RT60 ≤ 1.5 seconds; (b) sound source frequency is concentrated in the range of 2kHz-8kHz; (c) signal-to-noise ratio (SNR) > 10dB; (d) sound source distance is in the range of 1m-5m. When the conditions are not met, the distance estimation is downgraded to a preset reference value.

[0070] This invention addresses the limitation of traditional sound source localization, which can only obtain angle but not distance, by introducing direct sound and reverberant sound energy ratio analysis technology. By mapping distance information to the diffusion pattern of a heat map (converging at close range and diverging at distant range), a quasi-three-dimensional auditory spatial perception is successfully constructed on a two-dimensional monitoring screen, effectively avoiding misjudgments caused by distance confusion and significantly improving the physical realism of the reconstructed monitoring scene.

[0071] Furthermore, based on the sound source trajectory coordinates and the amplitude characteristics of the sound wave signal, a continuous dynamic saliency heatmap is generated within the time-deficient segment. Specifically, this includes: determining the duration of the time-deficient segment and calculating the total number of transition video frames to be generated within the time-deficient segment according to a preset video output frame rate (usually 10fps to 60fps, used to define the temporal resolution of the synthesized video stream); establishing a time index sequence synchronized with the transition video frame sequence; for each time point in the time index sequence, extracting the instantaneous amplitude characteristics of the corresponding moment from the audio stream segment and indexing the sound source trajectory coordinates of the corresponding moment from the continuous two-dimensional trajectory; using the extracted instantaneous amplitude characteristics and the sound source trajectory coordinates as input parameters, rendering the corresponding heatmap image frame by frame, and combining them into a continuous dynamic saliency heatmap sequence in time sequence.

[0072] Specifically, the cloud first determines the time gap between two discrete image frames to be 5 seconds. If the target video frame rate is 15fps, a total of 75 heatmap frames need to be generated. A time index sequence from 0 to 74 is established. For index 10 (corresponding to 0.66 seconds), the following operations are performed: Feature extraction: The instantaneous amplitude at this moment is read from the audio stream as -15dBFS (higher volume), and the coordinates are read from the trajectory as (300, 610). Sequence generation: This set of coordinates and amplitude parameters is passed as input to the rendering engine. For index 40 (corresponding to 2.66 seconds, the scraping ends, and the sound becomes quieter), the instantaneous amplitude is extracted as -35dBFS (lower volume), and the coordinates are moved to (900, 660). The above operations are performed 75 times in a loop, generating 75 transparent background image sequences containing heatmaps of different shapes. These sequences are continuous in time, moving in space, and fluctuating in intensity.

[0073] This invention achieves frame-by-frame alignment of audio features with visual trajectories by establishing a strict time index sequence. This serialization generation mechanism breaks the limitations of static images, constructing a "time dimension" that allows the originally static monitoring footage to display the dynamic evolution (start, climax, and end) of sound events in the form of a video stream, providing users with an intuitive "event playback" experience, rather than simply viewing a static on-site photo.

[0074] Furthermore, the position of the highlighted area in the dynamic saliency heatmap moves with the coordinates of the sound source trajectory, including: constructing a two-dimensional Gaussian distribution rendering model and anchoring the geometric center to the coordinates of the sound source trajectory; if the sound source trajectory coordinates are in an edge-attached state, locking the center of the two-dimensional Gaussian distribution pattern on the image boundary, and only rendering the breathing halo of the semi-circular area cut by the image boundary; if the sound source trajectory coordinates are not in an edge-attached state, rendering the complete two-dimensional Gaussian distribution pattern; establishing a first mapping relationship between the amplitude of the sound wave signal and the standard deviation of the Gaussian distribution, and a second mapping relationship between the amplitude of the sound wave signal and the peak opacity of the Gaussian distribution; and adjusting the parameters of the two-dimensional Gaussian distribution in real time according to the first and second mapping relationships to generate the dynamic saliency heatmap.

[0075] The two-dimensional Gaussian distribution, used as the rendering model for heatmaps, employs a linear mapping logic where the standard deviation equals a preset minimum standard deviation base value, plus the product of the standard deviation scaling factor and the change in sound wave amplitude. This calculation logic is a simplified processing based on engineering experience, primarily aimed at enhancing users' visual perception and understanding of sound source intensity information through intuitive dynamic changes in the heatmap, rather than precisely replicating the rigorous mathematical model of sound pressure attenuating logarithmically with distance in acoustic physics.

[0076] The first mapping relationship employs linear growth logic: a base radius value is preset as the minimum standard deviation of the Gaussian distribution; when the amplitude of the real-time acoustic signal exceeds a preset silence threshold, the difference between the current amplitude and the silence threshold is calculated; this difference is multiplied by a preset radius scaling factor to obtain a dynamic increment; finally, the dynamic increment is added to the base radius value to obtain the standard deviation of the Gaussian distribution for the current frame. The second mapping relationship employs non-linear saturation logic to ensure that the peak opacity of the heatmap does not exceed a preset upper limit under high amplitude, avoiding complete occlusion of the background image.

[0077] The parameter mapping relationship of the 2D Gaussian distribution rendering model is defined as follows: The first mapping relationship is used to adjust the standard deviation of the Gaussian distribution according to the sound wave amplitude. First, the amplitude deviation is calculated, which is the difference between the current sound wave signal amplitude and the preset silence threshold (usually -40 to -35 dBFS). The standard deviation calculation follows a linear growth logic: the amplitude deviation is multiplied by a preset radius scaling factor (e.g., 2 pixels for every 1 dB increase), and the product is added to a preset minimum standard deviation base value (e.g., 20 pixels). The system also sets an upper limit for the standard deviation (e.g., 200 pixels) to prevent excessive diffusion of the light spot; The second mapping relationship is used to adjust the peak opacity of the Gaussian distribution according to the sound wave amplitude. This calculation follows an exponential saturation logic: First, the amplitude deviation is multiplied by a preset saturation rate coefficient (e.g., 0.2 dB); then, the natural exponential function value of the negative value of the product is calculated; then, the saturation factor is obtained by subtracting the natural exponential function value; finally, the saturation factor is multiplied by a preset upper limit for peak opacity (e.g., 0.8). According to this logic, when the amplitude approaches the mute threshold, the opacity approaches zero; as the amplitude increases, the opacity rises rapidly and non-linearly, approaching the upper limit.

[0078] Specifically, a two-dimensional Gaussian distribution model is constructed, and shape control and parameter mapping are performed. Regarding shape control, when the coordinates of frame 10 are not in an edge-attached state (within the field of view), the rendering engine draws a complete circular Gaussian spot centered on that point. When the coordinates of frame 0 are in an edge-attached state (outside the field of view), the rendering engine locks the center at the left boundary, drawing a semi-circular spot cut off by the screen edge, forming an edge breathing halo effect similar to a "breathing light," visually indicating to the user that "the sound comes from outside the left side of the screen." Simultaneously, the diffusion range and color intensity are adjusted through parameter mapping: a linear mapping relationship is established, where the amplitude value plus the bias constant is multiplied by the scaling factor to obtain the standard deviation. For example, a large amplitude corresponds to a larger standard deviation, resulting in a spot coverage radius of 150 pixels, while a small amplitude corresponds to a smaller standard deviation, resulting in a spot radius of only 30 pixels. A color intensity mapping relationship is also established, where a large amplitude corresponds to high opacity and warm tones (red / orange), and a small amplitude corresponds to low opacity and cool tones (green / blue). Based on the above logic, the final generated heatmap for frame 10 is a bright orange-red spot with a large radius located in the lower left corner, while frame 40 becomes a faint light spot with a small radius located on the right side and a pale green color.

[0079] This invention transforms abstract sound signals into a concrete visual language through a mathematical model. By using a Gaussian distribution model and the morphological switching of edge halos, it intuitively solves the spatial indication problem of "inside and outside the screen"; through a nonlinear mapping from amplitude to geometric parameters, it achieves the visual perception of sound intensity. This design allows users to intuitively perceive the rhythm and location of sound even when watching silent images through the "scaling" and "color transitions" of heatmaps, greatly enhancing the information transmission efficiency of surveillance videos.

[0080] Furthermore, the dynamic saliency heatmap sequence is overlaid and fused frame by frame using discrete visual keyframes acquired at the trigger time as a static background base map. This includes: obtaining the discrete visual keyframe preceding the start of the time-missing segment as the static base map; using an Alpha blending algorithm to overlay the dynamic saliency heatmap onto the static base map; setting a global attenuation coefficient for the heatmap layer, which is inversely proportional to the duration of the time-missing segment; and reducing the overall opacity of the heatmap layer according to the global attenuation coefficient as the transition video frame sequence progresses, until the heatmap layer is completely transparent at the end of the time-missing segment.

[0081] Specifically, the cloud-based compositing engine reads the original JPEG image captured at the beginning of the time-gap segment as a static background. For the generated Nth frame heatmap, a standard alpha blending algorithm is used: the pixel values ​​of the heatmap are multiplied by a transparency factor, and the background image pixel values ​​are multiplied by the complement of the transparency factor to obtain the final pixel values. To prevent visual abruptness, a global attenuation factor is set; assuming the gap segment is 5 seconds, the heatmap maintains high visibility for the first 4 seconds; in the last second (just before switching to the next real-shot image), the global opacity decreases linearly over time, causing the heatmap to gradually dissipate and eventually become completely transparent. This processing results in the user seeing, in the final synthesized MP4 video stream: on a static garage background image, a red halo representing the sound of scraping moves along the bumper; as the sound disappears, the halo naturally fades out, and then the scene smoothly transitions to the next photo.

[0082] This invention solves the visual abruptness of heterogeneous data (static base map and dynamic heatmap) synthesis by using layer blending and dynamic transparency control technology. The alpha blending algorithm ensures that the heatmap does not completely obscure key details in the base map, while the global attenuation mechanism provides a smooth visual transition between discontinuous image frames, eliminating the discomfort of "frame skipping" during traditional low frame rate monitoring switching, making the synthesized video more visually smooth and natural.

[0083] The cross-modal audiovisual coordinate mapping and heatmap generation steps include a multi-source parallel processing mechanism: the cloud server uses a blind source separation algorithm or beamforming post-filtering technology to separate the audio stream segment into N independent sub-audio tracks, where N is the number of simultaneously detected sound sources; for each sub-audio track, its corresponding spatial azimuth angle and trajectory coordinates are calculated independently; a distinct heatmap color spectrum is assigned to each sub-audio track; within the time gap, N layers of dynamic saliency heatmaps are generated based on the amplitude characteristics and trajectory coordinates of each sub-audio track, and these N layers of heatmaps are superimposed and fused onto the same discrete visual keyframe to achieve parallel trajectory display of multiple targets.

[0084] Specifically, in a complex attempted theft, suspect A used a tool to pry open the lock on the left side of the vehicle, while suspect B acted as lookout on the right side, continuously knocking on the window to signal. After the mixed audio collected by the in-vehicle terminal was uploaded to the cloud, the cloud server initiated a rapid independent component analysis algorithm; successfully separating two independent audio tracks: track 1 contained high-frequency metallic prying sounds, and track 2 contained low-frequency glass-knocking sounds; parallel calculations revealed that the sound source of track 1 was located at 45° to the left front of the vehicle, and track 2 was located at exactly 120° to the right rear. The rendering engine then assigned a "warning red" color spectrum to track 1 and a "hint yellow" color spectrum to track 2. In the final synthesized video, the user can see a red heat trail vibrating at the left door, while a yellow heat spot flashes at the right window.

[0085] This invention overcomes the technical challenge of location confusion caused by multi-target aliasing in complex acoustic environments through blind source separation technology. Traditional solutions often calculate incorrect "average angles" or jump between different sound sources when facing multiple sound sources. However, this invention can clearly isolate the spatiotemporal behavior of each independent sound source and display them in different colors on the same screen, greatly improving the situational awareness and evidence analysis capabilities of the monitoring system under complex environmental interference.

[0086] The step of independently calculating the corresponding spatial azimuth angle of the sound source includes a masking localization step based on time-frequency sparsity: converting the audio stream segment to the short-time Fourier transform domain to generate a time-frequency spectrum of the mixed signal; for each time-frequency unit in the time-frequency spectrum, calculating the energy proportion of each separated sub-audio track in that unit; generating a binarized or soft-decision time-frequency masking matrix, classifying each time-frequency unit to the dominant sound source with the highest energy proportion; when calculating the spatial azimuth angle of the sound source of a specific sub-audio track, only selecting the data of those time-frequency units of that sub-audio track that are determined to be the dominant sound source to participate in the generalized cross-correlation calculation, and eliminating other interfered time-frequency units to improve the localization accuracy when there is overlapping sound.

[0087] Specifically, the suspect scraped the vehicle at a 45° angle to the left, while a car happened to be passing by at a 45° angle to the right, honking its horn. Due to the high energy of the horn, traditional full-band localization algorithms would be "biased" by the sound, causing the localization result of the scraping sound to be biased to the right or to jump erratically left and right. Analysis revealed that the scraping sound was mainly concentrated in the high-frequency range of 3kHz-8kHz, while the horn sound was mainly concentrated in the mid-frequency range of 500Hz-2kHz. Time-frequency masking was generated, utilizing only the "pure pixels" belonging to the scraping sound within the 3kHz-8kHz frequency band to calculate the location. Ultimately, despite the loud horn sound, the scraping sound source at a stable 45° angle to the left was located, and the generated trajectory was not interfered with by the horn sound from the right.

[0088] This invention utilizes the sparsity of sound signals in the time-frequency domain to solve the technical problem of positioning failure caused by strong interference sources masking weak target sound sources. By using only the purest data to calculate angles, it significantly improves the robustness of positioning in multi-source aliasing scenarios, ensuring accurate reconstruction of the minute trajectories of key sound sources even in noisy environments. Example 2

[0089] This embodiment provides a low-power vehicle monitoring system based on audio and image frame synthesis. The system adopts an edge-cloud collaborative architecture, in which the audio-visual acquisition module is deployed on the vehicle's low-power terminal, while the coordinate mapping module, heat map generation module, and video synthesis module are deployed on the cloud server.

[0090] As one embodiment of the present invention, refer to Figure 1 The flowchart of a low-power vehicle monitoring method based on audio and image frame synthesis is shown below. Figure 2 A schematic diagram of a low-power vehicle monitoring system based on audio and image frame synthesis, referring to... Figure 3 A schematic diagram illustrating the principle of cross-modal audiovisual coordinate mapping and heatmap generation.

[0091] The audio-visual acquisition module is deployed in the vehicle-mounted monitoring terminal. Its core hardware includes a linearly or circularly arranged microelectromechanical system (MEMS) microphone array and a low-power CMOS image sensor. Logically, the module keeps the audio acquisition circuit constantly open, converting analog signals to digital signals and continuously writing them into a pre-defined circular audio buffer. The module's built-in digital signal processor monitors the short-time energy value of the input sound wave in real time. When the detected energy value exceeds a preset trigger threshold (-20dBFS), an abnormal event is determined. At this point, the module performs two parallel operations: First, it sends a hard interrupt wake-up signal to the image signal processor, controlling it to skip the conventional H.264 / H.265 video encoding process and directly capture a discrete visual keyframe (e.g., JPEG format) without inter-frame predictive coding. Second, using the multi-channel audio data acquired by the microphone array, it calculates the time difference of the sound wave reaching each microphone using a generalized cross-correlation algorithm combined with a phase transform weighting function, thereby determining the spatial azimuth angle of the sound source relative to the vehicle. Finally, the module packages the captured audio stream segments containing information before and after the event, the calculated spatial azimuth of the sound source, and the captured discrete visual keyframes, and uploads them to the cloud via a 4G / 5G network.

[0092] The coordinate mapping module, deployed on a cloud server, receives uploaded data packets and processes time gaps (e.g., a 5-second blind spot) between adjacent discrete visual keyframes. This module pre-stores the intrinsic parameter matrix (including focal length, principal point, and distortion coefficients) and extrinsic parameter matrix (including the rotation and translation relationship of the microphone array relative to the camera) of the vehicle camera. In practice, the module first transforms the spatial azimuth of the sound source from the microphone coordinate system to the camera's optical center coordinate system based on the extrinsic parameter matrix; then, it uses the intrinsic parameter matrix to project the 3D line-of-sight vector onto the 2D image plane, generating the corresponding pixel coordinates. The module also features boundary detection: if the calculated pixel coordinates are within the image resolution range, they are marked as valid sound source trajectory coordinates; if they are outside the range, the intersection of the line-of-sight vector and the image boundary is calculated and marked as edge sound source trajectory coordinates, indicating the direction of sound sources outside the field of view.

[0093] The heatmap generation module, deployed on a cloud server, transforms abstract coordinates and sound features into concrete visual elements. It first performs time-frequency analysis on audio stream segments, extracting high-frequency instantaneous amplitude features, and then performs temporal interpolation (e.g., Kalman filtering) on ​​the discrete coordinate points output by the coordinate mapping module to generate a smooth sound source trajectory. Based on this, the module generates dynamic saliency heatmaps frame-by-frame within time-gap segments. The rendering logic employs a two-dimensional Gaussian distribution model: the geometric center of the Gaussian distribution is anchored to the current sound source trajectory coordinates; a mapping function is established to positively correlate the instantaneous amplitude of the audio with the standard deviation of the Gaussian distribution (determining the spot size) and the alpha channel value (determining the spot brightness). Specifically, when the coordinates are in an edge-attached state, the module only renders the semi-circular halo cut by the image boundary, creating an "edge breathing light" effect; when the coordinates are within the frame, it renders a complete circular spot, thus generating a continuous heatmap sequence that moves with the sound rhythm.

[0094] The video compositing module, deployed on a cloud server, is responsible for the final video stream reconstruction. It reads the discrete visual keyframe at the beginning of a time-gap segment as a static background image. For each frame of dynamic heatmap output by the heatmap generation module, this module uses an alpha blending algorithm to semi-transparently overlay it onto the static background image. To ensure a natural visual transition, a global decay mechanism is introduced: as the time-gap segment progresses (e.g., from second 1 to second 5), the overall opacity of the heatmap layer is gradually reduced, allowing it to fade out naturally as it approaches the next captured keyframe. Finally, this module concatenates and encodes this series of frames overlaid with dynamic heatmaps with the original discrete visual keyframes to generate a standard MP4 video stream, allowing users to smoothly play back surveillance videos containing sound source trajectories on mobile devices.

[0095] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A low-power vehicle monitoring method based on audio and image frame synthesis, characterized in that, include: The system continuously collects environmental sound slices to generate audio stream segments and monitors sound wave signals. When the amplitude of the sound wave signal exceeds a preset trigger threshold, it collects discrete visual keyframes that have not been inter-frame predictive coding. It calculates the spatial azimuth angle of the sound source of the sound wave signal and uploads the audio stream segments, the spatial azimuth angle of the sound source, and the discrete visual keyframes to the cloud. For the time gap between adjacent discrete visual keyframes, the cloud performs cross-modal audiovisual coordinate mapping; based on the camera's intrinsic parameter matrix and installation position, the spatial azimuth of the sound source is projected onto the pixel coordinate system of the discrete visual keyframe to generate the sound source trajectory coordinates. Based on the coordinates of the sound source trajectory and the amplitude characteristics of the sound wave signal, a continuous dynamic saliency heatmap is generated within the time gap. The position of the highlighted area in the dynamic saliency heatmap moves with the coordinates of the sound source trajectory. The dynamic saliency heatmap sequence is overlaid and fused frame by frame with the discrete visual keyframes collected at the trigger time as the static background map to generate a transition video frame sequence that fills the time gaps. This sequence is then concatenated with the discrete visual keyframes to synthesize a monitoring video stream.

2. The low-power vehicle monitoring method based on audio and image frame synthesis according to claim 1, characterized in that, The system continuously collects ambient sound slices to generate audio stream segments and monitors sound wave signals. Specifically, the vehicle-mounted monitoring terminal keeps the microphone array always on in low-power standby mode, converts the collected analog sound wave signals into digital signals and writes them into a circular audio buffer, and calculates the short-time energy value of the sound wave signals in real time. When the short-time energy value exceeds a preset silence threshold, it is determined that an audio stream segment needs to be generated, and the trigger moment is locked as a time anchor point. Data from a preset time period before and after the time anchor point is extracted from the circular audio buffer and spliced ​​together to generate the audio stream segment, and marked with a unified Coordinated Universal Time (UTC) timestamp.

3. The low-power vehicle monitoring method based on audio and image frame synthesis according to claim 1, characterized in that, When the amplitude of the acoustic signal exceeds a preset trigger threshold, discrete visual keyframes without inter-frame predictive coding are acquired, including: when the amplitude of the acoustic signal exceeds the preset trigger threshold, a wake-up command is sent to the image signal processor; the image signal processor directly generates single-frame RAW format raw data as discrete visual keyframes; after completing one acquisition, a forced cooling timer is started, and before the forced cooling timer expires, subsequent wake-up commands are blocked, and only audio data and sound source location information are recorded.

4. The low-power vehicle monitoring method based on audio and image frame synthesis according to claim 1, characterized in that, The method for calculating the spatial azimuth angle of a sound source using a microphone array includes: framing and windowing the microphone array signal; calculating the time difference between microphone units using a generalized cross-correlation algorithm; constructing a geometric trigonometric relationship model based on the time difference, the physical spacing between microphone units in the microphone array, and the sound speed constant in air; calculating the incident angle of the sound source relative to the normal of the microphone array based on the geometric trigonometric relationship model, which is used as the spatial azimuth angle of the sound source; and if the calculated incident angle has multiple solutions and ambiguity, introducing signal data from redundant microphone units for data verification.

5. The low-power vehicle monitoring method based on audio and image frame synthesis according to claim 1, characterized in that, Projecting the spatial azimuth angle of the sound source onto the pixel coordinate system of the discrete visual keyframe includes: retrieving a pre-calibrated extrinsic parameter matrix, which represents the rigid body rotation and translation transformation relationship of the microphone array coordinate system relative to the camera optical center coordinate system; constructing a three-dimensional unit direction vector in the microphone array coordinate system based on the spatial azimuth angle of the sound source; performing a coordinate space transformation on the three-dimensional unit direction vector using the extrinsic parameter matrix to generate a line-of-sight vector in the camera optical center coordinate system; retrieving the camera's intrinsic parameter matrix, which includes focal length parameters, principal point coordinate parameters, and radial distortion coefficients; constructing a three-dimensional sound source coordinate system by combining a preset sound source depth reference surface; projecting the three-dimensional sound source coordinate system onto the camera's two-dimensional imaging plane using the intrinsic parameter matrix, and calculating the corresponding two-dimensional pixel coordinate points; determining whether the two-dimensional pixel coordinate points are within the resolution boundary range of the discrete visual keyframe; if within the range, marking the original pixel coordinate points as valid projection anchor points; if outside the range, calculating the geometric intersection of the three-dimensional line-of-sight vector and the image boundary line, and marking it as an edge projection anchor point.

6. The low-power vehicle monitoring method based on audio and image frame synthesis according to claim 1, characterized in that, The process of generating sound source trajectory coordinates that change continuously over time includes: performing time-frequency analysis on audio stream segments in the cloud, extracting instantaneous sound source location data at multiple discrete time points using a preset sampling rate, and generating multiple corresponding discrete projection anchor points; using a Kalman filter algorithm, performing time-series fitting with the discrete projection anchor points as control points to generate a smooth, continuous two-dimensional trajectory; for each transition video frame to be generated within the time gap, linearly indexing the corresponding coordinate position on the continuous two-dimensional trajectory according to time, and defining it as the sound source trajectory coordinates; when the sound source trajectory coordinates originate from the fitting path of the edge projection anchor points, they are marked as edge-attached.

7. The low-power vehicle monitoring method based on audio and image frame synthesis according to claim 1, characterized in that, Based on the sound source trajectory coordinates and the amplitude characteristics of the sound wave signal, a continuous dynamic saliency heatmap is generated within a time-deficient segment. Specifically, this includes: determining the duration of the time-deficient segment and calculating the total number of transition video frames to be generated within the time-deficient segment according to a preset video output frame rate; establishing a time index sequence synchronized with the transition video frame sequence; for each time point in the time index sequence, extracting the instantaneous amplitude characteristics of the corresponding moment from the audio stream segment and indexing the sound source trajectory coordinates of the corresponding moment from the continuous two-dimensional trajectory; using the extracted instantaneous amplitude characteristics and the sound source trajectory coordinates as input parameters, rendering the corresponding heatmap image frame by frame, and combining them into a continuous dynamic saliency heatmap sequence according to time sequence.

8. The low-power vehicle monitoring method based on audio and image frame synthesis according to claim 1, characterized in that, The position of the highlighted area in the dynamic saliency heatmap moves with the coordinates of the sound source trajectory, including: constructing a two-dimensional Gaussian distribution rendering model and anchoring the geometric center to the coordinates of the sound source trajectory; if the sound source trajectory coordinates are in an edge-attached state, locking the center of the two-dimensional Gaussian distribution pattern on the image boundary, and only rendering the breathing halo of the semi-circular area cut by the image boundary; if the sound source trajectory coordinates are not in an edge-attached state, rendering the complete two-dimensional Gaussian distribution pattern; establishing a first mapping relationship between the amplitude of the sound wave signal and the standard deviation of the Gaussian distribution, and a second mapping relationship between the amplitude of the sound wave signal and the peak opacity of the Gaussian distribution; and adjusting the parameters of the two-dimensional Gaussian distribution in real time according to the first and second mapping relationships to generate the dynamic saliency heatmap.

9. The low-power vehicle monitoring method based on audio and image frame synthesis according to claim 1, characterized in that, The dynamic saliency heatmap sequence is overlaid and fused frame by frame using discrete visual keyframes acquired at the trigger time as a static background base map. This includes: obtaining the discrete visual keyframe preceding the start of the time-gap segment as the static base map; using an Alpha blending algorithm to overlay the dynamic saliency heatmap onto the static base map; setting a global attenuation coefficient for the heatmap layer, which is inversely proportional to the duration of the time-gap segment; and reducing the overall opacity of the heatmap layer according to the global attenuation coefficient as the transition video frame sequence progresses, until the heatmap layer is completely transparent at the end of the time-gap segment.

10. A low-power vehicle monitoring system based on audio and image frame synthesis, characterized in that, include: Audio-visual acquisition module: continuously acquires ambient sound slices to generate audio stream segments, and monitors sound wave signals. When the amplitude of the sound wave signal exceeds a preset trigger threshold, it acquires discrete visual keyframes that have not been inter-frame predictive coding; it uses a microphone array to calculate the spatial azimuth angle of the sound source of the sound wave signal, and associates, packages, and uploads the audio stream segments, spatial azimuth angle of the sound source, and discrete visual keyframes to the cloud. Coordinate mapping module: The cloud performs cross-modal audiovisual coordinate mapping for the time gap between adjacent discrete visual keyframes; based on the camera's intrinsic parameter matrix and installation position, the spatial azimuth of the sound source is projected onto the pixel coordinate system of the discrete visual keyframe to generate the sound source trajectory coordinates; Heatmap generation module: Based on the sound source trajectory coordinates and the amplitude characteristics of the sound wave signal, it generates a continuous dynamic saliency heatmap within the time gap. The position of the highlighted area in the dynamic saliency heatmap moves with the sound source trajectory coordinates. Video synthesis module: The dynamic saliency heatmap sequence is overlaid and fused frame by frame with discrete visual keyframes collected at the trigger time as static background base map to generate a transition video frame sequence that fills the time missing segment, and is concatenated with discrete visual keyframes to synthesize a monitoring video stream.

Images