Vehicle-mounted DVR abnormality detection and video protection method and system based on hardware and software cooperation
Patent Information
- Application Number
- CN202611192852.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-07
- Publication Date
- 2026-10-16
AI Technical Summary
[0004]（1）无法区分减速带和真实碰撞
[0131] Innovation Point 1: Task scheduling and hard real-time guarantee architecture based on heterogeneous dual-core SoC
Smart Images
Figure CN122929294A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of automotive electronics, in-vehicle video surveillance, and embedded heterogeneous computing technology. Specifically, it relates to a technical solution for detecting abnormal events and recording videos in in-vehicle video surveillance (including streaming rearview mirrors, 360-degree panoramic images, and dashcams) by using a heterogeneous dual-core chip (ARM application core + RISC-V real-time control core) and a three-stage cascaded discrimination topology of three types of features in the time domain, attitude domain, and frequency domain. Background Technology
[0002] Existing in-vehicle dashcams, integrated streaming rearview mirrors, and 360° surround view systems serve as crucial evidence-gathering and safety aids in vehicle accidents. Their exceptional detection accuracy and reliable video preservation directly determine the validity of driving evidence. Current in-vehicle dashcams typically suffer from the following problems and shortcomings:
[0003] Existing triaxial acceleration schemes typically only calculate the composite triaxial acceleration value, triggering event recording lock when it exceeds a fixed threshold. This scheme has the following drawbacks.
[0004] (1) Inability to distinguish between speed bumps and real collisions. When a vehicle passes over a speed bump, it will generate a large vertical impact. The composite acceleration value may exceed the collision detection threshold, causing the system to misjudge it as a collision event and lock it. At the same time, if speed bumps are triggered frequently, a large number of invalid event recordings will quickly fill the SD card storage space, causing real collision event recordings to be unable to be saved or to be forced to overwrite earlier valid data.
[0005] (2) Missed minor collisions or collisions at special angles. The peak acceleration of some scrapes and low-speed side collisions may be lower than the threshold, leading to missed detection. Increasing the sensitivity will increase false alarms.
[0006] The reason why the industry currently adopts the three-axis acceleration synthesis value scheme is that this design has low hardware requirements (only MCU is needed), simple calculation, and fully meets the requirements of real-time performance. However, due to limited computing power, it is also impossible to further establish an anomaly confidence level system, and it is impossible to distinguish between "definite anomaly", "suspected anomaly" and "normal interference", resulting in a one-size-fits-all approach to video preservation and low utilization of storage resources.
[0007] The reason for not using a SoC solution is that the real-time performance of an ARM main processor with an operating system is poor. If an anomaly is detected, it is often insufficient to immediately switch to emergency recording and storage mode, resulting in the failure to properly store critical video clips. Furthermore, the ARM main processor itself has a high load when handling video capture, encoding, audio / video synchronization, and storage writing. Performing multi-factor anomaly analysis under these conditions can easily lead to abnormal contention, affecting video stability.
[0008] While the MCU+SoC dual-chip separation solution has solved the above problems to some extent, the dual-chip architecture leads to an increase in PCB area and hardware BOM cost, and introduces new failure points such as inter-chip communication delay and additional power consumption.
[0009] In response to the aforementioned drawbacks, the industry urgently needs a solution that can ensure sufficient computing power to complete real-time processing of multi-dimensional features and perform video processing-related operations without increasing material costs, thereby solving industry problems such as false alarms, missed alarms, and storage waste in vehicle-mounted DVRs. Summary of the Invention
[0010] This invention aims to solve the problems of the prior art mentioned above. It proposes a method for anomaly detection and video preservation in vehicle-mounted DVRs based on hardware and software coordination. The technical solution of this invention is as follows:
[0011] A method for anomaly detection and video preservation in an in-vehicle DVR based on hardware and software coordination, the method running on a heterogeneous SoC architecture, the heterogeneous SoC including an ARM application core and a RISC-V real-time core, the method comprising the following steps:
[0012] S1: Hardware Initialization: The IMU device is mounted onto the RISC-V real-time core of the heterogeneous SoC via the IIC protocol. The controller is triggered by an interrupt to synchronously acquire triaxial acceleration and triaxial gyroscope signals. A hardware double-buffering mechanism is used to construct the original sampling sequence within the dynamic time window. S2: First-Level Decision Mode: Calculate the composite triaxial acceleration value. and peak .when , When the threshold T is exceeded, a Level 1 response is immediately triggered, and if an active time window exists at this time, the response is reset; when , If the threshold for triggering the second level is exceeded but the threshold for triggering the first level is not exceeded, proceed to S3;
[0013] S3: Dynamic Time Window Management: After activating the dynamic time window, the minimum observation period is 200ms. A "watchdog-feeding" mechanism is used to extend the window. Whenever synthetic acceleration is detected... When the extension threshold T is exceeded, the window will be extended by 20ms; the window termination conditions include triggering the first-level judgment, the signal being below the threshold for 50ms for normal closure, and the window terminating after a cumulative timeout of more than 500ms.
[0014] S4: After the window is closed, calculate the composite acceleration sequentially from the sampled data within the window and the preceding data back 100ms. Effective impact event count (Nimpact), low-frequency energy concentration based on Goertzel algorithm Angular velocity composite value When calculating the frequency domain features, Hanning window preprocessing is applied to the window to eliminate spectral leakage. Then, the Goertzel algorithm is used to calculate the preset low frequency points {5Hz, 11Hz, 13Hz, 16Hz}. In the above algorithm process, RISC-V RVV vector instructions are fully utilized for parallel operation.
[0015] S5: First, obtain the weight matrix based on historical data, and after normalizing the extracted feature vectors, input them into the PL-SoftMax classifier. After RISC-V RVV vector matrix multiplication, obtain the probability distribution of four types of events: NORMAL, SPEEDBUMP, COLLISION, and ATTITUDE. Take the event with the highest probability as the classification result, and its probability value as the confidence level.
[0016] S6: Based on the received event classification and confidence level, the data is packaged into packets according to a predetermined format and sent to the ARM application core via the RPMsg asynchronous message mechanism between RISC-V and the ARM core. S7: In the ARM application core, a differentiated storage strategy is executed based on the received event type and confidence level. To ensure timely response, a circular buffer based on frames needs to be created in the ARM application core program, and a file handle (fd) opened with the O_DIRECT flag needs to be created in advance. When an abnormal event is received, the current handle is immediately closed, and the data in the current circular buffer is traced back, writing the current abnormal information (event type, confidence level, timestamp, etc.) into the H.264 SEI field.
[0017] Hardware connection and initialization settings:
[0018] The inertial sensor is connected to the RISC-V controller via the IIC bus to maintain a fixed sampling period. The sensor register is continuously polled. Two equal-sized buffers, denoted as buffer A and buffer B, are used in on-chip memory. These buffers alternately handle data writing and processing, forming a ping-pong buffer structure. The buffer size is:
[0019]
[0020] in The maximum value of the dynamic time window is 500ms. The backtracking time is 100ms. For sampling period and =10ms. Therefore The maximum value is 60.
[0021] Level 1 Judgment Mode:
[0022] The S2 level-one determination mechanism is as follows:
[0023] Continuously calculate the combined values of the three-axis linear accelerations:
[0024]
[0025] Combined triaxial angular velocity values:
[0026]
[0027] set up Default threshold (medium): T = 4 × For Level 1 judgment, users can set it to the low setting of 3.5×. High-end 4.5× . The threshold T = 300° / s. Because exceeding the angular velocity limit increases the risk of rollover. Not accessible to users and cannot be adjusted independently. or If the value exceeds the threshold, the secondary judgment mode is not entered. If a dynamic time window has already been activated, the time window is closed.
[0028] Dynamic Time Window Management
[0029] Once the secondary judgment mode is entered, the dynamic time window management mechanism specifically involves recording the current trigger time. To start a dynamic window, perform the following initialization steps: Set the earliest time for the scheduled window to end: The minimum step extension Simultaneously, a timer will be used to record when the synthetic acceleration is continuously below the extension threshold. The initial value is set to 0.
[0030] If a detection occurs during the window's duration... If the first-level judgment is entered, the current window will be forcibly truncated, and an exception message will be immediately sent to the application core.
[0031] During the normal window holding period, at each sampling time:
[0032] ·like Then update At the same time, the continuous timer Reset to zero.
[0033] ·otherwise , .
[0034] Window termination conditions (window can be terminated if any one of the following conditions is met):
[0035] • Condition 1 (Silent Termination): Current Time And the duration of consecutive low values That is, 5 consecutive sampling points ≤ T means the window can be closed.
[0036] • Condition 2 (Terminates upon timeout): Total window length This means that the system will be forcibly terminated after a timeout.
[0037] After the window terminates, it is recorded as At this time, the actual window length If at this time < Then, the window data is filled to make its length reach... ; like Otherwise, keep the original window data unchanged.
[0038] Claim 5: Temporal Feature Extraction
[0039] The method according to claim 1, the dynamic time window data according to claim 3, the sampling period =10ms, the total number of sampling points within the window is:
[0040]
[0041] Where L is the dynamic window size. Indicates rounding down, sampling point index .
[0042] The temporal feature extraction specifically involves:
[0043] • Time-domain characteristics: Over-threshold impact counting
[0044]
[0045] in The calculation is the same as shown in claim 3. The impact threshold for the secondary judgment under the medium-range condition is an empirical value. Although users can set high, medium, and low levels for the primary judgment, the threshold here remains unchanged and will be uniformly compensated for in subsequent steps.
[0046] 1[·] is an indicator function that takes the value 1 when the condition inside the parentheses is true, and takes the value 0 otherwise. In the time domain, it reflects the persistence of high-intensity impacts within a window; collision events typically... Smaller (single, brief impact), but bumpy road surface Significant (multiple consecutive impacts).
[0047] Frequency domain feature extraction
[0048] The frequency domain features are calculated using the Goertzel algorithm to determine the target frequency point. The signal power spectral density, where the input signal is a composite value of linear acceleration. The reason for using frequency domain analysis is that it is not possible to further distinguish between minor collisions and speed bumps in the time domain, while it is easier to distinguish them in the frequency domain.
[0049] • Frequency domain characteristics: Hanning window preprocessing
[0050] Because dynamic time windows do not capture complete data at different frequency points, a Hanning window is needed for preprocessing to address spectral leakage and non-integer period issues caused by the dynamic window length. The formula is as follows:
[0051]
[0052] Since the maximum number of sampled data points supported is 60, the Hanning window coefficient N=64 can be used.
[0053] • Frequency domain characteristics: power spectrum
[0054] Because the natural frequency of a speed bump is typically in the low-frequency energy band (10Hz-15Hz due to wheel movement), while a minor collision is a direct impact, approximating an impact function, its spectrum exhibits a broadband, uniform energy distribution, extending from 0Hz to the Nyquist frequency (50Hz). This difference leads to the selection of the Goertzel algorithm to detect minor collision events in the frequency domain, ultimately based on the proportion of low-frequency energy. Distinguish between speed bumps and minor collisions.
[0055] Among them frequency points
[0056] For each frequency point Pre-calculated coefficients:
[0057]
[0058] According to Goertzel's recursive formula:
[0059]
[0060]
[0061]
[0062] in = 100 , For instantaneous model closure using triaxial acceleration, the initial value is... , Based on the above analysis, we can conclude that:
[0063] Power spectral density:
[0064] * +
[0065] Low-frequency energy concentration
[0066] Total energy of window signal
[0067]
[0068] Input source
[0069]
[0070] Where ε is a minimal quantity to prevent the denominator from being zero. It reflects the proportion of low-frequency energy (5 / 11 / 13 / 16Hz) in the total energy of the entire frequency band, with a value range of [0,1].
[0071] Typical scenarios The eigenvalues are shown in the table below:
[0072]
[0073] These are the key frequency domain features that distinguish speed bumps / bumps (high values) from collision events (low values), along with their time domain features. To form complementarity: Distinguishing between the duration of the impact, Distinguish vibration frequency patterns.
[0074] Pose domain feature extraction:
[0075] Based on dynamic time window data, sampling period =10ms, the total number of sampling points within the window is:
[0076]
[0077] Where L is the dynamic window size. Indicates rounding down, sampling point index .
[0078] The specific method for extracting the attitude domain features is as follows:
[0079] Input source have to:
[0080]
[0081] Synthetic peak value obtained:
[0082]
[0083] Reflecting the drastic degree of vehicle attitude change is the core feature for detecting abnormal attitude events.
[0084] Claim 8: Feature Matrix and Scene Fusion Determination (PL-SoftMax)
[0085] Extract the above 4-dimensional original feature vector Normalization is performed:
[0086]
[0087] in For a very small amount, after normalization For features ∈ [0, 1], the reference values for the normalization parameters are as follows: compensate
[0088]
[0089] • Scene fusion determination (PL-SoftMax)
[0090] The probability of different discrete events can be obtained by using native SoftMax to perform logical judgments on each event, where:
[0091]
[0092] However, because the above calculation method is complex, computationally expensive, and unsuitable for embedded systems, this paper actually uses PL-SoftMax for simulation approximation. All satisfy... The weight matrix Bias matrix b These correspond to four types of events: Normal (normal driving), Speedbump (speed bumps, bumps), Collision (collision, scrapes), and Attitude (abnormal posture, rollover, fishtailing).
[0093] Before performing calculations using PL-SoftMax, the weight matrix W and bias matrix must be obtained through training on the collected test dataset.
[0094] b. Then, the system is ported to the embedded platform using the weight matrix W and the bias matrix b.
[0095] The probability of the i-th class in the sample, obtained using PL-SoftMax for the above four types of events, is as follows:
[0096]
[0097] in This represents the original score for the i-th class; The exponential function in Softmax Replace it with a ReLU function with an offset. Each offset of +2 means that even if the original score of a class is very low, its probability will not be 0, thus increasing the discriminative power; the coefficient T=0.75 is actually equivalent to... The overall scale was enlarged, and the differences between scores in different categories were adjusted.
[0098] Because cross-entropy directly optimizes the classification probability distribution, it is suitable for the output of PL-SoftMax; and because this paper only has 4 sets of features with clear physical meaning, sparsification is not required, so L2 regularization is adopted (L1 regularization will compress some weights to 0 to produce sparse solutions, which is suitable for higher-dimensional features).
[0099] Therefore, the model training formula is:
[0100]
[0101] Where n is the total number of samples, It is a one-hot label, that is, the true label of the i-th sample in the k-th class; the... In represents the probability that the i-th sample is predicted to be of class k; In() is the logarithmic penalty, the lower the probability, the heavier the penalty; for F-norm, Let be the norm of matrix b.
[0102] Then, a two-stage hybrid strategy is adopted: the first stage adopts a full-set random search strategy, and in order to improve computational efficiency, the sign and approximate range of the weights of each row of the initial constraint [W,b] matrix are shown in the table below: then the optimal parameters are found in 5000 iterations of sampling.
[0103]
[0104] The second stage uses the results of the first stage as the initial point and calculates the gradient using finite differences:
[0105]
[0106]
[0107]
[0108] Then with the learning rate After fine-tuning the loss 300 times to a value of 0.005, local convergence was achieved, yielding the final matrices W and b. The accuracy on the test set reached over 94%.
[0109] The weight matrix is as follows:
[0110]
[0111] The bias matrix is as follows:
[0112]
[0113] On the embedded platform, using the W and b obtained above, the four sets of parameters acquired in real time are... , , , Normalization is performed on the feature vectors. Then use a linear classifier Calculate the confidence level:
[0114] The final score is:
[0115]
[0116] The obtained score is then further shifted to zero (ReLU truncation).
[0117]
[0118] The confidence level of the current event is obtained after normalization.
[0119]
[0120] Event Classification and Confidence Level: The event type EVT and confidence level LEVEL rules in S5 are as follows:
[0121]
[0122] Based on the events obtained and their confidence levels, they can be further categorized as follows:
[0123]
[0124] The dual-core asynchronous communication is implemented based on the OpenAMP framework and uses the RPMsg protocol; specifically, it includes:
[0125] The ARM-side driver is responsible for receiving data to be sent by the application layer, writing the data into a message queue, and passing it to the RISC-V side through heterogeneous communication (the underlying layer is a virtual circular buffer);
[0126] The RISC-V side is responsible for interrupt handling, writing the processed message into a message queue, and passing it to the ARM through heterogeneous communication. The ARM then sends the data to the application layer, thus realizing asynchronous communication based on the OpenAMP framework.
[0127] The differentiated storage strategy in S7 includes: First, a circular buffer queue with frames is pre-created in the ARM application core, and an emergency file handle opened with the O_DIRECT flag is created.
[0128] When a high-confidence (LEVEL=2) collision or attitude anomaly event is received, the current ordinary file handle is immediately closed and switched to the emergency file handle. The event information is written back to the key video frame in the current buffer queue as SEI metadata. After fsync is executed to force disk write, the file is closed and a timer is started to continuously write the event information to the newly created video file for the next 10 seconds.
[0129] When a high-confidence (LEVEL=2) collision or attitude anomaly event is received, the priority is increased, and it has the ability to interrupt other medium-confidence and low-confidence timers. When a medium-confidence (LEVEL=1) event is received, the current ordinary file handle is immediately closed and reopened. The event information is written back to the key video frame in the current buffer queue as SEI metadata, and a timer is started to continuously write the event information to the newly created video file for the next 10 seconds. When a low-confidence (LEVEL=0) event is received, the normal segmented storage strategy is executed, and no special locking is performed.
[0130] The advantages and beneficial effects of this invention are as follows:
[0131] Innovation Point 1: Task scheduling and hard real-time guarantee architecture based on heterogeneous dual-core SoC
[0132] This invention proposes a dedicated architecture for vehicle-mounted DVRs based on a heterogeneous dual-core (RISC-V real-time core + ARM application core) SoC. Through functional decoupling and efficient inter-core communication (claim 9), task-level hardware and software collaboration is achieved. Specifically, time-sensitive tasks (IMU interrupt acquisition, sliding window threshold initial judgment, dynamic window control, feature classification processing) are deployed on the RISC-V real-time core, utilizing its deterministic interrupt response and Vector Extended Instruction Set (RVV) to ensure millisecond-level latency. Computationally intensive tasks with lower real-time requirements (H.264 / H.265 video encoding, cache queue management, file storage) are deployed on the ARM application core.
[0133] Technical Benefits: Compared to the "MCU+SoC" dual-chip solution, this invention reduces PCB area by approximately 30% and BOM cost by 15%-20% through single-chip heterogeneous integration; compared to a pure MCU solution, computing power is increased by tens of times; and compared to a pure SoC solution, it possesses millisecond-level low-latency sensing capabilities. This architecture is the first to establish a complete closed loop of "real-time sensing - accurate classification - differentiated storage" in the field of automotive DVRs, possessing significant engineering barriers.
[0134] Innovation Point 2: An anomaly feature extraction method based on the fusion of "time-domain dynamic window" and "frequency-domain fixed-point Goertzel".
[0135] To address the issue of traditional methods that truncate abnormal events (such as secondary impacts from collisions, tail-flipping, etc.) in the time and attitude domains, this invention, based on this fixed model, further proposes a strategy for dynamically extracting features from the time, frequency, and attitude domains in a coordinated manner, according to the actual situation. Specifically, the following mechanisms are employed in the time and frequency domains:
[0136] • Time domain level (claim 3): Design a "watchdog feeding" dynamic window extension mechanism - when the acceleration exceeds the start threshold, the window automatically extends the preset observation time to ensure complete capture of the main impact of the collision and subsequent aftershocks, and avoid feature distortion caused by event truncation.
[0137] • Frequency domain level (claims 4-6): Addressing the main energy concentration characteristics of low-frequency vibrations (5-20Hz) of the vehicle body / wheels in automotive scenarios, a novel Hanning window preprocessing technique is employed to suppress spectral leakage caused by aperiodic sampling. Subsequently, the Goertzel fixed-point algorithm is used to calculate the energy concentration index for only four characteristic frequency points (5Hz, 11Hz, 13Hz, 16Hz). This avoids the high computational cost of traditional FFT.
[0138] Technical effects: Among them, the time-frequency domain closed-loop complementary scheme utilizes time-domain impulse counting ( ) and frequency domain The combined evaluation of indicators successfully resolved the issue of misinterpreting speed bumps as real collisions. Actual testing showed an anomaly detection accuracy exceeding 94%, and the Goertzel algorithm's computational cost is only 1 / 10 that of FFT, making it suitable for the limited computing resources of RISC-V real-time cores.
[0139] Innovation Point 3: A lightweight 4D SoftMax classifier based on offline learning and RVV acceleration
[0140] To address the shortcomings of traditional fixed threshold methods (low accuracy, high false positive rate) and the uncontrollable inference latency of general machine learning models (such as fully connected neural networks) on embedded platforms, this invention designs a two-stage lightweight classification mechanism:
[0141] • Offline training phase (preprocessing): Based on a large sample dataset collected historically, an optimal 4×4 weight matrix is learned offline using random search and gradient descent methods. This matrix is then used to train the four-dimensional feature vector (peak acceleration). Impact counting Low-frequency energy concentration Peak angular velocity This is mapped to the collision probability output.
[0142] • Online inference stage (claims 5-8): The trained fixed-weight matrix is deployed on a RISC-V real-time core, requiring only one matrix multiplication and PL-SoftMax normalization operation during inference. Specifically, the RVV vector extension instruction of RISC-V is used to achieve a one-time parallel acceleration of the 4×4 matrix multiplication.
[0143] • Technical effect: Compared with traditional + The dual-threshold scheme reduces the false alarm rate by approximately 60%; compared to general neural network schemes, under limited computing power, the inference latency is reduced to the millisecond level and has complete determinism; compared to scalar loop computation, RVV acceleration improves matrix operation efficiency by more than 3 times, achieving a balance between computing power and performance.
[0144] Innovation Point 4: A hierarchical video preservation strategy based on O_DIRECT bypass caching and SEI metadata injection
[0145] To address the two major pain points of traditional DVRs—rapid storage depletion due to "one-size-fits-all" recording and loss of critical video due to delayed system page cache write-to-disk—this invention proposes a tiered storage and rapid disk write-to-disk mechanism:
[0146] • Emergency disk write mechanism: During system initialization, an emergency file handle with the O_DIRECT flag is pre-created to bypass the operating system's Page Cache and interact directly with the block device (eMMC / SD card). When an emergency signal is received or a high-confidence collision event is output by the classifier, fsync is immediately triggered to force synchronization, and the historical video (N seconds before the collision to M seconds after the collision) in the backtracking circular buffer is directly written to disk.
[0147] • Tiered storage strategy: Based on the confidence probability output by the classifier (e.g., low / medium / high levels), storage space and locking priority are allocated differently; O_DIRECT emergency disk write is only enabled when an emergency signal is triggered or a "high confidence" event is triggered, and low confidence events (e.g., speed bumps) are simply marked, effectively reducing the encroachment of invalid data on storage space.
[0148] • Evidence solidification: At the same time as the video is dropped, key evidence such as the collision time, confidence score, and IMU three-axis acceleration curve are embedded into the video stream through SEI (Supplemental Enhancement Information) metadata injection technology to ensure the traceability and legal validity of the video evidence.
[0149] • Technical benefits: The latency for critical video data transfer to disk has been drastically reduced from ~200ms in traditional solutions to ≤50ms, greatly reducing the risk of video loss during abnormal power outages or system crashes; the tiered strategy improves the effective utilization of storage space, while SEI metadata provides an immutable, judicial-grade chain of evidence for post-incident liability determination. Attached Figure Description
[0150] Figure 1 This diagram illustrates a dual-core communication architecture.
[0151] Figure 2 This diagram illustrates the RPMsg message mechanism.
[0152] Figure 3 This diagram illustrates the hierarchical processing architecture of the real-time kernel software.
[0153] Figure 4 This diagram illustrates the dynamic extension of the time window.
[0154] Figure 5 Frequency domain Feature extraction diagram;
[0155] Figure 6 This diagram illustrates the calculation of the weight matrix during the training phase.
[0156] Figure 7 A schematic diagram showing the statistical information of the weight matrix;
[0157] Figure 8 This diagram illustrates the accuracy of calculations using the current weight matrix.
[0158] Figure 9 This diagram illustrates the confusion that occurs when using the current weight matrix.
[0159] Figure 10 This represents the dual-core communication flowchart. Detailed Implementation
[0160] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.
[0161] The technical solution of the present invention to solve the above-mentioned technical problems is:
[0162] This invention employs a dual-core asynchronous system architecture chip, consisting of a RISC-V real-time core and an ARM application core. High-real-time tasks such as signal feature extraction and signal processing, which require RISC-V processing, are assigned to the RISC-V real-time core for execution, while I / O-intensive tasks such as video encoding and file management are handled by the ARM application core. The two cores communicate decoupled via an asynchronous RPMsg message mechanism. This design ensures real-time performance while fully utilizing the vector operation capabilities of RISC-V to process complex signals. It also avoids coupling anomaly detection tasks with video stream processing tasks and eliminates the additional communication latency and BOM cost associated with MCU+SoC dual-chip solutions.
[0163] This invention collects data from the MPU6050 sensor and sets up a two-level management system. The first level provides a rapid response to clearly defined information; the second level uses dynamic time windows instead of fixed time windows for data interception and analysis, enabling it to adapt to abnormal events of varying durations. The time window increments dynamically using a "watchdog timer" mechanism, with the window length determined by rules such as 100ms backtracking to capture the leading edge, a minimum of 200ms, and dynamic extension.
[0164] This invention takes current performance into full consideration, using Goertzel to replace FFT to reduce complexity from O(M×N) to O(N log N). For computationally intensive steps such as feature extraction, normalization, and matrix multiplication, the RISC-V Vector Extensions instruction set (RVV) is used for vectorization to accelerate the operation, enabling the originally complex calculations to run on embedded terminals and meeting the latency requirements of real-time vehicle detection.
[0165] In this invention, the impact force transmitted through the tires and suspension system is similar to the spring-damping characteristics, with its natural frequency typically in the low-frequency energy band (10Hz-15Hz for wheel bounce). Minor collisions, however, are direct impacts, approximating an impact function, and their spectrum exhibits a broadband, uniform energy distribution, extending from 0Hz to the Nyquist frequency (50Hz). This difference leads to the selection of the Goertzel algorithm for detecting minor collision events in the frequency domain. The algorithm utilizes a Hanning window for preprocessing to address spectral leakage and non-integer period issues caused by the dynamic window length. Finally, the low-frequency energy proportion is used to determine the optimal frequency response. Distinguish between speed bumps and minor collisions.
[0166] This invention uses PL-SoftMax multi-scene classification. By importing raw data, it obtains the optimal weight matrix based on a hybrid optimization strategy, and outputs confidence based on the fused time domain, frequency domain, and attitude domain features, supporting hierarchical responses.
[0167] This invention employs a three-tiered video preservation mechanism. Compared to the existing "one-size-fits-all" locking and recording method, it can differentiate the handling of storage resources based on the event's confidence level. Low-confidence events do not occupy locked storage space, effectively reducing the invalid locking and recording rate; medium-confidence events are effectively assisted in subsequent incident analysis by using SEI metadata in H.264 data; high-confidence events not only preserve SEI metadata but also ensure that critical evidence is not overwritten through fsync forced disk write and file protection mechanisms.
[0168] The core value of this invention lies in the organic combination of real-time performance (RISC-V bare metal), accuracy (multi-domain feature fusion), and efficiency (RVV acceleration), forming a complete vehicle collision detection solution.
[0169] 1. System Architecture Design
[0170] This solution adopts a heterogeneous SoC architecture, and the system runs on the same heterogeneous SoC chip containing an ARM application core (main processor) and a RISC-V real-time core (coprocessor).
[0171] The ARM application core runs the Linux operating system and executes the core business logic of the DVR, including video acquisition, encoding, audio and video synchronization, storage writing (SD card / eMMC), and network transmission. The RISC-V real-time core runs the bare-metal real-time program, adopting a "real-time perception-feature extraction-hierarchical decision-result transmission" approach. It is responsible for IMU data acquisition, dynamic time window management, multi-dimensional feature extraction, confidence level determination, etc., and reports the results to the ARM through the RPMsg asynchronous message mechanism.
[0172] The two cores communicate via RPMsg messaging to exchange control commands and event notifications, achieving loosely coupled collaborative operation without blocking each other. The overall system data flow is as follows: Figure 1 As shown:
[0173] 1.1 Hardware Interface Initialization
[0174] The three-axis accelerometer and three-axis gyroscope sensors use the MPU6050, which is connected to the TWI bus as a slave device with a bus address of 0x68. When in use, the hal_twi_init() function is first used to initialize the IIC interface and configure the clock, interrupts, pins, etc. When the program ends, the hal_twi_uninit() function is used to destroy it.
[0175] 1.2 Timer Interrupt Sampling Design
[0176] Configure the default parameters of MPU6050 via IIC, set the gyroscope range to ±500° / s via FS_SEL[1:0] in register 0x1B, and set the accelerometer range to ±8g via AFS_SEL[1:0] in register 0x1C.
[0177] During each interruption, the MPU6050 collects data in 14 bytes from 0x3B to 0x48. Among them, 0x3B, 0x3D, and 0x3F correspond to acceleration in the X, Y, and Z axes, respectively; 0x41 corresponds to temperature (not actually used); and 0x43, 0x45, and 0x47 correspond to angular velocities in the X, Y, and Z axes, respectively.
[0178] When an interrupt is triggered, 14 bytes of raw data are read continuously from the starting address of the 0x3B register through the hal_twi_read() function interface, written into the raw_buf[2]
[14] array, and the index is switched to point to another free buffer; the entire switch must be completed under interrupt protection to ensure state consistency.
[0179] 2. Software logic judgment
[0180] 2.1 Software Architecture Design
[0181] As shown, a two-stage judgment mode is adopted, where the first-stage judgment only considers the data obtained by the NPU6050, i.e., the data obtained by the NPU5060. , The quick determination is completed. If the window is enabled at this time, it needs to be reset. When entering the secondary determination mode, the dynamic window is opened. After the window is closed, the feature vector is calculated and extracted from the buffer for secondary matrix determination. Finally, the result is sent to the real-time kernel for processing via RPMsg.
[0182] Finally, based on the scene fusion matrix, four states are obtained: NORMAL (normal driving), SPEEDBUMP (speed bump), COLLISION (collision), and ATTITUDE (abnormal attitude).
[0183] 3. Time window design
[0184] 3.1 Circular Buffer Data Structure
[0185] Data acquisition uses a two-buffer design. Since only the MPU6050 data is located in the 14 bytes from 0x3B to 0x48, the first buffer is designed as raw_buf[2]
[14] . The reason for this design is that there is a risk that the interrupt program is reading data from IIC and writing it to the buffer, while the main loop happens to need to read data and copy it at the same time; it also avoids the delay caused by the main loop having to wait for IIC to finish reading before it can safely read data.
[0186] The second buffer is designed as a circular buffer. Since the maximum supported time window size is 500ms (50 sampling points), and considering the need to capture complete leading-edge data (e.g., the rising edge of the impact signal may begin before the trigger threshold), this redundant circular buffer design can capture the complete waveform and avoid losing low-frequency components (such as vehicle resonance). Furthermore, 10 sampling points (100ms) are sufficient to cover the typical initial acceleration phase of a collision, so the buffer size is designed to be 60 sampling points. The data structure is as follows:
[0187]
[0188] 3.2 Writing data to the circular buffer
[0189] After each interrupt, the data in the first-level buffer is written to the current position data[head] in the second-level buffer;
[0190] The head always points to the next data position. When the buffer is full, i.e., the current idx = (buffer->head - buffer->count + i + 60) % 60, the pointer automatically rotates according to the FIFO strategy, and the new data overwrites the old data.
[0191] If the window is already active ( , (If the threshold is not exceeded), the count of each new window data is increased by window_count++; and in order to capture complete data, the analysis is started by window_start_idx = (buffer->head - 10 + 60) %60, that is, the system backtracks 100ms.
[0192] 3.3 Window Extension Design
[0193] Once the advanced judgment mode is entered, it indicates a non-emergency situation. Even if abnormal posture, speed bumps, or minor collisions occur, they generally won't cause external changes leading to a power outage. Therefore, a long time window can be selected for analysis and judgment. Furthermore, because the required window time varies for different situations, this paper designs a dynamic time window. For example... Figure 3 As shown:
[0194] The start time t_start is fixed at t_trigger - 100ms, which means that the first 10 sample points after triggering are used as the physical starting point of the window.
[0195] Once triggered, the minimum observation period is set to 200ms. This is because if the selected sampling point is too small during subsequent frequency domain determination, the frequency domain will exhibit a "main lobe adhesion" phenomenon, resulting in a high calculation error.
[0196] The actual shutdown time t_actual_end is then determined according to the extended algorithm:
[0197] Find the maximum value of a and b: MAX(a,b) ((a) > (b) ? (a) : (b))
[0198] t_end_min[n+1] = (g_syn > T) ? MAX(t_end_min[n], t_now + delta_t_step) : t_end_min[n];
[0199] This employs a watchdog-like feeding mechanism, where the dog is fed whenever a valid signal is detected. When T(4g), regardless of whether the remaining window time is 0ms or 10ms, the window will be extended by 20ms (i.e., the number of sampling points will be increased by 2).
[0200] Using 20ms here, which is twice the sampling period, can better balance response sensitivity and noise immunity (e.g., at time n). < T(4g), but at time n+1 > T(4g)), the time window t_end_min is monotonically increasing, and the window length is: 200ms + (signal duration)*10ms + 20ms.
[0201] 3.4 Window Termination Design
[0202] The window termination conditions are divided into three modes: emergency termination, normal mode, and timeout protection exit.
[0203] In order to detect the signal and trigger it in time when a collision occurs, a method is adopted. >T(4g) immediately ends the current time window, without considering subsequent time window processing, and immediately triggers a signal to be sent to the application core;
[0204] To avoid misjudging the bottom signal of vehicle body damping oscillation (i.e., when a vehicle collides, it causes mechanical vibration of the vehicle body, and this vibration does not disappear instantly, but presents as a decaying oscillation), a 50ms silence judgment is adopted; that is, when the signal is lower than the threshold, the low signal duration t_low is accumulated. If t_low continuously reaches 50ms (5 sampling points), the impact process is considered to have truly ended.
[0205] To avoid the window becoming infinitely long when driving on bumpy roads for extended periods, 500ms is usually sufficient to cover real collision, speed bump, and other signals. After the timeout, the window is forcibly closed to prevent memory overflow.
[0206] 4. Feature Extraction and Analysis
[0207] After the window is closed, the valid data in the buffer is extracted in chronological order, and the feature extraction and analysis phase begins.
[0208] 4.1 Temporal Feature Extraction
[0209] Read the latest data from the first-level cache. , , Because at this point only a single-point operation is performed to determine whether to enter the advanced decision mode, RVV vector operation is not used; only [other operations are employed]. That's fine. However, once the decision is made in advanced mode, simultaneous sampling is required, and the RVV vector instruction set supports parallel processing rules, allowing for simultaneous loading and computation. , , Therefore, using RVV vectorization is the best approach, as shown in the following formula:
[0210]
[0211] It should be noted that the Allwinner RISC-V architecture used in this solution is the XuanTie C906, where VLEN = 128 bits; and the MPU6050 outputs 16-bit ADC sampled values, with a range of -32768 to 32767. Therefore, its element width SEW = 16. To balance performance and register usage, LMUL = 2 is used, meaning each vector register occupies 2 register groups for calculation. Actual number of elements processed: (VLEN / SEW) × LMUL = (128 / 16) × 2 = 16 elements.
[0212] The steps are as follows:
[0213] First, configure the vector length using the `vsetvli` command, which loads 16 groups of elements simultaneously. However, it should be noted that if there are fewer than 16 tail samples, the actual elements can be loaded as needed.
[0214] Use the vle16.v instruction to load the x-axis data (i,...,i+15) into registers v4-v5, and similarly load the y-axis data (i,...,i+15) into registers v6-v7, and the z-axis data (i,...,i+15) into registers v8-v9;
[0215] It's important to note that since the input samples and Hanning window coefficients are 16 bits, bit extension needs to be considered when multiplying. The `vwmul.vv` command should be used to calculate the square. , , That is, (int16 × int16 → int32). Where... Stored in v10-v13 Stored in v14-v17 Store up to v18-v21;
[0216] Then, vadd.vv is used to perform two addition operations to complete the three-axis summation, resulting in 16 sets. Value (int32)
[0217] Then use the vredmax.vs vector reduction command to reduce the vectors to 16 groups. Value reduction yields a scalar maximum value.
[0218] Then, repeat the above steps to calculate the remaining data in a loop until the calculation is complete.
[0219] 4.2 Temporal Feature Extraction
[0220] To calculate the overthreshold impact number
[0221] First, set the threshold using Equ: Then use vmv.vx to set the broadcast threshold;
[0222] Use vmsgt.vv to compare the current value with the threshold;
[0223] Use vpopc.m to count the number of items exceeding the threshold and store it in a general-purpose register, then use add to accumulate it into the total counter;
[0224] 4.3 Pose Domain Feature Extraction
[0225] The attitude domain steps and calculation steps are the same as in the advanced decision mode. No further analysis will be conducted this time.
[0226] 4.4 Frequency Domain Feature Extraction
[0227] Because the time-domain characteristics of a vehicle are sensitive to sensor noise and road vibration interference during operation, when the road surface is bumpy, some signals in the time domain may be affected.
[0228] The location could be a minor collision, or it could be going over a speed bump, etc., which cannot be further distinguished. Therefore, this paper uses the frequency domain for further analysis.
[0229] In general, frequency domain analysis is usually implemented by Fast Fourier Transform (FFT). FFT calculates N frequency points in the full frequency band with a complexity of O(NlogN), which requires high computing power for embedded terminals. Considering our actual situation, that is, when encountering speed bumps or bumpy road sections, the impact force is transmitted through the tires and suspension system, similar to the spring-damping characteristics. Its natural frequency is usually in the low-frequency energy band (wheel bounce 10Hz-15Hz, body vibration 5Hz-10Hz), while a slight collision is a direct impact, approximating the impact function. The spectrum shows a broadband energy uniform distribution, extending from 0Hz to the Nyquist frequency (50Hz), (sampling once every 10ms), or even higher. Therefore, Goertzel is used for calculation. If Goertzel is used to process M frequency points and N samples, the complexity is O(M*N). Its data flow is shown in Figure [5]:
[0230] 4.4.1 Frequency Domain Signal Preprocessing
[0231] Considering the automotive scenario, if only the Goertzel method is used for calculation... Then there is a possibility of spectrum leakage. For example, in a minor collision, the frequency domain occurs in the 30Hz-50Hz range. Without windowing, the sidelobes of the frequency domain decay relatively slowly, and the energy at 30Hz will overflow to the Goertzel frequencies of 5Hz, 11Hz, 13Hz, and 16Hz. This will cause the low-frequency energy PSD to be artificially high. As molecules grow larger, they may misinterpret a "collision" as a "speed bump".
[0232] Meanwhile, because the time window length is dynamic, the periods of the four frequency points 5Hz, 11Hz, 13Hz, and 16Hz are different. For example, when the window length is 250ms, the four frequency points 5Hz, 11Hz, 13Hz, and 16Hz are not complete cycles. The discontinuity of the beginning and end signals will cause frequency domain anomalies and generate extremely strong parasitic broadband interference.
[0233] Based on the above analysis, this paper first uses the Hanning window to preprocess the current data when calculating Goertzel, and the formula is as follows:
[0234]
[0235] Since the maximum number of sampled data points supported is 60, the Hanning window coefficient N=64 can be used.
[0236] The window function is implemented as follows:
[0237] First, the `vsetvli` directive configures the vector length, meaning it loads 16 groups of elements simultaneously.
[0238] Using vle16.v, the input sample data (gsyn[i..i+15] (16 int16 samples)) is loaded into v4-v5, and the Hanning window coefficients (w[i..i+15] (16 int16 coefficients, Q15 format)) are loaded into v6-v7 respectively;
[0239] The sample and window coefficients are calculated using vwmul.vv, and the data is stored in registers v8-v11 while taking into account bit extension.
[0240] Then, right-shift by 15 bits to restore precision and narrow it to 16 bits. The result is stored back into v4-v5 to obtain 16 weighted samples.
[0241] Repeat the above steps to calculate the remaining data in a loop until the calculation is complete.
[0242] 4.4.2 Total Energy of the Window
[0243] The formula for calculating the total energy of the window signal is as follows:
[0244]
[0245] Its main steps are:
[0246] First, the `vsetvli` directive configures the vector length, meaning it loads 16 groups of elements simultaneously.
[0247] Load the weighted (processed by window functions) data into v4-v5 using vle16.v;
[0248] The sample square value is calculated using vwmul.vv, taking into account bit extension, and the data is saved to registers v8-v11.
[0249] Using the vwredsum.vs bit-width extended reduction summation instruction, the 16 sets of data are summed and accumulated into the v0 register (scalar register LMUL=1).
[0250] Repeat the above steps to calculate the remaining data in a loop until the summation is complete.
[0251] 4.4.3 Goertzel Iteration and Calculation:
[0252] This paper calculates the goertzel, which is essentially an IIR filter, by recursively calculating the DFT coefficients at specific frequencies. Although a single frequency is calculated serially, multiple frequencies are independent, so RVV vector parallel computation is used. Unlike the above, because goertzel uses recursive state variables, there are multiplications and four frequencies (5Hz, 11Hz, 13Hz, and 16Hz) are processed at once; therefore, SEW=32 and LMUL=1 in this case.
[0253] The main steps are as follows:
[0254] Configure four int32 elements using vsetvli. Set register v4 to store the frequency coefficient, register v8 to store the state s1[n-1], and register v10 to store s2[n-1].
[0255] The lh function loads the sample values processed in the current window. Since four groups of samples are processed simultaneously, the values are broadcast to the vector register v0 via vmv.vx.
[0256] Calculate c_k*s1[n-1] using vsuml.vv and save it to register v12.
[0257] Use vadd.vv to calculate gsyn[n] + c_k × s1[n-1], and save it back to register v12;
[0258] Use vsub.vv to calculate s0[n] = gsyn[n] + c_k × s1[n-1] - s2[n-1], and save it back to the v12 register;
[0259] Update the state s2[n] = s1[n-1] using vmv.vv, that is, vmv.vv v10, v8
[0260] Update the state s1[n] = s0[n] using vmv.vv, i.e., vmv.vv v8, v12
[0261] Finally, using addi, move to the next sample and continue processing, repeating the above steps until the summation is complete.
[0262] 4.4.4 Low-frequency energy concentration calculate:
[0263] After calculating the total signal energy After calculating the power spectral density (PSD) at four frequency points (5Hz, 11Hz, 13Hz, and 16Hz), a for loop is used to calculate the sum and divide by... get ;
[0264] If (lfconc < 0), then lfconc = 0; if (lfconc > 1), then lfconc = 1.
[0265] 5. Scene fusion determination (PL-SoftMax)
[0266] Before using it on an embedded platform, the weight matrix W and bias matrix b must first be trained using the test set data. The main optimization steps are as follows:
[0267] Use NumPy to initialize and normalize the data.
[0268] The first stage involves randomly generating weights within the constraint interval using random search (random_search), then relying on _pl_softmax for forward computation, _loss for loss calculation, and _accuracy for accuracy verification, and iterating in a loop to select the optimal initial parameters [W,b].
[0269] The second stage uses gradient descent to calculate and update [W,b] along the negative gradient. Accuracy is calculated to verify the precision. This process is repeated iteratively until the optimal solution [W,b] is finally output. The main process is as follows:
[0270] The weight matrix is calculated as follows: Figure 7 As shown:
[0271]
[0272] Where the bias matrix is:
[0273]
[0274] The accuracy rate is shown in Figure [8], which is 94.7% on the test set and 90% on the boundary target. The anomalies mainly occur in the distinction between Normal and Speedbump.
[0275] The following calculations are required on the embedded side:
[0276] 5.2 Input Normalized Feature Vector Processing:
[0277] Because of the input data The range of values is not uniform. For ease of calculation, we first define the input four-dimensional data {g}. max N impact , Normalization is performed. This involves using four sets of for loops based on the formula. ,calculate
[0278] range = params[i].max_val - params[i].min_val ; normalized[i] = (raw[i] - params[i].min_val) / range;
[0279] Among them, targeting Parameters, using
[0280] if (normalized[i] < 0.0f) normalized[i] = 0.0f; if (normalized[i] >1.0f) normalized[i] = 1.0f;
[0281] To determine the range between [0,1].
[0282] 5.3 Matrix-vector multiplication and addition:
[0283] The matrix operator rules for merging row operations are as follows: Here, we choose to use RVV vector operations. The main steps are as follows:
[0284] This section uses `vsetvli` to configure four `float32` elements for floating-point numbers.
[0285] Use vle32.v to load matrix W into register v1 and matrix X into register v2;
[0286] Calculate v3 = v2 * v1 element by element and save it to the v3 register. Then use vfedsum,vs to sum the results to obtain a scalar. Use vfmv.fs to extract the scalar result to a floating-point register and save it.
[0287] After the entire row of clicks is calculated, continue to offset 2 bits with slli, use flw to load the b vector into the f2 register, and use fadd.s to add the vector obtained by multiplying the above vectors to the b vector;
[0288] Repeat the above steps to calculate the remaining data in a loop until the calculation is complete.
[0289] 5.4 Zero-point shift + ReLU + normalization:
[0290] First, find the minimum value by traversing the list, i.e., if (scores[i] < s_min) { s_min = scores[i]};
[0291] Then, the zero point is shifted by using scores[i] = scores[i] - s_min;
[0292] The truncation is achieved using the statement: if (scores[i] < 0.0f) { scores[i] = 0.0f;}.
[0293] Finally, the sum is calculated using a loop, and then the probability value for each scenario is obtained by looping again and calculating probs[i] = scores[i] / sum.
[0294] 5.4 Event Classification and Confidence Level:
[0295] The probability values corresponding to the four scenarios are obtained by iterating through them. The scenario with the highest probability is determined as the most likely scenario, and its confidence level is the current probability p value. That is, the initial probability value is set as max_prob = probs[0] and the corresponding index is set as max_idx = 0. Then, the loop iterates through the scenarios if (probs[i] > max_prob) {max_prob = probs[i]; max_idx = i}, and finds the specific event according to the index. Then, the corresponding LEVEL level is obtained according to the classification rules.
[0296] 6. Dual-core communication
[0297] 6.1 Dual-core communication steps:
[0298] This invention implements heterogeneous dual-core communication based on OpenAMP, with inter-core communication using the RPMsg communication protocol. The ARM application core is responsible for creating device nodes and enabling timestamp synchronization; the RISC-V real-time core sends scene fusion judgment data according to the actual situation, and its frame data structure is shown below:
[0299]
[0300] In ARM application cores:
[0301] First, the device node is initialized using the rpmsg_alloc_ept() function, and then the specific device file, such as / dev / rpmsg0, is opened using the open() function.
[0302] Data is sent from the ARM application core. The ARM-side Linux system writes frame data to the RPMsg device node through the write() function. When the ARM-side Linux system sends timestamp information, the valid data frame only contains the ID and timestamp. Then, an interrupt notification is sent to the RISC-V real-time core.
[0303] When receiving data from the ARM side, the poll mechanism and the polling read mechanism are used to listen to the device node and wait for the data to be sent back.
[0304] After the program ends, the ARM-side Linux system releases the communication node using the rpmsg_free_ept() function;
[0305] In RISC-V real-time cores:
[0306] First, the openAMP architecture is initialized using the openamp_init() function, and then rpmsg_ctrldev_create() is called to create an RPMsg control breakpoint to complete the link synchronization with the ARM real-time core;
[0307] Receiving data from the RISC-V real-time core: Bind the business endpoint through rpmsg_client_bind(), register three types of callback functions: binding, unbinding, and data reception, and wait for the ARM real-time core to initiate data communication after it is ready;
[0308] The RISC-V real-time core sends data via the rpmsg_send() function;
[0309] Once the program terminates, in the unbinding callback function, mark the communication as stopped and release memory resources.
[0310] The overall process is shown in the following figure
[10] :
[0311] 6.2 Dual-core communication time synchronization:
[0312] On the Linux side, `clock_gettime()` is used for timing. On the RISC-V side, the `hal_timer_init()` function is used to initialize the timer, with a single trigger precision of 1ms. The `hal_timer_uninit()` function is used to destroy the timer after the program ends. When the RISC-V enters the time window, the `hal_timer_start()` function starts the timer, and when the window ends, the `hal_timer_stop()` function stops the timer.
[0313] When the system starts up, a timestamp calibration is performed. The time when the message is initiated from the Linux side. The time it takes for the message to be received on the Linux side; (This is a calculation method.) The time when the message is received by the ARM terminal. The message initiation time for the ARM side. Because the heterogeneous architecture shares a crystal oscillator, the clock speeds of both cores are fixed and will not drift. The actual round-trip time (RTT) for both cores is = The video encoding process uses 25fps with a 40ms interval between frames. Since the RTT is much less than 40ms, this invention only requires a time reference determination once at system startup.
[0314] 7. Storage Solution Design
[0315] In this design, video encoding adopts a full I-frame design, and the encapsulation format uses MPEG-TS stream. This choice offers two key advantages: based on the PAT and PMT table structure of MPEG-TS, even in the event of a collision breakpoint, the written TS data packets can be independently parsed; and using I-frame encoding allows each frame to be decoded independently, eliminating the need for GOP alignment, and enabling SEI data to be inserted instantly before any frame. The main points include the following:
[0316] 7.1 Frame Buffer Circular Queue
[0317] Because the real-time detection window of the RISC-V real-time core processing terminal is a maximum of 600ms (including 100ms backtracking), and taking into account the memory usage of the ARM side, the design maintains the frame buffer as a NAL circular buffer queue encoded with 75 frames (i.e. 3 seconds of data), which ensures that real events are necessarily in the buffer.
[0318] 7.2 Recording File Management Mechanism
[0319] After system startup, two file handles are pre-created. A regular file handle `fd` is created using the `O_WRONLY|O_CREAT` method, and `YYYYMMDD_HHMMSS.ts` is opened as the output file. An emergency file handle `emergency_fd` is created using the `O_WRONLY|O_CREAT|O_DIRECT` method (direct write to disk, power-loss safe), and the file `emergency_YYYYMMDD_HHMMSS_evt.ts` is opened for later use. Here, `evt` indicates an emergency file and deletion is prohibited.
[0320] In normal mode (LEVEL=0), a 3-minute timer is started. Each frame output by the encoder is continuously pushed into the queue, while older video frames are written to the file. After 3 minutes, the current file descriptor (fd) is terminated, and a new regular file descriptor (fd) is created using the O_WRONLY|O_CREAT method. The file YYYYMMDD_HHMMSS.ts is then opened, and data writing continues. After the normal operation concludes, a 3-minute video recording file is obtained.
[0321] When the received event is in LEVEL=1 mode, the following steps are executed in sequence:
[0322] Immediately after calling fsync, close the normal file handle fd and change the file permissions to read-only;
[0323] Create a regular file handle fd using the O_WRONLY|O_CREAT method, and open YYYYMMDD_HHMMSS_warn.ts as the output file, where warn indicates a suspected warning file;
[0324] The scene fusion judgment frame data obtained by the real-time kernel is backtracked and written into the H.264 SEI information of the current buffer. The current ordinary queue is still used, and H.264 SEI information is written when each subsequent frame is encoded and output. Then, a 10-second timer is started, and each frame output by the encoder is continuously pushed into the queue and H.264 SEI information is written at the same time.
[0325] After 10 seconds, fsync will be executed, followed by close to shut down and return to normal mode.
[0326] After the normal process ends, you will get a 13-second video file (3 seconds of buffer + 10 seconds of newly recorded video).
[0327] When LEVEL=2 mode, i.e., the event is COLLSION or ATTITUTE and the confidence level prob≥0.7, the following steps are executed sequentially:
[0328] Immediately after calling fsync, close the normal file handle fd and change the file permissions to read-only;
[0329] If other events are being processed at this time, the timers for those other events are forcibly shut down. Then, the already created emergency file handle `emergency_fd` is immediately switched, and the scene fusion decision frame data obtained from the RISC-V real-time core is back-written to the SEI field of the H.264 NAL video frame in the current buffer queue. Then, the data is immediately written to the file, followed by `fsync`, and then `close` to close the buffer, while clearing the current buffer.
[0330] Start a timer. Do not process information from the RISC-V end for 10 seconds. Recreate emergency_fd and open emergency_YYYYMMDD_HHMMSS_evt.ts. At this time, the buffer adopts the priority queue first-in-first-out mode, that is, when a frame of data enters, the current data is immediately popped out. The scene fusion judgment frame data obtained by the real-time core will still be written. After 10 seconds, fsync is executed and then close to close the buffer, clear the current buffer, and restore the normal mode.
[0331] After the process is complete, you will get two video files: a 3-second video file and a 10-second video file.
[0332] It should be noted that the user information (including but not limited to user device information, personal user information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the laws, regulations and standards of relevant countries and regions.
[0333] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.
[0334] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0335] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.
Claims
1. A hardware-software co-processing method for anomaly detection and video preservation in an in-vehicle DVR, the method running on a heterogeneous SoC architecture integrating an ARM application core and a RISC-V real-time core, comprising the following steps: S1. Hardware initialization: The IMU device is mounted to the RISC-V real-time core via the IIC protocol. The three-axis accelerometer and three-axis gyroscope signals are synchronously acquired with the help of interrupts. The original sampling sequence of the dynamic time window is constructed using a hardware double buffering mechanism. S2, First-level rapid determination: Calculate the combined peak value of triaxial acceleration separately. Three-axis angular velocity composite peak ; If either exceeds the first-level threshold, a first-level response is immediately triggered and the active time window is reset; if only the second-level trigger threshold is exceeded, proceed to step S3. S3. Dynamic Time Window Management: Activate the dynamic time window with a minimum observation duration of 200ms. Employ a watchdog-style window extension strategy, extending the window by 20ms whenever the synthesized acceleration exceeds the extension threshold. Window termination conditions include three categories: Level 1 judgment trigger, signal below the threshold for 50ms consecutively, and timeout when the cumulative window duration exceeds 500ms. After the window ends, synchronously capture the data within the window and the preceding 100ms of backtracking data. S4. Multidimensional feature extraction: Extract four types of features from the truncated data in sequence: time domain, attitude domain, and frequency domain. Frequency domain processing first eliminates spectral leakage through Hanning window preprocessing, and then uses the Goertzel algorithm to replace FFT to calculate the power spectrum at low frequency points of 5Hz, 11Hz, 13Hz, and 16Hz. The entire feature computation process is accelerated in parallel by calling RISC-V RVV vector extension instructions; S5. PL-SoftMax Fusion Classification: The normalized four-dimensional feature vector is input into the pre-trained PL-SoftMax classifier, and the vector matrix operation outputs four types of events: normal driving, speed bump, collision, and abnormal posture, along with their corresponding confidence scores. The PL-SoftMax classifier uses the ReLU function to replace the original SoftMax exponential operation, and combines L2 regularization with a two-stage hybrid optimization training strategy to obtain fixed weights and bias matrices. S6, Inter-core asynchronous transmission: The event type, confidence level, and timestamp are encapsulated in a data packet, and the RPMsg asynchronous message mechanism based on the OpenAMP framework is sent from the RISC-V real-time core to the ARM application core; S7, Tiered Video Preservation: The ARM application core pre-allocates frame-level circular buffers and pre-creates O_DIRECT emergency file handles; it performs differentiated storage based on event type and confidence level, writes exception information to H.264 SEI metadata, and completes video evidence storage to disk with fsync forced flushing.
2. The method for anomaly detection and video preservation of in-vehicle DVR based on hardware and software collaboration as described in claim 1, characterized in that, The hardware double buffer in step S1 is a ping-pong buffer structure; IMU sampling period The sensor register is continuously polled, and two equal-sized buffers, denoted as buffer A and buffer B, are stored in on-chip memory. The buffer sizes are: ； in The maximum value of the dynamic time window is 500ms. The backtracking time is 100ms.
3. The method for anomaly detection and video preservation of in-vehicle DVR based on hardware and software collaboration as described in claim 1, characterized in that, Step S2: The first-level judgment threshold is configured as follows: set up Default threshold (medium): T = 4 × For Level 1 judgment, users can set it to the low setting of 3.5×. High-end 4.5× ; Threshold T = 300° / s; when or If the value exceeds the threshold, the secondary judgment mode will not be entered. If a dynamic time window has already been activated, the time window will be closed and the abnormal event will be output directly for a quick response.
4. The method for anomaly detection and video preservation of in-vehicle DVR based on hardware and software collaboration as described in claim 1, characterized in that, Step S3, the dynamic time window control logic, includes: Minimum end time of initialization window Minimum step extension The continuous low value timer is cleared; if the synthesized acceleration is higher than the extension threshold within the sampling period, the minimum end time of the window is updated and the continuous low value timer is reset; if it remains lower than the threshold, the continuous low value timer is incremented. Window termination criteria: Silent closure is triggered by a continuous low signal for 50ms, or timeout closure is triggered by a cumulative duration of 500ms; if the actual window duration after termination is less than 200ms, data is automatically filled in to make up to the minimum observation duration.
5. The method for anomaly detection and video preservation of in-vehicle DVR based on hardware and software collaboration as described in claim 1, characterized in that, The time-domain feature of step S4 is the impact count exceeding the 2g acceleration threshold within the window. An indicator function is used to count the number of impacts within a single window to distinguish between continuous turbulence and single-collision impacts; the attitude domain feature is the composite peak value of the three-axis angular velocities within the window. It is used to identify abnormal vehicle postures such as rollover and fishtailing.
6. The method for anomaly detection and video preservation of in-vehicle DVR based on hardware and software collaboration according to claim 1, characterized in that, Step S4: Frequency domain characteristics include low-frequency energy concentration. The calculation formula is the ratio of the total power spectrum of the four target frequency points to the total energy of the window acceleration; Speed bump scene >0.6, collision scenario <0.35, complementing the time-domain impact count, distinguishes two easily confused scenarios.
7. The method for anomaly detection and video preservation of in-vehicle DVR based on hardware and software collaboration as described in claim 1, characterized in that, The four-dimensional feature vector in step S5 is a quantity During feature normalization, an independent compensation coefficient is configured for the peak acceleration, while other features are constrained to the [0,1] range using a uniform normalization interval. The classification output confidence level is graded according to the following rules: LEVEL 2 high confidence anomaly: collision / attitude anomaly with a maximum probability ≥ 0.7; LEVEL 1 Suspected anomaly: Collision / attitude anomaly probability 0.4~0.7, or speed bump probability ≥0.7; LEVEL 0 Normal: All other scenarios.
8. The method for anomaly detection and video preservation of in-vehicle DVR based on hardware and software collaboration according to claim 1, characterized in that, Step S7, the tiered video preservation strategy, is as follows: 1) LEVEL 2 High Confidence Collision / Attitude Anomaly: Interrupts the executing low-level event timer, immediately closes the normal recording file, switches the O_DIRECT emergency handle, backtracks the video frames in the circular buffer, writes SEI metadata and forces it to disk, and continuously saves the emergency recording with the anomaly mark for the next 10 seconds. 2) LEVEL 1 suspected event: Close the current ordinary file, backtrack the cache to write SEI information, and start a 10-second special recording timer; 3) LEVEL 0 Normal driving: Performs regular segmented recording, without adding abnormal markers or starting a special latching process.
9. The method for anomaly detection and video preservation of in-vehicle DVR based on hardware and software collaboration according to claim 1, characterized in that, The heterogeneous dual-core RPMsg communication process is as follows: the ARM application core and the RISC-V real-time core register message queues and asynchronous callback interfaces respectively. After the real-time core completes feature classification, it encapsulates a meta data packet containing fragment ID, timestamp, event type, and confidence level, and pushes it to the ARM core without blocking through a virtual ring buffer channel. The two cores only perform clock synchronization calibration once upon power-on, and the round-trip communication delay is less than the interval between single video frames.
10. A hardware-software co-processing system for anomaly detection and video preservation in a vehicle-mounted DVR, characterized in that, To implement the method according to any one of claims 1-9, comprising: Heterogeneous SoC chip, with built-in ARM application core and RISC-V real-time core; The IMU inertial sensing module is connected to the RISC-V real-time core via the IIC bus, and outputs raw sampled data of triaxial acceleration and angular velocity. The real-time signal processing unit, deployed on the RISC-V real-time core, includes a ping-pong buffer module, a dynamic time window controller, an RVV vector acceleration unit, and a PL-SoftMax embedded inference module. The inter-core communication unit implements asynchronous message interaction between the two cores based on OpenAMP and the RPMsg protocol; The ARM video preservation unit has a built-in frame circular buffer, a dual-type file handle manager, an SEI metadata writing module, and an fsync forced disk write control module.