Intelligent data annotation system and method based on multi-modal large model

Abstract

The invention relates to the technical field of neural network application, and discloses an intelligent data annotation system and method based on a multi-modal large model, and the system comprises a pre-cognition triggering module and a multi-modal annotation module. The pre-cognition triggering module judges abnormal events based on physical causal mismatch by continuously monitoring the internal state of a vehicle and cross-domain baseline signals of the external environment, and generates an asynchronous interrupt signal carrying threat level information in combination with acoustic characteristics of impact audio so as to wake up the multi-mode labeling module which is in a low-power-consumption mode by default; according to the method, high-cost attention distribution is preposed into a low-cost pre-cognitive filtering mode based on physical causal mismatch, so that internal consumption of computing power generated by processing massive redundant information in conventional driving of the system is avoided, and multi-dimensional information such as acoustics and optics is utilized, so that the attention distribution is more accurate. Sensitive and reliable detection and labeling of tiny high-risk long-tail events and threat level discrimination are realized.

Description

Technical Field

[0001] This invention relates to an intelligent data annotation system based on a multimodal large model, belonging to the field of neural network application technology. Background Technology

[0002] Currently, in the development of intelligent systems such as autonomous driving and advanced driver assistance systems, perception and decision-making systems based on multimodal large models have become the mainstream technical path for improving environmental understanding. They typically follow a comprehensive perception-deep fusion-integrated decision-making approach, which involves collecting comprehensive information about the surrounding environment without discrimination through multiple sensors such as high-resolution cameras and LiDAR, and then running complex neural network models on a computing platform with powerful computing capabilities to perform deep feature extraction and multimodal information fusion on these massive amounts of data, ultimately generating cognitive results or data annotations about the environment.

[0003] However, as such systems become more widespread in mass-production applications with strict cost and power consumption constraints, a fundamental limitation inherent in the aforementioned operating method is becoming increasingly apparent: its continuous and intense reliance on high computing power resources leads to enormous energy consumption and hardware costs. This internal consumption of computing power stems from the fact that the system spends over 90% of its computing resources on analyzing routine background information or redundant dynamic elements that do not substantially contribute to the final decision, thus forming a key bottleneck restricting the universality of the technology. More importantly, this comprehensive perception operating method exposes its deep-seated logical flaws when dealing with specific challenges, especially when dealing with static, small, high-risk targets that are physically small, have inconspicuous optical features, and do not have obvious motion characteristics, such as metal fragments or irregularly shaped obstacles that suddenly appear on a highway. Existing systems, because their attention is diverted to more significant routine traffic elements, are prone to drowning out such key danger information with low signal strength in massive amounts of background data, thus constituting a silent risk in decision-making.

[0004] Specifically, existing technologies suffer from the following shortcomings: 1. The system's attention allocation method distributes high-cost computing resources evenly or based on simple saliency across all perceived information, lacking a low-cost filtering mechanism that pre-judges the value of information in the pre-cognitive stage, leading to severe misallocation of computing resources; 2. The detection of small, static, low signal-to-noise ratio long-tail dangerous events relies excessively on brute-force analysis with high computing power, lacking an effective way to sensitively perceive the existence of such unexpected physical entities from different physical dimensions; 3. The system continuously increases model complexity and computing power requirements in pursuit of high perception accuracy, which constitutes an irreconcilable inherent contradiction with the practical need for large-scale, low-cost, and low-power deployment at the edge. Therefore, how to construct a novel neural network system operation mechanism that can reduce conventional computing power consumption while maintaining or even improving the detection capability of small, high-risk long-tail events, thereby solving the cost and reliability bottlenecks of high-order intelligent perception in universal applications, becomes the technical problem to be solved by this invention. Summary of the Invention

[0005] This invention provides an intelligent data annotation system based on a multimodal large model. Its main purpose is to solve the problems of existing intelligent perception systems, which, due to their comprehensive perception operation, suffer from serious waste of computing resources in normal scenarios, and lack a low-cost and reliable detection mechanism for small, high-risk long-tail events.

[0006] To achieve the above objectives, this invention provides an intelligent data annotation system based on a multimodal large model, the system comprising:

[0007] The first sensing and monitoring unit is configured to continuously monitor the internal state baseline signal that characterizes the internal motion state of the vehicle itself.

[0008] The second sensing and monitoring unit is configured to continuously monitor the external environment baseline signal that characterizes the physical state of the external environment in front of the vehicle;

[0009] An audio capture unit; an asynchronous mismatch triggering module, connected to a first sensing monitoring unit, a second sensing monitoring unit, and the audio capture unit, is configured to: continuously compare the real-time change rates of the two baseline signals based on a preset time-series relationship model reflecting the normal physical causal relationship between the internal state baseline signal and the external environment baseline signal, to determine whether a mismatch event violating the time-series relationship model has occurred; when a mismatch event is determined to have occurred, a high-priority interrupt signal is generated, and the audio capture unit is instructed to synchronously capture the impact sound associated with the mismatch event. The system includes a frequency signal and a spectrum analysis of the impact audio signal to extract acoustic feature parameters characterizing the material properties of the impact audio signal. These acoustic feature parameters are then appended to a high-priority interrupt signal. A multimodal annotation module, which is in low-power mode by default, is configured to be activated only when a high-priority interrupt signal is received. Based on the acoustic feature parameters, it pre-determines the threat level of the mismatch event, retrieves multimodal sensor data associated with the timestamp of the mismatch event, and annotates the physical spatial region related to the mismatch event.

[0010] Preferably, the system also includes a matrix light source composed of multiple independently controllable units; after being activated, the multimodal annotation module is also configured to control the matrix light source to instantaneously project a preset structured light pattern onto the physical space region related to the mismatch event, capture the object image illuminated by the structured light pattern through a visual sensor, and enhance the annotation of the object's shape based on the distortion features presented by the structured light pattern on the object image.

[0011] Preferably, the asynchronous mismatch triggering module is further configured to determine whether the vehicle is in a continuous high-entropy driving condition based on the temporal stability characteristics of the internal state baseline signal or the external environment baseline signal within a preset time window; and when it is determined that the vehicle is in a continuous high-entropy driving condition, the triggering logic for generating a high-priority interrupt signal is temporarily adjusted to suppress or reduce false triggering caused by continuous fluctuations in the baseline signal under this condition.

[0012] Preferably, the second sensing and monitoring unit is further configured to, after the asynchronous mismatch triggering module determines that a mismatch event has occurred, respond to an instruction to send an active detection pulse to the tire sidewall of a specific wheel associated with the mismatch event, and receive the echo signal returned from the tire sidewall; the asynchronous mismatch triggering module is further configured to determine whether the tire is in a fault state based on one or more physical characteristic parameters of the echo signal, and perform attribution analysis on the mismatch event accordingly, so as to attach the attribution result to the high-priority interrupt signal.

[0013] Preferably, the asynchronous mismatch triggering module is further configured to extract parameters characterizing the current vehicle speed based on the internal state baseline signal. And based on speed parameters Dynamically adjust the time parameters in the time series model. Adjustments are made according to preset response time parameters. With speed parameters Calibration rules for the inverse proportional relationship between them.

[0014] Preferably, the calibration rules are defined as follows: ,in, To preset the effective physical distance between the detection point of the second sensing monitoring unit and the wheel contact point, For speed parameters.

[0015] Preferably, the system further includes: a verification system independent of the first sensing monitoring unit, the second sensing monitoring unit, the audio capture unit, and the asynchronous mismatch triggering module, the verification system being configured to generate a verification interrupt signal based on the sensing signal from the other side of the vehicle using the same logic; and a hardware decision arbitration gate, the hardware decision arbitration gate being configured to output a final effective interrupt command to the multimodal labeling module only when a high-priority interrupt signal generated by the asynchronous mismatch triggering module and a verification interrupt signal generated by the verification system are received simultaneously within a preset time window.

[0016] Preferably, the first sensing monitoring unit is a wheel speed sensor, the second sensing monitoring unit is a one-dimensional distance measuring sensor, and the time sequence relationship model defines that under normal driving conditions, the road unevenness event detected by the second sensing monitoring unit should precede the corresponding vehicle bump event detected by the first sensing monitoring unit by a preset time window.

[0017] Preferably, after the multimodal annotation module is activated, the attention mechanism of its neural network model is forcibly guided to the image region or point cloud cluster associated with the specific wheel that triggered the mismatch event.

[0018] A smart data annotation method based on a multimodal large model, comprising the following steps:

[0019] Step a: The first sensing and monitoring unit continuously monitors the internal state baseline signal, which represents the internal motion state of the vehicle itself, and the second sensing and monitoring unit continuously monitors the external environment baseline signal, which represents the physical state of the external environment in front of the vehicle.

[0020] Step b involves the asynchronous mismatch triggering module performing the following operations: based on a preset time-series relationship model that reflects the normal physical causal relationship between the internal state baseline signal and the external environment baseline signal, the real-time change rate of the two baseline signals is continuously compared to determine whether a mismatch event that violates the time-series relationship model has occurred; when a mismatch event is determined to have occurred, a high-priority interrupt signal is generated, and the impact audio signal associated with the mismatch event is captured synchronously.

[0021] Step c: Perform spectral analysis on the impact audio signal to extract acoustic feature parameters that characterize the material properties of the impact audio signal, and attach these acoustic feature parameters to the high-priority interrupt signal;

[0022] Step d: Perform the following operations through the multimodal annotation module: Activate the system from the default low-power mode in response to a high-priority interrupt signal;

[0023] Step e: After pre-judging the threat level of the mismatch event based on the acoustic feature parameters attached to the high-priority interrupt signal, retrieve the multimodal sensor data associated with the timestamp of the mismatch event and label the physical spatial region related to the mismatch event.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] 1. Through the first and second sensing monitoring units, two cross-domain baseline signals representing the internal motion state of the vehicle and the physical state of the external environment in front are continuously monitored. The asynchronous mismatch triggering module continuously compares the temporal relationship between these two signal streams based on physical causal correlation. Only when the temporal relationship model representing physical regularity is violated is the multimodal labeling module in the default low-power mode activated by hardware interruption. This approach shifts the high-cost attention allocation from the post-cognition focus after comprehensive perception in existing technologies to a low-power pre-cognition filtering method, enabling the system to avoid continuous computational consumption of massive redundant background information without sacrificing sensitivity to small high-risk targets.

[0026] 2. When an asynchronous mismatch triggering module determines that a mismatch event has occurred, it further instructs the audio capture unit to synchronously capture the impact audio signal and perform spectrum analysis on the signal to extract acoustic feature parameters that characterize the material properties. These parameters are then added to the high-priority interrupt signal, introducing an independent and orthogonal verification dimension into the physical anomaly detection mechanism of the main solution. This allows the multimodal annotation module to pre-judge the potential threat level of the event based on the acoustic feature parameters the moment it is activated. As a result, the system no longer performs indiscriminate and costly analysis on all anomalies, but can prioritize the allocation of computing resources to events whose acoustic features indicate high risk. Its attention allocation logic has also evolved from a simple on / off state to an operation mode with both preliminary threat ranking capability and threat level ranking capability.

[0027] 3. By controlling the matrix light source to instantaneously project structured light patterns onto the mismatch event-related area after the multimodal annotation module is activated, and capturing the distortion features of the pattern on the object by the visual sensor, the system has the ability to actively intervene and reshape local observation conditions when facing extreme conditions with low signal-to-noise ratios such as night or rain and fog. It no longer relies entirely on passive natural light imaging, but can create stable optical textures for itself. At this time, the highly difficult and environmentally susceptible object shape recognition task is transformed into a relatively simple task of analyzing the distortion features of known patterns. Thus, the annotation of the three-dimensional contours of abnormal objects largely avoids dependence on external ambient lighting conditions.

[0028] 4. The system of this invention continuously monitors the temporal stability characteristics of the baseline signal through an asynchronous mismatch triggering module, enabling it to intrinsically determine whether the vehicle is in a continuous high-entropy driving condition such as a bumpy road surface. When entering such a condition, it temporarily adjusts its triggering logic. This condition-adaptive arbitration mechanism avoids the system being frequently awakened due to erroneous false interruptions in a continuous physical noise environment, ensuring the reliability of the core detection principle in a wider range of more complex real-world driving scenarios. At the same time, by actively detecting the tire sidewall using a second sensing monitoring unit after a mismatch event occurs, and performing attribution analysis on the event based on the physical characteristics of the echo signal, the system can clearly distinguish between external object impact and tire malfunction, two completely different events. This avoids attribution ambiguity in the decision-making direction, allowing subsequent labeling and response to point to the correct cause.

[0029] 5. The asynchronous mismatch triggering module of this invention also extracts vehicle speed parameters based on the internal state baseline signal, and dynamically and in a closed loop adjusts the core time parameters in its timing relationship model based on this. This enables the physical causal timing mismatch judgment principle, which is the core of the invention, to automatically adapt to changes in the vehicle's speed range from crawling to high speed, ensuring the effectiveness of its judgment logic under all driving conditions. In addition, by setting up an independent verification system and a hardware decision arbitration gate, the interrupt signal generated by the main triggering module is redundantly checked and arbitrated at the hardware level. This ensures that any false triggering caused by single-point failure or random interference can be effectively filtered out, so that the decision output of the entire system has the independent verification reliability necessary for safety-critical applications. Attached Figure Description

[0030] Figure 1 This is a structural diagram of the intelligent data annotation system based on a multimodal large model according to the present invention;

[0031] Figure 2 This is a comparison graph of the frequency response of different materials in this invention;

[0032] Figure 3 This is a flowchart of the state of the intelligent data annotation system of the present invention.

[0033] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0035] The present invention discloses an intelligent data annotation system and method based on a multimodal large model. The system includes a first sensing and monitoring unit, a second sensing and monitoring unit, an audio capture unit, an asynchronous mismatch triggering module, and a multimodal annotation module. The asynchronous mismatch triggering module is connected to the first sensing and monitoring unit, the second sensing and monitoring unit, and the audio capture unit, and is configured to send a high-priority interrupt signal to the multimodal annotation module, which is in low-power or sleep mode by default, to activate it. The operation of the entire system aims to solve the problem that the existing technology is prone to submerging key danger information in massive background data when dealing with static small high-risk targets that suddenly appear at high speeds, because the attention is diverted by conventional traffic elements. At the same time, it avoids the continuous high computing power consumption brought about by the comprehensive perception mode.In a typical application scenario, where a vehicle is traveling at high speed on a well-paved road, the computing platform of existing neural network systems needs to continuously process massive amounts of data collected by various sensors to maintain comprehensive environmental perception, leading to significant computational power consumption. This is especially true when a piece of optically inconspicuous metal debris appears on the road ahead of the vehicle, where traditional vision or lidar modalities struggle to reliably identify it. To address this challenge, this system is configured to execute a pre-cognitive filtering process based on physical causal mismatch. Specifically, the first sensing monitoring unit, preferably a wheel speed sensor, is configured to continuously monitor the internal state baseline signal characterizing the vehicle's internal motion state, i.e., the real-time wheel speed data sequence. Simultaneously, the second sensing monitoring unit, preferably a one-dimensional ranging sensor fixedly aligned with the ground at a specific distance in front of the vehicle, is configured to continuously monitor the external environment baseline signal characterizing the physical state of the external environment ahead of the vehicle, i.e., the distance value sequence representing the road surface smoothness. The asynchronous mismatch triggering module, typically a low-power microcontroller, internally contains a time-series model reflecting the normal physical causal relationship between the internal state baseline signal and the external environment baseline signal. The model defines that, under normal driving conditions, a road unevenness event detected by the second sensing unit should precede a corresponding vehicle bump event detected by the first sensing unit by a preset time window. This asynchronous mismatch triggering module continuously compares the real-time change rates of the two baseline signals. When it detects a drastic change in the internal state baseline signal, i.e., the wheel speed, which does not follow a temporal relationship model but precedes or significantly precedes the change in the external environmental baseline signal, it determines that a mismatch event violating physical norms has occurred. This mismatch strongly suggests a wheel rolling... If a suddenly appearing object is passed over by a one-dimensional ranging sensor that fails to detect it due to its physical limitations, the asynchronous mismatch triggering module immediately generates a high-priority interrupt signal upon detecting a mismatch event. This interrupt signal is then sent to the main computing platform via a hardware interface, such as a general-purpose input / output pin, to activate the multimodal labeling module, which is in low-power mode by default. In this way, the system prioritizes costly attention allocation to a filtering method based on physical causal mismatch, enabling it to sensitively detect small, high-risk, long-tail events without processing massive amounts of redundant information.

[0036] However, the triggering of the aforementioned mismatch events is essentially indiscriminate; whether it's running over a dangerous sharp metal object or a harmless empty plastic bottle, both will wake up the system, potentially causing unnecessary instantaneous computational power activation. Therefore, the system is also configured to implement a threat level prediction mechanism based on acoustic features. When the asynchronous mismatch triggering module determines that a mismatch event has occurred, it will instruct the audio capture unit (i.e., the vehicle-mounted microphone) in parallel to synchronously capture an impact audio signal within a short time window associated with the mismatch event's timestamp. The asynchronous mismatch triggering module then performs a Fast Fourier Transform on this impact audio signal for spectral analysis. Then, one or more acoustic feature parameters that can characterize the material properties of the impactor are extracted, such as the peak frequency where the energy is most concentrated and the proportion of high-frequency energy. These acoustic feature parameters are then attached to a high-priority interrupt signal and sent together to the multimodal annotation module. After the module is activated, it can make a preliminary judgment on the threat level of the event based on these parameters. For example, an event with high-frequency dominant acoustic features is judged as a high-risk event, so computing resources are prioritized. In this way, the system's attention allocation logic evolves from a simple binary switch to a more refined operation mode with preliminary threat ranking capabilities. Furthermore, even if the system is successfully activated and focused on the correct physical space, in extreme conditions with low signal-to-noise ratios such as at night or in rain and fog, passive optical imaging alone is still insufficient to accurately label the shape of abnormal objects. To address this challenge, the system is also equipped with a matrix light source consisting of multiple independently controllable units, typically the matrix adaptive LED headlights of a vehicle. After being activated, the multimodal labeling module is configured to send instructions to the controller of the matrix light source via the vehicle bus, causing it to instantaneously project a preset structured light pattern with a high-contrast geometry onto the physical space area associated with the mismatch event. For example, a checkerboard pattern or parallel stripes; at the same time, the vehicle's vision sensors simultaneously capture images that are illuminated by the structured light pattern and distorted by the three-dimensional contours of the object. The multimodal annotation module then analyzes and enhances the annotation of the object's three-dimensional shape based on the distortion features presented by the structured light pattern on the object image, such as the degree of bending or breakage of the pattern. This procedure enables the system to actively intervene and reshape local observation conditions, transforming the highly difficult task of object shape recognition into an analysis task of known pattern distortion features, so that the annotation of abnormal object contours is largely free from dependence on external ambient lighting conditions.

[0037] Furthermore, when a vehicle travels for extended periods on bumpy, unpaved roads or other persistently high-entropy driving conditions, the baseline signal itself will fluctuate dramatically. Maintaining the original triggering logic would lead to numerous false interruptions. To ensure system reliability in a wider range of driving scenarios, the asynchronous mismatch triggering module is configured to determine whether the vehicle is in this persistent high-entropy driving condition based on the temporal stability characteristics of the internal state baseline signal or the external environment baseline signal within a preset time window. Specifically, this module quantifies the stability or entropy of the baseline signal sequence by performing a sliding window-style standard deviation calculation. When the calculated standard deviation consistently exceeds a preset stability entropy threshold, it is determined that the vehicle has entered a high-entropy condition. At this point, the asynchronous mismatch triggering module automatically adjusts its triggering logic for generating high-priority interrupt signals, such as temporarily suspending the interrupt signal transmission function or proportionally increasing the triggering threshold for mismatch events based on the magnitude of the stability entropy, to suppress or reduce false triggers caused by continuous fluctuations in the baseline signal under this condition. This condition-adaptive arbitration mechanism avoids frequent erroneous wake-ups of the system in continuous physical noise environments; simultaneously, The root cause of a secondary physical mismatch event may be ambiguous, meaning it's impossible to distinguish whether the wheel ran over an external object or experienced a sudden tire malfunction. To circumvent this ambiguity, the second sensing and monitoring unit is further configured to, upon the asynchronous mismatch triggering module determining that a mismatch event has occurred, respond to a command by transmitting an active detection pulse to the tire sidewall of the specific wheel associated with the mismatch event and receiving the echo signal returned from the tire sidewall. The asynchronous mismatch triggering module is further configured to base its response on one or more physical characteristic parameters of the echo signal, such as the energy attenuation rate or envelope of the echo signal. The morphology of a tire is used to determine whether it is in a faulty state. There is a significant difference in the reflection and absorption characteristics of a probe pulse between a taut sidewall with normal tire pressure and a loose sidewall with insufficient tire pressure. The former produces an echo with concentrated energy and slow decay, while the latter produces an echo with diffuse energy and fast decay. Based on this difference, after attribution analysis, the asynchronous mismatch trigger module can attach the attribution results, such as external object impact or tire failure, to a high-priority interrupt signal. This allows the system to clearly distinguish between two completely different events, so that subsequent labeling and response can point to the correct cause.

[0038] Core time parameters in time series relationship models This refers to the time difference between external environmental events and internal state events. Its effectiveness is directly related to the vehicle's speed; a fixed... Since the value cannot adapt to changes across the entire speed range, the asynchronous mismatch triggering module is also configured to extract parameters characterizing the current vehicle speed based on the internal state baseline signal, namely the signal from the wheel speed sensor. Specifically, this can be obtained by calculating the average value of the signal within a sliding window. Subsequently, the module determines the value based on the velocity parameters. Dynamically and in a closed loop, adjust the time parameters in the time series model. The adjustment follows a preset time-reflecting parameter. With speed parameters The calibration rule for the inverse proportional relationship between them is specifically defined by a physical formula: ,in, The effective physical distance between the detection point of the second sensing monitoring unit and the wheel contact point is determined through offline calibration experiments during the design phase, based on a pre-set plan. To extract speed parameters in real time, this dynamic self-calibration mechanism enables the physical causal timing mismatch judgment principle, which is the core of the invention, to automatically adapt to changes in the vehicle's speed range from crawling to high speed, ensuring the effectiveness of its judgment logic under all driving conditions.

[0039] To ensure the reliability of the entire system's decision output, especially the independently verified reliability necessary for safety-critical applications, the system also includes a verification system independent of the aforementioned sensing and triggering modules, and a hardware decision arbitration gate. This verification system is configured to generate a verification interrupt signal using identical logic based on sensor signals from the other side of the vehicle, such as the rotational speed of the left wheel and the corresponding one-dimensional ranging signal. The hardware decision arbitration gate, typically a basic hardware AND gate, receives a high-priority interrupt signal generated by the master-asynchronous mismatch triggering module and a verification interrupt signal generated by the verification system at its two inputs, respectively. This hardware decision arbitration gate is configured to output a final, effective interrupt command to the multimodal labeling module only when both interrupt signals are received simultaneously within a short time window. This decision arbitration gateway, based on physical redundancy and logical AND gates, ensures that any single point of failure or random electromagnetic interference does not compromise the reliability of the system's output. All false triggers caused by mismatches can be effectively filtered at the hardware level, ensuring the reliability of the system's decision output through independent verification. Finally, when the multimodal annotation module is activated by the final interrupt command, it retrieves high-resolution multimodal sensor data such as those from cameras and LiDAR associated with the timestamp of the mismatch event, and annotates the physical spatial regions related to the mismatch event. Specifically, the attention mechanism of its neural network model is forcibly guided to the image region or point cloud cluster associated with the specific wheel that triggered the mismatch event, so as to perform high-precision multimodal annotation on the physical entity that caused the mismatch interruption. Through this event-triggered operation mode driven by physical causal mismatch, the dataset generated by the system naturally filters out a large number of ordinary driving scenarios without annotation value. Each annotated data point originates from a special event verified by physical laws, thus providing samples strongly correlated with physical anomalies for the training of the neural network model.

[0040] Example 1: In an autonomous driving test scenario on a nighttime highway, a test vehicle equipped with the intelligent data annotation system of this invention is driving stably at a speed of 100 kilometers per hour. Its multimodal annotation module is in the default low-power mode, and only the asynchronous mismatch triggering module continues to operate in the background with extremely low power consumption. At this time, there is an irregular static metal part that has fallen from a truck and is similar in color to the road surface. Traditional vision and lidar systems fail to identify it as an obstacle because they do not have significant optical or geometric features. During the vehicle's movement, the asynchronous mismatch triggering module continuously receives wheel speed signals from the first sensing monitoring unit and one-dimensional ranging signals from the second sensing monitoring unit, and based on the real-time extracted vehicle speed parameters... Core time parameters in the time series model Dynamic self-calibration is performed; when the vehicle is about to run over the metal part, since the height of the part is lower than the detection threshold of the one-dimensional ranging sensor, the external environmental baseline signal does not produce any warning fluctuations. However, at the moment when the right front wheel of the vehicle makes physical impact with the metal part, the internal state baseline signal output by the first sensing monitoring unit, i.e., the wheel speed, produces a violent high-frequency change. Since this change cannot be interpreted a priori by the external environmental baseline signal in front, it is immediately judged by the asynchronous mismatch triggering module as a mismatch event that violates the preset timing relationship model.

[0041] At the same moment this mismatch event is determined, a collaborative mechanism is triggered. The asynchronous mismatch triggering module sends a high-priority interrupt signal to the main computing platform and simultaneously instructs the audio capture unit to synchronously capture the audio signal at the moment of impact. By performing a Fast Fourier Transform on the audio signal, the asynchronous mismatch triggering module extracts acoustic feature parameters characterized by a high proportion of high-frequency energy and adds these parameters to the interrupt signal. This parallel mechanism of physical mismatch detection and acoustic material verification constitutes a two-stage information filtering and threat classification system. It not only resolves the industry contradiction between high-sensitivity detection capabilities and the requirements for low power consumption and low false alarm rate, but also makes the original system, which only included information filtering and threat classification, more comprehensive. The normal interrupt signal is given supplementary information about the abnormal physical properties. After receiving this information-rich hardware interrupt, the attention mechanism of the multimodal annotation module's neural network model is forcibly guided to the image region and point cloud cluster associated with the right front wheel that triggered the mismatch event. At this time, the system's computational task has been redefined from a high-cost open search problem in a broad perception field to a low-cost closed problem in a local spatiotemporal domain aimed at verifying physical events. The system retrieves high-resolution sensor data within a very short window before and after the interrupt signal's timestamp, performs focused analysis on this local space, and finally generates high-value annotation data containing pre-classification of the object's spatiotemporal location, three-dimensional shape, and material properties.

[0042] Example 2: This example aims to objectively verify the actual detection effectiveness and system power consumption performance of the intelligent data annotation system based on physical causal mismatch of the present invention through a quantitative test including a control group in a controlled test field. The test platform is a test vehicle equipped with the complete system of the present invention, on which a traditional multimodal perception system with high computing power and continuous operation is deployed in parallel as a control group. The two systems share the same set of high-resolution cameras and lidar sensors, but the asynchronous mismatch triggering module of the present invention and its associated first and second sensing monitoring units operate independently. The key parameters used in the test, such as the threshold for the rate of change of the internal state baseline signal used to judge mismatch events in the asynchronous mismatch triggering module, are set according to a structured decision logic. That is, the setting of this threshold aims to achieve a balance between the sensitivity of detecting weak physical impacts and the stability of suppressing background noise during normal vehicle operation. Therefore, prior to the experiment, a statistical distribution model of the background noise signal was obtained by collecting baseline driving data on the test vehicle over a long period of time. The final trigger threshold was set to be several standard deviations higher than the mean of the background noise statistical distribution, so as to judge the event at a predictable signal-to-noise ratio level. The experiment was conducted on a closed, flat test road. The test vehicle was driven at a constant speed of 80 km / h and repeated the test under four conditions: Condition 1 was baseline driving without obstacles; Condition 2 had a 5 cm high dark steel block placed on the path; Condition 3 had a hard rubber block of the same size placed on the path; Condition 4 had a flattened empty plastic bottle placed on the path. The size and optical characteristics of these obstacles were designed to be difficult for the visual or lidar modules of the control group system to identify stably. During the experiment, the alarm or labeling results of the two systems and the real-time power consumption of the computing platform were recorded simultaneously.

[0043] In repeated tests, the system exhibited a highly consistent response pattern, as detailed in Table 1. During the baseline driving in Condition 1, neither system generated any alarms, and the asynchronous mismatch triggering module of the present invention remained in a low standby power consumption state. In Conditions 2 and 3, when the vehicle ran over steel blocks and rubber blocks, the control group system failed to generate effective annotations, while the present invention system generated a high-priority interrupt signal through the asynchronous mismatch triggering module at the moment of each roll, activating the multimodal annotation module. Simultaneously, the impact audio signals collected by the audio capture unit, after spectral analysis, showed energy concentration characteristics belonging to different frequency bands. The impact of the steel blocks produced acoustic characteristics dominated by high frequencies, while the rubber blocks produced characteristics dominated by mid-to-low frequencies. In Condition 4, when rolling over an empty plastic bottle, the asynchronous mismatch triggering module also generated an interrupt signal, but its additional acoustic characteristic parameters exhibited a different spectral shape than those in Conditions 2 and 3. Throughout the entire test, the average power consumption of the present invention system was significantly lower than that of the continuously running control group system.

[0044] Table 1: Performance comparison between the system of the present invention and the control group system under different operating conditions.

[0045]

[0046] Experimental data shows that the system of this invention can reliably detect physical obstacles that traditional sensing systems miss due to the insignificance of the target's optical or geometric features. Its detection capability stems from the direct judgment of causal mismatch in the physical world, rather than relying on high-cost image recognition. In addition, by introducing acoustic feature analysis, the system gives preliminary material property differentiation to indiscriminate physical mismatch events, providing a basis for subsequent threat level judgment and resource allocation. Finally, while effectively detecting small, high-risk, long-tail events, the system's event-driven operation mode reduces its power consumption by nearly two orders of magnitude compared to traditional continuously running systems during most of the normal operating time.

[0047] Example 3: This example combines Figures 1 to 3 This section describes the intelligent data annotation system and methods based on multimodal large models, such as... Figure 1 As shown, the first sensing monitoring unit, a wheel speed sensor, continuously monitors the vehicle's internal motion state, providing an internal state baseline signal; the second sensing monitoring unit, a one-dimensional ranging sensor, continuously monitors the physical state of the external environment in front of the vehicle, providing an external environment baseline signal; the audio capture unit captures the impact audio signal and performs spectral analysis for subsequent sound feature extraction; the asynchronous mismatch triggering module determines whether a mismatch event has occurred based on a time-series model of physical causal mismatch; when a mismatch event is detected, a high-priority interrupt signal is generated, and the audio capture unit is instructed to synchronously capture the impact audio signal for further analysis of its acoustic characteristics; the redundancy verification system ensures the reliability of the system's judgment by monitoring the verification interrupt signal generated by the sensor on the other side; the hardware decision arbitration gate arbitrates the interrupt signals from the main triggering module and the verification system, ensuring that only simultaneous interrupts occur. Only when two valid signals are received is a final interrupt command sent to the multimodal annotation module. Upon receiving a valid interrupt signal, the multimodal annotation module activates and pre-determines the threat level of the mismatch event based on acoustic characteristics. It then retrieves multimodal sensor data related to the event timestamp and performs precise data annotation. The matrix light source projects structured light patterns onto the target area to enhance the annotation capability of object shapes, especially improving annotation accuracy in low signal-to-noise ratio environments. High-value annotation data is ultimately generated, including real-time location information, the three-dimensional shape of the object, and material properties. This system optimizes the data annotation process by combining physical causal mismatch judgment with multimodal information, which can improve the detection capability of small, high-risk, long-tail events, while avoiding the computational power consumption caused by processing a large amount of redundant information in traditional systems.

[0048] like Figure 2 As shown, the horizontal axis represents the frequency range, from 0.5kHz to 10kHz; the vertical axis represents the relative noise intensity, with the unit being relative noise intensity. The three curves in the figure represent different materials: the solid black line represents a steel block, the dashed black line represents a rubber block, and the dotted black line represents a plastic bottle. The relative noise intensity of different materials varies significantly with frequency. The steel block has a higher noise intensity in the lower frequency range of 0.5kHz to 2kHz, and the noise intensity gradually decreases as the frequency increases. The noise intensity of the rubber block shows a relatively stable decreasing trend, especially after 5kHz. The noise intensity of the plastic bottle shows a more significant decreasing trend in the high-frequency range above 5kHz.

[0049] like Figure 3 As shown, the system first starts from a low-power monitoring state and continuously monitors the internal state baseline signal based on a time-series relationship model to determine whether it has entered a continuous high-entropy driving condition. If it has not entered a continuous high-entropy condition, the system maintains the low-power mode and continues to perform routine monitoring. In this state, if a physical causal mismatch event is detected, the system will enter a mismatch event arbitration state. In this state, the system instructs the audio capture unit to synchronously capture the impact audio signal and extract acoustic feature parameters. At the same time, the system performs a primary / backup system redundancy check and generates a valid interrupt signal based on the verification result. After the arbitration is passed, the system enters a high-power activation and annotation state. In the high-power activation and annotation state, the system activates the multimodal annotation module and performs efficient annotation operations based on the acoustic feature parameters. At the same time, the system controls the matrix light source to project a structured light pattern onto the target area to further enhance the annotation effect on the target. Finally, high-value annotation data is generated and stored. If the system detects a condition for entering the low-power mode or a signal for exiting the high-power state at any time, the system will return to the low-power monitoring state for re-monitoring to ensure efficient operation and energy management of the system in different scenarios.

[0050] Example 4: This example aims to describe the offline calibration of a series of key parameters and the deterministic procedure of internal logic required for the core module, especially the asynchronous mismatch triggering module, of the aforementioned technical solution before its deployment on a specific vehicle platform, to ensure its reliable operational performance in subsequent practical applications. The entire calibration process is carried out in a laboratory equipped with an adjustable speed driving platform and a configurable road environment. The calibration object is a vehicle under test with integrated system hardware. The first step of the calibration is to determine the core parameter in the timing relationship model, namely the effective physical distance between the detection point of the second sensing monitoring unit and the wheel contact point. In this procedure, the vehicle under test is fixed on an adjustable-speed driving platform, and a standard obstacle of known geometry, clearly detectable by a second sensing monitoring unit, is placed on its tire rolling path. Subsequently, the vehicle travels at a set of precisely calibrated constant speeds covering a range from 30 km / h to 100 km / h. The system then repeatedly passes through the standard obstacle; each time it passes, the system records with high precision the timestamp of the obstacle detected by the second sensing monitoring unit. And the peak timestamp of signal changes caused by crushing impact, fed back by the first sensing monitoring unit. For each test, a measured time difference is calculated, i.e. Based on physical relationships A corresponding effective physical distance The measured values ​​can be calculated; ultimately, by analyzing multiple measurements obtained at all different speeds... The measured values ​​are statistically averaged to obtain an effective physical distance for this specific vehicle model and sensor layout. The final calibration value will be stored in the non-volatile memory of the asynchronous mismatch trigger module.

[0051] Next, the adaptive arbitration mechanism of the system under continuous high-entropy driving conditions is calibrated. The test vehicle is simulated to drive on unpaved roads with different bump levels on the test platform. The gravel road condition is simulated by placing continuous irregular low obstacles on the driving platform. During this process, the asynchronous mismatch triggering module continuously calculates the temporal stability characteristics of the internal state baseline signal and the external environment baseline signal within a preset time window, specifically a sliding window standard deviation. The system records this standard deviation under different levels of bumpiness. The numerical distribution range is determined to establish a mapping relationship between operating conditions and signal entropy; based on this mapping relationship, a stability entropy threshold is determined. It is set to a value taken as the minimum measured under a predictable, lowest level of sustained high-entropy driving condition. The mean value is used to ensure sensitive identification of this operating condition; simultaneously, when the system determines that it has entered a high-entropy operating condition, its threshold for triggering mismatch events is set. The adjustment logic is defined as a specific functional relationship. ,in, This represents the basic threshold under normal operating conditions, while the coefficient... Based on the trade-off between noise suppression and abnormal signal throughput, the algorithm is derived through fitting experimental data, thus transforming the adjustment logic into a quantifiable and verifiable adaptive algorithm.

[0052] Finally, the attribution analysis logic for mismatch events, especially the tire failure judgment mechanism, is calibrated. The vehicle under test is stationary, and the tire pressure of one wheel can be precisely controlled via external equipment. The asynchronous mismatch triggering module is instructed to enter a manually triggered diagnostic mode. In this mode, the second sensing monitoring unit emits an active detection pulse to the tire sidewall of that wheel and receives the echo signal. This process is repeated at multiple different and known tire pressure points, covering a range from standard pressure to complete underinflation. For each measurement, the asynchronous mismatch triggering module quantifies one or more physical characteristic parameters of the received echo signal, specifically calculating the energy integral of the echo signal within its effective time window. This experiment provides a clear correlation between tire pressure and... The relationship curve shows that as air pressure decreases, The value decreases due to increased absorption and scattering of the probe pulse by the tire sidewall; based on this curve, an energy threshold is used to distinguish between normal and faulty tire conditions. The system is positioned at a safety margin between the corresponding values ​​for normal air pressure and severe air deficit, thus providing a deterministic, physical measurement-based basis for attribution analysis. Through the aforementioned series of deterministic calibration procedures, before deployment, all key algorithm parameters and logical thresholds related to specific hardware and physical environments within the intelligent data annotation system are assigned traceable and objective values ​​based on experimental data. This transforms a general technical solution into an engineering deployment instance that is highly consistent and reliable for a specific platform.

[0053] Example 5: Before the intelligent data annotation system is deployed on a mass-produced vehicle platform, a standardized offline data filling and online parameter calibration procedure must be executed to ensure that the decision-making logic within the system has a high degree of consistency and traceability. In a standardized offline calibration process, for the logic in the multimodal annotation module used to predict the threat level of mismatch events, repeated and standardized physical impact tests are conducted on the vehicle in the laboratory using objects of different materials, masses, and shapes. In each test, the system synchronously records a set of multi-dimensional acoustic feature parameters associated with the impact, which are parsed by the asynchronous mismatch triggering module, and associates the event with a preset baseline threat level label. By collecting a large number of such acoustic feature-threat level data pairs, a lookup table or decision tree model for mapping the continuous acoustic feature parameter space to discrete threat level classification is systematically constructed. This model is then solidified into the storage unit of the multimodal annotation module, thereby establishing its threat level prediction function on a quantifiable empirical data basis.

[0054] Secondly, the time window parameter for the hardware decision arbitration gate also follows a deterministic online calibration process. In this process, deployed vehicles pass through a standard obstacle at different speeds multiple times in the test site. This obstacle is wide enough to be detected by the sensor systems on both sides of the vehicle simultaneously. In such scenarios, which inevitably trigger interrupt signals from both the primary and backup systems, the system accurately measures and records the time difference between the high-priority interrupt signal generated by the primary asynchronous mismatch trigger module and the verification interrupt signal generated by the redundant verification system. Through repeated testing, the system obtained a time difference. Based on the statistical distribution under different operating conditions, the duration of the time window used by the hardware decision arbitration gate to determine whether two interrupt signals are effectively correlated is determined to be the sum of the mean and three standard deviations of the statistical distribution. This procedure ensures that the time window of the arbitration gate can accommodate minor time jitter caused by differences in physical paths in real synchronous events, and can effectively reject two unrelated signals that are far apart in time and caused by random interference or independent faults, thus providing reliable operating parameters for the system's redundancy and safety mechanism.

[0055] Example 6: In scenarios where the intelligent data annotation system is integrated into a vehicle platform and put into daily operation, to ensure the reliability and compliance of its various functions, especially the active intervention and redundancy safety mechanisms, the system executes a standardized pre-test and online operation strategy configuration procedure before each startup or entry into working state. The first step of this procedure is that the main system controller sends a status query command in parallel to the main asynchronous mismatch trigger module and the independent redundancy verification system, requiring them to report the integrity status of themselves and the first and second sensor monitoring units connected to them within a preset timeout window. Only when both systems return to normal status and their internal self-test programs do not report any fault codes is the hardware decision arbitration gate authorized to enter normal working mode. If either system reports a sensor failure or communication timeout, the function of the hardware decision arbitration gate will be temporarily suspended, and a status flag representing redundancy degradation will be recorded in the on-board diagnostic system. This procedure ensures that the core redundancy safety logic of the system is built on a verified and intact hardware foundation.

[0056] Simultaneously, the system also performs online evaluation of the hardware and environmental conditions related to active optical texture detection to configure its operating strategy. The system first queries the vehicle hardware configuration list to confirm the existence of a usable and normal matrix light source, and simultaneously acquires the value representing the current ambient illuminance level output by the vehicle's ambient light sensor. Based on this, the execution logic of the instantaneous projection of structured light patterns in the multimodal annotation module is preset to a set of strict AND gate condition judgments. That is, the function will only be enabled when three conditions are met simultaneously: the matrix light source exists and is healthy, the ambient illuminance is lower than the preset activation threshold, and the vehicle's near-field environmental perception system does not detect other traffic participants within the safety boundary calculated based on vehicle speed and projection direction. In any other case, even if a mismatch event is triggered, the active intervention function remains disabled, and the system will only rely on passive sensing data for annotation. This online operating strategy with multiple condition constraints ensures the environmental adaptability and safety of the active intervention behavior.

[0057] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An intelligent data annotation system based on a multimodal large model, characterized in that, The system includes: The first sensing and monitoring unit is configured to continuously monitor the internal state baseline signal that characterizes the internal motion state of the vehicle itself. The second sensing and monitoring unit is configured to continuously monitor the external environment baseline signal that characterizes the physical state of the external environment in front of the vehicle; An audio capture unit; an asynchronous mismatch triggering module, connected to a first sensing monitoring unit, a second sensing monitoring unit, and the audio capture unit, is configured to: continuously compare the real-time change rates of the two baseline signals based on a preset time-series relationship model reflecting the normal physical causal relationship between the internal state baseline signal and the external environment baseline signal, to determine whether a mismatch event violating the time-series relationship model has occurred; when a mismatch event is determined to have occurred, a high-priority interrupt signal is generated, and the audio capture unit is instructed to synchronously capture the impact sound associated with the mismatch event. The system includes a frequency signal and a spectrum analysis of the impact audio signal to extract acoustic feature parameters characterizing the material properties of the impact audio signal. These acoustic feature parameters are then appended to a high-priority interrupt signal. A multimodal annotation module, which is in low-power mode by default, is configured to be activated only when a high-priority interrupt signal is received. Based on the acoustic feature parameters, it pre-determines the threat level of the mismatch event, retrieves multimodal sensor data associated with the timestamp of the mismatch event, and annotates the physical spatial region related to the mismatch event.

2. The intelligent data annotation system based on a multimodal large model according to claim 1, characterized in that, The system also includes a matrix light source composed of multiple independently controllable units; after being activated, the multimodal annotation module is also configured to control the matrix light source to instantaneously project a preset structured light pattern onto the physical space region related to the mismatch event, capture the image of the object illuminated by the structured light pattern through a visual sensor, and based on the distortion features presented on the object image by the structured light pattern.

3. The intelligent data annotation system based on a multimodal large model according to claim 1, characterized in that, The asynchronous mismatch triggering module is also configured to determine whether the vehicle is in a continuous high-entropy driving condition based on the time-domain stability characteristics of the internal state baseline signal or the external environment baseline signal within a preset time window; and when it is determined that the vehicle is in a continuous high-entropy driving condition, its triggering logic for generating a high-priority interrupt signal is temporarily adjusted.

4. The intelligent data annotation system based on a multimodal large model according to claim 1, characterized in that, The second sensing and monitoring unit is also configured to, after the asynchronous mismatch triggering module determines that a mismatch event has occurred, respond to the instruction to send an active detection pulse to the tire sidewall of the specific wheel associated with the mismatch event, and receive the echo signal returned from the tire sidewall; the asynchronous mismatch triggering module is further configured to determine whether the tire is in a fault state based on one or more physical characteristic parameters of the echo signal, and perform attribution analysis on the mismatch event accordingly, so as to attach the attribution result to the high-priority interrupt signal.

5. The intelligent data annotation system based on a multimodal large model according to claim 1, characterized in that, The asynchronous mismatch triggering module is also configured to extract parameters characterizing the current vehicle speed based on the internal state baseline signal. And based on speed parameters Dynamically adjust the time parameters in the time series model. Adjustments are made according to preset response time parameters. With speed parameters Calibration rules for the inverse proportional relationship between them.

6. The intelligent data annotation system based on a multimodal large model according to claim 5, characterized in that, The calibration rules are limited to: ,in, To preset the effective physical distance between the detection point of the second sensing monitoring unit and the wheel contact point, For speed parameters.

7. The intelligent data annotation system based on a multimodal large model according to claim 1, characterized in that, The system also includes: a verification system independent of a first sensing monitoring unit, a second sensing monitoring unit, an audio capture unit, and an asynchronous mismatch triggering module, which is configured to generate a verification interrupt signal based on the sensing signal from the other side of the vehicle using the same logic; and a hardware decision arbitration gate, which is configured to output a final effective interrupt command to the multimodal labeling module only when a high-priority interrupt signal generated by the asynchronous mismatch triggering module and a verification interrupt signal generated by the verification system are received simultaneously within a preset time window.

8. The intelligent data annotation system based on a multimodal large model according to claim 1, characterized in that, The first sensing unit is a wheel speed sensor, and the second sensing unit is a one-dimensional distance measuring sensor. The time-series relationship model defines that, under normal driving conditions, the road unevenness event detected by the second sensing unit should precede the corresponding vehicle bump event detected by the first sensing unit by a preset time window.

9. The intelligent data annotation system based on a multimodal large model according to claim 1, characterized in that, Once the multimodal annotation module is activated, the attention mechanism of its neural network model is forced to be directed to the image region or point cloud cluster associated with the specific wheel that triggered the mismatch event.

10. A smart data annotation method based on a multimodal large model, characterized in that, The method includes the following steps: Step a: The first sensing and monitoring unit continuously monitors the internal state baseline signal, which represents the internal motion state of the vehicle itself, and the second sensing and monitoring unit continuously monitors the external environment baseline signal, which represents the physical state of the external environment in front of the vehicle. Step b involves the asynchronous mismatch triggering module performing the following operations: based on a preset time-series relationship model that reflects the normal physical causal relationship between the internal state baseline signal and the external environment baseline signal, the real-time change rate of the two baseline signals is continuously compared to determine whether a mismatch event that violates the time-series relationship model has occurred; when a mismatch event is determined to have occurred, a high-priority interrupt signal is generated, and the impact audio signal associated with the mismatch event is captured synchronously. Step c: Perform spectral analysis on the impact audio signal to extract acoustic feature parameters that characterize the material properties of the impact audio signal, and attach these acoustic feature parameters to the high-priority interrupt signal; Step d: Perform the following operations through the multimodal annotation module: Activate the system from the default low-power mode in response to a high-priority interrupt signal; Step e: After pre-judging the threat level of the mismatch event based on the acoustic feature parameters attached to the high-priority interrupt signal, retrieve the multimodal sensor data associated with the timestamp of the mismatch event and label the physical spatial region related to the mismatch event.

Images