Forward collision warning method and apparatus

Abstract

The application provides a forward collision warning method and device, wherein the method comprises: acquiring asynchronous perception data of a warning area; updating semantic state components in a target state vector based on observation of first modal data; in a time interval between two adjacent times of updating the semantic state components, performing recursive prediction and correction on motion state components in the target state vector based on continuous observation of second modal data; and determining a current target state vector based on the corrected motion state components and the current semantic state components in the target state vector to perform forward collision warning. The method realizes high-frequency and semantic target state output by fusing high-frame-rate sparse motion data and low-frame-rate dense semantic data, solves the problem of warning lag caused by image frame processing delay in the traditional method, avoids the shortcomings of large calculation overhead and difficulty in deployment of a complex fusion model, and ensures the robustness of the warning system in extreme scenes such as severe light changes.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a forward collision warning method and device. Background Technology

[0002] Collision warning is a core function to ensure the safe operation of intelligent agents such as autonomous vehicles, robots, and drones. Existing warning technologies mainly revolve around three paths: physical models, visual perception, and novel sensors. However, none of these can simultaneously meet the requirements of low latency, high robustness, and low computational complexity in complex and dynamic edge scenarios.

[0003] Physical model-based methods, such as using millimeter-wave radar or lidar to measure relative distance and velocity to calculate the estimated collision time, rely heavily on the timeliness and accuracy of sensor observations in the early stages of target appearance for their warning timeliness and accuracy. In sudden scenarios such as pedestrians suddenly appearing out of nowhere, the initial sensor signals are often unstable, which can easily lead to warning delays. Furthermore, model parameters calibrated for specific platforms (such as automobiles) are difficult to transfer to platforms with other motion configurations (such as drones).

[0004] Traditional vision-based methods utilize image sequences for target detection, tracking, and risk calculation. While these methods can provide rich semantic information, they suffer from inherent processing delays, causing the system output to lag behind the real world. Furthermore, they are prone to motion blur under high-speed motion and overexposure or underexposure in high dynamic range scenes such as strong light and backlight, leading to recognition algorithm failure and insufficient system robustness.

[0005] To overcome the limitations of traditional cameras, neuromorphic vision sensors (event cameras) have been introduced. Their asynchronous event streams are sensitive to changes in brightness, which can alleviate motion blur. However, event streams are essentially sparse temporal differential signals, lacking spatial structure and semantic information, and cannot independently provide the complete target attributes required for early warning. Although there have been attempts to fuse events and RGB images, existing solutions mostly rely on complex neural network models with large numbers of parameters and difficult training, or involve computationally intensive preprocessing steps, making it difficult to meet the stringent requirements of edge devices for real-time performance and low power consumption.

[0006] Therefore, there is an urgent need in this field for a collision warning solution that can balance high timeliness, high environmental adaptability, and low resource consumption. Summary of the Invention

[0007] This invention provides a forward collision warning method and apparatus to solve the problem that existing collision warning technologies cannot simultaneously achieve ultra-low latency perception, high environmental robustness and low computational complexity. By asynchronously fusing and propagating state of high frame rate sparse motion data and low frame rate dense semantic data, it achieves low latency, high robustness and low complexity semantic-level perception and collision warning of risky targets in complex dynamic environments.

[0008] On one hand, the present invention provides a forward collision warning method, comprising: acquiring asynchronous sensing data of a warning area, the asynchronous sensing data including first modal data providing target semantic observation at a first frequency, and second modal data providing target motion change observation at a second frequency higher than the first frequency; determining and updating semantic state components in a target state vector based on the observations of the first modal data; recursively predicting and correcting motion state components in the target state vector based on continuous observations of the second modal data within the time interval between two adjacent updates of the semantic state components; determining a current target state vector based on the corrected motion state components and the current semantic state components in the target state vector, and performing a forward collision warning based on the current target state vector.

[0009] Further, the first modal data includes RGB image data, and the second modal data includes temporal difference data and spatial difference data; correspondingly, the asynchronous sensing data of the warning area is obtained by: synchronously acquiring the RGB image data, the temporal difference data, and the spatial difference data through a hybrid pixel sensor; or, acquiring an RGB image sequence output by an image sensor; obtaining the RGB image data by performing down-sampling processing on the RGB image sequence; and obtaining the temporal difference data and the spatial difference data by calculating the offset gradient of the RGB image sequence in the temporal and spatial dimensions, respectively.

[0010] Further, the step of determining and updating the semantic state components in the target state vector based on the observation of the first modality data includes: performing representation mapping processing on the first modality data to obtain initial semantic features; performing feature encoding on the initial semantic features to extract semantic and contextual information to obtain semantic observation information; and updating the semantic state components in the target state vector based on the semantic observation information; wherein the semantic state components include the target's width, height, and category probability.

[0011] Furthermore, the recursive prediction and correction of the motion state component in the target state vector based on continuous observations of the second modality data during the time interval between two adjacent updates of the semantic state component includes: predicting the target's motion state at the next moment using a state transition model based on the target state vector at the current moment to obtain the predicted motion state; obtaining the target's motion observation information based on the second modality data at the current moment; and correcting the predicted motion state based on the motion observation information to update the motion state component in the target state vector; wherein the motion state component includes the target's position and velocity.

[0012] Furthermore, the step of obtaining the target's motion observation information based on the second modal data at the current moment includes: performing characterization mapping on the temporal difference data and the spatial difference data respectively to obtain initial motion features; performing weighted fusion on the initial motion features to obtain a spatiotemporal change response map; and obtaining the motion observation information based on the spatiotemporal change response map.

[0013] Furthermore, the forward collision warning based on the current target state vector includes: predicting the future trajectory of the target based on the current target state vector, and calculating the estimated collision time based on the future trajectory; and outputting a collision warning if the estimated collision time is less than a preset time threshold.

[0014] Further, the forward collision warning based on the current target state vector includes: predicting the future trajectory of the target based on the current target state vector and determining the target's category probability; calculating the expected collision time based on the future trajectory and calculating the overlap between the future trajectory and the risk area; calculating a comprehensive risk score based on the overlap, the expected collision time, and the category probability; and outputting a collision warning if the comprehensive risk score is greater than a preset risk threshold.

[0015] Furthermore, the step of calculating the comprehensive risk score based on the overlap, the expected collision time, and the category probability includes: multiplying the overlap, the reciprocal of the expected collision time, and the category probability by their respective weighting coefficients and then summing them to obtain the comprehensive risk score.

[0016] Furthermore, the initial semantic features are feature-encoded using one or more combinations of convolutional structures, temporal modeling structures, attention structures, graph structures, frequency domain transformation structures, and pyramid structures to obtain the semantic observation information.

[0017] Secondly, the present invention also provides a forward collision warning device, comprising: an asynchronous sensing data acquisition module, configured to acquire asynchronous sensing data of a warning area, the asynchronous sensing data including first modal data providing target semantic observation at a first frequency, and second modal data providing target motion change observation at a second frequency higher than the first frequency; a target state vector update module, configured to determine and update semantic state components in a target state vector based on observations of the first modal data; and, within the time interval between two adjacent updates of the semantic state components, recursively predict and correct motion state components in the target state vector based on continuous observations of the second modal data; and a forward collision warning decision module, configured to determine a current target state vector based on the corrected motion state components and the current semantic state components in the target state vector, and to perform a forward collision warning based on the current target state vector.

[0018] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the forward collision warning method as described above.

[0019] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the forward collision warning method as described above.

[0020] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the forward collision warning method as described above.

[0021] The forward collision warning method provided by this invention acquires asynchronous sensing data of the warning area. This asynchronous sensing data includes first modal data providing target semantic observations at a first frequency, and second modal data providing target motion change observations at a second frequency higher than the first frequency. Based on the observations of the first modal data, the semantic state components in the target state vector are determined and updated. During the time interval between two adjacent updates of the semantic state components, based on continuous observations of the second modal data, the motion state components in the target state vector are recursively predicted and corrected. Based on the corrected motion state components and the current semantic state components in the target state vector, the current target state vector is determined for forward collision warning. This method fuses high-frame-rate sparse motion data and low-frame-rate dense semantic data, continuously predicting and correcting the target motion state using motion data during the intervals between semantic data updates. Combined with the semantic tags provided by the semantic data, it achieves high-frequency, semantically-enabled target state output. This fundamentally solves the warning lag problem caused by image frame processing delays in traditional methods, while avoiding the drawbacks of complex fusion models such as high computational overhead and difficulty in deployment. It also ensures the robustness of the warning system in extreme scenarios such as drastic changes in lighting. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic flowchart of the forward collision warning method provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the preprocessing and fusion process of temporal difference data and spatial difference data provided in the embodiments of the present invention.

[0025] Figure 3 This is a schematic diagram of the multimodal asynchronous state estimation and fusion mechanism of the forward collision warning method provided in the embodiments of the present invention.

[0026] Figure 4 This is a schematic diagram of the forward collision warning device provided in an embodiment of the present invention.

[0027] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] It's easy to understand that methods based on traditional physics models calculate the estimated collision time using relative distance and relative velocity (including relative acceleration), and compare it with a threshold to trigger an early warning. This method heavily relies on timely and reliable distance and velocity estimates, but the observation quality is poor when the risky target or obstacle first appears, leading to unstable distance or velocity estimates and causing a delay in threshold triggering. At the same time, to suppress false alarms, thresholds are often set conservatively, making it difficult for risk warnings to take effect promptly.

[0030] Traditional frame-based camera-based methods improve the accuracy of occluded target detection and the stability of risk target warnings by optimizing temporal correlation methods, improving candidate box quality, or refining the loss function. However, the end-to-end perception chain introduces blind time: from sampling, ISP processing, detection inference, tracking updates to risk calculation, there is an unavoidable processing window, and the system output corresponds to past observations, which differs significantly from the actual values ​​in the real world. In addition, traditional frame-based cameras are prone to motion blur and failure under high dynamic range, leading to a significant decrease in the accuracy and robustness of target recognition.

[0031] Existing multimodal fusion methods mainly alleviate detection degradation caused by motion blur and extreme lighting by fusing frame images and event images. However, existing methods for early detection of occluded targets have high computational costs and high model optimization complexity, making them difficult to cope with some extreme scenarios.

[0032] In light of this, this invention proposes a novel forward collision warning method. By asynchronously fusing and propagating state from high-frame-rate sparse spatiotemporal differential data and low-frame-rate dense RGB data, it can achieve ultra-low latency identification and early warning of risky targets in complex environments. This method can be widely applied to environmental perception and risk avoidance in open-world automatic control platforms. It is of great significance for promoting the agile operation and timely risk avoidance of autonomous intelligent systems in complex environments.

[0033] Specifically, Figure 1 A flowchart illustrating the forward collision warning method provided in an embodiment of the present invention is shown.

[0034] like Figure 1As shown, the method includes: S110, acquiring asynchronous sensing data of the warning area, the asynchronous sensing data including first modal data providing target semantic observation at a first frequency, and second modal data providing target motion change observation at a second frequency higher than the first frequency; S120, determining and updating the semantic state components in the target state vector based on the observation of the first modal data; within the time interval between two adjacent updates of the semantic state components, recursively predicting and correcting the motion state components in the target state vector based on continuous observation of the second modal data; S130, determining the current target state vector based on the corrected motion state components and the current semantic state components in the target state vector, and performing a forward collision warning based on the current target state vector.

[0035] The following will provide a detailed description of steps S110-S130 and related steps.

[0036] S110, acquire asynchronous sensing data of the warning area, the asynchronous sensing data including first modal data that provides target semantic observation at a first frequency, and second modal data that provides target motion change observation at a second frequency higher than the first frequency.

[0037] This step is the data input basis for realizing the forward collision warning method provided in this embodiment. It provides the system with two sensing data streams that have essential differences in update time frequency and information content, namely the first modal data and the second modal data of the warning area.

[0038] The warning zone is a pre-defined spatial area in front of the vehicle where collision risk monitoring is required. It can be defined by the field of view and effective detection distance of the onboard perception system (such as a forward-facing camera, millimeter-wave radar, etc.) and is used to filter and focus on moving or stationary targets (such as vehicles in front, pedestrians, etc.) that pose a potential threat to the vehicle.

[0039] First-modal data refers to data sampled at a relatively low first frequency (e.g., 30 frames per second). First-modal data provides semantic observations of targets in the imaging scene, containing rich information for identifying "what the target is," such as global features like texture, color, and shape. In practice, first-modal data can be RGB image data output from a traditional frame-based camera, or infrared / thermal imaging data, depth image / point cloud data; no specific limitation is made here.

[0040] Second modal data refers to data sampled at a significantly higher frequency than the first frequency (e.g., up to 10,000 times per second). Second modal data does not provide the complete semantics of the target, but rather specifically provides observations reflecting changes in target motion. Essentially, it is a differential signal of scene brightness in time or space, extremely sensitive to target motion, capable of capturing minute, instantaneous motion signatures. In specific implementations, second modal data can be temporal and spatial differential data output from a neuromorphic vision sensor, an asynchronous event stream output from a neuromorphic vision sensor, or raw intermediate frequency signals or velocity spectra provided by millimeter-wave radar or lidar; no specific limitations are imposed here.

[0041] The first and second frequencies are not fixed values. As long as the second frequency (e.g., the range is 757-10000 FPS) is significantly higher than the first frequency (e.g., the range is 10-60 FPS), the specific values ​​can be set according to the actual scenario. No specific limit is set here.

[0042] It should be noted that first-modal data (such as RGB images) is typically acquired periodically by visual sensors. Its processing involves complex calculations such as object detection and semantic segmentation, resulting in a low output frame rate. However, the data itself contains rich pixel-level texture and semantic information, thus first-modal data is low-frame-rate dense data. Second-modal data (such as spatiotemporal difference data) is directly generated from the temporal difference of the sensor's original signal, without the need for complex semantic parsing, and has extremely low processing latency, thus enabling high-frame-rate output. However, it only contains pixel-level motion change observations, with a single information dimension, thus second-modal data is high-frame-rate sparse data.

[0043] Based on the asynchronous sensing data of the warning area obtained in step S110, step S120 is further executed.

[0044] S120, based on the observation of the first modal data, determine and update the semantic state components in the target state vector; within the time interval between two adjacent updates of the semantic state components, based on the continuous observation of the second modal data, recursively predict and correct the motion state components in the target state vector.

[0045] This step is the core of the forward collision warning method provided in this embodiment to achieve ultra-low latency perception. The core concept is to decouple and asynchronously execute the target state update task based on the characteristics of the first modal data and the second modal data.

[0046] Specifically, firstly, a target state vector is initialized. The semantic state components of the target and motion state components Together they constitute, at least including the location of the target centroid. Target speed Target geometric dimensions and target category probability Information of equal dimensions.

[0047] The target centroid position can be determined by the observation results of the first modality data (such as the center point of the bounding box) and / or the preliminary localization of the motion response of the second modality data, and is represented as two-dimensional coordinates in the image coordinate system or vehicle coordinate system. The target's velocity can be set to the zero vector during initialization. Alternatively, an initial velocity value can be obtained by performing preliminary optical flow / motion estimation based on the second modal data within a short time window; the target geometry is directly obtained from the observation results of the first modal data, and is represented as the target width. and height The target category probability can be output by the semantic recognition network (such as a classifier) ​​of the first modality data, representing the probability distribution vector of the target belonging to each category (such as pedestrian, vehicle, non-motorized vehicle, etc.) in a predefined category set. .

[0048] When a new frame of first-modal data (such as an RGB image) is received, the system determines the semantic attributes of the target (such as class probability and size) based on the observations of the first-modal data through feature extraction and recognition, and updates the corresponding semantic state components in the target state vector. For example, the identified target class probability, bounding box width, and height are assigned to the semantic state components in the target state vector.

[0049] Within the time interval between two consecutive updates of semantic state components based on the first modality data, recursive prediction and correction of motion state components are continuously and frequently performed. Specifically, within this time window, the first modality data is in a blank or blind state, while the second modality data arrives continuously at an extremely high frequency. The system utilizes these continuous second modality data observations to perform a recursive processing on the motion state components (such as the target's position coordinates and velocity) in the target's state vector. This includes: first, predicting the motion state based on the target's kinematic model (such as a uniform velocity model) to calculate its possible state at the next moment; second, correcting the prediction results using the currently acquired second modality data observations to eliminate prediction errors, thereby achieving stable and high-frequency tracking of the target's trajectory. In this way, the target's motion information can be rapidly and continuously updated during the intervals when semantic information updates are slow.

[0050] Based on updating the target state vector in step S120, step S130 is further executed.

[0051] S130, determine the current target state vector based on the corrected motion state components and the current semantic state components in the target state vector, and perform forward collision warning based on the current target state vector.

[0052] This step is used to merge the asynchronous processing results obtained in the previous step S120, and make a final risk assessment based on the merged results.

[0053] Specifically, after the recursive correction in step S120, the latest corrected motion state components of the target are obtained. Simultaneously, in the system's state management, a current semantic state component is always maintained, which originates from the semantic information of the target most recently updated by the first modality data. This step combines these two pieces of information (the corrected motion state component and the current semantic state component) to form a complete current target state vector that represents the target's complete attributes at the current moment. The current target state vector simultaneously includes the target's "where and at what speed" (motion information) and "what it is and how big it is" (semantic information).

[0054] Based on the fused current target state vector, accurate risk assessment can be performed. For example, the future trajectory of the target can be predicted based on its position, speed, size, and the vehicle's status, and the estimated collision time can be calculated, or a comprehensive risk assessment can be performed by combining the target's category risk weights. Finally, when the assessment result meets the preset warning conditions (such as the estimated collision time being lower than a threshold), the system triggers and outputs a forward collision warning signal, thereby providing the driver or autonomous driving system with additional reaction time.

[0055] In this embodiment, asynchronous sensing data of the warning area is acquired. This asynchronous sensing data includes first modal data providing target semantic observation at a first frequency, and second modal data providing target motion change observation at a second frequency higher than the first frequency. Based on the observation of the first modal data, the semantic state components in the target state vector are determined and updated. During the time interval between two adjacent updates of the semantic state components, based on continuous observation of the second modal data, the motion state components in the target state vector are recursively predicted and corrected. Based on the corrected motion state components and the current semantic state components in the target state vector, the current target state vector is determined for forward collision warning. This method fuses high-frame-rate sparse motion data and low-frame-rate dense semantic data, continuously predicting and correcting the target motion state using motion data during the intervals of semantic data updates. Combined with the semantic tags provided by the semantic data, it achieves high-frequency, semantically-enabled target state output. This fundamentally solves the warning lag problem caused by image frame processing delay in traditional methods, while avoiding the disadvantages of large computational overhead and difficulty in deployment of complex fusion models, and ensuring the robustness of the warning system in extreme scenarios such as drastic changes in lighting.

[0056] Based on the above embodiments, the process of obtaining asynchronous sensing data within the warning area in step S110 will be described in detail below.

[0057] The first modality data includes RGB image data, and the second modality data includes temporal difference data and spatial difference data. Accordingly, asynchronous sensing data of the warning area is acquired, including: synchronously acquiring RGB image data, temporal difference data, and spatial difference data through a hybrid pixel sensor; or, acquiring an RGB image sequence output by an image sensor; obtaining RGB image data by downsampling the RGB image sequence; and obtaining temporal difference data and spatial difference data respectively by calculating the offset gradient of the RGB image sequence in the temporal and spatial dimensions.

[0058] It is easy to understand that the first modal data in this embodiment is RGB image data. RGB image data can provide rich semantic observations, such as color, texture, and target category. Its low frame rate and dense characteristics make it an ideal choice for providing reliable semantic anchors.

[0059] In this embodiment, the second modal data consists of temporal difference data and spatial difference data, which together constitute an efficient representation of the target's motion changes.

[0060] The core of temporal difference data lies in capturing the change in brightness over time at the same spatial location, i.e. This data is extremely sensitive to any movement of the target and can generate a response within the first millisecond of the target's appearance. It is the physical basis for achieving ultra-low latency sensing and directly reflects whether a change has occurred.

[0061] The core of spatial difference data lies in capturing the brightness gradient over the diagonal neighborhood at the same point in time, i.e. This data describes the target's contours, edges, and other structural information. Even if the target is temporarily stationary, its spatial differential signal still exists. Combining spatial differential data with temporal differential data can provide more accurate positioning and contour constraints for targets in different motion states, that is, it reflects what the changing contours are.

[0062] Temporal and spatial differential data can be acquired synchronously using hybrid pixel sensors. These sensors construct complementary visual pathways on the same focal plane—a COP pathway oriented towards cognition (color, absolute intensity) and an AOP pathway oriented towards motion (differential signals). A typical example is the Tianmouc sensor, which simultaneously transmits low-frequency RGB data and high-frequency spatiotemporal differential data through a hybrid pixel array and parallel readout architecture.

[0063] Specifically, the hybrid pixel sensor outputs first and second modal data at a fixed frame rate through global exposure. Typically, the frame rate of the first modal data is much lower than the frame rate of the second modal data. This is set within a time window. Inside, the sensor output is ,in Indicates the first The differential signal of the target in the spatial neighborhood at any given time; Indicates the first The differential signal of the target at any given moment in the time neighborhood; Indicates in RGB images captured at any time.

[0064] Synchronous acquisition via hybrid pixel sensors ensures that RGB images and spatiotemporal difference images have a unified temporal and spatial reference system.

[0065] Based on the Tianmouc chip, the pixels output by the COP and AOP paths are located on the same chip, in the same optical imaging plane, and share a unified optical coordinate system. Therefore, their output data... , and It has a fixed alignment spatial mapping relationship.

[0066] Furthermore, using the same photoelectric conversion front end, incident light is converted into charge through a photodiode, and charge transfer and sampling are completed under global shutter control, so that color / intensity data and differential data correspond to the same scene sampling time or sampling time window.

[0067] Ultimately, in each processing cycle (RGB frame cycle), a set of differential data with spatiotemporal alignment and spatial alignment of RGB and differential data input data can be obtained. This is used as input into the subsequent feature encoding and state estimation processes. Used to update the semantic state components in the target state vector, and and This is used for the entire period before the next RGB frame arrives. Within the time interval, the motion state components in the target state vector are continuously recursively predicted and corrected.

[0068] The temporal and spatial difference data can also be generated based on the RGB image frame sequence output by a traditional high-frame-rate CMOS image sensor.

[0069] Specifically, it acquires the RGB image sequence output by the CMOS camera at consecutive time points. Low-frequency first-mode data were obtained by downsampling and normalization. ,in Subsequently, calculations were performed based on the original RGB image sequences at adjacent time points. time( Pixel intensity variation at the same spatial location Time difference data was obtained. ,in The number of frames within this window is much greater than the number of RGB frames; and the spatial offset gradient is calculated based on the intensity difference between the target pixel in the RGB image at the current moment and its four diagonal neighbors (top left, bottom left, top right, and bottom right). This allows for the direct extraction of edge and contour features from the image. Similarly, within the time window... Within this, a series of spatial difference frames can be acquired, forming a set. . The overall characterization is within the time window Structural outline information of the interior scene.

[0070] The first and second modal data obtained through simulation naturally possess a unified temporal and spatial reference frame. For RGB image data... and , and Just captured All motion changes and structural information that occur in an RGB image during blind time within a given time period.

[0071] Finally, in each processing cycle (RGB frame cycle). Used in subsequent feature encoding and state estimation processes to update the semantic state components in the target state vector, and Used for the entire period before the next RGB frame arrives. Within the time interval, the motion state components in the target state vector are continuously recursively predicted and corrected.

[0072] Based on the above embodiments, the following will further describe in detail the process of determining and updating the target state vector in step S120.

[0073] What is easy to understand is the initial target state vector The semantic state components of the target and motion state components Together they constitute, at least including the location of the target centroid. Target speed Target geometric dimensions and target category probability Dimensional information.

[0074] The update of the target state vector includes the update of its semantic state components and the update of its motion state components. The semantic state components are updated based on the observation of RGB image data, while the motion state components are updated based on the observation of temporal difference data and spatial difference data. The following will take the first modality data as RGB image data and the second modality data as temporal difference data and spatial difference data as examples to explain the update process of the semantic state components and motion state components respectively.

[0075] In a specific embodiment, determining and updating the semantic state components in the target state vector based on the observations of the first modality data includes: performing representation mapping processing on the first modality data to obtain initial semantic features; performing feature encoding on the initial semantic features to extract semantic and contextual information to obtain semantic observation information; and updating the semantic state components in the target state vector based on the semantic observation information; wherein the semantic state components include the target's width, height, and category probability.

[0076] Specifically, when a new frame of RGB image data is received, the first representation function is used. The RGB image data is processed by representation mapping to convert it into a standard form suitable for subsequent deep network coding.

[0077] First characterization function The operations performed include, but are not limited to, pixel value normalization, image resizing, and tensor deformation. Pixel normalization refers to scaling intensity values ​​from [0,255] to [0,1] or performing mean-variance normalization; image resizing refers to uniformly scaling the image to a fixed resolution, such as 224x224; tensor deformation refers to converting H×W×C image data into a batch tensor format of B×C×H×W, where B is the batch size.

[0078] First characterization function The output is the initial semantic features. It is a normalized tensor that preserves the texture and color information of the original RGB image data.

[0079] Subsequently, the initial semantic features Input into a feature encoding network Feature coding network It is a deep neural network whose structure can be designed according to specific needs. In practical implementation, the feature encoding network... It can be constructed based on one or more of the following: convolutional structure, temporal modeling structure, attention structure, graph structure, frequency domain transform structure, and pyramid structure.

[0080] Among these, convolutional structures, such as those using CNN backbone networks like ResNet and VGG, extract multi-scale spatial features through multi-layer convolution and downsampling; attention structures, such as those introducing SE modules, CBAM, or the self-attention mechanism in Transformers, enable the network to focus on salient regions related to the target, enhancing feature discrimination; temporal coding structures, if processing consecutive image frames, can incorporate LSTM, GRU, or 3D convolutions to fuse temporal context information; and feature pyramid structures, such as FPN, are used to simultaneously acquire high-level features containing rich semantics and low-level features containing precise localization.

[0081] Feature coding network The output is a high-level feature map rich in semantic and contextual information. Based on this, by connecting specific detection heads (such as bounding box regression heads and classification heads), it is possible to obtain... Decoding specific semantic observation information .in, and These are the width and height observations of the target bounding box predicted by the network. It is a probability vector representing the observation probability that the target belongs to each of the preset categories (such as pedestrians, vehicles, and non-motorized vehicles).

[0082] After obtaining the semantic observation information, it is used to update the unified target state vector. The semantic state components in the target state vector. Specifically, the width of the target state vector. w ,high h and category probability p The data will be updated, and the update strategy can be to directly assign values ​​to the observations. This specific embodiment ensures that the semantic information (identity and size) of the target is reflected in the latest and most reliable way in the state vector.

[0083] In another specific embodiment, the recursive prediction and correction of the motion state components in the target state vector is achieved by a state estimation algorithm, which includes Kalman filtering or extended Kalman filtering.

[0084] Within the time interval between two consecutive updates of semantic state components, based on continuous observations of the second modality data, the motion state components in the target state vector are recursively predicted and corrected. This includes: based on the target state vector at the current moment, using a state transition model to predict the target's motion state at the next moment, thus obtaining the predicted motion state; based on the second modality data at the current moment, obtaining the target's motion observation information; and based on the motion observation information, correcting the predicted motion state to update the motion state components in the target state vector. The motion state components include the target's position and velocity.

[0085] It is easy to understand that at any high-frequency processing moment Based on the previous moment The estimated target state vector The predicted state is obtained by extrapolating and predicting the motion state using a state transition model. State transition models describe the intrinsic laws governing the motion of a target, such as the most commonly used uniform motion model: , This model predicts the target's motion state (position and velocity) at the next time step. It's worth noting the semantic components in the state vector. This prediction step typically remains unchanged, i.e. ， ， This reflects the inheritance of semantic information.

[0086] At the same time, based on the current situation Time difference data of the arrival of the blind time before the arrival of the next RGB frame. Spatial difference data It is necessary to extract motion observation information that can be used to correct predictions. .

[0087] Specifically, firstly, the time difference data were analyzed separately. Spatial difference data Applying the second representation function and the third representation function The second characterization function and the third representation function Operations such as noise suppression (e.g., LVTF denoising, time-domain filtering), signal enhancement, and feature aggregation (e.g., DBSCAN clustering) are performed to extract robust initial motion features from sparse, noisy raw differential signals. and .

[0088] Next, the extracted initial motion features of time will be... and initial motion characteristics in space Through a difference fusion function The fusion is performed, and the output of the difference fusion function is the spatiotemporal change response map. This is a feature map where the highlighted areas comprehensively represent the salient location and contour of moving targets in the scene. The difference fusion function... Various strategies can be employed, such as weighted summation, concatenation, or gating operations, without specific limitations here.

[0089] For the preprocessing and fusion process of temporal difference data and spatial difference data, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This diagram illustrates the preprocessing and fusion process of temporal and spatial difference data provided in an embodiment of the present invention. Figure 2 As shown, the input temporal and spatial difference data are first preprocessed separately. Temporal difference data flows through a foreground enhancement module to strengthen areas of significant dynamic change, while spatial difference data is refined through a contour refinement module to enhance target edges and structural details. Subsequently, the temporal difference branch enters a normalization module to unify the data scale, while the spatial difference branch enters a contour enhancement module to further enhance edge saliency. Furthermore, a background compensation module eliminates static background interference, ensuring that the focus remains on the dynamic target. Finally, the processed temporal and spatial difference features are fed into a spatiotemporal fusion module, which deeply integrates the motion dynamics information in the temporal dimension with the structural contour information in the spatial dimension, outputting the final spatiotemporal change response map. This response map not only preserves the target's high-frequency motion trajectory but also integrates clear spatial contours and foreground saliency, providing high-quality, low-latency perceptual input for subsequent asynchronous state updates and collision warnings.

[0090] Finally, from the spatiotemporal change response diagram The motion observation information of the target is decoded through centroid calculation, optical flow method, or a small regression network. That is, the observed target position and velocity.

[0091] Obtain motion observation information Then, a state estimation algorithm is used to predict the state. The state estimation algorithm here can be either Kalman filtering or extended Kalman filtering. The correction process is the core update step of the Kalman filter: calculating the predicted state. Predicted motion state (position and velocity) and motion observation information The residuals between them are then used to calculate an optimal Kalman gain. The residuals are weighted, and the weighted residuals are used to correct the predicted state. The predicted motion state (position and velocity) is obtained to obtain the current state. The current target state vector estimated at the optimal time. The motion state components in the equation.

[0092] The Kalman gain is dynamically calculated based on the uncertainty (covariance) of the predicted state and the observation noise, ensuring that the prediction is trusted when it is reliable, and the observation is trusted when the observation is reliable. Through the cyclic recursion of prediction-observation-correction, the motion state components (position and velocity) in the target motion state vector can be continuously, smoothly and with high precision tracked and updated.

[0093] It should be noted that Kalman filtering is an optimal recursive state estimation algorithm suitable for linear Gaussian systems. It maintains a state vector and its error covariance matrix to optimally estimate the state of a dynamic system under noisy observations. Extended Kalman filtering is a generalization of Kalman filtering to nonlinear systems. By approximating the nonlinear function with a first-order Taylor expansion, it makes it applicable to the nonlinear motion or observation models that may exist in this embodiment. Both algorithms can achieve efficient state recursion with extremely low computational complexity, perfectly meeting the requirements of low computational resource cost and high-frequency updates in this embodiment. They can effectively fuse high-frequency differential observations with low-frequency semantic anchors asynchronously.

[0094] Based on the above embodiments, the following will further describe in detail the process of forward collision warning in step S130 based on the current target state vector.

[0095] It is important to understand that after the recursive correction in step S120, the latest corrected motion state components of the target are obtained. Simultaneously, in the system's state management, a current semantic state component is always maintained. This component originates from the semantic information of the target most recently updated by the first modality data. By combining these two pieces of information (the corrected motion state component and the current semantic state component), a complete current target state vector representing the target's full attributes at the current moment can be obtained. The current target state vector simultaneously contains both the target's "where and at what speed" (motion information) and "what it is and how big it is" (semantic information).

[0096] Based on this, this embodiment provides two warning methods: threshold warning and comprehensive risk assessment warning, which correspond to warning requirements with different levels of complexity and accuracy.

[0097] In one specific embodiment, a threshold-based early warning method is described.

[0098] Forward collision warning is performed based on the current target state vector, including: predicting the future trajectory of the target based on the current target state vector, and calculating the estimated collision time based on the future trajectory; if the estimated collision time is less than a preset time threshold, a collision warning is output.

[0099] Specifically, the current target state vector Includes the target in The optimal estimated state at time t, including but not limited to its centroid coordinates. Speed ​​of movement And the possible motion trends. Using these motion state components, a multi-step motion state prediction function is used. It can project the target into a future time window. the movement trajectory within. The multi-step motion state prediction function is constructed based on preset kinematic models (such as constant velocity model and constant acceleration model), whose input is the current state (particularly referring to the current motion state component), and the output is the k th step (corresponding to the future time t+k ) predicted state . Through continuous calculation k=1,2,... until ΔT , the predicted position sequence of the target in a future period of time, that is, the future movement trajectory, can be obtained.

[0100] After obtaining the future movement trajectory of the target, combined with the current motion state (position, speed, heading) of the ego vehicle and its predicted trajectory (usually assumed to be constant linear velocity or traveling along the lane line), the time when the ego vehicle and the target first spatially overlap on the future trajectories can be calculated. The time difference between this time and the current time is the estimated time to collision. Time To Collision (TTC) is an intuitive physical quantity that directly reflects the urgency of collision risk, and its calculation formula can be simplified as the relative distance between the two parties divided by the relative speed (component along the collision direction).

[0101] Compare the calculated TTC value with one or more preset time thresholds. The preset time threshold is a safety boundary calibrated during system design according to factors such as vehicle braking performance, driver reaction time, and safety redundancy. For example, a two-level warning can be set: when TTC < T1 (e.g., 2.5s), a primary warning such as an audio prompt is triggered; when TTC < T2 (e.g., 1.5s), emergency braking is triggered and the system intervenes in braking. Generally, T1 is greater than T2. Once TTC is lower than any set threshold, a collision warning signal is output immediately. The threshold warning method provided in this embodiment has clear logic, simple calculation and rapid response, and is an effective means to meet basic warning functions.

[0102] In another specific embodiment, the comprehensive risk assessment and warning method is described.

[0103] Performing forward collision warning according to the current target state vector includes: predicting the future movement trajectory of the target based on the current target state vector, and determining the category probability of the target; calculating the estimated time to collision according to the future movement trajectory, and calculating the overlap degree between the future movement trajectory and the risk area; calculating a comprehensive risk score according to the overlap degree, the estimated time to collision and the category probability; outputting a collision warning when the comprehensive risk score is greater than a preset risk threshold.

[0104] Specifically, the future trajectory of the target is predicted based on the current target state vector, and the Time To Chance (TTC) is calculated as the time dimension basis for risk assessment. This process can be referenced from the implementation of the threshold-based early warning method, and will not be elaborated here.

[0105] Based on the prediction of future motion trajectories, for each predicted future moment... Calculate the predicted bounding box of the target. With predefined risk areas The degree of spatial overlap between them. The risk area is a dynamic or static area used to characterize the safe space of the vehicle, which typically includes: the vehicle's driving lane area, i.e., the lane area that the vehicle currently occupies and plans to occupy in the future; the braking safety area, i.e., the area that can be safely stopped in the future after considering the vehicle's maximum deceleration; and the collision hazard area, i.e., a tightly coupled buffer zone around the vehicle's outline.

[0106] Calculate the predicted bounding box With predefined risk areas The intersection-union ratio, i.e. This value quantifies the degree of spatial intrusion of the target's future location into the vehicle's safe space, and is a key indicator of the spatial dimension of risk assessment.

[0107] Simultaneously, the category probability vector in the semantic state component is directly obtained from the current target state vector. It also extracts the category probability of the target belonging to a high-risk category (such as pedestrians, non-motorized vehicles, two-wheeled vehicles, etc.). The probability This reflects the inherent risk attributes of the target, as different categories of traffic participants have varying vulnerabilities in accidents and different impacts on the severity of collision outcomes.

[0108] The risk indicators from the three dimensions mentioned above are weighted and integrated to calculate a comprehensive risk score. The specific formula is as follows .in, This indicates the overlap between the predicted target area (predicted bounding box) and the risk area; the larger the value, the higher the risk. TTC indicates the estimated collision time; the smaller the value, the higher the risk. To prevent a small constant with a denominator of 0 when TTC is 0; This indicates the probability that the target belongs to a high-risk category such as pedestrians, vehicles, or non-motorized vehicles. This is a weighting coefficient used to balance the contribution ratio of the three dimensions of risk in the overall score, and can be adjusted according to vehicle type, road conditions, driving strategy, etc.

[0109] The calculated comprehensive risk score Compare with a preset risk threshold. The preset risk threshold was calibrated through extensive scenario testing and data-driven analysis. When When the risk level exceeds the preset risk threshold, it indicates that the overall risk level of the target has exceeded the safety boundary set by the system, and the system will then output a collision warning signal.

[0110] This embodiment, through the fusion of multi-dimensional information, can identify scenarios that may not yet have fallen below the emergency threshold but are potentially high-risk due to obvious spatial intrusion or high-risk target types earlier and more accurately, thus achieving a smarter and more robust early warning.

[0111] In some embodiments, a detailed description is provided of the high-frequency semantic detection and early warning process for "ghost peek" scenarios.

[0112] It's easy to understand that when pedestrians, non-motorized vehicles, or other targets suddenly veer laterally into the vehicle's path from behind static obstructions (such as parked vehicles, building walls, or green belts), this scenario is known as a "ghost peek." This scenario presents several challenges: the time it takes for a target to go from being completely obscured to being fully exposed to the risk path is extremely short; traditional RGB-frame-based perception systems cannot detect the target before it is fully exposed to clear vision due to the obstruction; and from the point where the target becomes partially visible and a collision is possible, the system has only a few hundred milliseconds to react.

[0113] In other words, relying on a single perceptual modality (whether RGB only or event only) is insufficient in the "ghost peek" scenario: RGB images have incomplete information when the target is occluded or just peeks out, and there is a large inter-frame delay; while pure event streams have a fast response, they lack stable semantic labels. Therefore, fusing low-frame-rate dense RGB data with high-frame-rate sparse differential data becomes the key to achieving timely and reliable early warning.

[0114] Specifically, firstly, step 1 is executed, using a binocular hybrid pixel sensor (specifically, the Tianmouc sensor can be selected) deployed in front of the vehicle to simultaneously acquire multi-source input data streams of the warning area, including low-frame-rate dense RGB image data and high-frame-rate sparse differential data. The dense RGB image data, for example, outputs standard color image frames at a fixed frequency of 30 FPS, providing rich color and texture information; the sparse differential data includes temporal and spatial differential data, with a theoretically maximum event output frequency of 10,000 FPS, capable of capturing brightness changes at any point in the scene with sub-millisecond resolution.

[0115] Then, step 2 is performed to extract features in parallel from the synchronously acquired multi-data stream. The acquired RGB image data is input into a pre-trained ResNet50 convolutional neural network, which serves as a semantic feature encoder. Deep feature extraction is performed on RGB image data to output the initial semantic observation state of the target. . It includes preliminary estimates of semantic attributes such as category and size of potential targets (even partially occluded ones) in RGB image data using the ResNet50 convolutional neural network.

[0116] Simultaneously, the synchronously acquired high-frequency temporal and spatial difference data (usually accumulated into event frames) are input into a lighter-weight ResNet18 network. This network acts as a motion feature encoder, used to extract motion patterns from the sparse difference signal and output the initial motion observation state of the target. . It mainly includes initial estimates of the location and movement trend of the moving target.

[0117] Based on the initial semantic observation state and initial motion observation state Perform state initialization and construct the initial unified state vector of the target. And initialize the state covariance matrix for subsequent Kalman filtering. .

[0118] Next, steps 3-5 are executed, performing asynchronous state recursion updates based on Kalman filtering. After initialization, an asynchronous state recursion loop driven by high-frequency differential data is entered. This loop runs continuously throughout the entire time interval between two adjacent RGB image frames, and its core is the prediction-update process of Kalman filtering.

[0119] Perform step 3 (prediction) at any high-frequency recursion time. Based on the state estimation of the previous time step Through the state transition model (by the state transition matrix) The description (usually based on the assumption of uniform motion) is extrapolated to obtain the predicted state at the current moment. .in, Represents the scale-guided mapping matrix. Indicates the scale guide item.

[0120] Perform step 4 (prediction covariance) to calculate the uncertainty of the predicted state, i.e., the prediction error covariance matrix at the current time. = .in It is the state covariance matrix of the previous time step. It is the process noise covariance matrix, representing the uncertainty of the model.

[0121] Perform step 5 (updated), utilizing the current... The differential data arriving at each time point are quickly encoded using a ResNet18 network to obtain new motion observation information. Then, a Kalman update is performed: the Kalman gain is calculated. It is based on the prediction error covariance matrix and observation noise Determine the extent to which new observations should be trusted; calculate the observation residuals. That is, the difference between actual observations and predicted observations, among which The observation matrix is ​​used; the predicted state is corrected using Kalman gain-weighted residuals to obtain the high-frequency semantic state after differential observation correction. Update the estimated error covariance matrix. In the update, the motion state components (position, velocity) are... Significant corrections were made, while the semantic state components (category, size) largely retained their estimates from the previous RGB frame. The unchanged nature of semantics allows for its propagation over time. After each recursive loop in steps 3-5 is completed, a high-frequency semantic state containing the latest motion information and current semantic information is obtained. Based on this continuous state flow, risk assessment can be performed in real time, determining whether the target appears from behind the occlusion area (based on historical trajectories and scene priors), whether the target's movement trend points towards the vehicle's risk area (driving lane), and calculating instantaneous risk indicators, such as based on... TTC.

[0122] Once a continuous sequence of states indicates that the target meets the preset "ghost peek" risk conditions, such as appearing from an occluded area, having a high lateral velocity, or having a TTC less than a threshold, the "ghost peek" detection results and warning information are immediately output without waiting for the processing of the next frame of RGB image to be completed.

[0123] It is worth mentioning that in the "ghost peek" scenario of this embodiment, when the edge of a pedestrian's body just begins to emerge from the occlusion, the RGB image may not yet provide a sufficiently clear and complete outline for the detection network to recognize. However, this tiny, rapidly moving edge will immediately generate a strong response signal in the temporal difference data and can outline a preliminary outline in the spatial difference data. This embodiment, through the Kalman filter framework, can quickly initialize and update the state of a moving target in this early stage using these sparse but timely differential observations, and can replace the semantic label of the pedestrian provided by the RGB in the previous frame (or at initialization) with the semantic label of the pedestrian. This trajectory is assigned to the target. Therefore, before the target fully enters the clear RGB field of view and is confirmed by a traditional detector, this embodiment performs continuous tracking and risk assessment with semantic tags for hundreds of milliseconds, thereby achieving ultra-low latency early warning of "ghost peek" events.

[0124] In some embodiments, the process of estimating and recognizing high-frequency motion information in scenarios with extreme lighting changes is described in detail.

[0125] It's easy to understand that in the actual operation of autonomous driving and advanced driver assistance systems, vehicles often encounter extreme environments with drastic and rapid changes in lighting conditions. For example, at tunnel entrances and exits, vehicles instantly transition from a high-brightness environment to a low-brightness tunnel (or vice versa), requiring the camera to quickly adjust the exposure. During this process, the image is prone to large areas of overexposure or underexposure. At night, when meeting oncoming traffic or in strong backlight, the high beams of oncoming vehicles or strong ambient backlight can cause local or global saturation of the image sensor, obscuring the target outline in the halo and completely losing texture details. During instantaneous flashes of light and shadows, such as when traversing forest roads, the image brightness fluctuates drastically and irregularly between frames.

[0126] In the aforementioned scenarios, perception schemes relying on traditional RGB images face severe challenges: lag in automatic exposure control leads to overexposure, underexposure, and local saturation; high dynamic range scenes exceed the sensor's linear response range, causing loss of detail and blurred contours; and dramatic inter-frame brightness changes introduce significant noise. These factors collectively cause a severe decline in the performance of target detection, recognition, and tracking algorithms based on image texture and color, or even complete failure. However, the core mechanism of differential signals (especially temporal differential) lies in detecting brightness changes rather than absolute brightness values, thus exhibiting inherent insensitivity to the aforementioned large-scale global or local brightness shifts and saturation. As long as the target is moving, the relative brightness changes caused by its edges can be effectively captured by the differential sensor. This allows for the continued perception of dynamic targets even under extreme lighting conditions where RGB modes fail, relying on spatiotemporal differential data.

[0127] Specifically, in the extreme lighting scenarios addressed in this embodiment, RGB image data may be temporarily marked as unreliable or invalid by the system due to severe quality degradation. In this case, the system switches to a degraded sensing mode that primarily uses spatiotemporal difference data. Through a hybrid pixel sensor, high frame rates (theoretically up to 10000 FPS) of sparse spatiotemporal difference data are continuously acquired, including temporal and spatial difference data. Even if the RGB channels output invalid frames due to exposure issues, the sensor's differential data path typically still functions normally.

[0128] Since RGB semantic information is unreliable, the initialization of this embodiment relies entirely on spatiotemporal difference data: (1) Motion trend and contour extraction: Analyze the spatial difference data within the current time window to extract significant edge and contour information in the scene. These contours may correspond to the boundaries of targets such as vehicles and pedestrians. At the same time, analyze the temporal difference data to determine which spatial locations are continuously generating brightness changes, thereby identifying the motion area; (2) Initial state estimation: Spatially associate and fuse the significant contours extracted by spatial difference with the motion areas identified by temporal difference to initially determine the position of potential moving targets. Furthermore, by analyzing the distribution changes of time-difference events over a short period of time, the target's velocity can be roughly estimated. Based on this, an initial state estimate of the target is constructed. The initial state contains only motion state components because reliable semantics (category, precise size) are difficult to obtain in the absence of a clear RGB image.

[0129] In the absence of reliable RGB image data updates, it enters a high-frequency recursive tracking loop in a pure motion state, relying entirely on continuous differential observations to maintain and correct the target state.

[0130] First, state prediction is performed. This occurs at each high-frequency processing time. Based on the optimal motion state estimate of the previous moment Using the state transition model (derived from the state transition matrix) This indicates that the predicted motion state at the current moment is obtained by extrapolation using a uniform velocity model. Simultaneously, the uncertainty of the state prediction is updated, namely the prediction error covariance matrix at the current moment. .in It is the state covariance matrix of the previous time step. It is the process noise covariance matrix, representing the uncertainty of the model.

[0131] Then, perform a state update. Based on the current... The spatiotemporal difference data at each moment is used to extract the target's motion observation information through lightweight processing (such as event accumulation, centroid calculation, and optical flow estimation). It usually includes the observed location. Sometimes it may also include the observation speed. ; Calculate Kalman gain and utilize motion observation information For predicting motion state After correction, the optimal motion state estimate updated at the current time is obtained. And update the error covariance matrix. In this loop, the state vector contains only... semantic state components It will not be updated or will remain as the default / unknown value.

[0132] It is worth mentioning that although this embodiment cannot obtain the precise category and size of the target under extreme lighting conditions, the continuous and reliable high-frequency motion state estimation itself can provide crucial risk avoidance information, such as: outputting high-frequency weak semantic information, including the target's continuous motion trajectory, instantaneous velocity, and direction of motion; and based on the recursively obtained motion state sequence... The system performs a risk assessment based on motion attributes; determines whether the predicted trajectory of the target enters the vehicle's preset risk zone; calculates the estimated time of collision (TTC) based on the motion state; analyzes the target's direction of motion, acceleration, etc., to identify whether it exhibits a dangerous trend related to collision (such as sudden lane changes, accelerating towards the target); and calculates a risk level based on motion attributes by combining the target's trajectory, TTC, and relationship with the risk zone. When the risk level exceeds a preset threshold, a downgrade warning is triggered. This warning may not specify the target type, but it will clearly indicate that there is a fast-moving object ahead and a collision risk, thereby reminding the driver to intervene or trigger a more conservative automatic emergency braking strategy.

[0133] In other words, although this embodiment cannot obtain the complete semantic information of RGB image data under extreme lighting conditions, it can still achieve continuous motion estimation and motion attribute recognition of dynamic targets based on high-frequency differential signals and state transition models, thereby ensuring that the system still has a certain perception and early warning capability under the condition that RGB image data fails.

[0134] In other embodiments, Figure 3 The diagram illustrates the multimodal asynchronous state estimation and fusion mechanism of the forward collision warning method provided in this embodiment of the invention. It specifically describes how to combine low-frequency RGB semantics and high-frequency differential motion information to generate a target motion state with high temporal resolution and semantic attributes.

[0135] like Figure 3 As shown, firstly, low-frequency RGB image data is received as input. A deep semantic feature extraction network is used to extract the target's deep semantic features, and semantic state initialization is completed accordingly, establishing the target's initial category, size, and other strong semantic attributes. Simultaneously, high-frequency differential data is received as input. Motion response and change intensity are extracted through differential feature encoding to complete motion state initialization, establishing the target's position, velocity, and other dynamic attributes.

[0136] Subsequently, at consecutive high-frequency moments, the core high-frequency semantic state estimation is performed. On one hand, the semantic state is used to predict the semantic prior at the current high-frequency moment through a state transition model; on the other hand, the motion state is used to provide geometric and dynamic constraints for semantic updates through motion constraints. Based on this, new high-frequency observations are received, and the high-frequency motion observations are fused with the low-frequency semantic priors through a state update mechanism (such as the correction step of Kalman filtering), thereby outputting semantically meaningful high-frequency motion information at each high-frequency moment. The entire process forms a closed loop, which not only ensures the coherence of the semantics (from low-frequency RGB) but also endows the semantic information with extremely high temporal resolution (from high-frequency difference), achieving accurate and continuous estimation of the target state with extremely low latency.

[0137] Corresponding to the forward collision warning methods described in the above embodiments, the present invention also proposes a forward collision warning device. Specifically, Figure 4 A schematic diagram of the forward collision warning device provided in an embodiment of the present invention is shown.

[0138] like Figure 4 As shown, the device includes: an asynchronous sensing data acquisition module 410, used to acquire asynchronous sensing data of the warning area, the asynchronous sensing data including first modal data providing target semantic observation at a first frequency, and second modal data providing target motion change observation at a second frequency higher than the first frequency; a target state vector update module 420, used to determine and update the semantic state components in the target state vector based on the observation of the first modal data; and to recursively predict and correct the motion state components in the target state vector based on continuous observation of the second modal data within the time interval between two adjacent updates of the semantic state components; and a forward collision warning decision module 430, used to determine the current target state vector based on the corrected motion state components and the current semantic state components in the target state vector, and to perform a forward collision warning based on the current target state vector.

[0139] In this embodiment, asynchronous sensing data of the warning area is acquired. This asynchronous sensing data includes first modal data providing target semantic observation at a first frequency, and second modal data providing target motion change observation at a second frequency higher than the first frequency. Based on the observation of the first modal data, the semantic state components in the target state vector are determined and updated. During the time interval between two adjacent updates of the semantic state components, based on continuous observation of the second modal data, the motion state components in the target state vector are recursively predicted and corrected. Based on the corrected motion state components and the current semantic state components in the target state vector, the current target state vector is determined, and a forward collision warning is performed based on the current target state vector. This device, by fusing high-frame-rate sparse motion data and low-frame-rate dense semantic data, continuously predicts and corrects the target motion state using motion data during the intervals between semantic data updates. Combined with the semantic tags provided by the semantic data, it achieves high-frequency, semantically-enabled target state output. This fundamentally solves the warning lag problem caused by image frame processing delay in traditional methods, while avoiding the disadvantages of large computational overhead and difficulty in deployment of complex fusion models, and ensuring the robustness of the warning system in extreme scenarios such as drastic changes in lighting.

[0140] It should be noted that the forward collision warning device provided in the embodiments of the present invention can be referred to in correspondence with the forward collision warning methods described in the above embodiments, and will not be repeated here.

[0141] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 830, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the forward collision warning method provided in the above embodiments.

[0142] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the forward collision warning method provided in the above embodiments.

[0144] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the forward collision warning method provided in the above embodiments.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A forward collision warning method, characterized in that, include: Acquire asynchronous sensing data of the warning area, wherein the asynchronous sensing data includes first modal data that provides target semantic observation at a first frequency, and second modal data that provides target motion change observation at a second frequency higher than the first frequency; Based on the observations of the first modal data, the semantic state components in the target state vector are determined and updated; within the time interval between two consecutive updates of the semantic state components, based on the continuous observations of the second modal data, the motion state components in the target state vector are recursively predicted and corrected. The current target state vector is determined based on the corrected motion state components and the current semantic state components in the target state vector, and a forward collision warning is performed based on the current target state vector.

2. The forward collision warning method according to claim 1, characterized in that, The first modal data includes RGB image data, and the second modal data includes temporal difference data and spatial difference data; Accordingly, the asynchronous sensing data of the early warning area includes: The RGB image data, the temporal difference data, and the spatial difference data are acquired synchronously using a hybrid pixel sensor. or, The process involves acquiring an RGB image sequence output from an image sensor; performing downsampling on the RGB image sequence to obtain RGB image data; and calculating the offset gradients of the RGB image sequence in the time and spatial dimensions to obtain the temporal difference data and the spatial difference data, respectively.

3. The forward collision warning method according to claim 1, characterized in that, The observation based on the first modality data, determining and updating the semantic state components in the target state vector, includes: The first modality data is subjected to representation mapping processing to obtain initial semantic features; The initial semantic features are encoded to extract semantic and contextual information, thereby obtaining semantic observation information; Based on the semantic observation information, update the semantic state components in the target state vector; The semantic state components include the target's width, height, and category probability.

4. The forward collision warning method according to claim 2, characterized in that, The recursive prediction and correction of the motion state components in the target state vector based on continuous observations of the second modality data during the time interval between two adjacent updates of the semantic state components includes: Based on the target state vector at the current moment, the motion state of the target at the next moment is predicted by using the state transition model, and the predicted motion state is obtained. Based on the second modal data at the current moment, obtain the target's motion observation information; Based on the motion observation information, the predicted motion state is corrected to update the motion state components in the target state vector; The motion state components include the target's position and velocity.

5. The forward collision warning method according to claim 4, characterized in that, The acquisition of target motion observation information based on the second modal data at the current moment includes: The temporal difference data and the spatial difference data are respectively characterized and mapped to obtain the initial motion features; The initial motion features are weighted and fused to obtain a spatiotemporal change response map; The motion observation information is obtained based on the spatiotemporal change response map.

6. The forward collision warning method according to claim 1, characterized in that, The forward collision warning based on the current target state vector includes: Predict the future trajectory of the target based on the current target state vector, and calculate the estimated collision time based on the future trajectory; If the expected collision time is less than a preset time threshold, a collision warning will be output.

7. The forward collision warning method according to claim 1, characterized in that, The forward collision warning based on the current target state vector includes: Predict the future trajectory of the target based on the current target state vector, and determine the target's category probability; The estimated collision time is calculated based on the future trajectory, and the overlap between the future trajectory and the risk area is also calculated. A comprehensive risk score is calculated based on the overlap, the estimated collision time, and the category probability. If the overall risk score is greater than a preset risk threshold, a collision warning will be output.

8. The forward collision warning method according to claim 7, characterized in that, The calculation of a comprehensive risk score based on the overlap, the estimated collision time, and the category probability includes: The comprehensive risk score is obtained by multiplying the overlap, the reciprocal of the expected collision time, and the category probability by their respective weighting coefficients and then summing them.

9. The forward collision warning method according to claim 3, characterized in that, The initial semantic features are encoded using one or more of the following structures: convolutional structure, temporal modeling structure, attention structure, graph structure, frequency domain transform structure, and pyramid structure, to obtain the semantic observation information.

10. A forward collision warning device, characterized in that, include: An asynchronous sensing data acquisition module is used to acquire asynchronous sensing data of the warning area. The asynchronous sensing data includes first modal data that provides target semantic observation at a first frequency, and second modal data that provides target motion change observation at a second frequency higher than the first frequency. The target state vector update module is used to determine and update the semantic state components in the target state vector based on the observation of the first modality data; and to recursively predict and correct the motion state components in the target state vector based on the continuous observation of the second modality data during the time interval between two adjacent updates of the semantic state components. The forward collision warning decision module is used to determine the current target state vector based on the corrected motion state components and the current semantic state components in the target state vector, and to perform forward collision warning based on the current target state vector.

Images