A vehicle control method and system based on multi-source perception data fusion

Abstract

The application relates to the technical field of vehicle control, and provides a vehicle control method and system based on multi-source perception data fusion, which comprises the following steps: acquiring image data of the front of a vehicle collected by a camera in a storage device of the vehicle; judging whether a front vehicle of the vehicle is an intrusion vehicle according to the image data; when the front vehicle of the vehicle is the intrusion vehicle, acquiring radar data of the front of the vehicle collected by a millimeter wave radar in vehicle control based on multi-source perception data fusion of the storage device, and determining a real relative distance between the intrusion vehicle and the vehicle along a driving direction of the vehicle according to the latest image data and the radar data in the storage device; the latest image data is image data with the latest time stamp in the storage device; and control strategy information of the vehicle is determined according to the real relative distance between the intrusion vehicle and the vehicle. Through the scheme, the efficiency of vehicle control can be improved.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and in particular to a vehicle control method and system based on multi-source perception data fusion. Background Technology

[0002] In highway driving environments, vehicle control systems typically rely on multi-source perception data fusion to cope with complex traffic interaction scenarios. The conventional processing flow involves the fusion module performing spatiotemporal integration and trajectory state estimation on lane lines, relative distances, speeds, and spatial contours output by sensors such as cameras and millimeter-wave radar. The control module then determines actions such as following, decelerating, or maintaining lane position.

[0003] However, when vehicles merge from highway ramps onto the main road, there are often high-dynamic scenarios where merging vehicles suddenly accelerate and cut into front of the vehicle within a short period of time. In this scenario, because cameras observe the merging vehicle's tendency to cross the line earlier, while millimeter-wave radar is more sensitive to longitudinal relative speed changes, there is an inherent time lag between the updates of the lateral and longitudinal motion states of the same target by different sensors. Existing fusion mechanisms, in order to meet real-time requirements, usually synthesize the data from each source directly at a uniform current period. This leads to asynchrony in the lateral and longitudinal motion states of the merging vehicle during fusion. The target vehicle may still be judged as an approaching target in an adjacent lane in the fusion output, rather than a risky target that has already entered the front of the vehicle's lane, and its merging depth will also be underestimated.

[0004] This existing method, which relies on fixed-cycle synchronization, is difficult to accurately compensate for the time difference in the update of each sensor for the lateral intrusion and longitudinal speed changes of highly dynamic targets in a very short time. This results in a lag in the priority judgment of risk objects, a late triggering of longitudinal deceleration of the vehicle, and a serious mismatch between the chassis's actions and the actual entry situation. Summary of the Invention

[0005] This application provides a vehicle control method and system based on multi-source perception data fusion, aiming to solve the technical problems in the prior art that cause the risk assessment of the vehicle cutting in and the control strategy to be inaccurate due to the asynchronous update of camera and millimeter-wave radar data.

[0006] The technical solution of this application is as follows:

[0007] In a first aspect, this application discloses a vehicle control method based on multi-source perception data fusion, comprising: acquiring image data of the front of the vehicle collected by a camera in the vehicle's storage device; determining whether the vehicle in front of the vehicle is an intruding vehicle based on the image data; when the vehicle in front of the vehicle is an intruding vehicle, acquiring radar data of the front of the vehicle collected by the vehicle's millimeter-wave radar in the storage device, and determining the true relative distance between the intruding vehicle and the vehicle along the vehicle's driving direction based on the latest image data and radar data in the storage device; the latest image data is the image data with the latest timestamp in the storage device; and determining the vehicle's control strategy information based on the true relative distance between the intruding vehicle and the vehicle.

[0008] Optionally, the image data includes the latest image data and the first image data stored in the storage device. The first image data is the image data of the frame preceding the latest image data. Determining whether the vehicle in front of the vehicle is an intruding vehicle based on the image data includes: determining a first distance based on the latest image data; the first distance is the distance between the vehicle in front of the vehicle and the lane line of the lane where the vehicle is located along a first direction; the first direction is a direction perpendicular to the vehicle's direction of travel on the ground plane; determining a second distance based on the first image data; the second distance is the distance between the vehicle in front of the vehicle and the lane line of the lane where the vehicle is located along the first direction; using the difference between the first distance and the second distance as a first physical distance, and using the ratio of the first physical distance to a first duration as a first speed of the vehicle; the first duration is the frame duration between the latest image data and the first image data; and determining whether the vehicle in front of the vehicle is an intruding vehicle based on the first speed.

[0009] Optionally, determining whether the vehicle ahead of the vehicle is an intruding vehicle based on the first speed includes: determining whether the first speed is greater than a first preset speed threshold; determining that the vehicle ahead is not an intruding vehicle when the first speed is less than or equal to the first preset speed threshold; obtaining the width of the target lane when the first speed is greater than the first preset speed threshold; the target lane is the lane where the vehicle is located; determining the minimum distance between the intruding vehicle and the edge of the target lane based on the latest image data; using the ratio of the target distance to the first speed as a second duration; the target distance is the sum of the target width and the minimum distance; the target width is half the width of the target lane; and determining whether the vehicle ahead of the vehicle is an intruding vehicle based on the second duration.

[0010] Optionally, determining whether the vehicle in front of the vehicle is an intruding vehicle based on the second duration includes: obtaining the vehicle's current speed and a first correspondence; the first correspondence includes a one-to-one correspondence between multiple speed ranges and multiple first duration thresholds; taking the first duration threshold corresponding to the speed range where the vehicle's current speed is located in the first correspondence as the target first duration threshold; determining whether the second duration is greater than the target first duration threshold; if the second duration is greater than the target first duration threshold, determining that the vehicle in front of the vehicle is not an intruding vehicle; if the second duration is less than or equal to the target first duration threshold, determining that the vehicle in front of the vehicle is an intruding vehicle.

[0011] Optionally, the radar data includes the latest radar data in the storage device, where the latest radar data is the radar data with the latest timestamp in the storage device. Based on the latest image data and radar data in the storage device, the true relative distance between the intruding vehicles along the vehicle's travel direction is determined, including: using the duration between the timestamp of the latest image data and the timestamp of the latest radar data as the data deviation duration; and determining the true relative distance between the intruding vehicles along the vehicle's travel direction based on the radar data and the data deviation duration.

[0012] Optionally, the true relative distance between the intruding vehicles along the vehicle's travel direction is determined based on radar data and data deviation duration, including: determining the relative distance between the intruding vehicles and the speed of the intruding vehicles along the vehicle's travel direction based on radar data; and determining the true relative distance between the intruding vehicles along the vehicle's travel direction based on radar data, the relative distance between the intruding vehicles, the speed of the intruding vehicles along the vehicle's travel direction, and data deviation duration.

[0013] Optionally, the radar data also includes first radar data stored in a storage device. The first radar data is the radar data preceding the latest radar data. Determining the relative distance between the intruding vehicles and the speed of the intruding vehicles along the vehicle's travel direction based on the radar data includes: determining a first distance based on the latest radar data and using the first distance as the radar data to determine the relative distance between the intruding vehicles; the first distance is the distance between the intruding vehicles and the vehicles along the vehicle's travel direction; the difference between the first distance and a second distance is used as the target displacement change of the intruding vehicle; the second distance is the distance between the intruding vehicles and the vehicles along the vehicle's travel direction determined based on the first radar data; the ratio of the target displacement change to a third duration is used as the speed of the intruding vehicles along the vehicle's travel direction; the third duration is the acquisition interval between the latest radar data and the first radar data.

[0014] Optionally, the radar data also includes second radar data, which is the radar data preceding the first radar data. Based on the radar data, the relative distance between the intruding vehicles, the speed of the intruding vehicles along the vehicle's travel direction, and the data deviation duration, the true relative distance between the intruding vehicles along the vehicle's travel direction is determined, including: determining the acceleration of the intruding vehicles along the vehicle's travel direction based on the latest radar data, the first radar data, and the second radar data; using the product of the speed of the intruding vehicles along the vehicle's travel direction and the data deviation duration as a first displacement compensation amount; using the product of the acceleration and a first value as a second displacement compensation amount; the first value being the product of the square of the data deviation duration and 0.5; using the sum of the first displacement compensation amount and the second displacement compensation amount as a target displacement compensation amount; and extending the relative distance between the intruding vehicles along the vehicle's travel direction by the target displacement compensation amount to obtain the true relative distance between the intruding vehicles.

[0015] Optionally, the vehicle control strategy information is determined based on the actual relative distance between the intruding vehicles, including: calling a preset vehicle control model; the preset vehicle control model is used to determine the vehicle control strategy information based on vehicle information, including the vehicle's current speed and the actual relative distance between the intruding vehicles; the current speed of the vehicle and the actual relative distance between the intruding vehicles are input into the preset vehicle control model to obtain the vehicle control strategy information output by the preset vehicle control model.

[0016] Secondly, this application also discloses a vehicle control system based on multi-source perception data fusion, comprising: an acquisition device and a processing device; the acquisition device is used to acquire image data of the front of the vehicle collected by a camera in the vehicle's storage device; the processing device is used to determine whether the vehicle in front of the vehicle is an intruding vehicle based on the image data; the vehicle in front is the vehicle located in front of the vehicle; the processing device is further used to acquire radar data of the front of the vehicle collected by the vehicle's millimeter-wave radar in the storage device when the vehicle in front of the vehicle is an intruding vehicle, and determine the true relative distance between the intruding vehicle and the vehicle along the vehicle's driving direction based on the latest image data and radar data in the storage device; the latest image data is the image data with the latest timestamp in the storage device; the processing device is further used to determine the vehicle's control strategy information based on the true relative distance between the intruding vehicle and the vehicle.

[0017] Beneficial effects

[0018] This application, after determining that the vehicle ahead is an intruding vehicle, acquires the image data with the latest timestamp from the storage device and radar data, calculates the data deviation duration based on the timestamp difference between the two, and then determines the true relative distance between the intruding vehicle and the vehicle itself based on the radar data and this deviation duration. This technical solution overcomes the limitations of fixed-period synchronous fusion in existing technologies. Through timestamp alignment and dynamic displacement compensation mechanisms, it accurately corrects the longitudinal distance deviation caused by the asynchronous sampling times of the camera and millimeter-wave radar. In highly dynamic scenarios such as merging on highway ramps, this method can timely and accurately identify the actual position and movement state of the merging vehicle, avoiding the problem of delayed risk object priority judgment. This allows the vehicle control strategy to be generated based on the true relative distance, achieving precise matching of deceleration trigger timing and path-keeping control, significantly improving the safety and real-time response of the autonomous driving system in complex traffic environments. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a vehicle control method based on multi-source sensing data fusion provided in this application;

[0020] Figure 2 A schematic diagram of the architecture of a vehicle control system based on multi-source perception data fusion provided for this application;

[0021] Figure 3 A flowchart illustrating another vehicle control method based on multi-source sensing data fusion provided in this application;

[0022] Figure 4 This is a schematic diagram of another vehicle control system architecture based on multi-source perception data fusion provided in this application. Detailed Implementation

[0023] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] To better understand this application, the following detailed description will be provided with reference to the accompanying drawings and specific embodiments. It should be noted that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0025] Several key concepts are introduced in the technical solution of this application. First, "intruding vehicle" does not simply refer to any vehicle ahead, but specifically to those vehicles whose movement trajectory reveals a clear and high-risk intent to intrude. Second, "true relative distance" is a concept used to distinguish it from the distance directly measured by sensors at a given moment. True relative distance refers to the actual physical distance in the direction of travel between the intruding vehicle and the vehicle at the precise moment the system determines a target to be an intruding vehicle. This distance is obtained through dynamic compensation of older sensor data, making it more accurate than directly using unsynchronized measurement data.

[0026] In a typical scenario, a vehicle equipped with an advanced driver assistance system is traveling steadily at 120 km / h in the middle lane of a highway. At this moment, a vehicle on the right-hand ramp intends to merge into the main road. The driver suddenly accelerates and sharply turns the steering wheel to the left, attempting to cut in front of the vehicle.

[0027] This is a highly dynamic event that presents a significant challenge for the vehicle's driver assistance system. Both the front-facing camera and millimeter-wave radar continuously monitor the environment ahead. Due to its high sensitivity to image contours and lines, the camera is able to detect first the lateral movement trend of a vehicle crossing the lane line. However, while the millimeter-wave radar can accurately measure longitudinal distance and speed, its data update cycle is not synchronized with the camera, and it also requires time to confirm and track the target. Under traditional fusion mechanisms, the system may force the fusion of data from the two sensors within the same processing cycle due to inconsistent data timestamps. This approach can lead to misjudgments of the vehicle's true state on the ramp. For example, even when the camera has already identified a vehicle entering its lane, the fusion system might still rely on slightly earlier radar data, believing the vehicle is still in an adjacent lane, thus underestimating the depth and danger of its incursion. This lag in judgment will ultimately cause the vehicle's automatic emergency braking or avoidance strategy to be triggered too late, significantly increasing the risk of a collision.

[0028] To address this safety hazard caused by asynchronous data from multiple sensors, such as... Figure 1 As shown, this application provides a solution, as detailed below:

[0029] S101. Obtain image data of the front of the vehicle captured by the camera in the vehicle's storage device.

[0030] In one implementation, after the camera in the vehicle's storage device captures image data, the data can be obtained in real time from the camera in the vehicle's storage device.

[0031] S102. Determine whether the vehicle in front of the vehicle is an intruding vehicle based on the image data.

[0032] In one implementation, the image data includes the latest image data stored in the storage device and first image data, wherein the first image data is the image data of the frame preceding the latest image data. Determining whether the vehicle in front of the vehicle is an intruding vehicle based on the image data includes:

[0033] A first distance is determined based on the latest image data; the first distance is the distance between the vehicle in front of the vehicle and the lane line of the lane where the vehicle is located along a first direction; the first direction is the direction perpendicular to the vehicle's direction of travel on the ground plane; a second distance is determined based on the first image data; the second distance is the distance between the vehicle in front of the vehicle and the lane line of the lane where the vehicle is located along the first direction; the difference between the first distance and the second distance is taken as the first physical distance, and the ratio of the first physical distance to the first duration is taken as the vehicle's first speed; the first duration is the frame duration between the latest image data and the first image data; whether the vehicle in front of the vehicle is an intruding vehicle is determined based on the first speed.

[0034] It should be noted that a detailed explanation of this implementation method can be found in the subsequent related instructions.

[0035] S103. When the vehicle in front of the vehicle is an intruding vehicle, obtain the radar data collected by the vehicle's millimeter-wave radar in the storage device, and determine the true relative distance between the intruding vehicle and the vehicle along the vehicle's driving direction based on the latest image data and radar data in the storage device.

[0036] The latest image data is the image data with the latest timestamp in the storage device.

[0037] In one implementation, the radar data includes the latest radar data stored in a storage device, where the latest radar data is the radar data with the latest timestamp in the storage device. Based on the latest image data and radar data stored in the storage device, the true relative distance between the intruding vehicles along the vehicle's direction of travel is determined, including:

[0038] The time interval between the timestamp of the latest image data and the timestamp of the latest radar data is used as the data deviation time interval; based on the radar data and the data deviation time interval, the true relative distance between the intruding vehicles along the vehicle's driving direction is determined.

[0039] It should be noted that a detailed explanation of this implementation method can be found in the subsequent related instructions.

[0040] S104. Determine the vehicle control strategy information based on the actual relative distance between the intruding vehicles.

[0041] In one implementation, a preset vehicle control model is invoked; the preset vehicle control model is used to determine the vehicle control strategy information based on vehicle information, including the vehicle's current speed and the actual relative distance between the intruding vehicle and the vehicle; the vehicle's current speed and the actual relative distance between the intruding vehicle and the vehicle are input into the preset vehicle control model to obtain the vehicle control strategy information output by the preset vehicle control model.

[0042] It should be noted that a detailed explanation of this implementation method can be found in the subsequent related instructions.

[0043] This application, after determining that the vehicle ahead is an intruding vehicle, acquires the image data with the latest timestamp from the storage device and radar data, calculates the data deviation duration based on the timestamp difference between the two, and then determines the true relative distance between the intruding vehicle and the vehicle itself based on the radar data and this deviation duration. This technical solution overcomes the limitations of fixed-period synchronous fusion in existing technologies. Through timestamp alignment and dynamic displacement compensation mechanisms, it accurately corrects the longitudinal distance deviation caused by the asynchronous sampling times of the camera and millimeter-wave radar. In highly dynamic scenarios such as merging on highway ramps, this method can timely and accurately identify the actual position and movement state of the merging vehicle, avoiding the problem of delayed risk object priority judgment. This allows the vehicle control strategy to be generated based on the true relative distance, achieving precise matching of deceleration trigger timing and path-keeping control, significantly improving the safety and real-time response of the autonomous driving system in complex traffic environments.

[0044] The following will combine Figure 2 The proposed solution will be explained in detail.

[0045] In some specific implementations, relying solely on whether a vehicle crosses the line in a single frame image is insufficient to accurately determine whether a vehicle ahead is an intruding vehicle, as this could be due to unintentional minor movements by the driver. A more reliable approach is to analyze the trend of its lateral movement.

[0046] In this regard, the image data includes the latest image data in the storage device and the first image data, where the first image data is the image data of the frame preceding the latest image data, such as... Figure 3 As shown, determining whether the vehicle in front of the vehicle is an intruding vehicle based on image data includes:

[0047] S301. Determine the first distance based on the latest image data.

[0048] The first distance is the distance between the vehicle in front of the vehicle and the lane line of the lane in which the vehicle is located along the first direction; the first direction is the direction perpendicular to the direction of travel of the vehicle in the ground plane.

[0049] S302. Determine the second distance based on the first image data.

[0050] The second distance is the distance between the vehicle in front of the vehicle and the lane line of the lane in which the vehicle is located, along the first direction.

[0051] S303. The difference between the first distance and the second distance is taken as the first physical distance, and the ratio of the first physical distance to the first duration is taken as the first speed of the vehicle.

[0052] The first duration is the frame duration between the latest image data and the first image data.

[0053] S304. Determine whether the vehicle in front of the vehicle is an intruding vehicle based on the first speed.

[0054] The essence of this process is calculating the lateral velocity of the vehicle ahead approaching the current lane. The onboard camera continuously captures images at a fixed frame rate, and each frame can be used by image recognition algorithms to pinpoint the lane lines and the outline of the vehicle ahead. Specifically, the system first preprocesses each frame, including denoising and grayscale conversion. Then, it uses edge detection algorithms to extract lane line features from the image and combines lane line detection, semantic segmentation, or fitting algorithms to identify the mathematical model of the lane lines. Simultaneously, the system uses object detection algorithms to identify the bounding box of the vehicle ahead and extracts the vehicle's side edge facing the current lane, tire area, or other locations that can stably represent the vehicle's lateral boundary. The system continuously analyzes two frames: the latest frame and the frame preceding it. In each frame, the system calculates the lateral distance between the boundary of the vehicle ahead facing the current lane and the adjacent lane lines of the current lane—the first distance and the second distance.

[0055] The difference between the first distance and the second distance is taken as the first physical distance, and the ratio of the first physical distance to the first time duration is taken as the vehicle's first speed. This lateral speed value intuitively reflects the urgency of the other party's entry, providing a key quantitative basis for subsequent judgment on whether it constitutes an intrusion threat. In this way, the system can make judgments based on dynamic trends rather than static positions, thereby identifying dangerous vehicles with genuine entry intentions earlier and more accurately.

[0056] Of course, simply calculating lateral speed is insufficient to make a final intrusion judgment. While a high lateral speed is a warning sign, it needs to be combined with more information to assess the urgency of the threat. For example, a vehicle with a high lateral speed but still far from the lane line is clearly less threatening than a vehicle with a slightly slower lateral speed but already close to the lane line.

[0057] Therefore, based on the above scheme, the judgment logic can be further optimized. Determining whether the vehicle ahead is an intruding vehicle based on the first speed includes: determining whether the first speed is greater than a first preset speed threshold; if the first speed is less than or equal to the first preset speed threshold, determining that the vehicle ahead is not an intruding vehicle; if the first speed is greater than the first preset speed threshold, obtaining the width of the target lane; the target lane is the lane where the vehicle is located; determining the minimum distance between the intruding vehicle and the edge of the target lane based on the latest image data; using the ratio of the target distance to the first speed as a second duration; the target distance is the sum of the target width and the minimum distance; the target width is half the width of the target lane; and determining whether the vehicle ahead is an intruding vehicle based on the second duration.

[0058] This solution introduces a two-step verification mechanism. First, the system sets a preset threshold for lateral speed. This threshold is determined based on statistical analysis of normal driving behavior and dangerous lane-changing behavior. It involves collecting extensive real-world road test data, including the lateral speed distribution during normal lane changes and the lateral speed distribution during emergency lane changes, and using the statistical value near the boundary between these two scenarios as the threshold. If the calculated first speed does not exceed this threshold, the system considers it merely normal vehicle swaying or a slow lane-changing intention, not constituting an emergency threat. Only when the first speed is exceeded will the system initiate the second step of verification.

[0059] In the second verification step, the system calculates a parameter called the second duration, which represents the estimated time required for the vehicle to reach the predetermined lateral reference position within the target lane. The calculation process is as follows: First, the system obtains the width of the current lane, which can be obtained from high-precision map data or calculated in real time based on the distance between the left and right lane lines identified by the camera. Then, based on the latest image data, the system determines the minimum distance between the intruding vehicle and the edge of the target lane. This distance is calculated through the geometric relationship between the vehicle boundary and the lane line model in image processing. The system adds the target width to the minimum distance to obtain a target distance. This target distance represents the total lateral distance that the vehicle needs to traverse from its current position to the predetermined lateral reference position within the target lane. Finally, dividing this by the effective value of the first speed yields the second duration.

[0060] This second duration directly reflects the proximity of the threat. In this way, the system comprehensively considers the speed of the intrusion and the remaining lateral space, making the intrusion detection more accurate and effectively avoiding false alarms for vehicles that, although moving laterally at high speeds, will not immediately enter the lane.

[0061] In real-world high-speed driving scenarios, the criteria for judging danger should not be static. At 120 km / h, a short reaction time may already be very tight; while on urban roads at 60 km / h, the same amount of reaction time is relatively ample.

[0062] Therefore, an excellent decision-making system should be able to dynamically adjust its judgment criteria based on the vehicle's driving status. In this regard, this application provides a more preferred implementation: determining whether the vehicle in front of the vehicle is an intruding vehicle based on a second duration includes: obtaining the vehicle's current speed and a first correspondence; the first correspondence includes a one-to-one correspondence between multiple speed ranges and multiple first duration thresholds; using the first duration threshold corresponding to the speed range where the vehicle's current speed is located in the first correspondence as a target first duration threshold; determining whether the second duration is greater than the target first duration threshold; if the second duration is greater than the target first duration threshold, determining that the vehicle in front of the vehicle is not an intruding vehicle; if the second duration is less than or equal to the target first duration threshold, determining that the vehicle in front of the vehicle is an intruding vehicle.

[0063] The core of this solution lies in introducing a dynamic threshold related to the vehicle's speed. A mapping table is pre-stored in the vehicle's control unit, obtained through offline calibration and extensive real-vehicle testing. The specific establishment process is as follows: Within different speed ranges, testers simulate various intrusion scenarios, recording the minimum warning time required by the system, while considering factors such as system perception processing latency, actuator response time, and vehicle dynamics constraints. By statistically analyzing the minimum time margin required to avoid collisions or reduce risk at different speeds, a mapping relationship between speed ranges and the first duration threshold is established. This table defines the minimum intrusion duration threshold that the system can tolerate within different speed ranges.

[0064] Once the system calculates the second duration for the vehicle ahead, it doesn't compare it to a fixed value. Instead, it first obtains the vehicle's current speed via the vehicle bus, such as the CAN bus. Assuming the vehicle is traveling at a certain speed, the system looks up the range of that speed in a corresponding table to determine the corresponding target first duration threshold. Then, the system compares the calculated value to this dynamic threshold. If it's less than or equal to the threshold, the system ultimately confirms the vehicle ahead as an intruder and immediately initiates subsequent distance calculations and control strategies. Otherwise, if it's greater than the threshold, monitoring continues.

[0065] Through this adaptive judgment mechanism, this application enables the vehicle's risk assessment capabilities to dynamically match the hazard level of the actual driving scenario. At high speeds, the system typically needs to identify potential risks earlier to allow sufficient time for response; while at low speeds, it can appropriately adjust the triggering criteria based on the actual risk level to avoid unnecessary emergency braking due to oversensitivity. This ensures safety while improving passenger comfort and the system's intelligence.

[0066] After identifying the vehicle ahead as an intruder, the next crucial step is to accurately calculate the true relative distance between it and the vehicle in order to formulate the most appropriate control strategy. As mentioned earlier, due to the time lag between data acquisition and processing by cameras and millimeter-wave radar, directly using the distance value measured by radar will introduce errors. To address this issue, this application proposes a distance calculation method based on timestamp compensation.

[0067] In a preferred embodiment of this application, the radar data includes the latest radar data in the storage device, which is the radar data with the latest timestamp in the storage device. Determining the true relative distance between the intruding vehicles along the vehicle's driving direction based on the latest image data and radar data in the storage device includes: using the duration between the timestamp of the latest image data and the timestamp of the latest radar data as the data deviation duration; and determining the true relative distance between the intruding vehicles along the vehicle's driving direction based on the radar data and the data deviation duration.

[0068] The core of this specific implementation plan lies in its explicit transformation of the time asynchrony issue between the camera and radar into a quantifiable "data deviation duration." The system's time synchronization module is responsible for accurately timestamping each frame of data from different sensors. This timestamp can originate from the GPS clock or the vehicle's main control unit's internal high-precision clock. Specifically, the vehicle's main control unit is equipped with a hardware time synchronization interface. When the camera completes the acquisition of a frame of image via the MIPI interface and transmits it to the main control unit, the main control unit records the system clock value corresponding to the acquisition time or sensor output time of that frame and writes this timestamp into the header metadata area of ​​the image data packet.

[0069] Similarly, when the millimeter-wave radar sends a target data packet via the CAN bus or Ethernet interface, the system records the measurement time, output time, or time converted to a unified time base as the timestamp of the radar data packet. When the image processing module determines that a vehicle in a certain image frame is an intruding vehicle, the system records the timestamp of this frame of "latest image data." Simultaneously, the system searches the data cache for the "latest radar data" provided by the millimeter-wave radar and its timestamp. The system uses a timestamp parsing module to extract the corresponding timestamp values ​​from the headers of the image data packet and the radar data packet, respectively. The difference between the timestamp of the latest image data and the timestamp of the latest radar data is calculated, and the corresponding duration is taken as the data deviation duration. This data deviation duration can be accurate to the time resolution supported by the system clock. This deviation duration provides the most basic time basis for subsequent motion state extrapolation and distance compensation.

[0070] This specific implementation scheme eliminates the need to avoid or roughly handle time differences between sensors; instead, it treats these differences as a critical parameter for precise calculation and utilization. This approach lays the foundation for more refined motion compensation algorithms, enabling the final calculated relative distance to more accurately reflect the true location of the intruding vehicle at the moment it is identified as a threat, thus gaining valuable decision-making and reaction time for the control system.

[0071] After obtaining the duration of the data deviation, the key to achieving accurate compensation lies in how to use this duration to correct the distance measurement. A simple method is to assume that the intruding vehicle maintains a constant speed during the deviation duration.

[0072] Optionally, the true relative distance between the intruding vehicles along the vehicle's travel direction is determined based on radar data and data deviation duration, including: determining the relative distance between the intruding vehicles and the speed of the intruding vehicles along the vehicle's travel direction based on radar data; and determining the true relative distance between the intruding vehicles along the vehicle's travel direction based on radar data, the relative distance between the intruding vehicles, the speed of the intruding vehicles along the vehicle's travel direction, and data deviation duration.

[0073] This specific implementation plan clarifies the two core motion parameters required for distance compensation: relative distance and relative velocity. Both parameters can be directly obtained from millimeter-wave radar data. Millimeter-wave radar determines the relative distance of a target by measuring the flight time of electromagnetic waves, and simultaneously measures the radial relative velocity of the target using the Doppler effect. In practice, the system parses the longitudinal distance field value corresponding to the target ID of the intruding vehicle from the latest radar data packet as the relative distance, and parses the radial velocity field value or its component along the vehicle's travel direction as the relative velocity along the vehicle's travel direction. After the system acquires the latest radar data packet, it parses these two key pieces of information about the intruding vehicle. Then, the system calculates the displacement compensation amount based on the relative velocity and the data deviation duration. The calculation relationship can be expressed as: where represents the displacement compensation amount, represents the relative velocity along the vehicle's travel direction, and represents the data deviation duration. This compensation amount represents the longitudinal displacement change of the intruding vehicle relative to the vehicle within the data deviation duration.

[0074] Finally, based on the speed sign convention and the temporal relationship corresponding to the data deviation duration, the displacement compensation amount is added to or subtracted from the original relative distance measured by the radar to obtain the time-calibrated true relative distance. The specific calculation process is as follows: the displacement compensation amount equals the product of the relative speed and the corresponding duration of the data deviation; if, according to the adopted sign convention, the target is closer to the vehicle at the image time, the displacement compensation amount is subtracted from the original relative distance to obtain the true relative distance; if, according to the adopted sign convention, the target is farther away from the vehicle at the image time, the displacement compensation amount is added to the original relative distance to obtain the true relative distance.

[0075] For example, suppose radar data shows a relative distance of 40m, the relative speed of the intruding vehicle relative to the vehicle along the direction of travel is 15m / s, and the data deviation duration is 50ms. If, according to the used sign convention and time sequence, the intruding vehicle is closer to the vehicle within this data deviation duration, then the displacement compensation is m. Therefore, the true relative distance is m. Although a difference of 0.75m may seem small, at high speeds, this distance can determine whether a safe avoidance is possible or a collision occurs. This scheme achieves an effective correction to the distance data through simple linear extrapolation, significantly improving the accuracy of distance estimation in high-dynamic scenarios.

[0076] To achieve the aforementioned distance compensation, the longitudinal relative speed of the intruding vehicle must first be accurately calculated from the radar data. Although modern radars can directly output speed information, in some cases, calculating the speed through continuous distance measurements can provide a supplementary or verification method.

[0077] Optionally, the radar data also includes first radar data stored in a storage device. The first radar data is the radar data preceding the latest radar data. Determining the relative distance between the intruding vehicles and the speed of the intruding vehicles along the vehicle's travel direction based on the radar data includes: determining a first distance based on the latest radar data and using the first distance as the radar data to determine the relative distance between the intruding vehicles; the first distance is the distance between the intruding vehicles and the vehicles along the vehicle's travel direction; the difference between the first distance and a second distance is used as the target displacement change of the intruding vehicle; the second distance is the distance between the intruding vehicles and the vehicles along the vehicle's travel direction determined based on the first radar data; the ratio of the target displacement change to a third duration is used as the speed of the intruding vehicles along the vehicle's travel direction; the third duration is the acquisition interval between the latest radar data and the first radar data.

[0078] This specific implementation describes a process for calculating relative velocity using a differential method. The system retains at least two consecutive frames of radar data. Specifically, the system maintains a circular buffer in memory to store the most recently received radar data frames. When new radar data is received, it is added to the head of the buffer, and the oldest data frame is removed. When velocity calculation is needed, the system retrieves the latest radar data frame (containing the first distance and timestamp) and its preceding frame (the first radar data frame, containing the second distance and timestamp). Each frame contains the relative distance measured at that time. The system subtracts the first distance from the second distance; the difference is the target displacement change. Simultaneously, the system calculates the time difference between the timestamp of the latest radar data and the timestamp of the first radar data; this time difference is the third duration, which is the acquisition interval between two adjacent radar data frames. Finally, the target displacement change is divided by the third duration; the quotient is the average relative velocity between these two frames, and the calculation relationship can be expressed as follows. This calculated velocity value can be used for subsequent range compensation calculations. This method does not rely on the Doppler velocity output inside the radar, providing a redundant approach to velocity acquisition and enhancing the robustness of the system.

[0079] However, in high-dynamic scenarios such as emergency lane incursions on ramps, the oncoming vehicle is often in a state of rapid acceleration or deceleration, and the simple assumption of uniform motion may not be sufficient to accurately describe its motion state. The acceleration of the intruding vehicle will cause deviations in the linear extrapolation results. To further improve the accuracy of compensation, an acceleration term can be introduced to perform higher-order motion compensation.

[0080] In another preferred embodiment of this application, the radar data further includes second radar data, which is the radar data preceding the first radar data. Based on the radar data, the relative distance between the intruding vehicles, the speed of the intruding vehicles along the vehicle's travel direction, and the data deviation duration, the true relative distance between the intruding vehicles along the vehicle's travel direction is determined. This includes: determining the acceleration of the intruding vehicles along the vehicle's travel direction based on the latest radar data, the first radar data, and the second radar data; using the product of the speed of the intruding vehicles along the vehicle's travel direction and the data deviation duration as a first displacement compensation amount; using the product of the acceleration and a first value as a second displacement compensation amount; where the first value is the product of the square of the data deviation duration and 0.5; using the sum of the first and second displacement compensation amounts as a target displacement compensation amount; and extending the relative distance between the intruding vehicles along the vehicle's travel direction by the target displacement compensation amount to obtain the true relative distance between the intruding vehicles.

[0081] This specific implementation scheme introduces a second-order motion model for compensation. To calculate acceleration, the system requires at least three consecutive frames of radar data. Specifically, the system reads the latest radar data (range, time), the first radar data (range, time), and the second radar data (range, time) from the circular buffer. The system first calculates a first velocity value using the latest and first radar data, as follows: [Calculation method omitted]. Then, it calculates a second velocity value using the first and second radar data, as follows: [Calculation method omitted]. Finally, it subtracts the second velocity value from the first velocity value to obtain the velocity change, and divides this by the corresponding time interval to obtain the acceleration estimate of the intruding vehicle. When adjacent sampling intervals are approximately equal, the acceleration can be expressed as: [Equation omitted], where represents the time interval corresponding to adjacent velocity estimates; in a more general case, the acceleration can be determined based on the reference time difference between [Equation omitted] and [Equation omitted].

[0082] After obtaining the acceleration, the calculation of displacement compensation becomes more precise. The compensation amount is no longer a simple product of velocity and time, but is divided into two parts: the first displacement compensation amount, which is the displacement generated by the initial velocity, equal to the relative velocity multiplied by the data deviation duration; and the second displacement compensation amount, which is the displacement generated by the acceleration, equal to 0.5 multiplied by the acceleration and then multiplied by the square of the data deviation duration. In specific calculations, the square of the data deviation duration is first calculated, and this square is multiplied by 0.5 to obtain the first value. Then, the first value is multiplied by the acceleration to obtain the second displacement compensation amount. Adding the first and second displacement compensation amounts yields the target displacement compensation amount that takes into account the accelerated motion, and its calculation relationship can be expressed as follows. Finally, based on the velocity sign convention and the temporal sequence corresponding to the data deviation duration, the target displacement compensation amount is subtracted from or added to the original radar range measurement value, resulting in a true relative distance that is closer to the physical reality.

[0083] This application further proposes an optimization scheme, which aims to efficiently and accurately convert the real relative distance calculated by the aforementioned method into control commands that can be executed by the vehicle.

[0084] In a preferred embodiment of this application, determining vehicle control strategy information based on the actual relative distance between the intruding vehicle and the vehicle includes: calling a preset vehicle control model; the preset vehicle control model is used to determine the vehicle control strategy information based on vehicle information, including the vehicle's current speed and the actual relative distance between the intruding vehicle and the vehicle; inputting the vehicle's current speed and the actual relative distance between the intruding vehicle and the vehicle into the preset vehicle control model to obtain the vehicle control strategy information output by the preset vehicle control model.

[0085] This specific implementation plan introduces a pre-defined vehicle control model as an intelligent hub connecting perception results and execution actions. This model is not a simple lookup table or linear mapping, but a decision-making unit capable of comprehensively assessing the current driving situation and outputting the optimal control strategy. The vehicle information includes the vehicle's current speed and the actual relative distance, precisely calculated through the aforementioned steps. These two parameters are the core basis for assessing collision risk and formulating response strategies.

[0086] In practice, this pre-defined vehicle control model can be implemented in several ways. One approach is to use a rule-based model, which pre-defines a series of conditional judgments and corresponding control strategies based on extensive expert experience and safety regulations. For example, the model can define a safe distance threshold function, which takes the vehicle's current speed as input and outputs a dynamic minimum safe distance. When the actual relative distance is less than this safe distance, the model triggers a deceleration strategy. Furthermore, different deceleration levels can be defined based on the ratio of distance to safe distance, such as light deceleration, moderate deceleration, and emergency braking. Another more advanced approach is to use a data-driven model, such as a neural network or reinforcement learning model. These models learn the optimal control strategy under different speeds, relative distances, and road conditions through extensive training in simulation environments or real-world scenarios. For example, a deep neural network can be used, taking the vehicle's speed, the relative speed of the intruding vehicle (which can be estimated from the distance change rate), and the actual relative distance as input. The network's output layer can contain multiple neurons, each corresponding to a different control command, such as the desired deceleration or the desired steering angle. The training dataset for the model can be generated by simulating various entry scenarios in a simulation platform, or obtained from real vehicle data collected from test vehicles equipped with professional drivers.

[0087] In practical applications, after the system identifies the intruding vehicle and calculates the true relative distance, it packages these two key pieces of information, along with the vehicle's current speed, and inputs them into a preset vehicle control model. The model, after internal calculations, outputs specific control strategy information. This information can be a desired longitudinal acceleration value (positive values ​​represent acceleration, negative values ​​represent deceleration), a desired vehicle speed, or a comprehensive command including deceleration level and steering suggestions. For example, if the vehicle is traveling at 110 km / h and the calculated true relative distance is 35 meters, the model might output a desired deceleration of -3 m / s², prompting the vehicle to begin moderate deceleration. As the distance decreases further, the model may gradually increase the deceleration until emergency braking is triggered.

[0088] By introducing a pre-defined vehicle control model, this application effectively transforms precise perception data into specific vehicle control actions. This model can dynamically adjust the control strategy based on real-time changes in driving conditions, achieving a closed loop from "perception" to "decision-making" to "control." This not only improves the vehicle's automation level in emergency situations, but more importantly, through model optimization, it can maintain driving comfort and stability as much as possible while ensuring safety. It avoids sudden braking or oversteering caused by simple, rigid control logic, significantly improving the overall performance and user experience of the intelligent driving system.

[0089] like Figure 4 As shown, this application also proposes a vehicle control system based on multi-source perception data fusion, including: an acquisition device and a processing device; the acquisition device is used to acquire image data of the front of the vehicle collected by a camera in the vehicle's storage device; the processing device is used to determine whether the vehicle in front of the vehicle is an intruding vehicle based on the image data; the vehicle in front is the vehicle located in front of the vehicle; the processing device is further used to acquire radar data of the front of the vehicle collected by the vehicle's millimeter-wave radar in the storage device when the vehicle in front is an intruding vehicle, and determine the true relative distance between the intruding vehicle and the vehicle along the vehicle's driving direction based on the latest image data and radar data in the storage device; the latest image data is the image data with the latest timestamp in the storage device; the processing device is further used to determine the vehicle's control strategy information based on the true relative distance between the intruding vehicle and the vehicle.

[0090] Through the above technical solution, this application achieves hardware-level support for the aforementioned vehicle control method. The acquisition device, as the system's front-end perception interface, is responsible for real-time acquisition and caching of image data from the camera; the processing device, as the system's core computing unit, undertakes all processing tasks from image analysis, intrusion detection, distance calculation to policy generation. This hardware-software collaborative architecture design ensures a fast and reliable link from environmental perception to control execution, enabling the entire control method to operate stably and in real-time in the vehicle environment, providing reliable active safety protection capabilities for the vehicle.

[0091] At the hardware implementation level, the acquisition device may include interface circuits for direct communication with cameras and millimeter-wave radar, a data receiving and buffering module, and a read / write module connected to the vehicle's storage device. The processing device can consist of an onboard central processing unit, a graphics processing unit, or a dedicated autonomous driving chip, running software programs that implement the functions described above. The acquisition and processing devices can be connected via a high-speed internal bus to ensure low-latency transmission of image and radar data. Through this system architecture, the vehicle can accurately and in real-time identify intrusion risks in complex traffic environments and take timely control measures based on precisely calculated distance information, significantly improving driving safety.

[0092] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A vehicle control method based on multi-source sensing data fusion, characterized in that, include: Acquire image data of the front of the vehicle captured by the camera in the vehicle's storage device; Determine whether the vehicle in front of the vehicle is an intruding vehicle based on image data; When the vehicle in front of the vehicle is an intruding vehicle, the vehicle's millimeter-wave radar collects radar data in front of the vehicle from the storage device, and determines the true relative distance between the intruding vehicle and the vehicle along the vehicle's driving direction based on the latest image data and radar data in the storage device. The latest image data is the image data with the latest timestamp in the storage device; Based on the actual relative distance between the intruding vehicles, determine the vehicle control strategy information; The radar data includes the latest radar data in the storage device, which is the radar data with the latest timestamp in the storage device. Based on the latest image data and radar data in the storage device, the true relative distance between the intruding vehicles along the vehicle's direction of travel is determined, including: The duration between the timestamp of the latest image data and the timestamp of the latest radar data is used as the data deviation duration. The true relative distance between the intruding vehicles is determined based on radar data and data deviation duration along the vehicle's travel direction. Based on radar data and data deviation duration, the true relative distance between the intruding vehicles along the vehicle's travel direction is determined, including: The relative distance between the intruding vehicles and the speed of the intruding vehicles along their direction of travel are determined based on radar data. Based on radar data, the relative distance between intruding vehicles, the speed of the intruding vehicles along the vehicle's direction of travel, and the duration of data deviation, the true relative distance between the intruding vehicles along the vehicle's direction of travel is determined.

2. The vehicle control method based on multi-source sensing data fusion according to claim 1, characterized in that, Image data includes the latest image data and first image data stored in the storage device. The first image data is the image data of the frame preceding the latest image data. Determining whether the vehicle in front of the vehicle is an intruding vehicle based on the image data includes: The first distance is determined based on the latest image data; the first distance is the distance between the vehicle in front of the vehicle and the lane line of the lane in which the vehicle is located along a first direction; the first direction is the direction perpendicular to the vehicle's direction of travel in the ground plane; The second distance is determined based on the first image data; the second distance is the distance between the vehicle in front of the vehicle and the lane line of the lane in which the vehicle is located along the first direction. The difference between the first distance and the second distance is taken as the first physical distance, and the ratio of the first physical distance to the first duration is taken as the first speed of the vehicle; the first duration is the frame duration between the latest image data and the first image data. Determine whether the vehicle in front of the vehicle is an intruding vehicle based on the first speed.

3. The vehicle control method based on multi-source sensing data fusion according to claim 2, characterized in that, Determining whether the vehicle in front of the vehicle is an intruding vehicle based on the first speed includes: Determine whether the first speed is greater than a first preset speed threshold; When the first speed is less than or equal to a first preset speed threshold, it is determined that the vehicle in front is not an intruding vehicle; When the first speed is greater than a first preset speed threshold, the width of the target lane is obtained; the target lane is the lane where the vehicle is located. Determine the minimum distance between the intruding vehicle and the edge of the target lane based on the latest image data; The ratio of the target distance to the first speed is used as the second duration; the target distance is the sum of the target width and the minimum distance; the target width is half the width of the target lane. The second time interval is used to determine whether the vehicle in front of the vehicle is an intruding vehicle.

4. The vehicle control method based on multi-source perception data fusion according to claim 3, characterized in that, Determining whether the vehicle in front of the vehicle is an intruding vehicle based on the second duration includes: Obtain the vehicle's current speed and a first correspondence; the first correspondence includes a one-to-one correspondence between multiple speed ranges and multiple first duration thresholds; The first duration threshold corresponding to the speed range of the vehicle's current speed in the first correspondence is used as the target first duration threshold; Determine whether the second duration is greater than the target first duration threshold; When the second duration exceeds the target first duration threshold, determine that the vehicle in front of the vehicle is not an intruding vehicle; When the second duration is less than or equal to the target first duration threshold, the vehicle in front of the vehicle is identified as an intruding vehicle. The first duration threshold is the minimum intrusion duration threshold.

5. The vehicle control method based on multi-source perception data fusion according to claim 1, characterized in that, The radar data also includes first radar data stored in the storage device, which is the radar data preceding the latest radar data. Based on the radar data, the relative distance between the intruding vehicles and the speed of the intruding vehicles along their direction of travel are determined, including: The first distance is determined based on the latest radar data, and the first distance is used as radar data to determine the relative distance between the intruding vehicle and the vehicle; the first distance is the distance between the intruding vehicle and the vehicle along the vehicle's direction of travel; The difference between the first distance and the second distance is taken as the target displacement change of the intruding vehicle; the second distance is the distance between the intruding vehicle and the vehicle along the vehicle's direction of travel, determined based on the first radar data. The ratio of the target displacement change to the third time interval is used as the speed of the intruding vehicle along the vehicle's direction of travel; the third time interval is the acquisition interval between the latest radar data and the first radar data.

6. The vehicle control method based on multi-source perception data fusion according to claim 1, characterized in that, The radar data also includes second radar data, which is the radar data preceding the first radar data. Based on the radar data, the relative distance between the intruding vehicles, the speed of the intruding vehicles along the vehicle's direction of travel, and the duration of data deviation, the true relative distance between the intruding vehicles along the vehicle's direction of travel is determined, including: Based on the latest radar data, first radar data, and second radar data, determine the acceleration of the intruding vehicle along the vehicle's direction of travel; The product of the speed of the intruding vehicle along the vehicle's direction of travel and the duration of the data deviation is used as the first displacement compensation amount. The product of acceleration and the first value is used as the second displacement compensation; the first value is the product of the square of the data deviation duration and 0.

5. The sum of the first displacement compensation and the second displacement compensation is taken as the target displacement compensation. The relative distance between the intruding vehicles is extended along the vehicle's direction of travel by the target displacement compensation amount to obtain the true relative distance between the intruding vehicles.

7. The vehicle control method based on multi-source perception data fusion according to claim 1, characterized in that, Based on the actual relative distance between the intruding vehicles, determine the vehicle control strategy information, including: Invoke the preset vehicle control model; the preset vehicle control model is used to determine the vehicle control strategy information based on the vehicle information, including the vehicle's current speed and the actual relative distance between the intruding vehicle and the vehicle. Input the vehicle's current speed and the actual relative distance between the intruding vehicle and the vehicle into the preset vehicle control model to obtain the vehicle control strategy information output by the preset vehicle control model.

8. A vehicle control system based on multi-source sensing data fusion, characterized in that, include: Acquisition device and processing device; Acquisition device, used to acquire image data of the front of the vehicle captured by a camera in the vehicle's storage device; The processing device is used to determine whether the vehicle in front of the vehicle is an intruding vehicle based on image data. The vehicle in front is the vehicle located in front of the vehicle. The processing device is also used to acquire radar data of the vehicle in front of the vehicle collected by the millimeter-wave radar in the storage device when the vehicle in front of the vehicle is an intruding vehicle, and to determine the real relative distance between the intruding vehicle and the vehicle along the driving direction of the vehicle based on the latest image data and radar data in the storage device. The latest image data is the image data with the latest timestamp in the storage device; The processing device is also used to determine vehicle control strategy information based on the actual relative distance between the intruding vehicles; The radar data includes the latest radar data in the storage device, which is the radar data with the latest timestamp in the storage device. Based on the latest image data and radar data in the storage device, the true relative distance between the intruding vehicles along the vehicle's direction of travel is determined, including: The duration between the timestamp of the latest image data and the timestamp of the latest radar data is used as the data deviation duration. The true relative distance between the intruding vehicles is determined based on radar data and data deviation duration along the vehicle's travel direction. Based on radar data and data deviation duration, the true relative distance between the intruding vehicles along the vehicle's travel direction is determined, including: The relative distance between the intruding vehicles and the speed of the intruding vehicles along their direction of travel are determined based on radar data. Based on radar data, the relative distance between intruding vehicles, the speed of the intruding vehicles along the vehicle's direction of travel, and the duration of data deviation, the true relative distance between the intruding vehicles along the vehicle's direction of travel is determined.

Images