An electronic rearview mirror radar and vision fusion blind filling method and system based on FPGA

Abstract

The application provides a kind of based on FPGA electronic rearview mirror radar vision fusion blind filling method and system, method includes: obtaining continuous image frame, continuous radar echo data, vehicle motion state data and electronic rearview mirror display refresh state data, original acquisition record is generated by field integrity verification and abnormal state marker, and multi-agent system is established, generates blind filling data cache sequence, cache effective marker and physical freshness search population;Through visual freshness agent, extract visual physical micro change characteristics and generate visual freshness record, through radar freshness agent, extract radar physical micro disturbance characteristics and generate radar freshness record;According to visual freshness record, radar freshness record, vehicle motion state data and electronic rearview mirror display refresh state data, cross-modal evidence alignment and physical consistency check are carried out, and physical freshness determination result and fusion channel control instruction are generated.

Description

Technical Field

[0001] This invention relates to the field of electric vehicle electronic rearview mirrors and vehicle-mounted perception fusion technology, and in particular to an FPGA-based method and system for blind spot compensation in electric vehicle electronic rearview mirrors using radar-visual fusion. Background Technology

[0002] With the development of intelligent electric vehicles, electronic rearview mirrors are gradually replacing traditional rearview mirrors. To improve the reliability of blind spot compensation in complex scenarios such as rain, fog, strong light changes, and rapid approach of targets from the side and rear, existing solutions often incorporate millimeter-wave radar. This radar integrates target distance, speed, and angle information with image information to form a radar-visual fusion blind spot compensation result. Current radar-visual fusion blind spot compensation technologies primarily focus on image target recognition accuracy, radar detection stability, and multimodal spatial matching. For FPGA-based vehicle blind spot compensation systems, existing solutions generally utilize the parallel processing capabilities of FPGAs to buffer image frames, radar echoes, and display data for low-latency output.

[0003] However, in actual operation, the radar-visual fusion link is susceptible to factors such as short-term congestion of the vehicle bus, low temperature, electromagnetic interference, display cache blockage, or local hardware state machine lockup, leading to a highly insidious "false real-time" problem. In this situation, the camera or millimeter-wave radar may repeatedly output historical data from the previous cycle, or the electronic rearview mirror display cache may not have been properly refreshed. At this time, even if the data packet format remains intact and the frame sequence number or timestamp continues to increase, conventional verification mechanisms still cannot detect the anomaly, resulting in the actual blind spot compensation result deviating from the current real road condition.

[0004] This type of pseudo-real-time failure differs from explicit black screens, distorted screens, or radar outages. The driver still perceives a seemingly normal scene, and the system may continuously output blind spot safety warnings. In high-dynamic scenarios such as high-speed lane changes, ramp merging, rapid approach by motorcycles, or sudden acceleration from behind, brief data stagnation can cause the displayed content to lag significantly behind the actual traffic conditions, thus misleading the driver's judgment.

[0005] Therefore, this invention proposes an FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation method and system. The information disclosed in the background section is only for enhancing understanding of the background of this disclosure and may therefore contain prior art information that is not common knowledge to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing an FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation method and system, thereby solving the technical problems mentioned in the background section.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A radar-visual fusion method for blind spot compensation in electric vehicle electronic rearview mirrors based on FPGA includes the following steps: S1. Acquire continuous image frames, continuous radar echo data, vehicle motion status data, and electronic rearview mirror display refresh status data. Generate original acquisition records after field integrity verification and abnormal status marking. Establish a multi-agent system to generate blind spot data cache sequence, cache validity markers, and physical freshness search population. S2. The visual freshness agent performs hardware-level differential detection on adjacent consecutive image frames and uses the wolf pack algorithm to filter visual physical micro-change features to generate visual freshness records. S3. The radar freshness agent performs periodic comparisons of continuous radar echo data and uses the wolf pack algorithm to filter radar physical micro-perturbation features to generate radar freshness records. S4. Generate a cross-modal evidence alignment table based on visual freshness records, radar freshness records, vehicle motion status data, and electronic rearview mirror display refresh status data; perform physical consistency verification; and generate physical freshness judgment results and fusion channel control commands. S5. Select the fusion channel based on the physical freshness judgment result and the fusion channel control instruction, generate blind spot display information or conservative blind spot warning information, write it into the electronic rearview mirror display cache and generate blind spot link feedback record.

[0008] S1 specifically includes: acquiring image frames, radar echo data, vehicle motion status data, and display refresh status data according to the acquisition cycle; verifying the number, acquisition time, and cache address; removing data with missing necessary fields from the normal binding queue; writing a frozen suspicion flag to data with complete fields but stagnant data; generating the original acquisition record; establishing a multi-agent system on the FPGA; allocating the original acquisition record to the visual freshness agent, radar freshness agent, vehicle motion agent, display monitoring agent, and fusion gatekeeper agent; generating an agent task allocation table; establishing a blind spot data cache sequence based on the original acquisition record and agent task allocation table; generating a cache validity flag and initializing the physical freshness search population.

[0009] S2 specifically includes: the visual freshness agent reads the blind spot data cache sequence, cache validity markers, and physical freshness search population; performs hardware-level differential detection on adjacent consecutive image frames with valid cache, generates a set of visual candidate micro-change features, and generates visual skip records for acquisition cycles with invalid cache; uses a wolf pack algorithm to filter visual physical micro-change features based on image differential intensity, orientation matching degree, continuous occurrence degree, and noise suspicion value, and deletes single-frame noise, compressed block boundaries, and display afterimage candidates; calculates visual freeze suspicion value based on the set of visual physical micro-change features, generates visual freshness records, and sends them to the fusion gatekeeper agent as visual input for subsequent cross-modal evidence alignment and physical consistency verification.

[0010] S3 specifically includes: the radar freshness agent reads the blind spot data cache sequence, cache validity markers, physical freshness search population, and vehicle motion state data; performs periodic comparisons on the cached continuous radar echo data that is valid, generates a radar candidate micro-perturbation feature set, and generates radar skip records for acquisition cycles with invalid caches; uses a wolf pack algorithm to filter radar physical micro-perturbation features based on phase perturbation intensity, ground scattering band drift degree, vehicle motion matching degree, echo energy validity degree, and repeated cache suspicion value, and deletes candidates with fixed format repeated output, single-cycle jumps, and low-energy noise; calculates radar freeze suspicion value based on the radar physical micro-perturbation feature set, generates radar freshness records, and sends them to the fusion gatekeeper agent as radar-side input for subsequent cross-modal evidence alignment and physical consistency verification.

[0011] S4 specifically includes: the vehicle motion agent reads visual freshness records, radar freshness records, vehicle motion state data, and display refresh state data, performs time deviation verification according to the collection cycle, and generates a cross-modal evidence alignment table; the fusion gatekeeper agent performs physical consistency verification on the visual side, radar side, display side, and vehicle motion side based on the cross-modal evidence alignment table, and generates a physical freshness scoring table and a false real-time risk level; the fusion gatekeeper agent generates a physical freshness judgment result and a fusion channel control command based on the physical freshness scoring table and the false real-time risk level. The physical freshness judgment result includes real-time validity, single-sided downgrade validity, and non-real-time unavailability. The fusion channel control command is used for subsequent channel selection.

[0012] S5 specifically includes: the fusion gatekeeper agent reads the physical freshness judgment result, fusion channel control command, and pseudo-real-time risk level, generates a fusion channel status record, and selects a normal fusion channel, a pseudo-real-time fusion channel, or a conservative blind spot filling channel based on the real-time effective, unilateral degradation effective, or non-real-time unavailable status; it generates electronic rearview mirror blind spot filling display information, unilateral degradation blind spot filling information, or conservative blind spot filling warning information based on the fusion channel status record, and prohibits the output of blind spot safety prompts under severe pseudo-real-time risk; the display monitoring agent writes the corresponding display information into the electronic rearview mirror display cache, generates a blind spot filling link feedback record, and participates in the generation of the original acquisition record in the next acquisition cycle.

[0013] An FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation system includes: a data acquisition and initialization module, a visual freshness processing module, a radar freshness processing module, a physical consistency verification module, and a fusion display feedback module.

[0014] The beneficial effects of this invention are as follows: This invention achieves unified acquisition, cache binding, and field verification of continuous image frames, continuous radar echo data, vehicle motion status data, and electronic rearview mirror display refresh status data within an FPGA. This enables the identification of data with complete format but stagnant content, avoiding misjudgments of blind spot data as real-time and valid solely based on timestamps or data packet integrity. By extracting visual physical micro-change features such as road texture flow, target edge displacement, ambient light changes, and subtle image shifts through a visual freshness agent, and by extracting radar physical micro-perturbation features such as phase disturbances, ground scattering band drift, target distance and velocity changes, and echo energy fluctuations through a radar freshness agent, the invention provides verifiable physical evidence for the authenticity of blind spot data updates.

[0015] This invention utilizes a multi-agent system and a wolf pack algorithm to screen visual physical micro-change features and radar physical micro-perturbation features, respectively. This removes invalid candidates such as single-frame noise, compressed block boundaries, fixed-format repetitive output, buffer replay, and low-energy noise, improving the accuracy and anti-interference capability of physical freshness verification. Through cross-modal evidence alignment tables and physical consistency checks, cross-validation is performed on the visual, radar, display, and vehicle motion sides, enabling timely identification of false real-time risks such as image freezing, radar list freezing, display buffer repetitive output, and outdated content despite normal formatting.

[0016] This invention selects a normal fusion channel, a unilateral downgraded fusion channel, or a conservative blind spot compensation channel based on the physical freshness determination result and the false real-time risk level. When non-real-time unavailable data appears, it blocks its entry into the normal fusion process, avoiding unreliable blind spot safety warnings from the electronic rearview mirror output. The blind spot compensation link feedback record displays a valid marker, fusion channel status code, and false real-time risk level, which are then fed back to the next acquisition cycle. This creates a closed loop between data acquisition, physical verification, channel control, display output, and feedback feedback, improving the continuous safety and operational reliability of the electronic rearview mirror radar-based fusion blind spot compensation system. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a radar-visual fusion blind spot compensation method for electronic rearview mirrors based on FPGA according to the present invention; Figure 2 This is a schematic diagram of the framework of an FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation system according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1: As Figure 1 As shown, this embodiment provides a FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation method, including the following steps: S1. Acquire continuous image frames, continuous radar echo data, vehicle motion status data, and electronic rearview mirror display refresh status data. Generate original acquisition records after field integrity verification and abnormal status marking. Establish a multi-agent system to generate blind spot data cache sequence, cache validity markers, and physical freshness search population. S2. The visual freshness agent performs hardware-level differential detection on adjacent consecutive image frames and uses the wolf pack algorithm to filter visual physical micro-change features to generate visual freshness records. S3. The radar freshness agent performs periodic comparisons of continuous radar echo data and uses the wolf pack algorithm to filter radar physical micro-perturbation features to generate radar freshness records. S4. Generate a cross-modal evidence alignment table based on visual freshness records, radar freshness records, vehicle motion status data, and electronic rearview mirror display refresh status data; perform physical consistency verification; and generate physical freshness judgment results and fusion channel control commands. S5. Select the fusion channel based on the physical freshness judgment result and the fusion channel control instruction, generate blind spot display information or conservative blind spot warning information, write it into the electronic rearview mirror display cache and generate blind spot link feedback record.

[0020] S1 specifically includes the following sub-steps: S110. Acquire continuous image frames output by the electronic rearview mirror camera, continuous radar echo data output by the millimeter-wave radar, vehicle motion status data, and electronic rearview mirror display refresh status data. Perform field integrity verification and abnormal status marking on the above data to generate the original acquisition record.

[0021] Specifically, the FPGA is a field-programmable gate array. Internally, it receives data from the electronic rearview mirror camera, millimeter-wave radar, vehicle motion status acquisition unit, and electronic rearview mirror display control unit at fixed acquisition cycles, and writes the acquisition cycle number for the data within the same receiving window.

[0022] For continuous image frames, at least the image frame number, image acquisition count, image buffer address, image acquisition time, and image verification field should be recorded; for continuous radar echo data, at least the radar cycle number, radar acquisition time, target range, target velocity, target angle, echo phase, and echo energy should be recorded. Vehicle motion status data should at least record the vehicle motion status number, vehicle motion status acquisition time, vehicle speed, yaw rate, steering angle, and vehicle vibration status; electronic rearview mirror display refresh status data should at least record the display cache number, display refresh time, refresh count, and display output status.

[0023] Field integrity verification is used to determine whether each acquisition cycle has the necessary fields required for subsequent physical freshness verification. The necessary fields include image frame number, radar cycle number, vehicle motion status number, display cache number, corresponding acquisition time, and corresponding cache address.

[0024] If any necessary field is missing, the FPGA will remove the data combination corresponding to that acquisition cycle from the normal binding queue and generate a field missing deletion record. The field missing deletion record will not enter the normal blind data buffer sequence, but will only be saved as a basis for link diagnosis.

[0025] If the necessary fields are complete, but the image verification field changes while the image acquisition count does not increase, or the radar cycle number increases while the echo phase and echo energy do not change accordingly within the continuous acquisition cycle, or the display refresh count does not increase for two consecutive acquisition cycles, then the corresponding data will not be directly removed, but a freeze suspect mark will be written into the original acquisition record.

[0026] The suspected freeze flag is used to record whether there is a suspicion of data stagnation, and the suspected freeze value is used to record the degree of suspicion of data stagnation. The suspected freeze flag is a status field, and the suspected freeze value is a numerical field. The suspected freeze value is used as the basis for subsequent threshold judgment.

[0027] The original acquisition records serve as the common input for S120 to establish the task flow of the multi-agent system, S130 to establish the blind data cache sequence, and subsequent physical freshness verification.

[0028] S120. Establish a multi-agent system inside the FPGA, allocate tasks based on the data source of the original acquisition records and the frozen suspect markers, and generate an agent task allocation table.

[0029] Specifically, a multi-agent system refers to a processing structure that runs in the form of a parallel hardware state machine inside an FPGA and exchanges data through on-chip cache. It includes at least a visual freshness agent, a radar freshness agent, a vehicle motion agent, a display monitoring agent, and a fusion gatekeeper agent.

[0030] The visual freshness agent reads continuous image frames, image acquisition counts, image cache addresses, and image verification fields for subsequent extraction of road texture flow, target edge displacement, ambient light changes, and image micro-shifts; the radar freshness agent reads continuous radar echo data for subsequent extraction of phase perturbations, ground scattering zone drift, target echo changes, and echo energy fluctuations. The vehicle motion agent reads vehicle motion state data to provide motion constraints for visual physical micro-change features and radar physical micro-perturbation features, and performs cross-modal evidence alignment; the display monitoring agent reads electronic rearview mirror display refresh status data to determine whether the display cache is repeatedly output and whether the display write is valid; the fusion gatekeeper agent receives records output by each agent to perform physical consistency verification and fusion channel control.

[0031] The agent task allocation table records at least the agent ID, input data type, input cache address, processing cycle, output record name, output receiving agent, and exception return field. For data marked with a "frozen" flag, the agent task allocation table does not exclude it; instead, it synchronously assigns the "frozen" flag to the corresponding agent, enabling the visual freshness agent, radar freshness agent, and display monitoring agent to perform focused verification based on this flag during subsequent processing.

[0032] In this way, each type of data in the original acquisition record has a clear processing subject, output object, and abnormal return path, preventing unattended buffering or cross-cycle mismatch in continuous image frames, continuous radar echo data, vehicle motion status data, and electronic rearview mirror display refresh status data within the FPGA.

[0033] The agent task assignment table serves as the basis for S130 to establish the blind data cache sequence and the physical freshness search population.

[0034] Furthermore, to enable the parallel operation of the hardware state machine of the aforementioned multi-agent system, in the FPGA on-chip resource configuration, continuous image frames and continuous radar echo data are streamed and buffered in a ping-pong buffer mode through a dual-port block-level random access memory (DRAM).

[0035] The visual freshness agent and the radar freshness agent are directly mounted on the read port of the corresponding BRAM based on the AXI4-Stream bus protocol.

[0036] The agent task allocation table is instantiated as a global register array within the FPGA. It triggers a clock enable signal through state machine state transitions, thereby controlling the activation and sleep of each agent. This avoids the ineffective consumption of FPGA multiply-accumulator (DSP Slice) resources during periods of spurious real-time data stagnation.

[0037] S130. Based on the original acquisition records and the agent task allocation table, establish a blind data cache sequence, generate cache valid markers, and initialize the physical freshness search population of the wolf pack algorithm.

[0038] Specifically, the FPGA binds the image frame number, radar cycle number, vehicle motion status number, and display cache number into a blind spot data cache unit according to the acquisition cycle number, and arranges multiple blind spot data cache units in chronological order to form a blind spot data cache sequence.

[0039] When binding, first calculate the maximum time deviation between different data within the same collection period: In the formula, This represents the maximum time deviation in the k-th acquisition cycle, where k is the acquisition cycle number. This represents the image acquisition time corresponding to the kth acquisition cycle. This represents the radar acquisition time corresponding to the kth acquisition cycle. The vehicle motion state acquisition time corresponds to the kth acquisition cycle. This represents the display refresh time corresponding to the kth acquisition cycle. This is for calculating the maximum value.

[0040] If the maximum time deviation is not greater than the preset binding threshold and the necessary fields are complete, the data for that collection period is written into the normal binding queue; if the maximum time deviation is greater than the preset binding threshold, a record to be aligned is generated and used as the basis for subsequent skip processing or downgrade processing.

[0041] Cache validity tags are generated using the following formula: In the formula, This is a valid cache marker for the k-th acquisition cycle, where 1 indicates that the corresponding blind data cache unit is valid, and 0 indicates that the corresponding blind data cache unit is invalid. This is the preset binding threshold.

[0042] The FPGA only allows blind filler data cache units with a valid cache mark of 1 to enter visual freshness extraction and radar freshness extraction. For blind filler data cache units with a valid cache mark of 0, the cause of the anomaly is retained but not used as valid evidence in real time.

[0043] The wolf pack algorithm is a heuristic search algorithm that uses alpha wolf candidates to guide, explorer wolves to expand, and encircle and converge to select effective physical evidence from a set of candidate physical micro-change features.

[0044] The physical freshness search population consists of image texture candidate points, target edge candidate points, ambient light candidate points, display refresh candidate points, radar phase candidate points, ground scattering candidate points, and target echo candidate points. Among them, the display refresh candidate points are used by the display monitoring agent in subsequent display refresh scoring and display write validity judgment.

[0045] Each candidate point is recorded with its candidate number, source type, cache address, initial confidence value, frozen suspect flag, and corresponding collection period. The initial search score for a candidate point is calculated using the following formula: In the formula, Let i be the initial search score for the i-th candidate point, where i is the candidate point number. The degree of consecutive occurrence of the i-th candidate point. The degree of matching between the i-th candidate point and the vehicle motion state data. Let i be the data quality value of the i-th candidate point. Let i be the freezing suspicion value of the i-th candidate point. , , , These are the weighting coefficients for the degree of continuous occurrence, the degree of matching, the data quality value, and the suspected freezing value, respectively.

[0046] The FPGA sets the candidate point with the highest initial search score and a valid cache mark of 1 as the alpha candidate, sets the candidate point in the adjacent image area or adjacent radar range unit as the scout candidate, and sets the candidate point with an initial search score lower than the preset score threshold and a freezing suspicion value higher than the preset freezing threshold as the candidate to be deleted.

[0047] The resulting blinding data cache sequence, cache valid markers, and physical freshness search population serve as inputs to S210, S310, and S410, respectively, enabling S110-S130 to form an executable closed loop for data acquisition, task allocation, and candidate evidence initialization.

[0048] S2 specifically includes the following sub-steps: S210, the visual freshness agent reads the blind spot data cache sequence, cached valid tags and physical freshness search population, performs hardware-level differential detection on adjacent consecutive image frames, and generates a set of visual candidate micro-change features.

[0049] Specifically, the visual freshness agent first reads the blind spot data cache sequence according to the acquisition cycle number and verifies the cache validity flag corresponding to each blind spot data cache unit; when the cache validity flag is 1, it reads the continuous image frames, image acquisition count, image cache address, image verification field, frozen suspect flag, and image texture candidate points, target edge candidate points, and ambient light candidate points in the physical freshness search population in that blind spot data cache unit. When the cache validity flag is 0, hardware-level differential detection is not performed, and a visual skip record is generated. The visual skip record includes at least the image frame number, acquisition cycle number, skip reason, and cache validity flag, which is used by the subsequent fusion gatekeeper agent to identify that the acquisition cycle cannot be used as real-time visual evidence.

[0050] Hardware-level differential detection is performed in pixel-parallel mode within the FPGA. It differentiates the brightness values ​​at the same pixel location in adjacent consecutive image frames to obtain the image difference value. In the formula, For the kth acquisition cycle at pixel position The image difference value at point k, where k is the acquisition cycle number. Image pixel position, For the kth acquisition cycle at pixel position The brightness value at that location, For the first Each acquisition cycle at pixel position The brightness value at that location.

[0051] The visual freshness agent compares the image difference value with a preset difference threshold. If the image difference value reaches the preset difference threshold and the area it is located in corresponds to a road texture candidate point, a target edge candidate point, or an ambient light candidate point, then the area is written into the visual candidate micro-change feature set. If the image acquisition count or image verification field has changed, but no corresponding difference has been generated in the road texture, target edge, and ambient light candidate areas, then the corresponding area is written into the frozen suspected candidate area.

[0052] The set of visual candidate micro-change features includes at least candidate number, candidate source type, candidate location, image difference intensity, change direction, consecutive occurrence count, noise suspicion value, frozen suspicion value, frozen suspicion marker, and corresponding acquisition period, which serve as inputs for the wolf pack algorithm in S220 to screen visual physical micro-change features.

[0053] S220. The wolf pack algorithm is used to filter visual physical micro-change features from the visual candidate micro-change feature set to generate a visual physical micro-change feature set.

[0054] Specifically, the wolf pack algorithm filters valid visual evidence from the set of visual candidate micro-change features in the order of alpha wolf candidate guidance, scout wolf candidate expansion, and encirclement convergence deletion. The visual freshness agent first calculates the visual side fitness score for each visual candidate point: In the formula, The visual side fitness score is given to the i-th visual candidate point, where i is the visual candidate point number. Let be the image difference intensity of the i-th visual candidate point. The degree of directional matching between the i-th visual candidate point and the vehicle motion state data. Let represent the degree of consecutive occurrence of the i-th visual candidate point in consecutive image frames. Let be the noise suspicion value of the i-th visual candidate point. , , , These are the weighting coefficients for image difference intensity, orientation matching degree, continuity of occurrence degree, and noise suspicion value, respectively.

[0055] The visual freshness agent sets the candidate points with the highest visual adaptation score, effective cache marking of 1, and freezing suspicion value below the preset freezing threshold as alpha wolf candidates; sets the candidate points located in adjacent image regions with the alpha wolf candidates, with the same direction of change and a direction matching degree reaching the preset matching threshold as scout wolf candidates; and then expands the search for adjacent candidate points along the road texture flow direction, target edge movement direction, and ambient light change direction.

[0056] The specific formula for updating the position of the wolf-hunting candidates during the extended search process is as follows: In the formula, Let h be the coordinates of the new pixel after the h-th wolf probe expansion. Here, h represents the pixel coordinates before the h-th wolf exploration expansion, where h is the number of search iterations. The step size for visual spatial search is negatively correlated with the local differential gradient of the image, meaning the step size decreases in regions with strong gradients. The direction vector is guided by the alpha candidate, and its value is determined by the main optical flow direction of the feature points in two consecutive frames; rand(-1,1) is the output of a hardware random number generator that generates random numbers between -1 and 1. This hardware-level search process is executed within a set number of iterations until a new candidate point with a higher visual fitness score is found, or the local maximum boundary is reached, at which point the search stops.

[0057] During the encirclement and convergence process, the visual freshness agent deletes candidate points that appear only in a single frame, have insufficient continuous occurrence, are located at the boundary of a compressed block, are located in a fixed noise region, whose change direction does not match the vehicle speed or yaw rate, and are formed solely by display afterimages.

[0058] The candidate points retained after deletion are written into a visual physical micro-change feature set, which includes at least feature number, feature location, evidence type, image difference intensity, change direction, number of consecutive occurrences, visual side fitness score, and frozen suspect marker, for S230 to generate visual freshness record.

[0059] S230. Generate a visual freshness record based on the set of visual physical micro-change features, and send the visual freshness record to the fusion gatekeeper agent.

[0060] Specifically, the visual freshness agent counts the number of visual physical micro-change features confirmed as valid within each acquisition cycle and calculates the visual freeze suspicion value: In the formula, This represents the visual freeze suspicion value for the k-th acquisition period, where k is the acquisition period number. This represents the number of visual physical micro-change features confirmed as valid within the k-th acquisition period. To preset a threshold for the number of valid visual evidence, The operation is performed to select the smaller value.

[0061] This calculation is used to indicate that the suspicion of freezing increases when visual evidence is insufficient and decreases when visual evidence reaches a threshold. For example, when the preset threshold for the number of valid visual evidences is 6 and the number of valid visual physical micro-change features is 2, the visual freezing suspicion value is 0.67, which can be marked as visual non-real-time unavailable.

[0062] Visual freshness records include at least the image frame number, acquisition cycle number, visual evidence location, visual evidence type, visual update magnitude, visual continuity count, visual freeze suspicion value, freeze suspicion marker, cache valid marker, and output receiver agent.

[0063] If the visual freeze suspicion value is lower than the preset visual freeze threshold, the visual freshness record is marked as visually real-time valid; if the visual freeze suspicion value is not lower than the preset visual freeze threshold, the visual freshness record is marked as visually non-real-time unavailable.

[0064] Regardless of whether the visual freshness record is valid in real time, the visual freshness agent sends it to the fusion gatekeeper agent. Records of visually unavailable but not real-time conditions cannot be used alone as security evidence for the normal fusion channel; they are only used as risk evidence for cross-modal evidence alignment in S410 and physical consistency verification in S420. Visual skip records are sent to the fusion gatekeeper agent as anomalous sub-records of the visual freshness record, enabling the fusion gatekeeper agent to read the reason for visual skipping from the visual freshness record in S410.

[0065] S3 specifically includes the following sub-steps: S310, the radar freshness agent reads the blind spot data cache sequence, cached valid tags, physical freshness search population and vehicle motion state data, performs periodic comparison on continuous radar echo data, and generates a set of radar candidate micro-perturbation features.

[0066] Specifically, the radar freshness agent reads the blind spot data cache sequence according to the acquisition cycle number, and first verifies the cache validity flag corresponding to each blind spot data cache unit. When the cache validity flag is 1, the radar cycle number, target distance, target speed, target angle, echo phase, echo energy, frozen suspect flag, and vehicle speed, yaw rate, steering angle, and vehicle vibration state in the vehicle motion state data of the blind spot data cache unit are read. When the cache validity flag is 0, radar cycle comparison is not performed, and a radar skip record is generated. The radar skip record includes at least the radar cycle number, acquisition cycle number, skip reason, and cache validity flag, which is used by the subsequent fusion gatekeeper agent to identify that the acquisition cycle cannot be used as real-time radar evidence.

[0067] The radar freshness agent performs periodic comparisons on continuous radar echo data from adjacent acquisition cycles. The periodic comparisons include echo phase perturbation comparison, ground scattering band drift comparison, target range change comparison, target velocity change comparison, target angle change comparison, and echo energy fluctuation comparison.

[0068] The echo phase disturbance value is calculated using the following formula: In the formula, Let be the echo phase disturbance value in the k-th acquisition cycle, where k is the acquisition cycle number. The echo phase of the kth acquisition cycle, For the first The echo phase of each acquisition cycle.

[0069] The radar freshness agent compares the echo phase disturbance value, the change in the ground scattering zone position, the change in the target distance, the change in the target speed, and the change in the echo energy with the corresponding thresholds. When the vehicle motion status data shows changes in vehicle speed, yaw rate, or steering angle, and the aforementioned radar-side changes are all below the corresponding thresholds, the radar cycle is written into the suspected freeze cycle. When the radar cycle number increases but the target distance, target speed, target angle, echo phase, and echo energy remain the same or only show fixed-format repetitive changes in more than two consecutive acquisition cycles, the radar cycle is also written into the suspected freeze cycle.

[0070] The radar candidate micro-perturbation feature set includes at least the candidate number, candidate source type, radar evidence location, range perturbation, velocity perturbation, angle perturbation, phase perturbation, energy perturbation, repeated cached suspected values, frozen suspected markers, and corresponding acquisition period, which serve as inputs for the wolf pack algorithm in S320 to screen radar physical micro-perturbation features.

[0071] S320. The wolf pack algorithm is used to filter radar physical perturbation features from the radar candidate perturbation feature set to generate a radar physical perturbation feature set.

[0072] Specifically, the wolf pack algorithm filters valid radar evidence from the radar candidate micro-perturbation feature set in the order of alpha wolf candidate selection, scout wolf candidate expansion, and encirclement convergence deletion. The radar freshness agent first calculates the radar-side fitness score for each radar candidate point: In the formula, The radar-side fitness score is given for the j-th radar candidate point, where j is the radar candidate point number. Let be the phase perturbation intensity of the j-th radar candidate point. The degree of ground scattering band drift of the j-th radar candidate point. The degree of matching between the j-th radar candidate point and the vehicle motion state data. The effectiveness of the echo energy of the j-th radar candidate point. For the j-th radar candidate point, a duplicate cached suspected value is provided. , , , , These are the weighting coefficients for phase perturbation intensity, ground scattering band drift, matching degree, echo energy effectiveness, and suspected duplicate buffering value, respectively.

[0073] The radar freshness agent sets the candidate point with the highest radar-side fitness score, a cache effective mark of 1, and a duplicate cache suspicion value below a preset duplicate threshold as the alpha candidate; it sets the candidate point located in adjacent distance, angle, or speed cells and matching the changes in vehicle speed, yaw rate, or steering angle as the scout candidate; the scout candidate expands its search along the distance, angle, and speed cells to search for adjacent candidate points to cover the continuous performance of the same real radar disturbance in different echo cells.

[0074] In the radar multidimensional data matrix, the extended search for wolf detection candidates is strictly constrained by physical resolution in the three orthogonal dimensions of range, velocity, and angle. Its search update rules are as follows: In the formula, This is the index of radar candidate cells after the b-th expansion (containing three-dimensional coordinates of range, velocity, and angle). Here is the radar candidate cell index before the update, and b is the number of radar domain expansion walks; The search step size vector has components in the range, velocity, and angle dimensions that are not greater than the corresponding physical resolution threshold of the radar hardware. The physical prediction of changes in vehicle motion state data mapped to the radar coordinate system. This is the sign function. This mechanism ensures that the perturbation characteristics searched by the algorithm are strictly in the same direction in geometric space as the current physical motion trend of the vehicle.

[0075] During the convergence process, the radar freshness agent deletes candidate points that are repeatedly output in a fixed format, candidate points that jump only in a single acquisition cycle, phase jump candidate points that do not correspond to vehicle motion state data, candidate points whose echo energy is lower than the noise threshold, and duplicate candidate points formed by buffer playback.

[0076] The candidate points retained after deletion are written into the radar physical micro-perturbation feature set, which includes at least the feature number, radar evidence location, evidence type, range perturbation, velocity perturbation, angle perturbation, phase perturbation, energy perturbation, radar side fitness score, number of consecutive occurrences, and frozen suspect markers, for S330 to generate radar freshness records.

[0077] S330: Generate a radar freshness record based on the radar physical micro-perturbation feature set, and send the radar freshness record to the fusion gatekeeper agent.

[0078] Specifically, the radar freshness agent counts the number of radar physical micro-perturbations confirmed as valid within each acquisition cycle and calculates the radar freeze suspicion value: In the formula, This represents the radar freeze suspect value for the k-th acquisition cycle, where k is the acquisition cycle number. This represents the number of radar physical micro-perturbation features confirmed as valid within the k-th acquisition period. To preset a threshold for the number of valid radar evidence, The operation is performed to select the smaller value.

[0079] This calculation is used to indicate that the suspicion of radar freezing increases when there are insufficient effective radar physical perturbation features, and decreases when the suspicion of radar freezing decreases when the effective radar physical perturbation features reach a threshold. For example, when the preset threshold for the number of effective radar evidences is 5 and the number of effective radar physical perturbation features is 1, the radar freezing suspicion value is 0.8, which can be marked as radar not being available in real time.

[0080] The radar freshness record includes at least the radar cycle number, acquisition cycle number, radar evidence location, radar evidence type, range perturbation, velocity perturbation, angle perturbation, phase perturbation, energy perturbation, radar continuity count, radar freeze suspicion value, freeze suspicion flag, buffer validity flag, and output receiver agent. If the radar freeze suspicion value is lower than the preset radar freeze threshold, the radar freshness record is marked as radar real-time valid; if the radar freeze suspicion value is not lower than the preset radar freeze threshold, the radar freshness record is marked as radar non-real-time unavailable.

[0081] Regardless of whether the radar freshness record is valid in real time, the radar freshness agent sends it to the fusion gatekeeper agent. Notable radar records that are not available in real time cannot be used alone as security evidence for the normal fusion channel; they are only used as risk evidence for cross-modal evidence alignment in S410 and physical consistency verification in S420. Radar skip records are sent to the fusion gatekeeper agent as anomalous sub-records of the radar freshness record, enabling the fusion gatekeeper agent to read the reason for radar skipping from the radar freshness record in S410.

[0082] S4 specifically includes the following sub-steps: S410: The vehicle motion agent reads visual freshness records, radar freshness records, vehicle motion state data, and electronic rearview mirror display refresh state data, aligns visual evidence, radar evidence, vehicle motion state, and display refresh state to the same acquisition cycle, and generates a cross-modal evidence alignment table.

[0083] Specifically, the vehicle motion agent reads the visual freshness record generated by S230, the radar freshness record generated by S330, the vehicle motion state data generated by S110, and the electronic rearview mirror display refresh status data according to the acquisition cycle sequence number. It then verifies whether the image frame sequence number, radar cycle sequence number, vehicle motion state number, and display cache number in the above records correspond to the same blind spot data cache unit.

[0084] The cross-modal evidence alignment table is a data table used to record the correspondence between visual evidence, radar evidence, vehicle motion state, and display refresh state. It includes at least the acquisition cycle number, image frame number, radar cycle number, vehicle motion state number, display cache number, visual real-time valid marker, radar real-time valid marker, visual frozen suspected value, radar frozen suspected value, display refresh count, visual evidence position, radar evidence position, visual skip reason, radar skip reason, and alignment status field.

[0085] The vehicle motion agent performs cross-modal time deviation verification for the same acquisition cycle. The cross-modal alignment deviation is calculated using the following formula: In the formula, This represents the cross-modal alignment deviation in the k-th acquisition cycle, where k is the acquisition cycle number. The image acquisition time for the kth acquisition cycle is... Let k be the radar acquisition time for the kth acquisition cycle. The time for collecting vehicle motion status data in the k-th collection cycle is denoted as . This represents the display refresh time for the kth acquisition cycle. This is for calculating the maximum value.

[0086] Cross-modal alignment deviation Used for secondary verification of the data that has already been initially bound in S130; it does not replace the maximum time deviation in S130. ,in, Used for initial binding Used for cross-modal verification before subsequent physical consistency checks.

[0087] When the cross-modal alignment deviation is not greater than the preset alignment threshold, the vehicle motion agent marks the acquisition period as aligned and valid; when the cross-modal alignment deviation is greater than the preset alignment threshold, the acquisition period is marked as aligned and the reason for the alignment error is written into the cross-modal evidence alignment table.

[0088] For acquisition cycles with visual skip records, radar skip records, visual non-real-time unavailable markers, or radar non-real-time unavailable markers, the vehicle motion agent does not delete the corresponding records. Instead, it writes the corresponding risk sources into the cross-modal evidence alignment table for the S420 to continue participating in physical consistency verification.

[0089] The S420 integrates the gatekeeper agent to read the cross-modal evidence alignment table, performs physical consistency verification between the visual side, radar side, display side, and vehicle motion side, and generates a physical freshness score table and a false real-time risk level.

[0090] Specifically, the fusion gatekeeper agent does not judge whether the blind spot data is real-time and effective based solely on data format, timestamp, or single sensor results. Instead, it judges whether continuous image frames, continuous radar echo data, and electronic rearview mirror display refresh status data jointly present physical changes that match the vehicle's motion status data.

[0091] When vehicle motion data indicates changes in vehicle speed, yaw rate, steering angle, or vehicle vibration, the visual side should show corresponding evidence in road texture flow, target edge displacement, ambient light changes, or slight image shifts. The radar side should show corresponding evidence in phase disturbances, ground scattering zone drift, target distance and speed changes, or echo energy fluctuations. The display side should show increasing display refresh count, switching display cache number, or normal display output status.

[0092] If either side only exhibits format updates without any interpretable physical changes, the fusion gatekeeper agent will write that side into the source of false real-time risk. The overall score for physical freshness is calculated using the following formula: In the formula, The comprehensive score for physical freshness in the k-th collection period, where k is the collection period number. The visual freshness score for the k-th acquisition period is given. The radar freshness score is given for the k-th acquisition cycle. The score is refreshed for the kth collection period. The cross-modal consistency score for the k-th acquisition cycle. This is the pseudo-real-time risk deduction value for the k-th acquisition period. , , , , These are the weighting coefficients for visual freshness score, radar freshness score, display refresh score, cross-modal consistency score, and false real-time risk deduction value, respectively.

[0093] The visual freshness score is determined by the reverse of the visual freeze suspicion value, the radar freshness score is determined by the reverse of the radar freeze suspicion value, the display refresh score is determined by whether the display refresh count increases, whether the display cache number is switched, and whether the display output status is normal, and the cross-modal consistency score is determined by the matching relationship between the visual evidence position, the radar evidence position, and the vehicle motion state.

[0094] The pseudo-real-time risk level is generated by the following formula: In the formula, This represents the false real-time risk level for the k-th data collection period, where 0 indicates no false real-time risk, 1 indicates mild false real-time risk, and 2 indicates severe false real-time risk. For high scoring thresholds, The low scoring threshold is set, and the high scoring threshold is greater than the low scoring threshold.

[0095] When both the visual and radar sides are marked as non-real-time unavailable, or when the display refresh count does not increase for two consecutive acquisition cycles, the fusion gatekeeper agent will set the false real-time risk level to at least 2, even if the data format is complete.

[0096] The physical freshness scoring table includes at least visual freshness score, radar freshness score, display refresh score, cross-modal consistency score, physical freshness comprehensive score, false real-time risk deduction value, and false real-time risk level, which serve as the basis for S430 to generate control results.

[0097] S430 and the fusion gatekeeper agent generate physical freshness judgment results and fusion channel control instructions based on the physical freshness scoring table and the false real-time risk level, and send them to the corresponding fusion channel control process of S510.

[0098] Specifically, the physical freshness determination results include at least three categories: real-time valid, unilaterally downgraded valid, and non-real-time unusable. Real-time valid means that the visual freshness record, radar freshness record, and electronic rearview mirror display refresh status data have all passed the physical consistency check, and the false real-time risk level is 0; A single-sided downgrade is valid if either the visual or radar side has a non-real-time unavailable flag, but the other side is consistent with the vehicle motion status data and the electronic rearview mirror display refresh status data, and the false real-time risk level is no higher than 1. Non-real-time unavailability indicates that there are freezes, cached duplicate outputs, cross-modal alignment anomalies, or display refresh anomalies on the visual, radar, or display sides that cannot be explained by the vehicle's motion state, and the pseudo-real-time risk level is 2.

[0099] The fusion channel control commands include at least the normal fusion command, the unilateral degradation fusion command, the conservative blind spot filling command, the blocking fusion command, and the risk warning display command. The normal fusion command is used by the S510 to open the normal fusion channel, allowing real-time valid continuous image frames and continuous radar echo data to enter the radar-view fusion blind spot filling channel; the unilateral degradation fusion command is used by the S510 to block data from the side that is not available in real time, and retain the real-time valid data from the side to enter the conservative blind spot filling channel; The conservative blind spot filling instruction is used by S520 to generate conservative blind spot filling warning information based on the most recent real-time valid record, vehicle motion status data, and blind spot risk boundary; the blocking fusion instruction is used by S510 to block blind spot filling data that is determined to be non-real-time and unusable from entering the normal fusion channel; the display risk warning instruction is used by S530 to write the false real-time risk level and corresponding warning into the electronic rearview mirror display cache.

[0100] Therefore, the physical freshness determination result and fusion channel control command output by S430 can directly control the fusion channel selection, blind spot display generation and display cache writing of S510-S530, without generating intermediate results that are irrelevant to subsequent steps.

[0101] S5 specifically includes the following sub-steps: S510: The fusion gatekeeper agent reads the physical freshness determination result, fusion channel control instructions and pseudo-real-time risk level, selects the corresponding fusion channel, and generates a fusion channel status record.

[0102] Specifically, the fusion gatekeeper agent reads the physical freshness judgment result, fusion channel control command and false real-time risk level output by S430 according to the acquisition cycle number, and verifies whether the visual freshness record, radar freshness record, cross-modal evidence alignment table and physical freshness scoring table corresponding to the acquisition cycle have the same acquisition cycle number.

[0103] To prevent non-real-time and unavailable data from continuing to enter the normal fusion process, the fusion gatekeeper agent generates a fusion channel status code for each acquisition cycle: In the formula, This is the status code of the fusion channel in the kth acquisition cycle, where k is the acquisition cycle number, 0 indicates a normal fusion channel, 1 indicates a single-sided downgraded fusion channel, and 2 indicates that normal fusion is blocked and a conservative blind spot filling channel is enabled.

[0104] When the fusion channel status code is 0, the fusion gate agent allows real-time valid continuous image frames and continuous radar echo data to enter the normal fusion channel simultaneously, and writes the data numbers of both into the allowed data number field. At the same time, the fusion gate agent writes the visual freshness record, radar freshness record, vehicle motion state data and blind spot risk boundary corresponding to the acquisition cycle into the real-time valid record cache, as the most recent real-time valid record for subsequent conservative blind spot filling channel calls.

[0105] When the fusion channel status code is 1, the fusion gatekeeper agent only allows real-time valid data from one side to enter the single-side downgraded fusion channel, writes non-real-time unavailable data from the other side into the blocked data list, and retains its image frame number, radar cycle number, frozen suspected value, and risk source.

[0106] When the fusion channel status code is 2, the fusion gatekeeper agent blocks the continuous image frames and continuous radar echo data of the current acquisition cycle as the basis for normal fusion safety, and only retains the most recent real-time valid record, vehicle motion status data and blind spot risk boundary to enter the conservative blind spot filling channel.

[0107] The fusion channel status record includes at least the acquisition cycle number, fusion channel status code, physical freshness judgment result, false real-time risk level, allowed data number, blocked data number, blocking reason, reserved diagnostic fields, and output object, which serve as direct inputs for the S520 to generate and display content.

[0108] S520: Generate electronic rearview mirror blind spot display information, unilateral downgrade blind spot information, or conservative blind spot warning information based on the fusion channel status record.

[0109] Specifically, when the fusion channel status code is 0, the fusion gatekeeper agent reads real-time valid continuous image frames, continuous radar echo data, visual freshness records and radar freshness records, performs radar-visual fusion blind spot filling, and generates electronic rearview mirror blind spot filling display information.

[0110] The blind spot information displayed by the electronic rearview mirror includes at least the target display position, target risk level, blind spot occupancy marker, target distance, target speed, display overlay area, and data source marker. The data source marker indicates that the displayed information is derived from both real-time, valid visual data and radar data.

[0111] When the fusion channel status code is 1, the fusion gatekeeper agent does not use non-real-time unavailable data in the blocked data list to generate a safety conclusion. Instead, it generates unilateral degradation and blind spot filling information based on real-time effective one-sided data, vehicle motion status data, and blind spot risk boundaries, and writes a data source restricted flag into the displayed content.

[0112] When the fusion channel status code is 2, the fusion gatekeeper agent does not output a conclusion such as "blind spot safety". Instead, it generates conservative blind spot filling warning information based on the most recent real-time effective record, the current vehicle motion status data and the blind spot risk boundary.

[0113] The conservative blind zone boundary is expanded as follows: In the formula, This represents the conservative blind zone boundary for the k-th acquisition cycle, where k is the acquisition cycle number. For the most recent real-time effective record corresponding to the blind zone boundary, The conservative expansion coefficient is used. The vehicle speed in the kth data collection cycle. This is the time interval between the current acquisition cycle and the most recent valid real-time acquisition cycle.

[0114] This calculation indicates that the higher the vehicle speed or the longer the time since the most recent valid real-time record, the more the conservative blind zone boundary expands outward, in order to avoid outputting overly optimistic blind spot compensation results in a pseudo-real-time state.

[0115] When electronic rearview mirror blind spot compensation information, unilateral downgraded blind spot compensation information, and conservative blind spot compensation warning information are all present, the display priority is as follows: conservative blind spot compensation warning information, unilateral downgraded blind spot compensation information, and electronic rearview mirror blind spot compensation information. When the false real-time risk level is 2, the electronic rearview mirror display must include a false real-time risk warning and must not display a blind spot safety warning. The generated display information is sent to the display monitoring agent for the S530 to perform display cache write and display refresh verification.

[0116] S530 and the display monitoring agent write the electronic rearview mirror blind spot information, unilateral downgrade blind spot information, or conservative blind spot warning information into the electronic rearview mirror display cache, and generate blind spot link feedback records.

[0117] Specifically, after receiving the display information output by the S520, the display monitoring agent writes it into the electronic rearview mirror display cache according to the display priority, and records the corresponding display cache number, display refresh count and display output status.

[0118] After writing, the display monitoring agent reads the electronic rearview mirror display refresh status data again to confirm whether the display cache has been switched, whether the display refresh count has increased, and whether the display output status is normal. The display write validity flag is generated according to the following formula: In the formula, This is a valid display write flag for the kth acquisition cycle, where k is the acquisition cycle number, 1 indicates a valid display write, and 0 indicates a display write error.

[0119] When the display write validity flag is 0, the display monitoring agent marks the acquisition cycle as a display refresh anomaly and writes the display refresh anomaly as a new source of false real-time risk into the blind spot link feedback record; when the display write validity flag is 1, the corresponding display information is marked as output. The blind spot link feedback record includes at least the acquisition cycle number, display cache number, display refresh count, display write validity flag, physical freshness judgment result, fusion channel status code, blind spot display type, false real-time risk level, blocked data number, risk warning type, and next cycle reflow flag.

[0120] The display monitoring agent writes the blind spot link feedback record into the feedback cache, so that it can be reacquired by S110 as electronic rearview mirror display refresh status data in the next acquisition cycle. It then generates a new original acquisition record together with continuous image frames, continuous radar echo data and vehicle motion status data, so as to continue to participate in field integrity verification, physical freshness verification, fusion channel control and display refresh anomaly judgment, thus forming a closed loop from data acquisition, physical verification, channel control, display output to feedback return.

[0121] In a preferred embodiment based on a real-vehicle engineering project, the core preset parameters and weight configurations involved in the above method are as follows: preset binding threshold The value is set to 50ms to accommodate asynchronous refresh rates between radar and cameras; the threshold for the number of valid visual evidence. Set to 6, the threshold for the number of valid radar evidence. Set to 4; The preferred weighting coefficients in the initial search scoring formula are: =0.3, =0.4, =0.2, =0.1, this configuration tends to prioritize extracting evidence that strongly matches vehicle motion; the preferred weighting coefficient in the physical freshness comprehensive scoring formula is... =0.3, =0.3, =0.15, =0.25.

[0122] High scoring threshold for pseudo-real-time risk level Set to 0.85, low scoring threshold Set to 0.5; conservative expansion coefficient Set to 0.5s. Using the above parameter combination, when bus congestion causes data stagnation for 100ms, the system can accurately trigger a conservative blind spot warning with the highest priority, ensuring the safety of the electronic rearview mirror output image.

[0123] Example 2: Figure 2 As shown, this embodiment provides an FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation system, including: The system includes a data acquisition initialization module, a visual freshness processing module, a radar freshness processing module, a physical consistency verification module, and a fusion display feedback module. The data acquisition initialization module is used to acquire continuous image frames, continuous radar echo data, vehicle motion status data, and electronic rearview mirror display refresh status data. After field integrity verification and abnormal status marking, the original acquisition record is generated. A multi-agent system including visual freshness agent, radar freshness agent, vehicle motion agent, display monitoring agent, and fusion gatekeeper agent is established inside the FPGA. The system generates blind spot data cache sequence, cache validity marker, and physical freshness search population. The visual freshness processing module is used to perform hardware-level differential detection on adjacent consecutive image frames and uses the wolf pack algorithm to filter visual physical micro-change features to generate visual freshness records. The radar freshness processing module is used to periodically compare continuous radar echo data and use the wolf pack algorithm to filter radar physical micro-perturbation features to generate radar freshness records. The physical consistency verification module is used to generate a cross-modal evidence alignment table based on visual freshness records, radar freshness records, vehicle motion state data, and electronic rearview mirror display refresh state data, perform physical consistency verification, and generate physical freshness judgment results and fusion channel control instructions. The fusion display feedback module is used to select the fusion channel based on the physical freshness judgment result and the fusion channel control command, generate electronic rearview mirror blind spot display information, unilateral downgrade blind spot information or conservative blind spot warning information, write it into the electronic rearview mirror display cache and generate blind spot link feedback record.

[0124] All the above formulas are performed using dimensionless numerical calculations; the relevant formulas are based on empirical models that approximate the real situation, obtained through extensive data collection and software simulation fitting. The preset parameters and thresholds involved in the formulas can be conventionally set and adjusted by those skilled in the art according to the physical constraints of the actual application scenario.

[0125] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0127] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for blind spot compensation using radar-visual fusion in electronic rearview mirrors based on FPGA, characterized in that, Includes the following steps: S1. Acquire continuous image frames, continuous radar echo data, vehicle motion status data, and electronic rearview mirror display refresh status data. Generate raw acquisition records after field integrity verification and abnormal status marking. Establish a multi-agent system to generate a blind spot data cache sequence, cache validity markers, and a physical freshness search population. Specifically, this includes: Acquire image frames, radar echo data, vehicle motion status data, and display refresh status data according to the acquisition cycle, verify the number, acquisition time, and cache address, remove data with missing necessary fields from the normal binding queue, write frozen suspicion flags to data with complete fields but stagnant data, and generate the original acquisition record. A multi-agent system is established on the FPGA. The original acquisition records are assigned to the visual freshness agent, radar freshness agent, vehicle motion agent, display monitoring agent, and fusion gatekeeper agent, generating an agent task allocation table. The agent task allocation table records the agent number, input data type, input buffer address, processing cycle, output record name, output receiving agent, and exception return field. A blind spot data cache sequence is established based on the original acquisition records and the agent task allocation table. Valid cache markers are generated and the physical freshness search population is initialized. The physical freshness search population consists of image texture candidate points, target edge candidate points, ambient light candidate points, display refresh candidate points, radar phase candidate points, ground scattering candidate points, and target echo candidate points. S2. The visual freshness agent performs hardware-level differential detection on adjacent consecutive image frames and uses a wolf pack algorithm to filter visual physical micro-change features to generate visual freshness records; specifically including: The visual freshness agent reads the blind spot data cache sequence, cache valid markers and physical freshness search population, performs hardware-level differential detection on adjacent consecutive image frames with valid cache, generates a set of visual candidate micro-change features, and generates visual skip records for acquisition cycles with invalid cache. The wolf pack algorithm is used to filter visual physical micro-change features based on image difference intensity, orientation matching degree, continuous occurrence degree and noise suspicion value, and remove single frame noise, compressed block boundaries and display afterimage candidates; The visual freeze suspicion value is calculated based on the set of visual physical micro-change features, a visual freshness record is generated, and sent to the fusion gatekeeper agent as the visual input for subsequent cross-modal evidence alignment and physical consistency verification. S3. The radar freshness agent performs periodic comparisons of continuous radar echo data and uses a wolf pack algorithm to filter radar physical micro-perturbation features, generating radar freshness records, specifically including: The radar freshness agent reads the blind spot data cache sequence, cache valid markers, physical freshness search population and vehicle motion state data, performs periodic comparison on the cached valid continuous radar echo data, generates a radar candidate micro-perturbation feature set, and generates radar skip records for the acquisition period with invalid cache. The wolf pack algorithm is used to screen radar physical micro-perturbation features based on phase perturbation intensity, ground scattering band drift, vehicle motion matching degree, echo energy effectiveness, and suspected repeated buffering value, and remove candidates with fixed format repeated output, single-cycle jump and low energy noise. The radar freeze suspicion value is calculated based on the radar physical micro-perturbation feature set, a radar freshness record is generated, and sent to the fusion gatekeeper agent as the radar-side input for subsequent cross-modal evidence alignment and physical consistency verification. S4. The vehicle motion agent generates a cross-modal evidence alignment table based on visual freshness records, radar freshness records, vehicle motion state data, and electronic rearview mirror display refresh status data. The fusion gatekeeper agent reads the cross-modal evidence alignment table to perform physical consistency verification and generates physical freshness judgment results and fusion channel control commands.

2. The FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation method according to claim 1, characterized in that, Also includes: S5. Based on the physical freshness determination result and the fusion channel control command, select the fusion channel, generate blind spot display information or conservative blind spot warning information, write it into the electronic rearview mirror display cache, and generate a blind spot link feedback record, specifically including: The fusion gatekeeper agent reads the physical freshness judgment result, fusion channel control command and false real-time risk level, generates fusion channel status record, and selects normal fusion channel, unilaterally downgraded fusion channel or conservative blind-filling channel according to the real-time effective, unilaterally downgraded effective or non-real-time unavailable status. Based on the status record of the fusion channel, electronic rearview mirror blind spot display information, unilateral downgrade blind spot information, or conservative blind spot warning information are generated, and blind spot safety prompts are prohibited from being output under heavy real-time risk conditions. The display monitoring agent writes the corresponding display information into the electronic rearview mirror display cache, generates a blind spot feedback record, and then flows back to participate in the generation of the original acquisition record in the next acquisition cycle.

3. The FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation method according to claim 1, characterized in that, S4 specifically includes: The vehicle motion agent reads visual freshness records, radar freshness records, vehicle motion status data, and display refresh status data, performs time deviation verification according to the collection cycle, and generates a cross-modal evidence alignment table. The fusion gatekeeper agent performs physical consistency checks on the visual, radar, display, and vehicle motion sides based on a cross-modal evidence alignment table, generating a physical freshness score table and a false real-time risk level. The fusion gatekeeper agent generates physical freshness judgment results and fusion channel control instructions based on the physical freshness scoring table and the pseudo-real-time risk level. The physical freshness judgment results include real-time effective, unilateral downgrade effective, and non-real-time unavailable. The fusion channel control instructions are used for subsequent channel selection.

4. An FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation system, employing the FPGA-based electronic rearview mirror radar-visual fusion blind spot compensation method as described in any one of claims 1 to 3, characterized in that, include: The system includes a data acquisition initialization module, a visual freshness processing module, a radar freshness processing module, a physical consistency verification module, and a fusion display feedback module.

Images