A method and device based on 4D millimeter wave radar and V2X application fusion

Abstract

The application discloses a kind of based on 4D millimeter wave radar and V2X application fusion method and device, the method includes: using 4D millimeter wave radar to carry out active detection to environment surrounding target, obtains the distance, angle, speed and height information of target, generates point cloud data;The point cloud data is transmitted to V2X equipment;The computing unit of the V2X equipment uses pre-training model to analyze the point cloud data, identifies the type and position information of target;The V2X communication unit of the V2X equipment, the type and position information of target identified are broadcast to surrounding support V2X communication equipment by V2X communication technology.The method can strengthen the reliability of car-road cooperation application, improve road efficiency, the ability of safe traffic.

Description

Technical Field

[0001] This invention relates to the fields of 4D millimeter-wave radar sensing technology, artificial intelligence technology, and V2X communication technology. More specifically, it relates to a method and apparatus based on the integration of 4D millimeter-wave radar and V2X applications, used to achieve vehicle-road cooperation to improve road traffic efficiency and safety. Background Technology

[0002] V2X (Vehicle to Everything) is a communication technology that connects vehicles to everything, including vehicles, infrastructure, networks, and people. It facilitates information exchange and data transmission through vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and vehicle-to-person (V2P) methods. This technology aims to improve road safety, reduce traffic congestion, lower energy consumption, and enhance the driving experience.

[0003] Existing radar systems can only detect obstacles wirelessly over long distances, but cannot transmit the detection results wirelessly over long distances. They must transmit the results via Ethernet cables or RS485 buses, and further transmit the signal through an intermediate link called MEC (Multi-Access Edge Computing) before sending it via Ethernet to the RSU (Road Side Unit). Only then can the radar detection results be fused and broadcast encrypted via V2X signals (V2X (Vehicle to Everything) refers to the information exchange technology between a vehicle and various devices in its surrounding environment in the field of autonomous driving). Such systems not only waste resources and increase costs, but also increase product size, hindering widespread application. Summary of the Invention

[0004] In view of this, the present invention provides a method and apparatus based on the integration of 4D millimeter-wave radar and V2X applications. The present invention uses V2X communication technology to make up for the short transmission distance of millimeter-wave radar sensing results, and uses millimeter-wave radar all-weather high-precision active sensing technology to make up for the lack of high-precision active sensing capability of V2X equipment. Through the heterogeneous integration of the two, the complexity of equipment in the process of realizing vehicle-road cooperation is reduced, the reliability of vehicle-road cooperation applications is enhanced, and the efficiency and safety of road traffic are improved.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, embodiments of the present invention provide a method for fusing 4D millimeter-wave radar with V2X applications, comprising the following steps:

[0007] S1. Use 4D millimeter-wave radar to actively detect targets in the surrounding environment, obtain distance, angle, speed and height information of the targets, and generate point cloud data;

[0008] S2. Transmit the point cloud data to the V2X device;

[0009] S3. The computing unit of the V2X device uses a pre-trained model to analyze the point cloud data and identify the type and location information of the target object;

[0010] S4. The V2X communication unit of the V2X device broadcasts the type and location information of the identified target object to surrounding V2X communication-enabled devices through V2X communication technology.

[0011] Further, step S1 includes the following steps:

[0012] S11. Use 4D millimeter-wave radar to transmit linear frequency modulated continuous wave (FMCW) signals and transmit the signals into the environment through an antenna array;

[0013] The S12 and 4D millimeter-wave radars receive signals reflected back from targets and mix them with transmitted linear frequency modulated continuous wave (FMCW) signals to generate intermediate frequency signals.

[0014] S13. Perform frequency analysis on the intermediate frequency signal to extract target information, including: distance, speed, angle and height information;

[0015] S14. Perform three-dimensional coordinate transformation on the extracted target information to generate point cloud data.

[0016] Furthermore, in step S13, the distance information is calculated by measuring the time delay or frequency offset of the intermediate frequency signal.

[0017] Furthermore, in step S13, the angle information is calculated by measuring the phase difference or beamform between adjacent receiving channels.

[0018] Furthermore, in step S13, the velocity information is calculated by measuring the Doppler frequency shift of the intermediate frequency signal.

[0019] Furthermore, in step S13, the height information is measured:

[0020] The height of the target in three-dimensional space is calculated based on the angle and distance.

[0021] Further, step S3 includes:

[0022] A pre-trained model is used to extract features from point cloud data, extracting features related to the target object type, including: the target object's size, shape, texture, and motion state;

[0023] The extracted features are matched with the target object point cloud model in the pre-trained model, and the type of target object is identified based on the matching results;

[0024] After identifying the type of target object, the target object's location information is extracted again, including its distance, angle, speed, and height.

[0025] Based on the type and location information of the identified target, a fusion calculation is performed to predict potential safety hazards.

[0026] In a second aspect, embodiments of the present invention also provide an apparatus based on the fusion of 4D millimeter-wave radar and V2X applications, using the method for the fusion of 4D millimeter-wave radar and V2X applications as described in any of the first aspects, the apparatus comprising: a 4D millimeter-wave radar and a V2X device interconnected.

[0027] The V2X device is integrated onto the data processing board of the 4D millimeter-wave radar via a B2B dock; the V2X device has an onboard bus interface for communication and interaction with other device units.

[0028] The 4D millimeter-wave radar is used to actively detect targets in the surrounding environment and generate point cloud data;

[0029] The V2X device is used to analyze point cloud data and identify target objects, and broadcast the identified target object information to surrounding devices through V2X communication technology.

[0030] Furthermore, the V2X device has an OTA upgrade interface.

[0031] As can be seen from the above technical solution, compared with the prior art, the present invention has the following technical advantages:

[0032] This invention heterogeneously integrates millimeter-wave radar sensing equipment with V2X equipment. Leveraging the all-weather, high-precision sensing capability of millimeter-wave radar, the sensed data is directly transmitted to the V2X equipment via a communication bus. Utilizing the high-performance NPU (Neural Network Processing Unit) of the V2X equipment, a pre-trained model based on the radar data is loaded for target matching, enabling rapid acquisition of information on road users. Furthermore, the communication frequency bands in V2X technology allow for long-distance transmission of the sensed data, enabling single-point deployment and multi-point sensing, thereby enhancing the active safety capabilities of vehicles. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 The schematic diagram of the method based on the fusion of 4D millimeter-wave radar and V2X application provided by the present invention.

[0035] Figure 2 A detailed process diagram of radar detection and V2X equipment analysis and calculation provided for this invention.

[0036] Figure 3 This is a schematic diagram of a linear frequency modulated pulse signal with amplitude as a function of time, provided by the present invention.

[0037] Figure 4 This is a schematic diagram of a linear frequency modulated pulse signal with frequency as a function of time, provided by the present invention.

[0038] Figure 5 This is a schematic diagram of Chirp signal ranging provided by the present invention.

[0039] Figure 6 This is a schematic diagram of the intermediate frequency signal provided by the present invention.

[0040] Figure 7 A schematic diagram of the Fast Fourier Transform provided by this invention.

[0041] Figure 8 This is a schematic diagram of a uniform linear antenna array provided by the present invention.

[0042] Figure 9 This is a schematic diagram of a TDMA-MIMO signal provided by the present invention.

[0043] Figure 10 This is a schematic diagram of the radar matrix provided by the present invention.

[0044] Figure 11 This is a schematic diagram of the overall radar algorithm provided by the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] This invention discloses a method for fusing 4D millimeter-wave radar and V2X applications, involving 4D millimeter-wave radar and V2X devices. The 4D millimeter-wave radar acts as the front end to actively detect surrounding targets, while the V2X device acts as the back end, using its computing unit to analyze the perceived targets. Through a pre-trained model, it identifies traffic participants with different spatial positions, speeds, and directions of movement, as well as obstacles hindering safe traffic operation. Then, through the V2X communication unit, it broadcasts the identified environmental targets to surrounding V2X-enabled devices, achieving a high-precision, high-reliability vehicle-road cooperative safety scenario.

[0047] Its overall processing flow is as follows Figure 1 As shown, it includes the following steps:

[0048] S1. Use 4D millimeter-wave radar to actively detect targets in the surrounding environment, obtain distance, angle, speed and height information of the targets, and generate point cloud data;

[0049] S2. Transmit the point cloud data to the V2X device;

[0050] S3. The computing unit of the V2X device uses a pre-trained model to analyze the point cloud data and identify the type and location information of the target object;

[0051] S4. The V2X communication unit of the V2X device broadcasts the type and location information of the identified target object to surrounding V2X communication-enabled devices through V2X communication technology.

[0052] This method, based on the fusion of 4D millimeter-wave radar and V2X applications, leverages the all-weather, high-precision perception capabilities of millimeter-wave radar and utilizes V2X communication technology to achieve long-distance transmission and sharing of perception data, thereby realizing high-precision and high-reliability vehicle-road cooperative safety scenarios. Furthermore, it identifies traffic participant information through a pre-trained model, enabling driving safety warning and control functions. This method contributes to promoting the development and application of vehicle-road cooperative technology.

[0053] The technical solution of this invention will be described in detail below, and can be broadly divided into two parts: millimeter-wave radar front-end radio frequency detection and V2X back-end model analysis and calculation. These two parts will be described in detail, and the detailed data processing flow is as follows: Figure 2 As shown, it mainly includes: active detection of the surrounding environment by the front-end radar; analysis of the radar-detected targets by the V2X equipment computing unit, realizing target classification and security analysis for key targets; and data sharing of perceived targets by the V2X communication unit, combined with independent V2X equipment to realize driving safety warnings and related control functions. The overall process consists of... Figure 2 The process will be explained in detail below, followed by a detailed explanation of each part.

[0054] I. Detailed Explanation of Radar Front-End:

[0055] 1. Transmit radar waves

[0056] Signal generation: Millimeter-wave radar generates radar waveforms through frequency modulated continuous wave (FMCW). This produces a linear frequency modulated (LFM) signal that scans frequencies within a specific frequency band.

[0057] Signal transmission: The generated frequency-modulated or pulse signal is transmitted into the environment through an antenna array, usually at a certain beam angle.

[0058] The method for generating the FMCW signal waveform is shown in the following formula:

[0059]

[0060] Where A represents the signal amplitude, controlling the signal strength. In practical systems, it is related to the transmit power and is usually a fixed value.

[0061] f0: The starting frequency of the signal (also called the carrier frequency), which is the frequency of the signal at t=0.

[0062] β: Linear frequency modulation rate (frequency change rate), measured in Hz / s or GHz / s, represents the rate at which the signal frequency changes over time. It determines the sweep range (bandwidth) and sweep period of the FMCW signal. The frequency change within one period is: linear frequency modulation rate × period duration.

[0063] t: Time variable, representing the signal propagation time. In the formula, the frequency of the signal changes linearly with time t.

[0064] This describes the quadratic phase change caused by a linear change in frequency. It is the cumulative phase change, and its first derivative is directly related to the frequency.

[0065] Figure 3 This is a graph with an amplitude of 1, an initial frequency of 1 Hz, an ending frequency of 10 Hz, and a sampling rate of 1000 Hz for 1 second (too high a frequency cannot be used for the graph as it would be difficult to see; this is just a brief illustration). The horizontal axis is Time, representing time in seconds; the vertical axis is Amplitude, representing amplitude; and Linear Chirp Signal represents a linear frequency modulated pulse signal.

[0066] Figure 4This is a graph of a linear frequency modulated pulse signal (frequency as a function of time) in the range of 77GHz-81GHz; where B is the bandwidth of 4GHz; the horizontal axis is Time, representing time in μs; and the vertical axis is Frequency (GHz), representing frequency.

[0067] The formula for calculating the phase is:

[0068]

[0069] f start It is the initial frequency, f end This is the end frequency, and k is the frequency modulation rate. (Signal duration T), where t is time.

[0070] 2. Receive reflected signals

[0071] Reflected wave reception: When radar waves encounter target objects (such as vehicles, pedestrians, obstacles, etc.) during propagation, they are reflected. The radar receiver receives these reflected signals through an antenna array.

[0072] Signal mixing: The received echo signal is mixed with the transmitted signal to generate a mixed signal, which is then used to analyze information such as the target's distance and speed.

[0073] 3. Signal demodulation and frequency processing

[0074] Frequency offset calculation: By comparing the frequency difference between the transmitted and received signals, the radar system can calculate the relative speed and distance of the target.

[0075] For FMCW radar, the frequency difference between the received and transmitted signals (also called frequency offset) can be used to calculate the target's distance. By measuring this frequency offset, the radar system can deduce the target's range information.

[0076] Doppler effect calculation: Due to relative motion, the frequency of the reflected wave will shift (Doppler effect). By calculating the frequency change, the relative velocity of the target can be inferred.

[0077] 4. Extraction of distance and velocity

[0078] ① Distance calculation: For FMCW radar, the distance to the target can be calculated by measuring the time delay (or frequency offset) of the signal.

[0079] The 4D millimeter-wave radar transmits a linear frequency modulated sawtooth wave signal, i.e., a chirp signal, such as... Figure 5 As shown, the distance measurement principle is as follows: where T C T is the signal period time. RF T is the frequency sweep time.I This is the idle time. The Chrip signal frequency is 76GHz~81GHz. This embodiment uses the IWR1843 radar chip, whose ADC sampling frequency is 5MHz, while the received signal frequency is 76~81GHz. Direct processing of the received signal is not possible; a mixer is needed to mix the received and transmitted signals. After mixing, the down-converted signal is retained as the intermediate frequency signal, such as... Figure 6 As shown.

[0080] The expression for the intermediate frequency signal is:

[0081]

[0082] ω1 and ω2 represent the angular frequencies of the transmitted and received signals. For linear frequency modulated (LFM) continuous wave radar, the difference between the frequency of the received signal and the frequency of the transmitted signal is usually related to the target's range or velocity. For LFM signals, the frequency difference is proportional to the target's range or velocity; the transmitted signal frequency changes linearly with time, while the received signal frequency shifts due to the target's relative velocity (Doppler effect) or range (frequency shift caused by delay).

[0083] t: Time variable, representing the moment of the signal.

[0084] and This indicates the initial phase of the transmitted and received signals. The phase difference between the two reflects the influence of target distance or other environmental characteristics. The phase difference contains distance information related to the target, playing a crucial role, especially in pulse compression or precise ranging.

[0085] Intermediate frequency signal x out It is a sine wave that varies with time, and its frequency difference and phase difference contain information about the target's motion and position.

[0086] From the intermediate frequency signal diagram, we can obtain:

[0087]

[0088] S is the frequency sweep slope, τ is the reception delay, and d is the target distance. The formula for measuring the target distance, derived from the previous formula, is:

[0089]

[0090] c: indicates the transmission speed of millimeter-wave radar, i.e., the speed of light;

[0091] f IF : represents the intermediate frequency signal obtained by formula (4);

[0092] S: represents the sweep slope, i.e. the relationship between frequency change and time.

[0093] From formula (5), it can be seen that the intermediate frequency signal frequency f IF Proportional to the target distance. The intermediate frequency signal is sampled to obtain a time-domain sequence, and after adding a Hanning window, a Range-FFT (Range-Fast Fourier Transform) is performed on the distance dimension, as follows: Figure 7 As shown:

[0094] If we take the number of sampling points N as the number of points in the FFT, then we can obtain:

[0095]

[0096] f s The sampling rate for the intermediate frequency signal;

[0097] k n ∈[0,N-1] represents the sampling point number of the N-point FFT;

[0098] S: represents the sweep slope, i.e. the relationship between frequency change and time.

[0099] And the sampling period T s The relationship between bandwidth B and the expression is as follows:

[0100]

[0101] B=ST s (8)

[0102] Substituting formula (7) into formulas (6) and (8), we get:

[0103]

[0104] From formula (9), it can be seen that when B is constant, d and k n Since the distance is proportional to the gate, the distance resolution can be obtained:

[0105]

[0106] The radar's maximum measurement range d max Limited by the intermediate frequency signal frequency:

[0107]

[0108] Among them, f IFmax For the maximum intermediate frequency signal, in Complex 2x and real sampling modes, f IFmax ≤0.9f s / 2,f s This represents the sampling rate of the intermediate frequency signal.

[0109] ② Angle calculation: In radar systems with multiple antenna arrays, the angles (azimuth and elevation) of a target can be estimated using phase difference or beamforming techniques, which helps to pinpoint the target's precise location.

[0110] The principle of angle measurement, such as Figure 8 As shown, Range-FFT is performed on the sampled intermediate frequency signal to obtain the target's range information. However, the range value obtained from the radar still does not reveal the target's specific azimuth relative to the radar, resulting in range ambiguity. Further signal processing is needed to obtain the radar's azimuth information. This embodiment uses the phase method for angle measurement. For uniformly distributed antenna array elements, such as... Figure 9 As shown, the path difference ΔR between adjacent echo signals caused by the spacing between the receiving antennas can be expressed as:

[0111] ΔR=d a sinθ (12)

[0112] Where, d a θ is the array spacing, and θ is the incident angle.

[0113] Even a small path difference can cause a phase change in the signals received by different arrays. The phase difference ΔΦ between adjacent receiving channels is:

[0114]

[0115] Where f0 is the initial frequency of the chrip signal, c is the signal velocity (taken as the speed of light), and d... a θ is the array spacing, and θ is the incident angle.

[0116] The results were:

[0117]

[0118] Where, d a θ is the array spacing, θ is the incident angle, f0 is the initial frequency of the chrip signal, and ΔΦ is the phase difference derived from Equation 13.

[0119] Signal transmission employs TMDA-MIMO mode, with three transmitting antennas activated at different time intervals. This reduces mutual interference caused by multiple antennas operating simultaneously, thereby improving the system's signal-to-noise ratio and detectability. Simultaneously, N... d =N t *N r The effect of virtual antennas avoids the high cost and design complexity associated with using a large number of physical antennas, enabling radar systems to achieve high-performance target detection at a lower cost.

[0120] Performing an angular-dimensional FFT (Angle-FFT) on the received data sequence from different virtual channels (green part) yields the radar matrix diagram, such as the radar matrix. Figure 10 As shown, the two-dimensional matrix is ​​a range-angle matrix. After performing Angle-FFT, the phase difference between adjacent receiving channels can be obtained, thus yielding the target's azimuth information. The corresponding actual expression for ΔΦ is:

[0121]

[0122] ΔΦ: The phase difference between signals received by different virtual channels in the antenna array.

[0123] k a : FFT index, related to the antenna channel.

[0124] N d : The total number of channels in the antenna array.

[0125] Substituting formula (15) into formula (14) yields the angular resolution:

[0126]

[0127] θ res Radar angular resolution refers to the minimum interval at which the angles of a target can be distinguished under the physical configuration of the array.

[0128] c: speed of light.

[0129] f0: The initial frequency of the frequency-modulated signal.

[0130] N d : The number of channels in the antenna array.

[0131] d d The spacing between adjacent antennas in an antenna array.

[0132] Angle ambiguity occurs when the phase difference exceeds ΔΦ < π. Substituting the maximum unambiguous phase difference into formula (14) yields the maximum unambiguous angle θ. M expression:

[0133]

[0134] θ M : Maximum unambiguous angle range; when the phase difference ΔΦ exceeds 2π, the angle measurement will become blurred. This formula defines the upper limit range of the angle measurement.

[0135] ③ Velocity calculation: The velocity of the target relative to the radar is calculated using the Doppler frequency shift principle.

[0136] Principle of measuring the radial velocity of a target:

[0137] 3.1 Doppler effect

[0138] When a target object moves relative to the radar, its velocity v causes a frequency shift in the echo signal, known as the Doppler shift f. d .

[0139] The formula for Doppler frequency shift is:

[0140]

[0141] v: Radial velocity of the target (velocity component along the direction of radar wave propagation, unit: m / s).

[0142] λ: Wavelength of the millimeter-wave signal (unit: m), relative to the carrier frequency f c The relationship is

[0143] c: speed of light, approximately 3 * 10 8 m / s.

[0144] 3.2 Linear Frequency Modulated Continuous Wave (Chirp Signal)

[0145] FMCW radar acquires both range and velocity information of a target by transmitting a linear frequency modulated (LFM) signal. The frequency of the chirp signal increases or decreases linearly with time, and its frequency expression is as follows:

[0146] f(t) = f c +k*t (19)

[0147] Among them, f c : Carrier frequency (center frequency, usually 77GHz or 79GHz);

[0148] Frequency modulation slope (unit: Hz / s) represents the rate at which the signal frequency changes.

[0149] B: Bandwidth, typically 4GHz;

[0150] T c The modulation period is typically tens of microseconds.

[0151] 3.3 Target echo signal

[0152] When the signal emitted by the radar is reflected back by the target, the echo signal received by the radar will simultaneously include the delay time t. d and Doppler frequency shift f d The reflected signal can be represented as:

[0153]

[0154] in, The time delay of the signal to and from the target is related to the target distance R;

[0155] c is the speed of light, and the specific value is given above.

[0156] f d Doppler shift caused by the target velocity.

[0157] 3.4 Intermediate Frequency Signal Analysis (Mixer Processing)

[0158] FMCW radar mixes the received signal with the transmitted signal (difference frequency) to obtain an intermediate frequency (beat signal) with a frequency of f. b The intermediate frequency signal consists of two parts:

[0159] f b =f range +f d (twenty one)

[0160] Distance frequency f range Caused by signal delay time, indicating target distance: in

[0161] Doppler frequency f d : Caused by the relative motion of the target, indicating the target velocity.

[0162] 3.5 Distance and velocity separation

[0163] In order to determine the target's range frequency f range With Doppler frequency f d Separately, FMCW radar typically employs positive and negative slope frequency modulation (bidirectional chirp):

[0164] Up-chirp: Frequency from low to high;

[0165] Down-chirp: Frequency from high to low;

[0166] The intermediate frequencies of the echo signals from the two frequency sweeps are as follows:

[0167]

[0168] By calculating the sum and difference of the uplink and downlink frequency sweeps, the target's range and velocity can be obtained respectively:

[0169] Target distance R:

[0170]

[0171] Target velocity v:

[0172]

[0173] The target speed information is obtained through the above processing.

[0174] ④ Height measurement

[0175] Altitude measurement principle: After obtaining the target's pitch angle, this angle can be combined with the target's range to calculate the target's altitude in three-dimensional space.

[0176] Assuming the radar is located at the origin, and the target object's coordinates are (x, y, z), where z is the altitude, the target's altitude can be calculated using the following formula:

[0177] z = R*sin(θ) elevation (26)

[0178] Where R is the distance between the target and the radar;

[0179] θ elevation It is the elevation angle of the target relative to the radar.

[0180] In summary, based on the principles outlined in ①, ②, ③, and ④, the concise overall algorithm flow for the radar front-end is as follows: Figure 11 As shown.

[0181] 5. Target Detection and Recognition

[0182] Target detection: After obtaining information such as distance, speed, and angle, the radar system processes the data to detect multiple targets in the environment (such as other vehicles, pedestrians, obstacles, etc.). Target detection algorithms include cluster analysis, threshold setting, and signal filtering.

[0183] Data denoising and filtering: Signals may contain noise or interference, especially in complex environments (such as multi-target environments). Therefore, the system applies filtering techniques (such as Kalman filtering, wavelet transform, etc.) to suppress noise and retain valid target information.

[0184] 6. Data Fusion and Point Cloud Generation

[0185] Data fusion: In order to improve detection accuracy and robustness, information from multiple radar sensors (multi-point radar) or other sensors (such as lidar, cameras, etc.) is usually fused.

[0186] Point cloud generation: Information such as the target's distance, velocity, and angle is combined to form coordinate points in space, which constitute the so-called "point cloud". Each point in the point cloud represents the position of a target, usually having (X, Y, Z) coordinates and other attributes (such as velocity, reflection intensity, etc.).

[0187] II. V2X Device Backend Model Analysis and Calculation Instructions

[0188] The V2X backend model analysis and computation module mainly consists of V2X computing units and fused data components:

[0189] ①V2X computing unit

[0190] This part acquires point cloud data detected by the radar front end via a bus, which includes the distance, angle, speed, and height information of the target. However, due to the heterogeneous design, the radar data processing part uses a low-power, high-performance processor with an NPU for processing. Therefore, this device does not have the powerful computing capabilities of a graphics processor like NVIDIA's. Thus, this embodiment uses a cluster-based graphics computing unit to learn from massive millimeter-wave radar data and generate models of radar echo point cloud data of most of the target objects that need to be used. These models include: adults (including different body positions such as lying down, standing, and squatting), children (including different body positions such as lying down, standing, and squatting), cars, buses, trucks, small trains, bicycles, motorcycles, traffic cones, water-filled barriers, municipal engineering vehicles, etc. This method can not only greatly improve computing efficiency, but also update the target object models as needed.

[0191] The model recognition process is as follows:

[0192] Feature extraction: The pre-trained model first performs feature extraction on the point cloud data, extracting features related to the type of the target object, such as the size, shape, texture, and motion state of the target object.

[0193] Model matching: The extracted features are matched with the target object model in the pre-trained model, and the type of the target object is identified based on the matching results.

[0194] Location information extraction: After identifying the type of target object, the model further extracts the location information of the target object, including the distance, angle, speed and height of the target object.

[0195] The computing unit integrates the target object's position, angle, and velocity information from the received point cloud data with the equipment's carrier information. Its main purpose is to identify any predictable safety hazards based on various information about the carrier on which the equipment is located. If such hazards exist, the effective data is fused together, and through the communication between the equipment and the carrier, the personnel operating the carrier can be informed more accurately and promptly, thereby improving road traffic safety and increasing road traffic efficiency.

[0196] The following describes the security risk principle of the computing device's carrier based on V2X device data and point cloud data detected by a 4D millimeter-wave radar front end, using a simple scenario:

[0197] A. Description of forward collision warning

[0198] By calculating parameters such as the relative position, speed, and deceleration between the vehicle and the vehicle in front in real time, potential collision risks are predicted. The key steps are as follows:

[0199] a. Obtain carrier-related data for the device body, including velocity v. s acceleration a s Location information x s ;

[0200] b. Sequentially extract the velocity v of a target object from the point cloud data provided by the radar front end. f acceleration a f Location information x f ;

[0201] c. Calculate the relative distance:

[0202] d rel =x f -x s (27)

[0203] d. Calculate the collision time TTC:

[0204]

[0205] When TTC is small (e.g., TTC < 5s), the warning system triggers a warning; when TTC = ∞, it means that a collision will not occur (i.e., the speed of the device itself is less than or equal to the speed of the target object in front).

[0206] e. Collision deceleration calculation: In an emergency, to avoid a collision, the braking speed required for the device itself is:

[0207]

[0208] If the current braking acceleration speed a of the carrier on which the device is located is... s Greater than a req (The braking acceleration is negative, so it is greater than), triggering warnings, etc.

[0209] f. If steps d and e do not trigger the warning, proceed to step b to continue running.

[0210] B. Description of blind spot / lane change warning

[0211] The risk of collision is determined by the distance and relative speed between the vehicle and vehicles in the blind spot or adjacent lanes.

[0212] a. Obtain carrier-related data for the device body, including velocity v.s Device carrier location information x s y s The width and length W of the device carrier s L s ;

[0213] b. Sequentially extract relevant data of a target object from the point cloud data provided by the radar front end, including velocity v. t Device carrier location information x t y t The width and length W of the device carrier t L t ;

[0214] c. Blind Spot Calculation: A blind spot is a fixed area within the vehicle itself, typically defined as:

[0215] Horizontal position: Starting from the rear of the vehicle, to a certain distance behind the vehicle (usually 3 to 5 meters);

[0216] Vertical position: Extends a certain width (usually 1 to 2 meters) on both sides of the vehicle;

[0217] The criteria for determining whether a target vehicle is in the blind spot are:

[0218] x s -L s ≤x t ≤x s # (30)

[0219]

[0220] Wherein: ΔW represents the width expansion of the blind zone, which is generally taken as 0.5 to 1 meter.

[0221] d. Lane change collision prediction: If the vehicle plans to change lanes, the collision risk with vehicles in adjacent lanes must be considered. Judgment criteria include:

[0222] Relative distance:

[0223] d rel =|x t -x s | (32)

[0224] Relative speed:

[0225] v rel =|v s -x t | (33)

[0226] Collision event calculation (TTC):

[0227]

[0228] If the target vehicle is within the planned lane change area of ​​the device carrier (usually defined as a range extending 5 meters in front and behind), and the TTC is small (e.g., TTC < 5s), the warning system triggers a lane change warning;

[0229] e. If steps c and d do not trigger the warning, proceed to step b to continue running.

[0230] In actual implementation, this device can have dozens of built-in safety warning scenarios. Due to space limitations, they will not be described one by one. Only the above two scenarios will be used to briefly describe the calculation logic.

[0231] ② Data fusion

[0232] In the data fusion section, data on targets that may hinder safety, calculated by the computing unit, can be extracted. Through V2X wireless communication, the corresponding targets can be shared with other V2X devices in the vicinity via the communication frequency band, thereby improving traffic efficiency and enhancing traffic safety.

[0233] In this invention, the core implementation includes:

[0234] 1. Enhance the active perception capability of V2X devices (equipment) of surrounding targets. Traditional V2X devices or equipment do not have active perception capabilities.

[0235] 2. Utilize cluster computing to extract key target object models, replacing traditional traffic detection and recognition equipment with lower-cost hardware components, thereby enhancing the low-carbon and energy-saving effect;

[0236] 3. Significantly reduces the transmission time consumption of traditional V2X network-connected sensing targets;

[0237] 4. Data fusion and sharing: Target data acquired from the millimeter-wave radar sensor at the front end of the device is directly transmitted in encrypted form from the V2X device at the back end of the device, enhancing information sharing capabilities.

[0238] 5. Trajectory Prediction: Predicts the future position of the target vehicle and calculates its relative position and speed.

[0239] 6. Real-time calculation: Calculates collision time (TTC) and relative distance (d) in real time using vehicle kinematic equations and dynamic update algorithms. rel Relative velocity v rel Braking speed a rel Parameters such as these.

[0240] 7. Early warning decision: Based on the calculation results and thresholds, trigger early warning information of different levels.

[0241] The method based on the fusion of 4D millimeter-wave radar and V2X application provided by this invention can not only achieve all-weather high-precision perception by combining 4D millimeter-wave radar with V2X technology, but also make full use of V2X communication technology to broadcast the perception capability of millimeter-wave radar over a wide range, enabling exploration from a higher starting point.

[0242] Based on the same inventive concept, this invention also provides an apparatus based on the fusion of 4D millimeter-wave radar and V2X applications. Using the method of the above embodiment based on the fusion of 4D millimeter-wave radar and V2X applications, the apparatus includes: a 4D millimeter-wave radar and a V2X device connected to each other.

[0243] The V2X module is integrated under the millimeter-wave radar data processing board via a B2B hub and communicates with other equipment units through the vehicle bus. This effectively avoids interference and saves space, achieving perfect structural integration.

[0244] The radar antenna and V2X antenna are positioned on two different structural surfaces, which effectively avoids the possibility of crosstalk.

[0245] Data flow: The radar transmits structured data such as position, speed, azimuth, and altitude to the V2X for calculation and analysis, and then broadcasts the fused data to other vehicles or RSUs.

[0246] 4D millimeter-wave radar is used to actively detect targets in the surrounding environment and generate point cloud data;

[0247] V2X devices are used to analyze point cloud data and identify targets, broadcasting the identified target information to surrounding devices via V2X communication technology. V2X devices have an OTA (Over-The-Air) upgrade interface, allowing for updates to the corresponding recognition models and programs to handle dynamic changes in vehicles and target objects according to different needs.

[0248] The present invention aims to solve the problem that V2X devices cannot actively detect traffic participants in the surrounding environment during vehicle operation. By combining 4D millimeter-wave radar with V2X devices, the active detection capability of V2X devices is improved, the detection cost and data processing time of traditional V2X communication are reduced, and the safety of vehicles and traffic participants in vehicle-road cooperation is achieved with higher accuracy and higher reliability.

[0249] This device has the following technical advantages:

[0250] Improving perception accuracy: 4D millimeter-wave radar can provide high-precision target position, speed and direction information, while V2X communication can acquire information about the surrounding environment in real time. By fusing the two, the environment around the vehicle can be perceived more accurately, thus improving perception accuracy.

[0251] Enhanced perception capabilities: Both 4D millimeter-wave radar and V2X communication have a large perception range. By fusing the two, the perception range can be expanded, target objects can be detected and tracked better, and perception capabilities can be enhanced.

[0252] Improved accuracy of decision-making and control: Based on more accurate environmental perception information, 4D millimeter-wave radar and V2X fusion technology can formulate more accurate driving decisions and control strategies, improving the safety and stability of vehicle driving.

[0253] Enhancing traffic efficiency: V2X communication enables real-time communication between vehicles, between vehicles and infrastructure, and between vehicles and pedestrians. Through information sharing and collaborative decision-making, traffic flow can be optimized and traffic efficiency improved.

[0254] Reduce accident risk: Through more accurate perception and smarter decision-making and control, 4D millimeter-wave radar combined with V2X technology can reduce the risk of vehicle accidents and improve road traffic safety.

[0255] Reduce overall cost: Integrating 4D millimeter-wave radar with V2X functionality can effectively reduce overall cost, decrease size, and expand its application areas.

[0256] In summary, the fusion of 4D millimeter-wave radar and V2X can improve perception accuracy, enhance perception capabilities, improve the accuracy of decision-making and control, enhance traffic efficiency, and reduce accident risks.

[0257] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0258] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for fusing 4D millimeter-wave radar with V2X applications, characterized in that, The method is based on a fusion device comprising interconnected 4D millimeter-wave radar and V2X equipment. The V2X equipment is integrated onto the data processing board of the 4D millimeter-wave radar via a B2B hub. The 4D millimeter-wave radar acts as the front end to actively detect targets in the surrounding environment, while the V2X equipment acts as the back end, using its computing unit to analyze the sensed targets. Target data acquired from the millimeter-wave radar sensor at the front end is directly encrypted and transmitted from the V2X device at the back end, enhancing information sharing capabilities. The method includes the following steps: S1. Use 4D millimeter-wave radar to actively detect targets in the surrounding environment, obtain distance, angle, speed and height information of the targets, and generate point cloud data; S2. Transmit the point cloud data to the V2X device; S3. The computing unit of the V2X device uses a pre-trained model to analyze the point cloud data and identify the type and location information of the target object; S4. The V2X communication unit of the V2X device broadcasts the type and location information of the identified target object to surrounding V2X communication-enabled devices through V2X communication technology.

2. The method for fusing 4D millimeter-wave radar and V2X applications according to claim 1, characterized in that, Step S1 includes the following steps: S11. Use 4D millimeter-wave radar to transmit linear frequency modulated continuous wave (FMCW) signals and transmit the signals into the environment through an antenna array; The S12 and 4D millimeter-wave radars receive signals reflected back from targets and mix them with transmitted linear frequency modulated continuous wave (FMCW) signals to generate intermediate frequency signals. S13. Perform frequency analysis on the intermediate frequency signal to extract target information, including: distance, speed, angle and height information; S14. Perform three-dimensional coordinate transformation on the extracted target information to generate point cloud data.

3. The method for fusing 4D millimeter-wave radar and V2X applications according to claim 2, characterized in that, In step S13, the distance information is calculated by measuring the time delay or frequency offset of the intermediate frequency signal.

4. The method for fusing 4D millimeter-wave radar and V2X applications according to claim 2, characterized in that, In step S13, the angle information is obtained by measuring the phase difference or beamform between adjacent receiving channels.

5. The method for fusing 4D millimeter-wave radar and V2X applications according to claim 2, characterized in that, In step S13, the velocity information is calculated by measuring the Doppler frequency shift of the intermediate frequency signal.

6. The method for fusing 4D millimeter-wave radar and V2X applications according to claim 2, characterized in that, In step S13, the height information is measured: The height of the target in three-dimensional space is calculated based on the angle and distance.

7. The method for fusing 4D millimeter-wave radar and V2X applications according to claim 1, characterized in that, Step S3 includes: A pre-trained model is used to extract features from point cloud data, extracting features related to the target object type, including: the target object's size, shape, texture, and motion state; The extracted features are matched with the target object point cloud model in the pre-trained model, and the type of target object is identified based on the matching results; After identifying the type of target object, the target object's location information is extracted again, including its distance, angle, speed, and height. Based on the type and location information of the identified target, a fusion calculation is performed to predict potential safety hazards.

Images