Millimeter wave radar speed measurement calibration method based on visual flow measurement

Abstract

The invention is suitable for the technical field of hydrographic survey, and provides a millimeter wave radar speed measurement calibration method based on visual flow measurement, which comprises the following steps: acquiring data, constructing a database and preprocessing an image, performing target detection on the preprocessed image based on YOLO v8, performing region segmentation on a water surface region after target extraction is completed, and obtaining a millimeter wave radar speed measurement calibration result. Visual flow measurement calculation is carried out based on an optical flow method, when optical flow calculation is carried out, calculation should be carried out in each area, space-time synchronization is carried out on a visual flow measurement result and a flow velocity measured by a millimeter wave radar, credible areas are screened, abnormal data are processed, and linear coefficients are fitted for each credible area k through a least square method; according to the method, under the condition that millimeter wave radar speed measurement single-point measurement is limited, area speed measurement information is provided through visual flow measurement, and the problem that the millimeter wave measurement range is insufficient is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hydrological measurement, more particularly, it relates to a millimeter wave radar speed calibration method based on visual flow measurement. BACKGROUND

[0002] Modern hydrological flow measurement research often focuses on non-contact flow measurement methods, including measurement by acoustic Doppler effect, optics and radar. Millimeter wave radar has always been an important method for river speed measurement due to its all-weather operation and small environmental impact. In recent years, visual flow measurement has become a popular means of hydrological monitoring due to its high resolution and area measurement capability.

[0003] However, millimeter wave radar speed measurement and visual flow measurement each have their own defects. The problems of millimeter wave speed measurement include: 1. Large speed measurement error. Millimeter wave speed measurement relies on Doppler shift to calculate flow speed, which may be affected by wind speed, waves, flow field turbulence, etc., resulting in errors. 2. Limitations of single-point millimeter wave speed measurement. Millimeter wave speed measurement usually measures the average flow speed at a certain point or a small range, and cannot effectively obtain the flow speed distribution of the entire water surface. Visual flow speed is easily affected by light and weather, and its flow measurement stability is poor.

[0004] Therefore, the present application combines the two flow measurement methods, and uses the fine resolution capability of visual speed measurement to calibrate the millimeter wave speed measurement error, in order to improve the speed measurement accuracy. SUMMARY

[0005] In view of the deficiencies of the prior art, the purpose of the present application is to provide a millimeter wave radar speed calibration method based on visual flow measurement, which corrects the millimeter wave radar speed measurement error through visual speed measurement to improve the millimeter wave radar speed measurement.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] A millimeter wave radar speed calibration method based on visual flow measurement, comprising the following steps:

[0008] S1, collecting data and constructing a database, including obtaining river and channel video flow and radar speed measurement data and performing image preprocessing;

[0009] S2, target detection on the preprocessed image based on YOLO v8, and after target extraction, region segmentation is performed on the water surface area;

[0010] S3, visual flow calculation based on the optical flow method. When performing optical flow calculation, it should be calculated in each region respectively;

[0011] In each sub-region divided, an improved Shi-Tomasi algorithm is used for corner detection;

[0012] The Pyramidal Lucas-Kanade algorithm is used to calculate the motion vector of each corner point in consecutive frames, and Δx and Δy are solved by minimizing the sum of squared errors in a local window.

[0013] Construct a three-level image pyramid and iteratively calculate the optical flow vector layer by layer;

[0014] Based on the pre-calibrated camera parameters (mounting height H, focal length f, pitch angle θ), the pixel velocity is mapped to the actual flow velocity;

[0015] S4. Synchronize the visual flow measurement results with the flow velocity measured by the millimeter-wave radar in time and space, align the video and radar data timestamps through the Beidou PPS signal, and the error is ≤10ms; set up at least three reflective targets with known positions in the monitored waters, establish a mapping relationship between the radar polar coordinate system and the visual pixel coordinate system, and map the radar velocity matrix to the visual division area according to the spatial coordinates, to ensure that each area k contains both: visual velocity Radar speed and radar signal-to-noise ratio SNR (k) ;

[0016] S5. Screen the trusted area and process abnormal data;

[0017] Calculate the difference index between visual flow measurement and radar flow measurement results;

[0018] Determine the credible area, process the abnormal data, and perform neighborhood interpolation on the excluded area;

[0019] During interpolation, if the same area is eliminated for three consecutive frames, the area will be re-divided;

[0020] S6. For each credible region k, the linear coefficient is fitted by the least squares method, the radar velocity is corrected, and the calibration results are fused and output to obtain the overall river surface velocity.

[0021] A millimeter-wave radar speed measurement calibration system based on visual flow measurement includes a perception module, a visual flow measurement module, a data calibration module and a calculation output module.

[0022] The advantages of the present invention are:

[0023] First, the present invention integrates multi-source data and performs visual flow measurement based on the YOLOv8-optical flow method to ensure the accuracy of feature point tracking and calibrate the millimeter-wave radar data. Under the limitation of single-point measurement of millimeter-wave radar speed measurement, visual flow measurement provides surface speed measurement information to make up for the problem of insufficient millimeter-wave measurement range.

[0024] Secondly, the present invention uses visual recognition flow velocity detection technology to calibrate millimeter-wave radar flow velocity detection, realizing cross-modal data fusion. It not only makes up for the defect that a single sensor is easily disturbed in complex hydrological environments, but also solves the problem of poor stability of visual flow measurement under unfavorable lighting conditions. The spatiotemporal alignment of the two realizes data complementarity and greatly improves the measurement accuracy.

[0025] Third, the present invention considers dynamic segmentation of the calculation area when designing the algorithm to achieve dynamic matching of measurement and scene characteristics. For water surfaces with different hydraulic conditions, the regional segmentation method is selected according to their respective characteristics for data matching, making the calibration more accurate and avoiding the overall error diffusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of a method for calibrating millimeter-wave radar speed measurement based on visual flow measurement according to the present invention;

[0027] Figure 2 This is a block diagram of the millimeter-wave radar speed measurement calibration system based on visual flow measurement in the present invention. DETAILED DESCRIPTION

[0028] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the invention are shown in the accompanying drawings.

[0029] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] See also Figure 1 , the present invention provides the following technical solutions:

[0031] Specifically, it refers to a millimeter wave radar speed measurement calibration method based on visual flow measurement, which includes the following steps:

[0032] S1. Collect data and build a database. This includes acquiring river and channel video streams and radar speed measurement data and performing image preprocessing.

[0033] S1-1. Complete equipment deployment and set up cameras above rivers and channels to capture water flow video streams with a frame rate ≥ 30fps and a resolution > 1080p.

[0034] S1-2. Perform image preprocessing on the video streams of rivers and channels. Convert the video streams to grayscale using OpenCV. Then, perform image enhancement and noise reduction on the converted grayscale images. Perform filtering on the data acquired by the millimeter-wave radar.

[0035] S1-3. Deploy millimeter-wave radar in the monitoring area, set its speed measurement center point as the reference point, output the original point cloud data, and analyze and generate the speed matrix.

[0036] S2. Perform target detection on the preprocessed image based on YOLO v8. After target extraction, divide the water surface area.

[0037] S2-1. Build a target detection model based on the YOLO v8s model, using water surface ripples, vortices, etc. as detection objects. Prepare training images with labeled water surface ripples, vortices, etc., train the built model, use GloU to optimize the water surface area detection frame to improve the accuracy of water surface area detection, introduce Focal Loss to reduce the impact of interference such as reflections and floating objects, and add gradient perception loss to make the detected water surface area more consistent with the texture characteristics of actual water flow. The YOLOv8s model includes: Backbone, Neck, Head, and Post-processing.

[0038] S2-2. When performing regional segmentation, the center point of the millimeter-wave radar speed measurement should be used as the reference point. Two segmentation methods are often considered. When the flow velocity of the scene to be measured is uniform and the boundary is regular, the nine-square grid segmentation can be used. When the flow velocity of the scene to be measured varies greatly with distance, the annular diffusion segmentation can better match the actual flow field characteristics.

[0039] The specific steps of using the nine-square grid segmentation are: taking the center point of the radar speed measurement as the benchmark, divide it into 3×3 sub-areas, and the grid size matches the radar beam width.

[0040] The specific steps of using annular diffusion segmentation are: taking the center point of the radar speed measurement as the center of the circle, expand multiple annular areas outward and set the radius interval according to needs.

[0041] S3. Perform visual flow measurement calculation based on the optical flow method. When performing optical flow calculation, calculation should be performed separately in each area.

[0042] S3-1. In each divided sub-region, the improved Shi-Tomasi algorithm is used to detect corners, and its response function is:

[0043] R=min(λ1,λ2)

[0044] Among them, λ1 and λ2 are the eigenvalues ​​of the grayscale covariance matrix of the local window of the image, and only the grayscale covariance matrix that satisfies R>R is retained. th By dynamically adjusting the threshold, the auxiliary detection algorithm is automatically used in sparse areas to supplement the feature points, and non-maximum suppression is used to eliminate artifacts in highly reflective areas.

[0045] S3-2. Use the Pyramidal Lucas-Kanade algorithm to calculate the motion vector of each corner point in consecutive frames, and solve Δx and Δy by minimizing the sum of squared errors in a local window. The local minimization formula of Lucas-Kanade optical flow is as follows:

[0046] In the optical flow calculation, it is assumed that the grayscale of the image remains unchanged in the local window, that is, for the pixel located at (x, y), its grayscale value at time t and t+Δt satisfies:

[0047] I(x,y,t)≈I(x+Δx,y+Δy,t+Δt).

[0048] In order to solve the pixel displacement (Δx, Δy), the sum of squared errors is minimized within the local window w. The formula is expressed as:

[0049] E(Δx, Δy)=∑ (x,y)∈w [I(x+Δx,y+Δy,t+Δt)-I(x,y,t)] 2 .

[0050] A three-level image pyramid is constructed, and the optical flow vector is iteratively calculated layer by layer to solve the problem of cross-frame loss of large displacement feature points. Specifically, the image is down-sampled at multiple levels to form a resolution hierarchy from coarse to fine, and an image pyramid is constructed. Then, the initial displacement is estimated starting from the top low-resolution image, and it is transferred and optimized layer by layer to the original resolution of the bottom layer. After completing the forward tracking, reverse tracking is performed, and then abnormal points with excessive displacement deviation are eliminated.

[0051] The formula for calculating pixel speed is:

[0052]

[0053] Among them, v pixel is the pixel velocity, (Δx, Δy) is the displacement of feature points in adjacent frames, and Δt is the time interval.

[0054] After calculating the pixel velocity, outliers are removed based on the median absolute deviation (MAD) and the pixels that meet the effective speed.

[0055] S3-3. Based on the pre-calibrated camera parameters (installation height H, focal length f, pitch angle θ), the pixel velocity is mapped to the actual flow velocity. The calculation formula is:

[0056]

[0057] Among them, v s is the actual flow velocity, (v-v0) is the vertical pixel offset of the feature point, and the far-field error is corrected by the lens distortion coefficient.

[0058] S4. Synchronize the visual flow measurement results with the flow velocity measured by the millimeter-wave radar in time and space, align the video and radar data timestamps through the Beidou PPS signal, and the error is ≤10ms; set up at least three reflective targets with known positions in the monitored waters, establish a mapping relationship between the radar polar coordinate system and the visual pixel coordinate system, and map the radar velocity matrix to the visual division area according to the spatial coordinates, to ensure that each area k contains both: visual velocity Radar speed and radar signal-to-noise ratio SNR (k) .

[0059] S5. Screen the trusted area and process abnormal data.

[0060] S5-1. Calculate the difference index between the visual flow measurement and the radar flow measurement results. The calculation method is as follows:

[0061]

[0062] In the above formula, the denominator is 0.1 to prevent division by zero.

[0063] S5-2, determine the credible area, if the data has high consistency and high signal-to-noise ratio, that is, C k <0.15 and SNR (k) >10dB is used as the reserved area. If the data shows that the signal-to-noise ratio is low due to interference, that is, C k ≥0.15 and SNR (k) When the value is ≤10dB, it is regarded as the rejection area.

[0064] S5-3: When processing abnormal data, the neighborhood interpolation is performed on the excluded area. The interpolation method is as follows:

[0065]

[0066] During interpolation, if the same area is eliminated for three consecutive frames, area redivision will be triggered.

[0067] S6. Fit the linear coefficients for each credible region k by the least squares method, calibrate the radar velocity, and then fuse and output the calibration results.

[0068] S6-1. Error modeling is performed by region using a linear calibration model, as follows:

[0069]

[0070] S6-2. Output the calibration results and perform global velocity fusion to obtain the overall water surface velocity. Based on the above calculation results, the regional calibration results can be output as follows:

[0071]

[0072] The final data fusion of the output results of each area is performed to obtain the overall water surface velocity, which is calculated as follows:

[0073]

[0074] Among them, w k is the regional weight, and the credible regional weight is: w k =1-C k , the excluded area does not participate in the calculation, so the weight of the excluded area w k =0.

[0075] See also Figure 2 According to the above solution, the present invention provides another new technical solution: a millimeter wave radar speed measurement calibration system based on visual flow measurement, comprising:

[0076] Perception module: By setting up cameras and installing millimeter-wave radars, images and flow rates of the water area to be measured are acquired. The acquired image data is preprocessed and a coordinate system is established. A velocity matrix is ​​generated for the flow rate data measured by the millimeter-wave radar.

[0077] Visual flow measurement module: The preprocessed image is used to identify targets based on the YOLOv8s model for optical flow calculation. After target identification is completed, the area is divided by the radar center point, and optical flow calculation is performed within the divided area to obtain the visual flow velocity.

[0078] Data calibration module: synchronizes the flow velocity measured by the millimeter-wave radar with the flow velocity results obtained by visual flow measurement in time and space, establishes the radar coordinate system and the visual coordinate system, and establishes a mapping relationship between the two.

[0079] Calculation output module: Calculate the difference index of each area, screen out the credible areas, and perform neighborhood interpolation processing on the excluded areas. Then, perform error modeling on each credible area, and finally calculate the overall water surface velocity based on the weight.

[0080] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A millimeter-wave radar velocity measurement calibration method based on visual flow measurement, characterized by: The steps include: S1. Collect data and build a database, including obtaining river and channel video streams and radar speed measurement data and performing image preprocessing; S2. Perform target detection on the preprocessed image based on YOLO v8. After target extraction, perform region segmentation on the water surface area. S3. Perform visual flow measurement calculation based on the optical flow method. When performing optical flow calculation, calculation should be performed separately in each area; In each divided sub-region, the improved Shi-Tomasi algorithm is used to detect corners; The Pyramidal Lucas-Kanade algorithm is used to calculate the motion vector of each corner point in consecutive frames, and Δx and Δy are solved by minimizing the sum of squared errors in a local window. Construct a three-level image pyramid and iteratively calculate the optical flow vector layer by layer; Based on the pre-calibrated camera parameters (mounting height H, focal length f, pitch angle θ), the pixel velocity is mapped to the actual flow velocity; S4. Synchronize the visual flow measurement results with the flow velocity measured by the millimeter-wave radar in time and space, align the video and radar data timestamps through the Beidou PPS signal, and the error is ≤10ms; set up at least three reflective targets with known positions in the monitored waters, establish a mapping relationship between the radar polar coordinate system and the visual pixel coordinate system, and when mapping the radar velocity matrix to the visual division area according to the spatial coordinates, ensure that each area k contains both: visual velocity Radar speed and radar signal-to-noise ratio SNR (k) ; S5. Screen the trusted area and process abnormal data; Calculate the difference index between visual flow measurement and radar flow measurement results; Determine the credible area, process the abnormal data, and perform neighborhood interpolation on the excluded area; During interpolation, if the same area is eliminated for three consecutive frames, the area will be re-divided; S6. For each credible region k, the linear coefficient is fitted by the least squares method, the radar velocity is corrected, and the calibration results are fused and output to obtain the overall river surface velocity.

2. The millimeter-wave radar speed measurement calibration method based on visual flow measurement according to claim 1 is characterized in that: The image preprocessing in S1 includes performing image preprocessing on the obtained video stream of rivers and channels, performing grayscale conversion on the video stream in OpenCV, and then performing image enhancement and noise reduction on the converted gray image; filtering the data obtained by the millimeter wave radar; deploying the millimeter wave radar in the monitoring area, setting its speed measurement center point as the reference point, and outputting the original point cloud data to analyze and generate a speed matrix.

3. The millimeter-wave radar velocity measurement calibration method based on visual flow measurement according to claim 1 is characterized in that: The specific steps in S2 are to build a target detection model based on the YOLO v8s model, take water surface ripples, vortices, etc. as detection objects, prepare training pictures with marked water surface ripples, vortices, etc., train the built model, use GloU to optimize the water surface area detection frame to improve the water surface area detection accuracy, introduce Focal Loss to reduce the influence of interference such as reflections and floating objects, and add gradient perception loss to make the detected water surface area more consistent with the texture characteristics of actual water flow. The YOLOv8s model includes: Backbone, Neck, Head, and Post-processing; When performing region segmentation, the center point of the millimeter-wave radar velocity measurement should be used as the reference point. Two segmentation methods are often considered. When the flow velocity of the scene to be measured is uniform and the boundary is regular, the nine-square grid segmentation can be used. When the flow velocity of the scene to be measured varies greatly with distance, the annular diffusion segmentation can better match the actual flow field characteristics. The specific steps of using the nine-square grid segmentation are as follows: the center point of the radar speed measurement is used as the reference, and 3×3 sub-areas are divided. The grid size matches the radar beam width. The specific steps of using annular diffusion segmentation are: taking the center point of the radar speed measurement as the center of the circle, expand multiple annular areas outward and set the radius interval according to needs.

4. The millimeter-wave radar speed measurement calibration method based on visual flow measurement according to claim 1 is characterized in that: In S3, the improved Shi-Tomasi algorithm is used to detect corners in each divided sub-region, and its response function is: R=min(λ1,λ2) Among them, λ1 and λ2 are the eigenvalues ​​of the grayscale covariance matrix of the local window of the image, and only the grayscale covariance matrix that satisfies R>R is retained. th By dynamically adjusting the threshold, the auxiliary detection algorithm is automatically used to supplement the feature points in sparse areas, and non-maximum suppression is used to eliminate artifacts in highly reflective areas. The Pyramidal Lucas-Kanade algorithm is used to calculate the motion vector of each corner point in consecutive frames, and Δx and Δy are solved by minimizing the sum of squared errors in a local window. The specific calculation formula is: Lucas-Kanade optical flow local minimization formula, as follows: In the optical flow calculation, it is assumed that the grayscale of the image remains unchanged in the local window, that is, for the pixel located at (x, y), its grayscale value at time t and t+Δt satisfies: I(x,y,t)≈I(x+Δx,y+Δy,t+Δt) In order to solve the pixel displacement (Δx, Δy), the sum of squared errors is minimized within the local window w. The formula is expressed as: E(Δx,Δy)=∑ (x,y)∈w [I(x+Δx,y+Δy,t+Δt)-I(x,y,t)] 2 Construct a three-level image pyramid and iteratively calculate the optical flow vector layer by layer to solve the problem of large displacement feature points being lost across frames. Specifically, the image is downsampled at multiple levels to form a resolution hierarchy from coarse to fine. The image pyramid is then constructed. The initial displacement is estimated starting from the top low-resolution image, and is transferred and optimized layer by layer to the original resolution of the bottom layer. After completing the forward tracking, the reverse tracking is performed, and outliers with excessive displacement deviations are removed. The formula for calculating pixel speed is: Among them, v pixel is the pixel velocity, (Δx, Δy) is the displacement of feature points in adjacent frames, and Δt is the time interval; After calculating the pixel velocity, outliers are removed based on the median absolute deviation (MAD) and the pixels that meet the The effective speed of Based on the pre-calibrated camera parameters (installation height H, focal length f, pitch angle θ), the pixel velocity is mapped to the actual flow velocity, and the calculation formula is: Among them, v s is the actual flow velocity, (v-v0) is the vertical pixel offset of the feature point, and the far-field error is corrected by the lens distortion coefficient.

5. The millimeter-wave radar speed measurement calibration method based on visual flow measurement according to claim 1 is characterized in that: The S4 specifically includes: aligning the video and radar data timestamps through the Beidou PPS signal, with an error of ≤10ms; placing at least three reflective targets with known positions in the monitored waters, establishing a mapping relationship between the radar polar coordinate system and the visual pixel coordinate system, and mapping the radar velocity matrix to the visual division area according to the spatial coordinates, to ensure that each area k contains both: visual velocity Radar speed and radar signal-to-noise ratio SNR (k) .

6. The millimeter wave radar speed measurement calibration method based on visual flow measurement according to claim 1 is characterized in that: In S5, the difference index between the visual flow measurement result and the radar flow measurement result is calculated as follows: (The denominator is 0.1 to prevent division by zero) The method for determining the credible area is as follows: if the data has high consistency and high signal-to-noise ratio, that is, C k <0.15 and SNR (k) >10dB is used as the reserved area. If the data shows that the signal-to-noise ratio is low due to interference, that is, C k ≥0.15 and SNR (k) When ≤10dB, it is regarded as the rejection area; When processing abnormal data, neighborhood interpolation is performed on the excluded area. The interpolation method is as follows: During interpolation, if the same area is eliminated for three consecutive frames, area redivision will be triggered.

7. The millimeter-wave radar speed measurement calibration method based on visual flow measurement according to claim 1 is characterized in that: The specific steps of S6 are as follows: The error modeling is performed in the sub-regions using a linear calibration model, as follows: The output calibration results are then combined with global velocity fusion to obtain the overall water surface velocity. Based on the above calculation results, the regional calibration results can be output as follows: The final data fusion of the output results of each area is performed to obtain the overall water surface velocity, which is calculated as follows: Among them, w k is the regional weight, and the credible regional weight is: w k =1-C k , the excluded area does not participate in the calculation, so the weight of the excluded area w k =0.

8. A millimeter-wave radar speed measurement calibration system based on visual flow measurement, based on the millimeter-wave radar speed measurement calibration method based on visual flow measurement according to any one of claims 1 to 7, characterized in that: It includes perception module, visual flow measurement module, data calibration module and calculation output module.

Images