Image scale reconstruction method and related device

Abstract

The invention discloses an image scale reconstruction method and a related device, and the method comprises the steps: carrying out the timestamp matching of a camera and a millimeter-wave radar, obtaining a radar-image matching frame pair, carrying out the monocular depth estimation of an image frame, obtaining a depth map, and carrying out the calculation of the depth map according to the external parameters from the millimeter-wave radar to the camera; transforming radar point cloud in the radar-image matching frame into a camera coordinate system to obtain target radar point cloud, performing perspective projection according to internal reference of a camera, mapping to an image plane, performing ratio operation on a physical depth value detected by the millimeter wave radar and a relative depth value in a depth map to obtain a scale factor, and performing calculation on the scale factor; and determining a scale factor median from all scale factors corresponding to the current image frame as a scale reference value of the current image frame. The millimeter-wave radar and the camera are introduced to work cooperatively, physical distance information measured by the radar is mapped to an image plane through timestamp synchronization, and the actual distance between an object in an image and the camera is determined by combining the advantages of the millimeter-wave radar.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, more particularly, to an image scale reconstruction method and related device. BACKGROUND

[0002] In the visual perception scene, due to the difference of camera intrinsic parameters, combined with the continuous change of scene semantics, the same object may present different visual scales in different images. However, these visual scales cannot truly reflect the actual size of the object in reality.

[0003] In the application scenarios such as automatic driving, intelligent inspection and security monitoring, the visual perception system has accurate and stable requirements for the visual scale, and needs to accurately determine the actual distance between the object in the image and the camera. To solve this problem, monocular depth estimation technology is generally used at present, which outputs normalized relative depth. However, this method has obvious limitations, which leads to the fact that only the front and back relationship between objects can be perceived, but the real physical size of the object cannot be obtained. At the same time, the monocular depth estimation method is not good in universality among different scenes, and is extremely sensitive to factors such as light, shadow and weather changes, so it is difficult to achieve stable and reliable depth perception effect, and thus it is also difficult to support advanced decision-making tasks that rely on accurate image scale. SUMMARY

[0004] Therefore, the present application discloses an image scale reconstruction method and related device to make full use of the advantages of millimeter wave radar, such as being not limited by light conditions, being suitable for harsh weather, having low power consumption and low cost, so as to be suitable for long-term actual scene deployment, effectively solving the problem that the traditional image scale estimation is affected by scene semantic changes and environmental interference, thereby improving the precision and robustness of the depth estimation system in complex dynamic scenes, and thus being able to provide data support for advanced decision-making tasks that rely on accurate image scale.

[0005] An image scale reconstruction method, comprising:

[0006] Timestamp matching is performed on the camera and the millimeter wave radar to obtain a radar-image matching frame pair;

[0007] Monocular depth estimation is performed on the image frame of the radar-image matching frame to obtain a depth map corresponding to the image frame;

[0008] According to the calibrated extrinsic parameters of the millimeter wave radar to the camera, the corresponding radar point cloud in the radar-image matching frame is transformed from the radar coordinate system to the camera coordinate system to obtain a target radar point cloud;

[0009] According to the calibrated intrinsic parameters of the camera, perspective projection is performed on the target radar point cloud transformed to the camera coordinate system to map the target radar point cloud to the image plane;

[0010] For each radar point successfully mapped to the image plane, the physical depth value detected by the millimeter-wave radar is compared with the corresponding relative depth value in the depth map to obtain the scale factor of each radar point.

[0011] Obtain all scale factors corresponding to the current image frame, and determine the value of the scale factor from all scale factors as the scale reference value of the current image frame.

[0012] Optionally, the step of performing timestamp matching on the camera and millimeter-wave radar to obtain radar-image matching frame pairs includes:

[0013] Acquire the current point cloud frame data of the millimeter-wave radar, and extract the radar frame timestamp from the current point cloud frame data;

[0014] Acquire the image frame data continuously output by the camera, and extract the corresponding image frame timestamp from each image frame data;

[0015] The method of minimum absolute time difference matching is adopted to select the image frame with the smallest difference between the image frame timestamp and the radar frame timestamp from all candidate image frames, and the absolute value of the difference is not greater than the maximum tolerance threshold, as the best matching image frame for the current radar frame.

[0016] Based on the current radar frame and its corresponding best matching image frame, a radar-image matching frame pair is obtained.

[0017] Optionally, after performing perspective projection on the target radar point cloud transformed into the camera coordinate system according to the calibrated intrinsic parameters of the camera, and mapping the target radar point cloud onto the image plane, the method further includes:

[0018] The target radar point cloud is mapped onto the image plane to obtain the projection point set;

[0019] A one-to-one correspondence is established between the projection point set and the image pixel coordinates to form a point cloud projection-image pixel correspondence table.

[0020] Optionally, for each radar point successfully mapped to the image plane, the ratio of the physical depth value detected by the millimeter-wave radar to the corresponding relative depth value in the depth map is calculated to obtain the scale factor for each radar point, including:

[0021] From the point cloud projection-image pixel correspondence table, extract the three-dimensional coordinates of each radar point in the camera coordinate system, and extract the depth component from the three-dimensional coordinates as the physical depth value of the point cloud pixel.

[0022] Based on the image pixel positions recorded in the point cloud projection-image pixel correspondence table, the normalized relative depth value of the corresponding point cloud pixel is extracted from the depth map, wherein the relative depth value reflects the relative distance of the point cloud pixel relative to the camera.

[0023] The scale factor of each radar point is obtained by comparing the physical depth value with the corresponding relative depth value.

[0024] Optionally, obtaining all scale factors corresponding to the current image frame and determining the value of the scale factor from all scale factors as the scale reference value of the current image frame includes:

[0025] Obtain all scale factors corresponding to the current image frame to form a scale value set;

[0026] Outlier detection is performed on the scale value set to identify abnormal scale factors whose distribution deviates significantly from the normal range, and the abnormal scale factors are removed to obtain a cleaned target scale factor set.

[0027] The scale factors in the target scale factor set are sorted in ascending order of their numerical values ​​to form an ordered scale sequence;

[0028] The median of the scale factor is extracted from the ordered scale sequence, and the median of the scale factor is used as the scale reference value of the current image frame.

[0029] Optionally, it also includes:

[0030] Calculate the rate of change of the median scale factor relative to the median scale factor of the previous frame;

[0031] If the scale change rate exceeds the change rate threshold, it is determined that the median value of the scale factor is abnormal, and the image brightness feature time series is extracted from the image frames of the radar-image matching frame;

[0032] Anomaly detection is performed on the time series of the image brightness features;

[0033] When it is determined from the image brightness feature time series that the brightness of the current frame is not abnormal, a texture complexity index is constructed and recorded in the texture complexity time series;

[0034] Image texture anomaly detection is performed on the texture complexity time series to determine the cause of scale anomalies in the current frame.

[0035] Optionally, extracting the image brightness feature time series from the corresponding image frame in the radar-image matching frame includes:

[0036] The color space of the corresponding image frame in the radar-image matching frame is converted, and the luminance channel image is extracted from the converted color space.

[0037] The average value of the overall brightness of the current image frame is obtained by averaging all pixel values ​​in the brightness channel image.

[0038] Based on the pixel value distribution in the luminance channel, count the number of pixels at all luminance levels and construct a luminance histogram.

[0039] The overall average brightness and the brightness histogram are added to the image brightness feature time series.

[0040] Optionally, anomaly detection is performed on the time series of the image brightness features, including:

[0041] Extract the mean brightness sequence and brightness histogram sequence of the N nearest frames to the current image frame from the image brightness feature time series;

[0042] Calculate the sequence mean and sequence standard deviation of the brightness mean sequence, and based on the sequence standard deviation, standardize the difference between the current frame brightness mean and the sequence mean to obtain the standardized difference.

[0043] Calculate the Kullback-Leibler divergence between the current frame's luminance histogram and the average historical histogram of the nearest N-1 frames in the luminance histogram sequence.

[0044] The standardized difference and the Kullback-Leibler divergence are compared with their respective thresholds, and the anomaly of the image brightness feature time series is determined based on the comparison results.

[0045] Optionally, the texture construction complexity metric includes:

[0046] Convert the image frames of the radar-image matching frames into grayscale images;

[0047] Determine the horizontal and vertical gradients of the grayscale image, and calculate the gradient magnitude map based on the horizontal and vertical gradients;

[0048] The texture complexity index of the image frame is obtained by averaging the values ​​of all pixels in the gradient magnitude map.

[0049] Optionally, it also includes:

[0050] If the scale change rate does not exceed the change rate threshold, or if the image texture is determined to be normal according to the texture complexity index, the scene is determined to be unchanged, and the global scale factor is updated using an exponential moving average method.

[0051] If the scale change rate exceeds the change rate threshold, or if an image texture anomaly is determined based on the texture complexity index, and the scene is determined to have changed, the median value of the scale factor of the current image frame is directly used as the new scene scale to complete the size factor update.

[0052] An image scale reconstruction apparatus, comprising:

[0053] The timestamp matching unit is used to perform timestamp matching between the camera and the millimeter-wave radar to obtain radar-image matching frame pairs.

[0054] A depth estimation unit is used to perform monocular depth estimation on the image frame of the radar-image matching frame to obtain a depth map corresponding to the image frame.

[0055] The coordinate system transformation unit is used to transform the radar point cloud corresponding to the radar-image matching frame from the radar coordinate system to the camera coordinate system according to the calibrated extrinsic parameters of the millimeter-wave radar to the camera, so as to obtain the target radar point cloud.

[0056] The projection unit is used to perform perspective projection on the target radar point cloud transformed into the camera coordinate system according to the calibrated intrinsic parameters of the camera, and to map the target radar point cloud onto the image plane.

[0057] The scale factor determination unit is used to perform a ratio calculation between the physical depth value detected by the millimeter-wave radar and the corresponding relative depth value in the depth map for each radar point successfully mapped to the image plane, so as to obtain the scale factor of each radar point.

[0058] The scale determination unit is used to obtain all scale factors corresponding to the current image frame, and determine the value of the scale factor from all scale factors as the scale reference value of the current image frame.

[0059] A computer storage medium storing at least one instruction that, when executed by a processor, implements the image scale reconstruction method described above.

[0060] An electronic device, comprising: a memory and a processor;

[0061] The memory is used to store at least one instruction;

[0062] The processor is used to execute the at least one instruction to implement the image scale reconstruction method described above.

[0063] As can be seen from the above technical solution, the present invention discloses an image scale reconstruction method and related apparatus. The method involves time-stamp matching of a camera and a millimeter-wave radar to obtain radar-image matching frame pairs. Monocular depth estimation is performed on the image frames of the radar-image matching frames to obtain a depth map corresponding to the image frames. Based on the extrinsic parameters of the calibrated millimeter-wave radar to the camera, the radar point cloud corresponding to the radar-image matching frame is transformed from the radar coordinate system to the camera coordinate system to obtain the target radar point cloud. Based on the calibrated intrinsic parameters of the camera, perspective projection is performed on the target radar point cloud transformed to the camera coordinate system to map the target radar point cloud onto the image plane. For each radar point successfully mapped to the image plane, the ratio of the physical depth value detected by the millimeter-wave radar to the corresponding relative depth value in the depth map is calculated to obtain the scale factor of each radar point. All scale factors corresponding to the current image frame are obtained, and the median value of the scale factors is determined from all scale factors as the scale reference value of the current image frame. This invention introduces a collaborative approach between millimeter-wave radar and a camera. By calibrating the extrinsic parameters of the millimeter-wave radar and the camera, and synchronizing the timestamps of their data, the physical distance information measured by the radar is mapped onto the image plane. This establishes a stable correspondence between pixel coordinates and physical scale, obtaining highly reliable scale data and enabling the determination of the actual distance between objects in the image and the camera. This invention fully leverages the advantages of millimeter-wave radar—its lack of limitation due to lighting conditions, adaptability to harsh weather, low power consumption, and low cost—making it suitable for long-term deployment in real-world scenarios. It effectively solves the problem of traditional image scale estimation being affected by scene semantic changes and environmental interference, thereby improving the accuracy and robustness of depth estimation systems in complex dynamic scenes. This provides data support for advanced decision-making tasks that rely on precise image scale. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0065] Figure 1 This is a flowchart of an image scale reconstruction method disclosed in an embodiment of the present invention;

[0066] Figure 2 This is a flowchart of another image scale reconstruction method disclosed in an embodiment of the present invention;

[0067] Figure 3 This is a schematic diagram of the structure of an image scale reconstruction device disclosed in an embodiment of the present invention;

[0068] Figure 4This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] This invention discloses an image scale reconstruction method and related apparatus. It introduces a millimeter-wave radar and camera working collaboratively. By calibrating the extrinsic parameters of the millimeter-wave radar and camera and synchronizing the timestamps of their data, the physical distance information measured by the radar is mapped onto the image plane. This establishes a stable correspondence between pixel coordinates and physical scale, obtaining highly reliable scale data and enabling the determination of the actual distance between objects in the image and the camera. This invention fully utilizes the advantages of millimeter-wave radar—its lack of limitation due to lighting conditions, adaptability to harsh weather, low power consumption, and low cost—making it suitable for long-term deployment in real-world scenarios. It effectively solves the problem of traditional image scale estimation being affected by scene semantic changes and environmental interference, thereby improving the accuracy and robustness of depth estimation systems in complex dynamic scenes. This provides data support for advanced decision-making tasks that rely on precise image scale.

[0071] See Figure 1 The present invention discloses a flowchart of an image scale reconstruction method, which includes:

[0072] Step S101: Perform timestamp matching on the camera and millimeter-wave radar to obtain radar-image matching frame pairs.

[0073] The radar-image matching frame pair includes: radar point cloud and the corresponding image frame.

[0074] Millimeter-wave radar has advantages such as being unaffected by light, adapting to harsh weather, low power consumption, and low cost, making it more suitable for long-term deployment and use in real-world scenarios.

[0075] This embodiment performs timestamp matching between the camera and the millimeter-wave radar. In fact, it matches the synchronization frames of the camera and the millimeter-wave radar based on their timestamps to ensure temporal consistency between the images acquired by the camera and the point clouds acquired by the millimeter-wave radar, providing the necessary prerequisite for subsequent image spatial mapping and scale factor calculation.

[0076] Step S102: Perform monocular depth estimation on the image frame of the radar-image matching frame to obtain the depth map corresponding to the image frame.

[0077] In practical applications, deep neural networks can be used to perform monocular depth estimation on camera image frames to obtain the relative depth of each pixel in the image frame, thereby obtaining the depth map corresponding to the image frame. This provides a basis for constructing a dense depth map from the camera's perspective and serves as the image source for subsequent scale estimation.

[0078] Step S103: Based on the calibrated extrinsic parameters of the millimeter-wave radar to the camera, transform the radar point cloud corresponding to the radar-image matching frame from the radar coordinate system to the camera coordinate system to obtain the target radar point cloud.

[0079] In practical applications, the intrinsic parameters of the camera and the extrinsic parameters of the millimeter-wave radar can be obtained through calibration. These intrinsic and extrinsic parameters provide the geometric basis for subsequent image and radar data fusion. Both the intrinsic and extrinsic parameters are matrices, with the extrinsic parameters including a rotation matrix R and a translation vector T.

[0080] In this embodiment, based on the extrinsic parameters of millimeter-wave radar to the camera, the three-dimensional points in the radar coordinate system are transformed to the camera coordinate system. The three-dimensional points are the radar point clouds corresponding to the radar-image matching frames.

[0081] Step S104: Based on the calibrated intrinsic parameters of the camera, perform perspective projection on the target radar point cloud transformed into the camera coordinate system, and map the target radar point cloud onto the image plane.

[0082] This embodiment performs perspective projection on the target radar point cloud transformed into the camera coordinate system based on the camera's intrinsic parameters, so as to map the target radar point cloud onto the image plane, thereby completing the mapping of three-dimensional spatial points onto the two-dimensional image plane and realizing cross-modal data alignment and fusion.

[0083] Step S105: For each radar point successfully mapped to the image plane, the ratio of the physical depth value detected by the millimeter-wave radar to the corresponding relative depth value in the depth map is calculated to obtain the scale factor of each radar point.

[0084] The scale factor of radar points is a key parameter used in radar data processing to adjust the scale ratio of point cloud data. Essentially, it's a scaling factor that converts relative depth values ​​in a depth map into actual physical distances. This embodiment utilizes the scale factor to subsequently infer the true size of the image depth map.

[0085] Step S106: Obtain all scale factors corresponding to the current image frame, and determine the value of the scale factor from all scale factors as the scale reference value of the current image frame.

[0086] In practical applications, all scale factors corresponding to the current image frame can be sorted in descending order. After removing outliers, the median of the scale factors can be determined and used as the scale reference value of the current image frame. This scale reference value is also the representative scale estimate of the current image frame.

[0087] The median scaling factor exhibits strong robustness and resistance to interference, effectively suppressing the impact of local outliers or single-point errors on the overall estimation, and can represent the scaling level of most matched point pairs.

[0088] In summary, this invention discloses an image scale reconstruction method. It involves timestamp matching between a camera and a millimeter-wave radar to obtain radar-image matching frame pairs. Monocular depth estimation is performed on the image frames of the radar-image matching frames to obtain the corresponding depth maps. Based on the extrinsic parameters of the calibrated millimeter-wave radar and camera, the radar point cloud corresponding to the radar-image matching frame is transformed from the radar coordinate system to the camera coordinate system to obtain the target radar point cloud. Based on the calibrated intrinsic parameters of the camera, perspective projection is performed on the target radar point cloud transformed to the camera coordinate system to map the target radar point cloud onto the image plane. For each radar point successfully mapped to the image plane, the ratio of the physical depth value detected by the millimeter-wave radar to the corresponding relative depth value in the depth map is calculated to obtain the scale factor of each radar point. All scale factors corresponding to the current image frame are obtained, and the median value of the scale factors is determined from all scale factors as the scale reference value for the current image frame. This invention introduces a collaborative approach between millimeter-wave radar and a camera. By calibrating the extrinsic parameters of the millimeter-wave radar and the camera, and synchronizing the timestamps of their data, the physical distance information measured by the radar is mapped onto the image plane. This establishes a stable correspondence between pixel coordinates and physical scale, obtaining highly reliable scale data and enabling the determination of the actual distance between objects in the image and the camera. This invention fully leverages the advantages of millimeter-wave radar—its lack of limitation due to lighting conditions, adaptability to harsh weather, low power consumption, and low cost—making it suitable for long-term deployment in real-world scenarios. It effectively solves the problem of traditional image scale estimation being affected by scene semantic changes and environmental interference, thereby improving the accuracy and robustness of depth estimation systems in complex dynamic scenes. This provides data support for advanced decision-making tasks that rely on precise image scale.

[0089] In one embodiment, step S101 may specifically include:

[0090] (1) Obtain the current point cloud frame data of the millimeter-wave radar and extract the radar frame timestamp from the current point cloud frame data.

[0091] Specifically, the current point cloud frame data of the millimeter-wave radar is acquired, and the radar frame timestamp of the current point cloud frame data is extracted. The format is a UNIX timestamp (in milliseconds or microseconds).

[0092] (2) Obtain the image frame data continuously output by the camera, and extract the corresponding image frame timestamp from each image frame data.

[0093] Acquire image frame data continuously output by the camera and from each image frame data Extract image frame timestamps Image frame timestamp With radar frame timestamp The format is consistent.

[0094] (3) Using the minimum absolute time difference matching method, the image frame with the smallest difference between the timestamp of the image frame and the timestamp of the radar frame and the absolute value of the difference is not greater than the maximum tolerance threshold is selected from all candidate image frames and used as the best matching image frame for the current radar frame.

[0095] radar frame timestamps Image frame timestamp The minimum absolute time difference matching method is used for processing, and the maximum tolerance threshold can be set to... .

[0096] From all candidate image frames, select the image frame with the smallest difference between its timestamp and the radar frame timestamp, and The image frame is selected as the best matching image frame for the current radar frame. Radar-image matching frame pairs are obtained based on the matching results.

[0097] (4) Based on the current radar frame and its corresponding best matching image frame, obtain the radar-image matching frame pair.

[0098] The radar-image matching frame pair contains information such as radar point cloud, radar frame timestamp, image frame, image frame timestamp, and the time difference between the two.

[0099] In one embodiment, the calibration process for the camera's intrinsic parameters and the millimeter-wave radar's extrinsic parameters can be as follows:

[0100] (1) Acquire camera calibration image data.

[0101] Specifically, in a laboratory environment, a standard checkerboard calibration board is placed, and multiple images are acquired at different angles and distances to obtain camera calibration image data.

[0102] (2) The camera calibration image data is processed using the checkerboard calibration method. OpenCV's findChessboardCorners is used to detect the corner positions of the checkerboard, and calibrateCamera is called to calculate the camera's intrinsic parameters.

[0103] OpenCV (an open-source computer vision library) treats images as pixel matrices and uses matrix operations and optimization algorithms to achieve efficient image processing. It covers multiple aspects, including underlying data structures, core algorithms, performance optimization, and application support.

[0104] findChessboardCorners is a core function in the OpenCV library used to detect the coordinates of the interior corner points of a chessboard image. It is often used in high-precision scenarios such as camera calibration and 3D reconstruction.

[0105] calibrateCamera is the core function in the OpenCV library for camera calibration. It calculates the camera's intrinsic parameters (focal length, distortion coefficients, etc.) and extrinsic parameters (rotation matrix, translation vector) from multi-view images.

[0106] (3) Acquire camera image data containing planar features collected by the camera.

[0107] In practical applications, camera image data can be lane image data containing planar features such as lane lines and ground markings.

[0108] (4) The camera image data is processed using the findHomography method of OpenCV. Based on the correspondence between the feature points between the camera image coordinates and the actual physical coordinates of the ground, the homography matrix from the camera to the ground is calculated.

[0109] `findHomography` is a function in OpenCV used to calculate the homography matrix between two planes, commonly used for tasks such as image registration and stitching. Its core function is to estimate the transformation matrix describing the projection relationship between images by matching feature point pairs.

[0110] (5) Obtain millimeter-wave radar point cloud data. By measuring the installation position relationship between the camera and the millimeter-wave radar on the vehicle, obtain the approximate relative rotation matrix and translation vector between the camera and the millimeter-wave radar, and use the approximate relative rotation matrix and translation vector as the initial external parameters of the radar to the camera.

[0111] (6) The camera’s intrinsic parameters, identity matrix and initial extrinsic parameters are processed using manual_calib. The rotation matrix and translation vector are adjusted in the interactive interface to align the projection of the millimeter-wave radar point cloud onto the camera image with static environmental features such as lane lines and road boundaries, so as to obtain the extrinsic parameters of the radar to the camera.

[0112] manual_calib is an open-source tool for calibrating sensor extrinsic parameters. It is mainly used for extrinsic parameter calibration of LiDAR and cameras, and supports manual parameter adjustment and visualization of calibration results.

[0113] In one embodiment, after step S104, the image scale reconstruction method may further include:

[0114] The target radar point cloud is mapped onto the image plane to obtain the projection point set;

[0115] A one-to-one correspondence is established between the projection point set and the image pixel coordinates to form a point cloud projection-image pixel correspondence table.

[0116] Correspondingly, step S105 may specifically include:

[0117] (1) Extract the three-dimensional coordinates of each radar point in the camera coordinate system from the point cloud projection-image pixel correspondence table, and extract the depth component from the three-dimensional coordinates as the physical depth value of the point cloud pixel.

[0118] From the point cloud projection-image pixel correspondence table, the 3D coordinates of each radar point in the camera coordinate system are extracted, and the depth component (Z-axis component) is extracted from the 3D coordinates as the physical depth value. The physical depth value has physical units (meters) and represents the actual distance of the point cloud pixel in real space.

[0119] (2) Based on the image pixel positions recorded in the point cloud projection-image pixel correspondence table, extract the normalized relative depth values ​​of the corresponding point cloud pixels from the depth map.

[0120] Based on the image pixel positions recorded in the point cloud projection-image pixel correspondence table, the depth map Extract the normalized relative depth value of the corresponding point cloud pixels. The relative depth value is a dimensionless relative depth that reflects the relative distance of point cloud pixels relative to the camera.

[0121] (3) The physical depth value is compared with the corresponding relative depth value to obtain the scale factor of each radar point.

[0122] For each pair of point cloud pixels, calculate the scale factor. , where Z is the physical depth value (in meters) of a point cloud pixel, and d is the relative depth value of a point cloud pixel. The scale factor S represents the scaling factor required to restore the relative depth to the physical depth.

[0123] In one embodiment, step S106 may specifically include:

[0124] (1) Obtain all scale factors corresponding to the current image frame to form a scale value set.

[0125] From the scale factors of each pair of point cloud pixel pairs, obtain all the scale factors S corresponding to the current image frame to form a scale value set.

[0126] (2) Perform outlier detection on the scale value set, identify abnormal scale factors whose distribution deviates significantly from the normal range, and remove the abnormal scale factors to obtain the cleaned target scale factor set.

[0127] In practical applications, the IsolationForest algorithm can be used to detect outliers in the scale value set, identify abnormal scale factors whose distribution deviates significantly from the normal range, and remove them to obtain a cleaned target scale factor set.

[0128] (3) Sort the scale factors in the target scale factor set in ascending order according to their numerical values ​​to form an ordered scale sequence.

[0129] (4) Extract the median of the scale factor from the ordered scale sequence and use the median of the scale factor as the scale reference value of the current image frame.

[0130] The median typically exhibits strong robustness and resistance to interference. Based on this, this application extracts the median scale factor from ordered scale sequences. As a scale reference value for the current image frame, it effectively suppresses the influence of local outliers, thereby characterizing the scale level of most matching point pairs.

[0131] In practical applications, when camera systems frequently encounter scale drift and depth instability caused by environmental changes (such as switching between indoor and outdoor environments, drastic changes in lighting, and severe weather), the robustness and reliability of the camera system are easily affected.

[0132] To maintain the temporal continuity and spatial stability of scale information, this invention introduces a dynamic scale control strategy based on median statistics and exponential decay smoothing. When scene changes or significant inter-frame differences occur, the system automatically switches to a multi-point fusion scale estimation method, effectively mitigating scale drift and abrupt changes. During scene changes (such as switching from indoor to outdoor), the system automatically switches scale and employs a multi-point scale averaging mechanism when necessary to ensure the continuity and stability of the scale estimation process. The overall technical approach integrates the stable ranging capability of millimeter-wave radar with the high-resolution characteristics of visual images, ultimately achieving a scale perception capability that meets operational requirements and can operate stably for extended periods.

[0133] For details, see Figure 2 The present invention discloses another image scale reconstruction method flowchart. Based on the above embodiments, the image scale reconstruction method may further include:

[0134] Step S107: Calculate the rate of change of the median scale factor relative to the median scale factor of the previous frame.

[0135] Median scale factor of the current image frame Median scale factor relative to the previous frame scale change rate The calculation formula is as follows:

[0136] (1);

[0137] By calculating the scale change rate between two adjacent frames This quantifies the degree of scale difference between adjacent frames.

[0138] Step S108: If the scale change rate exceeds the change rate threshold, it is determined that the median value of the scale factor is abnormal, and the image brightness feature time series is extracted from the image frame of the radar-image matching frame.

[0139] This embodiment will calculate the scale change rate. Compared with the set rate of change threshold Compare, if > If the median value of the scale factor is abnormal, it is preliminarily determined that there is a possibility of scene switching or sudden environmental change in the current image frame; otherwise, it is determined that the scale change of the current image frame is not significant.

[0140] When the median value of the judgment scale factor is abnormal, the image brightness feature time series is extracted from the image frame of the radar-image matching frame. This is then used to conduct image-assisted analysis in combination with brightness distribution, edge features, and key area texture changes to confirm whether a scene switch or abrupt change has indeed occurred.

[0141] Step S109: Perform anomaly detection on the image brightness feature time series.

[0142] Step S110: When it is determined from the image brightness feature time series that the brightness of the current frame is not abnormal, a texture complexity index is constructed and recorded in the texture complexity time series.

[0143] Texture complexity metrics are used to quantify the richness or variation of textures in an image, typically reflecting local or global features such as grayscale distribution, edge density, and structural repetition.

[0144] Step S111: Perform image texture anomaly detection on the texture complexity time series to determine the cause of the current frame scale anomaly.

[0145] In practical applications, the texture complexity of the most recent N frames is extracted from the texture complexity time series, and they are as follows: And calculate the mean texture complexity of the most recent N frames. and standard deviation mean The calculation expression is shown in formula (2), and the standard deviation is... The calculation expression is shown in formula (3).

[0146] (2);

[0147] (3).

[0148] Calculate the texture complexity index of the current image frame. with the mean The difference, and the standard deviation Compare: ,in, The threshold is an empirical threshold. A value of 2 indicates that a texture change has occurred in the current image frame, and the scene has changed. Otherwise, the texture change is considered to be continuous, and the scene remains stable.

[0149] In one embodiment, step S108, which involves extracting the image brightness feature time series from the image frames of the radar-image matching frame, may specifically include:

[0150] (1) Perform color space conversion on the corresponding image frame in the radar-image matching frame, and extract the luminance channel image from the converted color space.

[0151] In practical applications, the corresponding image frame in the radar-image matching frame can be converted from the RGB (Red, Green, Blue) color space to the HSV (Hue, Saturation, Value) color space, and the V channel can be extracted to obtain the luminance channel image L.

[0152] (2) Average all pixel values ​​in the brightness channel image to obtain the overall average brightness of the current image frame.

[0153] The average value of all pixels in the luminance channel image L is calculated to obtain the overall luminance mean of the current image frame. The expression is as follows:

[0154] (4);

[0155] In the formula, Indicates the image height. Indicates the image width. Indicates pixel brightness value. and It represents the two-dimensional coordinate position of a pixel in an image.

[0156] (3) Based on the distribution of pixel values ​​in the brightness channel, count the number of pixels at all brightness levels and construct a brightness histogram.

[0157] In practical applications, a luminance histogram can be constructed by counting the number of pixels at the 256 luminance levels (0-255) based on the pixel value distribution in the luminance channel LLL. The expression is as follows:

[0158] ;

[0159] Brightness histogram It reflects the overall brightness distribution.

[0160] (4) Add the overall average brightness and the brightness histogram to the image brightness feature time series.

[0161] In one embodiment, step S109 may specifically include:

[0162] (1) Extract the mean brightness sequence and brightness histogram sequence of the nearest N frames to the current image frame from the image brightness feature time series.

[0163] In practical applications, the average brightness sequence corresponding to the N nearest frames to the current image frame is extracted from the image brightness feature time series. and brightness histogram sequence It is used to calculate statistics and detect anomalies.

[0164] (2) Calculate the sequence mean and sequence standard deviation of the brightness mean sequence, and standardize the difference between the current frame brightness mean and the sequence mean based on the sequence standard deviation to obtain the standardized difference.

[0165] Calculate the mean of the most recent N frame luminance mean sequence with standard deviation ,as follows:

[0166] (5);

[0167] (6);

[0168] In the formula, t represents time.

[0169] Average brightness of the current frame The mean of the most recent N frame luminance mean sequence The difference is standardized according to formula (7), and the standard value is... The expression is as follows:

[0170] (7).

[0171] (3) Calculate the Kullback-Leibler divergence between the current frame luminance histogram and the historical average histogram of the nearest N-1 frames in the luminance histogram sequence.

[0172] Specifically, calculate the brightness histogram of the current frame. Compared with the historical average histogram of the most recent N-1 frames The Kullback-Leibler divergence between them.

[0173] The historical average histogram of the most recent N-1 frames The expression is as follows:

[0174] = (8).

[0175] Kullback-Leibler divergence The expression is as follows:

[0176] (9);

[0177] In the formula, , To prevent small values ​​from being divided by zero.

[0178] (4) Compare the standardized difference and the Kullback-Leibler divergence with the corresponding thresholds respectively, and determine whether the image brightness feature time series is abnormal based on the comparison results.

[0179] like and , The preset difference threshold, If the predefined divergence threshold is used, the current frame brightness is determined to be abnormal, indicating a change in the scene; otherwise, the current frame brightness is determined to be normal.

[0180] In one embodiment, the process of constructing the texture complexity index in step S110 is as follows:

[0181] (1) Convert the image frames of radar-image matching frames into grayscale images.

[0182] (2) Determine the horizontal and vertical gradients of the grayscale image, and calculate the gradient magnitude map based on the horizontal and vertical gradients.

[0183] In practical applications, the Sobel operator can be used to determine grayscale images. horizontal gradient with vertical gradient Then based on the horizontal gradient with vertical gradient Calculate gradient magnitude map Gradient magnitude plot The expression is as follows:

[0184] (10).

[0185] (3) Take the average value of all pixels in the gradient magnitude map to obtain the texture complexity index of the image frame.

[0186] Gradient magnitude map The texture complexity index of the image frame is obtained by averaging all pixels. Texture complexity index The expression is as follows:

[0187] (11);

[0188] In the formula, and These are the row and column dimensions of the image, respectively. and It represents the two-dimensional coordinate position of a pixel in an image.

[0189] In practical applications, texture complexity metrics can be used... Recorded in the texture complexity time series.

[0190] In one embodiment, the image scale reconstruction method may further include:

[0191] (1) If the scale change rate does not exceed the change rate threshold, or if the image texture is determined to be normal according to the texture complexity index, the scene is determined to be unchanged, and the global scale factor is updated by exponential moving average.

[0192] The global scaling factor is updated using the Exponential Moving Average (EMA) method, as shown in the following expression:

[0193] (12);

[0194] In the formula, This represents the updated global scale factor at time t. For smoothing coefficients, Indicates the median of the scaling factor. This represents the global scale factor at time t-1 before the update.

[0195] (2) If the scale change rate exceeds the change rate threshold, or if the image texture is abnormal according to the texture complexity index, and the scene changes, the median value of the scale factor of the current image frame is directly used as the new scale of the scene to complete the size factor update.

[0196] If the rate of change of scale exceeds the rate of change threshold, or if an image texture anomaly is determined based on the texture complexity index, indicating a change in the scene, then the median value of the scale factor is directly adjusted. As a new scale for scenarios, namely .

[0197] Corresponding to the above method embodiments, the present invention also discloses an image scale reconstruction apparatus.

[0198] See Figure 3 The present invention discloses a schematic diagram of an image scale reconstruction device, which may include:

[0199] The timestamp matching unit 201 is used to perform timestamp matching between the camera and the millimeter-wave radar to obtain radar-image matching frame pairs.

[0200] The radar-image matching frame pair includes: radar point cloud and the corresponding image frame.

[0201] Millimeter-wave radar has advantages such as being unaffected by light, adapting to harsh weather, low power consumption, and low cost, making it more suitable for long-term deployment and use in real-world scenarios.

[0202] This embodiment performs timestamp matching between the camera and the millimeter-wave radar. In fact, it matches the synchronization frames of the camera and the millimeter-wave radar based on their timestamps to ensure temporal consistency between the images acquired by the camera and the point clouds acquired by the millimeter-wave radar, providing the necessary prerequisite for subsequent image spatial mapping and scale factor calculation.

[0203] The depth estimation unit 202 is used to perform monocular depth estimation on the image frame of the radar-image matching frame to obtain the depth map corresponding to the image frame;

[0204] In practical applications, deep neural networks can be used to perform monocular depth estimation on camera image frames to obtain the relative depth of each pixel in the image frame, thereby obtaining the depth map corresponding to the image frame. This provides a basis for constructing a dense depth map from the camera's perspective and serves as the image source for subsequent scale estimation.

[0205] The coordinate system transformation unit 203 is used to transform the radar point cloud corresponding to the radar-image matching frame from the radar coordinate system to the camera coordinate system according to the calibrated extrinsic parameters of the millimeter-wave radar to the camera, so as to obtain the target radar point cloud.

[0206] In practical applications, the intrinsic parameters of the camera and the extrinsic parameters of the millimeter-wave radar can be obtained through calibration. These intrinsic and extrinsic parameters provide the geometric basis for subsequent image and radar data fusion. Both the intrinsic and extrinsic parameters are matrices, with the extrinsic parameters including a rotation matrix R and a translation vector T.

[0207] In this embodiment, based on the extrinsic parameters of millimeter-wave radar to the camera, the three-dimensional points in the radar coordinate system are transformed to the camera coordinate system. The three-dimensional points are the radar point clouds corresponding to the radar-image matching frames.

[0208] The projection unit 204 is used to perform perspective projection on the target radar point cloud transformed into the camera coordinate system according to the calibrated intrinsic parameters of the camera, and to map the target radar point cloud onto the image plane.

[0209] This embodiment performs perspective projection on the target radar point cloud transformed into the camera coordinate system based on the camera's intrinsic parameters, so as to map the target radar point cloud onto the image plane, thereby completing the mapping of three-dimensional spatial points onto the two-dimensional image plane and realizing cross-modal data alignment and fusion.

[0210] The scale factor determination unit 205 is used to perform a ratio calculation between the physical depth value detected by the millimeter-wave radar and the corresponding relative depth value in the depth map for each radar point successfully mapped to the image plane, so as to obtain the scale factor of each radar point.

[0211] The scale factor of radar points is a key parameter used in radar data processing to adjust the scale ratio of point cloud data. Essentially, it's a scaling factor that converts relative depth values ​​in a depth map into actual physical distances. This embodiment utilizes the scale factor to subsequently infer the true size of the image depth map.

[0212] The scale determination unit 206 is used to obtain all scale factors corresponding to the current image frame, and determine the value of the scale factor from all scale factors as the scale reference value of the current image frame.

[0213] In practical applications, all scale factors corresponding to the current image frame can be sorted in descending order. After removing outliers, the median of the scale factors can be determined and used as the scale reference value of the current image frame. This scale reference value is also the representative scale estimate of the current image frame.

[0214] The median scaling factor exhibits strong robustness and resistance to interference, effectively suppressing the impact of local outliers or single-point errors on the overall estimation, and can represent the scaling level of most matched point pairs.

[0215] In summary, this invention discloses an image scale reconstruction device that performs timestamp matching between a camera and a millimeter-wave radar to obtain radar-image matching frame pairs. Monocular depth estimation is performed on the image frames of the radar-image matching frames to obtain a depth map corresponding to the image frames. Based on the extrinsic parameters of the calibrated millimeter-wave radar to the camera, the radar point cloud corresponding to the radar-image matching frame is transformed from the radar coordinate system to the camera coordinate system to obtain the target radar point cloud. Based on the calibrated intrinsic parameters of the camera, perspective projection is performed on the target radar point cloud transformed to the camera coordinate system to map the target radar point cloud onto the image plane. For each radar point successfully mapped to the image plane, the ratio of the physical depth value detected by the millimeter-wave radar to the corresponding relative depth value in the depth map is calculated to obtain the scale factor of each radar point. All scale factors corresponding to the current image frame are obtained, and the median value of the scale factors is determined from all scale factors as the scale reference value of the current image frame. This invention introduces a collaborative approach between millimeter-wave radar and a camera. By calibrating the extrinsic parameters of the millimeter-wave radar and the camera, and synchronizing the timestamps of their data, the physical distance information measured by the radar is mapped onto the image plane. This establishes a stable correspondence between pixel coordinates and physical scale, obtaining highly reliable scale data and enabling the determination of the actual distance between objects in the image and the camera. This invention fully leverages the advantages of millimeter-wave radar—its lack of limitation due to lighting conditions, adaptability to harsh weather, low power consumption, and low cost—making it suitable for long-term deployment in real-world scenarios. It effectively solves the problem of traditional image scale estimation being affected by scene semantic changes and environmental interference, thereby improving the accuracy and robustness of depth estimation systems in complex dynamic scenes. This provides data support for advanced decision-making tasks that rely on precise image scale.

[0216] In one embodiment, the timestamp matching unit 201 can be specifically used for:

[0217] Acquire the current point cloud frame data of the millimeter-wave radar, and extract the radar frame timestamp from the current point cloud frame data;

[0218] Acquire the image frame data continuously output by the camera, and extract the corresponding image frame timestamp from each image frame data;

[0219] The method of minimum absolute time difference matching is adopted to select the image frame with the smallest difference between the image frame timestamp and the radar frame timestamp from all candidate image frames, and the absolute value of the difference is not greater than the maximum tolerance threshold, as the best matching image frame for the current radar frame.

[0220] Based on the current radar frame and its corresponding best matching image frame, a radar-image matching frame pair is obtained.

[0221] In one embodiment, the image scale reconstruction apparatus may further include:

[0222] The mapping unit is used to map the target radar point cloud onto the image plane after the projection unit 204 maps the target radar point cloud onto the image plane to obtain the projection point set.

[0223] The relationship table determination unit is used to establish a one-to-one correspondence between the projection point set and the image pixel coordinates, forming a point cloud projection-image pixel correspondence table.

[0224] In one embodiment, the scaling factor determination unit 205 can specifically be used for:

[0225] From the point cloud projection-image pixel correspondence table, extract the three-dimensional coordinates of each radar point in the camera coordinate system, and extract the depth component from the three-dimensional coordinates as the physical depth value of the point cloud pixel.

[0226] Based on the image pixel positions recorded in the point cloud projection-image pixel correspondence table, the normalized relative depth value of the corresponding point cloud pixel is extracted from the depth map, wherein the relative depth value reflects the relative distance of the point cloud pixel relative to the camera.

[0227] The scale factor of each radar point is obtained by comparing the physical depth value with the corresponding relative depth value.

[0228] In one embodiment, the scale determination unit 206 can specifically be used for:

[0229] Obtain all scale factors corresponding to the current image frame to form a scale value set;

[0230] Outlier detection is performed on the scale value set to identify abnormal scale factors whose distribution deviates significantly from the normal range, and the abnormal scale factors are removed to obtain a cleaned target scale factor set.

[0231] The scale factors in the target scale factor set are sorted in ascending order of their numerical values ​​to form an ordered scale sequence;

[0232] The median of the scale factor is extracted from the ordered scale sequence, and the median of the scale factor is used as the scale reference value of the current image frame.

[0233] In one embodiment, the image scale reconstruction apparatus may further include:

[0234] The rate of change calculation unit is used to calculate the rate of change of the median scale factor relative to the median scale factor of the previous frame.

[0235] The feature extraction unit is used to determine that the median value of the scale factor is abnormal if the scale change rate exceeds the change rate threshold, and to extract the image brightness feature time series from the image frames of the radar-image matching frame;

[0236] A feature anomaly detection unit is used to detect anomalies in the time series of the image brightness features;

[0237] The index construction unit is used to construct a texture complexity index and record it in the texture complexity time series when it is determined that the brightness of the current frame is not abnormal based on the image brightness feature time series.

[0238] The texture anomaly detection unit is used to perform image texture anomaly detection on the texture complexity time series in order to determine the cause of the current frame scale anomaly.

[0239] In one embodiment, the feature extraction unit can specifically be used for:

[0240] The color space of the corresponding image frame in the radar-image matching frame is converted, and the luminance channel image is extracted from the converted color space.

[0241] The average value of the overall brightness of the current image frame is obtained by averaging all pixel values ​​in the brightness channel image.

[0242] Based on the pixel value distribution in the luminance channel, count the number of pixels at all luminance levels and construct a luminance histogram.

[0243] The overall average brightness and the brightness histogram are added to the image brightness feature time series.

[0244] In one embodiment, the feature anomaly detection unit can be specifically used for:

[0245] Extract the mean brightness sequence and brightness histogram sequence of the N nearest frames to the current image frame from the image brightness feature time series;

[0246] Calculate the sequence mean and sequence standard deviation of the brightness mean sequence, and based on the sequence standard deviation, standardize the difference between the current frame brightness mean and the sequence mean to obtain the standardized difference.

[0247] Calculate the Kullback-Leibler divergence between the current frame's luminance histogram and the average historical histogram of the nearest N-1 frames in the luminance histogram sequence.

[0248] The standardized difference and the Kullback-Leibler divergence are compared with their respective thresholds, and the anomaly of the image brightness feature time series is determined based on the comparison results.

[0249] In one embodiment, the indicator building unit can be specifically used for:

[0250] Convert the image frames of the radar-image matching frames into grayscale images;

[0251] Determine the horizontal and vertical gradients of the grayscale image, and calculate the gradient magnitude map based on the horizontal and vertical gradients;

[0252] The texture complexity index of the image frame is obtained by averaging the values ​​of all pixels in the gradient magnitude map.

[0253] In one embodiment, the image scale reconstruction apparatus may further include:

[0254] The first update unit is used to update the global scale factor by using an exponential moving average method if the scale change rate does not exceed the change rate threshold, or if the image texture is determined to be normal according to the texture complexity index and the scene is determined to be unchanged.

[0255] The second update unit is used to determine that if the scale change rate exceeds the change rate threshold, or if the image texture is abnormal according to the texture complexity index, and the scene changes, the median value of the scale factor of the current image frame is directly used as the new scale of the scene to complete the size factor update.

[0256] It should be noted that for the specific working principles of each component in the device embodiment, please refer to the corresponding section of the method embodiment, which will not be repeated here.

[0257] Corresponding to the above embodiments, the present invention also discloses a computer storage medium that stores at least one instruction, which, when executed by a processor, implements the steps shown in the embodiments of the image scale reconstruction method.

[0258] Computer storage media can be tangible media that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. Computer storage media can be machine-readable signal media or machine-readable storage media. Computer storage media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0259] Corresponding to the above embodiments, such as Figure 4As shown, the present invention also provides a schematic diagram of the structure of an electronic device, which may include: a processor 1 and a memory 2;

[0260] The processor 1 and memory 2 communicate with each other via communication bus 3.

[0261] Processor 1, for executing at least one instruction;

[0262] Memory 2 is used to store at least one instruction;

[0263] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0264] Memory 2 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0265] The processor executes at least one instruction to implement the steps shown in the embodiment of the image scale reconstruction method.

[0266] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0267] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0268] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image scale reconstruction method, characterized in that, include: Timestamp matching is performed between the camera and the millimeter-wave radar to obtain radar-image matching frame pairs; Monocular depth estimation is performed on the image frames of the radar-image matching frame to obtain the depth map corresponding to the image frame; Based on the calibrated extrinsic parameters of the millimeter-wave radar to the camera, the radar point cloud corresponding to the radar-image matching frame is transformed from the radar coordinate system to the camera coordinate system to obtain the target radar point cloud; Based on the calibrated intrinsic parameters of the camera, perspective projection is performed on the target radar point cloud transformed into the camera coordinate system, and the target radar point cloud is mapped onto the image plane. For each radar point successfully mapped to the image plane, the physical depth value detected by the millimeter-wave radar is compared with the corresponding relative depth value in the depth map to obtain the scale factor of each radar point. Obtain all scale factors corresponding to the current image frame, and determine the value of the scale factor from all scale factors as the scale reference value of the current image frame.

2. The image scale reconstruction method according to claim 1, characterized in that, The process of time-stamp matching between the camera and millimeter-wave radar to obtain radar-image matching frame pairs includes: Acquire the current point cloud frame data of the millimeter-wave radar, and extract the radar frame timestamp from the current point cloud frame data; Acquire the image frame data continuously output by the camera, and extract the corresponding image frame timestamp from each image frame data; The method of minimum absolute time difference matching is adopted to select the image frame with the smallest difference between the image frame timestamp and the radar frame timestamp from all candidate image frames, and the absolute value of the difference is not greater than the maximum tolerance threshold, as the best matching image frame for the current radar frame. Based on the current radar frame and its corresponding best matching image frame, a radar-image matching frame pair is obtained.

3. The image scale reconstruction method according to claim 1 or 2, characterized in that, After performing perspective projection on the target radar point cloud transformed into the camera coordinate system according to the calibrated intrinsic parameters of the camera, and mapping the target radar point cloud onto the image plane, the method further includes: The target radar point cloud is mapped onto the image plane to obtain the projection point set; A one-to-one correspondence is established between the projection point set and the image pixel coordinates to form a point cloud projection-image pixel correspondence table.

4. The image scale reconstruction method according to claim 3, characterized in that, For each radar point successfully mapped to the image plane, the physical depth value detected by the millimeter-wave radar is compared with the corresponding relative depth value in the depth map to obtain the scale factor for each radar point, including: From the point cloud projection-image pixel correspondence table, extract the three-dimensional coordinates of each radar point in the camera coordinate system, and extract the depth component from the three-dimensional coordinates as the physical depth value of the point cloud pixel. Based on the image pixel positions recorded in the point cloud projection-image pixel correspondence table, the normalized relative depth value of the corresponding point cloud pixel is extracted from the depth map, wherein the relative depth value reflects the relative distance of the point cloud pixel relative to the camera. The scale factor of each radar point is obtained by comparing the physical depth value with the corresponding relative depth value.

5. The image scale reconstruction method according to claim 1, characterized in that, The step of obtaining all scale factors corresponding to the current image frame and determining the value of the scale factor from all scale factors as the scale reference value of the current image frame includes: Obtain all scale factors corresponding to the current image frame to form a scale value set; Outlier detection is performed on the scale value set to identify abnormal scale factors whose distribution deviates significantly from the normal range, and the abnormal scale factors are removed to obtain a cleaned target scale factor set. The scale factors in the target scale factor set are sorted in ascending order of their numerical values ​​to form an ordered scale sequence; The median of the scale factor is extracted from the ordered scale sequence, and the median of the scale factor is used as the scale reference value of the current image frame.

6. The image scale reconstruction method according to claim 1, characterized in that, Also includes: Calculate the rate of change of the median scale factor relative to the median scale factor of the previous frame; If the scale change rate exceeds the change rate threshold, it is determined that the median value of the scale factor is abnormal, and the image brightness feature time series is extracted from the image frames of the radar-image matching frame; Anomaly detection is performed on the time series of the image brightness features; When it is determined from the image brightness feature time series that the brightness of the current frame is not abnormal, a texture complexity index is constructed and recorded in the texture complexity time series; Image texture anomaly detection is performed on the texture complexity time series to determine the cause of the current frame scale anomaly.

7. The image scale reconstruction method according to claim 6, characterized in that, The step of extracting the image brightness feature time series from the corresponding image frame in the radar-image matching frame includes: The color space of the corresponding image frame in the radar-image matching frame is converted, and the luminance channel image is extracted from the converted color space. The average value of all pixel values ​​in the brightness channel image is obtained by averaging the values ​​of the current image frame. Based on the pixel value distribution in the luminance channel, count the number of pixels at all luminance levels and construct a luminance histogram. The overall average brightness and the brightness histogram are added to the image brightness feature time series.

8. The image scale reconstruction method according to claim 6 or 7, characterized in that, Anomaly detection is performed on the time series of the image brightness features, including: Extract the mean brightness sequence and brightness histogram sequence of the N nearest frames to the current image frame from the image brightness feature time series; Calculate the sequence mean and sequence standard deviation of the brightness mean sequence, and based on the sequence standard deviation, standardize the difference between the current frame brightness mean and the sequence mean to obtain the standardized difference. Calculate the Kullback-Leibler divergence between the current frame luminance histogram and the average historical histogram of the nearest N-1 frames in the luminance histogram sequence. The standardized difference and the Kullback-Leibler divergence are compared with their respective thresholds, and the anomaly of the image brightness feature time series is determined based on the comparison results.

9. The image scale reconstruction method according to claim 6 or 7, characterized in that, The texture construction complexity metrics include: Convert the image frames of the radar-image matching frames into grayscale images; Determine the horizontal and vertical gradients of the grayscale image, and calculate the gradient magnitude map based on the horizontal and vertical gradients; The texture complexity index of the image frame is obtained by averaging the values ​​of all pixels in the gradient magnitude map.

10. The image scale reconstruction method according to claim 6, characterized in that, Also includes: If the scale change rate does not exceed the change rate threshold, or if the image texture is determined to be normal according to the texture complexity index, the scene is determined to be unchanged, and the global scale factor is updated using an exponential moving average method. If the scale change rate exceeds the change rate threshold, or if an image texture anomaly is determined based on the texture complexity index, and the scene is determined to have changed, the median value of the scale factor of the current image frame is directly used as the new scene scale to complete the size factor update.

11. An image scale reconstruction apparatus, characterized in that, include: The timestamp matching unit is used to perform timestamp matching between the camera and the millimeter-wave radar to obtain radar-image matching frame pairs. A depth estimation unit is used to perform monocular depth estimation on the image frame of the radar-image matching frame to obtain a depth map corresponding to the image frame. The coordinate system transformation unit is used to transform the radar point cloud corresponding to the radar-image matching frame from the radar coordinate system to the camera coordinate system according to the calibrated extrinsic parameters of the millimeter-wave radar to the camera, so as to obtain the target radar point cloud. The projection unit is used to perform perspective projection on the target radar point cloud transformed into the camera coordinate system according to the calibrated intrinsic parameters of the camera, and to map the target radar point cloud onto the image plane. The scale factor determination unit is used to perform a ratio calculation between the physical depth value detected by the millimeter-wave radar and the corresponding relative depth value in the depth map for each radar point successfully mapped to the image plane, so as to obtain the scale factor of each radar point. The scale determination unit is used to obtain all scale factors corresponding to the current image frame, and determine the value of the scale factor from all scale factors as the scale reference value of the current image frame.

12. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction, which, when executed by a processor, implements the image scale reconstruction method as described in any one of claims 1 to 10.

13. An electronic device, characterized in that, The electronic device includes: a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the image scale reconstruction method as described in any one of claims 1 to 10.

Images