Method and system for optimizing single image ranging accuracy based on multi-sensor fusion

Through the multi-sensor fusion optimization single-picture ranging method, the registration and fusion of multi-sensor data and image information is used to solve the problem of insufficient accuracy of single-picture ranging in complex environments, and achieve higher ranging accuracy and robustness .

CN118642096BActive Publication Date: 2024-11-19JIANGSU HAOHAN INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411124831.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-19
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

The existing single-picture ranging method is difficult to achieve high-precision ranging in complex environments, due to factors such as ambient lighting changes, camera internal parameter error, image noise and occlusion.

Method used

采用多传感器融合优化单图测距方法,通过获取目标图像和传感记录,进行时序特征分析和配准融合处理,提取像素点特征信息,结合传感数据中的距离信息,利用多源信息的 Complementarity improves ranging accuracy and robustness.

Benefits of technology

It achieves improving the accuracy and robustness of ranging in complex environments, and can better cope with the ranging requirements under complex environment conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118642096B_ABST
    Figure CN118642096B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for optimizing the ranging accuracy of a single image based on multi-sensor fusion, and relates to the field of ranging and positioning. The method comprises: obtaining a target image; obtaining a target sensing record, which is a record of dynamic sensing monitoring of a target space corresponding to the target image by a multi-sensor group; performing time series feature analysis on the target sensing record to obtain target sensing information at the target shooting time; calling a registration fusion model to perform intelligent registration fusion processing on the target sensing information and the target image to obtain a target fusion result; obtaining target pixel feature information of the target object in the target fusion result, and obtaining the target distance according to the target pixel feature information. The method solves the technical problem that the existing single-image ranging is difficult to cope with complex environments, resulting in insufficient ranging accuracy, and achieves the technical effect of improving the accuracy and robustness of ranging, and better coping with ranging requirements under complex environmental conditions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of ranging and positioning, and in particular to a method and system for optimizing the ranging accuracy of a single image based on multi-sensor fusion. Background Art

[0002] In the field of modern computer vision and perception, accurately measuring the distance of target objects in a scene is a core requirement for many applications (such as autonomous driving, robot navigation, drone obstacle avoidance, augmented reality, etc.). Although traditional single-image ranging methods, such as stereo vision based on image processing and monocular vision ranging, can achieve distance measurement to a certain extent, they are often difficult to achieve the requirements of high-precision applications due to factors such as ambient lighting changes, camera intrinsic parameter errors, image noise and occlusion.

[0003] Among the current related technologies, single-image ranging has the technical problem of being difficult to cope with complex environments, resulting in insufficient ranging accuracy. Summary of the invention

[0004] The present application provides a method and system for optimizing the ranging accuracy of a single image based on multi-sensor fusion, which obtains target images and sensor records, extracts information matching the target shooting time from the dynamic sensor records, uses a registration fusion model to align and fuse the sensor information with the image information, extracts pixel features of the target object from the fusion result, combines the distance information in the sensor data, calculates the target distance, and other technical means, and utilizes the complementarity of multi-source information to achieve the technical effect of improving the accuracy and robustness of ranging and better responding to ranging needs under complex environmental conditions.

[0005] This application provides a method for optimizing single image ranging accuracy based on multi-sensor fusion, including:

[0006] A target image is acquired, wherein the target image has an identification of the target shooting time; a target sensing record is acquired, wherein the target sensing record refers to a record of dynamic sensing monitoring of a target space corresponding to the target image by a multi-sensor group; a time series feature analysis is performed on the target sensing record to obtain target sensing information at the target shooting time; a registration and fusion model is called to perform intelligent registration and fusion processing on the target sensing information and the target image to obtain a target fusion result; target pixel feature information of the target object is acquired in the target fusion result, and a target distance is obtained according to the target pixel feature information.

[0007] In a possible implementation, the following processing is performed:

[0008] The multi-sensor group includes a temperature sensor, a humidity sensor and a wind speed sensor; the target environment temperature dynamically monitored by the temperature sensor, the target environment humidity dynamically monitored by the humidity sensor and the target environment wind speed dynamically monitored by the wind speed sensor are traversed and matched in the target sensing record to form target environment information; a predetermined camera feature is read, and target camera information of a target camera is acquired based on the predetermined camera feature, wherein the target camera refers to a camera that captures the target image; an error correction analysis function is introduced, and the error analysis of the target distance is performed in combination with the target environment information and the target camera information to obtain a target correction distance.

[0009] In a possible implementation, the following processing is performed:

[0010] The predetermined camera features include the target focal length of the target camera and the target comprehensive distortion index of the target lens. The expression of the error correction analysis function is as follows:

[0011] ;in, is the target correction distance, is the target distance, It refers to the environmental correction coefficient of the target environmental information to the target distance, refers to the i-th environmental characteristic parameter in the target environmental information, refers to the i-th coefficient of variation corresponding to the i-th environmental characteristic parameter, It refers to the camera correction coefficient of the characteristic of the target camera to the target distance, refers to the jth feature of the target camera, It refers to the correlation index between the j-th feature and the target distance, and are the first coefficient and the second coefficient respectively, and , and They refer to the environmental error coefficient and the camera error coefficient respectively.

[0012] In a possible implementation, the following processing is performed:

[0013] The multi-sensor group includes a laser radar and a millimeter-wave radar; the laser point cloud time sequence of the laser radar is obtained in the target sensing record, and the first point cloud data group at the first time in the laser point cloud time sequence is extracted; the first time difference between the first time and the target shooting time is normalized to obtain a first interpolation coefficient; the first point cloud data group is weighted calculated based on the first interpolation coefficient to obtain a target interpolation point cloud data group at the target shooting time; the target interpolation millimeter wave information of the millimeter wave radar at the target shooting time is obtained; the target interpolation point cloud data group and the target interpolation millimeter wave information constitute the target sensing information.

[0014] In a possible implementation, the following processing is performed:

[0015] The target interpolation point cloud data group is subjected to registration and fusion analysis through the registration and fusion model to obtain a target point cloud model; the target interpolation millimeter wave information is rendered to the target point cloud model to obtain a target initial model of the target space; the target image is mapped to the target initial model to obtain a target three-dimensional model, and the target three-dimensional model is used as the target fusion result.

[0016] In a possible implementation, the following processing is performed:

[0017] A first point cloud sample is randomly extracted from the target interpolation point cloud data group; the first point cloud sample is analyzed by the registration fusion model to obtain a first model parameter of a first point cloud model; the first point cloud sample is removed from the target interpolation point cloud data group to obtain a first test sample, wherein the first test sample includes a plurality of test point clouds; a first distance from a first test point cloud in the plurality of test point clouds to the first point cloud model is obtained; the first distance is analyzed to obtain a first test result, and the target point cloud model is determined according to the first test result.

[0018] In a possible implementation, the following processing is performed:

[0019] According to the first inspection result, if the first distance meets the predetermined distance threshold, the first inspection point cloud is added to the consistency list; if the first distance does not meet the predetermined distance threshold, the first inspection point cloud is added to the inconsistency list; the ratio of the number of point clouds in the consistency list to the number of point clouds in the inconsistency list is recorded as the first inspection result.

[0020] The present application also provides a system for optimizing the ranging accuracy of a single image based on multi-sensor fusion, including:

[0021] A target image acquisition module, the target image acquisition module is used to acquire a target image, the target image has an identification of the target shooting time; a target sensor record acquisition module, the target sensor record acquisition module is used to acquire a target sensor record, the target sensor record refers to a record of dynamic sensor monitoring of the target space corresponding to the target image through a multi-sensor group; a timing feature analysis module, the timing feature analysis module is used to perform timing feature analysis on the target sensor record to obtain the target sensor information at the target shooting time; an intelligent registration and fusion processing module, the intelligent registration and fusion processing module is used to call the registration and fusion model to perform intelligent registration and fusion processing on the target sensor information and the target image to obtain a target fusion result; a target distance acquisition module, the target distance acquisition module is used to obtain target pixel feature information of the target object in the target fusion result, and obtain the target distance according to the target pixel feature information.

[0022] The method and system for optimizing the ranging accuracy of a single image based on multi-sensor fusion proposed in this application first obtain a target image, which has an identification of the target shooting time, and then obtain a target sensing record, which refers to a record of dynamic sensing monitoring of the target space corresponding to the target image by a multi-sensor group, and then perform a time series feature analysis on the target sensing record to obtain the target sensing information at the target shooting time, and then call the registration fusion model to perform intelligent registration and fusion processing on the target sensing information and the target image to obtain a target fusion result, and finally obtain the target pixel feature information of the target object in the target fusion result, and obtain the target distance based on the target pixel feature information, thereby achieving the technical effect of improving the accuracy and robustness of ranging and better coping with ranging requirements under complex environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the accompanying drawings of the embodiment of the present invention will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0024] Figure 1 A schematic diagram of the flow of a method for optimizing single-image ranging accuracy based on multi-sensor fusion provided in an embodiment of the present application.

[0025] Figure 2 A schematic diagram of the structure of a system for optimizing single-image ranging accuracy based on multi-sensor fusion provided in an embodiment of the present application.

[0026] Explanation of reference numerals: target image acquisition module 10 , target sensor record acquisition module 20 , time series feature analysis module 30 , intelligent registration and fusion processing module 40 , target distance acquisition module 50 . DETAILED DESCRIPTION

[0027] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0029] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0030] The present application embodiment provides a method for optimizing the ranging accuracy of a single image based on multi-sensor fusion, such as Figure 1 As shown, the method includes:

[0031] Step S100, obtaining a target image, wherein the target image has an identification of a target shooting time. Specifically, a camera or other image acquisition device is used to aim at the target scene for shooting, and the camera settings (such as resolution, exposure time, focal length, etc.) are ensured to be suitable for the current environmental conditions to obtain a clear image. At the moment the image is captured, the current timestamp is obtained through the camera's built-in clock or an external time server synchronized therewith, and the image with the timestamp is stored in a designated database or file system. Among them, the target image refers to a specific image that needs to be processed for distance measurement, including one or more target objects that need to measure the distance; the target shooting time refers to the specific time point when the target image is captured.

[0032] Step S200, obtaining a target sensing record, wherein the target sensing record refers to a record of dynamic sensing monitoring of the target space corresponding to the target image by a multi-sensor group. Specifically, the multi-sensor group is a set of multiple sensors of different types, which are used to obtain information of the target space from different angles or dimensions. These sensors include laser radar, millimeter wave radar, etc. The multi-sensor group is connected to the data acquisition system, receives a synchronization signal (such as an image shooting timestamp) from the image capture system, or triggers the monitoring cycle of the multi-sensor group according to a preset schedule. Each sensor in the multi-sensor group works independently according to its characteristics to perform dynamic sensing monitoring on the target space. The dynamic sensing monitoring refers to continuous or periodic measurement of physical quantities (such as distance, speed, acceleration, temperature, etc.) in the target space to obtain real-time information about the change of the state of the target space. The multi-sensor group transmits the monitored data to the data acquisition system in real time, and the data acquisition system records it in chronological order, and when recording the data, adds a timestamp to each data point (so as to match the timestamp of the target image) to form a target sensing record, and stores the recorded sensing data in a database or file system that is the same as or associated with the image.

[0033] Step S300, performing time series feature analysis on the target sensing record to obtain the target sensing information at the target shooting time. Specifically, the acquired target sensing record is preliminarily processed, including data cleaning (removing noise, outliers, etc.), data format unification (ensuring comparability between different sensor data) and data alignment (sorting the sensing data according to the timestamp). Using the timestamp of the target image as a reference, each data point in the target sensing record is matched with its closest timestamp to find the sensing data closest to the target shooting time. In the sensing data after time synchronization, features related to target ranging are extracted according to the needs of the application scenario. These features include distance reading, speed change, acceleration, etc., depending on the type of sensor used and the ranging principle. The extracted features are further analyzed using methods such as time series analysis, signal processing or pattern recognition to identify patterns or trends related to the target shooting time. According to the results of the time series feature analysis, the target sensing information at the target shooting time is determined, including parameters such as the distance, position, and speed of the target object at a specific time point.

[0034] In a possible implementation, step S300 further includes step S310, wherein the multi-sensor group includes a laser radar and a millimeter wave radar. Specifically, the laser radar is a sensor that measures distance by emitting laser pulses and measuring their reflection time, and can generate accurate three-dimensional point cloud data; the millimeter wave radar uses the millimeter wave frequency band for detection and has the ability to penetrate bad weather such as rain and fog. Step S320, obtain the laser point cloud time sequence of the laser radar in the target sensing record, and extract the first point cloud data group at the first time in the laser point cloud time sequence. Specifically, read the laser point cloud time sequence data from the sensing record of the laser radar, and the laser point cloud time sequence is a three-dimensional point cloud data sequence generated by the laser radar over time. Find the first time closest to the target shooting time according to the timestamp, that is, the first time is a time point in the laser radar record that is closest to the target shooting time, and extract the first point cloud data group at the first time (the three-dimensional point cloud data set captured by the laser radar at the first time).

[0035] Step S330, normalize the first time difference between the first time and the target shooting time to obtain a first interpolation coefficient. Specifically, the time difference between the first time and the target shooting time is calculated, and the time difference is normalized to a certain time unit (such as milliseconds, seconds, etc.) to obtain the first interpolation coefficient, that is, the data is scaled so that it falls into a small specific interval. The obtained first interpolation coefficient is a weight factor used for interpolation calculation, which reflects the degree of influence of data at different time points on the interpolation result. Step S340, weighted calculation is performed on the first point cloud data group based on the first interpolation coefficient to obtain the target interpolation point cloud data group at the target shooting time. Specifically, the first interpolation coefficient is used to perform weighted processing on each point in the first point cloud data group, and the point cloud data set at the target shooting time is calculated by weighted averaging or other interpolation methods, that is, the target interpolation point cloud data group. Step S350, obtain the target interpolation millimeter wave information of the millimeter wave radar at the target shooting time. Specifically, the millimeter wave information at the target shooting time is obtained by interpolation from the sensing record of the millimeter wave radar using the same processing method as the laser radar, and the target interpolation millimeter wave information is obtained, including the distance, speed and other information detected by the millimeter wave radar. Step S360, the target interpolation point cloud data group and the target interpolation millimeter wave information constitute the target sensing information. Specifically, the target interpolation point cloud data group and the target interpolation millimeter wave information are merged to form complete target sensing information, which is used for registration and fusion processing with the target image. Laser radar and millimeter wave radar each have their own advantages and disadvantages. Laser radar provides high-precision three-dimensional point cloud data, but is affected by factors such as ambient light and smoke; millimeter wave radar has the characteristics of strong penetration and all-weather operation, but its ranging accuracy and resolution are relatively low. This implementation method aligns the data of the two in time through interpolation processing and fuses them into unified target sensing information, making full use of their respective advantages, providing more accurate and reliable data support for image registration and ranging, and achieving the technical effect of improving ranging accuracy and robustness.

[0036] Step S400, calling the registration fusion model to perform intelligent registration and fusion processing on the target sensor information and the target image to obtain a target fusion result. Specifically, a pre-trained registration fusion model is loaded, and the registration fusion model is used to align and fuse data from different sources (such as sensors and cameras), including two parts: a registration algorithm and a fusion algorithm. The formats of the target sensor information and the target image are converted into a unified format that can be processed by the registration fusion model to meet the input requirements of the model. Key feature information related to the target image, such as depth information, temperature, light intensity, etc., is extracted from the target sensor information, and visual features, such as edges, textures, colors, etc., are extracted from the target image. The sensor data is aligned with the spatial position in the image using spatial transformations (such as affine transformations, perspective transformations, etc.), and the precise correspondence between the sensor data and the image frame is ensured through temporal registration. The corresponding relationship between the sensor data and the image is found through feature descriptors (such as SIFT, SURF, etc.) or deep learning features. The registered sensor data is fused with the target image at the pixel level or feature level, including weighted average, Bayesian fusion, deep learning fusion, and other methods. Use sensor data to enhance the information in the image, such as improving the three-dimensional perception of the image through depth information, or adjusting the brightness contrast of the image through illumination data. Evaluate the quality of the fusion result, check whether there are obvious artifacts or distortions, adjust the parameters of the registration and fusion algorithms based on the evaluation results to optimize the results, generate and output the fused image or a data set containing the fusion information, and obtain the target fusion result.

[0037] In a possible implementation, step S400 further includes step S410, performing registration fusion analysis on the target interpolation point cloud data group through the registration fusion model to obtain a target point cloud model. Specifically, the registration algorithm is determined according to the characteristics and requirements of the point cloud data. For point cloud data from the same time but different angles or different sensors, ICP (iterative closest point) or its variants can be selected, and the selected registration algorithm is applied to align each point cloud in the target interpolation point cloud data group, that is, to find the optimal transformation matrix (such as rotation and translation) to minimize the error between different point clouds. The registered point clouds are merged into a unified point cloud model, including removing duplicate points, filling holes, and smoothing, etc., to generate a complete and continuous target point cloud model. Step S420, the target interpolation millimeter wave information is rendered to the target point cloud model to obtain the target initial model of the target space. Specifically, the coordinate system of the millimeter wave data and the coordinate system of the point cloud model are converted to map the key information (such as distance information) in the millimeter wave data to the corresponding position of the target point cloud model. The millimeter wave information is presented on the point cloud model in an appropriate manner (such as color coding, height coding, etc.) by using visualization technology to enhance the description capability of the target initial model on the physical characteristics of the target space. Step S430, the target image is mapped to the target initial model to obtain a target three-dimensional model, and the target three-dimensional model is used as the target fusion result. Specifically, necessary preprocessing operations are performed on the target image, such as correcting distortion, adjusting color balance, etc., to ensure image quality. Through feature matching or known spatial correspondence, the perspective transformation or projection transformation of the image is used to align the target image with the target initial model (the point cloud model with rendered millimeter wave information). By calculating the texture coordinates of each model vertex and sampling the corresponding image pixels, the pixel information of the target image is mapped to the surface of the target initial model, and real texture details are added to the model. After texture mapping, the target initial model is transformed into a target three-dimensional model with rich visual information. This model combines the three-dimensional geometric information of the laser point cloud, the physical quantity information of the millimeter wave, and the visual information of the image. This implementation method performs image mapping after the geometric structure has been stabilized and the key physical quantity information has been rendered, avoiding the mismatch problem caused by structural changes and achieving the technical effect of improving the accuracy of building the target three-dimensional model.

[0038] In a possible implementation, step S410 further includes step S411, randomly extracting the first point cloud sample from the target interpolation point cloud data set. Specifically, randomly selecting a point cloud data from the target interpolation point cloud data set as the initial processing object, that is, the first point cloud sample. Step S412, analyzing the first point cloud sample through the registration fusion model to obtain the first model parameters of the first point cloud model. Specifically, the registration fusion model receives the first point cloud sample as input, and calculates the transformation relationship between the point clouds in the target interpolation point cloud data set through a series of algorithms (such as iterative closest point algorithm ICP, feature matching, etc.) to achieve registration. The first model parameters are the parameters of the first point cloud model calculated during the registration and fusion process, and define the properties of the first point cloud model in space, such as position, direction, and scale. Step S413, removing the first point cloud sample from the target interpolation point cloud data set to obtain a first test sample, and the first test sample includes multiple test point clouds. Specifically, the first point cloud sample that has been used to construct the first point cloud model is removed from the original target interpolation point cloud data group to avoid bias in the verification process. The remaining point cloud data constitutes a first test sample, which is used to verify the accuracy and completeness of the first point cloud model. Step S414, obtain a first distance from the first test point cloud in the multiple test point clouds to the first point cloud model. Specifically, a point cloud data randomly selected from the multiple test point clouds is used as the first test point cloud. The shortest distance from each point in the first test point cloud to the first point cloud model is calculated to obtain a first distance, and the first distance reflects the degree of matching between the first test point cloud and the first point cloud model. Step S415, analyze the first distance to obtain a first test result, and determine the target point cloud model according to the first test result. Specifically, according to the first distance, the similarity or difference between the first test point cloud and the first point cloud model is evaluated through statistical analysis (such as calculating the average distance, standard deviation, etc.) and threshold comparison. The first test result is obtained by analysis, and the first test result shows to what extent the first point cloud model can represent or match the test point cloud. If the first inspection result meets the preset standard (such as the average distance is less than a certain threshold), the first point cloud model is considered to be a preliminary version of the target point cloud model. If not, repeat the above process until a satisfactory target point cloud model is obtained. This implementation method is based on the idea of ​​iterative optimization and verification. By randomly extracting and eliminating point cloud samples, the accuracy and completeness of the point cloud model are gradually constructed and verified. The application of the registration fusion model ensures that point cloud data from different sources can be fused into a unified three-dimensional model. The introduction of inspection samples and distance analysis provide real-time feedback on the quality of the model, ensuring that the final target point cloud model is both accurate and reliable. This iterative construction and verification process reduces errors and deviations, and achieves the technical effect of improving the accuracy and stability of the target point cloud model.

[0039] In a possible implementation, step S415 further includes step S4151, according to the first inspection result, if the first distance meets the predetermined distance threshold, then the first inspection point cloud is added to the consistency list, and if the first distance does not meet the predetermined distance threshold, then the first inspection point cloud is added to the inconsistency list. Specifically, the calculated first distance is compared with the predetermined distance threshold, and the distance threshold is a preset value representing the maximum acceptable deviation between the first inspection point cloud and the first point cloud model. If the predetermined distance threshold is met, it means that the matching degree between the first inspection point cloud and the first point cloud model is high, that is, the distances of most points in the first inspection point cloud to the first point cloud model are within an acceptable range. Therefore, the first inspection point cloud is added to the consistency list, which is used to collect point cloud data that is highly consistent with the first point cloud model. If the predetermined distance threshold is not met, it indicates that there is a large mismatch between the first inspection point cloud and the first point cloud model. Therefore, the first inspection point cloud is added to the inconsistency list, which is used to collect point cloud data that is inconsistent with the first point cloud model or has significant deviations. Step S4152, record the ratio of the number of point clouds in the consistency list to the number of point clouds in the inconsistency list as the first inspection result. Specifically, count the number of point clouds in the consistency list and the inconsistency list respectively. Divide the number of point clouds in the consistency list by the total number of point clouds (i.e., the sum of the number of point clouds in the consistency list and the inconsistency list) to obtain a ratio, which reflects the overall consistency between the first point cloud model and the inspection point cloud data. This implementation method considers the accuracy and coverage of the first point cloud model by classifying the inspection point clouds into the consistency list and the inconsistency list. The large number of point clouds in the consistency list indicates that the first point cloud model can well represent these point cloud data. The small number of point clouds in the inconsistency list indicates that the first point cloud model is accurate in most cases. Representing this relationship in the form of a ratio provides a quantitative indicator for measuring the overall consistency and reliability of the first point cloud model, thereby achieving the technical effect of comprehensively evaluating the quality of the first point cloud model.

[0040] Step S500, obtain the target pixel feature information of the target object in the target fusion result, and obtain the target distance according to the target pixel feature information. Specifically, image processing technology, such as edge detection, corner detection, texture analysis, etc., is used to extract features from the target fusion result (i.e., the image after multi-sensor information and image registration fusion processing) to identify the significant features of the target object. On the basis of feature extraction, the target object is identified by matching the extracted features with a known target template, or by using a classifier for classification, that is, the object whose distance needs to be measured is identified from the image or data.

[0041] Once the target object is identified, the specific pixel or pixel area of ​​the target object in the fused image is located by calculating the centroid of the target object's outline, the center point of the bounding box, etc. According to the pixel feature information of the target object (such as pixel position, size, etc.) and the camera's intrinsic parameters (internal parameters of the camera, including focal length, optical center position, distortion coefficient, etc., which determine how the camera maps the three-dimensional world to a two-dimensional image) and extrinsic parameters (external parameters of the camera, which describe the position and posture of the camera in the world coordinate system), the distance of the target object is calculated using the principle of geometry or triangulation. The embodiment of the present application adopts the method of acquiring the target image and sensor records, extracting information matching the target shooting time from the dynamic sensor records, aligning and fusing the sensor information with the image information using the registration fusion model, extracting the pixel features of the target object from the fusion result, combining the distance information in the sensor data, calculating the target distance and other technical means, and utilizing the complementarity of multi-source information to achieve the technical effect of improving the accuracy and robustness of ranging and better coping with the ranging requirements under complex environmental conditions.

[0042] In a possible implementation, step S500 further includes step S510, wherein the multi-sensor group includes a temperature sensor, a humidity sensor, and a wind speed sensor, and the target environment temperature dynamically monitored by the temperature sensor, the target environment humidity dynamically monitored by the humidity sensor, and the target environment wind speed dynamically monitored by the wind speed sensor are traversed and matched in the target sensing record, and the target environment information is formed. Specifically, a database or file storing target sensing records is accessed, and these records contain data dynamically monitored by the multi-sensor group (including temperature sensors, humidity sensors, and wind speed sensors). Through timestamp matching, the temperature, humidity, and wind speed data that match the target shooting time are filtered out from these records, and the matched temperature, humidity, and wind speed data are combined into a target environment information set. Since changes in environmental conditions will affect the imaging quality and ranging accuracy of the camera, the target environment information is used for error correction analysis. Step S520, read a predetermined camera feature, and collect the target camera information of the target camera based on the predetermined camera feature, wherein the target camera refers to the camera that shoots the target image. Specifically, the predetermined camera features are some inherent properties that have been determined during the design and manufacturing process of the camera, such as focal length, aperture size, etc. Access a database or configuration file storing camera features, retrieve specific information of the camera from the database according to the model or identifier of the target camera, and obtain specific parameters and features of the camera used when shooting the target image. Since different features and parameters of the camera will directly affect its imaging effect and ranging accuracy, the target camera information is also used for error correction calculation. Step S530, introduce an error correction analysis function, and combine the target environment information with the target camera information to perform error analysis of the target distance to obtain the target correction distance. Specifically, according to the known influence of camera characteristics and environmental conditions on ranging errors, define an error correction analysis function, take the target environment information obtained in step S510 and the target camera information obtained in step S520 as input, pass them to the error correction analysis function, and calculate the ranging error caused by environmental conditions and camera characteristics by running the error correction analysis function. Finally, correct the original ranging result (target distance) according to the error analysis result to obtain a more accurate target correction distance. Due to the influence of environmental conditions and camera characteristics, the distance calculated directly from image information often has certain errors. This implementation method introduces an error correction analysis function and combines the actual environmental information and camera information to more accurately evaluate and correct these errors, thereby achieving the technical effect of improving the accuracy and reliability of ranging.

[0043] In a possible implementation, step S530 further includes step S531, the predetermined camera feature includes a target focal length of the target camera and a target comprehensive distortion index of the target lens, and the expression of the error correction analysis function is as follows:

[0044] ;

[0045] in, is the target correction distance, is the target distance, It refers to the environmental correction coefficient of the target environmental information to the target distance, refers to the i-th environmental characteristic parameter in the target environmental information, refers to the i-th coefficient of variation corresponding to the i-th environmental characteristic parameter, It refers to the camera correction coefficient of the characteristic of the target camera to the target distance, refers to the jth feature of the target camera, It refers to the correlation index between the j-th feature and the target distance, and are the first coefficient and the second coefficient respectively, and , and They refer to the environmental error coefficient and the camera error coefficient respectively.

[0046] Specifically, the target focal length is the focal length value set for the target camera lens, which affects the magnification and depth of field of the image. The target comprehensive distortion index is a comprehensive indicator of the degree of image distortion at a specific focal length, which integrates the distortion of multiple distortion types (such as barrel distortion, pincushion distortion, etc.) and multiple regions (such as center, edge). The target environmental information includes three characteristics: temperature, humidity and wind speed, and the target camera includes two characteristics: target focal length and target comprehensive distortion index. The coefficient of variation is an indicator that quantifies the degree of fluctuation of environmental characteristic parameters. The environmental correction coefficient reflects the overall impact of the environment on image quality. The camera correction coefficient reflects the contribution of camera characteristics to image errors. The correlation index indicates the strength of the relationship between camera characteristics and target distance. Substitute all the above parameters and coefficients into the error correction analysis function to calculate the target correction distance under given environmental and camera conditions. This distance integrates all known error sources and provides a more accurate distance estimate. This implementation achieves the technical effect of improving the accuracy of distance estimation by constructing an error correction analysis function.

[0047] In the above, refer to Figure 1 The method for optimizing the ranging accuracy of a single image based on multi-sensor fusion according to an embodiment of the present invention is described in detail. Figure 2A system for optimizing single-image ranging accuracy based on multi-sensor fusion according to an embodiment of the present invention is described.

[0048] The system based on multi-sensor fusion to optimize the single-image ranging accuracy according to the embodiment of the present invention is used to solve the technical problem that the existing single-image ranging is difficult to cope with complex environments, resulting in insufficient ranging accuracy, so as to improve the accuracy and robustness of ranging and better cope with the ranging requirements under complex environmental conditions. The system based on multi-sensor fusion to optimize the single-image ranging accuracy includes: a target image acquisition module 10, a target sensor record acquisition module 20, a time series feature analysis module 30, an intelligent registration fusion processing module 40, and a target distance acquisition module 50.

[0049] The target image acquisition module 10 is used to acquire the target image, and the target image has an identification of the target shooting time; the target sensor record acquisition module 20 is used to acquire the target sensor record, and the target sensor record refers to the record of dynamic sensor monitoring of the target space corresponding to the target image through a multi-sensor group; the timing feature analysis module 30 is used to perform timing feature analysis on the target sensor record to obtain the target sensor information at the target shooting time; the intelligent registration and fusion processing module 40 is used to call the registration and fusion model to perform intelligent registration and fusion processing on the target sensor information and the target image to obtain the target fusion result; the target distance acquisition module 50 is used to obtain the target pixel feature information of the target object in the target fusion result, and obtain the target distance based on the target pixel feature information.

[0050] The specific configuration of the target distance acquisition module 50 will be described in detail below. As described above, the target distance acquisition module 50 may further include: a target environment information acquisition unit for the multi-sensor group including a temperature sensor, a humidity sensor and a wind speed sensor, traversing and matching the target environment temperature dynamically monitored by the temperature sensor, the target environment humidity dynamically monitored by the humidity sensor and the target environment wind speed dynamically monitored by the wind speed sensor in the target sensing record, and forming the target environment information; a target camera information acquisition unit for reading a predetermined camera feature, and acquiring the target camera information of the target camera based on the predetermined camera feature, wherein the target camera refers to the camera that takes the target image; a distance correction unit for introducing an error correction analysis function, and combining the target environment information with the target camera information to perform error analysis of the target distance to obtain the target correction distance.

[0051] The distance correction unit may further include: an error correction analysis function construction subunit for the predetermined camera feature including the target focal length of the target camera and the target comprehensive distortion index of the target lens, and the expression of the error correction analysis function is as follows:

[0052] ;

[0053] in, is the target correction distance, is the target distance, It refers to the environmental correction coefficient of the target environmental information to the target distance, refers to the i-th environmental characteristic parameter in the target environmental information, refers to the i-th coefficient of variation corresponding to the i-th environmental characteristic parameter, It refers to the camera correction coefficient of the characteristic of the target camera to the target distance, refers to the jth feature of the target camera, It refers to the correlation index between the j-th feature and the target distance, and are the first coefficient and the second coefficient respectively, and , and They refer to the environmental error coefficient and the camera error coefficient respectively.

[0054] The specific configuration of the timing feature analysis module 30 will be described in detail below. As described above, the timing feature analysis module 30 may further include: a first point cloud data group extraction unit for the multi-sensor group including a laser radar and a millimeter wave radar, obtaining the laser point cloud timing of the laser radar in the target sensing record, and extracting the first point cloud data group at the first time in the laser point cloud timing; a first interpolation coefficient acquisition unit for normalizing the first time difference between the first time and the target shooting time to obtain a first interpolation coefficient; a weighted calculation unit for performing weighted calculation on the first point cloud data group based on the first interpolation coefficient to obtain a target interpolation point cloud data group at the target shooting time; a target interpolation millimeter wave information acquisition unit for obtaining the target interpolation millimeter wave information of the millimeter wave radar at the target shooting time; and a target sensing information assembly unit for composing the target sensing information with the target interpolation point cloud data group and the target interpolation millimeter wave information.

[0055] The specific configuration of the intelligent registration and fusion processing module 40 will be described in detail below. As described above, the intelligent registration and fusion processing module 40 may further include: a target point cloud model acquisition unit for performing registration and fusion analysis on the target interpolation point cloud data group through the registration and fusion model to obtain a target point cloud model; a target initial model acquisition unit for rendering the target interpolation millimeter wave information to the target point cloud model to obtain a target initial model of the target space; a target three-dimensional model acquisition unit for mapping the target image to the target initial model to obtain a target three-dimensional model, and using the target three-dimensional model as the target fusion result.

[0056] Among them, the target point cloud model acquisition unit may further include: a first point cloud sample extraction subunit is used to randomly extract the first point cloud sample from the target interpolation point cloud data group; a first model parameter acquisition subunit is used to analyze the first point cloud sample through the registration fusion model to obtain the first model parameter of the first point cloud model; a first verification sample acquisition subunit is used to remove the first point cloud sample from the target interpolation point cloud data group to obtain a first verification sample, and the first verification sample includes multiple verification point clouds; a first distance acquisition subunit is used to obtain a first distance from a first verification point cloud in the multiple verification point clouds to the first point cloud model; a first verification result acquisition subunit is used to analyze the first distance to obtain a first verification result, and determine the target point cloud model according to the first verification result.

[0057] Among them, the first inspection result acquisition subunit may further include: a first inspection result analysis microunit is used to add the first inspection point cloud to a consistency list based on the first inspection result if the first distance meets the predetermined distance threshold, and add the first inspection point cloud to an inconsistency list if the first distance does not meet the predetermined distance threshold; a ratio calculation microunit is used to record the ratio of the number of point clouds in the consistency list to that in the inconsistency list as the first inspection result.

[0058] The system for optimizing single-image ranging accuracy based on multi-sensor fusion provided by an embodiment of the present invention can execute the method for optimizing single-image ranging accuracy based on multi-sensor fusion provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0059] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules may be used and run on the user terminal and / or server, and the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0060] The above specific implementation manner does not constitute a limitation to the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiment and can still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for optimizing single-image ranging accuracy based on multi-sensor fusion, characterized in that: include: Acquire a target image, wherein the target image has an identification of the target shooting time; Acquire a target sensing record, wherein the target sensing record refers to a record of dynamic sensing monitoring of a target space corresponding to the target image by a multi-sensor group; Performing time series feature analysis on the target sensing record to obtain target sensing information at the target shooting time; Calling the registration and fusion model to perform intelligent registration and fusion processing on the target sensor information and the target image to obtain a target fusion result; Acquire target pixel feature information of the target object in the target fusion result, and obtain the target distance according to the target pixel feature information; Also includes: The multi-sensor group includes a temperature sensor, a humidity sensor and a wind speed sensor; Traversing and matching the target environment temperature dynamically monitored by the temperature sensor, the target environment humidity dynamically monitored by the humidity sensor, and the target environment wind speed dynamically monitored by the wind speed sensor in the target sensing record, and forming target environment information; Reading a predetermined camera feature, and acquiring target camera information of a target camera based on the predetermined camera feature, wherein the target camera refers to a camera that captures the target image; Introducing an error correction analysis function, and combining the target environment information with the target camera information to perform error analysis on the target distance to obtain a target correction distance; Wherein, the predetermined camera feature includes the target focal length of the target camera and the target comprehensive distortion index of the target lens, and the expression of the error correction analysis function is as follows: ; in, is the target correction distance, is the target distance, It refers to the environmental correction coefficient of the target environmental information to the target distance, Refers to the target environment information Environmental characteristic parameters, refers to the The environmental characteristic parameter corresponds to The coefficient of variation, It refers to the camera correction coefficient of the characteristic of the target camera to the target distance, refers to the target camera's Features, refers to the The correlation index between the feature and the target distance, and are the first coefficient and the second coefficient respectively, and , and They refer to the environmental error coefficient and the camera error coefficient respectively.

2. The method for optimizing single image ranging accuracy based on multi-sensor fusion according to claim 1, characterized in that: include: The multi-sensor group includes a laser radar and a millimeter-wave radar; Acquire a laser point cloud time sequence of the laser radar in the target sensing record, and extract a first point cloud data group at a first time in the laser point cloud time sequence; Normalizing a first time difference between the first time and the target shooting time to obtain a first interpolation coefficient; Performing weighted calculation on the first point cloud data set based on the first interpolation coefficient to obtain a target interpolation point cloud data set at the target shooting time; Obtaining target interpolation millimeter wave information of the millimeter wave radar at the target shooting time; The target interpolation point cloud data group and the target interpolation millimeter wave information constitute the target sensing information.

3. The method for optimizing single image ranging accuracy based on multi-sensor fusion according to claim 2 is characterized in that: include: Performing registration and fusion analysis on the target interpolation point cloud data set through the registration and fusion model to obtain a target point cloud model; Rendering the target interpolation millimeter wave information to the target point cloud model to obtain a target initial model of the target space; The target image is mapped to the target initial model to obtain a target three-dimensional model, and the target three-dimensional model is used as the target fusion result.

4. The method for optimizing single image ranging accuracy based on multi-sensor fusion according to claim 3 is characterized in that: include: Randomly extracting a first point cloud sample from the target interpolation point cloud data group; Analyzing the first point cloud sample by using the registration fusion model to obtain first model parameters of a first point cloud model; Eliminate the first point cloud sample from the target interpolation point cloud data set to obtain a first inspection sample, wherein the first inspection sample includes a plurality of inspection point clouds; Acquire a first distance from a first inspection point cloud among the multiple inspection point clouds to the first point cloud model; The first distance is analyzed to obtain a first inspection result, and the target point cloud model is determined according to the first inspection result.

5. The method for optimizing single image ranging accuracy based on multi-sensor fusion according to claim 4 is characterized in that: include: According to the first inspection result, if the first distance meets a predetermined distance threshold, the first inspection point cloud is added to a consistency list; if the first distance does not meet the predetermined distance threshold, the first inspection point cloud is added to an inconsistency list; The ratio of the number of point clouds in the consistency list to the number of point clouds in the inconsistency list is recorded as the first inspection result.

6. A system for optimizing single-image ranging accuracy based on multi-sensor fusion, characterized in that: The system is used to implement the method for optimizing single-image ranging accuracy based on multi-sensor fusion according to any one of claims 1 to 5, and the system comprises: A target image acquisition module, the target image acquisition module is used to acquire a target image, the target image has an identification of the target shooting time; A target sensing record acquisition module, wherein the target sensing record acquisition module is used to acquire a target sensing record, wherein the target sensing record refers to a record of dynamic sensing monitoring of a target space corresponding to the target image by a multi-sensor group; A timing feature analysis module, which is used to perform timing feature analysis on the target sensing record to obtain target sensing information at the target shooting time; An intelligent registration and fusion processing module, wherein the intelligent registration and fusion processing module is used to call a registration and fusion model to perform intelligent registration and fusion processing on the target sensor information and the target image to obtain a target fusion result; The target distance acquisition module is used to obtain target pixel feature information of the target object in the target fusion result, and obtain the target distance according to the target pixel feature information.