Automatic calibration dual-light fusion image configuration method, device, equipment and medium
By using an automatically calibrated dual-light fusion image configuration method, matching point pairs and confidence levels of visible light and infrared images are obtained, and geometric consistency checks and quality assessments are performed. This solves the problems of high labor costs, insufficient cross-spectral robustness, and difficult online maintenance in existing technologies, and achieves efficient and stable image fusion and target detection and tracking tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2025-12-19
- Publication Date
- 2026-05-12
AI Technical Summary
Existing dual-light system calibration methods suffer from problems such as high labor costs, limited on-site conditions, poor repeatability, difficulty in timely correction of external parameter drift, insufficient cross-spectral robustness, difficulty in online maintenance, and lack of engineering configuration.
By acquiring visible light and infrared images of the same road area, feature extraction is performed to obtain matching point pairs and confidence scores. Geometric consistency checks and fitting are then performed to obtain the homography matrix, and quality assessment is conducted to achieve pixel-level alignment and fusion.
It achieves automatic calibration without manual target placement and offline calibration, significantly reducing labor costs and deployment difficulty, improving calibration efficiency and the quality and consistency of fused images, and enhancing the application performance and robustness of the dual-light fusion system in complex scenarios.
Smart Images
Figure CN122023474A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and intelligent transportation technology. More specifically, this invention relates to an automatically calibrated dual-light fusion image configuration method, apparatus, device, and medium. Background Technology
[0002] In engineering deployments, dual-spectral systems (visible light VIS / infrared IR) require cross-spectral geometric pairing. Existing calibration methods include the following, but they also present the following technical problems: 1) Artificial Target (CharUco / AprilTag Heated Version): Offline calculation of internal and external parameters, followed by fine-tuning after on-site installation. This process suffers from high labor costs, limited on-site conditions, poor repeatability, and difficulty in timely correction of external parameter drift during long-term operation.
[0003] 2) Classical Feature Method (SIFT / ORB + RANSAC): Limited cross-spectral matching capability; poor stability in low-texture / backlight / thermal saturation scenes. Specifically, visible light and infrared belong to cross-spectral imaging, and the difference in appearance distribution leads to weak matching capability and high mismatch rate of classic handcrafted features (such as SIFT / ORB) in cross-spectral matching; in traffic scenes, a large amount of dynamic foreground caused by traffic flow and pedestrians will further interfere with geometric estimation, resulting in unstable homography / mapping solutions.
[0004] In addition, the two existing technologies mentioned above also have the following technical problems: Engineering pain points: When deploying a large number of devices, repeated target placement and manual operation are time-consuming; external parameters are prone to drift after scene migration; parameter expression lacks a unified "configuration" for reuse and rollback.
[0005] It is highly dependent on manual labor: it requires target placement, framing, lighting control, and multiple rounds of shooting.
[0006] Insufficient cross-spectral robustness: Traditional features exhibit sparse matching and numerous mismatches when there are large differences between VIS and IR.
[0007] Online maintenance is difficult: there is no automatic update mechanism after the external parameters drift, and it is difficult to suppress fusion tearing and ghosting in a timely manner.
[0008] Lack of engineered configuration: parameters are difficult to migrate across devices, and rollback and traceability are poor. Summary of the Invention
[0009] The technical problem to be solved by the present invention is to provide an automatically calibrated dual-light fusion image configuration method, apparatus, device and medium, which aims to solve at least one of the above-mentioned technical problems.
[0010] In a first aspect, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: an automatically calibrated dual-light fusion image configuration method, the method comprising: Acquire visible light and infrared images of the same road area, where the visible light and infrared images contain the same scene and target; Feature extraction was performed on the visible light image and the infrared image respectively to obtain the matching point pairs and confidence scores between the visible light image and the infrared image; Based on the confidence level of the matching point pairs, a geometric consistency check and fitting are performed on the matching point pairs to obtain a homography matrix describing the geometric relationship between the visible light image and the infrared image. The homography matrix is quality evaluated to obtain the homography matrix and quality index that pass the evaluation. Based on the homography matrix and quality index that pass the evaluation, the visible light image and infrared image to be processed are aligned and fused at the pixel level.
[0011] The beneficial effects of this invention are as follows: By acquiring visible light and infrared images of the same road area and extracting features from these two images respectively, matching point pairs and confidence scores are obtained. Then, based on the confidence scores of the matching point pairs, geometric consistency checks and fitting are performed to obtain a homography matrix describing the geometric relationship between the visible light and infrared images. Finally, the homography matrix is quality evaluated to obtain a homography matrix that passes the quality evaluation and a quality index, thereby achieving pixel-level alignment and fusion of the visible light and infrared images to be processed. This method achieves automatic calibration of dual-light images without the need for manual target placement and offline calibration, significantly reducing labor costs and deployment difficulty, while improving calibration efficiency. Through feature extraction and confidence score screening, combined with geometric consistency checks, the accuracy and reliability of matching point pairs are ensured, thereby improving the fitting accuracy and stability of the homography matrix. Furthermore, the quality evaluation step further guarantees the quality of the homography matrix, enabling it to be better applied to image fusion, improving the quality and consistency of the fused image, providing a higher-quality image foundation for subsequent tasks such as target detection, tracking, and event recognition, and enhancing the application performance and robustness of the dual-light fusion system in complex scenarios.
[0012] Based on the above technical solution, the present invention can be further improved as follows.
[0013] Furthermore, feature extraction is performed on the visible light image and the infrared image respectively to obtain the matching point pairs and confidence scores between the visible light image and the infrared image, including: Multiple sparse feature points and descriptors for each sparse feature point are extracted from visible light and infrared images, respectively. Feature matching is performed on the sparse feature points corresponding to the visible light image and the infrared image, as well as the descriptor of each sparse feature point, to obtain the matching point pairs and confidence scores between the visible light image and the infrared image.
[0014] Furthermore, based on the confidence level of the matching point pairs, a geometric consistency check and fitting are performed on the matching point pairs to obtain a homography matrix describing the geometric relationship between the visible light image and the infrared image, including: Based on the confidence level of the matching point pairs, a geometric consistency check is performed on the matching point pairs to filter out the interior points that conform to the geometric model; By fitting the interior points that conform to the geometric model, the homography matrix is obtained.
[0015] Furthermore, the method also includes: Based on the homography matrix and the matching point pairs, quality indicators are determined, including reprojection error, inlier rate, and spatial coverage. Quality assessment of the homography matrix includes: The homography matrix is evaluated for quality based on the quality indicators.
[0016] Furthermore, the quality assessment of the homography matrix based on the aforementioned quality indicators includes: Based on the reprojection error and the first set threshold, determine whether the fitting accuracy of the homography matrix meets the requirements; Based on the inlier rate and the second set threshold, determine whether the reliability of the homography matrix meets the requirements; Based on spatial coverage and a third set threshold, determine whether the distribution of matching point pairs corresponding to the homography matrix meets the requirements; The homography matrix that meets the corresponding requirements in terms of fitting accuracy, reliability, and distribution is determined as the homography matrix that passes the quality assessment.
[0017] Secondly, to solve the above-mentioned technical problems, the present invention also provides an automatically calibrated dual-light fusion image configuration device, the device comprising: The acquisition module is used to acquire visible light and infrared images of the same road area, where the visible light and infrared images include the same scene and target; The matching module is used to extract features from visible light images and infrared images respectively, and obtain matching point pairs and confidence scores between visible light images and infrared images; The homography matrix preliminary determination module is used to perform geometric consistency checks and fitting on the matching point pairs based on the confidence level of the matching point pairs, so as to obtain the homography matrix describing the geometric relationship between the visible light image and the infrared image. The homography matrix final determination module is used to perform quality assessment on the homography matrix, obtain the homography matrix that passes the quality assessment and the quality index, and then perform pixel-level alignment and fusion on the visible light image and infrared image to be processed based on the homography matrix that passes the quality assessment and the quality index.
[0018] Thirdly, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the automatic calibration dual-light fusion image configuration method of the present application.
[0019] Fourthly, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the automatic calibration dual-light fusion image configuration method of the present application.
[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.
[0022] Figure 1 This is a flowchart illustrating an automatic calibration method for configuring dual-light fusion images according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating another automatic calibration method for dual-light fusion image configuration according to an embodiment of the present invention; Figure 3 A schematic diagram of the structure of an automatically calibrated dual-light fusion image configuration device provided in one embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation
[0023] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.
[0024] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.
[0025] The solution provided in this invention can be applied to any application scenario requiring dual-light fusion image configuration. The solution provided in this invention can be executed by any electronic device, such as a user's terminal device, including at least one of the following: smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart TV, or smart in-vehicle device.
[0026] This invention provides a possible implementation, such as... Figure 1 The diagram shows a flowchart of an automatically calibrated dual-light fusion image configuration method. This method can be executed by any electronic device, such as a terminal device, or jointly executed by a terminal device and a server. For ease of description, the method provided in this embodiment will be described below using a terminal device as the execution subject as an example. Figure 1 The flowchart shown indicates that the method may include the following steps: S10, acquire visible light and infrared images of the same road area, where the visible light and infrared images include the same scene and target; S20, feature extraction is performed on the visible light image and the infrared image respectively to obtain the matching point pairs and confidence scores between the visible light image and the infrared image; S30, Based on the confidence level of the matching point pairs, perform geometric consistency checks and fitting on the matching point pairs to obtain the homography matrix describing the geometric relationship between the visible light image and the infrared image; S40, perform quality assessment on the homography matrix to obtain the homography matrix and quality index that pass the quality assessment, and perform pixel-level alignment and fusion on the visible light image and infrared image to be processed based on the homography matrix and quality index that pass the quality assessment.
[0027] The method of this invention acquires visible light and infrared images of the same road area, extracts features from these two images to obtain matching point pairs and confidence scores, and then performs geometric consistency checks and fitting based on the confidence scores of the matching point pairs to obtain a homography matrix describing the geometric relationship between the visible light and infrared images. Finally, the homography matrix is quality evaluated to obtain a homography matrix that passes the quality evaluation and a quality index, thereby achieving pixel-level alignment and fusion of the visible light and infrared images to be processed. This method achieves automatic calibration of dual-light images without the need for manual target placement and offline calibration, significantly reducing labor costs and deployment difficulty, while improving calibration efficiency. Through feature extraction and confidence score screening, combined with geometric consistency checks, the accuracy and reliability of matching point pairs are ensured, thereby improving the fitting accuracy and stability of the homography matrix. In addition, the quality evaluation step further guarantees the quality of the homography matrix, enabling it to be better applied to image fusion, improving the quality and consistency of the fused image, providing a higher quality image foundation for subsequent tasks such as target detection, tracking, and event recognition, and enhancing the application performance and robustness of the dual-light fusion system in complex scenarios.
[0028] The following specific embodiments further illustrate the solution of the present invention. In these embodiments, the present invention aims to automatically obtain high-confidence cross-spectral correspondence points using graph neural network matchers such as SuperGlue in natural or weak target scenarios. Unattended solution is used to solve pixel-level mapping (homography / lookup table), and the results are stored in the image configuration in a structured manner. Closed-loop fine-tuning can be performed online, which significantly reduces the cost of manual calibration and improves cross-device consistency and robustness.
[0029] This method employs a cross-spectral feature matching backbone of "SuperPoint features + SuperGlue learning matching", combined with YOLO dynamic target elimination strategy and robust model estimation (RANSAC / MAGSAC++) in traffic scenarios, to obtain geometric models such as homography matrix or sparse grid mapping, forming structured image configuration and version management; online phase periodic monitoring of reprojection error, triggering small-step optimization or recalibration when the threshold is exceeded, to achieve closed-loop maintenance.
[0030] This invention relates to the fields of computer vision and intelligent transportation technology, specifically to an automatic calibration and image configuration method for visible light (VIS) and infrared (IR) dual-light cameras based on the learning feature matcher SuperGlue in transportation industry scenarios. This method achieves cross-spectral geometric alignment and stable fusion, providing a high-quality geometric consistency foundation for subsequent target detection, tracking, and event recognition. Visible light and infrared imaging are significantly complementary: VIS provides rich details and textures, while IR exhibits stronger robustness under complex lighting conditions such as nighttime, backlighting, smoke, or rain and fog. In traffic monitoring and event detection, dual-light fusion can improve all-weather detection and recognition performance.
[0031] Based on this, combined Figure 2 The automatic calibration method for configuring dual-light fusion images provided in this embodiment may include the following steps: S10, acquire visible light and infrared images of the same road area, where the visible light and infrared images include the same scene and target; The aforementioned visible light and infrared images can be acquired using synchronous acquisition devices to ensure temporal consistency and avoid scene changes caused by time differences. The targets can include vehicles, pedestrians, etc.
[0032] S20, feature extraction is performed on the visible light image and the infrared image respectively to obtain the matching point pairs and confidence scores between the visible light image and the infrared image; Here, a matching point pair refers to a pair of points that match in a visible light image and an infrared image. Each matching point pair includes a feature point from the visible light image and a feature point from the infrared image. Confidence refers to the reliability of each matching point pair.
[0033] Alternatively, one possible implementation of the above S20 is as follows: S201, extract multiple sparse feature points and a descriptor for each sparse feature point from the visible light image and the infrared image respectively; For each sparse feature point, a descriptor can be extracted, typically a 256-dimensional vector, to describe the local features around that feature point.
[0034] Specifically, SuperPoint can be used to extract multiple sparse feature points and descriptors for each sparse feature point from visible light and infrared images, respectively. SuperPoint is a sparse feature network trained by a self-supervised framework, suitable for multi-view geometry tasks, robust, and efficient inference.
[0035] S202, perform feature matching on the sparse feature points corresponding to the visible light image and the infrared image, as well as the descriptor of each sparse feature point, to obtain the matching point pairs and confidence scores between the visible light image and the infrared image.
[0036] Specifically, SuperGlue can be used to perform feature matching on sparse feature points and descriptors for each sparse feature point in visible light and infrared images. SuperGlue is a feature matching algorithm based on graph neural networks (GNNs) that can learn the geometric relationships between feature points and output matching point pairs and their confidence scores. The confidence scores are calculated using graph neural networks (GNNs) and attention mechanisms, taking into account the geometric relationships and contextual information between feature points.
[0037] SuperGlue uses a graph neural network (GNN) with attention to aggregate context and simultaneously rejects matching and unmatched points through differentiable optimal transmission, significantly reducing mismatches under cross-spectral / wide baseline conditions.
[0038] Optionally, prior to S202, the method further includes: Inflate the YOLO bounding box by a certain number of pixels (safe boundary), apply SuperPoint filtering to the original VIS / IR image, and then feed it into SuperGlue to extract sparse feature points and descriptors; this reduces the number of "foreground points" from the source.
[0039] In traffic scenarios, dynamic targets (such as vehicles and pedestrians) can interfere with feature matching and geometric estimation. Therefore, before or after feature matching, feature points in dynamic target regions can be removed by combining object detection algorithms (such as YOLO). Use object detection algorithms (such as YOLO) to detect moving targets in visible light and infrared images, for example, to detect the bounding boxes (bboxes) of moving targets.
[0040] Feature points in dynamic target regions are removed from matching point pairs to reduce the impact of dynamic interference on geometric estimation. For example, feature points within the bounding box are removed from the matching point pairs.
[0041] S30, Based on the confidence level of the matching point pairs, perform geometric consistency checks and fitting on the matching point pairs to obtain the homography matrix describing the geometric relationship between the visible light image and the infrared image; Alternatively, one possible implementation of S30 is as follows: S301, Based on the confidence level of the matching point pairs, perform a geometric consistency check on the matching point pairs and filter out the interior points that conform to the geometric model; Specifically, among all matching point pairs, those with a confidence level greater than a set value are identified as interior points that conform to the geometric model.
[0042] S302, fits the interior points that conform to the geometric model to obtain the homography matrix.
[0043] Robust fitting algorithms (such as RANSAC or MAGSAC++) can be used to fit the homography matrix H, ensuring its robustness to anomalous matching points. The fitting results can also be normalized and optimized to improve the accuracy of the homography matrix.
[0044] Optionally, the above-mentioned fitting of interior points conforming to the geometric model yields a homography matrix, including: Perform a normalized DLT (centering + scale normalization) on the set of matching points formed by all interior points conforming to the geometric model, so that the average distance is Then, least squares / LM refinement is performed to obtain indicators such as reprojection error (RPE) quantiles, interior point ratio, and spatial coverage. Specifically, based on the homography matrix and matching point pairs, normalized DLT (centering + scale normalization, making the average distance equal to 1) is applied to the matching point pairs. Then perform least squares / LM refinement, specifically referring to: The matching point set is normalized (centering and scale normalization) to make the average distance of the point set equal to 1. .
[0045] The DLT method is applied to the normalized point set to estimate the initial homography matrix H.
[0046] The initial homography matrix H is optimized using the least squares method or the LM algorithm to further improve the fitting accuracy and obtain the homography matrix.
[0047] S40, perform quality assessment on the homography matrix to obtain the homography matrix that passes the quality assessment and the quality index, and perform pixel-level alignment and fusion on the visible light image and infrared image to be processed based on the homography matrix that passes the quality assessment (which can be a 3×3 matrix) and the quality index.
[0048] Pixel-level alignment refers to mapping the infrared image onto the pixel plane of the visible light image using a homography matrix H that has passed quality assessment, achieving geometric alignment between the two images. Fusion refers to fusing the aligned images to generate a fused image containing information from both spectral sources, used for subsequent tasks such as target detection, tracking, and event recognition. The visible light image and infrared image to be processed refer to the two images that need to be geometrically aligned and fused, originating from a visible light camera and an infrared camera, respectively. These two images capture the same scene or target but have different imaging characteristics. Through the dual-light fusion image configuration method, geometric alignment and fusion of the two images can be achieved, thereby improving the performance of target detection, tracking, and event recognition.
[0049] Optionally, the method further includes: Based on the homography matrix and the matching point pairs, quality indicators are determined, including reprojection error, inlier rate, and spatial coverage. The above quality assessment of the homography matrix includes: The homography matrix is evaluated for quality based on the quality indicators.
[0050] Optionally, the quality indicators are determined based on the homography matrix and matching point pairs. These quality indicators include reprojection error, inlier rate, and spatial coverage, including: For each matching point pair (pVIS, pIR), where pVIS is a feature point in the visible light image and pIR is a feature point in the infrared image, the homography matrix H is used to map the feature point pIR from the infrared image onto the pixel plane of the visible light image: ;in, These are the points mapped through the homography matrix H. Calculate the mapped points. Euclidean distance from the actual matching point pVIS: The reprojection error is determined based on the Euclidean distances between all mapped points and the actual matching points; for example, the mean of all Euclidean distances is calculated as the reprojection error. Based on the confidence level of the matching point pairs, geometric consistency checks and fitting are performed on the matching point pairs to obtain the homography matrix describing the geometric relationship between the visible light image and the infrared image. Simultaneously, an interior point mask is also obtained. Based on the interior point mask, the number of interior points N is calculated. inliers Total number of matching point pairs N total The ratio of these values is determined as the interior point ratio. Divide any one of the infrared or visible light images into several grids (e.g., an M×M grid), and count the number N grids containing interior point matching pairs. covered Spatial coverage is calculated based on the following formula: In this context, inlier match pairs refer to matching point pairs that, after geometric consistency checks during feature matching, are considered to conform to a geometric model (such as homography matrix H, essential matrix E, or fundamental matrix F), i.e., inlier points that conform to the geometric model.
[0051] Optionally, the above-mentioned quality assessment of the homography matrix based on quality indicators includes: Based on the reprojection error and the first set threshold, it is determined whether the fitting accuracy of the homography matrix meets the requirements; for example, if the reprojection error is lower than the first set threshold, it is determined that the fitting accuracy of the homography matrix meets the requirements. The reliability of the homography matrix is determined based on the inlier rate and a second set threshold. For example, if the inlier rate is higher than the second set threshold, the reliability of the homography matrix is determined to be acceptable. Based on spatial coverage and a third set threshold, determine whether the distribution of matching point pairs corresponding to the homography matrix meets the requirements; for example, if the spatial coverage is greater than the third set threshold, determine whether the distribution of matching point pairs corresponding to the homography matrix meets the requirements. The homography matrix that meets the corresponding requirements in terms of fitting accuracy, reliability, and distribution is determined as the homography matrix that passes the quality assessment.
[0052] Optionally, if any one of the fitting accuracy, reliability, and distribution does not meet the corresponding requirements, the method may include: Re-extract sparse feature points and descriptors: This may be due to inaccurate feature extraction or poor quality of matching point pairs. You can try using different feature extraction algorithms or adjusting the extraction parameters.
[0053] Adjust the matching algorithm: Optimize the parameters of the matching algorithm, such as the confidence threshold of SuperGlue, to improve the quality of the matched point pairs.
[0054] Increase the number of samples: If the number of matching point pairs is small, you can try increasing the number of image pairs to improve the fitting accuracy of the homography matrix.
[0055] Refit the homography matrix: Refit the high-quality matching point pairs after screening with geometric fit, such as using RANSAC or MAGSAC++ algorithms.
[0056] Noise reduction: The image is denoised to reduce the impact of noise on feature extraction and matching.
[0057] The solution of the present invention has the following advantages compared with the prior art: 1) Automatic calibration for H: Directly outputs the homography matrix for pixel-level fusion, with short engineering links and low integration costs.
[0058] 2) Cross-spectral robustness: SuperGlue context and geometric prior improve cross-spectral matching quality and remain stable in low-texture / backlight scenes.
[0059] 3) Resistant to dynamic interference: YOLO removes matches within vehicle / pedestrian areas to avoid geometric estimation offsets caused by traffic flow.
[0060] 4) Robust estimation: RANSAC / MAGSAC++ is used, which is not sensitive to outlier matches and has higher accuracy and stability in H.
[0061] 5) Maintainable: Configurable, versioned, and online incremental optimization facilitates large-scale deployment and long-term stable operation.
[0062] Based on and Figure 1Based on the same principle as the method shown, this embodiment of the invention also provides an automatically calibrated dual-light fusion image configuration device 20, such as... Figure 3 As shown, the automatically calibrated dual-light fusion image configuration device 20 may include an acquisition module 210, a matching module 220, a homography matrix preliminary determination module 230, and a homography matrix final determination module 240, wherein: The acquisition module 210 is used to acquire visible light images and infrared images of the same road area, wherein the visible light images and infrared images include the same scene and target. The matching module 220 is used to extract features from the visible light image and the infrared image respectively, and obtain the matching point pairs and confidence scores between the visible light image and the infrared image. The homography matrix preliminary determination module 230 is used to perform geometric consistency checks and fitting on the matching point pairs based on the confidence level of the matching point pairs, so as to obtain a homography matrix describing the geometric relationship between the visible light image and the infrared image. The homography matrix final determination module 240 is used to perform quality assessment on the homography matrix to obtain a homography matrix that passes the quality assessment and a quality index, so as to perform pixel-level alignment and fusion of the visible light image and infrared image to be processed based on the homography matrix that passes the quality assessment and the quality index.
[0063] Optionally, when the matching module 220 performs feature extraction on the visible light image and the infrared image respectively to obtain the matching point pairs and confidence scores between the visible light image and the infrared image, it is specifically used for: Multiple sparse feature points and descriptors for each sparse feature point are extracted from visible light and infrared images, respectively. Feature matching is performed on the sparse feature points corresponding to the visible light image and the infrared image, as well as the descriptor of each sparse feature point, to obtain the matching point pairs and confidence scores between the visible light image and the infrared image.
[0064] Optionally, when the above-mentioned homography matrix preliminary determination module 230 performs geometric consistency checks and fitting on the matching point pairs based on the confidence level of the matching point pairs to obtain the homography matrix describing the geometric relationship between the visible light image and the infrared image, it is specifically used for: Based on the confidence level of the matching point pairs, a geometric consistency check is performed on the matching point pairs to filter out the interior points that conform to the geometric model; By fitting the interior points that conform to the geometric model, the homography matrix is obtained.
[0065] Optionally, the above-mentioned device further includes: The quality index determination module is used to determine quality indices based on the homography matrix and matching point pairs. The quality indices include reprojection error, inlier rate, and spatial coverage. The homography matrix final determination module 240 described above is specifically used for: When performing quality assessment on the homography matrix, it is used to: The homography matrix is evaluated for quality based on the quality indicators.
[0066] Optionally, the homography matrix final determination module 240, when performing a quality assessment of the homography matrix based on quality indicators, is specifically used for: Based on the reprojection error and the first set threshold, determine whether the fitting accuracy of the homography matrix meets the requirements; Based on the inlier rate and the second set threshold, determine whether the reliability of the homography matrix meets the requirements; Based on spatial coverage and a third set threshold, determine whether the distribution of matching point pairs corresponding to the homography matrix meets the requirements; The homography matrix that meets the corresponding requirements in terms of fitting accuracy, reliability, and distribution is determined as the homography matrix that passes the quality assessment.
[0067] The automatically calibrated dual-light fusion image configuration device of this invention can execute the automatically calibrated dual-light fusion image configuration method provided in this invention. The implementation principle is similar. The actions performed by each module and unit in the automatically calibrated dual-light fusion image configuration device in each embodiment of this invention correspond to the steps in the automatically calibrated dual-light fusion image configuration method in each embodiment of this invention. For detailed functional descriptions of each module of the automatically calibrated dual-light fusion image configuration device, please refer to the descriptions in the corresponding automatically calibrated dual-light fusion image configuration methods shown above, which will not be repeated here.
[0068] The aforementioned automatically calibrated dual-light fusion image configuration device can be a computer program (including program code) running on a computer device. For example, the automatically calibrated dual-light fusion image configuration device is an application software. The device can be used to execute the corresponding steps in the method provided in the embodiments of the present invention.
[0069] In some embodiments, the automatically calibrated dual-light fusion image configuration device provided in this invention can be implemented using a combination of hardware and software. As an example, the automatically calibrated dual-light fusion image configuration device provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the automatically calibrated dual-light fusion image configuration method provided in this invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.
[0070] In other embodiments, the automatically calibrated dual-light fusion image configuration device provided in this invention can be implemented in software. Figure 3 An automatically calibrated dual-light fusion image configuration device stored in a memory is shown. It can be software in the form of programs and plug-ins, and includes a series of modules, including an acquisition module 210, a matching module 220, a homography matrix preliminary determination module 230, and a homography matrix final determination module 240, for implementing the automatically calibrated dual-light fusion image configuration method provided in the embodiments of the present invention.
[0071] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.
[0072] Based on the same principles as the methods shown in the embodiments of the present invention, the embodiments of the present invention also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the methods shown in any embodiment of the present invention by invoking the computer programs.
[0073] In one alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.
[0074] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0075] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0076] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0077] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.
[0078] Among these, electronic devices can also be terminal devices. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.
[0079] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.
[0080] According to another aspect of the present invention, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.
[0081] Computer program code for performing the operations of this invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0082] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0083] The computer-readable storage medium provided in this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0084] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.
[0085] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.
Claims
1. An automatically calibrated method for configuring dual-light fusion images, characterized in that, Includes the following steps: Acquire visible light and infrared images of the same road area, wherein the visible light and infrared images include the same scene and target; Feature extraction is performed on the visible light image and the infrared image respectively to obtain the matching point pairs and confidence scores between the visible light image and the infrared image; Based on the confidence level of the matching point pairs, a geometric consistency check and fitting are performed on the matching point pairs to obtain a homography matrix describing the geometric relationship between the visible light image and the infrared image; The homography matrix is subjected to quality evaluation to obtain a homography matrix that passes the quality evaluation and a quality index. Based on the homography matrix that passes the quality evaluation and the quality index, the visible light image and infrared image to be processed are aligned and fused at the pixel level.
2. The method according to claim 1, characterized in that, The step of extracting features from the visible light image and the infrared image respectively to obtain the matching point pairs and confidence scores between the visible light image and the infrared image includes: Multiple sparse feature points and a descriptor for each sparse feature point are extracted from the visible light image and the infrared image, respectively. Feature matching is performed on the sparse feature points corresponding to the visible light image and the infrared image, as well as the descriptor of each sparse feature point, to obtain the matching point pairs and confidence scores between the visible light image and the infrared image.
3. The method according to claim 1, characterized in that, The step of performing geometric consistency checks and fitting on the matching point pairs based on their confidence levels to obtain a homography matrix describing the geometric relationship between the visible light image and the infrared image includes: Based on the confidence level of the matching point pairs, a geometric consistency check is performed on the matching point pairs to filter out interior points that conform to the geometric model; The homography matrix is obtained by fitting the interior points of the geometric model.
4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the homography matrix and the matching point pairs, quality indicators are determined, including reprojection error, inlier rate, and spatial coverage. The quality assessment of the homography matrix includes: The homography matrix is evaluated for quality based on the stated quality indicators.
5. The method according to claim 4, characterized in that, The quality assessment of the homography matrix based on the quality indicators includes: Based on the reprojection error and the first set threshold, determine whether the fitting accuracy of the homography matrix meets the requirements; Based on the inlier rate and the second set threshold, determine whether the reliability of the homography matrix meets the requirements; Based on the spatial coverage and the third set threshold, determine whether the distribution of the matching point pairs corresponding to the homography matrix meets the requirements; The homography matrix that meets the corresponding requirements in terms of fitting accuracy, reliability, and distribution is determined as the homography matrix that passes the quality assessment.
6. An automatically calibrated dual-light fusion image configuration device, characterized in that, include: An acquisition module is used to acquire visible light images and infrared images of the same road area, wherein the visible light images and the infrared images include the same scene and target; The matching module is used to extract features from the visible light image and the infrared image respectively, and obtain the matching point pairs and confidence scores between the visible light image and the infrared image; The homography matrix preliminary determination module is used to perform geometric consistency checks and fitting on the matching point pairs based on the confidence level of the matching point pairs, so as to obtain a homography matrix describing the geometric relationship between the visible light image and the infrared image; The homography matrix final determination module is used to perform quality assessment on the homography matrix to obtain a homography matrix that passes the quality assessment and a quality index, so as to perform pixel-level alignment and fusion of the visible light image and infrared image to be processed based on the homography matrix that passes the quality assessment and the quality index.
7. The apparatus according to claim 6, characterized in that, When the matching module extracts features from the visible light image and the infrared image respectively to obtain the matching point pairs and confidence scores between the visible light image and the infrared image, it is specifically used for: Multiple sparse feature points and a descriptor for each sparse feature point are extracted from the visible light image and the infrared image, respectively. Feature matching is performed on the sparse feature points corresponding to the visible light image and the infrared image, as well as the descriptor of each sparse feature point, to obtain the matching point pairs and confidence scores between the visible light image and the infrared image.
8. The apparatus according to claim 6, characterized in that, The homography matrix preliminary determination module, when performing geometric consistency checks and fitting on the matching point pairs based on their confidence levels to obtain a homography matrix describing the geometric relationship between the visible light image and the infrared image, is specifically used for: Based on the confidence level of the matching point pairs, a geometric consistency check is performed on the matching point pairs to filter out interior points that conform to the geometric model; The homography matrix is obtained by fitting the interior points of the geometric model.
9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of claims 1-5.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-5.