Wavelength division imaging device and image registration following method

Abstract

The embodiment of the invention discloses wavelength division imaging equipment and an image registration following method. According to one specific embodiment, the wavelength division imaging device comprises a visible light camera, an infrared camera, a dichroscope, a wide spectrum window and a mechanical shell, and one side of the mechanical shell comprises a light inlet; the visible light camera is mounted on one side in the mechanical shell, comprises a visible light imaging lens and is used for receiving a visible wave band and carrying out visible light imaging; the infrared camera is mounted on the other side of the interior of the mechanical shell, comprises an infrared lens and is used for receiving an infrared band and performing infrared imaging; the wide spectrum window is installed at a light inlet included in the mechanical shell and used for preventing dust and rainwater from entering the mechanical shell. The dichroscope is installed in the mechanical shell and used for reflecting infrared band light and transmitting visible band light. According to the embodiment, the situation of view angle difference caused by multi-band lens superposition imaging is avoided.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of light imaging, specifically to wavelength-division imaging devices and image registration and following methods. Background Technology

[0002] With the continuous advancement of imaging technology, monitoring lenses are increasingly being applied to various monitoring scenarios. Currently, when using monitoring lenses for scene monitoring, the common methods are: monitoring through a single-band monitoring lens or using multi-band lenses to create segmented, stitched images.

[0003] However, when using the above method to perform scene monitoring with a monitoring lens, the following technical problems often occur:

[0004] When using a single-band lens for scene monitoring, image information may be lost, resulting in incomplete scene monitoring information. When using a multi-band lens for window stitching imaging, each band forms its own optical path, causing differences in the viewing angle of the imaging results of each band, which in turn leads to inconsistent imaging information.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure propose wavelength-division imaging devices and image registration and following methods to solve one or more of the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a wavelength-splitting imaging device, which includes: a visible light camera, an infrared camera, a dichroic mirror, a broadband window, and a mechanical housing. One side of the mechanical housing includes a light inlet for receiving mixed imaging natural light through the light inlet. The visible light camera is mounted inside one side of the mechanical housing and includes a visible light imaging lens for receiving visible wavelengths and performing visible light imaging. The infrared camera is mounted inside the other side of the mechanical housing and includes an infrared lens for receiving infrared wavelengths and performing infrared imaging. The broadband window is mounted at the light inlet of the mechanical housing to prevent dust and rainwater from entering the mechanical housing. The dichroic mirror is mounted inside the mechanical housing for reflecting infrared light and transmitting visible light.

[0009] Optionally, the aforementioned wavelength-division imaging device further includes a broadband achromatic collimator, which is installed in front of the light inlet of the aforementioned mechanical housing to reduce the incident light beam.

[0010] Optionally, the infrared camera and the visible light camera are respectively installed at a positive 45-degree angle and a negative 45-degree angle on the dichroic mirror.

[0011] Alternatively, the aforementioned broad spectral window may include one of the following: a fluorite window and a diamond sheet coating.

[0012] Optionally, the aforementioned wavelength division imaging device further includes a bracket, which is installed inside the aforementioned mechanical housing. The bracket is made of a material with high rigidity and low coefficient of thermal expansion, and is used to fix the aforementioned visible light camera, the aforementioned infrared camera, and the aforementioned dichroic mirror.

[0013] Secondly, some embodiments of this disclosure provide an image registration and following method, which includes: receiving an initial video frame pair sent by a wavelength division imaging device in real time; performing feature point matching processing on the initial video frame pair based on coordinate consistency to generate a matching feature point set; splitting the matching feature point set to generate a basic control point set and a backup control point set; performing dynamic registration and following on the initial video frame pair based on the basic control point set and the backup control point set to generate a registered video frame pair; and sending the registered video frame pair to an associated video receiving terminal for display.

[0014] Optionally, the above-mentioned feature point matching processing based on coordinate consistency to generate a matching feature point set includes: performing special point detection processing on each initial video frame in the above-mentioned initial video frame pair to generate a feature point group; performing feature point matching processing on each generated feature point group based on coordinate consistency to generate a matched feature point set; performing deduplication processing on the matched feature point set to generate a deduplicated feature point set; and selecting a preset number of deduplicated feature point groups from the deduplicated feature point set as matching feature point groups to obtain the matching feature point set.

[0015] Optionally, the above-mentioned dynamic registration and following of the initial video frame pair based on the basic control point set and the backup control point set to generate a registered video frame pair includes: determining the basic control point neighborhood corresponding to the basic control point set to obtain the control point neighborhood range; performing control point search processing on the control point neighborhood range based on the basic control point set to generate a search result; in response to a change in the image represented by the search result, determining whether the basic control point set meets a preset description condition, wherein the preset description condition is that the description corresponding to each basic control point group in the basic control point set changes; in response to a change in the image represented by the basic control point set... If the set of basic control points meets the above-mentioned preset description conditions, and based on the consistency of the above-mentioned backup control point set and coordinates, the above-mentioned basic control point set is updated to generate an updated control point set; based on the above-mentioned updated control point set, the above-mentioned initial video frame pair is aligned to generate an aligned video frame pair, which serves as a registered video frame pair; the above-mentioned backup control point set is supplemented to generate a supplemented backup control point set, which serves as a backup control point set; in response to the above-mentioned basic control point set not meeting the above-mentioned preset description conditions, based on the above-mentioned basic control point set, the above-mentioned initial video frame pair is aligned to generate an aligned video frame pair, which serves as a registered video frame pair.

[0016] Optionally, the above method further includes: in response to the search results indicating that the set of matching feature points meets the preset registration conditions, determining the initial video frame pair as the registered video frame pair.

[0017] The various embodiments disclosed above have the following beneficial effects: the wavelength division imaging devices of some embodiments of this disclosure avoid the difference in viewing angle caused by the superposition of multi-band lenses. Specifically, the reason for the difference in viewing angle caused by the superposition of multi-band lenses is that infrared optical components are mostly opaque in the visible band and have a wide spectral color difference due to changes in refractive index. Furthermore, since multiple cameras monitor the scene from different angles, the final image has a difference in viewing angle when the images from different cameras are superimposed. Based on this, some embodiments of the wavelength-splitting imaging device disclosed herein include a visible light camera, an infrared camera, a dichroic mirror, a broadband window, and a mechanical housing. One side of the mechanical housing includes a light inlet for receiving mixed imaging natural light. The visible light camera is mounted inside the mechanical housing on one side and includes a visible light imaging lens for receiving visible light and performing visible light imaging. The infrared camera is mounted inside the mechanical housing on the other side and includes an infrared lens for receiving infrared light and performing infrared imaging. The broadband window is mounted at the light inlet of the mechanical housing to prevent dust and rainwater from entering the mechanical housing. The dichroic mirror is mounted inside the mechanical housing to reflect infrared light and transmit visible light. Thus, the wavelength boundary between visible and infrared light can be used as the cutoff band of the dichroic mirror, achieving the effect of transmitting visible light and reflecting infrared light, thereby realizing wavelength-splitting beam splitting in a common optical path. Imaging devices for corresponding wavelengths are installed at positive and negative 45-degree angles to the dichroic mirror, achieving a parallax-free dual-band wavelength-splitting imaging effect. This avoids the situation where differences in perspective arise from the superposition of multiple band lenses. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0019] Figure 1 These are schematic diagrams illustrating the structure of some embodiments of the wavelength-division imaging apparatus according to this disclosure;

[0020] Figure 2 This is a flowchart of some embodiments of the image registration and following method according to the present disclosure. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0023] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0024] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0026] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] Figure 1 A schematic diagram 100 of some embodiments of a wavelength-division imaging device according to the present disclosure is shown. The wavelength-division imaging device includes: a visible light camera 101, an infrared camera 102, a dichroic mirror 103, a broadband window 104, and a mechanical housing 105.

[0028] In some embodiments, one side of the aforementioned mechanical housing includes a light inlet for receiving mixed imaging natural light through the light inlet; the aforementioned visible light camera is mounted inside one side of the aforementioned mechanical housing, including a visible light imaging lens for receiving visible light and performing visible light imaging; the aforementioned infrared camera is mounted inside the other side of the aforementioned mechanical housing, including an infrared lens for receiving infrared light and performing infrared imaging; the aforementioned broadband window is mounted at the light inlet included in the aforementioned mechanical housing to prevent dust and rainwater from entering the aforementioned mechanical housing; the aforementioned dichroic mirror is mounted inside the aforementioned mechanical housing to reflect infrared light and transmit visible light.

[0029] Optionally, the wavelength division imaging device also includes a broadband achromatic collimator, which is installed in front of the light inlet of the aforementioned mechanical housing to reduce the incident light beam.

[0030] Optionally, the infrared camera and the visible light camera are respectively installed at a positive 45-degree angle and a negative 45-degree angle on the dichroic mirror.

[0031] Alternatively, the aforementioned broad spectral window includes one of the following: a fluorite window and a diamond sheet coating.

[0032] Optionally, the aforementioned wavelength division imaging device further includes a bracket, which is installed inside the aforementioned mechanical housing. The bracket is made of a material with high rigidity and low coefficient of thermal expansion, and is used to fix the aforementioned visible light camera, the aforementioned infrared camera, and the aforementioned dichroic mirror.

[0033] The various embodiments disclosed above have the following beneficial effects: the wavelength division imaging devices of some embodiments of this disclosure avoid the difference in viewing angle caused by the superposition of multi-band lenses. Specifically, the reason for the difference in viewing angle caused by the superposition of multi-band lenses is that infrared optical components are mostly opaque in the visible band and have a wide spectral color difference due to changes in refractive index. Furthermore, since multiple cameras monitor the scene from different angles, the final image has a difference in viewing angle when the images from different cameras are superimposed. Based on this, some embodiments of the wavelength-splitting imaging device disclosed herein include a visible light camera, an infrared camera, a dichroic mirror, a fluorite window, and a mechanical housing. One side of the mechanical housing includes a light inlet for receiving mixed imaging natural light. The visible light camera is mounted inside the mechanical housing on one side and includes a visible light imaging lens for receiving visible light and performing visible light imaging. The infrared camera is mounted inside the mechanical housing on the other side and includes an infrared lens for receiving infrared light and performing infrared imaging. The fluorite window is installed at the light inlet of the mechanical housing to prevent dust and rainwater from entering the mechanical housing. The dichroic mirror is mounted inside the mechanical housing to reflect infrared light and transmit visible light. Thus, the wavelength boundary between visible and infrared light can be used as the cutoff band of the dichroic mirror, achieving the effect of transmitting visible light and reflecting infrared light, thereby realizing wavelength-splitting beam splitting in a common optical path. Imaging devices for corresponding wavelengths are installed at positive and negative 45-degree angles to the dichroic mirror, achieving a parallax-free dual-band wavelength-splitting imaging effect. This avoids the situation where differences in perspective arise from the superposition of multiple band lenses.

[0034] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of an image registration and following method according to the present disclosure. The image registration and following method includes the following steps:

[0035] Step 201: Receive the initial video frame pairs sent by the wavelength division imaging device in real time.

[0036] In some embodiments, the execution entity of the image registration and following method (e.g., a server) can receive initial video frame pairs sent by a wavelength-division imaging device in real time. These initial video frame pairs can be visible light initial video frame images and infrared initial video frame images. The visible light initial video frame images and the infrared initial video frame images can be video frame images generated by the wavelength-division imaging device.

[0037] Step 202: Based on coordinate consistency, perform feature point matching processing on the above initial video frame pairs to generate a set of matching feature points.

[0038] In some embodiments, the execution entity may perform feature point matching processing on the initial video frame pairs based on coordinate consistency to generate a set of matching feature points.

[0039] In practice, the initial video frame pairs described above can be processed using the following steps to perform feature point matching to generate a set of matching feature points:

[0040] The first step involves performing feature point detection processing on each initial video frame in the aforementioned initial video frame pair to generate a feature point group. This feature point detection processing can be performed on each initial video frame in the initial video frame pair using a feature point detection algorithm. For example, the feature point detection algorithm could be the FAST feature point detection algorithm.

[0041] The second step, based on coordinate consistency, involves feature point matching processing on each generated feature point group to produce a matched feature point set. In practice, feature point matching can be performed using consistent initial coordinates.

[0042] The third step is to deduplicate the matched feature point set to generate a deduplicated feature point set. In practice, a spatial algorithm can be used to remove matching points that are too close together to obtain the deduplicated feature point set. This spatial algorithm can be a quadtree algorithm.

[0043] The fourth step involves selecting a predetermined number of deduplicated feature point groups from the aforementioned set of deduplicated feature point groups to form a matching feature point set. This predetermined number can be a pre-set number of deduplicated feature point groups to be selected. For example, this predetermined number could be 6.

[0044] Step 203: The above-mentioned matching feature point set is split to generate a basic control point set and a backup control point set.

[0045] In some embodiments, the executing entity may split the matching feature point set to generate a basic control point set and a backup control point set. In practice, the matching feature point set can be randomly split in half to create a basic control point set and a backup control point set. Here, the matching feature point set may include six matching feature point groups; therefore, the matching feature point set can be split into three basic control point sets and three backup control point sets.

[0046] Step 204: Based on the above-mentioned basic control point set and the above-mentioned backup control point set, perform dynamic registration and following on the above-mentioned initial video frame pair to generate a registered video frame pair.

[0047] In some embodiments, the execution entity may perform dynamic registration and following of the initial video frame pair based on the basic control point set and the backup control point set to generate a registered video frame pair.

[0048] In some optional implementations of certain embodiments, the execution entity may perform dynamic registration and following of the initial video frame pair using the following steps to generate a registered video frame pair:

[0049] The first step is to determine the neighborhood of the basic control points corresponding to the aforementioned set of basic control points, thus obtaining the range of the control point neighborhood. The neighborhood of the basic control points can be a neighborhood of a pre-defined size. For example, the neighborhood of the basic control points can be a 7x7 or 9x9 neighborhood corresponding to the aforementioned set of basic control points.

[0050] In the process of adopting technical solutions to address the aforementioned technical problems, the following issues often arise: When registering visible light and infrared images, changes in image description may lead to the loss of original basic control points, making image registration impossible using the selected basic control points, resulting in image distortion in the output. In response to these technical problems, and considering the current state of available technology, the following solution can be adopted.

[0051] In practice, the neighborhood of the basic control points corresponding to the above set of basic control points can be determined through the following sub-steps, thus obtaining the range of the control point neighborhood:

[0052] The first sub-step involves performing the following processing steps for each basic control point in the aforementioned basic control point set:

[0053] The first processing step involves performing a control point search based on the aforementioned basic control points to generate control point search results. This control point search can be performed by searching for control points in the coordinates of the corresponding registered video frames based on the coordinates of the basic control points on the screen.

[0054] The second processing step, in response to the control point search result indicating that no target control point was found, predicts the position of the control point corresponding to the target control point based on a preset position algorithm to generate a predicted control point position. The target control point can be a control point in the registered video frame that corresponds to the basic control point.

[0055] The third processing step involves determining the search radius for the base control point and the target control point based on the predicted control point locations, and defining the search range corresponding to the search radius as the neighborhood of the sub-control point. In practice, the distance between the predicted control point location and the base control point can be defined as the search radius, and the circular area centered on the base control point and with the search radius as its radius can be defined as the neighborhood of the sub-control point.

[0056] Here, the search radius for the aforementioned basic control points and target control points can be determined using the following methods:

[0057] Let the initial control point pixel grayscale value be I(x, y, t). When Δt is sufficiently small, it can be assumed that the grayscale value of the new point has not changed relative to the initial point, resulting in the following formula:

[0058] I(x,y,t)=I(x+Δx,y+Δy,t+Δt).

[0059] Where Δt represents a very small time interval, u and v represent the optical flow of the new control point relative to the original point in the horizontal and vertical directions, respectively, and x and y represent the x-coordinate and y-coordinate of the initial control point I, respectively. A Taylor expansion of the small changes yields the following formula:

[0060]

[0061] Where ε represents terms of degree two or higher. When Δt approaches 0, the formula becomes:

[0062] I x ·u+I y ·v+I t =0.

[0063] in, Assuming that within a 5x5 neighborhood centered on the initial control point, all pixels have the same motion vector (optical flow), then the optical flow constraint equation can be written for each pixel in that neighborhood:

[0064] I xi ·u+I yi ·v+I ti =0.

[0065] Where i takes values ​​from 1 to 25 (all pixels in the 5x5 neighborhood), then the optical flow constraint equations for this neighborhood are:

[0066]

[0067] make Then we have:

[0068] A T A·U=A T b.

[0069] Where A, U, and b represent the pixel brightness matrix, optical flow matrix, and time variation matrix, respectively. T denotes the transpose. Solving the above formula using the least squares method yields:

[0070] U = (A T A) -1 ·A T b.

[0071] Here, the combined optical flow U of the motion vectors of the neighborhood point group can be obtained. Based on the optical flow, the motion vector of the new point relative to the initial point is obtained, and the vector magnitude is used as the initial search radius.

[0072] The fourth processing step involves performing a control point search based on the aforementioned sub-control point neighborhood to generate a second control point search result. In practice, a control point search can be performed again within the aforementioned sub-control point neighborhood.

[0073] The fifth processing step, in response to the second control point search result indicating that no target control point was found, updates the neighborhood range of the sub-control points to generate an updated search range. In practice, the radius corresponding to the neighborhood range of the sub-control points can be increased by 50% to update the neighborhood range of the sub-control points and obtain the updated search range.

[0074] The sixth processing step involves re-exercising the control point search process based on the updated search range described above, to generate the third control point search result.

[0075] The seventh processing step is to delete the basic control point from the basic control point set in response to the above-mentioned third control point search result indicating that no target control point was found, so as to generate a deleted basic control point set.

[0076] The eighth processing step involves selecting target matching feature points from the aforementioned set of matching feature points and adding them to the aforementioned set of deleted basic control points to update the aforementioned set of deleted basic control points, thus obtaining an updated control point set. Here, matching feature points can be randomly selected from the aforementioned set of matching feature points and added to the aforementioned set of deleted basic control points.

[0077] The second sub-step involves merging the sub-basic control point neighborhoods corresponding to each updated control point in the aforementioned updated control point set into the basic control point neighborhood.

[0078] The aforementioned first to second sub-steps, as an inventive point of this disclosure, combined with step "Step 205" below, solve the technical problem: "When registering a visible light image and an infrared image, the image description may change, resulting in the loss of the original basic control points. This makes image registration impossible using the selected basic control points, leading to image distortion in the output." The reason for the output image distortion is as follows: When registering a visible light image and an infrared image, the image description may change, resulting in the loss of the original basic control points. This makes image registration impossible using the selected basic control points, leading to image distortion in the output. Solving the above factors can avoid output image distortion. To achieve this effect, this disclosure firstly performs the following processing steps for each basic control point in the aforementioned basic control point set: First, based on the aforementioned basic control points, a control point search process is performed to generate control point search results. This generates control points corresponding to the visible light image and the infrared image. Secondly, in response to the control point search results indicating that no target control point was found, the position of the control point corresponding to the target control point is predicted based on a preset position algorithm to generate a predicted control point position. Thus, the position where the control point should exist can be predicted using a position algorithm. Then, based on the predicted control point position, the search radius for the basic control point and the target control point is determined, and the search range corresponding to the search radius is defined as the neighborhood range of the sub-control point. Thus, control point searching can be performed within the search range of the sub-control point. Next, based on the neighborhood range of the sub-control point, control point search processing is performed to generate a second control point search result; in response to the second control point search result indicating that no target control point was found, the neighborhood range of the sub-control point is updated to generate an updated search range. Thus, the search range can be expanded when no control point is found. Then, based on the updated search range, control point search processing is performed again to generate a third control point search result; in response to the third control point search result indicating that no target control point was found, the basic control point is deleted from the basic control point set to generate a deleted basic control point set. Therefore, if no control point is found after expanding the search area, the control point can be deleted. Then, a target backup control point group is selected from the aforementioned backup control point group and added to the deleted basic control point group to update the deleted basic control point group, resulting in an updated control point group. Thus, the basic control points can be updated using alternative control point groups. Finally, the sub-basic control point neighborhoods corresponding to each updated control point in the updated control point group are merged into the basic control point neighborhood. Combined with "Step 205", the registered video frame pairs are sent to the associated video receiving terminal for display.Therefore, by searching for control points and updating control points using backup control points when no basic control point is found, visible light and infrared images can be registered, thus avoiding image distortion in the output.

[0079] The second step involves performing a control point search on the neighborhood of the aforementioned control points, based on the basic control point set, to generate search results.

[0080] The third step is to determine, based on the search results above, whether the aforementioned set of basic control points meets the preset description conditions. These preset description conditions may include changes in the descriptions corresponding to different basic control point groups within the set.

[0081] Fourth, in response to the aforementioned basic control point set satisfying the preset description conditions, and based on the consistency of the aforementioned backup control point set and coordinates, the basic control point set is updated to generate an updated control point set. In practice, basic control point sets that exceed the screen area can be removed from the aforementioned basic control point set, and backup control point sets with the same number of removed basic control point sets can be selected from the aforementioned backup control point set. The selected backup control point sets are then merged into the basic control point set to update the basic control point set, resulting in the updated control point set.

[0082] Fifth, based on the updated control point set, the initial video frame pairs are aligned to generate aligned video frame pairs, which serve as the registered video frame pairs. The alignment process can involve calculating transformation matrices based on the various basic control point groups included in the basic control point set to achieve alignment.

[0083] The sixth step involves supplementing the aforementioned set of backup control points to generate a supplemented set of backup control points. This supplementation process may involve re-detecting feature points and selecting feature point groups from the detected groups, merging them into the backup control point set, so that the number of backup control point groups in the backup control point set is the same as the number of updated control point groups in the updated control point set.

[0084] Step 7: In response to the fact that the above-mentioned basic control point set does not meet the above-mentioned preset description conditions, the above-mentioned initial video frame pair is subjected to image alignment processing based on the above-mentioned basic control point set to generate aligned video frame pairs as registered video frame pairs.

[0085] Optionally, after step seven, in response to the above search result characterization screen matching feature point set satisfying preset registration conditions, the above initial video frame pair is determined as the registered video frame pair.

[0086] The aforementioned steps one through seven, as an inventive point of this disclosure, combined with step "step 205" below, solve the technical problem: "When performing image registration, registration is usually performed through description consistency. However, for multimodal images, due to the inconsistency of the brightness gradient matrices of the feature points, image registration cannot be performed, resulting in image distortion." The reasons for the inability to perform image registration and the resulting image distortion are as follows: When performing image registration, registration is usually performed through description consistency. However, for multimodal images, due to the inconsistency of the brightness gradient matrices of the feature points, image registration cannot be performed, resulting in image distortion. If the above factors are resolved, the effect of avoiding output image distortion can be achieved. To achieve this effect, this disclosure firstly determines the neighborhood of the basic control points corresponding to the aforementioned set of basic control points, obtaining the control point neighborhood range. Thus, the neighborhood range of control points can be determined. Secondly, based on the aforementioned set of basic control points, control point search processing is performed on the aforementioned control point neighborhood range to generate search results. Thus, the corresponding control point search can be performed through the set neighborhood range. Third, in response to changes in the search results representing the image, it is determined whether the basic control point set meets the preset description conditions. This determines whether the description of the control points changes after the image changes. Fourth, in response to the basic control point set meeting the preset description conditions, based on the consistency of the backup control point set and coordinates, the basic control point set is updated to generate an updated control point set. This updates the coordinates of the control points through coordinate consistency, avoiding situations where registration fails due to image changes. Fifth, based on the updated control point set, the initial video frame pairs are aligned to generate aligned video frame pairs, which serve as the registered video frame pairs. This allows for image alignment and registration using the updated control points. Sixth, the backup control point set is supplemented to generate a supplemented backup control point set. This allows the number of backup control points to be increased to a preset amount. Seventh, in response to the aforementioned basic control point set not meeting the preset description conditions, the initial video frame pair is aligned based on the aforementioned basic control point set to generate an aligned video frame pair, which serves as the registered video frame pair. Thus, the aligned image can be used as the registered video frame pair. Combined with "step 205", the registered video frame pair is sent to the associated video receiving terminal for display. Therefore, by searching for control points and updating control points using backup control points when no basic control points are found, the registered visible light and infrared images can be registered, avoiding image distortion in the output.

[0087] Step 205: Send the registered video frame pairs to the associated video receiving terminal for display.

[0088] In some embodiments, the executing entity may send the registered video frame pairs to an associated video receiving terminal for display. The associated video receiving terminal may be a user terminal connected to the executing entity via a wired or wireless connection.

[0089] The above embodiments of this disclosure have the following beneficial effects: the image registration and following method of some embodiments of this disclosure avoids the situation where multimodal camera imaging cannot match, thereby avoiding the situation where the monitoring screen cannot dynamically follow. Specifically, the reason why multimodal camera imaging cannot match and the monitoring screen cannot dynamically follow is that the inherent differences between multimodal images make it impossible for the algorithm to search for consistent feature points. The only registration method is to mechanically align the images during hardware assembly, thus making it impossible to achieve dynamic image following. Based on this, the image registration and following method of some embodiments of this disclosure firstly receives the initial video frame pair sent by the wavelength division imaging device in real time. Thus, the images sent by the wavelength division imaging device can be received in real time. Secondly, feature point matching processing is performed on the initial video frame pair to generate a matching feature point set. Thus, by performing feature point matching on the initial video frames in the initial video frame pair, the feature points corresponding to each initial video frame in the initial video frame pair can be determined. Then, the matching feature point set is split to generate a basic control point set and a backup control point set. Thus, basic control points and backup control points can be separated. Subsequently, based on the aforementioned basic control point set and the aforementioned backup control point set, the initial video frame pair is dynamically registered and followed to generate a registered video frame pair. This allows for registration of the initial video frames within the initial video frame pair using consistent feature points. Finally, the registered video frame pair is sent to the associated video receiving terminal for display. This completes the image registration and following, preventing mismatches in multimodal camera imaging and thus avoiding situations where the monitoring image cannot dynamically follow the target.

[0090] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A wavelength-division imaging device, the wavelength-division imaging device comprising: Visible light camera, infrared camera, dichroic mirror, wide-spectrum window, and mechanical housing, among which, One side of the mechanical housing includes a light inlet for receiving mixed imaging natural light through the light inlet; The visible light camera is installed inside the mechanical housing and includes a visible light imaging lens for receiving visible wavelengths and performing visible light imaging. The infrared camera is installed inside the mechanical housing on the other side, and includes an infrared lens for receiving infrared wavelengths and performing infrared imaging. The broadband window is installed at the light inlet of the mechanical housing to prevent dust and rainwater from entering the mechanical housing; The dichroic mirror is installed inside the mechanical housing and is used to reflect infrared light and transmit visible light.

2. The wavelength-division imaging device according to claim 1, wherein, The wavelength division imaging device further includes a broadband achromatic collimator, which is installed in front of the light inlet of the mechanical housing to reduce the incident light beam.

3. The wavelength-division imaging device according to claim 1, wherein, The infrared camera and the visible light camera are respectively installed at a positive 45-degree angle and a negative 45-degree angle on the dichroic mirror.

4. The wavelength-division imaging device according to claim 1, wherein, The broadband window includes one of the following: a fluorite window and a diamond sheet coating.

5. The wavelength-division imaging device according to claim 1, wherein, The wavelength division imaging device also includes a bracket, which is installed inside the mechanical housing. The bracket is made of a material with high rigidity and low coefficient of thermal expansion and is used to fix the visible light camera, the infrared camera and the dichroic mirror.

6. An image registration and following method, applicable to the wavelength division imaging device as described in any one of claims 1-5, the image registration and following method comprising: Real-time reception of initial video frame pairs sent by wavelength division imaging equipment; Based on coordinate consistency, feature point matching processing is performed on the initial video frame pairs to generate a set of matching feature points; The matching feature point set is split to generate a basic control point set and a backup control point set; Based on the basic control point set and the backup control point set, the initial video frame pair is dynamically registered and followed to generate a registered video frame pair. The registered video frame pairs are sent to the associated video receiving terminal for display.

7. The image registration and following method according to claim 6, wherein, The step of performing feature point matching processing on the initial video frame pairs based on coordinate consistency to generate a set of matching feature points includes: Feature point detection processing is performed on each initial video frame in the initial video frame pair to generate a feature point group; Based on coordinate consistency, feature point matching processing is performed on each generated feature point group to generate a set of matched feature points. The matched feature point set is deduplicated to generate a deduplicated feature point set. A predetermined number of deduplicated feature point groups are selected from the deduplicated feature point group set as matching feature point groups to obtain the matching feature point group set.

Images