Super-resolution data set, model and image acquisition method, system, equipment and medium

Abstract

The invention provides a super-resolution data set, model and image acquisition method, system and device and a medium, and the method comprises the steps: obtaining a first low-resolution image and a second low-resolution image based on a first camera and a second camera which are equal in focal length and are disposed at two sides of a third camera at equal intervals; after a high-resolution image is obtained based on a third camera, a first virtual low-resolution image of a virtual camera sharing the same optical axis with the third camera is obtained based on the first low-resolution image and the second low-resolution image; a second virtual low-resolution image which is obtained by cutting the first virtual low-resolution image and has the same size as the high-resolution image and the high-resolution image form a first image pair; based on the designed infrared thermal imaging image acquisition structure, the problem of scene mismatching between high-resolution images and low-resolution images caused by time lapse and different spatial view angles is solved, and the construction quality of a super-resolution data set in a real scene is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared thermal imaging, and particularly to a super-resolution data set, a model, an image acquisition method, a system, a device, and a medium. Background Art

[0002] Infrared thermal imaging technology can be applied to many fields such as night vision observation, fire fighting and rescue, fire detection, and power detection. Infrared thermal imaging super-resolution technology, as a key technology that can improve the image quality and detail expressiveness of infrared thermal imaging devices, has become an object of enthusiastic research in the field. Infrared thermal imaging super-resolution technology usually uses a super-resolution model obtained by deep learning training, and the performance of the super-resolution model is often affected by the quality of the super-resolution data set used for training.

[0003] Currently, in the application scenario of infrared thermal imaging super-resolution, there are mainly two schemes for obtaining a super-resolution data set of a real scene: 1) using a zoom lens for two-fold zoom; 2) using multiple cameras for spatial alignment and intercepting high and low resolutions. However, the method of using a zoom lens for two-fold zoom cannot solve the influence of time on the scene, resulting in Figure 1 image mismatch and object deviation problems between the images at time T1 and time T2 as shown; at the same time, the method of using multiple cameras for spatial alignment and interception cannot solve the influence of spatial perspective on the scene, and there will also be Figure 2 image mismatch problems caused by the difference in the observation surfaces of Camera1 and Camera2 as shown. Therefore, there is an urgent need to provide a method for constructing a super-resolution data set that can improve the application effect of infrared thermal imaging super-resolution technology in the real world scene. Summary of the Invention

[0004] In order to solve the problems of the prior art, it is necessary to provide a super-resolution data set, a model, an image acquisition method, a system, a device, and a medium for the above technical problems.

[0005] In a first aspect, an embodiment of the present invention provides a method for obtaining a super-resolution data set, the method including:

[0006] Obtaining a first low-resolution image based on a first camera, obtaining a second low-resolution image based on a second camera, and obtaining a high-resolution image based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal;

[0007] Obtaining a first virtual low-resolution image based on the virtual camera based on the first low-resolution image and the second low-resolution image, wherein the virtual camera and the third camera share the same optical axis;

[0008] Based on the cropping of the first virtual low-resolution image, a second virtual low-resolution image is obtained, where the size of the second virtual low-resolution image is the same as that of the high-resolution image;

[0009] Based on the second virtual low-resolution image and the high-resolution image, a first image pair is formed.

[0010] In a second aspect, an embodiment of the present invention provides a method for obtaining a super-resolution network model, the method comprising:

[0011] Based on a first camera, a first low-resolution image sample is obtained, based on a second camera, a second low-resolution image sample is obtained, and based on a third camera, a high-resolution image sample is obtained; wherein the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal;

[0012] Based on the virtual camera parameters coaxial with the third camera, the first low-resolution image sample, and the second low-resolution image sample, a first virtual low-resolution image sample based on the virtual camera is obtained;

[0013] Based on the cropping of the first virtual low-resolution image sample, a second virtual low-resolution image sample is obtained, where the size of the second virtual low-resolution image sample is the same as that of the high-resolution image sample;

[0014] Based on the reconstruction processing of the second virtual low-resolution image sample by a preset super-resolution network, a high-resolution reconstructed image corresponding to the second virtual low-resolution image sample is obtained;

[0015] Based on the error calculation of the high-resolution reconstructed image and the high-resolution image sample by a preset image effective area template, the image reconstruction loss of the preset super-resolution network is obtained;

[0016] Based on the backpropagation update of the network parameters of the preset super-resolution network by the image reconstruction loss, the super-resolution network model is obtained.

[0017] In a third aspect, an embodiment of the present invention provides an image acquisition method, the method comprising:

[0018] Obtain a first result image; the first result image is a low-resolution image obtained based on a low-resolution camera;

[0019] Input the first result image into a super-resolution network model for processing to obtain a second result image, where the super-resolution network model is obtained through the following steps:

[0020] Obtain a first low - resolution image sample based on a first camera, obtain a second low - resolution image sample based on a second camera, and obtain a high - resolution image sample based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal;

[0021] Based on the virtual camera parameters co - axial with the third camera, the first low - resolution image sample, and the second low - resolution image sample, obtain a first virtual low - resolution image sample based on the virtual camera;

[0022] Based on the cropping of the first virtual low - resolution image sample, obtain a second virtual low - resolution image sample, wherein the size of the second virtual low - resolution image sample is the same as that of the high - resolution image sample;

[0023] Based on the reconstruction processing of the second virtual low - resolution image sample by a preset super - resolution network, obtain a high - resolution reconstructed image corresponding to the second virtual low - resolution image sample;

[0024] Based on the error calculation of the high - resolution reconstructed image and the high - resolution image sample using a preset image valid region template, obtain the image reconstruction loss of the preset super - resolution network;

[0025] Based on the back - propagation update of the network parameters of the preset super - resolution network using the image reconstruction loss, obtain the super - resolution network model.

[0026] In a fourth aspect, an embodiment of the present invention further provides a super - resolution dataset acquisition system, and the system includes:

[0027] A first image acquisition module, configured to obtain a first low - resolution image based on a first camera, obtain a second low - resolution image based on a second camera, and obtain a high - resolution image based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal;

[0028] A first virtual camera calibration module, configured to obtain a first virtual low - resolution image based on the virtual camera based on the first low - resolution image and the second low - resolution image, wherein the virtual camera is co - axial with the third camera;

[0029] A first virtual image generation module, configured to obtain a second virtual low - resolution image based on the cropping of the first virtual low - resolution image, wherein the size of the second virtual low - resolution image is the same as that of the high - resolution image;

[0030] An image pair acquisition module, configured to form a first image pair based on the second virtual low-resolution image and the high-resolution image.

[0031] In a fifth aspect, an embodiment of the present invention further provides a super-resolution network model acquisition system, where the system includes:

[0032] A second image acquisition module, configured to acquire a first low-resolution image sample based on a first camera, acquire a second low-resolution image sample based on a second camera, and acquire a high-resolution image sample based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal;

[0033] A second virtual camera calibration module, configured to obtain a first virtual low-resolution image sample based on a virtual camera based on virtual camera parameters coaxial with the third camera, the first low-resolution image sample, and the second low-resolution image sample;

[0034] A second virtual image generation module, configured to obtain a second virtual low-resolution image sample based on cropping of the first virtual low-resolution image sample, where the size of the second virtual low-resolution image sample is the same as that of the high-resolution image sample;

[0035] An image reconstruction processing module, configured to perform reconstruction processing on the second virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the second virtual low-resolution image sample;

[0036] A reconstruction loss calculation module, configured to calculate an error between the high-resolution reconstructed image and the high-resolution image sample based on a preset image valid area template to obtain an image reconstruction loss of the preset super-resolution network;

[0037] A model acquisition module, configured to perform backpropagation update on network parameters of the preset super-resolution network based on the image reconstruction loss to obtain the super-resolution network model.

[0038] In a sixth aspect, an embodiment of the present invention further provides an image acquisition system, where the system includes:

[0039] A third image acquisition module, configured to acquire a first result image; the first result image is a low-resolution image acquired based on a low-resolution camera;

[0040] An image processing module, configured to input the first result image into a super-resolution network model for processing to obtain a second result image, where the super-resolution network model is obtained through the following steps:

[0041] Obtain a first low-resolution image sample based on a first camera, obtain a second low-resolution image sample based on a second camera, and obtain a high-resolution image sample based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal;

[0042] Based on the virtual camera parameters coaxial with the third camera, the first low-resolution image sample, and the second low-resolution image sample, obtain a first virtual low-resolution image sample based on the virtual camera;

[0043] Based on the cropping of the first virtual low-resolution image sample, obtain a second virtual low-resolution image sample, wherein the size of the second virtual low-resolution image sample is the same as that of the high-resolution image sample;

[0044] Based on the reconstruction processing of the second virtual low-resolution image sample by a preset super-resolution network, obtain a high-resolution reconstructed image corresponding to the second virtual low-resolution image sample;

[0045] Based on the error calculation of the high-resolution reconstructed image and the high-resolution image sample using a preset image valid region template, obtain the image reconstruction loss of the preset super-resolution network;

[0046] Based on the backpropagation update of the network parameters of the preset super-resolution network using the image reconstruction loss, obtain the super-resolution network model.

[0047] In a seventh aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any of the above methods are implemented.

[0048] In an eighth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the above method are implemented.

[0049] Compared with the prior art, the present invention acquires high- and low-resolution image data of a real-world scene by designing an infrared thermal imaging image acquisition structure including a third camera and two low-resolution cameras with equal focal lengths and equally spaced on both sides of the third camera, and uses the low-resolution images collected by the two cameras to fuse and generate a low-resolution image corresponding to a virtual camera coaxial with the third camera, and then constructs a high-low resolution image pair with the high-resolution image data obtained by the third camera. This method can simultaneously solve the problem of scene mismatch between high- and low-resolution images caused by time lapse and different spatial perspectives, effectively improve the construction quality of the super-resolution data set in the real scene, and thus provide a reliable guarantee for the construction of a super-resolution network model with higher application effects. Description of the Drawings

[0050] Figure 1 is a schematic diagram of the effect of collecting images at consecutive moments using a zoom lens in the prior art;

[0051] Figure 2 is a schematic diagram of the effect of collecting images using multiple cameras in the prior art;

[0052] Figure 3 is a schematic flowchart of the method for obtaining a super-resolution data set in an embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of the deployment configuration of the first camera, the second camera and the third camera in an embodiment of the present invention;

[0054] Figure 5 is a schematic diagram of the construction effect of the virtual camera in an embodiment of the present invention;

[0055] Figure 6 is a schematic diagram of the common field of view area of the first camera and the second camera in an embodiment of the present invention;

[0056] Figure 7 is a schematic diagram of the invalid area that cannot be matched in the common field of view area of the first camera and the second camera in an embodiment of the present invention;

[0057] Figure 8 is a schematic diagram of the implementation principle of the stereo matching algorithm in an embodiment of the present invention;

[0058] Figure 9 is a schematic diagram of the principle of obtaining the position coordinates of the matching points in the first matching point set in the reference camera coordinate system in an embodiment of the present invention;

[0059] Figure 10 is a schematic diagram of the equivalent model of the virtual camera and the third camera in an embodiment of the present invention;

[0060] Figure 11It is a schematic diagram of the edge deformation of the high-resolution image and the second virtual low-resolution image in the embodiments of the present invention;

[0061] Figure 12 It is a schematic diagram of the high-resolution image and the second virtual low-resolution image after cropping the effective area in the embodiments of the present invention;

[0062] Figure 13 It is a schematic diagram of the effect of feature point registration on the cropped second low-resolution image and high-resolution image based on the image registration algorithm in the embodiments of the present invention;

[0063] Figure 14 It is a schematic diagram of the effect of optimizing the matching result obtained based on the image registration algorithm by using the random sample consensus algorithm in the embodiments of the present invention;

[0064] Figure 15 It is a schematic diagram of the scene alignment effect between the second virtual low-resolution image and the high-resolution image in the embodiments of the present invention;

[0065] Figure 16 It is a schematic diagram of the brightness alignment effect between the second virtual low-resolution image and the high-resolution image in the embodiments of the present invention;

[0066] Figure 17 It is a schematic diagram of the flow of the method for obtaining the super-resolution network model in the embodiments of the present invention;

[0067] Figure 18 It is a schematic diagram of the flow of the image acquisition method in the embodiments of the present invention;

[0068] Figure 19 It is a schematic diagram of the structure of the super-resolution data set acquisition system in the embodiments of the present invention;

[0069] Figure 20 It is a schematic diagram of the structure of the super-resolution network model acquisition system in the embodiments of the present invention;

[0070] Figure 21 It is a schematic diagram of the structure of the image acquisition system in the embodiments of the present invention. Detailed implementation manners

[0071] In order to make the objectives, technical solutions and beneficial effects of the present application clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the following described embodiments are part of the embodiments of the present invention and are only used to illustrate the present invention, but not to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0072] In some embodiments, such as Figure 3As shown, a method for obtaining a super-resolution dataset is provided, and the method includes:

[0073] S11. Obtain a first low-resolution image based on a first camera, obtain a second low-resolution image based on a second camera, and obtain a high-resolution image based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal.

[0074] The first camera, the second camera, and the third camera are three infrared thermal imaging cameras with the same lens resolution parameters. As Figure 4 shown, the only difference is that when actually used to collect high- and low-resolution image data, their arranged positions are different and the corresponding field of view ranges of the actually collected images are different, that is, the first camera, the third camera, and the second camera are arranged in sequence on a fixed plate, and the lens focal lengths of the first camera and the second camera are equal, and the lens focal length of the third camera is greater than the lens focal lengths of the first camera and the second camera. It should be noted that Figure 4 in, Sensor1, Sensor2, and Sensor3 respectively represent the first camera, the second camera, and the third camera. The distances from Sensor1 and Sensor2 to Sensor3 are equal. The lens focal lengths used by Sensor1 and Sensor2 are both f1, the lens focal length used by Sensor3 is f2 (f2 > f1), and the respective fields of view FOV (Field of View) corresponding to Sensor1, Sensor2, and Sensor3 are FOV Sensor1 , FOV Sensor2 , and FOV Sensor3 .

[0075] In actual image acquisition applications, the first camera, the second camera, and the third camera are synchronously triggered by a trigger signal provided by a computer to simultaneously acquire images. This method ensures that the image acquisition of the three cameras is consistent in time, effectively overcoming the problem of scene changes in the images acquired by different cameras caused by the time-interval acquisition method (such as the existing zoom method of acquiring pictures at focal length f1 at time t1 and at focal length f2 at time t2, where the scenes in the camera's field of view have changed at different times, resulting in inconsistent imaging results for sequential photos, such as moving cars, people, etc.). To facilitate obtaining the required first low-resolution image, second low-resolution image, and high-resolution image at low cost and high quality, in some embodiments, the focal length f2 of the third camera is 2n times the focal length f1 of the first camera, where n is a positive integer; the third camera, the first camera, and the second camera have the same height, the same attitude, and their lens foci are aligned. It should be noted that in the actual data acquisition process, multi-field-of-view data acquisition can be achieved by replacing the fixed-focus lens group. If two-fold super-resolution data needs to be acquired, a lens group pair with f2 = 2×f1 can be used, and if four-fold super-resolution data acquisition is required, a lens group pair with f2 = 4×f1 can also be selected.

[0076] It should be noted that before the first camera and the second camera acquire data, they need to be calibrated to obtain the accurate positional relationship between the two cameras to prepare for the construction of the subsequent virtual camera; during the data acquisition process, three images corresponding to FOV Sensor1 , FOV Sensor2 , and FOV Sensor3 can be obtained synchronously. Among them, the images corresponding to FOV Sensor1 and FOV Sensor2 are the first low-resolution image and the second low-resolution image; the image corresponding to FOV Sensor3 is used as the high-resolution image. After the acquisition is completed, the three images need to be post-processed through the following method steps to obtain a high-quality image pair that can be used for super-resolution network training.

[0077] S12. Based on the first low-resolution image and the second low-resolution image, obtain a first virtual low-resolution image based on a virtual camera, where the virtual camera shares the same optical axis as the third camera; among them, the virtual camera can be understood as a camera fictionalized based on the overlapping field-of-view regions in the first low-resolution image and the second low-resolution image, used to generate a first virtual low-resolution image that shares the same optical axis as the high-resolution image corresponding to the third camera to obtain a reliable high-low resolution image pair.

[0078] The equivalent layout position, field-of-view range (FOV VirtualSensor ) and focal length information (equal to the focal lengths of the first camera and the second camera) of the virtual camera are as Figure 5As shown, before generating the first virtual low-resolution image, it is necessary to obtain the internal and external parameter information of the virtual camera based on the internal and external parameter information of the first camera and the second camera; in some embodiments, based on the internal parameter matrix K1 and the external parameter matrix Pose1 of the first camera and the internal parameter matrix K2 and the external parameter matrix Pose2 of the second camera, the parameters of the virtual camera are obtained, and based on the parameters of the virtual camera, the first low-resolution image, and the second low-resolution image, the first virtual low-resolution image based on the virtual camera is obtained.

[0079] In practical applications, the internal parameter matrix K1 and the external parameter matrix Pose1 of the first camera and the internal parameter matrix K2 and the external parameter matrix Pose2 of the second camera can both be obtained by the pre-calibration method as described above. According to Figure 5 the virtual camera construction method shown, it is easy to obtain the internal parameter matrix K3 and the external parameter matrix Pose3 of the virtual camera based on the parameter information of the first camera and the second camera; in some embodiments, the step of obtaining the parameters of the virtual camera includes:

[0080] Based on the stereo vision calibration method, the internal parameter matrix K1 and the external parameter matrix Pose1 of the first camera and the internal parameter matrix K2 and the external parameter matrix Pose2 of the second camera are obtained; among them, the stereo vision calibration process can be understood as using a checkerboard infrared camera calibration board as the calibration object for stereo vision calibration, and the Zhang Zhengyou calibration method or other similar calibration methods are used in the calibration process to obtain the internal parameter matrix external parameter matrix where, f x and f y are the focal lengths in the x-axis direction and the y-axis direction of the camera respectively, c x and c y are the image center coordinates in the x-axis direction and the y-axis direction of the camera respectively, and R and t are the rotation matrix and the translation matrix from the second camera to the first camera respectively. That is, based on the above stereo vision calibration method, the internal parameter matrices K1 and K2 of the first camera Sensor1 and the second camera Sensor2, and the external parameter matrices Pose1 and Pose2 are obtained.

[0081] Based on the average value of the internal parameter matrix K1 of the first camera and the internal parameter matrix K2 of the second camera, the internal parameter matrix K3 of the virtual camera is obtained; that is, the matrix elements in the internal parameter matrix K3 of the virtual camera are the average values of the corresponding position elements of the internal parameter matrix K1 of the first camera and the internal parameter matrix K2 of the second camera.

[0082] Based on the average value of the external parameter matrix Pose1 of the first camera and the external parameter matrix Pose2 of the second camera, the external parameter matrix Pose3 of the virtual camera is obtained; that is, the matrix elements in the external parameter matrix Pose3 of the virtual camera are the average values of the corresponding position elements of the external parameter matrix Pose1 of the first camera and the external parameter matrix Pose2 of the second camera.

[0083] After obtaining the parameters (internal parameter matrix and external parameter matrix) of the virtual camera through the above method steps, according to the conversion relationship from spatial coordinates to pixel coordinates, a set of spatial points (the matching points in the first low-resolution image and the second low-resolution image in the first camera coordinate system) constructed by the first camera and the second camera can form the image of the virtual camera according to Equation (1):

[0084]

[0085] In the formula, is a spatial point P in the first camera coordinate system s as an example, is the homogeneous coordinate of the image pixel of the virtual camera, and R1 is the transformation matrix from the Pose1 of the first camera to the Pose3 of the virtual camera. The imaging effect of the corresponding virtual camera is as Figure 5 shown. The virtual camera simultaneously has partial perspectives of the first camera and the second camera, and can effectively solve the problem of spatial scene changes caused by the perspective in this part of the area. It should be noted that in practical applications, the mapping relationship between the spatial point coordinates in the second camera coordinate system and the virtual camera can also be established based on Equation (1).

[0086] As described above, in the process of constructing the image of the virtual camera, it is necessary to first perform stereo matching on the first low-resolution image generated by the first camera and the second low-resolution image generated by the second camera to obtain the matching point set in the first low-resolution image and the second low-resolution image; in some embodiments, the step of obtaining the first virtual low-resolution image based on the virtual camera parameters, the first low-resolution image, and the second low-resolution image includes:

[0087] Based on the stereo matching of the first low-resolution image and the second low-resolution image, the first matching point set of the first low-resolution image and the second low-resolution image is obtained; among them, stereo matching can be understood as a binocular matching method in the prior art; as Figure 6 shown, the green box area is the common field of view (FOV Overlap area) of the first camera and the second camera. Only within the common field of view can the three-dimensional scene be effectively restored through the stereo matching algorithm. When using FOV OverlapWhen performing binocular matching on a region and attempting to recover the spatial information of all pixels in this region, there are often Figure 7 unmatched regions (usually featureless and weakly textured) shown as the red region in Figure 7 ; the green region in Figure 8 is the valid matching region (with stronger texture and more features); therefore, it is necessary to use Figure 7 the stereo matching algorithm shown in

[0088] (such as the Semi-Global Block Matching algorithm) to find the matching points in the valid matching regions of the first low-resolution image and the second low-resolution image shown in s :

[0089]

[0090] In the formula, (X, Y, Z) are the spatial coordinates Point of the matching point P s , b is the baseline, f is the camera focal length, X L is the x-axis pixel of the corresponding matching point in the first low-resolution image, X R is the x-axis pixel of the corresponding matching point in the second low-resolution image, (x, y) are the pixel coordinates, and (x0, y0) are the image center coordinates; as shown in Figure 9 : Figure 9 Figure (a) in L describes the relationship between the Z coordinate and the disparity X R -X Figure 9 、b and f, and figure (b) in s describes the relationship in the first camera coordinate system. The X-axis, Y-axis, and Z-axis are the camera coordinate system of the first camera, (X, Y, Z) are the spatial point coordinates Point in the first camera coordinate system s , and (x, y) are the coordinates of (X, Y, Z) on the first low-resolution image. After obtaining the spatial point (X, Y, Z), according to Equation (1), using the internal and external parameter matrices of the virtual camera, as shown by the red arrow in Figure 8 , project the Point of each matching point P s onto the virtual camera, and the first virtual low-resolution image corresponding to the virtual camera FOV VirtualSensor can be generated.

[0091] It should be noted that the above spatial point coordinates Point s The (X, Y, Z) in the calculation formula (2) can also be the spatial point coordinates in the second camera coordinate system. That is, in practical applications, each matching point in the first matching point set can be converted into the spatial position coordinates in the first camera coordinate system according to requirements, and then based on the transformation matrix R1 from the first camera to the virtual camera, the coordinates of each matching point in the virtual camera space coordinate system can be obtained. Each matching point in the first matching point set can also be converted into the spatial position coordinates in the second camera coordinate system, and then based on the transformation matrix (R2) from the second camera to the virtual camera, the coordinates of each matching point in the virtual camera space coordinate system can be obtained.

[0092] Performing projection calculation on the position coordinates of each matching point in the first matching point set based on the internal parameter matrix K3 and the external parameter matrix Pose3 of the virtual camera to obtain the first virtual low-resolution image; among them, the process of performing projection calculation on the position coordinates of each matching point in the first matching point set can be understood as follows: first, based on the transformation matrix from the reference camera (the first camera or the second camera) to the virtual camera, the position coordinates of each matching point in the first matching point set in the reference camera coordinate system are used to obtain the coordinates of each matching point in the virtual camera space coordinate system, and then based on the internal parameter matrix K3 and the external parameter matrix Pose3 of the virtual camera, the coordinates in the virtual camera space coordinate system are projected onto the image plane of the virtual camera to obtain the pixel coordinates of each matching point in the image coordinate system of the virtual camera, and finally the required first virtual low-resolution image is obtained.

[0093] In some embodiments, the step of performing projection calculation on the position coordinates of each matching point in the first matching point set based on the internal parameter matrix K3 and the external parameter matrix Pose3 of the virtual camera to obtain the first virtual low-resolution image includes:

[0094] Based on the external parameter matrix of the virtual camera and the external parameter matrix of the corresponding camera in the reference camera coordinate system, a virtual camera pose transformation matrix is obtained; among them, the virtual camera pose transformation matrix is expressed as: [Pose ref -1 represents the inverse matrix of the external parameter matrix Pose ref of the first camera or the second camera, and Pose3 represents the external parameter matrix of the virtual camera.

[0095] Based on the virtual camera pose transformation matrix and the internal parameter matrix of the corresponding camera in the reference camera coordinate system, the mapping relationship between the image coordinates of the virtual camera and the spatial coordinates in the reference camera coordinate system is obtained; among them, the mapping relationship is expressed as: ​

[0096]

[0097] In the formula, is the spatial point coordinates in the reference coordinate system, is the homogeneous coordinates of the image pixels in the corresponding virtual camera, is the virtual camera pose transformation matrix.

[0098] Based on the mapping relationship between the image coordinates of the virtual camera and the spatial coordinates in the reference camera coordinate system, the projection calculation is performed on the position coordinates of each matching point in the first matching point set to obtain the homogeneous coordinates of each matching point on the image coordinate system of the virtual camera; that is, substituting the position coordinates of each matching point in the first matching point set into Equation (3) can obtain its homogeneous coordinates on the image coordinate system of the virtual camera.

[0099] Based on the homogeneous coordinates of each matching point in the first matching point set on the image coordinate system of the virtual camera, the first virtual low-resolution image is obtained.

[0100] S13. Based on the cropping of the first virtual low-resolution image, a second virtual low-resolution image is obtained, where the size of the second virtual low-resolution image is the same as that of the high-resolution image.

[0101] In an ideal situation (the lenses of each camera fully satisfy the pinhole imaging principle and there is no installation error), the virtual camera Virtual Sensor and the third camera Sensor3 can be equivalent to Figure 10 the model shown in the figure. The red is the virtual camera VirtualSensor, and the gray is the third camera Sensor3. The virtual camera and the third camera share the same optical axis, and at the same time, it satisfies that the focal length of the third camera (the same as the focal lengths of the first camera and the second camera) is an even multiple of the focal length of the virtual camera. If f2 = 2 × f1, then according to the camera model, it can be known that the field of view areas of the virtual camera and the third camera satisfy Equation (4):

[0102]

[0103] In the formula, FOV Sensor3 and FOV VirtualSensor respectively represent the field of view areas of the third camera and the virtual camera; f2 and f1 respectively represent the lens focal lengths of the third camera and the virtual camera.

[0104] According to the characteristics of the camera model shown in Equation (4), it can be known that the second virtual low-resolution image and the high-resolution image share the same optical axis. Based on taking the image corresponding to FOV Sensor3 as the high-resolution image, and at the same time taking the part in FOV VirtualSensor that corresponds to FOV Sensor3According to the principle that the partial region image corresponding to the image overlap is used as the low-resolution image, the part of the first virtual low-resolution image that overlaps with the high-resolution image is cropped to obtain the second virtual low-resolution image.

[0105] S14. Based on the second virtual low-resolution image and the high-resolution image, a first image pair is formed; among them, after the second virtual low-resolution image is obtained through the above method steps, it can be used as the low-resolution image in the first image pair, and paired with the high-resolution image obtained by the third camera to generate the corresponding high-low resolution image pair, so as to ensure that the high-resolution image and the low-resolution image have consistent spatial information and effectively solve the problem of image scene mismatch between the two.

[0106] The first image pair obtained through the above method steps can already effectively solve the problems of image mismatch and object deviation existing in the existing super-resolution datasets. However, considering that the virtual camera equivalent model proposed in the present invention may be affected by factors such as installation errors of the first camera and / or the second camera, incomplete consistency of poses, resulting in deviation of the optical axis center of the actual virtual image from the ideal center, lens distortion, etc., there may be a problem of small alignment deviation between the actually obtained high-resolution image and the second virtual low-resolution image. In order to improve the production quality of the image pair as much as possible, in some embodiments, the method further includes: performing scene alignment on the second virtual low-resolution image based on the high-resolution image to obtain a third virtual low-resolution image; based on the third virtual low-resolution image and the high-resolution image, a second image pair is formed; among them, the steps of the scene alignment include:

[0107] Performing feature point registration on the second low-resolution image and the high-resolution image based on an image registration algorithm to obtain a second set of matching points; among them, the process of obtaining the second set of matching points is: first, considering that there may be a lens distortion problem and typical distortion occurs at the image edge, such as Figure 11 in the images (a) and (b), there is a deformation phenomenon around the image area within the red rectangular frame, which will affect the feature point registration effect. First, crop the high-resolution image according to a preset edge cropping ratio (such as 10% of the image edge as the invalid area); then, crop the corresponding field of view area in the second low-resolution image according to the cropped high-resolution image to obtain Figure 12 the image areas shown in the images (a) and (b) (red rectangular frame areas) in Figure 12 in the image (c) is a two-fold enlarged effect diagram of the area within the red rectangular frame in the image (b); then, perform feature point registration on the cropped second low-resolution image and the cropped high-resolution image based on the image registration algorithm to obtain an initial set of matching points, such as Figure 13as shown (the circles are SIFT feature points, the left figure is the cropped high-resolution image, and the right figure is the cropped second-lowest resolution image); as Figure 13 shown by the colored lines in, there are a large number of feature points in the second matching point set directly obtained by the registration method and there are many incorrect matches. Then, the random sample consensus algorithm is used for optimization to find more accurate SIFT feature points and obtain the required second matching point set. As Figure 14 shown, there are basically no incorrect matching points. It should be noted that the above image registration algorithm can be selected according to actual application requirements. For example, feature extraction operators (Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF)) or deep learning registration algorithms (Super Point), etc. are not specifically limited here.

[0108] Based on the feature points in the second matching point set and the least squares method, an image transformation matrix is obtained; wherein, the expression satisfied by the image transformation matrix is as follows:

[0109]

[0110] In the formula, and respectively represent the feature points in the second matching point set located in the cropped second-lowest resolution image and the high-resolution image; A represents the transformation matrix from the second-lowest resolution image to the high-resolution image, which can be calculated by the least squares method based on the matching points in the second matching point set.

[0111] Based on the image transformation matrix and the second-lowest resolution image, the third virtual low-resolution image is obtained; wherein, the third virtual low-resolution image can be understood as the low-resolution image after scene alignment obtained by multiplying the above-obtained image transformation matrix A by the pixel matrix of the above-cropped second-lowest resolution image; the final alignment effect is as Figure 15 shown, the high-resolution image is on the left and the low-resolution image is on the right.

[0112] In addition, considering that in actual applications, due to different camera fields of view, the camera will perform automatic gain adjustment on the current scene, resulting in Figure 15 the image gray-scale distribution differences shown, such that the high-resolution image cannot be used as the standard target for the low-resolution image. In order to ensure the brightness consistency between the high- and low-resolution image pairs, in some embodiments, the method further includes: performing brightness alignment on the third virtual low-resolution image based on the high-resolution image to obtain a fourth virtual low-resolution image; and forming a third image pair based on the fourth virtual low-resolution image and the high-resolution image; wherein, the step of the brightness alignment includes:

[0113] Obtain the first average brightness and the first brightness standard deviation of the high-resolution image, and obtain the second average brightness and the second brightness standard deviation of the third virtual low-resolution image; wherein, the first average brightness and the second average brightness can be understood as the average brightness of all pixel points in the corresponding image, and the corresponding first brightness standard deviation and the second brightness standard deviation can be understood as the brightness standard deviation of all pixel points in the corresponding image. The specific calculation method can be implemented with reference to relevant existing technologies and will not be elaborated here.

[0114] Based on the ratio of the second brightness standard deviation of the third virtual low-resolution image to the first brightness standard deviation of the high-resolution image, obtain the average brightness correction coefficient of the third virtual low-resolution image; wherein, the average brightness correction coefficient can be expressed as Std LI / Std HI , and Std LI and Std HI represent the second brightness standard deviation and the first brightness standard deviation respectively.

[0115] Based on the product of the average brightness correction coefficient of the third virtual low-resolution image and the second average brightness of the third virtual low-resolution image, obtain the average brightness correction value corresponding to the third virtual low-resolution image; wherein, the average brightness correction value is expressed as (Std LI / Std HI )×L LI , and L LI represents the second average brightness.

[0116] Based on the difference between the first average brightness of the high-resolution image and the average brightness correction value of the third virtual low-resolution image, obtain the pixel brightness deviation correction value of the third virtual low-resolution image; wherein, the pixel brightness deviation correction value is expressed as (L HI -(Std LI / Std HI )×L LI ), and L HI represents the first average brightness.

[0117] Based on the product of the average brightness correction coefficient of the third virtual low-resolution image and the pixel value of each pixel point in the third virtual low-resolution image, obtain the pixel correction value of the third virtual low-resolution image; wherein, the pixel correction value of the third virtual low-resolution image is expressed as (Std LI / Std HI )×LImage Aligned,i , and LImage Aligned,i represents the pixel value of the i-th pixel point in the third virtual low-resolution image.

[0118] Based on the sum of the pixel correction values of each pixel point in the third virtual low-resolution image and the pixel point brightness deviation correction value, the fourth virtual low-resolution image is obtained; among them, the fourth virtual low-resolution image can be understood as the low-resolution image after the brightness alignment process, and the brightness alignment effect is as Figure 16 shown; the pixel value of each pixel point in the fourth virtual low-resolution image can be expressed as: (Std LI / Std HI )×LImage Aligned,i +(L HI -(Std LI / Std HI )×L LI ).

[0119] It should be noted that the method steps given in the above embodiments are only an elaboration of the production process of a pair of high and low resolution images. In practical applications, according to the sample size requirements in the super-resolution data set, the method provided by the present invention can be used to collect images of real-world scenes multiple times, and through the same image processing, the required image pairs can be obtained, and finally a super-resolution data set of real-world scenes with multiple image pairs can be constructed.

[0120] The method provided by the embodiments of the present invention for collecting high- and low-resolution image data of real-world scenes by designing an infrared thermal imaging image acquisition structure including a third camera and two low-resolution cameras with equal focal lengths and equally spaced on both sides of the third camera, and fusing the low-resolution images collected by the two cameras to generate a low-resolution image corresponding to a virtual camera coaxial with the third camera, and then constructing a high-low resolution image pair with the high-resolution image data obtained by the third camera can simultaneously solve the problem of scene mismatch between high- and low-resolution images caused by time lapse and different spatial perspectives, so that the finally generated high- and low-resolution image pairs simultaneously meet spatial consistency and brightness consistency, effectively improving the construction quality of the super-resolution data set in the real scene, and further providing a reliable guarantee for the construction of a super-resolution network model with higher application effects.

[0121] The super-resolution data set obtained based on the super-resolution data set construction method provided in the above embodiments can be used for model training of supervised super-resolution networks in deep learning to ensure that the super-resolution network model obtained by training can obtain high-resolution images that more conform to the real-world thermal imaging scene after super-resolution calculation and processing of low-resolution images. In some embodiments, as Figure 17 shown, a method for obtaining a super-resolution network model is provided, and the method includes:

[0122] S21. Obtain a first low-resolution image sample based on the first camera, obtain a second low-resolution image sample based on the second camera, and obtain a high-resolution image sample based on the third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal; it should be noted that the setting limitations and usage descriptions of the first camera, the second camera, and the third camera can refer to the relevant limitations in the aforementioned super-resolution dataset acquisition method. Similarly, the relevant descriptions of using the first camera, the second camera, and the third camera to respectively obtain the first low-resolution image sample, the second low-resolution image sample, and the high-resolution image sample can also refer to the relevant limitations of the first low-resolution image, the second low-resolution image, and the high-resolution image in the aforementioned super-resolution dataset acquisition method, which will not be elaborated here.

[0123] In some embodiments, the focal length f2 of the third camera is 2n times the focal length f1 of the first camera, where n is a positive integer; the third camera, the first camera, and the second camera have the same height, the same pose, and the lens foci are flush.

[0124] S22. Based on the virtual camera parameters coaxial with the third camera, the first low-resolution image sample, and the second low-resolution image sample, obtain a first virtual low-resolution image sample based on the virtual camera.

[0125] In some embodiments, based on the internal parameter matrix K1 and the external parameter matrix Pose1 of the first camera and the internal parameter matrix K2 and the external parameter matrix Pose2 of the second camera, obtain the parameters of the virtual camera.

[0126] In some embodiments, the steps of obtaining the parameters of the virtual camera include:

[0127] Based on the stereo vision calibration method, obtain the internal parameter matrix K1 and the external parameter matrix Pose1 of the first camera and the internal parameter matrix K2 and the external parameter matrix Pose2 of the second camera;

[0128] Based on the average value of the internal parameter matrix K1 of the first camera and the internal parameter matrix K2 of the second camera, obtain the internal parameter matrix K3 of the virtual camera;

[0129] Based on the average value of the external parameter matrix Pose1 of the first camera and the external parameter matrix Pose2 of the second camera, obtain the external parameter matrix Pose3 of the virtual camera.

[0130] In some embodiments, the steps of obtaining a first virtual low-resolution image sample based on the virtual camera parameters coaxial with the third camera, the first low-resolution image sample, and the second low-resolution image sample include:

[0131] Based on the stereo matching of the first low-resolution image sample and the second low-resolution image sample, a first sample matching point set between the first low-resolution image sample and the second low-resolution image sample is obtained;

[0132] Based on the first sample matching point set between the first low-resolution image sample and the second low-resolution image sample, the position coordinates of each matching point in the first sample matching point set in the reference camera coordinate system are obtained, where the reference camera coordinate system is the camera coordinate system of the first camera or the second camera;

[0133] Based on the projection calculation of the position coordinates of each matching point in the first sample matching point set by the internal parameter matrix K3 and the external parameter matrix Pose3 of the virtual camera, the first virtual low-resolution image sample is obtained.

[0134] In some embodiments, the step of obtaining the first virtual low-resolution image sample by performing projection calculation on the position coordinates of each matching point in the first sample matching point set based on the internal parameter matrix K3 and the external parameter matrix Pose3 of the virtual camera includes:

[0135] Based on the external parameter matrix of the virtual camera and the external parameter matrix of the corresponding camera in the reference camera coordinate system, a virtual camera pose conversion matrix is obtained;

[0136] Based on the virtual camera pose conversion matrix and the internal parameter matrix of the corresponding camera in the reference camera coordinate system, the mapping relationship between the image coordinates of the virtual camera and the spatial coordinates in the reference camera coordinate system is obtained;

[0137] Based on the mapping relationship between the image coordinates of the virtual camera and the spatial coordinates in the reference camera coordinate system, projection calculation is performed on the position coordinates of each matching point in the first sample matching point set to obtain the homogeneous coordinates of each matching point on the image coordinate system of the virtual camera;

[0138] Based on the homogeneous coordinates of each matching point in the first sample matching point set on the image coordinate system of the virtual camera, the first virtual low-resolution image sample is obtained.

[0139] It should be noted that all the implementation processes of the above virtual camera construction and obtaining the first virtual low-resolution image sample based on the virtual camera can refer to the relevant definitions of virtual camera construction and obtaining the first virtual low-resolution image in the aforementioned super-resolution dataset acquisition method, and will not be elaborated here.

[0140] S23. Obtain a second virtual low - resolution image sample based on the cropping of the first virtual low - resolution image sample, where the size of the second virtual low - resolution image sample is the same as that of the high - resolution image sample. It should be noted that the method for obtaining the second virtual low - resolution image sample can refer to the relevant description of the second virtual low - resolution image acquisition in the aforementioned super - resolution dataset acquisition method, which will not be elaborated here.

[0141] S24. Obtain a high - resolution reconstructed image corresponding to the second virtual low - resolution image sample based on the reconstruction process of the second virtual low - resolution image sample by a preset super - resolution network. Among them, the preset super - resolution network can be understood as a network model that can perform super - resolution calculation processing on a low - resolution image to obtain a higher - resolution image. No specific limitation is made here. The corresponding high - resolution reconstructed image can be understood as an image with a higher resolution generated by performing super - resolution reconstruction calculation on the input second virtual low - resolution image sample based on the preset super - resolution network. The specific super - resolution reconstruction calculation process varies depending on the network structure of the preset super - resolution network, and no specific limitation is made here.

[0142] S25. Calculate the image reconstruction loss of the preset super - resolution network based on the error calculation between the high - resolution reconstructed image and the high - resolution image sample using a preset image valid region template. Among them, the preset image valid region template can be understood as a mask template that defines the valid pixel region in the high - resolution reconstructed image and the high - resolution image sample. The pixel mask value of the valid pixel region is 1, and the pixel mask value of the invalid pixel region is 0. The corresponding image reconstruction loss can be expressed as:

[0143]

[0144] In the formula, L1 represents the absolute error loss between all high - resolution reconstructed images and the corresponding high - resolution image samples; HImage i and PImage i represent the gray - scale values of the i - th pixel in the high - resolution image sample and the corresponding high - resolution reconstructed image respectively; Mask i represents the pixel mask value of the i - th pixel in the preset image valid region template, where 1 indicates that the pixel is valid and 0 indicates that the pixel is invalid.

[0145] S26. Backpropagation update is performed on the network parameters of the preset super-resolution network based on the image reconstruction loss to obtain the super-resolution network model, that is, backpropagation is performed based on the result of the image reconstruction loss to update the network parameters of the preset super-resolution network. When the value of the loss function (image reconstruction loss) reaches stability and no longer decreases, the training of the preset super-resolution network ends, and the super-resolution network model can be obtained for super-resolution prediction (calculation) of low-resolution images.

[0146] In some embodiments, the method further includes: performing scene alignment on the second virtual low-resolution image sample based on the high-resolution image sample to obtain a third virtual low-resolution image sample; performing reconstruction processing on the third virtual low-resolution image sample based on the preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the third virtual low-resolution image sample. Among them, the steps of the scene alignment include:

[0147] Performing feature point registration on the second low-resolution image sample and the high-resolution image sample based on an image registration algorithm to obtain a second sample matching point set;

[0148] Obtaining a sample image transformation matrix based on the feature points in the second sample matching point set and the least squares method;

[0149] Obtaining the third virtual low-resolution image sample based on the sample image transformation matrix and the second low-resolution image sample.

[0150] It should be noted that the relevant description of the process of obtaining the third virtual low-resolution image sample in the embodiment can also refer to the relevant description of obtaining the third virtual low-resolution image in the foregoing super-resolution dataset obtaining method, which will not be elaborated here.

[0151] In some embodiments, the method further includes: performing brightness alignment on the third virtual low-resolution image sample based on the high-resolution image sample to obtain a fourth virtual low-resolution image sample; performing reconstruction processing on the fourth virtual low-resolution image sample based on the preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the fourth virtual low-resolution image sample. Among them, the steps of the brightness alignment include:

[0152] Obtaining the first sample average brightness and the first sample brightness standard deviation of the high-resolution image sample, and obtaining the second sample average brightness and the second sample brightness standard deviation of the third virtual low-resolution image sample;

[0153] Obtain the sample average brightness correction coefficient of the third virtual low-resolution image sample based on the ratio of the second sample brightness standard deviation of the third virtual low-resolution image sample to the first sample brightness standard deviation of the high-resolution image sample;

[0154] Obtain the sample average brightness correction value corresponding to the third virtual low-resolution image sample based on the product of the sample average brightness correction coefficient of the third virtual low-resolution image sample and the second sample average brightness of the third virtual low-resolution image sample;

[0155] Obtain the sample pixel point brightness deviation correction value of the third virtual low-resolution image sample based on the difference between the first sample average brightness of the high-resolution image sample and the sample average brightness correction value of the third virtual low-resolution image sample;

[0156] Obtain the sample pixel correction value of the third virtual low-resolution image sample based on the product of the sample average brightness correction coefficient of the third virtual low-resolution image sample and the pixel value of each pixel point in the third virtual low-resolution image sample;

[0157] Obtain the fourth virtual low-resolution image sample based on the sum of the sample pixel correction values of each pixel point in the third virtual low-resolution image and the sample pixel point brightness deviation correction value.

[0158] It should be noted that the relevant descriptions of the process for obtaining the fourth virtual low-resolution image sample in the embodiments can also refer to the relevant descriptions of obtaining the fourth virtual low-resolution image in the aforementioned method for obtaining a super-resolution data set, which will not be elaborated here.

[0159] The super-resolution network model obtained based on the method for obtaining a super-resolution network model provided in the above embodiments has strong robustness and accurate and reliable super-resolution prediction calculation capabilities, thereby effectively ensuring that the high-resolution image obtained after performing super-resolution calculation processing on the low-resolution image is more in line with the real-world thermal imaging scenario.

[0160] In some embodiments, as Figure 18 shown, an image acquisition method is provided, and the method includes:

[0161] S31. Obtain a first result image; the first result image is a low-resolution image obtained by a low-resolution camera;

[0162] S32. Input the first result image into a super-resolution network model for processing to obtain a second result image, where the super-resolution network model is obtained through the following steps:

[0163] Obtain a first low-resolution image sample based on a first camera, obtain a second low-resolution image sample based on a second camera, and obtain a high-resolution image sample based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal. In some embodiments, the focal length f2 of the third camera is 2n times the focal length f1 of the first camera, where n is a positive integer; the third camera, the first camera, and the second camera have the same height, the same pose, and the lens foci are flush.

[0164] Based on the virtual camera parameters coaxial with the third camera, the first low-resolution image sample, and the second low-resolution image sample, obtain a first virtual low-resolution image sample based on the virtual camera. In some embodiments, based on the internal parameter matrix K1 and the external parameter matrix Pose1 of the first camera and the internal parameter matrix K2 and the external parameter matrix Pose2 of the second camera, obtain the parameters of the virtual camera.

[0165] In some embodiments, the steps of obtaining the parameters of the virtual camera include:

[0166] Based on the stereo vision calibration method, obtain the internal parameter matrix K1 and the external parameter matrix Pose1 of the first camera and the internal parameter matrix K2 and the external parameter matrix Pose2 of the second camera;

[0167] Based on the average value of the internal parameter matrix K1 of the first camera and the internal parameter matrix K2 of the second camera, obtain the internal parameter matrix K3 of the virtual camera;

[0168] Based on the average value of the external parameter matrix Pose1 of the first camera and the external parameter matrix Pose2 of the second camera, obtain the external parameter matrix Pose3 of the virtual camera.

[0169] In some embodiments, the step of obtaining a first virtual low-resolution image sample based on the virtual camera parameters coaxial with the third camera, the first low-resolution image sample, and the second low-resolution image sample includes:

[0170] Based on the stereo matching of the first low-resolution image sample and the second low-resolution image sample, obtain a first sample matching point set of the first low-resolution image sample and the second low-resolution image sample;

[0171] Based on the first sample matching point set of the first low-resolution image sample and the second low-resolution image sample, obtain the position coordinates of each matching point in the first sample matching point set in the reference camera coordinate system, where the reference camera coordinate system is the camera coordinate system of the first camera or the second camera;

[0172] Performing projection calculations on the position coordinates of each matching point in the first sample matching point set based on the internal parameter matrix K3 and the external parameter matrix Pose3 of the virtual camera to obtain the first virtual low-resolution image sample.

[0173] In some embodiments, the step of performing projection calculations on the position coordinates of each matching point in the first sample matching point set based on the internal parameter matrix K3 and the external parameter matrix Pose3 of the virtual camera to obtain the first virtual low-resolution image sample includes:

[0174] Based on the external parameter matrix of the virtual camera and the external parameter matrix of the corresponding camera in the reference camera coordinate system, obtaining a virtual camera pose transformation matrix;

[0175] Based on the virtual camera pose transformation matrix and the internal parameter matrix of the corresponding camera in the reference camera coordinate system, obtaining the mapping relationship between the image coordinates of the virtual camera and the spatial coordinates in the reference camera coordinate system;

[0176] Performing projection calculations on the position coordinates of each matching point in the first sample matching point set based on the mapping relationship between the image coordinates of the virtual camera and the spatial coordinates in the reference camera coordinate system to obtain the homogeneous coordinates of each matching point on the image coordinate system of the virtual camera;

[0177] Based on the homogeneous coordinates of each matching point in the first sample matching point set on the image coordinate system of the virtual camera, obtaining the first virtual low-resolution image sample.

[0178] Based on the cropping of the first virtual low-resolution image sample, obtaining a second virtual low-resolution image sample, where the size of the second virtual low-resolution image sample is the same as that of the high-resolution image sample;

[0179] Based on the reconstruction processing of the second virtual low-resolution image sample by a preset super-resolution network, obtaining the high-resolution reconstructed image corresponding to the second virtual low-resolution image sample;

[0180] Based on the error calculation between the high-resolution reconstructed image and the high-resolution image sample using a preset image valid region template, obtaining the image reconstruction loss of the preset super-resolution network;

[0181] Based on the backpropagation update of the network parameters of the preset super-resolution network using the image reconstruction loss, obtaining the super-resolution network model.

[0182] In some embodiments, the method further includes: performing scene alignment on the second virtual low-resolution image sample based on the high-resolution image sample to obtain a third virtual low-resolution image sample; performing reconstruction processing on the third virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the third virtual low-resolution image sample; wherein the step of scene alignment includes:

[0183] Performing feature point registration on the second low-resolution image sample and the high-resolution image sample based on an image registration algorithm to obtain a second sample matching point set;

[0184] Obtaining a sample image transformation matrix based on the feature points in the second sample matching point set and the least squares method;

[0185] Obtaining the third virtual low-resolution image sample based on the sample image transformation matrix and the second low-resolution image sample.

[0186] In some embodiments, the method further includes: performing brightness alignment on the third virtual low-resolution image sample based on the high-resolution image sample to obtain a fourth virtual low-resolution image sample; performing reconstruction processing on the fourth virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the fourth virtual low-resolution image sample; wherein the step of brightness alignment includes:

[0187] Obtaining a first sample average brightness and a first sample brightness standard deviation of the high-resolution image sample, and obtaining a second sample average brightness and a second sample brightness standard deviation of the third virtual low-resolution image sample;

[0188] Obtaining a sample average brightness correction coefficient of the third virtual low-resolution image sample based on the ratio of the second sample brightness standard deviation of the third virtual low-resolution image sample to the first sample brightness standard deviation of the high-resolution image sample;

[0189] Obtaining a sample average brightness correction value corresponding to the third virtual low-resolution image sample based on the product of the sample average brightness correction coefficient of the third virtual low-resolution image sample and the second sample average brightness of the third virtual low-resolution image sample;

[0190] Obtaining a sample pixel point brightness deviation correction value of the third virtual low-resolution image sample based on the difference between the first sample average brightness of the high-resolution image sample and the sample average brightness correction value of the third virtual low-resolution image sample;

[0191] Obtaining a sample pixel correction value of the third virtual low-resolution image sample based on the product of the sample average brightness correction coefficient of the third virtual low-resolution image sample and the pixel value of each pixel point in the third virtual low-resolution image sample;

[0192] Obtaining the fourth virtual low-resolution image sample based on the sum of the sample pixel correction value of each pixel point in the third virtual low-resolution image and the sample pixel point brightness deviation correction value.

[0193] It should be noted that the specific training process of the super-resolution network model used to perform super-resolution calculation processing on the first result image (low-resolution image) collected in actual applications in the above image acquisition method can refer to the relevant description of the aforementioned super-resolution network model acquisition method, which will not be elaborated here. The second result image (high-resolution image) obtained based on the image acquisition method provided in the above embodiment has a higher resolution and is more in line with the real-world thermal imaging scenario.

[0194] It should be noted that although each step in the flowcharts of the above super-resolution dataset acquisition method, super-resolution network model acquisition method, and image acquisition method is sequentially shown according to the indication of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders.

[0195] In some embodiments, as Figure 19 shown, a super-resolution dataset acquisition system is provided, and the system includes:

[0196] A first image acquisition module 11, configured to obtain a first low-resolution image based on a first camera, obtain a second low-resolution image based on a second camera, and obtain a high-resolution image based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal;

[0197] A first virtual camera calibration module 12, configured to obtain a first virtual low-resolution image based on the virtual camera based on the first low-resolution image and the second low-resolution image, wherein the virtual camera and the third camera share the same optical axis;

[0198] A first virtual image generation module 13, configured to obtain a second virtual low-resolution image based on the cropping of the first virtual low-resolution image, wherein the size of the second virtual low-resolution image is the same as that of the high-resolution image;

[0199] An image pair acquisition module 14, configured to form a first image pair based on the second virtual low-resolution image and the high-resolution image.

[0200] In some embodiments, as Figure 20 shown, a super-resolution network model acquisition system is provided, and the system includes:

[0201] A second image acquisition module 21, configured to acquire a first low-resolution image based on a first camera, acquire a second low-resolution image based on a second camera, and acquire a high-resolution image based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal;

[0202] A second virtual camera calibration module 22, configured to obtain a first virtual low-resolution image based on a virtual camera based on virtual camera parameters coaxial with the third camera, the first low-resolution image, and the second low-resolution image;

[0203] A second virtual image generation module 23, configured to obtain a second virtual low-resolution image based on cropping the first virtual low-resolution image, wherein the size of the second virtual low-resolution image is the same as that of the high-resolution image;

[0204] An image reconstruction processing module 24, configured to perform reconstruction processing on the second virtual low-resolution image based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the second virtual low-resolution image;

[0205] A reconstruction loss calculation module 25, configured to calculate an error between the high-resolution reconstructed image and the high-resolution image based on a preset image valid region template to obtain an image reconstruction loss of the preset super-resolution network;

[0206] A model acquisition module 26, configured to perform backpropagation update on network parameters of the preset super-resolution network based on the image reconstruction loss to obtain the super-resolution network model.

[0207] In some embodiments, as Figure 21 shown, an image acquisition system is provided, and the system includes:

[0208] A third image acquisition module 31, configured to acquire a first result image; the first result image is a low-resolution image acquired based on a low-resolution camera;

[0209] An image processing module 32, configured to input the first result image into a super-resolution network model for processing to obtain a second result image, wherein the super-resolution network model is obtained through the following steps:

[0210] Obtain a first low-resolution image based on a first camera, obtain a second low-resolution image based on a second camera, and obtain a high-resolution image based on a third camera; wherein, the first camera and the second camera are arranged on both sides of the third camera at equal intervals, and the focal lengths of the first camera and the second camera are equal;

[0211] Based on the virtual camera parameters coaxial with the third camera, the first low-resolution image and the second low-resolution image, obtain a first virtual low-resolution image based on the virtual camera;

[0212] Based on the cropping of the first virtual low-resolution image, obtain the second virtual low-resolution image, wherein the size of the second virtual low-resolution image is the same as that of the high-resolution image;

[0213] Based on the reconstruction processing of the second virtual low-resolution image by a preset super-resolution network, obtain a high-resolution reconstructed image corresponding to the second virtual low-resolution image;

[0214] Based on the error calculation of the high-resolution reconstructed image and the high-resolution image using a preset image valid region template, obtain the image reconstruction loss of the preset super-resolution network;

[0215] Based on the backpropagation update of the network parameters of the preset super-resolution network using the image reconstruction loss, obtain the super-resolution network model.

[0216] It should be noted that for the specific limitations of the super-resolution dataset acquisition system, the super-resolution network model acquisition system, and the image acquisition system, reference can be made to the limitations of the super-resolution dataset acquisition method, the super-resolution network model acquisition method, and the image acquisition method in the above text respectively, and the corresponding technical effects can also be equivalently obtained, which will not be elaborated here. Each module in the above super-resolution dataset acquisition system, super-resolution network model acquisition system, and image acquisition system can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0217] In some embodiments, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that when the processor executes the computer program, the steps of the above method are implemented.

[0218] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0219] In summary, compared with the prior art, the super-resolution dataset, model, image acquisition method, system, device and medium provided by the embodiments of the present invention can collect high- and low-resolution image data of the real-world scene by designing an infrared thermal imaging image acquisition structure including a third camera and two low-resolution cameras with equal focal lengths and equally spaced on both sides of the third camera, and generate a low-resolution image corresponding to a virtual camera coaxial with the third camera by fusing the low-resolution images collected by the two cameras, and then construct a high-low resolution image pair by combining it with the high-resolution image data obtained by the third camera, which can simultaneously solve the problem of scene mismatch between high- and low-resolution images caused by time lapse and different spatial perspectives, effectively improve the construction quality of the super-resolution dataset in the real scene, effectively ensure the training effect of the super-resolution network model, and thus provide a strong technical support for providing accurate and reliable super-resolution prediction calculation results.

[0220] Each embodiment in this specification is described in a progressive manner. For the parts that are the same or similar in each embodiment, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0221] The above-described embodiments only represent several preferred embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and substitutions can be made, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the protection scope of the claims.

Claims

1. A method for acquiring a super-resolution data set, characterized in that: The method comprises: Acquire a first low-resolution image based on a first camera, acquire a second low-resolution image based on a second camera, and acquire a high-resolution image based on a third camera; wherein the first camera and the second camera are arranged at equal intervals on both sides of the third camera, and the focal lengths of the first camera and the second camera are equal; Based on the first low-resolution image and the second low-resolution image, a first virtual low-resolution image based on a virtual camera is obtained, wherein the virtual camera and the third camera share a common optical axis; Based on cropping the first virtual low-resolution image, a second virtual low-resolution image is obtained, wherein the second virtual low-resolution image has the same size as the high-resolution image; A first image pair is constructed based on the second virtual low-resolution image and the high-resolution image.

2. The method for obtaining a super-resolution data set according to claim 1, wherein: The focal length f2 of the third camera is 2n times the focal length f1 of the first camera, where n is a positive integer; the third camera, the first camera and the second camera have the same height, consistent posture and aligned lens focal points.

3. The method for obtaining a super-resolution data set according to claim 1, wherein: Based on the intrinsic parameter matrix K1 and extrinsic parameter matrix Pose1 of the first camera and the intrinsic parameter matrix K2 and extrinsic parameter matrix Pose2 of the second camera, the parameters of the virtual camera are obtained, and based on the parameters of the virtual camera, the first low-resolution image and the second low-resolution image, a first virtual low-resolution image based on the virtual camera is obtained.

4. The method for obtaining a super-resolution data set according to claim 3, wherein: The step of obtaining the parameters of the virtual camera includes: Based on the stereo vision calibration method, the intrinsic parameter matrix K1 and the extrinsic parameter matrix Pose1 of the first camera and the intrinsic parameter matrix K2 and the extrinsic parameter matrix Pose2 of the second camera are obtained; Obtaining an intrinsic parameter matrix K3 of the virtual camera based on an average value of the intrinsic parameter matrix K1 of the first camera and the intrinsic parameter matrix K2 of the second camera; The extrinsic parameter matrix Pose3 of the virtual camera is obtained based on the average value of the extrinsic parameter matrix Pose1 of the first camera and the extrinsic parameter matrix Pose2 of the second camera.

5. The method for obtaining a super-resolution data set according to claim 3 or 4, characterized in that: The step of obtaining a first virtual low-resolution image based on the virtual camera based on the parameters of the virtual camera, the first low-resolution image and the second low-resolution image comprises: Obtaining a first matching point set between the first low-resolution image and the second low-resolution image based on stereo matching of the first low-resolution image and the second low-resolution image; Based on a first matching point set of the first low-resolution image and the second low-resolution image, obtaining position coordinates of each matching point in the first matching point set in a reference camera coordinate system, wherein the reference camera coordinate system is a camera coordinate system of the first camera or the second camera; The first virtual low-resolution image is obtained by performing projection calculation on the position coordinates of each matching point in the first matching point set based on the intrinsic parameter matrix K3 and the extrinsic parameter matrix Pose3 of the virtual camera.

6. The method for obtaining a super-resolution data set according to claim 5, wherein: The step of performing projection calculation on the position coordinates of each matching point in the first matching point set based on the intrinsic parameter matrix K3 and the extrinsic parameter matrix Pose3 of the virtual camera to obtain the first virtual low-resolution image comprises: Based on the extrinsic matrix of the virtual camera and the extrinsic matrix of the camera corresponding to the reference camera coordinate system, a virtual camera posture conversion matrix is ​​obtained; Based on the virtual camera posture conversion matrix and the intrinsic parameter matrix of the camera corresponding to the reference camera coordinate system, a mapping relationship between the image coordinates of the virtual camera and the space coordinates in the reference camera coordinate system is obtained; Based on the mapping relationship between the image coordinates of the virtual camera and the spatial coordinates in the reference camera coordinate system, a projection calculation is performed on the position coordinates of each matching point in the first matching point set to obtain homogeneous coordinates of each matching point in the image coordinate system of the virtual camera; The first virtual low-resolution image is obtained based on the homogeneous coordinates of each matching point in the first matching point set in the image coordinate system of the virtual camera.

7. The method for obtaining a super-resolution data set according to claim 1, wherein: The method further includes: performing scene alignment on the second virtual low-resolution image based on the high-resolution image to obtain a third virtual low-resolution image; and forming a second image pair based on the third virtual low-resolution image and the high-resolution image; wherein the scene alignment step includes: Performing feature point registration on the second low-resolution image and the high-resolution image based on an image registration algorithm to obtain a second matching point set; Based on the feature points in the second matching point set and the least square method, an image transformation matrix is ​​obtained; The third virtual low-resolution image is obtained based on the image conversion matrix and the second low-resolution image.

8. The method for obtaining a super-resolution data set according to claim 7, wherein: The method further includes: performing brightness alignment on the third virtual low-resolution image based on the high-resolution image to obtain a fourth virtual low-resolution image; and forming a third image pair based on the fourth virtual low-resolution image and the high-resolution image; wherein the brightness alignment step includes: Acquire a first average brightness and a first brightness standard deviation of the high-resolution image, and acquire a second average brightness and a second brightness standard deviation of the third virtual low-resolution image; Obtaining an average brightness correction coefficient of the third virtual low-resolution image based on a ratio of a second brightness standard deviation of the third virtual low-resolution image to a first brightness standard deviation of the high-resolution image; Obtaining an average brightness correction value corresponding to the third virtual low-resolution image based on a product of an average brightness correction coefficient of the third virtual low-resolution image and a second average brightness of the third virtual low-resolution image; Obtaining a pixel brightness deviation correction value of the third virtual low-resolution image based on a difference between a first average brightness of the high-resolution image and an average brightness correction value of the third virtual low-resolution image; Obtaining a pixel correction value of the third virtual low-resolution image based on a product of an average brightness correction coefficient of the third virtual low-resolution image and a pixel value of each pixel point in the third virtual low-resolution image; The fourth virtual low-resolution image is obtained based on the sum of the pixel correction value of each pixel in the third virtual low-resolution image and the pixel brightness deviation correction value.

9. A method for obtaining a super-resolution network model, characterized in that: The method comprises: Acquire a first low-resolution image sample based on the first camera, acquire a second low-resolution image sample based on the second camera, and acquire a high-resolution image sample based on the third camera; wherein the first camera and the second camera are arranged at equal intervals on both sides of the third camera, and the focal lengths of the first camera and the second camera are equal; Obtaining a first virtual low-resolution image sample based on a virtual camera based on parameters of a virtual camera having a common optical axis with the third camera, the first low-resolution image sample, and the second low-resolution image sample; Based on the clipping of the first virtual low-resolution image sample, a second virtual low-resolution image sample is obtained, wherein the second virtual low-resolution image sample has the same size as the high-resolution image sample; Reconstructing the second virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the second virtual low-resolution image sample; Calculating the error between the high-resolution reconstructed image and the high-resolution image sample based on a preset image valid area template to obtain the image reconstruction loss of the preset super-resolution network; The super-resolution network model is obtained by back-propagating and updating the network parameters of the preset super-resolution network based on the image reconstruction loss.

10. The super-resolution network model acquisition method according to claim 9, characterized in that: The focal length f2 of the third camera is 2n times the focal length f1 of the first camera; the third camera, the first camera and the second camera have the same height, the same posture and the same lens focus.

11. The super-resolution network model acquisition method according to claim 9, characterized in that: The parameters of the virtual camera are obtained based on the intrinsic parameter matrix K1 and the extrinsic parameter matrix Pose1 of the first camera and the intrinsic parameter matrix K2 and the extrinsic parameter matrix Pose2 of the second camera.

12. The super-resolution network model acquisition method according to claim 11, characterized in that: The step of obtaining the parameters of the virtual camera includes: Based on the stereo vision calibration method, the intrinsic parameter matrix K1 and the extrinsic parameter matrix Pose1 of the first camera and the intrinsic parameter matrix K2 and the extrinsic parameter matrix Pose2 of the second camera are obtained; Obtaining an intrinsic parameter matrix K3 of the virtual camera based on an average value of the intrinsic parameter matrix K1 of the first camera and the intrinsic parameter matrix K2 of the second camera; The extrinsic parameter matrix Pose3 of the virtual camera is obtained based on the average value of the extrinsic parameter matrix Pose1 of the first camera and the extrinsic parameter matrix Pose2 of the second camera.

13. The super-resolution network model acquisition method according to claim 11 or 12, characterized in that: The step of obtaining a first virtual low-resolution image sample based on a virtual camera based on the virtual camera parameters having the same optical axis as the third camera, the first low-resolution image sample and the second low-resolution image sample comprises: Obtaining a first sample matching point set of the first low-resolution image sample and the second low-resolution image sample based on stereo matching of the first low-resolution image sample and the second low-resolution image sample; Based on a first sample matching point set of the first low-resolution image sample and the second low-resolution image sample, obtaining position coordinates of each matching point in the first sample matching point set in a reference camera coordinate system, wherein the reference camera coordinate system is a camera coordinate system of the first camera or the second camera; The first virtual low-resolution image sample is obtained by performing projection calculation on the position coordinates of each matching point in the first sample matching point set based on the intrinsic parameter matrix K3 and the extrinsic parameter matrix Pose3 of the virtual camera.

14. The super-resolution network model acquisition method according to claim 13, characterized in that: The step of performing projection calculation on the position coordinates of each matching point in the first sample matching point set based on the intrinsic parameter matrix K3 and the extrinsic parameter matrix Pose3 of the virtual camera to obtain the first virtual low-resolution image sample comprises: Based on the extrinsic matrix of the virtual camera and the extrinsic matrix of the camera corresponding to the reference camera coordinate system, a virtual camera posture conversion matrix is ​​obtained; Based on the virtual camera posture conversion matrix and the intrinsic parameter matrix of the camera corresponding to the reference camera coordinate system, a mapping relationship between the image coordinates of the virtual camera and the space coordinates in the reference camera coordinate system is obtained; Based on the mapping relationship between the image coordinates of the virtual camera and the spatial coordinates in the reference camera coordinate system, a projection calculation is performed on the position coordinates of each matching point in the first sample matching point set to obtain homogeneous coordinates of each matching point in the image coordinate system of the virtual camera; The first virtual low-resolution image sample is obtained based on the homogeneous coordinates of each matching point in the first sample matching point set in the image coordinate system of the virtual camera.

15. The super-resolution network model acquisition method according to claim 9, characterized in that: The method further includes: performing scene alignment on the second virtual low-resolution image sample based on the high-resolution image sample to obtain a third virtual low-resolution image sample; performing reconstruction processing on the third virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the third virtual low-resolution image sample; wherein the scene alignment step includes: Performing feature point registration on the second low-resolution image sample and the high-resolution image sample based on an image registration algorithm to obtain a second sample matching point set; Based on the feature points in the second sample matching point set and the least square method, a sample image conversion matrix is ​​obtained; The third virtual low-resolution image samples are obtained based on the sample image conversion matrix and the second low-resolution image samples.

16. The super-resolution network model acquisition method according to claim 15, characterized in that: The method further includes: performing brightness alignment on the third virtual low-resolution image sample based on the high-resolution image sample to obtain a fourth virtual low-resolution image sample; performing reconstruction processing on the fourth virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the fourth virtual low-resolution image sample; wherein the brightness alignment step includes: Acquire a first sample average brightness and a first sample brightness standard deviation of the high-resolution image samples, and acquire a second sample average brightness and a second sample brightness standard deviation of the third virtual low-resolution image samples; Obtaining a sample average brightness correction coefficient of the third virtual low-resolution image sample based on a ratio of a second sample brightness standard deviation of the third virtual low-resolution image sample to a first sample brightness standard deviation of the high-resolution image sample; Obtaining a sample average brightness correction value corresponding to the third virtual low-resolution image sample based on a product of a sample average brightness correction coefficient of the third virtual low-resolution image sample and a second sample average brightness of the third virtual low-resolution image sample; Obtaining a sample pixel brightness deviation correction value of the third virtual low-resolution image sample based on a difference between a first sample average brightness of the high-resolution image sample and a sample average brightness correction value of the third virtual low-resolution image sample; Obtaining a sample pixel correction value of the third virtual low-resolution image sample based on a product of a sample average brightness correction coefficient of the third virtual low-resolution image sample and a pixel value of each pixel point in the third virtual low-resolution image sample; The fourth virtual low-resolution image sample is obtained based on the sum of the sample pixel correction value of each pixel point in the third virtual low-resolution image and the sample pixel point brightness deviation correction value.

17. An image acquisition method, characterized in that: The method comprises: Acquire a first result image; the first result image is a low-resolution image to be processed acquired based on a low-resolution camera; The first result image is input into a super-resolution network model for processing to obtain a second result image, wherein the super-resolution network model is obtained by the following steps: Acquire a first low-resolution image sample based on the first camera, acquire a second low-resolution image sample based on the second camera, and acquire a high-resolution image sample based on the third camera; wherein the first camera and the second camera are arranged at equal intervals on both sides of the third camera, and the focal lengths of the first camera and the second camera are equal; Obtaining a first virtual low-resolution image sample based on a virtual camera based on parameters of a virtual camera having a common optical axis with the third camera, the first low-resolution image sample, and the second low-resolution image sample; Based on the clipping of the first virtual low-resolution image sample, a second virtual low-resolution image sample is obtained, wherein the second virtual low-resolution image sample has the same size as the high-resolution image sample; Reconstructing the second virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the second virtual low-resolution image sample; Calculating the error between the high-resolution reconstructed image and the high-resolution image sample based on a preset image valid area template to obtain the image reconstruction loss of the preset super-resolution network; The super-resolution network model is obtained by back-propagating and updating the network parameters of the preset super-resolution network based on the image reconstruction loss.

18. The image acquisition method according to claim 17, characterized in that: The focal length f2 of the third camera is 2n times the focal length f1 of the first camera, where n is a positive integer; the third camera, the first camera and the second camera have the same height, consistent posture and aligned lens focal points.

19. The image acquisition method according to claim 17, characterized in that: The parameters of the virtual camera are obtained based on the intrinsic parameter matrix K1 and the extrinsic parameter matrix Pose1 of the first camera and the intrinsic parameter matrix K2 and the extrinsic parameter matrix Pose2 of the second camera.

20. The image acquisition method according to claim 19, characterized in that: The step of obtaining the parameters of the virtual camera includes: Based on the stereo vision calibration method, the intrinsic parameter matrix K1 and the extrinsic parameter matrix Pose1 of the first camera and the intrinsic parameter matrix K2 and the extrinsic parameter matrix Pose2 of the second camera are obtained; Obtaining an intrinsic parameter matrix K3 of the virtual camera based on an average value of the intrinsic parameter matrix K1 of the first camera and the intrinsic parameter matrix K2 of the second camera; The extrinsic parameter matrix Pose3 of the virtual camera is obtained based on the average value of the extrinsic parameter matrix Pose1 of the first camera and the extrinsic parameter matrix Pose2 of the second camera.

21. The image acquisition method according to claim 19 or 20, characterized in that: The step of obtaining a first virtual low-resolution image sample based on a virtual camera based on the virtual camera parameters having the same optical axis as the third camera, the first low-resolution image sample and the second low-resolution image sample comprises: Obtaining a first sample matching point set of the first low-resolution image sample and the second low-resolution image sample based on stereo matching of the first low-resolution image sample and the second low-resolution image sample; Based on a first sample matching point set of the first low-resolution image sample and the second low-resolution image sample, obtaining position coordinates of each matching point in the first sample matching point set in a reference camera coordinate system, wherein the reference camera coordinate system is a camera coordinate system of the first camera or the second camera; The first virtual low-resolution image sample is obtained by performing projection calculation on the position coordinates of each matching point in the first sample matching point set based on the intrinsic parameter matrix K3 and the extrinsic parameter matrix Pose3 of the virtual camera.

22. The image acquisition method according to claim 21, characterized in that: The step of performing projection calculation on the position coordinates of each matching point in the first sample matching point set based on the intrinsic parameter matrix K3 and the extrinsic parameter matrix Pose3 of the virtual camera to obtain the first virtual low-resolution image sample comprises: Based on the extrinsic matrix of the virtual camera and the extrinsic matrix of the camera corresponding to the reference camera coordinate system, a virtual camera posture conversion matrix is ​​obtained; Based on the virtual camera posture conversion matrix and the intrinsic parameter matrix of the camera corresponding to the reference camera coordinate system, a mapping relationship between the image coordinates of the virtual camera and the space coordinates in the reference camera coordinate system is obtained; Based on the mapping relationship between the image coordinates of the virtual camera and the spatial coordinates in the reference camera coordinate system, a projection calculation is performed on the position coordinates of each matching point in the first sample matching point set to obtain homogeneous coordinates of each matching point in the image coordinate system of the virtual camera; The first virtual low-resolution image sample is obtained based on the homogeneous coordinates of each matching point in the first sample matching point set in the image coordinate system of the virtual camera.

23. The image acquisition method according to claim 17, characterized in that: The method further includes: performing scene alignment on the second virtual low-resolution image sample based on the high-resolution image sample to obtain a third virtual low-resolution image sample; performing reconstruction processing on the third virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the third virtual low-resolution image sample; wherein the scene alignment step includes: Performing feature point registration on the second low-resolution image sample and the high-resolution image sample based on an image registration algorithm to obtain a second sample matching point set; Based on the feature points in the second sample matching point set and the least square method, a sample image conversion matrix is ​​obtained; The third virtual low-resolution image samples are obtained based on the sample image conversion matrix and the second low-resolution image samples.

24. The image acquisition method according to claim 23, characterized in that: The method further includes: performing brightness alignment on the third virtual low-resolution image sample based on the high-resolution image sample to obtain a fourth virtual low-resolution image sample; performing reconstruction processing on the fourth virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the fourth virtual low-resolution image sample; wherein the brightness alignment step includes: Acquire a first sample average brightness and a first sample brightness standard deviation of the high-resolution image samples, and acquire a second sample average brightness and a second sample brightness standard deviation of the third virtual low-resolution image samples; Obtaining a sample average brightness correction coefficient of the third virtual low-resolution image sample based on a ratio of a second sample brightness standard deviation of the third virtual low-resolution image sample to a first sample brightness standard deviation of the high-resolution image sample; Obtaining a sample average brightness correction value corresponding to the third virtual low-resolution image sample based on a product of a sample average brightness correction coefficient of the third virtual low-resolution image sample and a second sample average brightness of the third virtual low-resolution image sample; Obtaining a sample pixel brightness deviation correction value of the third virtual low-resolution image sample based on a difference between a first sample average brightness of the high-resolution image sample and a sample average brightness correction value of the third virtual low-resolution image sample; Obtaining a sample pixel correction value of the third virtual low-resolution image sample based on a product of a sample average brightness correction coefficient of the third virtual low-resolution image sample and a pixel value of each pixel point in the third virtual low-resolution image sample; The fourth virtual low-resolution image sample is obtained based on the sum of the sample pixel correction value of each pixel point in the third virtual low-resolution image and the sample pixel point brightness deviation correction value.

25. A super-resolution data set acquisition system, characterized in that: The system comprises: A first image acquisition module, used to acquire a first low-resolution image based on a first camera, acquire a second low-resolution image based on a second camera, and acquire a high-resolution image based on a third camera; wherein the first camera and the second camera are arranged at equal intervals on both sides of the third camera, and the focal lengths of the first camera and the second camera are equal; A first virtual camera calibration module, configured to obtain a first virtual low-resolution image based on a virtual camera based on the first low-resolution image and the second low-resolution image, wherein the virtual camera and the third camera share a common optical axis; A first virtual image generating module, configured to obtain a second virtual low-resolution image based on cropping of the first virtual low-resolution image, wherein the second virtual low-resolution image has the same size as the high-resolution image; The image pair acquisition module is used to form a first image pair based on the second virtual low-resolution image and the high-resolution image.

26. A super-resolution network model acquisition system, characterized in that: The system comprises: A second image acquisition module, used to acquire a first low-resolution image sample based on the first camera, acquire a second low-resolution image sample based on the second camera, and acquire a high-resolution image sample based on the third camera; wherein the first camera and the second camera are arranged at equal intervals on both sides of the third camera, and the focal lengths of the first camera and the second camera are equal; A second virtual camera calibration module is used to obtain a first virtual low-resolution image sample based on a virtual camera based on parameters of a virtual camera having a common optical axis with the third camera, the first low-resolution image sample and the second low-resolution image sample; A second virtual image generation module, configured to obtain a second virtual low-resolution image sample based on cropping of the first virtual low-resolution image sample, wherein the second virtual low-resolution image sample has the same size as the high-resolution image sample; An image reconstruction processing module, configured to reconstruct the second virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the second virtual low-resolution image sample; A reconstruction loss calculation module, used for calculating the error between the high-resolution reconstructed image and the high-resolution image sample based on a preset image valid area template to obtain the image reconstruction loss of the preset super-resolution network; A model acquisition module is used to back-propagate and update the network parameters of the preset super-resolution network based on the image reconstruction loss to obtain the super-resolution network model.

27. An image acquisition system, characterized in that: The system comprises: A third image acquisition module is used to obtain a first result image; the first result image is a low-resolution image to be processed obtained based on a low-resolution camera; An image processing module is used to input the first result image into a super-resolution network model for processing to obtain a second result image, wherein the super-resolution network model is obtained by the following steps: Acquire a first low-resolution image sample based on the first camera, acquire a second low-resolution image sample based on the second camera, and acquire a high-resolution image sample based on the third camera; wherein the first camera and the second camera are arranged at equal intervals on both sides of the third camera, and the focal lengths of the first camera and the second camera are equal; Obtaining a first virtual low-resolution image sample based on a virtual camera based on parameters of a virtual camera having a common optical axis with the third camera, the first low-resolution image sample, and the second low-resolution image sample; Based on the clipping of the first virtual low-resolution image sample, a second virtual low-resolution image sample is obtained, wherein the second virtual low-resolution image sample has the same size as the high-resolution image sample; Reconstructing the second virtual low-resolution image sample based on a preset super-resolution network to obtain a high-resolution reconstructed image corresponding to the second virtual low-resolution image sample; Calculating the error between the high-resolution reconstructed image and the high-resolution image sample based on a preset image valid area template to obtain the image reconstruction loss of the preset super-resolution network; The super-resolution network model is obtained by back-propagating and updating the network parameters of the preset super-resolution network based on the image reconstruction loss.

28. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 24 are implemented.

29. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 24 are implemented.

Images