Image reconstruction method, training method, detection method, device and storage medium
By constructing a reconstruction network and using feature matching and template matching techniques to generate one-to-one pairs of low-to-high-resolution infrared images, the reconstruction network is trained, which solves the problem of insufficient resolution and clarity of infrared images and improves the accuracy of target detection.
Patent Information
- Application Number
- CN202111165805.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-30
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-09-30
AI Technical Summary
Existing technologies are insufficient to effectively improve the resolution and clarity of infrared images, resulting in poor infrared target detection performance.
By constructing a reconstruction network and utilizing feature matching and template matching techniques from training sample images, one-to-one pairs of low-resolution and high-resolution infrared images are generated, and the reconstruction network is trained to improve image reconstruction quality.
It improves the resolution and texture detail recovery of infrared images, enhances the accuracy of target detection, and solves the problems of reconstruction and detection of low-quality infrared images.
Smart Images

Figure CN114004742B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image reconstruction method, a training method, a detection method, a device and a storage medium. Background Art
[0002] Compared with visible light images, infrared images are not affected by illumination and occlusion, and therefore are widely used in scenarios such as intelligent transportation or line crossing alarms. However, it is not easy to obtain high-resolution and clear infrared images. Most common infrared images are blurry and have low clarity, resulting in poor results when performing subsequent operations (such as infrared target detection tasks). Therefore, it is necessary to improve the visual effects of infrared images, but the solutions adopted in the relevant technical solutions have poor improvement effects on infrared images. Summary of the invention
[0003] The present application provides an image reconstruction method, a training method, a detection method, a device and a storage medium, which can improve the quality of image reconstruction.
[0004] In order to solve the above technical problems, the technical solution adopted in the present application is: to provide an image reconstruction method, the method comprising: obtaining an infrared image to be processed; reconstructing the infrared image to be processed based on a reconstruction network to obtain a reconstructed infrared image; wherein the reconstruction network is trained based on infrared sample images, the infrared sample images include a first infrared training image and a second infrared training image, the content of the first infrared training image is the same as the content of the second infrared training image, the resolution of the first infrared training image is less than the resolution of the second infrared training image, and the field of view angle of the first infrared training image matches the field of view angle of the second infrared training image.
[0005] In order to solve the above technical problems, the technical solution adopted in the present application is: to provide a training method for a reconstruction network, the method comprising: obtaining a first infrared image and a second infrared image, the resolution of the first infrared image being smaller than the resolution of the second infrared image; performing feature matching on the first infrared image and the second infrared image to generate a third infrared image; based on the third infrared image, performing template matching processing on the first infrared image to obtain a first infrared training image in the first infrared image; based on the first infrared training image, processing the third infrared image to obtain a second infrared training image in the third infrared image that matches the first infrared training image; using training samples to train the reconstruction network to obtain a trained reconstruction network, the training samples comprising the first infrared training image and the second infrared training image.
[0006] In order to solve the above technical problems, the technical solution adopted in the present application is: to provide an infrared target detection method, the method comprising: obtaining an infrared image to be processed; reconstructing the infrared image to be processed based on a reconstruction network to obtain a reconstructed infrared image; performing target detection processing on the reconstructed infrared image to obtain a target detection result; wherein the reconstruction network is trained using the reconstruction network training method in the above technical solution.
[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an image processing device, which includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the image reconstruction method, reconstruction network training method or infrared target detection method in the above technical solution.
[0008] To solve the above technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the image reconstruction method, reconstruction network training method or infrared target detection method in the above technical solution.
[0009] Through the above scheme, the beneficial effect of the present application is: the reconstruction network is trained using training samples to obtain a trained reconstruction network, the infrared sample image includes a first infrared training image and a second infrared training image, the content of the first infrared training image is the same as the content of the second infrared training image, the resolution of the first infrared training image is less than the resolution of the second infrared training image, and the field of view of the first infrared training image matches the field of view of the second infrared training image; in practical applications, the acquired infrared image to be processed is input into the trained reconstruction network to obtain a reconstructed infrared image. Since the image first input into the reconstruction network is an image with the same content, different resolution and matching field of view, the problem of mismatching field of view of the image input into the reconstruction network can be overcome, so that the reconstruction network can better learn the features of high-resolution infrared images, so as to better reconstruct low-quality infrared images in actual use, improve the reconstruction effect of the reconstruction network, and while improving the resolution of low-quality infrared images, the texture details of low-quality infrared images and the contours of targets in the images can also be restored, thereby improving the accuracy of target detection in low-quality infrared images. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:
[0011] Figure 1 It is a flowchart of an embodiment of a training method for a reconstruction network provided by the present application;
[0012] Figure 2 It is a flowchart of another embodiment of the training method for reconstructing a network provided by the present application;
[0013] Figure 3 It is a flowchart of an embodiment of an image reconstruction method provided by the present application;
[0014] Figure 4 is a flowchart of another embodiment of the image reconstruction method provided by the present application;
[0015] Figure 5 It is a flow chart of an embodiment of an infrared target detection method provided by the present application;
[0016] Figure 6 is a structural schematic diagram of an embodiment of an image processing device provided by the present application;
[0017] Figure 7 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION
[0018] The present application is further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present application, but are not intended to limit the scope of the present application. Similarly, the following examples are only some embodiments of the present application rather than all embodiments, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0019] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0020] It should be noted that the terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.
[0021] A common technology for improving image resolution is super-resolution technology. Super-resolution technology improves image quality through software algorithms and includes two categories: 1) Traditional image super-resolution methods, including bilinear interpolation or sparse dictionary-based methods, which have relatively poor image texture restoration; 2) Neural network-based super-resolution methods, including Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN) or Residual Channel Attention Network (RCAN). These methods can effectively restore image texture details while improving image resolution. By using super-resolution technology to first improve the image quality of low-quality blurred infrared images, and then combining them with existing algorithms for subsequent processing, the effect of subsequent processing can be improved; for example, taking target detection as an example of subsequent processing, the accuracy of target detection can be greatly improved. Although some methods in related technologies have enhanced infrared images, for low-quality blurred infrared images, it is far from enough to simply enhance the image contrast and use the Sobel operator to enhance the texture details, because low-quality images have more noise and worse image quality. For target detection tasks, the generalization is poor and the recognition rate of targets of interest is low.
[0022] Based on the above problems, the present application provides a new quality improvement algorithm for blurry and low-quality infrared images. The low-quality, blurry infrared images are first preprocessed to improve the clarity of the infrared images and enhance the texture details in the images; the preprocessed infrared images are then input into a reconstruction network in order to train a network that can transform low-quality infrared images into high-quality infrared images, thereby improving the reconstruction effect of the reconstruction network and improving the effects of subsequent detection or recognition tasks. The technical solution adopted in the present application is described in detail below.
[0023] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a training method for a reconstruction network provided by the present application, the method comprising:
[0024] Step 11: Get training samples.
[0025] The reconstruction network is trained based on infrared sample images, which include a first infrared training image and a second infrared training image. The content of the first infrared training image is the same as that of the second infrared training image. The resolution of the first infrared training image is smaller than that of the second infrared training image. The field of view of the first infrared training image matches that of the second infrared training image. Specifically, in order to obtain training samples, the following method can be used: Figure 2 The solution shown is processed, which specifically includes the following steps:
[0026] Step 111: Acquire a first infrared sample image and a second infrared sample image.
[0027] An infrared camera can be used to shoot the current monitoring scene to obtain a first infrared sample image and a second infrared sample image, or the first infrared sample image and the second infrared sample image can be directly read from an image database; specifically, the first infrared sample image corresponds to the second infrared sample image, and the first infrared sample image and the second infrared sample image are images generated by shooting the same target in the same monitoring scene, but the resolutions of the two are different, and the resolution of the first infrared sample image is smaller than the resolution of the second infrared sample image, so the first infrared image can be considered to be a low resolution (LowResolution, LR) infrared image, and the second infrared image is a high resolution (High Resolution, LR) infrared image.
[0028] In a specific embodiment, the collected first infrared sample image and second infrared sample image may be cropped first, so that the resolution of the cropped first infrared sample image is 160*120, and the resolution of the cropped second infrared sample image is 640*480.
[0029] Step 112: performing feature matching on the first infrared sample image and the second infrared sample image to generate a third infrared sample image.
[0030] After obtaining the first infrared sample image and the second infrared sample image, feature extraction processing can be performed on the first infrared sample image and the second infrared sample image respectively to obtain corresponding features; then the features of the two are matched to generate a third infrared sample image, which corresponds to the second infrared sample image, that is, the third infrared sample image can be regarded as an image generated by transforming the second infrared sample image.
[0031] Step 113: Based on the third infrared sample image, perform template matching processing on the first infrared sample image to obtain a first infrared training image in the first infrared sample image.
[0032] After acquiring the third infrared sample image, template matching processing can be performed on the third infrared sample image and the first infrared sample image. For example, the resolutions of the first infrared sample image and the third infrared sample image are adjusted to be consistent, and then matched according to pixel values to find the area in the first infrared sample image that matches the third infrared sample image. The image where the area is located is the first infrared training image.
[0033] Step 114: Based on the first infrared training image, the third infrared sample image is processed to obtain a second infrared training image in the third infrared sample image that matches the first infrared training image.
[0034] After acquiring the first infrared training image, the size ratio of the first infrared sample image to the third infrared sample image can be calculated, and then the position of the first infrared training image in the first infrared sample image can be enlarged according to the size ratio to obtain the position of the second infrared training image in the third infrared sample image. The second infrared training image is cut out from the third infrared sample image according to the position, and the field of view angle of the first infrared training image is the same as the field of view angle of the second infrared training image.
[0035] Step 12: Use the training samples to train the reconstruction network to obtain a trained reconstruction network.
[0036] To train the reconstruction network, we can use Figure 2 The solution shown is processed, which specifically includes the following steps:
[0037] Step 115: Use a reconstruction network to reconstruct the first infrared training image and the second infrared training image to obtain a reconstructed sample image.
[0038] The reconstruction network is trained using training samples, each training sample includes a first infrared training image and a second infrared training image corresponding to the first infrared training image, the second infrared training image is a label image of the first infrared training image, the first infrared training image and the second infrared training image are input into the reconstruction network, the reconstruction network learns the features of the high-quality infrared image from the second infrared training image to adjust the low-quality infrared image (i.e., the second infrared training image) to generate a reconstructed sample image corresponding to the first infrared training image.
[0039] In a specific embodiment, the reconstruction network includes a feature extraction layer and a reconstruction layer. During the training process of the reconstruction network, the feature extraction layer is used to receive a first infrared training image and a second infrared training image, and perform feature extraction processing on the first infrared training image and the second infrared training image respectively to obtain a first extracted feature and a second extracted feature; the reconstruction layer is used to process the first extracted feature and the second extracted feature to obtain a reconstructed sample image. The algorithm adopted by the reconstruction network may include an interpolation-based method (for example: nearest neighbor interpolation method or bilinear interpolation method), a reconstruction-based method (for example: iterative back projection method, convex set projection method or maximum a posteriori probability method) or a learning-based method (for example: neighborhood embedding method or sparse coding method).
[0040] Step 116: Determine whether a preset training termination condition is met based on the reconstructed sample image.
[0041] After the reconstruction network outputs the reconstructed sample image, the loss between the reconstructed sample image and the second infrared training image can be calculated (recorded as the current loss); then the current loss is used to determine whether the end condition of the reconstruction network (i.e., the preset training end condition) is met. If the preset training end condition is met, it indicates that the training is ended and a trained reconstruction network is obtained.
[0042] Furthermore, the preset stopping conditions include: loss value convergence, that is, the difference between the previous loss value and the current loss value is less than the set value; determining whether the current loss value is less than the preset loss value, the preset loss value is a preset loss threshold, if the current loss value is less than the preset loss value, it is determined that the preset stopping condition is met; the number of training times reaches the set value (for example: training 10,000 times); or the accuracy obtained when testing using the test set reaches the set conditions (for example: exceeding the preset accuracy), etc.
[0043] Step 117: If the preset training termination condition is not met, the model parameters of the reconstruction network are adjusted.
[0044] If it is determined that the preset training termination condition is not met at present, the model parameters of the reconstruction network can be adjusted, and the model parameters include parameters of the feature extraction layer and the reconstruction layer. For example, taking the reconstruction layer including multiple convolutional layers as an example, the model parameters can be the number of convolutional layers or the size of the convolutional layers; and it is also necessary to return to the step of obtaining the first infrared sample image and the second infrared sample image, that is, execute step 111 until the preset training termination condition is met.
[0045] Since the higher the resolution of an image, the more perfect the texture details of the image, the present embodiment uses multiple training samples to train the reconstruction network so as to obtain a reconstruction network that can improve the image resolution; firstly, the acquired first infrared sample image and second infrared sample image are preprocessed by using an image preprocessing method to obtain a first infrared training image and a second infrared training image; then, the first infrared training image and the second infrared training image are input into the reconstruction network to train the reconstruction network. The present embodiment proposes a new construction method of LR infrared image and HR infrared image. In view of the problem that the content of the LR infrared image and the HR infrared image obtained by real sampling does not match due to the difference in the field of view angle, feature matching is first used to solve the problem of the field of view angle mismatch between the LR infrared image and the HR infrared image, and then the template matching method is used to determine the area with the same content in the LR infrared image and the HR infrared image, so as to obtain a one-to-one corresponding LR-HR infrared image pair (i.e., the first infrared training image and the second infrared training image), so as to use the LR-HR infrared image pair to train the reconstruction network. Since the data used for training is preprocessed first, a reconstruction network with better reconstruction quality can be obtained, thereby improving the quality of image reconstruction.
[0046] See also Figure 3 , Figure 3 : is a flow chart of an embodiment of an image reconstruction method provided by the present application, the method comprising:
[0047] Step 31: Obtain the infrared image to be processed.
[0048] An infrared camera can be used to shoot the current monitoring scene in real time to obtain an infrared image to be processed.
[0049] Step 32: Reconstruct the infrared image to be processed based on the reconstruction network to obtain a reconstructed infrared image.
[0050] The training process of the reconstruction network is as follows:
[0051] 1) Obtain training samples.
[0052] A first infrared sample image and a second infrared sample image are obtained, wherein the resolution of the first infrared sample image is smaller than the resolution of the second infrared sample image; feature matching is performed on the first infrared sample image and the second infrared sample image to generate a third infrared sample image; based on the third infrared sample image, template matching processing is performed on the first infrared sample image to obtain a first infrared training image in the first infrared sample image; based on the first infrared training image, the third infrared sample image is processed to obtain a second infrared training image in the third infrared sample image that matches the first infrared training image.
[0053] 2) Use the training samples to train the reconstruction network to obtain a trained reconstruction network.
[0054] A reconstruction network is used to reconstruct the first infrared training image and the second infrared training image to obtain a reconstructed sample image, and the reconstruction network can be a super-resolution network; based on the reconstructed sample image, it is determined whether a preset training termination condition is met; if the preset training termination condition is not met at present, the model parameters of the reconstruction network are adjusted, and the step of obtaining the first infrared sample image and the second infrared sample image is returned until the preset training termination condition is met to obtain a trained reconstruction network.
[0055] In a specific embodiment, the reconstruction network includes a feature extraction layer and a reconstruction layer. After the reconstruction network is trained, the feature extraction layer is used to receive the infrared image to be processed, perform feature extraction on the infrared to be processed, and obtain a third extracted feature; the reconstruction layer is used to reconstruct the third extracted feature to obtain a reconstructed infrared image.
[0056] To address the problem of content mismatch between the LR infrared image and the HR infrared image obtained by real sampling due to the difference in field of view angle, this embodiment proposes a new method for constructing a pair of LR infrared images and HR infrared images, which can greatly improve the reconstruction performance of the super-resolution network for the infrared images obtained by real sampling and increase the generalization of the reconstruction network.
[0057] See also Figure 4 , Figure 4 : is a flow chart of another embodiment of the image reconstruction method provided by the present application, the method comprising:
[0058] Step 401: Acquire a first infrared sample image and a second infrared sample image.
[0059] Step 401 is the same as step 111 in the above embodiment and will not be described again.
[0060] Step 402: performing feature extraction processing on the first infrared sample image and the second infrared sample image respectively to obtain a first sample feature and a second sample feature.
[0061] The first infrared sample image and the second infrared sample image are processed respectively by a feature extraction method to obtain a first sample feature and a second sample feature. The feature extraction method is a feature extraction method in the related art. For example, a feature extraction method based on ORB (Oriented FAST and Rotated Binary Robust Independent Elementary Features) is used to extract features from the LR infrared image and the HR infrared image to obtain a feature point set F of the LR infrared image. LR (i.e., the first sample feature) and the feature point set F of the HR infrared image HR (i.e. the second sample feature).
[0062] Step 403: Calculate the homography matrix between the first sample feature and the second sample feature, and use the homography matrix to perform affine transformation on the second infrared sample image to obtain a third infrared sample image.
[0063] The homography matrix H of the image is calculated based on the homography change of the image. The calculation formula is as follows:
[0064] F HR =H*F LR (1)
[0065] Through the homography matrix H, the second infrared sample image is affine transformed to obtain the transformed image I HR1 (i.e. the third infrared sample image), as shown below:
[0066] I HR1 =H*I HR0 (2)
[0067] Among them, I HR0 is the second infrared sample image.
[0068] Step 404: down-sample the third infrared sample image to obtain a fourth infrared sample image, and select a matching image from the fourth infrared sample image.
[0069] The resolution of the fourth infrared sample image is the same as that of the first infrared sample image, the matching image is located in the central area of the fourth infrared sample image, and the ratio of the total number of pixels of the matching image to the total number of pixels of the fourth infrared sample image is a preset ratio, which is a ratio set according to application requirements or experience, such as 80%, 90% or 95%.
[0070] In a specific embodiment, the third infrared sample image I HR1 Perform 4 times downsampling to obtain the fourth infrared sample image F DR , F DRThe resolution is 160*120, select F DR The image where the central 80% area is located is used as the grayscale matching template (i.e. matching image) G DR , G DR The resolution is 128*96.
[0071] Step 405: Based on the template matching algorithm of the normalized correlation coefficient, perform template matching processing on the matching image and the first infrared sample image to obtain the first infrared training image.
[0072] The matching image and the first infrared sample image are template matched with normalized correlation coefficients, and an image corresponding to the matching image (ie, the first infrared training image) is matched in the first infrared sample image. The resolution of the first infrared training image may be 128*96.
[0073] Step 406: Obtain position information of the first infrared training image in the first infrared sample image; and generate a second infrared training image based on the position information and the third infrared sample image.
[0074] The image corresponding to the first infrared training image in the third infrared sample image (i.e., the second infrared training image) is determined through the coordinate correspondence between the first infrared training image and the third infrared sample image. Specifically, the resolution ratio is first obtained; then the position information is processed based on the resolution ratio to obtain matching position information, the position information including the upper left corner position of the first infrared training image, the length of the first infrared training image, and the width of the first infrared training image; the image corresponding to the matching position information is selected from the third infrared sample image to obtain the second infrared training image, and the ratio of the resolution of the second infrared training image to the resolution of the first infrared training image is the resolution ratio, for example, the resolution of the second infrared training image may be 512*384.
[0075] Furthermore, the resolution ratio is the ratio of the resolution of the first infrared sample image to the resolution of the third infrared sample image, the resolution of the first infrared sample image is a1×b1, a1 is the number of pixels in the first infrared sample image in the vertical direction, and b1 is the number of pixels in the first infrared sample image in the horizontal direction; the resolution of the third infrared sample image is a2×b2, a2 is the number of pixels in the third infrared sample image in the vertical direction, and b2 is the number of pixels in the third infrared sample image in the horizontal direction. The infrared sample images have the same aspect ratio, that is, b1 / a1=b2 / a2; for example, assuming that the upper left corner position, width and length of the first infrared training image are [x, y], m and n respectively, the resolution of the first infrared sample image is 160*120, and the resolution of the third infrared sample image is 640*480, then the resolution ratio=640 / 160=480 / 120=4, the upper left corner position of the second infrared training image is [4x, 4y], the length of the second infrared training image is 4x, and the width of the second infrared training image is 4y.
[0076] Through the above-mentioned preprocessing operations, a one-to-one corresponding and completely matched LR infrared image (i.e., the first infrared training image) and HR infrared image (i.e., the second infrared training image) are finally obtained, which serve as the input of the image super-resolution reconstruction network, that is, the training samples include the first infrared training image and the second infrared training image.
[0077] Step 407: performing normalization processing on the first infrared training image and the second infrared training image respectively to obtain a third infrared training image and a fourth infrared training image.
[0078] The preprocessed LR infrared image (i.e., the first infrared training image) and the preprocessed HR infrared image (i.e., the second infrared training image) are input into the input layer of the reconstruction network. The resolution of the first infrared training image is 128*96, and the resolution of the second infrared training image is 512*384. The three RGB channels of the first infrared training image and the second infrared training image are uniformly normalized.
[0079] Step 408: Perform super-resolution processing on the third infrared training image to obtain a reconstructed sample image.
[0080] The third infrared training image is processed using the following steps to obtain a reconstructed sample image.
[0081] 1) Perform feature extraction processing on the third infrared training image to obtain a third sample feature.
[0082] The main function of the feature extraction layer is to extract texture detail features in the third infrared training image. The super-resolution (SR) algorithm model in the related technology has huge parameters and calculation amount, which takes too long for the subsequent target detection network and is not conducive to the deployment of the model. Therefore, the feature extraction layer provided in this embodiment adopts a lightweight convolutional neural network to extract features, such as MobileNet V1 and MobileNet V2, to obtain high and low frequency information in the third infrared training image by fusing the features from shallow to deep layers in the image, and finally fuse the features of the third infrared training image.
[0083] 2) Based on the sub-pixel convolutional network (ESPCN), the third sample feature is reconstructed to obtain a reconstructed sample image.
[0084] Since the resolution of the fourth infrared training image is an integer multiple of the resolution of the third infrared training image, this embodiment chooses to use sub-pixel convolution (for example: ESPCN) to convolve the features of the third infrared training image. ESPCN is a learnable upsampling module that obtains the features of low LR infrared images through several layers of convolutional networks, and then increases the resolution of the LR infrared image to the same as that of the HR infrared image through periodic screening to obtain an SR image (i.e., a reconstructed sample image).
[0085] Step 409: Calculate the loss between the reconstructed sample image and the fourth infrared training image to obtain a loss value, and determine whether the loss value converges.
[0086] If the loss value converges, it is determined that the preset training termination condition is met, and the training is stopped to obtain the trained reconstruction network; specifically, the LI loss between the reconstructed sample image and the fourth infrared training image can be calculated; the gradient loss between the reconstructed sample image and the fourth infrared training image can be calculated; the LI loss and the gradient loss are weighted summed to obtain the loss value, that is, the loss of the entire network is optimized by the L1 loss function and the gradient loss function, and the calculation method is as follows:
[0087]
[0088] Lg=[G(I SR )-G(I hR )] 2 (4)
[0089] Loss=Lg+L1 (5)
[0090] Among them, I HR is the fourth infrared training image, ISR To reconstruct the sample image, G(I SR ) is the gradient of the reconstructed sample image, G(I HR ) is the gradient of the fourth infrared training image, the Sobel operator can be used to calculate the gradient, Loss is the gradient value, L1 is the L1 loss, and Lg is the gradient loss.
[0091] It can be understood that the reconstruction effect of the image can also be measured by the Peak Signal to Noise Ratio (PSNR) or the Structural Similarity (SSIM) between the fourth infrared training image and the reconstructed sample image.
[0092] Step 410: If the loss value has not converged, the model parameters of the reconstruction network are adjusted.
[0093] If the loss value has not converged, the model parameters of the reconstruction network are adjusted, and the step of obtaining the first infrared sample image and the second infrared sample image is returned to until the preset training termination condition is met to obtain a trained reconstruction network.
[0094] Step 411: If the loss value converges, the infrared image to be processed is obtained, and the infrared image to be processed is reconstructed based on the reconstruction network to obtain a reconstructed infrared image.
[0095] Step 411 is the same as steps 31-32 of the above embodiment and will not be described again.
[0096] This embodiment uses a super-resolution reconstruction network to reconstruct low-quality infrared images, so that the image texture output from the reconstruction network is clearer, and the target contour is more prominent during subsequent target detection, which helps to improve the target detection effect. In addition, this embodiment uses ORB feature matching and affine transformation to solve the problem of mismatching the field of view between the LR infrared image and the HR infrared image, and then uses the regional grayscale template matching method to determine the area with the same content in the LR infrared image and the HR infrared image, and obtains a one-to-one corresponding LR-HR infrared image pair, which can greatly improve the reconstruction performance of the super-resolution network for the infrared image obtained by real sampling, and increase the generalization of the reconstruction network.
[0097] After the reconstructed infrared image is generated, it can be applied in multiple fields, such as target detection, recognition or tracking. The following is an example of the application in the target detection field. The low-quality blurry infrared target detection technology can be applied to the category detection or behavior analysis of multiple targets in the video, such as providing comprehensive protection for perimeters such as fences or walls. First, the super-resolution technology is used to improve the resolution and clarity of the collected infrared image sequence, and then the infrared target detection technology is used to determine the target of interest in the image, which provides a basis for subsequent target behavior analysis. The following is a detailed description.
[0098] See also Figure 5 , Figure 5 : is a flow chart of an embodiment of an infrared target detection method provided by the present application, the method comprising:
[0099] Step 51: Obtain the infrared image to be processed.
[0100] The image to be processed includes a target to be detected, and the target to be detected may be a person, other animals or an object.
[0101] Step 52: reconstruct the infrared image to be processed based on the reconstruction network to obtain a reconstructed infrared image.
[0102] The reconstruction network is trained using the reconstruction network training method in the above embodiment, and the specific training process will not be described in detail here.
[0103] Step 53: Perform target detection processing on the reconstructed infrared image to obtain a target detection result.
[0104] By analyzing the reconstructed infrared image, specific targets in the infrared image to be processed can be effectively detected. Specifically, the target detection network includes a feature extraction layer and a detection layer. The feature extraction layer is used to extract features from the reconstructed infrared image to obtain the current features; the detection layer is used to detect the current features to obtain the target detection results.
[0105] In a specific embodiment, the target detection network may be a YOLO (You Only Look Once) network. The YOLO network is a real-time target detection model that processes the reconstructed infrared image through a convolutional neural network to obtain the position and category of the target object in the reconstructed infrared image. Specifically, the following steps may be used to train the YOLO network:
[0106] 1) Construct a YOLO network.
[0107] The YOLO network includes an input layer, a feature extraction layer (including a first feature extraction layer and a second feature extraction layer), a detection layer, a classification layer, and a regression layer. The first feature extraction layer and the second feature extraction layer are lightweight convolutional neural networks.
[0108] 2) Input training samples.
[0109] The training samples contain low-resolution infrared images of pedestrians, vehicles or non-motor vehicles, and the positions and categories of the target objects are marked in the form of annotation boxes. The training samples are input into the reconstruction network to obtain a reconstructed image, the resolution of which can be 512*384, and the reconstructed image is input into the YOLO network.
[0110] 3) Extract feature maps.
[0111] The first feature extraction layer performs multiple convolution and downsampling processes on the reconstructed image output by the reconstruction network to obtain a first feature map. The size of the first feature map can be 64*48*192.
[0112] 4) Training of YOLO network.
[0113] The second feature extraction layer is connected to the first feature extraction layer, and is used to perform convolution and downsampling processing on the first feature map, and input it to the detection layer, which is a two-layer fully connected layer.
[0114] The low-quality blurred infrared target detection method provided in this embodiment is suitable for scenes such as outdoor fences or traffic intersections to detect targets such as pedestrians or vehicles; it can be understood that it can also be applied to other scenes, and is not limited to this.
[0115] The infrared target detection method provided in this embodiment is based on traditional image processing technology, deep learning technology and convolutional neural network, and mainly includes three modules: a preprocessing module, a super-resolution module and a target detection module, but the preprocessing module is only used in the training process of the reconstruction network; this embodiment constructs two independent neural networks: a reconstruction network and a target detection network; this embodiment designs a new, lightweight infrared image super-resolution convolutional neural network, which can reconstruct low-quality and blurred infrared images, improve the clarity of infrared images and enhance the texture details in the images, so that the texture of the input image of the YOLO network is clearer, the target contour is more prominent, and the performance of the target detection network is improved; the reconstruction network is combined with the YOLO network to realize target detection of the processed image, thereby improving the detection accuracy of infrared targets, especially far and small infrared targets, and can solve the problem of low detection accuracy of low-quality and blurred infrared samples by infrared target detection methods in traffic scenes. In addition, this embodiment produces real LR-HR infrared images to train the reconstruction network, and the obtained super-resolution reconstruction network has better generalization. Compared with the related technology of using a generative adversarial network to convert visible light images into infrared images, it can adapt to more low-quality infrared scenes without retraining, and can avoid the false detection problem caused by shade.
[0116] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of an image processing device provided by the present application. The image processing device 60 includes a memory 61 and a processor 62 connected to each other. The memory 61 is used to store a computer program. When the computer program is executed by the processor 62, it is used to implement the image reconstruction method, the reconstruction network training method or the infrared target detection method in the above-mentioned embodiment.
[0117] See also Figure 7 , Figure 7 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 70 is used to store a computer program 71. When the computer program 71 is executed by a processor, it is used to implement the image reconstruction method, the reconstruction network training method or the infrared target detection method in the above-mentioned embodiments.
[0118] The computer-readable storage medium 70 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other media that can store program codes.
[0119] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0122] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. An image reconstruction method, characterized in that: include: Acquire an infrared image to be processed; Reconstructing the infrared image to be processed based on the reconstruction network to obtain a reconstructed infrared image; The reconstruction network is trained based on infrared sample images, the infrared sample images include a first infrared training image and a second infrared training image, the second infrared training image is a label image of the first infrared training image, the content of the first infrared training image is the same as the content of the second infrared training image, the resolution of the first infrared training image is smaller than the resolution of the second infrared training image, and the field of view of the first infrared training image matches the field of view of the second infrared training image; The method of acquiring the first infrared training image and the second infrared training image includes: Acquire a first infrared sample image and a second infrared sample image, wherein a resolution of the first infrared sample image is smaller than a resolution of the second infrared sample image; Performing feature extraction processing on the first infrared sample image and the second infrared sample image respectively to obtain a first sample feature and a second sample feature; Calculating a homography matrix between the first sample feature and the second sample feature; transforming the second infrared sample image by using a homography matrix to obtain a third infrared sample image; Adjusting the resolution of the first infrared sample image and the third infrared sample image to be consistent, and then performing template matching processing on the matching images in the first infrared sample image and the third infrared sample image to find an area in the first infrared sample image that matches the matching image, and the image corresponding to the area is the first infrared training image; The position information of the first infrared training image in the first infrared sample image is obtained; and the second infrared training image is generated based on the position information and the third infrared sample image.
2. The image reconstruction method according to claim 1, characterized in that: The step of adjusting the resolution of the first infrared sample image and the third infrared sample image to be consistent, and then performing template matching processing on the matching images in the first infrared sample image and the third infrared sample image to find an area in the first infrared sample image that matches the matching image, wherein the image corresponding to the area is the first infrared training image, includes: Downsampling the third infrared sample image to obtain a fourth infrared sample image, wherein a resolution of the fourth infrared sample image is the same as a resolution of the first infrared sample image; Selecting a matching image from the fourth infrared sample image; Based on a template matching algorithm of a normalized correlation coefficient, template matching processing is performed on the matching image and the first infrared sample image to obtain the first infrared training image.
3. The image reconstruction method according to claim 2, characterized in that: The matching image is located in a central area of the fourth infrared sample image, and a ratio of a total number of pixels of the matching image to a total number of pixels of the fourth infrared sample image is a preset ratio.
4. The image reconstruction method according to claim 1, characterized in that: The step of generating the second infrared training image based on the position information and the third infrared sample image comprises: Acquire a resolution ratio, where the resolution ratio is a ratio of a resolution of the first infrared sample image to a resolution of the third infrared sample image; Processing the position information based on the resolution ratio to obtain matching position information; An image corresponding to the matching position information is selected from the third infrared sample image to obtain the second infrared training image, and a ratio of a resolution of the second infrared training image to a resolution of the first infrared training image is the resolution ratio.
5. The image reconstruction method according to claim 1, characterized in that: The method further comprises: Reconstructing the first infrared training image and the second infrared training image using the reconstruction network to obtain a reconstructed sample image; Determining whether a preset training termination condition is met based on the reconstructed sample image; If not, the model parameters of the reconstruction network are adjusted, and the step of obtaining the first infrared sample image and the second infrared sample image is returned to until the preset training termination condition is met to obtain the trained reconstruction network.
6. The image reconstruction method according to claim 5, characterized in that: The step of using the reconstruction network to reconstruct the first infrared training image and the second infrared training image to obtain a reconstructed sample image includes: Normalizing the first infrared training image and the second infrared training image respectively to obtain a third infrared training image and a fourth infrared training image; The third infrared training image is subjected to super-resolution processing to obtain the reconstructed sample image.
7. The image reconstruction method according to claim 6, characterized in that: The step of performing super-resolution processing on the third infrared training image to obtain the reconstructed sample image comprises: Performing feature extraction processing on the third infrared training image to obtain a third sample feature; Based on a sub-pixel convolutional network, the third sample feature is reconstructed to obtain the reconstructed sample image.
8. The image reconstruction method according to claim 7, characterized in that: The method further comprises: Calculating the loss between the reconstructed sample image and the fourth infrared training image to obtain a loss value; Determining whether the loss value converges; If so, it is determined that the preset training termination condition is met.
9. The image reconstruction method according to claim 8, characterized in that: The step of calculating the loss between the reconstructed sample image and the fourth infrared training image to obtain the loss value comprises: Calculating the LI loss between the reconstructed sample image and the fourth infrared training image; Calculating a gradient loss between the reconstructed sample image and the fourth infrared training image; The LI loss and the gradient loss are weightedly summed to obtain the loss value.
10. A training method for a reconstruction network, characterized in that: include: Acquire a first infrared sample image and a second infrared sample image, wherein a resolution of the first infrared sample image is smaller than a resolution of the second infrared sample image; Performing feature extraction processing on the first infrared sample image and the second infrared sample image respectively to obtain a first sample feature and a second sample feature; Calculating a homography matrix between the first sample feature and the second sample feature; transforming the second infrared sample image by using a homography matrix to obtain a third infrared sample image; Adjusting the resolution of the first infrared sample image and the third infrared sample image to be consistent, and then performing template matching processing on the matching images in the first infrared sample image and the third infrared sample image to find an area in the first infrared sample image that matches the matching image, and the image corresponding to the area is the first infrared training image; Acquire position information of the first infrared training image in the first infrared sample image; generating a second infrared training image based on the position information and the third infrared sample image; The reconstruction network is trained using training samples to obtain a trained reconstruction network, wherein the training samples include the first infrared training image and the second infrared training image, and the second infrared training image is a label image of the first infrared training image.
11. An infrared target detection method, characterized in that: include: Acquire an infrared image to be processed; Processing the infrared image to be processed based on the reconstruction network according to any one of claims 1 to 10 to obtain a reconstructed infrared image; Perform target detection processing on the reconstructed infrared image to obtain a target detection result.
12. An image processing device, characterized in that: It comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the image reconstruction method described in any one of claims 1 to 9, the training method of the reconstruction network described in claim 10, or the infrared target detection method described in claim 11.
13. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the image reconstruction method described in any one of claims 1 to 9, the training method of the reconstruction network described in claim 10, or the infrared target detection method described in claim 11.
Citation Information
Patent Citations
Super-resolution method of infrared image and computer readable storage medium
CN112184547A
Network model training method, face image super-resolution reconstruction method and equipment
CN112508782A
