A method for generating image super-resolution training data for traffic signs
By using adjustable degradation models in the super-resolution of traffic sign images to simulate various degradation factors in the real imaging process, high-quality training data is generated, which solves the problem of low recognition rate in the prior art and improves traffic sign images clarity and recognition accuracy.
Patent Information
- Application Number
- CN202410903731.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-08
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-08
AI Technical Summary
The prior art fails to fully consider various degradation factors in the real imaging process in the super-resolution of traffic sign images, resulting in low recognition rate, especially in the high-frequency details of text information in traffic sign images, which is prone to strokes. The problem of stickiness and morphology changes.
The adjustable degradation model is used to generate degraded images close to real scenes, including blur, interval sampling, downsampling, noise and JPEG compression, and high-quality traffic sign image super-resolution data are generated through deep learning network training.
It improves the clarity and detection accuracy of traffic sign images, improves the text recognition rate, enhances the super resolution effect and robustness of the network, and is suitable for traffic sign images of different degrees of degradation.
Smart Images

Figure CN118446900B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of traffic sign image super-resolution, and in particular to a method for generating image super-resolution training data for traffic signs. Background Art
[0002] Scene text recognition (STR) aims to identify text information in images. It is widely used in optical character recognition (OCR), ancient book restoration, autonomous driving and other scenarios, and is an important research direction in computer vision. However, the biggest challenge facing scene text recognition algorithms at this stage is still the low resolution of the input image and the low recognition rate caused by multiple degradation factors.
[0003] Image super-resolution algorithms can improve the resolution of input images and enhance image quality, and have been applied to many scenarios. Text image super-resolution is an important branch of this. While improving image resolution and image quality, it is also necessary to retain the original structure of text information as much as possible, thereby improving the recognition rate in subsequent recognition links. Existing text image super-resolution algorithms are implemented using deep neural networks, which require a large amount of training data for model training. The data set must contain pairs of high-resolution and low-resolution images with completely identical image content in order to establish an accurate relationship between the two. However, in practice, it is extremely difficult to obtain such paired image data. Existing methods use a difference algorithm on a high-resolution image to generate a low-resolution image, but this method does not fully consider the various degradation factors in the actual imaging process, and the super-resolution effect on real scene images is poor.
[0004] Most of the datasets used by existing image super-resolution methods are constructed by using algorithms such as bicubic downsampling on high-resolution images to generate corresponding low-resolution images. However, when constructing such datasets, only the single degradation factor of downsampling is simulated, and the actual imaging degradation process is not considered. As a result, the model method trained using the dataset has a good reconstruction effect for the low-resolution images generated by bicubic downsampling, but the reconstruction effect for the images collected in the real scene is poor.
[0005] The formation of low-resolution (LR) images involves the synergy of multiple degradation factors, including blur, noise, color distortion, geometric distortion and compression. Taking mobile phone photography as an example, the degradation characteristics of different brands (such as Apple and Samsung) are different, and it is difficult to describe them uniformly with a single model. In addition, the compression method of image files will also have a significant impact on image quality. Therefore, in order to highly restore the real scene, a more comprehensive degradation model needs to be constructed. The model should comprehensively consider multiple degradation factors, such as blur, downsampling, JPEG compression and camera noise, to more accurately simulate the image degradation process in real scenes.
[0006] Super-resolution of text images is of great value in the fields of traffic sign recognition and autonomous driving. However, they also only consider the degradation factor of bicubic downsampling, without considering more complex imaging degradation, and the super-resolution effect on real-world traffic sign images is limited. In addition, the text information in traffic sign images is different from natural scenes, with a more complex structure and higher requirements for high-frequency details of traffic sign images. Often due to blurring and downsampling in traffic sign images, the text cannot meet the requirements of the sampling theorem, and jagged aliasing effects appear on the edges of the strokes, resulting in problems with sticky strokes and morphological changes in the results of the text super-resolution algorithm, affecting the super-resolution effect. Summary of the invention
[0007] In view of the shortcomings of the prior art described above, the present invention provides a method for generating image super-resolution training data based on traffic signs. Traffic sign image super-resolution is performed on traffic sign images collected by intelligent driving cars, thereby improving the clarity and detection accuracy of traffic sign images. An adjustable degradation model is used to generate degraded images close to text information in real scenes for training deep learning network models.
[0008] To achieve the above effects, the technical solution of the present invention is as follows:
[0009] The present invention provides a method for training and generating a traffic sign image super-resolution, comprising the following steps:
[0010] Step 1: construct a degradation model, wherein the degradation model is used to generate a degraded image and a degraded sequence; wherein the degradation types of the degradation model include blur degradation, interval sampling, downsampling, noise and compression;
[0011] The degradation type depicted by the degradation model is expressed as:
[0012] (1)
[0013] In the formula, represents the degraded image obtained after the degradation model, U n, U1, U2, Um represent the degradation types of different degradation models, σ g Indicates whether to enable the degradation model. n Indicates the level, m Indicates the number of categories available for selection of degradation models; y HR represents a high-resolution, non-degraded traffic sign image;
[0014] Among them, blur degradation is expressed as:
[0015] Isotropic Gaussian blur is used to simulate the blur degradation process; isotropic Gaussian filter f ( i , j )∈[- k , k ] Elements are sampled from an isotropic Gaussian distribution, an isotropic Gaussian filter f ( i , j ) has a blur kernel with a length and width of 2 k +1, k is an integer; f ( i , j ) is expressed as:
[0016] (2)
[0017] In the formula, N represents the normalization constant, C represents the spatial coordinate, represents the covariance matrix, where:
[0018] (3)
[0019] In the formula, represents the blur standard deviation of the isotropic Gaussian filter;
[0020] In the isotropic Gaussian blur, k Randomly sample from integers in [2,10], that is, k ∈[2,10], then 2 k +1∈[5,21], blur standard deviation of isotropic Gaussian filter Randomly sample from [0.2, 1.1] to approximate the random blurring in the real world;
[0021] The interval sampling includes:
[0022] The pixels of the original traffic sign image are sampled every other row, i.e., interval sampling, to simulate the aliasing process of the original traffic sign image;
[0023] Step 2: disrupt the order of the degraded sequence and apply it to the original traffic sign image to generate a degraded image;
[0024] Step 3: Use the original traffic sign image and the degraded image pair as training data to train the traffic sign image super-resolution reconstruction network until the network converges.
[0025] Furthermore, step 1 includes:
[0026] Step 1.1: Only the isotropic Gaussian blur kernel is considered in the blur processing. At 2, 3, and 4 times super-resolution magnification, the blur kernel width range is set to [0.2, 2], [0.2, 3], and [0.2, 4] to construct a low-resolution traffic sign image with blur degradation;
[0027] Step 1.2: The degradation model uses bicubic interpolation to downsample the high-resolution traffic sign image. By constructing a sampling weight matrix, the 16 pixels adjacent to each pixel are used as sampling points, and the sampling step size is 2, 3, or 4 times the downsampling ratio.
[0028] Step 1.3: In the degradation model, this process is modeled by noise, and the traffic sign image is degraded using noise with a mean of 0 and a variance range of [0,15];
[0029] Step 1.4: Randomly crop the low-resolution traffic sign image and the corresponding high-resolution traffic sign image; the low-resolution traffic sign image block size is 48×48×3, and the corresponding high-resolution traffic sign image block sizes are 96×96×3, 144×144×3, and 192×192×3 at 2, 3, and 4 times super-resolution magnifications, respectively;
[0030] Step 1.5: Perform data augmentation on the traffic sign image blocks and expand the training data set by rotating and flipping them.
[0031] Furthermore, the down sampling includes bilinear sampling and bicubic sampling. The bilinear sampling is implemented by a dual interpolation method, and the bicubic sampling is implemented by a bicubic interpolation method.
[0032] Furthermore, the noise is randomly extracted from Gaussian noise and Poisson noise; for Gaussian noise, its probability density function f(x) is the probability density function of Gaussian distribution:
[0033] (4)
[0034] in, x represents noise, represents the noise expectation, represents the blur standard deviation of the isotropic Gaussian filter;
[0035] The distribution of the Poisson noise is described by Poisson distribution:
[0036] (5)
[0037] Among them, y represents the number of times an event occurs in a certain time or space, corresponding to the intensity of the image to be processed, and λ represents the average occurrence rate of events per unit time or space;
[0038] The standard deviation of Gaussian noise is set to the range of [1,3], and the intensity coefficient of Poisson noise is set to the range of [0.05,0.3].
[0039] Furthermore, in step 1, JPEG compression is used for compression, and the process of JPEG compression is divided into discrete cosine transform and quantization.
[0040] Furthermore, step 2 includes: using a degradation sequence to simulate a degradation process that the original traffic sign image is subjected to in reality; constructing a degradation sequence S1, where the degradation sequence S1 is a sequential connection of K traditional degradation sequences {B, D, N, J}, that is:
[0041] (6)
[0042] Among them, B stands for blur, D stands for downsampling consisting of interval sampling, bicubic interpolation, and downsampling, N stands for noise, and J stands for JPEG compression;
[0043] Considering the order of the degenerate sequence S1, rotational degradation is introduced into the degenerate sequence S1, and the degenerate factors in the degenerate sequence S1 are shuffled to obtain the shuffled degenerate sequence S2. Shuffle represents the random shuffle operation, R represents the rotational degradation, and the degenerate sequence S2 is expressed as:
[0044] S2=shuffle(S1+R)(7)
[0045] Random deletion is used for the degradation sequence S2: for each different degradation type in the degradation sequence S2, the first occurrence in the degradation sequence is retained and randomly deleted with probability p in the appearing degradation sequence S2 to generate a degradation sequence S3; the degradation sequence S3 is applied to the original traffic sign image to generate a low-resolution degraded traffic sign image.
[0046] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0047] In order to solve the problems of low image clarity when recognizing license plates and traffic signs in intelligent driving, which leads to jagged aliasing effects on the edges of text strokes and sticky and morphological changes of strokes after super-resolution, the present invention constructs a degradation model, generates low-resolution traffic sign images by simulating the real degradation process, and adds aliasing degradation for text, so that the generated low-resolution traffic sign images are closer to the degradation of real application scenarios, so as to increase network training images and improve the super-resolution effect and detection accuracy of the network. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a schematic diagram of the degradation model of the present invention;
[0049] Figure 2 It is a schematic diagram of interval sampling of the present invention;
[0050] Figure 3 Schematic diagram of an example of the random parameter scheme of the present invention. DETAILED DESCRIPTION
[0051] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention is also implemented or applied through other different specific embodiments. The details in this specification are also modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.
[0052] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0053] Example
[0054] This embodiment proposes a method for generating image super-resolution training data for traffic signs. Figure 1 , including the following steps:
[0055] The images people see go through multiple degradation processes. For example, when posting captured images to Moments, the images are degraded when the camera captures the scene and form an image. The images are also degraded when they are transferred to a computer. When posted to Moments, the images are compressed and degraded again.
[0056] In view of the above process, the present invention proposes an adjustable degradation model (ADM) for traffic sign images. The adjustable degradation model aims to cover degradation instances in real scenes as much as possible by generating a variety of degradation combinations. The adjustable degradation model introduces a random parameter mechanism and high-order degradation. By repeating low-order degradation multiple times, it can more accurately simulate the repeated degradation process in real scenes, enhance the randomness of the degradation process, and make the degradation simulation closer to the actual situation. In addition, the adjustable degradation model introduces an interval sampling link to simulate the aliasing degradation of text information in traffic sign images, and expands the low-order degradation combination and introduces high-order degradation to more accurately simulate the complex degradation conditions in real scenes.
[0057] Step 1: construct a degradation model, wherein the degradation model is used to generate a degraded image and a degraded sequence; wherein the degradation types of the degradation model include blur degradation, interval sampling, downsampling, noise and compression;
[0058] The degradation type depicted by the degradation model is expressed as:
[0059] (1)
[0060] In the formula, represents the degraded image obtained after the degradation model, U n , U1, U2, Um represent the degradation types of different degradation models, σ g Indicates whether to enable the degradation model. n Indicates the level, m Indicates the number of categories available for selection of degradation models; y HR represents a high-resolution, non-degraded traffic sign image;
[0061] In the present invention, "degradation" and "degradation" have the same meaning, and the "degradation model" is composed of a sequence of multiple "degradation mechanisms" or "degradation factors". The "degradation mechanism" is a single factor, and the "degradation model" is an overall mathematical model. The present invention uses different degradation categories for testing, in which an interval sampling operation is performed before downsampling. By combining multiple groups of different degradation methods, a high degree of simulation of complex degradation processes in real scenes is achieved. At the same time, the introduction of the random parameter mechanism makes the model more flexible and changeable in the combination of different basic degradation types, supporting the random selection and application of first-order, second-order and even higher-order degradation, thereby more accurately simulating the degradation of real scenes.
[0062] Among them, blur degradation is expressed as:
[0063] Isotropic Gaussian blur is used to simulate the blur degradation process; isotropic Gaussian filter f (i , j )∈[- k , k ] Elements are sampled from an isotropic Gaussian distribution, an isotropic Gaussian filter f ( i , j ) has a blur kernel with a length and width of 2 k +1, k is an integer; f ( i , j ) is expressed as:
[0064] (2)
[0065] In the formula, N represents the normalization constant, C represents the spatial coordinate, represents the covariance matrix, where:
[0066] (3)
[0067] In the formula, represents the blur standard deviation of the isotropic Gaussian filter;
[0068] In the isotropic Gaussian blur, k Randomly sample from integers in [2,10], that is, k ∈[2,10], then 2 k +1∈[5,21], blur standard deviation of isotropic Gaussian filter Randomly sample from [0.2, 1.1] to approximate the random blurring in the real world;
[0069] The interval sampling includes:
[0070] like Figure 2 As shown, in the field of signal processing, aliasing originates from the high-frequency signal higher than half of the sampling frequency being mapped to the low-frequency part of the signal, superimposed with the original low-frequency signal, and affecting the integrity and accuracy of the signal. Aliasing will cause the text information in the image to be degraded, such as the strokes sticking together and the edges being jagged. In the present invention, the pixels of the original traffic sign image are sampled every other row, that is, interval sampling (the sampling interval is 1), to simulate the aliasing process of the original traffic sign image;
[0071] Step 2: disrupt the order of the degraded sequence and apply it to the original traffic sign image to generate a degraded image;
[0072] As a preferred technical solution, in this embodiment, step 1 includes:
[0073] Step 1.1: Only the isotropic Gaussian blur kernel is considered in the blurring process. At 2, 3, and 4 times super-resolution magnification, the blur kernel width range is set to [0.2, 2], [0.2, 3], and [0.2, 4] to construct a low-resolution image with blur degradation.
[0074] Step 1.2: The degradation model uses bicubic interpolation to downsample the high-resolution image. By constructing a sampling weight matrix, the 16 pixels adjacent to each pixel are used as sampling points, and the sampling stride size is 2, 3, or 4 times the downsampling ratio.
[0075] Step 1.3: In the degradation model, this process is modeled by noise, and the image is degraded using noise with a mean of 0 and a variance range of [0,15];
[0076] Step 1.4: Randomly crop the low-resolution image and the corresponding high-resolution image; the low-resolution image block size is 48×48×3, and the corresponding high-resolution image block sizes are 96×96×3, 144×144×3, and 192×192×3 at 2, 3, and 4 times super-resolution magnifications, respectively;
[0077] Step 1.5: Perform data augmentation on the image patches to expand the training dataset by rotating and flipping them.
[0078] As a preferred technical solution, in this embodiment, the downsampling includes bilinear sampling and bicubic sampling. The bilinear sampling is implemented by a dual interpolation method, and the bicubic sampling is implemented by a bicubic interpolation method.
[0079] Downsampling: The image may undergo multiple size changes during the acquisition and storage process, and each pixel needs to be downsampled each time. In the degradation process of the present invention, multiple random downsampling is achieved by randomly selecting from the following two downsampling methods:
[0080] (1) Bilinear sampling
[0081] Bilinear interpolation calculates the value of the interpolated pixel by taking a weighted average of the four pixels around the interpolated pixel; these four pixels are the pixel values of the upper left corner (x1, y1), the upper right corner (x2, y1), the lower left corner (x1, y2), and the lower right corner (x2, y2), where (x1, y1) and (x2, y2) are the coordinates of the two pixels closest to the interpolation point, and (x1, y2) and (x2, y1) are the coordinates of the two pixels farthest from the interpolation point;
[0082] (2) Bicubic sampling
[0083] The basic idea of bicubic sampling is to represent the value of each pixel as the weighted average of its surrounding 16 pixels, where the weight of each pixel is calculated by a cubic interpolation function.
[0084] As a preferred technical solution, in this embodiment, the noise is randomly extracted from Gaussian noise and Poisson noise; Gaussian noise and Poisson noise are respectively extracted by N G , N P express;
[0085] Gaussian noise is caused by random errors in the process of random photons arriving or transmitting signals in the image sensor; for Gaussian noise N G , whose probability density function f ( x ) is the probability density function of the Gaussian distribution:
[0086] (4)
[0087] in, x represents noise, represents the noise expectation (usually set to 0), represents the blur standard deviation of the isotropic Gaussian filter;
[0088] The Poisson noise N p It is a kind of random noise, also called counting noise. Poisson noise is caused by the random arrival of photons sampled by the image sensor during the process of signal measurement or sampling, which produces random errors in the image signal. The characteristic of Poisson noise is that the variance is equal to the mean of the signal, that is, the intensity of the noise varies with the intensity of the signal. The distribution of Poisson noise is described by Poisson distribution:
[0089] (5)
[0090] Among them, y represents the number of events that occur in a certain time or space, corresponding to the intensity of the image to be processed, λ represents the average occurrence rate of events per unit time or space, corresponding to the average intensity of the image; and N G Similarly, Poisson noise N P It is independent at each pixel;
[0091] When Poisson noise is added to the degradation model, the noise is randomly selected from Gaussian noise and Poisson noise; the standard deviation range of Gaussian noise is set to [1,3], the intensity coefficient range of Poisson noise is [0.05,0.3], and the two noise parameters are randomly sampled from the corresponding ranges.
[0092] As a preferred technical solution, in this embodiment, most image storage formats and images uploaded to the Internet will compress the image; JPEG (Joint Photographic Experts Group) is a common image compression format, and step 2 compression uses JPEG compression. The JPEG compression process is divided into discrete cosine transform (DCT) and quantization:
[0093] (1) Discrete cosine transform
[0094] Discrete cosine transform is the first step of JPEG compression, which is to convert the image from the spatial domain to the frequency domain for better compression. In this step, the image to be processed is divided into 8×8 image blocks, and each image block is converted into a set of frequency domain coefficients by discrete cosine transform. The frequency domain coefficients contain information of different frequencies in the image block. The frequency domain coefficients are compressed to better preserve the image information.
[0095] (2) Quantification
[0096] After the discrete cosine transform, a large number of frequency domain coefficients are obtained. Most of the coefficients are unimportant information, and only a small part is the really critical detail information. In the quantization step, the frequency domain coefficients are multiplied by a quantization matrix to reduce the amount of detail information, thereby achieving compression. This quantization matrix is adjusted according to different JPEG compression qualities to achieve different compression ratios.
[0097] It should be noted that JPEG compression is a lossy compression method because some information is lost during the discrete cosine transform and quantization process, which may cause some visible image quality loss. Therefore, JPEG compression is used in application scenarios that tolerate a certain degree of image quality loss, such as digital media, communications, and network transmission. The degree of JPEG compression is determined by the quality factor q∈[0,100], where a lower q means a higher compression ratio and lower quality. In traditional isotropic Gaussian blur, the JPEG compression quality factor is set to a fixed value, while in the isotropic Gaussian blur proposed in this invention, the quality factor q is selected from [35,100]. JPEG compression within this range will not cause excessive distortion to the image, but will bring some features after JPEG compression to the image (such as JPEG artifacts at the edges).
[0098] As a preferred technical solution, in this embodiment, step 2 includes: the degradation model proposed in the present invention is used to generate real degradation training data for the image super-resolution network by randomizing the order of five degradation types and combining them randomly; the degradation model is an adjustable degradation model. It is divided into three steps: initialization, randomization (randomization) and random deletion;
[0099] The degradation sequence is used to simulate the degradation process of the image to be processed in reality; the degradation sequence S1 is constructed, and the degradation sequence S1 is the sequential connection of K traditional degradation sequences {B, D, N, J}, that is:
[0100] (6)
[0101] Among them, B stands for blur, D stands for downsampling consisting of interval sampling, bicubic interpolation, and downsampling, N stands for noise, and J stands for JPEG compression;
[0102] Considering the order of the degraded sequence S1, rotation degradation is introduced into the degraded sequence S1 (taking the center point of the image as the center, randomly rotating left or right at an angle less than 1 degree, and downsampling the pixels), and the degradation factors in the degraded sequence S1 are shuffled to obtain the shuffled degraded sequence S2. Shuffle represents the random shuffle operation, R represents the rotation degradation, and the degraded sequence S2 is expressed as:
[0103] S2=shuffle(S1+R)(7)
[0104] Random deletion is used for the degradation sequence S2: for each different degradation type in the degradation sequence S2, the first occurrence in the degradation sequence is retained and randomly deleted with probability p in the appearing degradation sequence S2 to generate a degradation sequence S3; the degradation sequence S3 is applied to the original traffic sign image to generate a degraded traffic sign image.
[0105] Figure 3 The flow of the above steps when K=3 is illustrated; in S1, K traditional degradation sequences are generated and shuffled with an additional R in S2, where the first occurrence of the degradation type will be retained in the future (marked in dark blue), and other degradation processes will be randomly deleted in S3, some degradation processes (marked in light blue) will be retained in the final degradation sequence, while others (marked in gray) will be deleted; in summary, the random parameter scheme is divided into three steps: initialization, randomization (shuffle), and random deletion; the degradation sequence S1 is initialized, the rotation degradation is added to the sequence and shuffled to generate S2, and the random deletion strategy is applied to the sequence after shuffling, and finally the random degradation sequence S3 is obtained.
[0106] It should be noted that the present invention is tested on the Chinese Traffic Sign Database (CTSDB), which has 17,856 traffic sign images. The present invention uses the proposed isotropic Gaussian blur to generate a corresponding low-resolution degraded image for each traffic sign image. In the test results, "bic" is a 2x interval sampling + bicubic downsampling degradation process to obtain a low-resolution traffic sign image with any multiple downsampling, "blur" is a Gaussian blur with a level of 2.0, "noisy" is a Gaussian noise with a noise level of 20, "jpeg" is a JPEG compression with a compression factor of 50, and "all" is a complete probability scheme for all degradation factors, with probabilities of noise 0.8, blur 0.7, compression 0.6, and downsampling 0.2. In the high-order degradation model, the maximum degradation order is set to three orders, where the probability of third-order degradation is 0.1, the probability of second-order degradation is 0.2, and the probability of first-order degradation is 0.4. In terms of model optimization, the Adam optimizer was used, and the relevant parameters were set: b1=0.9, b2=0.97, and epsilon=10 −7 , the initial learning rate is 2×10 −5 The entire test was trained for 1000 rounds, the learning rate was halved every 100 rounds, and each round of training contained 200 iterations. The L1 loss function was used as the optimization target. Training and testing were performed on different super-resolution methods, and the peak signal-to-noise ratio (PSNR) test results are shown in Table 1.
[0107] Table 1 4x super-resolution PSNR (db) test results on CTSDB dataset
[0108]
[0109] Among them, Bicubic in the first column represents the bicubic interpolation method, SRCNN and SRGAN represent the SRCNN and SRGAN super-resolution methods trained on the data set generated by bicubic downsampling, and SRCNN-ADM and SRGAN-ADM represent the SRCNN and SRGAN super-resolution methods trained on the data set generated by the adjustable degradation model of the present invention. From the analysis of the results, the super-resolution method trained on the data set generated by the adjustable degradation model of the present invention has achieved better results than the super-resolution method trained on the training set generated by bicubic interpolation under various types of degradation factors. The model trained on the training set generated by the adjustable degradation model can improve the PSNR by more than 1db, which proves the effectiveness of the model of the present invention in alleviating the degradation of traffic sign images and improving the network reconstruction performance. At the same time, the results show that as the degree of degradation of traffic sign images deepens, the advantage of the data set generated by the adjustable degradation model in traffic sign image reconstruction becomes more and more obvious, demonstrating the robustness and wide applicability of the present invention in dealing with different degrees of degradation.
[0110] Furthermore, the present invention is applied to the CTSDB dataset for traffic sign recognition, and the accuracy is used as the evaluation index. The test results are shown in Table 2.
[0111] Table 2 The test results of the present invention applied to the traffic sign recognition test of the CTSDB dataset
[0112]
[0113] In the traffic sign recognition test, Yolov8 was used as the target recognition algorithm, and SRGAN, which had the best super-resolution effect in the previous test, was used as the super-resolution algorithm. Three groups of control tests were performed using bicubic interpolation upsampling: Bic+Yolov8, SRGAN+Yolov8, and SRGAN-ADM+Yolov8. The settings of SRGAN and SRGAN-ADM were the same as in the previous test. The results of the first two groups of tests show that the use of image super-resolution technology can effectively improve the accuracy of traffic sign target recognition. In the 2x super-resolution and 4x super-resolution scenarios, the accuracy rate has increased by more than 1 percentage point. The results of the last two groups of tests show that the SRGAN super-resolution network obtained by training with the adjustable degradation model of the present invention has an accuracy increase of 4.53 percentage points and 4.65 percentage points, respectively, at 2x super-resolution and 4x super-resolution, greatly improving the accuracy of traffic sign recognition.
[0114] The present invention also uses two hundred license plate images of real scenes taken independently, and adds them as expanded data to the training set, and fine-tunes the SRGAN-ADM in the previous test to obtain the test results of the trained super-resolution network on the license plate image test set. The network performs better in restoring text edge information, noise points are more effectively removed, the contrast between background and text is more significant, and the overall visual effect is significantly improved; this not only enhances the practicability of the super-resolution model, but also provides more valuable data support for subsequent research.
[0115] Implementing the embodiments of the present invention has the following beneficial effects:
[0116] The present invention belongs to the field of image processing and deep learning, and is a method for super-resolution of traffic sign images collected by intelligent driving cars to improve the clarity of traffic sign images and the accuracy of subsequent detection. Before the super-resolution model of traffic sign images is trained, the present invention uses an adjustable degradation model to generate degraded images close to the text information in the real scene, increases the training data of the model, and after training, the network improves the PSNR of traffic sign images from 32.28 to 33.35, and the recognition rate of text information such as traffic signs, speed limits, and road signs is improved from 89.61% to 94.14%.
[0117] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.
Claims
1. A method for generating image super-resolution training data for traffic signs, characterized in that: The following steps are involved: Step 1: construct a degradation model, wherein the degradation model is used to generate a degraded image and a degraded sequence; wherein the degradation types of the degradation model include blur degradation, interval sampling, downsampling, noise and compression; The degradation type depicted by the degradation model is expressed as: In the formula, represents the degraded image obtained after the degradation model, U n , U1, U2, Um represent the degradation types of different degradation models, σ g Indicates whether to enable the degradation model. n Indicates the level, m Indicates the number of categories available for selection of degradation models; y HR represents a high-resolution, non-degraded traffic sign image; Among them, blur degradation is expressed as: Isotropic Gaussian blur is used to simulate the blur degradation process; isotropic Gaussian filter f ( i , j )∈[- k , k ] Elements are sampled from an isotropic Gaussian distribution, an isotropic Gaussian filter f ( i , j ) has a blur kernel with a length and width of 2 k +1, k is an integer; f ( i , j ) is expressed as: In the formula, N represents the normalization constant, C represents the spatial coordinate, represents the covariance matrix, where: In the formula, represents the blur standard deviation of the isotropic Gaussian filter; In the isotropic Gaussian blur, k Randomly sample from integers in [2,10], that is, k ∈[2,10], then 2 k +1∈[5,21], blur standard deviation of isotropic Gaussian filter Randomly sample from [0.2, 1.1] to approximate the random blurring in the real world; The interval sampling includes: The pixels of the original traffic sign image are sampled every other row, i.e., interval sampling, to simulate the aliasing process of the original traffic sign image; Step 2: disrupt the order of the degraded sequence and apply it to the original traffic sign image to generate a degraded image; Step 3: Use the original traffic sign image and the degraded image pair as training data to train the traffic sign image super-resolution reconstruction network until the network converges; Wherein, step 1 comprises: Step 1.1: Only the isotropic Gaussian blur kernel is considered in the blur processing. At 2, 3, and 4 times super-resolution magnification, the blur kernel width range is set to [0.2, 2], [0.2, 3], and [0.2, 4] to construct a low-resolution traffic sign image with blur degradation; Step 1.2: The degradation model uses bicubic interpolation to downsample the high-resolution traffic sign image. By constructing a sampling weight matrix, the 16 pixels adjacent to each pixel are used as sampling points, and the sampling step size is 2, 3, or 4 times the downsampling ratio. Step 1.3: In the degradation model, this process is modeled by noise, and the traffic sign image is degraded using noise with a mean of 0 and a variance range of [0,15]; Step 1.4: Randomly crop the low-resolution traffic sign image and the corresponding high-resolution traffic sign image; the low-resolution traffic sign image block size is 48×48×3, and the corresponding high-resolution traffic sign image block sizes are 96×96×3, 144×144×3, and 192×192×3 at 2, 3, and 4 times super-resolution magnifications, respectively; Step 1.5: Perform data augmentation on the traffic sign image blocks by rotating and flipping them to expand the training data set. Among them, step 2 includes: using a degradation sequence to simulate a degradation process that the original traffic sign image is subjected to in reality; constructing a degradation sequence S1, which is a sequential connection of K traditional degradation sequences {B, D, N, J}, that is: Among them, B stands for blur, D stands for downsampling consisting of interval sampling, bicubic interpolation, and downsampling, N stands for noise, and J stands for JPEG compression; Considering the order of the degenerate sequence S1, rotational degradation is introduced into the degenerate sequence S1, and the degenerate factors in the degenerate sequence S1 are shuffled to obtain the shuffled degenerate sequence S2. Shuffle represents the random shuffle operation, R represents the rotational degradation, and the degenerate sequence S2 is expressed as: S2=shuffle(S1+R) Random deletion is used for the degradation sequence S2: for each different degradation type in the degradation sequence S2, the first occurrence in the degradation sequence is retained and randomly deleted with probability p in the appearing degradation sequence S2 to generate a degradation sequence S3; the degradation sequence S3 is applied to the original traffic sign image to generate a low-resolution degraded traffic sign image.
2. The method according to claim 1, characterized in that The down sampling includes bilinear sampling and bicubic sampling. The bilinear sampling is implemented by a dual interpolation method, and the bicubic sampling is implemented by a bicubic interpolation method.
3. The method according to claim 1, characterized in that The noise is randomly extracted from Gaussian noise and Poisson noise; for Gaussian noise, its probability density function is f ( x ) is the probability density function of the Gaussian distribution: in, x represents noise, represents the noise expectation, represents the blur standard deviation of the isotropic Gaussian filter; The distribution of the Poisson noise is described by Poisson distribution: Among them, y represents the number of times an event occurs in a certain time or space, corresponding to the intensity of the image to be processed, and λ represents the average occurrence rate of events per unit time or space; The standard deviation of Gaussian noise is set to the range of [1,3], and the intensity coefficient of Poisson noise is set to the range of [0.05,0.3].
4. The method according to claim 1, characterized in that: Step 2 uses JPEG compression, and the JPEG compression process is divided into discrete cosine transform and quantization.
