A method and apparatus for restoring a blurred image and an electronic device

Abstract

The application discloses a blurred image recovery method and device and electronic equipment. The blurred image recovery model is used to recover the blurred image to obtain the corresponding clear image. When the blurred image recovery model is trained, the noise processing is performed on the plurality of sample images in the training set to obtain the corresponding noise-added sample image groups of the sample images, and the noise-added sample image groups have corresponding sample clear images. The blurred image recovery model is trained based on the plurality of noise-added sample image groups and the corresponding sample clear images of the plurality of noise-added sample image groups, so that the blurred image recovery model can adapt to the image recovery of the blurred images collected in different environments. When the blurred image to be processed is obtained, the clear image corresponding to the blurred image is determined based on the blurred image recovery model. Compared with the existing image enhancement algorithm, the accuracy of image recovery can be improved. The technical scheme of the application has the characteristics of universality, reliability and real-time, and meets the reliable characteristics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a blurred image recovery method and device and electronic equipment. BACKGROUND

[0002] When monitoring equipment such as a monitoring camera and vehicle-mounted cameras capture license plate numbers of vehicles, roadside text information, road warning signs and traffic signs, the images obtained by the equipment may be blurred due to the following reasons: the distance between the monitoring equipment and the vehicle is too far, the fog is too thick, the weather is stormy or snowy, the monitoring camera shakes due to strong wind or the road bounces, or the camera lens is dirty. In this case, the image enhancement measures, such as rain and fog removal and image quality enhancement, cannot effectively remove the blur. Therefore, the existing blurred image recovery method has the problem of poor image recovery accuracy. SUMMARY

[0003] The present application provides a blurred image recovery method, device and electronic equipment to solve the problem of poor image recovery accuracy of the existing blurred image recovery method.

[0004] In a first aspect, the present application provides a blurred image recovery method, which comprises:

[0005] obtaining a blurred image to be processed, inputting the blurred image into a blurred image recovery model, and determining a clear image corresponding to the blurred image based on the blurred image recovery model;

[0006] The blurred image recovery model is obtained by the following method:

[0007] For a plurality of sample images in a training set, the sample images are subjected to noise addition processing to obtain a plurality of noise-added sample image groups corresponding to the sample images; the plurality of noise-added sample image groups and sample clear images corresponding to the plurality of noise-added sample image groups are input into the blurred image recovery model, and the blurred image recovery model is trained.

[0008] In a second aspect, the present application provides a blurred image recovery device, which comprises:

[0009] a model training module configured to, for a plurality of sample images in a training set, subject the sample images to noise addition processing to obtain a plurality of noise-added sample image groups corresponding to the sample images; input the plurality of noise-added sample image groups and sample clear images corresponding to the plurality of noise-added sample image groups into the blurred image recovery model, and train the blurred image recovery model;

[0010] An image recovery module is configured to acquire a blurred image to be processed, input the blurred image into a blurred image recovery model, and determine a clear image corresponding to the blurred image based on the blurred image recovery model.

[0011] In a third aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0012] The memory is configured to store a computer program.

[0013] The processor is configured to execute the program stored on the memory to implement the method steps.

[0014] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method steps.

[0015] The present application provides a blurred image recovery method and device and an electronic device. The method comprises: acquiring a blurred image to be processed, inputting the blurred image into a blurred image recovery model, and determining a clear image corresponding to the blurred image based on the blurred image recovery model. The blurred image recovery model is trained in the following manner: for a plurality of sample images in a training set, the sample images are subjected to noise addition processing to obtain a set of noise-added sample images corresponding to the sample images; the plurality of sets of noise-added sample images and sample clear images corresponding to the plurality of sets of noise-added sample images are input into the blurred image recovery model, and the blurred image recovery model is trained.

[0016] The technical solution has the following advantages or beneficial effects:

[0017] In the present application, a blurred image recovery model is first trained, and a clear image corresponding to a blurred image is obtained based on the blurred image recovery model. In training the blurred image recovery model, a plurality of sample images in a training set are subjected to noise processing to obtain a set of noise-added sample images corresponding to the sample images, and the set of noise-added sample images has corresponding sample clear images. The blurred image recovery model is trained based on the plurality of sets of noise-added sample images and the sample clear images corresponding to the plurality of sets of noise-added sample images, so that the blurred image recovery model can adapt to image recovery of blurred images collected in different environments. When a blurred image to be processed is acquired, a clear image corresponding to the blurred image is determined based on the blurred image recovery model. Compared with existing image enhancement algorithms, the accuracy of image recovery can be improved. The technical solution of the present application has the characteristics of universality, reliability and real-time, and meets the reliable characteristics. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 The schematic diagram of the blurred image restoration process provided by the present application is shown in FIG. 1.

[0020] Figure 2 The schematic diagram of the training process of the blurred image restoration model provided by the present application is shown in FIG. 2.

[0021] Figure 3 The schematic diagram of the training process of the blurred image restoration model provided by the present application is shown in FIG. 2.

[0022] Figure 4 The schematic diagram of the blurred image restoration process provided by the present application is shown in FIG. 1.

[0023] Figure 5 The schematic diagram of the blurred image restoration process provided by the present application is shown in FIG. 1.

[0024] Figure 6 The schematic diagram of the blurred image restoration process provided by the present application is shown in FIG. 1.

[0025] Figure 7 The schematic diagram of the training process of the language question and answer model provided by the present application is shown in FIG. 3.

[0026] Figure 8 The schematic diagram of the process of updating the blurred image restoration model provided by the present application is shown in FIG. 4.

[0027] Figure 9 The schematic diagram of the motor vehicle license plate standard provided by the present application is shown in FIG. 5.

[0028] Figure 10 The schematic diagram of the public data multi-resolution data set provided by the present application is shown in FIG. 6.

[0029] Figure 11 The schematic diagram of the text semantic question and answer example provided by the present application is shown in FIG. 7.

[0030] Figure 12 The schematic diagram of the blurred image restoration process provided by the present application is shown in FIG. 1.

[0031] Figure 13 The schematic diagram of the blurred image restoration algorithm flowchart provided by the present application is shown in FIG. 8.

[0032] Figure 14 The schematic diagram of the blurred image restoration effect provided by the present application is shown in FIG. 9.

[0033] Figure 15A structure schematic diagram of a fuzzy image recovery device provided in the present application is shown in the figure.

[0034] Figure 16 A structure schematic diagram of an electronic device provided in the present application is shown in the figure. DETAILED DESCRIPTION

[0035] In order to make the purpose and embodiments of the present application more clear, the following will combine the figures in the exemplary embodiments of the present application to clearly and completely describe the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, but not all the embodiments.

[0036] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0037] The terms "first", "second", "third" and the like in the specification and claims of the present application and the above-mentioned figures are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit the specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0038] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not necessarily limit to all the components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0039] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware or / and software code capable of performing functions associated with the element.

[0040] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0041] For convenience of explanation, the foregoing description has been made in conjunction with specific embodiments. However, the foregoing exemplary discussion is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Various modifications and variations are possible in light of the above teachings. The embodiments are chosen and described in order to best explain the principles of the application and its practical application and to thereby enable others skilled in the art best to utilize the application and various embodiments with various modifications as are suited to the particular use contemplated.

[0042] Figure 1 A schematic diagram of the blur image restoration process provided in the present application is shown in the following figure, which includes the following steps:

[0043] S101: For a plurality of sample images in a training set, the sample images are subjected to noise addition processing to obtain a plurality of sets of noise-added sample images corresponding to the sample images; the plurality of sets of noise-added sample images and the sample clear images corresponding to the plurality of sets of noise-added sample images are input into the blur image restoration model, and the blur image restoration model is trained.

[0044] S102: Obtain a blur image to be processed, input the blur image into the blur image restoration model, and determine a clear image corresponding to the blur image based on the blur image restoration model.

[0045] The blur image restoration method provided in the present application is applied to an electronic device, which can be a PC, a tablet computer, or the like, or a server.

[0046] An electronic device is deployed with a trained blur image restoration model. In training the blur image restoration model, a plurality of sample images in a training set are first obtained. The plurality of sample images can be images in a Mnist handwritten dataset, a Times New Roman, a boldface, and other uppercase and lowercase English letter datasets, and other public datasets. The plurality of sample images are respectively processed to add noise, thereby achieving training set enhancement. The noise adding processing can be one or more of adding Gaussian noise, additive noise ratio in color component, more intensive additive noise in brightness component, spatially dependent noise, masking noise, high-frequency noise, impulse noise, quantization noise, Gaussian blur, image denoising, JPEG compression, JPEG2000 compression, JPEG transmission error, JPEG2000 transmission error, non-biased pattern noise, local block distortion of different intensities, average bias, intensity bias, contrast variation, changing color saturation index, lossy compression, multiple Gaussian noise, color image quantization, sparse sampling, color difference, and comfort noise. After the noise adding processing of a sample image, a set of noisy sample images corresponding to the sample image is obtained. For example, for a sample image A1, after adding noise in different manners or combinations, images A2, A3, A4, and A5 are obtained. Thus, A1, A2, A3, A4, and A5 form a set of noisy sample images corresponding to the sample image A1. The training set stores a sample clear image corresponding to the sample image A1. Because the set of noisy sample images corresponding to the sample image A1 contains images with the same content but different blur degrees, the sample clear image corresponding to the sample image A1 is also the sample clear image corresponding to the set of noisy sample images.

[0047] For a plurality of sets of noisy sample images, the set of noisy sample images and the sample clear image corresponding to the set of noisy sample images are input into the blur image restoration model. The blur image restoration model extracts features from the set of noisy sample images, determines a predicted restored image, and then determines a model loss value based on the predicted restored image and the sample clear image. The blur image restoration model is trained based on the model loss value. When a preset number of model training iterations is reached or the model loss value meets the requirement, the training of the blur image restoration model is completed.

[0048] After the electronic device obtains a blur image to be processed, the blur image is input into the blur image restoration model. The blur image to be processed is subjected to feature extraction based on the blur image restoration model, and a clear image corresponding to the blur image is determined.

[0049] In the present application, a blurred image restoration model is first trained, and a corresponding clear image is obtained based on the blurred image restoration model. During the training of the blurred image restoration model, the multiple sample images in the training set are subjected to noise processing to obtain a set of noise-added sample images corresponding to the sample images, and the set of noise-added sample images has corresponding sample clear images. The blurred image restoration model is trained based on the multiple sets of noise-added sample images and the sample clear images corresponding to the multiple sets of noise-added sample images, so that the blurred image restoration model can adapt to the image restoration of blurred images collected in different environments. When a blurred image to be processed is obtained, the clear image corresponding to the blurred image is determined based on the blurred image restoration model. Compared with existing image enhancement algorithms, the accuracy of image restoration can be improved. The technical solution of the present application has the characteristics of universality, reliability and real-time, and meets the reliability characteristics.

[0050] Reliability characteristics:

[0051] (1) Universality: The present application provides a means for restoring blurred images containing text and numerical information, which can be used for image restoration of any such blurred images, meeting the universality;

[0052] (2) Reliability: The present application provides users with better blurred image text restoration capability than pure image recognition, and provides users with a reliable interface for uploading real and reliable high-definition pictures, enhancing model accuracy, meeting the reliability characteristics;

[0053] (3) Real-time: The present application can provide real-time feedback on the inference results of the obtained frames under the condition of extracting one frame every 5 frames from a video, meeting the real-time characteristics of the reliability characteristics.

[0054] In order to further improve the accuracy of the blurred image restoration model, in the present application, the training process of the blurred image restoration model includes:

[0055] For the multiple sample images in the training set, the resolution of the sample images is adjusted according to a plurality of preset resolutions to obtain a plurality of resolution-adjusted images corresponding to the sample images. The plurality of resolution-adjusted images are subjected to noise addition processing to obtain a set of noise-added sample images corresponding to the sample images. The multiple sets of noise-added sample images and the sample clear images corresponding to the multiple sets of noise-added sample images are input into the blurred image restoration model, and the blurred image restoration model is trained.

[0056] The preset multiple resolutions are, for example, 512*512, 256*256, 128*128, 64*64, 32*32, etc. The sample image is adjusted in resolution to obtain a sample image corresponding to a resolution of 512*512, a sample image corresponding to a resolution of 256*256, a sample image corresponding to a resolution of 128*128, a sample image corresponding to a resolution of 64*64, a sample image corresponding to a resolution of 32*32, etc. The multiple resolution image groups corresponding to the sample image include a sample image corresponding to a resolution of 512*512, a sample image corresponding to a resolution of 256*256, a sample image corresponding to a resolution of 128*128, a sample image corresponding to a resolution of 64*64, and a sample image corresponding to a resolution of 32*32, etc. Then, the sample images of each resolution in the multiple resolution image groups are respectively subjected to noise increasing processing, that is, the sample image corresponding to a resolution of 512*512, the sample image corresponding to a resolution of 256*256, the sample image corresponding to a resolution of 128*128, the sample image corresponding to a resolution of 64*64, and the sample image corresponding to a resolution of 32*32, etc. are respectively subjected to noise increasing processing, and then a group of noise-added sample images corresponding to the sample image is obtained.

[0057] The multiple groups of noise-added sample images and the sample clear images corresponding to the multiple groups of noise-added sample images are input into the blur image restoration model, and the blur image restoration model is trained. In the present application, the blur image restoration model is trained according to the multiple groups of noise-added sample images containing multiple scale images by performing multi-scale transformation on the sample images. The blur image restoration model obtained in this way can have the image restoration capability for multiple scale images, and further improves the accuracy of the blur image restoration model.

[0058] Figure 2 A training process diagram of the blur image restoration model provided in the present application includes the following steps:

[0059] S201: For multiple sample images in a training set, adjusting the sample images in resolution according to a preset multiple resolutions to obtain a multiple resolution image group corresponding to the sample images, and performing noise increasing processing on the multiple resolution image group to obtain a group of noise-added sample images corresponding to the sample images.

[0060] S202: Inputting multiple groups of noise-added sample images and sample clear images corresponding to the multiple groups of noise-added sample images into the blur image restoration model, and training the blur image restoration model.

[0061] In order to further improve the accuracy of the blur image restoration model, in the present application, the training process of the blur image restoration model includes:

[0062] For multiple sample images in the training set, adjust the resolution of the sample images according to multiple preset resolutions to obtain a multi-resolution image group corresponding to the sample images, and perform noise addition processing on the multi-resolution image group to obtain a noisy sample image group corresponding to the sample images; input the character arrangement specification information, multiple noisy sample image groups, the sample clear images corresponding to the multiple noisy sample image groups, and the text description information corresponding to the multiple noisy sample image groups into the blurred image restoration model to train the blurred image restoration model.

[0063] Taking license plate image restoration as an example, the character arrangement specification information can be motor vehicle license plate document information. The character arrangement specification information records that the first digit of the license plate number is the abbreviation of the province, autonomous region, or municipality directly under the Central Government, the second digit is the code of the license-issuing agency, there is an interval symbol after the code of the license-issuing agency, and there is a license plate serial number after the interval symbol, and there are regulations that O and I cannot be included in the license plate serial number. Inputting the character arrangement specification information into the blurred image restoration model can learn the character arrangement specification information. For example, in license plate restoration, the number "3" and the character "了" have a high similarity in the blurred image, but the blurred image restoration model will increase the recognition weight of the number "3" and reduce the recognition weight of the character "了".

[0064] For the character image restoration scenario, the text description information is the character information in the sample image. For example, if the sample image is a license plate image, the text description information is the text description of the license plate number. For example, the text description information is "京H*****". Inputting the text description information into the blurred image restoration model can better learn the character information in the image to be restored and improve the accuracy of the blurred image restoration model.

[0065] Figure 3 This is a schematic diagram of the training process of the blurred image restoration model provided by this application, including the following steps:

[0066] S301: For multiple sample images in the training set, adjust the resolution of the sample images according to multiple preset resolutions to obtain a multi-resolution image group corresponding to the sample images, and perform noise addition processing on the multi-resolution image group to obtain a noisy sample image group corresponding to the sample images.

[0067] S302: Input the character arrangement specification information, multiple noisy sample image groups, the sample clear images corresponding to the multiple noisy sample image groups, and the text description information corresponding to the multiple noisy sample image groups into the blurred image restoration model to train the blurred image restoration model.

[0068] In addition, the character arrangement specification information, the plurality of noise-added sample image groups, the sample clear images corresponding to the plurality of noise-added sample image groups, and the text description information corresponding to the plurality of noise-added sample image groups are input into the blurred image restoration model, and the blurred image restoration model is trained.

[0069] Optionally, the character arrangement specification information, the plurality of noise-added sample image groups, the sample clear images corresponding to the plurality of noise-added sample image groups, and the text description information corresponding to the plurality of noise-added sample image groups are input into the blurred image restoration model, the blurred image restoration model outputs predicted text description information, a loss value is calculated according to the predicted text description information and the target text description information, the parameter of the blurred image restoration model is adjusted according to the loss value, and the training of the blurred image restoration model is completed when a preset training iteration number is reached or the loss value meets a requirement. In this way, when the blurred image restoration model is applied, the blurred image to be processed is input into the blurred image restoration model, and the clear image corresponding to the blurred image and the target text description information corresponding to the blurred image are output based on the blurred image restoration model.

[0070] Figure 4 A blurred image restoration process provided in the present application includes the following steps:

[0071] S401: For a plurality of sample images in a training set, the resolution of the sample images is adjusted according to a plurality of preset resolutions to obtain a plurality of resolution-adjusted image groups corresponding to the sample images, and noise is added to the plurality of resolution-adjusted image groups to obtain a plurality of noise-added sample image groups corresponding to the sample images.

[0072] S402: The character arrangement specification information, the plurality of noise-added sample image groups, the sample clear images corresponding to the plurality of noise-added sample image groups, and the text description information corresponding to the plurality of noise-added sample image groups are input into the blurred image restoration model, and the blurred image restoration model is trained.

[0073] S403: A blurred image to be processed is obtained, the blurred image is input into the blurred image restoration model, and the clear image corresponding to the blurred image and the target text description information in the blurred image are determined based on the blurred image restoration model.

[0074] In the present application, obtaining the blurred image to be processed includes:

[0075] obtaining an image to be processed, if the image is a character region image, taking the character region image as the fuzzy image to be processed; if the image is a region of interest (ROI) image or a raw scene image, performing resolution adjustment on the image according to the preset multiple resolutions, and taking a multi-resolution image group corresponding to the obtained image as the fuzzy image to be processed.

[0076] If the image to be processed is a character region image, the character region image is directly subjected to recovery processing, that is, the character region image is directly taken as the fuzzy image to be processed. If the image is a region of interest (ROI) image or a raw scene image, in order to improve the accuracy of image recovery processing, resolution adjustment is performed on the region of interest (ROI) image or the raw scene image according to the preset multiple resolutions, and a multi-resolution image group corresponding to the obtained region of interest (ROI) image or the raw scene image is taken as the fuzzy image to be processed. Finally, each small image containing useful character information can be subjected to image recovery, thereby further improving the accuracy of image recovery.

[0077] Figure 5 A fuzzy image recovery process provided in the present application includes the following steps:

[0078] S501: For multiple sample images in a training set, performing resolution adjustment on the sample images according to preset multiple resolutions to obtain a multi-resolution image group corresponding to the sample images, and performing noise addition processing on the multi-resolution image group to obtain a noise-added sample image group corresponding to the sample images.

[0079] S502: Inputting character arrangement specification information, multiple noise-added sample image groups, sample clear images corresponding to the multiple noise-added sample image groups, and text description information corresponding to the multiple noise-added sample image groups into the fuzzy image recovery model, and training the fuzzy image recovery model.

[0080] S503: Obtaining an image to be processed, if the image is a character region image, taking the character region image as the fuzzy image to be processed; if the image is a region of interest (ROI) image or a raw scene image, performing resolution adjustment on the image according to the preset multiple resolutions, and taking a multi-resolution image group corresponding to the obtained image as the fuzzy image to be processed; inputting the fuzzy image into the fuzzy image recovery model, and determining a clear image corresponding to the fuzzy image and target text description information in the fuzzy image based on the fuzzy image recovery model.

[0081] In the image restoration in the intelligent traffic scene, in addition to paying attention to the clear license plate image, the traffic management personnel generally also pays attention to the related specification information of the license plate number. Based on the above consideration, in order to improve the user experience, in the present application, the method further comprises:

[0082] The target text description information is input into the language question and answer model, and the target question information and the target answer information corresponding to the target text description information are determined based on the language question and answer model.

[0083] The electronic device is pre-trained with a language question and answer model, which is used to generate question information and generate corresponding answer information according to the question information. The language question and answer model can be WenXin YiYan or GPT language question and answer model.

[0084] Figure 6 A fuzzy image restoration process diagram is provided for the present application, which comprises the following steps:

[0085] S601: For a plurality of sample images in a training set, the resolution of the sample images is adjusted according to a plurality of preset resolutions, to obtain a plurality of resolution images corresponding to the sample images, and the plurality of resolution images are processed to increase noise to obtain a plurality of noise-added sample image groups corresponding to the sample images.

[0086] S602: The character arrangement specification information, the plurality of noise-added sample image groups, the sample clear images corresponding to the plurality of noise-added sample image groups, and the text description information corresponding to the plurality of noise-added sample image groups are input into the fuzzy image restoration model, and the fuzzy image restoration model is trained.

[0087] S603: Obtain a fuzzy image to be processed, input the fuzzy image into the fuzzy image restoration model, and determine the clear image corresponding to the fuzzy image and the target text description information in the fuzzy image based on the fuzzy image restoration model.

[0088] S604: The target text description information is input into the language question and answer model, and the target question information and the target answer information corresponding to the target text description information are determined based on the language question and answer model.

[0089] Figure 7 A language question and answer model training process diagram is provided for the present application, which comprises the following steps:

[0090] S701: The language question and answer sample file is input into the language question and answer model, and a set number of predicted question and answer information is determined based on the language question and answer model.

[0091] S702: Determine the tags of the set number of predicted Q&A information, and train the language Q&A model according to the tags of the set number of predicted Q&A information, where the tags include correct Q&A tags and incorrect Q&A tags.

[0092] The language Q&A sample file is, for example, a motor vehicle license plate file, and the set number is, for example, 1000, 2000, etc. During the training process of the language Q&A model, within each iteration cycle, after the language Q&A model determines the predicted Q&A information, it determines the correct Q&A information and incorrect Q&A information in the predicted Q&A information. Among them, the correct Q&A information and incorrect Q&A information can be screened by humans. Then, correct Q&A tags are configured for the correct Q&A information, and incorrect Q&A tags are configured for the incorrect Q&A information. Then, the tagged Q&A information is input into the language Q&A model, and the voice Q&A model is trained iteratively in this way.

[0093] After the voice Q&A model is trained, the target text description information is input into the language Q&A model, and the target question information and target answer information corresponding to the target text description information are determined based on the language Q&A model. For example, when the target text description information is "京H*****", the target question information corresponding to the target text description information determined based on the language Q&A model is: "Is the license plate '京H*****' a legal motor vehicle license plate?", and the target answer information is: "'京H*****' meets the motor vehicle license plate standard and is a legal motor vehicle license plate". Another example, when the target text description information is "USH****", the target question information corresponding to the target text description information determined based on the language Q&A model is: "Is the license plate 'USH****' a legal motor vehicle license plate?", and the target answer information is: "'USH****' does not meet the motor vehicle license plate standard and the nationality is unknown".

[0094] To further make the blurred image restoration model more accurate, in this application, the method further includes:

[0095] Obtain the target clear image corresponding to the blurred image to be processed input by the user, determine the structural similarity between the clear image corresponding to the blurred image and the target clear image. If the structural similarity is greater than a preset first similarity threshold, use the clear image as a positive incentive prediction result to update the blurred image restoration model; if the structural similarity is less than a preset second similarity threshold, use the clear image as a negative incentive prediction result to update the blurred image restoration model; where the preset second similarity threshold is less than the preset first similarity threshold.

[0096] In some scenarios, the user inputs a target clear image corresponding to the blurred image to be processed to the electronic device. After the electronic device determines the clear image corresponding to the blurred image to be processed based on the blurred image restoration model, the structural similarity SSIM of the clear image corresponding to the blurred image and the target clear image is determined.

[0097] The structural similarity SSIM calculation process is as follows:

[0098] Given two images x and y, the structural similarity of the two images can be calculated as follows

[0099] Where μ x is the mean of x, μ y is the mean of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y. c1=(k1L) 2 , c2=(k2L) 2 is a constant used to maintain stability. L is the dynamic range of pixel values, k1=0.01, k2=0.03. The range of structural similarity is -1 to 1. When the two images are exactly the same, the value of SSIM is equal to 1.

[0100] If the structural similarity is greater than a preset first similarity threshold, the clear image is used as a positive incentive prediction result to update the blurred image restoration model. That is, it is considered that the accuracy of the clear image corresponding to the blurred image meets the requirements. The preset first similarity threshold is, for example, 0.7, 0.8, etc. If the structural similarity is less than a preset second similarity threshold, the clear image is used as a negative incentive prediction result to update the blurred image restoration model. That is, it is considered that the accuracy of the clear image corresponding to the blurred image does not meet the requirements. The preset second similarity threshold is, for example, 0.2, 0.3, etc.

[0101] Figure 8 The process of updating the blurred image restoration model provided in the present application is shown in the following steps:

[0102] S801: Obtain a target clear image corresponding to a blurred image to be processed input by a user, and determine the structural similarity of the clear image corresponding to the blurred image and the target clear image.

[0103] S802: If the structural similarity is greater than a preset first similarity threshold, the clear image is used as a positive incentive prediction result to update the blurred image restoration model.

[0104] S803: If the structural similarity is less than a preset second similarity threshold, update the blur image restoration model according to the clear image as a negative incentive prediction result.

[0105] The blur image restoration method provided in the application is applicable to the restoration of blurred images captured in traffic scenes, home scenes and work scenes, and is also applicable to the restoration of character blurred images such as text and numbers. The blur image restoration method provided in the application will be described in detail below by taking the restoration of character blurred images such as text and numbers as an example.

[0106] The application combines public data sets, actual captured video and picture data containing real license plates or characters, and these video and picture data containing license plate numbers and characters are composed of image pairs from high resolution to low resolution such as 1024*1024 images, 512*512 images, 256*256 images, 128*128 images and 64*64 images to provide potential constraint conditions of letter and number arrangement rules for the model, and to form complete training sets, evaluation sets and test sets; the self-supervised learning deep learning network is trained by combining the above training data sets, and after the model converges, the evaluation set is used for constraint to obtain the restoration model of high-definition text image text document, various license plates and the like, and the incremental and reinforced interface and channel are preset for subsequent blurred images and clear images, a blurred image restoration model and a training scheme of the model are provided, and the recognition success efficiency is improved by probability statistics in application.

[0107] The model combines the reinforcement incremental learning mode, and the incremental and reinforced interface and channel are preset for subsequent blurred images and clear images. When the algorithm is actually applied, the algorithm user can provide 5-10 target clear images of suspicious license plates to the algorithm, the algorithm outputs clear images through calculation, and the SSIM structural similarity of the 5-10 high-definition license plates is given to the target license plate probability, so as to narrow the scope of identification.

[0108] The application uses not only the Mnist handwritten data set, TimesNewRoman, uppercase and lowercase English letter data set, etc. These public data sets, national standard license plate number data specification, special font designed by the Ministry of Public Security, and the basic font is improved to bold, according to the current motor vehicle license plate standard, for example Figure 9As shown, a number or a character is taken as a 1024*1024 input image, and by means of long-distance shooting, adding 24 types of noise, reducing resolution, etc., 512*512 images, 256*256 images, 128*128 images, 64*64 images, etc. of the corresponding numbers and images in the above data set are provided respectively, thereby establishing a high-resolution training image pair data set of individual letters and specific individual Chinese characters. This data set restricts the generation range of letters and numbers in the fuzzy image restoration model.

[0109] For example, "a paragraph of Times New Roman font is as follows:", "Innovation in China, China's wisdom and global", "①-Pic01 resolution is 32*32", "②-Pic02 resolution is 64*64", "③-Pic03 resolution is 128*128", "④-Pic04 resolution is 256*256", "⑤-Pic05 resolution is 512*512", "⑥-Pic06 resolution is 1024*1024"; and so on.

[0110] The 24 types of noise, distortion and their addition to the picture are as follows: Gaussian noise, additive noise ratio in color components, more intensive additive noise in luminance components, spatially correlated noise, masking noise, high-frequency noise, impulse noise, quantization noise, Gaussian blur, image denoising, JPEG compression, JPEG2000 compression, JPEG transmission error, JPEG2000 transmission error, non-biased pattern noise, local block distortion of different intensities, average offset (intensity offset), contrast change, change of color saturation index, lossy compression, multiple Gaussian noise, color image quantization, sparse sampling, color difference and comfort noise.

[0111] Real license plate picture or video collection database, vehicle information database, vehicle picture or video data set, etc. are used for text annotation and semantic annotation to provide potential constraint conditions for the arrangement rules of letters and numbers for the model, so that the model can learn the feature information consistent with the character features and text features in the picture during subsequent model training, thereby learning the license plate and vehicle information.

[0112] Using a language-based large model algorithm such as Wenxin Yiyang or GPT4, by inputting the motor vehicle license plate specification file, the Mnist handwritten data set, Times New Roman, bold, etc. and reasonable questions, semantic text about license plate regulations and standards and its feature alignment are generated, thereby positively guiding the self-supervised training process, so that the fuzzy image restoration model is not only suitable for ordinary text and number fuzzy image recognition and identification, but also suitable for specific application scenarios such as license plate fuzzy recovery.

[0113] Figure 10Schematic diagram of the open data multi-resolution dataset provided by this application Figure 10 It contains 4 examples, two examples of Times New Roman, one example of the Mnist handwritten dataset, and one example of Beijing license plates. Among them, taking Times New Roman as an example, when annotating the dataset, it can but is not limited to being described as follows - "A passage of text in Times New Roman font is as follows:", "Innovation in China, Made in China, Benefiting the World", "① - Pic01 has a resolution of 32×32" in the atlas, "② - Pic02 has a resolution of 64×64", "③ - Pic03 has a resolution of 128×128", "④ - Pic04 has a resolution of 256×256", "⑤ - Pic05 has a resolution of 512×512", "⑥ - Pic06 has a resolution of 1024×1024". "⑥ - Pic06 with a resolution of 1024×1024" corresponds to a high-definition picture, and "① - Pic01 with a resolution of 32×32" corresponds to an ultra-low-resolution picture. The resolution of each picture or each frame of the video can but is not limited to be composed of resolutions such as 1024*1024, 512*512, 256*256, 128*128, 64*64 (or additionally 32×32) from high to low to form an image pair set containing various resolutions (such as the above 5 types (or 6 types)) to establish an image pair set for the various resolutions corresponding to the same picture (or the same frame of the video), and perform semantic annotation on the images of each resolution, for example: label them as high-definition, high, medium, low, extremely low (or ultra-low), etc., so that when training the subsequent model, it can learn the features of "high-definition, high, medium, low, extremely low (or ultra-low)" and match the above resolutions.

[0114] Figure 11 Schematic diagram of the text semantic question and answer example provided by this application. Input materials such as motor vehicle license plate specification documents, the Mnist handwritten dataset, Times New Roman, and boldface into the language question and answer model and ask reasonable questions. Finally, train the language question and answer model.

[0115] Examples of text questions and answers for language large models:

[0116] Question: "Is the license plate 'Beijing A33333' a legal motor vehicle license plate?";

[0117] Answer: "The license plate 'Beijing A33333' complies with the standard of GA 36-2018 'Motor Vehicle License Plates of the People's Republic of China' and is a legal motor vehicle license plate."

[0118] Question: "Is the license plate 'US.X77I66' a legal motor vehicle license plate?";

[0119] Answer: "US.X77I66" does not meet the GA 36-2018 "People's Republic of China Motor Vehicle License Plate" standard, is not a Chinese license plate, and the nationality is unknown.

[0120] The present application includes but is not limited to configuring question and answer templates in combination with the Text to SQL method, and generating questions and answers in batches.

[0121] The above question and answer are aligned again with the image feature encoding by feature encoding, and the self-attention mechanism is used to carry out deep learning training of the corresponding and alignment relationship between the question and the answer, obtain refined and accurate policy and regulation text alignment and feature extraction capability, answer more accurate and concise than the original question and answer language class model, use scripts to randomly combine questions and answers, and provide rich policy and regulation text feature data sets for further self-supervised training.

[0122] The final effect that can be achieved is:

[0123] Question: "What does 'Lu' mean in the license plate?";

[0124] Answer: "Lu is the abbreviation for Shandong Province, placed first before the English letter in the license plate, indicating that the vehicle belongs to Shandong Province."

[0125] Question: "Jing H*****";

[0126] Answer: "Jing H*****" meets the standard of motor vehicle license plate and is a legal motor vehicle license plate.

[0127] Thus, the "standard text composition of domestic license plates and the legal order specified in the regulations and documents" are learned, including the abbreviation of the province and its fixed position in the license plate, the position of the English letter of the subordinate region of each province, the color of the license plate, and the legal rules of the last 5-6 letters and numbers (such as the prohibition of O and I, etc.), so that the subsequent trained fuzzy image restoration model can truly understand and master the regulations and documents and recognize the license plate.

[0128] Figure 12 The fuzzy image restoration overall flowchart provided for the present application includes Figure 10 the disclosed data multi-resolution data set part shown in Figure 11 the text semantic question and answer example part shown in. In combination with the above training data set, and in combination with the question and answer semantic feature encoding, the above text, character, semantic, picture, etc. Feature encoding is matched and completely aligned one by one, and self-supervised learning deep learning network is used for training, and after the model converges, it is constrained by the evaluation set. Finally, the character and number fuzzy image restoration model is obtained.

[0129] Figure 13 The fuzzy image restoration algorithm flowchart provided for the present application.

[0130] ①Through artificial frame selection of fuzzy text and numbers in the image area, the small picture after frame selection is extracted, the fuzzy small picture is passed to the fuzzy image recovery model, and the top 5 candidate text probabilities of the possible content of the text are given (output end E), which is further manually judged by the algorithm user or on-site staff;

[0131] ②The input complete picture is segmented by multiple segmentation strategies such as 128*128, 64*64, 32*32 (or 16*16) resolution, and each small picture after segmentation is extracted. The fuzzy small picture is passed to the fuzzy image recovery algorithm, the algorithm detects the text output probability threshold to screen and judge whether the segment contains fuzzy text and numbers, and gives the top 5 candidate text probabilities of the possible content of the text, and retains the fuzzy segmented segment as one of the useful information small pictures. All segmented sub-segments of all segmentation strategies are screened in turn, and finally each useful information small picture and its top 5 probability results of text and numbers (output end E) are displayed to the algorithm user or on-site staff for further manual judgment. Figure 1

[0132] ③The algorithm user or on-site staff can narrow down the segmentation range of the large picture in step ② by frame selecting the ROI region (region of interest), and the remaining steps are similar to step ②, thereby improving the detection efficiency of the algorithm.

[0133] When fuzzy images containing text or text information are recovered, any one of the above ①-③ applications can obtain effective result pictures.

[0134] When the algorithm is applied to license plate fuzzy picture recovery, the algorithm user can provide 5-10 clear images of suspicious license plates to the algorithm, the algorithm calculates the combination of numbers and characters generated by the GAN network, and gives the target license plate probability by comparing the SSIM (Structural Similarity) of the 5-10 high-definition license plates, thereby narrowing down the scope of identification.

[0135] ​The model combines the reinforcement incremental learning mode, and pre-provisions an incremental and reinforced interface and channel (reinforced incremental learning information I) for subsequent fuzzy images and clear images. If the 5-10 high-definition license plates provided by the application layer are confirmed by manual confirmation to be consistent with the result of a high possibility of a certain license plate, or the SSIM structural similarity is very high and exceeds the threshold value (such as higher than 90%), at this time, the reinforced incremental learning extracts the feature of this part, and takes this as a positive +1 behavior, and gives it to the self-supervised training model to obtain reinforcement 1 time, and aligns the picture and text feature codes and the like, so that the fuzzy image recovery capability of the model is improved.

[0136] As shown in Figure 13 , if the fuzzy text target is directly framed, the fuzzy image recovery is directly performed on the fuzzy text target; if the ROI region is framed, the fuzzy image recovery is performed after multi-resolution processing of the ROI region; or the fuzzy image recovery is performed after multi-resolution processing of the original scene image. The text string of the recognized text is output, and the clear image and the percentage probability value thereof are output. It is judged whether the target clear image input by the human being is obtained, if the target clear image input by the human being is obtained, the fuzzy image recovery model is reinforced and incrementally learned according to the structural similarity of the clear image and the target clear image. If there is no target clear image input by the human being, the text string of the recognized text is output, and the clear image and the percentage probability value thereof are output.

[0137] In the application of the algorithm, the ROI region on the screen is dynamically selected, so that the detection and recognition capability can be further improved.

[0138] By using the model, the high-resolution recovery of the whole picture or the ROI region of the picture of the fuzzy text or character can be performed, and the possibility probability of the alternative license plate is given, useful information is provided for investigation and identification, and the investigation efficiency is improved.

[0139] The similarity is output, and the SSIM formula is used, and the details are as follows:

[0140] Given two images x and y, the structural similarity of the two images can be calculated in the following manner

[0141]

[0142] Where μ x is the mean of x, μ y is the mean of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y. c1=(k1L) 2 , c2=(k2L) 2is a constant used to maintain stability. L is the dynamic range of pixel values, k1 = 0.01, k2 = 0.03.

[0143] The range of structural similarity is -1 to 1. When two images are exactly the same, the value of SSIM is equal to 1. As an implementation of the theory of structural similarity, the structural similarity index defines structural information as independent of brightness and contrast from the perspective of image composition, reflects the properties of object structure in the scene, and models distortion as a combination of brightness, contrast, and structure. The mean value is used as an estimate of brightness, the standard deviation is used as an estimate of contrast, and the covariance is used as a measure of structural similarity.

[0144] Figure 14 The blurred image recovery effect diagram provided by the present application is shown in the following figure. Figure 14 It is found that the blurred image can be clearly recovered.

[0145] Figure 15 The blurred image recovery device structure diagram provided by the present application is shown in the following figure.

[0146] The model training module 151 is configured to, for a plurality of sample images in a training set, perform noise addition processing on the sample images to obtain a plurality of noise-added sample image groups corresponding to the sample images; input the plurality of noise-added sample image groups and sample clear images corresponding to the plurality of noise-added sample image groups into the blurred image recovery model, and train the blurred image recovery model.

[0147] The image recovery module 152 is configured to acquire a blurred image to be processed, input the blurred image into the blurred image recovery model, and determine a clear image corresponding to the blurred image based on the blurred image recovery model.

[0148] The model training module 151 is configured to, for a plurality of sample images in a training set, perform noise addition processing on the sample images to obtain a plurality of noise-added sample image groups corresponding to the sample images; input the plurality of noise-added sample image groups and sample clear images corresponding to the plurality of noise-added sample image groups into the blurred image recovery model, and train the blurred image recovery model.

[0149] The model training module 151 is configured to, for a plurality of sample images in a training set, perform resolution adjustment on the sample images according to a plurality of preset resolutions to obtain a plurality of multi-resolution image groups corresponding to the sample images, perform noise addition processing on the multi-resolution image groups to obtain a plurality of noise-added sample image groups corresponding to the sample images, input character arrangement specification information, the plurality of noise-added sample image groups, sample clear images corresponding to the plurality of noise-added sample image groups, and text description information corresponding to the plurality of noise-added sample image groups into the fuzzy image recovery model, and train the fuzzy image recovery model.

[0150] The image recovery module 152 is configured to input the fuzzy image into the fuzzy image recovery model, and determine a clear image corresponding to the fuzzy image and target text description information in the fuzzy image based on the fuzzy image recovery model.

[0151] The image recovery module 152 is configured to obtain an image to be processed, and if the image is a character region image, take the character region image as the fuzzy image to be processed, or if the image is a region of interest (ROI) image or an original scene image, perform resolution adjustment on the image according to the plurality of preset resolutions, and take a plurality of multi-resolution image groups corresponding to the image as the fuzzy image to be processed.

[0152] The image recovery module 152 is configured to input the target text description information into a language question and answer model, and determine target question information and target answer information corresponding to the target text description information based on the language question and answer model.

[0153] The model training module 151 is further configured to input a language question and answer sample file into the language question and answer model, determine a set number of predicted question and answer information based on the language question and answer model, determine labels of the set number of predicted question and answer information, and train the language question and answer model according to the labels of the set number of predicted question and answer information, where the labels include correct question and answer labels and incorrect question and answer labels.

[0154] The model training module 151 is further configured to obtain a target clear image corresponding to a fuzzy image to be processed input by a user, determine a structural similarity between the clear image corresponding to the fuzzy image and the target clear image, update the fuzzy image recovery model with the clear image as a positive excitation prediction result if the structural similarity is greater than a preset first similarity threshold, and update the fuzzy image recovery model with the clear image as a negative excitation prediction result if the structural similarity is less than a preset second similarity threshold, where the preset second similarity threshold is less than the preset first similarity threshold.

[0155] The application also provides an electronic device, such as Figure 16As shown, the electronic device includes a processor 161, a communication interface 162, a memory 163, and a communication bus 164, wherein the processor 161, the communication interface 162, and the memory 163 communicate with each other through the communication bus 164.

[0156] The memory 163 stores a computer program, which, when executed by the processor 161, causes the processor 161 to perform any of the above method steps.

[0157] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0158] The communication interface 162 is used for communication between the electronic device and other devices.

[0159] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Alternatively, the memory can also be at least one storage device located away from the aforementioned processor.

[0160] The processor can be a general-purpose processor, including a central processing unit, a network processor (NP), and the like; can also be a Digital Signal Processing (DSP), an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like.

[0161] The present application also provides a computer storage readable storage medium, which stores a computer program executable by an electronic device, and when the program runs on the electronic device, causes the electronic device to perform any of the above method steps.

[0162] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all changes and modifications falling within the scope of the present application.

[0163] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for restoring a blurred image, characterized in that, The method includes: A blurred image to be processed is acquired, the blurred image is input into a blurred image restoration model, and a clear image corresponding to the blurred image and target text description information in the blurred image are determined based on the blurred image restoration model. The blurred image restoration model is trained in the following way: For multiple sample images in the training set, the resolution of the sample images is adjusted according to multiple preset resolutions to obtain multi-resolution image groups corresponding to the sample images. Noise is added to the multi-resolution image groups to obtain noisy sample image groups corresponding to the sample images. Character arrangement standard information, multiple noisy sample image groups, clear sample images corresponding to the multiple noisy sample image groups, and text description information corresponding to the multiple noisy sample image groups are input into the blurred image restoration model to train the blurred image restoration model.

2. The method as described in claim 1, characterized in that, Obtaining the blurred image to be processed includes: Obtain the image to be processed. If the image is a character region image, then the character region image is used as the blurred image to be processed. If the image is a region of interest (ROI) image or an original scene image, adjust the resolution of the image according to the preset multiple resolutions, and use the resulting multi-resolution image group as the blurred image to be processed.

3. The method as described in claim 1, characterized in that, The method further includes: The target text description information is input into the language question answering model, and the target question information and target answer information corresponding to the target text description information are determined based on the language question answering model.

4. The method as described in claim 3, characterized in that, The training process of the language question-answering model includes: Input the language question answering sample file into the language question answering model, determine a set number of predicted question answering information based on the language question answering model, determine the labels of the set number of predicted question answering information, and train the language question answering model based on the labels of the set number of predicted question answering information, wherein the labels include correct question answer labels and incorrect question answer labels.

5. The method as described in claim 1, characterized in that, The method further includes: Obtain the target clear image corresponding to the blurred image input by the user, and determine the structural similarity between the clear image corresponding to the blurred image and the target clear image; if the structural similarity is greater than a preset first similarity threshold, use the clear image as a positive stimulus prediction result to update the blurred image restoration model; if the structural similarity is less than a preset second similarity threshold, use the clear image as a negative stimulus prediction result to update the blurred image restoration model; wherein, the preset second similarity threshold is less than the preset first similarity threshold.

6. A device for restoring a blurred image, characterized in that, The device includes: The model training module is used to adjust the resolution of multiple sample images in the training set according to multiple preset resolutions to obtain multi-resolution image groups corresponding to the sample images; add noise to the multi-resolution image groups to obtain noisy sample image groups corresponding to the sample images; input character arrangement standard information, multiple noisy sample image groups, clear sample images corresponding to the multiple noisy sample image groups, and text description information corresponding to the multiple noisy sample image groups into the blurred image restoration model to train the blurred image restoration model; The image restoration module is used to acquire the blurred image to be processed, input the blurred image into the blurred image restoration model, and determine the clear image corresponding to the blurred image and the target text description information in the blurred image based on the blurred image restoration model.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-5.

Images