Image deblurring method, device, equipment, medium and computer program product

Abstract

The present application discloses an image deblurring method, apparatus, device, medium, and computer program product, relating to the field of image processing. The method comprises: obtaining an image to be processed, which is an image to be deblurred; inputting the image to be processed into a target image processing model for clarity enhancement processing to obtain a target image with enhanced clarity; wherein the target image processing model is obtained by training an image processing model to be trained using sample images, the image processing model is used to perform clarity enhancement processing on the sample images to obtain a first predicted image, and blur enhancement processing to obtain a second predicted image, and the model parameters of the target image processing model are obtained by comparative learning of the clarity ranking and constraint ranking between the sample image, the first predicted image, and the second predicted image. This method can improve the processing effect of the trained target image processing model.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image deblurring method, apparatus, device, medium, and computer program product. Background Art

[0002] Due to unstable factors in the image acquisition process, or the loss of some information in the image data during image transmission, storage, and compression, the image may be blurred.

[0003] In related technologies, when implementing image deblurring based on deep learning, sample images labeled with image clarity are usually input into the model to drive the model to learn the blur kernel that makes the clear image blurry, thereby training a model that can deblur images.

[0004] However, in the implementation process of the above solution, the labels of sample images are often obtained through manual annotation, and the accuracy of the annotation affects the effect of the model obtained by downstream training. Summary of the Invention

[0005] The embodiments of the present application provide an image deblurring method, apparatus, device, medium, and computer program product, which can improve the processing effect of image deblurring. The technical solution is as follows:

[0006] In one aspect, a method for deblurring an image is provided, the method comprising:

[0007] Acquire an image to be processed, where the image to be processed is an image to be deblurred;

[0008] Inputting the image to be processed into a target image processing model for clarity enhancement processing to obtain a target image with enhanced clarity;

[0009] In which, the target image processing model is obtained by training the image processing model to be trained through sample images, and the image processing model is used to perform clarity enhancement processing on the sample image to obtain a first predicted image, and blur enhancement processing to obtain a second predicted image. The model parameters of the target image processing model are obtained by comparing and learning the clarity sorting and constraint sorting between the sample image, the first predicted image and the second predicted image.

[0010] In another aspect, a device for deblurring an image is provided, the device comprising:

[0011] An acquisition module, configured to acquire an image to be processed, wherein the image to be processed is an image to be deblurred;

[0012] A processing module, configured to input the image to be processed into a target image processing model for performing definition enhancement processing to obtain a target image with enhanced definition;

[0013] In which, the target image processing model is obtained by training the image processing model to be trained through sample images, and the image processing model is used to perform clarity enhancement processing on the sample image to obtain a first predicted image, and blur enhancement processing to obtain a second predicted image. The model parameters of the target image processing model are obtained by comparing and learning the clarity sorting and constraint sorting between the sample image, the first predicted image and the second predicted image.

[0014] On the other hand, a computer device is provided, wherein the terminal includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the image deblurring method described in any one of the embodiments of the present application.

[0015] On the other hand, a computer-readable storage medium is provided, wherein at least one program code is stored in the computer-readable storage medium, and the program code is loaded and executed by a processor to implement the image deblurring method described in any one of the embodiments of the present application.

[0016] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image deblurring method described in any of the above embodiments.

[0017] The technical solution provided by this application includes at least the following beneficial effects:

[0018] In the training process of the image processing model for image clarity enhancement, the sample image is processed for clarity enhancement and blur enhancement respectively, thereby obtaining a first predicted image and a second predicted image. The image processing model is trained by the clarity arrangement between the sample image, the first predicted image and the second predicted image, thereby realizing unsupervised training of the model using contrastive learning. There is no need to label the sample images, which improves the training efficiency of the model and avoids the problem of label accuracy affecting the training effect. When the trained target image processing model is applied to image deblurring, the processing effect of the model is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a flowchart of an image deblurring method provided by an exemplary embodiment of the present application;

[0022] Figure 3 is a flowchart of an image deblurring method provided by an exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of an image processing model to be trained provided by an exemplary embodiment of the present application;

[0024] Figure 5 is a flowchart of an image deblurring method provided by an exemplary embodiment of the present application;

[0025] Figure 6 is a schematic diagram of determining a clarity evaluation value provided by an exemplary embodiment of the present application;

[0026] Figure 7 is a flowchart of an image deblurring method provided by an exemplary embodiment of the present application;

[0027] Figure 8 is a schematic diagram of an encoder provided by an exemplary embodiment of the present application;

[0028] Figure 9 is a schematic diagram of a decoder provided by an exemplary embodiment of the present application;

[0029] Figure 10 1 is a schematic diagram of a framework of an image deblurring method provided by an exemplary embodiment of the present application;

[0030] Figure 11 is a flowchart of an image deblurring method provided by an exemplary embodiment of the present application;

[0031] Figure 12 This is a structural block diagram of an image deblurring device provided by an exemplary embodiment of the present application;

[0032] Figure 13 This is a structural block diagram of an image deblurring device provided by an exemplary embodiment of the present application;

[0033] Figure 14 It is a structural diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0035] First, a brief introduction to the terms involved in the embodiments of this application is given:

[0036] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0037] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0038] Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0039] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, map construction, autonomous driving, and smart transportation.

[0040] In combination with the above explanations, the application scenarios of the embodiments of the present application are schematically described. The image deblurring method can be applied to any of the following scenarios:

[0041] First, this method can be applied to camera applications. Specifically, when capturing images through a camera application, the aforementioned image deblurring method can be used to enhance the clarity of the captured images in real time, thereby reducing image blur caused by device jitter, object motion, and other factors.

[0042] Secondly, the method can be applied to image processing applications. Schematically, an image deblurring function is provided by an image processing application. That is, an image deblurring function module is provided in the image processing application. The user can upload an image as an image to be processed, and then use the image deblurring function module in the image processing application to process the image to be processed, thereby obtaining a target image with enhanced clarity, so as to achieve, for example, enhancing the clarity of old photos, improving the clarity of low-resolution images, and processing images that have blurred problems due to transmission or compression.

[0043] Third, this method can be applied to vehicle-mounted terminal applications in vehicle-mounted scenarios. Schematically, a vehicle is equipped with a vehicle-mounted terminal, which can be connected to an image acquisition device configured on the vehicle. When the image acquisition device captures images of the environment and transmits the captured images to the vehicle-mounted terminal, the vehicle-mounted terminal can enhance the clarity of the captured images using the above method. The above images can be images captured of the vehicle's surroundings, or images captured of the vehicle's interior, for example, in imaging and shooting of a vehicle's rearview mirror.

[0044] It is worth noting that the above three scenarios are only examples. The method can also be applied to other scenarios that require image deblurring processing and is not specifically limited here.

[0045] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The computer system of the implementation environment includes: a terminal device 110, a server 120 and a communication network 130.

[0046] The terminal device 110 includes various devices such as mobile phones, tablet computers, desktop computers, portable laptops, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. Illustratively, a target application is running in the terminal device 110, and the target application provides an image deblurring function. Optionally, the target application can be traditional application software, cloud application software, a mini-program or application module in a host application, or a web platform, without limitation here. Optionally, the target application can be a camera application, an image processing application, a map application, etc., without specific limitation here.

[0047] Server 120 provides backend services for the target application. Illustratively, server 120 trains the image processing model to be trained using sample images to obtain the target image processing model. Terminal device 110 sends an image processing request to server 120, requesting image deblurring processing from server 120. The request includes the image to be processed.

[0048] In other embodiments, after the server 120 obtains the target image processing model through training, the target image processing model can also be sent to the terminal device 110. The terminal device 110 establishes a functional module that can locally implement image deblurring processing based on the above-mentioned target image processing model. When deblurring the image to be processed, it is implemented by calling the local functional module.

[0049] When the target image processing model is trained, the clarity enhancement processing of the image can be achieved. Schematically, after receiving the image processing request, the server 120 calls the pre-trained target image processing model to enhance the clarity of the image to be processed to obtain the target image. The server 120 returns the target image to the terminal device 110.

[0050] It is worth noting that the above-mentioned server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud security, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0051] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology, all based on the cloud computing business model. It can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark and will need to be transmitted to backend systems for logical processing. Data of varying levels will be processed separately, and data from all industries will require a strong system backend, which can only be achieved through cloud computing.

[0052] In some embodiments, the server 120 may also be implemented as a node in a blockchain system. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0053] Illustratively, the terminal device 110 and the server 120 are connected via a communication network 130 , wherein the communication network 130 may be a wired network or a wireless network, which is not limited here.

[0054] Please refer to Figure 2 , which shows an image deblurring method shown in an embodiment of the present application. In the embodiment of the present application, the image deblurring application process is schematically described. Schematically, the method is applied to Figure 1 The server shown is used as an example for illustration. It is worth noting that the method can also be applied to terminal devices, which is not specifically limited here. The method includes:

[0055] Step 201: Obtain an image to be processed.

[0056] Illustratively, the image to be processed is an image to be deblurred.

[0057] Optionally, the image to be processed may be uploaded by a terminal device, or may be read by the server from a database.

[0058] In some embodiments, when the image to be processed is uploaded by the terminal device, the image to be processed may be carried in an image processing request, and the image processing request instructs the server to perform deblurring processing on the image to be processed.

[0059] Optionally, the image to be processed may also be an image frame extracted from a specified video. In one example, the server performs frame processing on the specified video to obtain at least two image frames, and determines at least one image to be processed from the at least two image frames.

[0060] Step 202: Input the image to be processed into the target image processing model for clarity enhancement processing to obtain a target image with enhanced clarity.

[0061] Illustratively, the server inputs the image to be processed into a functional module provided with image deblurring, and the functional module performs clarity enhancement processing on the image to be processed by calling a target image processing model, thereby outputting a target image.

[0062] In some embodiments, the server returns the output target image to the terminal device. In other embodiments, when the image to be processed is an image extracted from a specified video, the obtained target image is combined with an unprocessed image frame in the specified video to generate the target video.

[0063] Schematically, the target image processing model is obtained by training the image processing model to be trained using sample images. The image processing model is used to perform clarity enhancement processing on the sample image to obtain a first predicted image, and blur enhancement processing to obtain a second predicted image. The model parameters of the target image processing model are obtained by comparing and learning the clarity sorting and constraint sorting between the sample image, the first predicted image, and the second predicted image.

[0064] In some embodiments, the image processing model includes a deblurring subnetwork and a reblurring subnetwork. The deblurring subnetwork is used to enhance the clarity of the sample image to obtain a first predicted image, while the reblurring subnetwork is used to enhance the blurriness of the sample image to obtain a second predicted image.

[0065] In some embodiments, during the training of the image processing model, the network parameters of the deblurring subnetwork and the reblurring subnetwork are adjusted by comparing and learning the clarity ranking and constraint ranking between the sample image, the first predicted image, and the second predicted image, and finally the target image processing model is trained to obtain the target image processing model, which includes the trained target deblurring subnetwork and target reblurring subnetwork.

[0066] In some embodiments, during the application of the target image processing model, since only the clarity enhancement processing of the image is required, during the application process, only the target deblurring subnetwork is used to process the input image to be processed.

[0067] In other embodiments, after the target image processing model is trained, the target image processing model is transmitted to the terminal device, that is, steps 201 to 202 are applied to the terminal device, and the terminal device applies the above target image processing model to the functional module of the target application.

[0068] In one example, after the terminal device obtains the target image processing model, the target image processing model is applied to the image processing component in the camera application. When the camera application of the terminal device is in running state and the image processing component is called, the image captured by the camera application is input into the target image processing model as the image to be processed, thereby outputting the target image with improved clarity.

[0069] For example, when the camera captures the environmental image in real time, the captured image is first input into the above-mentioned target image processing model in real time, and then the obtained clarity-enhanced image is displayed in real time; for another example, when a selection operation for a specified image in the album is received, and a deblurring processing instruction for the specified image is received, the above-mentioned specified image is input into the above-mentioned target image processing model, and then the target image with enhanced clarity is output and displayed.

[0070] To summarize, the image deblurring method provided in the embodiment of the present application, in the process of training an image processing model for image clarity enhancement, performs clarity enhancement processing and blur enhancement processing on the sample image respectively, thereby obtaining a first predicted image and a second predicted image, and trains the image processing model through the clarity arrangement between the sample image, the first predicted image, and the second predicted image, thereby realizing unsupervised training of the model using contrastive learning, without the need to label the sample image, thereby improving the training efficiency of the model and avoiding the problem of label accuracy affecting the training effect, so that the target image processing model obtained by training can be applied to the processing effect when image clarity enhancement is achieved.

[0071] Please refer to Figure 3, which shows an image deblurring method according to an embodiment of the present application. In the embodiment of the present application, the training process of the image processing model to be trained is schematically described, that is, steps 301 to 304 are performed before step 201. The method includes:

[0072] Step 301: Acquire a sample image.

[0073] Illustratively, the above sample images are used to train the image processing model to be trained.

[0074] Optionally, the sample images may be uploaded by a terminal device, obtained from a database, or obtained from a public data set on the Internet.

[0075] Optionally, the model structure of the image processing model to be trained may be indicated by the terminal device or obtained from a database. Schematically, before training the image processing model to be processed, the model parameters are initialized to obtain the model initialization parameters for starting the model training.

[0076] In some embodiments, when deblurring an image using an image processing model, an unsupervised contrastive learning approach is employed during the training phase to train the model. Illustratively, the image processing model includes a deblurring subnetwork and a reblurring subnetwork, wherein the deblurring subnetwork is used to enhance clarity, and the reblurring subnetwork is used to enhance blur.

[0077] Step 302: Perform definition enhancement processing on the sample image through an image processing model to obtain a first predicted image.

[0078] In an embodiment of the present application, the sample image is subjected to clarity enhancement processing by the deblurring subnetwork in the above-mentioned image processing model to obtain a first predicted image, wherein, during the overall model training process of the image processing model, the training objective of the deblurring subnetwork is to obtain a first predicted image with higher clarity than the input sample image.

[0079] Optionally, the above-mentioned deblurring sub-network can use at least one network structure such as Convolutional Neural Networks (CNN), Recurrent Neural Network (RNN), Visual Geometry Group Network (VGG), and Residual Network (ResNet).

[0080] In some embodiments, before performing clarity enhancement on a sample image, the sample image may be preprocessed and then fed into the deblurring subnetwork. Optionally, the preprocessing operation may include at least one of edge detection, scaling, frequency domain processing, and histogram contrast enhancement.

[0081] Step 303: Perform blur enhancement processing on the sample image through the image processing model to obtain a second predicted image.

[0082] In an embodiment of the present application, the sample image is subjected to competitive blur enhancement processing through the re-blurring sub-network in the above-mentioned image processing model to obtain a second predicted image, wherein, during the overall model training process of the image processing model, the training objective of the re-blurring sub-network is to obtain a second predicted image with lower clarity than the input sample image.

[0083] Optionally, the defuzzification subnetwork may use at least one network structure such as CNN, RNN, VGG, or ResNet. In some embodiments, the defuzzification subnetwork and the refuzzification subnetwork may use the same network structure, or the defuzzification subnetwork and the refuzzification subnetwork may use different network structures, which are not limited herein.

[0084] In some embodiments, before blur enhancement is performed on a sample image, the sample image may be preprocessed and then input into the re-blurring subnetwork. Optionally, the preprocessing operation may include at least one image processing operation selected from edge detection, scaling, frequency domain processing, and histogram contrast enhancement.

[0085] Step 304 : Training the image processing model based on the ranking loss between the sample image, the first predicted image, and the second predicted image to obtain a target image processing model.

[0086] Schematically, the sorting loss is used to indicate the difference between the sorting situation of image clarity between the sample image, the first predicted image and the second predicted image and the constrained sorting situation, wherein the above-mentioned constrained sorting situation indicates that the second clarity of the first predicted image > the first clarity of the sample image > the third clarity of the second predicted image.

[0087] After obtaining the first predicted image and the second predicted image through the deblurring subnetwork and the reblurring subnetwork respectively, a triplet is formed by the sample image, the first predicted image and the second predicted image, and comparative learning is realized through the above triplet based on the mutual constraint relationship between the three, so as to train the overall model so that the first predicted image obtained by the deblurring subnetwork, the second predicted image obtained by the reblurring subnetwork and the sample image can meet the constraint conditions corresponding to the training objectives.

[0088] In schematic form, according to the training objectives corresponding to the deblurring subnetwork and the reblurring subnetwork respectively, it can be known that the deblurring subnetwork needs to improve the clarity of the input image as much as possible, and the reblurring subnetwork needs to improve the blurriness of the input image as much as possible. Therefore, for the triple consisting of the sample image, the first predicted image and the second predicted image, the corresponding specified constraints in the training process are that the second clarity of the first predicted image is greater than the first clarity of the sample image, and the first clarity of the sample image is greater than the third clarity of the second predicted image, that is, the second clarity > the first clarity > the third clarity.

[0089] The image processing model is trained according to the above constraints. Schematically, a first evaluation value of the sample image, a second evaluation value of the first predicted image, and a third evaluation value of the second predicted image are obtained. The first evaluation value is used to indicate the clarity of the sample image, the second evaluation value is used to indicate the clarity of the first predicted image, and the third evaluation value is used to indicate the clarity of the second predicted image. Based on the difference between the comparison results between the first evaluation value, the second evaluation value, and the third evaluation value and the specified constraints, a sorting loss value is determined. Based on the sorting loss value, the image processing model is trained to obtain a target image processing model.

[0090] In some embodiments, when determining the difference between the comparison results of the first evaluation value, the second evaluation value, and the third evaluation value and the specified constraint, a ranking loss value can be determined by pairwise comparison. For example, if the constraint is that the second evaluation value is greater than the first evaluation value, then the loss L1 corresponding to the constraint is as shown in Formula 1, where f2 indicates the second evaluation value, f1 indicates the first evaluation value, and m indicates a hyperparameter of the constraint.

[0091] Formula 1: L1 = max{0,m-(f2-f1)}

[0092] Similarly, the constraints of the second evaluation value > the third evaluation value, and the first evaluation value > the third evaluation value, correspond to losses L2 and L3 respectively. The sorting loss value corresponding to the triplet can be determined through the above three losses, so as to adjust the image processing model according to the sorting loss value, that is, adjust the network parameters of the deblurring subnetwork and the reblurring subnetwork, and finally train to obtain the target image processing model.

[0093] Indicative, such as Figure 4 , which shows a schematic diagram of an image processing model to be trained provided by an exemplary embodiment of the present application. The image processing model to be trained 400 includes a deblurring subnetwork 410 and a reblurring subnetwork 420. The sample image 401 is input into the deblurring subnetwork 410 and the reblurring subnetwork 420, respectively, and the first predicted image 411 and the second predicted image 421 are output. The sample image 401, the first predicted image 411, and the second predicted image 421 are input into the loss determination module 430, and the image processing model to be trained 400 is trained according to the ranking loss value determined by the loss determination module 430.

[0094] In some embodiments, after the image processing model to be trained is trained to obtain a target image processing model, the target image processing model includes a target deblurring subnetwork that has completed training. Schematically, the target deblurring subnetwork is applied as a network for clarity enhancement.

[0095] Schematically, the image to be processed is input into the above-mentioned target deblurring subnetwork, and the target image is output, wherein the clarity of the target image is higher than that of the image to be processed.

[0096] To summarize, the image deblurring method provided in the embodiment of the present application, in the process of training an image processing model for image clarity enhancement, performs clarity enhancement processing and blur enhancement processing on the sample image respectively, thereby obtaining a first predicted image and a second predicted image, and trains the image processing model through the clarity arrangement between the sample image, the first predicted image, and the second predicted image, thereby realizing unsupervised training of the model using contrastive learning, without the need to label the sample image, thereby improving the training efficiency of the model and avoiding the problem of label accuracy affecting the training effect, so that the target image processing model obtained by training can be applied to the processing effect when image clarity enhancement is achieved.

[0097] Please refer to Figure 5 , which shows an exemplary embodiment of the present application providing an image deblurring method. In the embodiment of the present application, the determination of the sorting loss is schematically illustrated. The method includes:

[0098] Step 501: Acquire a sample image.

[0099] Illustratively, the sample images are used to train an image processing model to be trained. The image processing model to be trained includes a deblurring subnetwork and a reblurring subnetwork to be trained, wherein the deblurring subnetwork is used to achieve clarity enhancement processing, and the reblurring subnetwork is used to achieve blur enhancement processing.

[0100] Step 502: Perform clarity enhancement processing on the sample image through the deblurring sub-network to obtain a first predicted image.

[0101] In an embodiment of the present application, the sample image is subjected to clarity enhancement processing by the deblurring subnetwork in the above-mentioned image processing model to obtain a first predicted image, wherein, during the overall model training process of the image processing model, the training objective of the deblurring subnetwork is to obtain a first predicted image with higher clarity than the input sample image.

[0102] Step 503: Perform blur enhancement processing on the sample image through the re-blurring sub-network to obtain a second predicted image.

[0103] In an embodiment of the present application, the sample image is subjected to competitive blur enhancement processing through the re-blurring sub-network in the above-mentioned image processing model to obtain a second predicted image, wherein, during the overall model training process of the image processing model, the training objective of the re-blurring sub-network is to obtain a second predicted image with lower clarity than the input sample image.

[0104] Step 504 : Obtain a first evaluation value of the sample image, a second evaluation value of the first predicted image, and a third evaluation value of the second predicted image.

[0105] In an embodiment of the present application, a first evaluation value is obtained for the sample image, a second evaluation value is obtained for the first predicted image, and a third evaluation value is obtained for the third predicted image, wherein the first evaluation value is used to indicate the clarity of the sample image, the second evaluation value is used to indicate the clarity of the first predicted image, and the third evaluation value is used to indicate the clarity of the second predicted image.

[0106] Optionally, the evaluation value of the image clarity may be determined by at least one of the following methods:

[0107] The first one determines the image clarity by calculating the gradient of the image.

[0108] For example, prior experience with image clarity shows that edges in high-resolution images are sharper, resulting in larger calculated gradients. In contrast, edges in blurry images are blurrier, resulting in smaller calculated gradients. Therefore, given two images with identical content, if one is sharp and the other is blurry, the calculated gradient information for the sharp image will be richer, meaning it will be larger, than that for the blurry image. Therefore, image clarity can be indicated by calculating its gradient information.

[0109] Optionally, when calculating the gradient information of an image, the gradient in a specified direction may be calculated, and the gradient information corresponding to the gradient in the specified direction may be used as the image clarity. Alternatively, the gradient in multiple directions may be calculated, and the gradient information corresponding to the gradients in the multiple directions may be combined to determine the image clarity. Illustratively, the gradient is a vector used to describe the rate of change of a pixel point in an image in a certain direction, and the gradient information indicates that the image gradient can be used as data for determining image clarity after appropriate intermediate processing. When determining image clarity only for the gradient in a specified direction, since each pixel only includes the gradient in one direction, the gradient in the specified direction may be directly used as the gradient information for determining image clarity. When determining image clarity for the gradient in multiple directions, since each pixel includes the gradient in multiple directions, when determining the gradient information corresponding to the gradient in the i-th direction, the gradient in the i-th direction may be intermediately processed to obtain the gradient information corresponding to the i-th direction, and the image clarity may be determined based on the gradient information in each direction. Optionally, the above-mentioned intermediate processing may include at least one processing method of determining the absolute value corresponding to the gradient in the i-th direction, determining the square corresponding to the gradient in the i-th direction, determining the mapping value of the gradient in the i-th direction in the normalized direction of multiple directions, etc.

[0110] Taking determining an evaluation value based on gradient information in two perpendicular directions as an example, illustratively, first gradient information of a sample image in a first direction and second gradient information in a second direction are determined, wherein the first direction and the second direction are perpendicular to each other; a first evaluation value is determined based on the first gradient information and the second gradient information; third gradient information of the first predicted image in the first direction and fourth gradient information in the second direction are determined; a second evaluation value is determined based on the third gradient information and the fourth gradient information; fifth gradient information of the second predicted image in the first direction and sixth gradient information in the second direction are determined; and a third evaluation value is determined based on the fifth gradient information and the sixth gradient information.

[0111] Schematically, the image gradient is calculated as shown in Formula 2 and Formula 3, wherein Formula 2 indicates the gradient G of the image I in the y direction: y Calculation method, x is the direction perpendicular to y, formula 3 indicates the gradient G of image I in the x direction x Calculation method.

[0112] Formula 2: G y =I(x,y+1)-I(x,y-1)

[0113] Formula 3: G x =I(x+1,y)-I(x-1,y)

[0114] Illustratively, when the evaluation value is determined by gradient information in two directions, the overall gradient information can be determined by Formula 4, and the overall gradient information is used as the evaluation value of the image clarity.

[0115] Formula 4:

[0116] In one example, the first direction and the second direction may be horizontal and vertical directions, respectively.

[0117] The second method is to determine the clarity of the image through a pre-trained clarity assessment subnetwork.

[0118] Illustratively, the image clarity can also be determined by data-driven means, that is, the clarity of the input image is predicted by a pre-trained clarity assessment subnetwork, and a corresponding assessment value is output.

[0119] Indicative, such as Figure 6 As shown, it shows a schematic diagram of determining the clarity evaluation value provided by an exemplary embodiment of the present application, wherein the image to be detected (including the sample image, the first predicted image and the second predicted image) 601 is input into the clarity evaluation subnetwork 610, and the clarity evaluation value 602 corresponding to the image to be detected 601 is output, wherein the clarity evaluation subnetwork 610 is pre-trained by big data 603.

[0120] In some embodiments, features are extracted from the sample image, the first predicted image, and the second predicted image to obtain a first feature representation corresponding to the sample image, a second feature representation corresponding to the first predicted image, and a third feature representation corresponding to the second predicted image, respectively; the first feature representation, the second feature representation, and the third feature representation are respectively input into a clarity evaluation subnetwork, and a prediction is made on the distribution of the feature representations in the feature space, and a first evaluation value of the sample image, a second evaluation value of the first predicted image, and a third evaluation value of the second predicted image are output.

[0121] Optionally, the clarity assessment subnetwork may be at least one network such as CNN, RNN, ResNet, or pulse coupled neural network that can be used for image clarity estimation.

[0122] Step 505 : Determine a sorting loss value based on the difference between the comparison result between the first evaluation value, the second evaluation value, and the third evaluation value and the specified constraint condition.

[0123] Schematically, after obtaining the first predicted image and the second predicted image through the deblurring subnetwork and the reblurring subnetwork respectively, a triplet is formed by the sample image, the first predicted image and the second predicted image, and comparative learning is realized through the above triplets based on the mutual constraint relationship between the three, so as to train the overall model so that the first predicted image obtained by the deblurring subnetwork, the second predicted image obtained by the reblurring subnetwork and the sample image can meet the constraint conditions corresponding to the training objectives.

[0124] Schematically, the ranking result of the evaluation values ​​expected to be achieved for the specified constraints corresponding to the image processing model to be trained is the second evaluation value > the first evaluation value > the third evaluation value. Therefore, the determination of the ranking loss can be broken down into the comparison relationship between the first evaluation value and the second evaluation value, the comparison relationship between the first evaluation value and the third evaluation value, and the comparison relationship between the second evaluation value and the third evaluation value. The overall ranking loss value of the model is determined by the loss values ​​corresponding to the above three comparison relationships. The specific determination method is shown in step 204 and will not be repeated here.

[0125] Step 506 : Train the network parameters of the defuzzification sub-network and the refuzzification sub-network based on the ranking loss value to obtain a target defuzzification sub-network and a target refuzzification sub-network.

[0126] In some embodiments, in response to the ranking loss value converging, it is determined that the training of the defuzzification sub-network and the refuzzification sub-network is completed, and the model parameters at the time of convergence are used as the network parameters of the target defuzzification sub-network and the target refuzzification sub-network.

[0127] Step 507: Input the image to be processed into the target deblurring subnetwork for clarity enhancement processing to obtain a target image with enhanced clarity.

[0128] Schematically, the image to be processed is input into the above-mentioned target deblurring subnetwork, and the target image is output, wherein the clarity of the target image is higher than that of the image to be processed.

[0129] Optionally, the image to be processed may be uploaded by a terminal device, or may be read by the server from a database.

[0130] To summarize, the image deblurring method provided in the embodiment of the present application, in the process of training an image processing model for image clarity enhancement, performs clarity enhancement processing and blur enhancement processing on the sample image respectively, thereby obtaining a first predicted image and a second predicted image, and trains the image processing model through the clarity arrangement between the sample image, the first predicted image, and the second predicted image, thereby realizing unsupervised training of the model using contrastive learning, without the need to label the sample image, thereby improving the training efficiency of the model and avoiding the problem of label accuracy affecting the training effect, so that the target image processing model obtained by training can be applied to the processing effect when image clarity enhancement is achieved.

[0131] Please refer to Figure 7 , which shows an image deblurring method provided by an exemplary embodiment of the present application. In this embodiment of the present application, the image processing model in the training phase includes a deblurring subnetwork, a reblurring subnetwork, a priori calculation module, a data-driven calculation module, and a loss determination module. Both the deblurring subnetwork and the reblurring subnetwork use an encoding-decoding network structure. The method includes:

[0132] Step 701: Acquire a sample image.

[0133] Illustratively, the above sample images are used to train the image processing model to be trained.

[0134] Schematically, before the sample image is processed by the deblurring sub-network and the reblurring sub-network, it is necessary to extract features from the sample image, and input the feature representation corresponding to the sample image into the deblurring sub-network and the reblurring sub-network for prediction, thereby obtaining the first predicted image and the second predicted image.

[0135] In some embodiments, in order to achieve better clarity enhancement or blur enhancement effects when performing image processing through the deblurring subnetwork and the reblurring subnetwork, before inputting the sample image into the deblurring subnetwork and the reblurring subnetwork, the image semantic features of the sample image are extracted as the input of the deblurring subnetwork and the reblurring subnetwork, thereby achieving assisted clarity enhancement and blur enhancement based on image semantics.

[0136] Schematically, the image semantics of the sample image are semantically classified in at least two semantic spaces to obtain image semantic features corresponding to the semantic spaces respectively; the image semantic features are feature-connected to obtain sample image features; the sample image features are input into a deblurring subnetwork, and the deblurring subnetwork enhances the clarity of the sample image features based on the image semantic guidance of the sample image to obtain a first predicted image; the sample image features are input into a reblurring subnetwork, and the reblurring subnetwork enhances the blurriness of the sample image features based on the image semantic guidance of the sample image to obtain a second predicted image.

[0137] In some embodiments, a CNN can be used to extract a visual feature map from a sample image. This visual feature map is then convolved to obtain image semantic features corresponding to different semantic spaces, thereby separating the semantics corresponding to different content in the image. Optionally, the semantic space can be divided based on the statement object corresponding to the content. The semantic space can include a subject semantic space, an object semantic space, an object semantic space, or a relationship semantic space.

[0138] Illustratively, image semantic features corresponding to the sample image are determined based on at least two semantic spaces, thereby indicating that the sample image carries spatial entity semantic structure information. In some embodiments, image semantic features from different semantic spaces are merged to obtain sample image features for input into the deblurring subnetwork and the reblurring subnetwork.

[0139] Specifically, when obtaining sample image features through image semantic features, feature extraction can be performed on the sample image through CNN to obtain a visual feature map, and the visual feature map is convolved into the above-mentioned at least two semantic spaces. Taking the semantic space including the subject semantic space and the object semantic space as an example, the convolution feature maps on the subject semantic space and the object semantic space are respectively obtained, and the convolution feature maps corresponding to the above-mentioned subject semantic space and object semantic space are used as leaf nodes of the feature tree. The leaf nodes are convolved and merged into the feature map of their corresponding parent nodes to obtain the overall feature map corresponding to the sample image. The sample image features are obtained by transforming the overall feature map through nonlinear functions, average pooling operations and full connection operations, wherein the sample image features can be in the form of a matrix or a vector.

[0140] Schematically, the above process of extracting image semantics as features is implemented by a pre-trained semantic feature extraction subnetwork, wherein, during the training process of the above semantic feature extraction subnetwork, supervised training can be performed according to the labels corresponding to the feature tree, and the above supervised training process is achieved by calculating the cross entropy loss between leaf nodes.

[0141] Step 7021: Input the sample image into a first encoder for encoding to obtain a first encoding feature.

[0142] Schematically, the deblurring subnetwork in the embodiment of the present application adopts an encoding-decoding structure, that is, the deblurring subnetwork is composed of a first encoder and a first decoder, wherein the above-mentioned first encoder is used to extract the features of the sample image and thus perform feature encoding, and the first decoder decodes the extracted features and returns the features to the image space, thereby outputting a first predicted image.

[0143] In some embodiments, the structure of the first encoder may include at least one convolution layer. Schematically, the sample image is convolved with the first encoder, thereby reducing the spatial resolution of the feature map of the sample image while increasing the number of channels.

[0144] In one example, if Figure 8 As shown, it shows a schematic diagram of an encoder provided by an exemplary embodiment of the present application, wherein the encoder 800 includes a first convolutional layer 810 and a second convolutional layer 820. After the sample image 801 is input into the first convolutional layer 810, an intermediate feature 802 with reduced resolution is obtained. The intermediate feature 802 passes through the second convolutional layer 820 to obtain a first encoded feature 803 with an increased number of channels.

[0145] Step 7022: Input the first encoding feature into a first decoder for decoding to obtain a first predicted image.

[0146] In some embodiments, the network structure of the first decoder can be symmetrical with that of the first encoder. Schematically, the first decoder improves the control resolution of the feature map through deconvolution, while reducing the number of feature channels, and finally decodes back to the image space corresponding to the sample image to obtain the first predicted image.

[0147] In one example, if Figure 9 As shown, it shows a schematic diagram of a decoder provided by an exemplary embodiment of the present application, wherein the decoder 900 includes a third convolutional layer 910 and a fourth convolutional layer 920. After the sample image 901 is input into the third convolutional layer 910, an intermediate feature 902 with improved resolution is obtained, and the intermediate feature 902 passes through the fourth convolutional layer 920 to obtain a first predicted image 903 with a reduced number of channels.

[0148] Step 7031: Input the sample image into the second encoder for encoding to obtain a second encoding feature;

[0149] Schematically, the re-blurring sub-network in the embodiment of the present application adopts an encoding-decoding structure, that is, the re-blurring sub-network is composed of a second encoder and a second decoder, wherein the second encoder is used to extract the features of the sample image and thus perform feature encoding, and the second decoder is used to decode the extracted features and return the features to the image space, thereby outputting a second predicted image.

[0150] Optionally, the structure of the second encoder may be the same as that of the first encoder. Schematically, the sample image is convolved by the second encoder, thereby reducing the spatial resolution of the feature map of the sample image while increasing the number of channels.

[0151] Step 7032: Input the second encoded feature into a second decoder for decoding to obtain a second predicted image.

[0152] Optionally, the structure of the second decoder can be the same as that of the first decoder. Schematically, the second decoder improves the control resolution of the feature map through deconvolution while reducing the number of feature channels, and finally decodes it back to the image space corresponding to the sample image to obtain the second predicted image.

[0153] Step 7041: Input the sample image, the first predicted image, and the second predicted image into the prior calculation module to obtain a first evaluation value corresponding to the sample image, a second evaluation value corresponding to the first predicted image, and a third evaluation value corresponding to the second predicted image.

[0154] Illustratively, the a priori calculation module determines image clarity by calculating the image's gradient. Optionally, when calculating the image's gradient information, the gradient in a specific direction can be calculated and used as the image clarity. Alternatively, the gradients in multiple directions can be calculated and combined to determine an evaluation value corresponding to the image clarity.

[0155] Taking determining an evaluation value based on gradient information in two perpendicular directions as an example, illustratively, first gradient information of a sample image in a first direction and second gradient information in a second direction are determined, wherein the first direction and the second direction are perpendicular to each other; a first evaluation value is determined based on the first gradient information and the second gradient information; third gradient information of the first predicted image in the first direction and fourth gradient information in the second direction are determined; a second evaluation value is determined based on the third gradient information and the fourth gradient information; fifth gradient information of the second predicted image in the first direction and sixth gradient information in the second direction are determined; and a third evaluation value is determined based on the fifth gradient information and the sixth gradient information.

[0156] Step 7042: Input the sample image, the first predicted image, and the second predicted image into the data-driven calculation module to obtain the fourth evaluation value corresponding to the sample image, the fifth evaluation value corresponding to the first predicted image, and the sixth evaluation value corresponding to the second predicted image.

[0157] Schematically, the data-driven computing module determines the clarity of the image through a pre-trained clarity assessment subnetwork.

[0158] In some embodiments, features are extracted from the sample image, the first predicted image, and the second predicted image to obtain a first feature representation corresponding to the sample image, a second feature representation corresponding to the first predicted image, and a third feature representation corresponding to the second predicted image, respectively; the first feature representation, the second feature representation, and the third feature representation are respectively input into a clarity evaluation subnetwork, and a prediction is made on the distribution of the feature representations in the feature space, and a fourth evaluation value of the sample image, a fifth evaluation value of the first predicted image, and a sixth evaluation value of the second predicted image are output.

[0159] Step 705 : Determine a sorting loss value based on the comparison results among the first evaluation value, the second evaluation value, and the third evaluation value, and the comparison results among the fourth evaluation value, the fifth evaluation value, and the sixth evaluation value, in combination with specified constraints.

[0160] Schematically, the ranking result of the evaluation values ​​expected to be achieved under the specified constraints is the second evaluation value > the first evaluation value > the third evaluation value, and the fifth evaluation value > the fourth evaluation value > the sixth evaluation value. Therefore, the ranking loss value can be determined by calculating the loss between the size relationship between the first evaluation value, the second evaluation value, and the third evaluation value and the specified constraints, and the loss between the size relationship between the fourth evaluation value, the fifth evaluation value, and the sixth evaluation value and the specified constraints. Optionally, the ranking loss value can be the sum of the losses corresponding to the two groups of evaluation values, or the average of the losses corresponding to the two groups of evaluation values.

[0161] Step 706: Train the image processing model based on the ranking loss value to obtain a target image processing model.

[0162] In some embodiments, in response to the ranking loss value converging, it is determined that the image processing model training is completed, and a target image processing model is obtained, wherein the target image processing model includes a target deblurring subnetwork and a target reblurring subnetwork.

[0163] In one example, if Figure 10As shown, it shows the framework of the image deblurring method provided by an exemplary embodiment of the present application. The sample image 1001 is input into the deblurring sub-network 1010 including the first encoder 1011 and the first decoder 1012, and the reblurring sub-network 1020 including the second encoder 1021 and the second decoder 1022, and the first predicted image 1002 and the second predicted image 903 are output. In addition to the direct connection between the first encoder 1011 and the first decoder 1012, a skip connection (Skip Connections, that is, the output of each convolutional layer in the first encoder 1011 is jumped to the convolutional layer in the first decoder 1012. For example, taking the first encoder 1011 including the first convolutional layer and the second convolutional layer, and the first decoder 1012 including the third convolutional layer and the fourth convolutional layer as an example, in addition to being directly connected to the second convolutional layer, the output of the first convolutional layer also needs to be input to the third and fourth convolutional layers. In addition to being directly connected to the third convolutional layer, the output of the second convolutional layer also needs to be input to the fourth convolutional layer. Jumping ensures the reusability of features and reduces the learning of redundant mappings. The sample image 1001, the first predicted image 1002, and the second predicted image 1003 are input to the prior calculation module 1030 and the data-driven calculation module 1040. The output evaluation value indicating the clarity of the image is determined by the loss determination module 1050 to determine the ranking loss value, so that the overall model is trained according to the ranking loss value.

[0164] Step 707: Input the image to be processed into the target deblurring subnetwork in the target image processing model for clarity enhancement processing to obtain a target image with enhanced clarity.

[0165] Schematically, the image to be processed is input into the above-mentioned target deblurring subnetwork, and the target image is output, wherein the clarity of the target image is higher than that of the image to be processed.

[0166] To summarize, the image deblurring method provided in the embodiment of the present application, in the process of training an image processing model for image clarity enhancement, performs clarity enhancement processing and blur enhancement processing on the sample image respectively, thereby obtaining a first predicted image and a second predicted image, and trains the image processing model through the clarity arrangement between the sample image, the first predicted image, and the second predicted image, thereby realizing unsupervised training of the model using contrastive learning, without the need to label the sample image, thereby improving the training efficiency of the model and avoiding the problem of label accuracy affecting the training effect, so that the target image processing model obtained by training can be applied to the processing effect when image clarity enhancement is achieved.

[0167] Please refer to Figure 11, which shows an image deblurring method provided by an exemplary embodiment of the present application. In the embodiment of the present application, the method is applied to a vehicle-mounted scene as an example for schematic illustration. The method includes:

[0168] Step 1101: Acquire an image to be processed collected by an onboard device.

[0169] In an embodiment of the present application, a trained target image processing model is applied to image processing in an in-vehicle scene. The training objective of the target image processing model during the training phase includes deblurring images exhibiting motion blur. Illustratively, during the training phase of the image processing model, the input sample images include images containing moving objects.

[0170] Illustratively, the image to be processed is an image acquired by the vehicle-mounted device from the surrounding environment. In some embodiments, the image to be processed includes a target area with motion blur.

[0171] In one example, the above-mentioned image to be processed may be a driving scene in which the environment surrounding the vehicle is displayed through an electronic rearview mirror. The image acquisition device corresponding to the electronic rearview mirror acquires images of the environment surrounding the vehicle to obtain continuous environmental image frames, and the above-mentioned environmental image frames are used as the image to be processed for clarity enhancement processing.

[0172] In another example, the image to be processed can be a map data generation scenario. Schematically, an information collection vehicle collects the road environment, and then generates map data for a specified target. For example, electronic cameras along the road are identified and aggregated to provide map data corresponding to the cameras for a map application. Schematically, the collection equipment onboard the information collection vehicle collects environmental images. Before generating map data for the environmental images, the target image processing model can be used to enhance the clarity of the collected environmental images to improve the accuracy of the subsequent map data generation.

[0173] Step 1102: input the image to be processed into the target image processing model, and the target image processing model performs clarity enhancement processing on the target area to obtain the target image.

[0174] Optionally, the vehicle-mounted terminal uploads the collected image to be processed to the server via the network, and the server calls the target image processing model to perform clarity enhancement processing on the above-mentioned image to be processed; or, the vehicle-mounted terminal is equipped with a component including the target image processing model, and the vehicle-mounted terminal calls the above-mentioned component to perform clarity enhancement processing on the image to be processed.

[0175] Schematically, taking the above-mentioned electronic rearview mirror displaying the vehicle's surrounding environment as an example, the image acquisition device corresponding to the electronic rearview mirror will capture the environmental image frame and input it into a chip or processing unit equipped with the above-mentioned target image processing model component for clarity enhancement processing, thereby obtaining a clarity-enhanced target image frame, and the vehicle's surrounding environment will be displayed on the electronic rearview mirror based on the target image frame.

[0176] Schematically, taking the above-mentioned information collection vehicle collecting the environment along the road as an example, the image collection device corresponding to the information collection vehicle collects the surrounding environment according to the specified collection frequency, and transmits the collected environmental image to the server for map data generation. After receiving the above-mentioned environmental image, the server first enhances the clarity of the environmental image through the target image processing model to obtain the target image, and uses the target image to generate map data. For example, the electronic eyes in multiple target images are aggregated to establish a connection between the electronic eyes in the image and the electronic eye entities in the real world.

[0177] In summary, the image deblurring method provided in the embodiment of the present application applies the trained target image processing model to the vehicle-mounted scene, and enhances the clarity of the environmental images collected in the vehicle-mounted scene, thereby improving the clarity of the environmental images used in the vehicle-mounted scene. For example, the clarity of the content displayed in the electronic rearview mirror is improved, or the accuracy of map data generation is improved by improving the clarity of the image used for map data generation.

[0178] It is worth noting that the image deblurring method provided in the embodiments of the present application can also be applied to camera applications. For example, in a camera application scenario, the camera application can input the captured image as the image to be processed into a target image processing model for clarity enhancement to obtain a target image. Optionally, the target image processing model can be set in a server or installed in a terminal device corresponding to the camera application.

[0179] In some embodiments, in order to improve the effect of the target image processing model when performing clarity enhancement, model training can also be performed based on the shooting habits of the photographer when shooting through the camera application in the terminal device, that is, the training objectives of the image processing model to be trained include learning the shooting habits corresponding to the target account, and applying the shooting habits to image clarity enhancement. The above-mentioned shooting habits are used to indicate the factors that cause image blur when the target account shoots images. In one example, it is determined based on historical captured images that the target account often has shaking and blurring when shooting images, so that it is known that the shooting habits of the target account include shaking habits. For example, the photographer often has hand shaking when holding the terminal device during the shooting process, resulting in shaking and blurring of the images captured by the camera. Therefore, when training the image processing model, the blur kernel corresponding to the shaking blur is learned through historical captured images with shaking and blurring, so that the trained deblurring network is more inclined to process shaking and blurring images, providing more targeted image processing functions for the target account. In another example, based on historical captured images, it is determined that the target account often encounters low brightness when taking images, so that it is known that the target account's shooting habits include the habit of shooting in low-brightness environments. For example, when the photographer takes pictures at night, the clarity of the image decreases due to the low brightness of the environment. Therefore, when training the image processing model, the blur kernel corresponding to low-brightness blur is learned through historical captured images with low brightness, so that the trained deblurring network is more inclined to process low-brightness blurred images, providing more targeted image processing functions for the target account.

[0180] Illustratively, with full authorization from the target account, the terminal device sends historical images to the server. These images are images captured by the target account through the terminal device during a historical period. The server receives the historical images sent by the target account and uses them as sample images to train the image processing model to be trained. The trained target image processing model is then used as the processing model for the target account to enhance image clarity through the camera application. Optionally, the target image processing model is sent to the terminal device corresponding to the target account, or the target image processing model and the target account's account ID are stored in a corresponding manner.

[0181] Optionally, the training frequency of the model training using the historical captured images of the target account may be specified by the target account or preset by the system, for example, the target image processing model may be updated at a frequency of once a month.

[0182] Please refer to Figure 12 , which shows a structural block diagram of an image deblurring device provided by an exemplary embodiment of the present application, the device includes the following modules:

[0183] An acquisition module 1210 is configured to acquire an image to be processed, where the image to be processed is an image to be deblurred;

[0184] The processing module 1220 is configured to input the image to be processed into a target image processing model for performing definition enhancement processing to obtain a target image with enhanced definition;

[0185] In which, the target image processing model is obtained by training the image processing model to be trained through sample images, and the image processing model is used to perform clarity enhancement processing on the sample image to obtain a first predicted image, and blur enhancement processing to obtain a second predicted image. The model parameters of the target image processing model are obtained by comparing and learning the clarity sorting and constraint sorting between the sample image, the first predicted image and the second predicted image.

[0186] In some optional embodiments, such as Figure 13 As shown, the apparatus further includes a training module 1230;

[0187] The acquisition module 1210 is further configured to acquire the sample image;

[0188] The training module 1230 includes:

[0189] A first processing submodule 1231 is configured to perform definition enhancement processing on the sample image using the image processing model to obtain the first predicted image;

[0190] A second processing submodule 1232 is configured to perform blur enhancement processing on the sample image using the image processing model to obtain the second predicted image;

[0191] The training submodule 1233 is used to train the image processing model based on the sorting loss between the sample image, the first predicted image and the second predicted image to obtain a target image processing model. The sorting loss is used to indicate the difference between the sorting situation and the constrained sorting situation of the image clarity between the sample image, the first predicted image and the second predicted image.

[0192] In some optional embodiments, the training submodule 1233 further includes:

[0193] a determining unit 1234 configured to obtain a first evaluation value of the sample image, a second evaluation value of the first predicted image, and a third evaluation value of the second predicted image, wherein the first evaluation value is used to indicate the clarity of the sample image, the second evaluation value is used to indicate the clarity of the first predicted image, and the third evaluation value is used to indicate the clarity of the second predicted image;

[0194] a loss determining unit 1235 for determining a ranking loss value based on a difference between a comparison result among the first evaluation value, the second evaluation value, and the third evaluation value and a specified constraint condition, the specified constraint condition indicating that the second definition of the first predicted image is greater than the first definition of the sample image, and the first definition of the sample image is greater than the third definition of the second predicted image;

[0195] The training unit 1236 is used to train the image processing model based on the sorting loss value to obtain the target image processing model.

[0196] In some optional embodiments, the determining unit 1234 is further configured to determine first gradient information of the sample image in a first direction and second gradient information in a second direction, where the first direction and the second direction are perpendicular to each other;

[0197] The determining unit 1234 is further configured to determine the first evaluation value based on the first gradient information and the second gradient information;

[0198] The determining unit 1234 is further configured to determine third gradient information of the first predicted image in the first direction and fourth gradient information in the second direction;

[0199] The determining unit 1234 is further configured to determine the second evaluation value based on the third gradient information and the fourth gradient information;

[0200] The determining unit 1234 is further configured to determine fifth gradient information of the second predicted image in the first direction and sixth gradient information of the second direction;

[0201] The determining unit 1234 is further configured to determine the third evaluation value based on the fifth gradient information and the sixth gradient information.

[0202] In some optional embodiments, the training submodule 1233 further includes:

[0203] an extraction unit 1237 configured to perform feature extraction on the sample image, the first predicted image, and the second predicted image to obtain a first feature representation corresponding to the sample image, a second feature representation corresponding to the first predicted image, and a third feature representation corresponding to the second predicted image, respectively;

[0204] The prediction unit 1238 is used to input the first feature representation, the second feature representation and the third feature representation into the clarity evaluation subnetwork respectively, predict the distribution of the feature representation in the feature space, and output the first evaluation value of the sample image, the second evaluation value of the first predicted image and the third evaluation value of the second predicted image.

[0205] In some optional embodiments, the image processing model includes a first encoder, a first decoder, a second encoder and a second decoder, the first encoder and the first decoder are used to enhance the clarity of the sample image, and the second encoder and the second decoder are used to enhance the blurriness of the sample image;

[0206] The first processing submodule 1231 is further configured to input the sample image into the first encoder for encoding to obtain a first encoding feature;

[0207] The first processing submodule 1231 is further configured to input the first coding feature into the first decoder for decoding to obtain the first predicted image;

[0208] The second processing submodule 1232 is further configured to input the sample image into the second encoder for encoding to obtain a second encoding feature;

[0209] The second processing submodule 1232 is further configured to input the second coding feature into the second decoder for decoding to obtain the second predicted image.

[0210] In some optional embodiments, the image processing model includes a deblurring subnetwork and a reblurring subnetwork, the deblurring subnetwork is used for clarity enhancement processing, and the reblurring subnetwork is used for blurriness enhancement processing;

[0211] The training module 1230 further includes:

[0212] An extraction submodule 1239 is configured to perform semantic classification on the image semantics of the sample image in at least two semantic spaces, and obtain image semantic features corresponding to the semantic spaces respectively;

[0213] The extraction submodule 1239 is further configured to perform feature connection on the image semantic features to obtain sample image features;

[0214] The first processing submodule 1231 is further configured to input the sample image features into the deblurring subnetwork, and the deblurring subnetwork enhances the clarity of the sample image features based on the image semantics of the sample image to obtain the first predicted image;

[0215] The second processing submodule 1232 is further configured to input the sample image features into the re-blurring subnetwork, and the re-blurring subnetwork enhances the blurriness of the sample image features based on the image semantics of the sample image to obtain the second predicted image.

[0216] In some optional embodiments, the training objective of the image processing model to be trained includes deblurring an image with motion blur;

[0217] The acquisition module 1210 is further configured to acquire an image to be processed acquired by the vehicle-mounted device, wherein the image to be processed is an image acquired by the vehicle-mounted device of the surrounding environment, and the image to be processed includes a target area with motion blur;

[0218] The processing module 1220 is further configured to input the image to be processed into the target image processing model, and the target image processing model performs definition enhancement processing on the target area to obtain the target image.

[0219] In some optional embodiments, the training objective of the image processing model to be trained includes learning the shooting habits corresponding to the target account and applying the shooting habits to image clarity enhancement, wherein the shooting habits are used to indicate factors that cause image blur when the target account shoots images;

[0220] The acquisition module 1210 is further configured to receive historical images sent by a target account, where the historical images are collected by the target account through a terminal device within a historical period.

[0221] The acquisition module 1210 is further configured to use the historical captured image as the sample image.

[0222] In summary, the image deblurring device provided in the embodiment of the present application, in the process of training the image processing model for image clarity enhancement, performs clarity enhancement processing and blur enhancement processing on the sample image respectively, thereby obtaining a first predicted image and a second predicted image, and trains the image processing model through the clarity arrangement between the sample image, the first predicted image, and the second predicted image, thereby realizing unsupervised training of the model using contrastive learning, without the need to label the sample image, thereby improving the training efficiency of the model and avoiding the problem of the training effect being affected by the accuracy of the label, so that the target image processing model obtained by training is applied to the processing effect when the image clarity is enhanced.

[0223] It should be noted that the image deblurring device provided in the above embodiment is merely an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image deblurring device provided in the above embodiment and the image deblurring method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0224] Figure 14 The following is a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. Specifically, it includes the following structure.

[0225] Server 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including a random access memory (RAM) 1402 and a read-only memory (ROM) 1403, and a system bus 1405 connecting system memory 1404 and CPU 1401. Server 1400 also includes a mass storage device 1406 for storing an operating system 1413, application programs 1414, and other program modules 1415.

[0226] The mass storage device 1406 is connected to the central processing unit 1401 through a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1406 and its associated computer-readable media provide non-volatile storage for the server 1400. In other words, the mass storage device 1406 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0227] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1404 and mass storage device 1406 can be collectively referred to as memory.

[0228] According to various embodiments of the present application, the server 1400 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server 1400 may be connected to the network 1412 via the network interface unit 1411 connected to the system bus 1405, or the network interface unit 1411 may be used to connect to other types of networks or remote computer systems (not shown).

[0229] The memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0230] Embodiments of the present application further provide a computer device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the image deblurring methods provided in the above-described method embodiments. Optionally, the computer device may be a terminal or a server.

[0231] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the image deblurring method provided by the above-mentioned method embodiments.

[0232] Embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image deblurring method described in any of the above embodiments.

[0233] Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0234] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0235] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for deblurring an image, characterized in that: The method comprises: Acquire an image to be processed, where the image to be processed is an image to be deblurred; Inputting the image to be processed into the target deblurring subnetwork for clarity enhancement processing to obtain a target image with enhanced clarity; The target deblurring subnetwork is a subnetwork in the target image processing model. The target image processing model is obtained by training the image processing model to be trained using sample images. The image processing model to be trained includes a deblurring subnetwork and a reblurring subnetwork. The training process of the image processing model to be trained includes: acquiring the sample image; Performing a definition enhancement process on the sample image through the deblurring sub-network to obtain a first predicted image; and performing a blur enhancement process on the sample image through the reblurring sub-network to obtain a second predicted image; The image processing model is trained based on the sorting loss between the sample image, the first predicted image and the second predicted image to obtain the target image processing model, the sorting loss is used to indicate the difference between the sorting situation of image clarity between the sample image, the first predicted image and the second predicted image and the constrained sorting situation, the constrained sorting situation is used to indicate that the second clarity of the first predicted image is greater than the first clarity of the sample image, and the first clarity of the sample image is greater than the third clarity of the second predicted image.

2. The method according to claim 1, characterized in that The training of the image processing model to be trained based on the ranking loss between the sample image, the first predicted image, and the second predicted image to obtain the target image processing model includes: Obtaining a first evaluation value of the sample image, a second evaluation value of the first predicted image, and a third evaluation value of the second predicted image, wherein the first evaluation value is used to indicate the clarity of the sample image, the second evaluation value is used to indicate the clarity of the first predicted image, and the third evaluation value is used to indicate the clarity of the second predicted image; determining a sorting loss value based on a difference between a comparison result among the first evaluation value, the second evaluation value, and the third evaluation value and a specified constraint condition, the specified constraint condition indicating that a second definition of the first predicted image is greater than a first definition of the sample image, and the first definition of the sample image is greater than a third definition of the second predicted image; The image processing model is trained based on the ranking loss value to obtain the target image processing model.

3. The method according to claim 2, characterized in that The obtaining of the first evaluation value of the sample image, the second evaluation value of the first predicted image, and the third evaluation value of the second predicted image includes: determining first gradient information of the sample image in a first direction and second gradient information in a second direction, wherein the first direction and the second direction are perpendicular to each other; determining the first evaluation value based on the first gradient information and the second gradient information; determining third gradient information of the first predicted image in the first direction and fourth gradient information in the second direction; determining the second evaluation value based on the third gradient information and the fourth gradient information; determining fifth gradient information of the second predicted image in the first direction and sixth gradient information of the second direction; The third evaluation value is determined based on the fifth gradient information and the sixth gradient information.

4. The method according to claim 2, characterized in that The obtaining of the first evaluation value of the sample image, the second evaluation value of the first predicted image, and the third evaluation value of the second predicted image includes: Performing feature extraction on the sample image, the first predicted image, and the second predicted image to obtain a first feature representation corresponding to the sample image, a second feature representation corresponding to the first predicted image, and a third feature representation corresponding to the second predicted image, respectively; The first feature representation, the second feature representation and the third feature representation are respectively input into the clarity evaluation subnetwork, and the distribution of the feature representation in the feature space is predicted, and the first evaluation value of the sample image, the second evaluation value of the first predicted image and the third evaluation value of the second predicted image are output.

5. The method according to any one of claims 1 to 4, characterized in that: The image processing model includes a first encoder, a first decoder, a second encoder and a second decoder, wherein the first encoder and the first decoder are used to enhance the clarity of the sample image, and the second encoder and the second decoder are used to enhance the blurriness of the sample image; The performing definition enhancement processing on the sample image by the deblurring sub-network to obtain a first predicted image includes: Inputting the sample image into the first encoder for encoding to obtain a first encoding feature; Inputting the first encoding feature into the first decoder for decoding to obtain the first predicted image; The performing blur enhancement processing on the sample image by the re-blurring sub-network to obtain a second predicted image includes: Inputting the sample image into the second encoder for encoding to obtain a second encoding feature; The second encoded feature is input into the second decoder for decoding to obtain the second predicted image.

6. The method according to any one of claims 1 to 4, characterized in that: The image processing model includes a defuzzification subnetwork and a refuzzification subnetwork, wherein the defuzzification subnetwork is used for clarity enhancement processing and the refuzzification subnetwork is used for blur enhancement processing; After acquiring the sample image, the method further includes: Performing semantic classification on the image semantics of the sample image in at least two semantic spaces to obtain image semantic features corresponding to the semantic spaces respectively; Performing feature connection on the image semantic features to obtain sample image features; The performing definition enhancement processing on the sample image by the deblurring sub-network to obtain a first predicted image includes: Inputting the sample image features into the deblurring sub-network, and having the deblurring sub-network enhance the clarity of the sample image features based on the image semantics of the sample image to obtain the first predicted image; The performing blur enhancement processing on the sample image by the re-blurring sub-network to obtain a second predicted image includes: The sample image features are input into the re-blurring sub-network, and the re-blurring sub-network enhances the blurriness of the sample image features based on the image semantic guidance of the sample image to obtain the second predicted image.

7. The method according to any one of claims 1 to 4, characterized in that: The training objective of the image processing model to be trained includes deblurring an image with motion blur; The step of inputting the image to be processed into a target deblurring subnetwork for clarity enhancement processing to obtain a target image with enhanced clarity includes: Acquire an image to be processed captured by a vehicle-mounted device, wherein the image to be processed is an image acquired by the vehicle-mounted device from a surrounding environment, and the image to be processed includes a target area with motion blur; The image to be processed is input into the target deblurring subnetwork, and the target image processing model performs clarity enhancement processing on the target area to obtain the target image.

8. The method according to any one of claims 1 to 4, characterized in that: The training objectives of the image processing model to be trained include learning the shooting habits corresponding to the target account and applying the shooting habits to image clarity enhancement, wherein the shooting habits are used to indicate factors that cause image blur when the target account shoots images; The acquiring of the sample image comprises: Receiving historical captured images sent by a target account, where the historical captured images are collected by the target account through a terminal device within a historical period; The historical captured image is used as the sample image.

9. An image deblurring device, characterized in that: The device comprises: An acquisition module, configured to acquire an image to be processed, wherein the image to be processed is an image to be deblurred; a processing module, configured to input the image to be processed into a target deblurring subnetwork for clarity enhancement processing to obtain a target image with enhanced clarity; wherein the target deblurring subnetwork is a subnetwork in a target image processing model, the target image processing model being obtained by training an image processing model to be trained using sample images, the image processing model to be trained comprising a deblurring subnetwork and a reblurring subnetwork; The acquisition module is further configured to acquire the sample image; The apparatus further comprises a training module, wherein the training module comprises: A first processing submodule, configured to perform definition enhancement processing on the sample image through the deblurring subnetwork to obtain a first predicted image; A second processing submodule is configured to perform blur enhancement processing on the sample image through the re-blurring subnetwork to obtain a second predicted image; A training submodule is used to train the image processing model based on the sorting loss between the sample image, the first predicted image and the second predicted image to obtain the target image processing model, wherein the sorting loss is used to indicate the difference between the sorting situation and the constrained sorting situation of the image clarity between the sample image, the first predicted image and the second predicted image, and the constrained sorting situation is used to indicate that the second clarity of the first predicted image is greater than the first clarity of the sample image, and the first clarity of the sample image is greater than the third clarity of the second predicted image.

10. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the image deblurring method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one program code, and the program code is loaded and executed by a processor to implement the image deblurring method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the image deblurring method according to any one of claims 1 to 8.

Images