Image processing method and related device

Abstract

The present disclosure provides an image processing method and a related device. The method comprises: acquiring an image to be processed; performing defect repairing on said image to obtain a first image; performing target detection on the first image; and in response to a target object being detected in the first image, performing enhancement processing on the first image to obtain a second image.

Description

Image processing method and related equipment

[0001] This application claims priority to the Chinese invention patent application entitled “Image Processing Method and Related Equipment” and application number 202311523751.7, filed on November 15, 2023. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to an image processing method and related equipment. Background Art

[0003] Photos preserved using early photography technology usually exist in physical form. Due to their own materials and improper storage, such photos may have various problems after being stored for a long time, such as blurred portraits, damaged backgrounds, scratches, poor details, discoloration, etc., which affect the visual perception.

[0004] Summary of the Invention

[0005] The present disclosure proposes an image processing method and related devices to solve or partially solve the above problems.

[0006] In a first aspect of the present disclosure, an image processing method is provided, comprising: acquiring an image to be processed; performing defect repair on the image to be processed to obtain a first image; performing target detection on the first image; and in response to detecting a target object in the first image, performing enhancement processing on the first image to obtain a second image.

[0007] In a second aspect of the present disclosure, an image processing device is provided, comprising: an acquisition module configured to acquire an image to be processed; a repair module configured to perform defect repair on the image to be processed to obtain a first image; a detection module configured to perform target detection on the first image; and an enhancement module configured to, in response to detecting a target object in the first image, perform enhancement processing on the first image to obtain a second image.

[0008] In a third aspect of the present disclosure, a computer device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect.

[0009] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors are caused to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] FIG1 shows a schematic diagram of an exemplary system provided by an embodiment of the present disclosure.

[0012] FIG2A shows a schematic flow chart of an exemplary method provided by an embodiment of the present disclosure.

[0013] FIG2B shows a flowchart of an exemplary method for restoring a first image according to an embodiment of the present disclosure.

[0014] FIG2C is a schematic flow chart showing an exemplary method for restoring an intermediate image according to an embodiment of the present disclosure.

[0015] FIG2D shows a flowchart of an exemplary method of enhancement processing according to an embodiment of the present disclosure.

[0016] FIG2E is a flowchart illustrating an exemplary method for determining whether an image is a decolorized photograph according to an embodiment of the present disclosure.

[0017] FIG3A shows a schematic diagram of an exemplary image to be processed according to an embodiment of the present disclosure.

[0018] FIG3B shows a schematic diagram of an exemplary scratch mask according to an embodiment of the present disclosure.

[0019] FIG3C shows a schematic diagram of an exemplary intermediate image according to an embodiment of the present disclosure.

[0020] FIG3D shows a schematic diagram of an exemplary first image according to an embodiment of the present disclosure.

[0021] FIG4 shows a schematic diagram of an exemplary model architecture provided by an embodiment of the present disclosure.

[0022] FIG5 shows a schematic diagram of the hardware structure of an exemplary computer device provided by an embodiment of the present disclosure.

[0023] FIG6 shows a schematic diagram of an exemplary device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0025] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0026] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0027] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0028] As an optional but non-limiting implementation, in response to a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0029] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0030] FIG1 shows a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.

[0031] As shown in FIG1 , system 100 may include a terminal device 102, a terminal device 104, a server 106, and a database server 108. A medium (e.g., a network) providing a communication link may be provided between the terminal device 102, the terminal device 104, the server 106, and the database server 108. The network may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0032] Various applications (APPs) can be installed on the terminal device 104, such as image processing applications, video conferencing applications, reading applications, video applications, social applications, payment applications, web browsers and instant messaging tools, etc. These applications can all be used to display the delivered information.

[0033] The terminal devices 102 and 104 herein can be either hardware or software. When the terminal devices 102 and 104 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, e-book readers, MP3 players, laptop computers (Laptops), and desktop computers (PCs), etc. When the terminal devices 102 and 104 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitations are given here.

[0034] Server 106 may be a server that provides various services, such as a backend server that supports various applications displayed on terminal devices 102 and 104. Database server 108 may also be a database server that provides various services. It is understood that if server 106 can implement the relevant functions of database server 108, database server 108 may not be provided in system 100.

[0035] The server 106 and database server 108 herein can also be hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or software module. No specific limitations are imposed herein.

[0036] It should be noted that the image processing method provided in the embodiments of the present disclosure can be executed by terminal device 102 and / or terminal device 104. It should be understood that the number of terminal devices, users, servers, and database servers in FIG1 is merely illustrative. Any number of terminal devices, users, servers, and database servers may be provided as required.

[0037] In one embodiment, the terminal device 104 may be installed with an image processing application or software, and the user 112 may use the application or software to process the image, for example, to enlarge the image and display the enlarged image.

[0038] In some embodiments, the system 100 can be used to train a machine learning model for image processing. For example, the user 110 can deploy a machine learning model to be trained on the server 106 and use the terminal device 102 to design training samples, and then use the training samples to train the machine learning model. During the training process, the terminal device 102 can be used to adjust the parameters of the machine learning model until the machine learning model is trained.

[0039] Optionally, a pre-trained machine learning model can be deployed on the server 106. In this case, when processing an image, the terminal device 104 can upload the image to the server 106 for processing, and the server 106 returns the processed image to the terminal device 104 for display. In some cases, if the pre-trained machine learning model can be lightweight, it can be deployed on the terminal device 104. When processing an image, the terminal device 104 can call the locally stored machine learning model to perform image processing without going through the server 106.

[0040] As mentioned earlier, due to their age, many old photos (or vintage photos) have problems such as blurred portraits, damaged backgrounds, scratches, poor details, and discoloration, which affect the visual experience.

[0041] Therefore, how to restore these old photos (or old photos) through image processing technology to restore their old glory is of great significance for preserving and reproducing memories.

[0042] In view of this, an embodiment of the present disclosure provides an image processing method, which can achieve better image restoration effects and improve user experience by repairing defects in the image to be processed and then enhancing the image when the target object is detected.

[0043] FIG2A is a flow chart illustrating an exemplary method 200 provided in an embodiment of the present disclosure. Method 200 can be used to process an image. Optionally, method 200 can be implemented independently by terminal devices 102 and 104 of FIG1 , or by system 100 of FIG1 . The following description uses terminal device 104 as an example to illustrate method 200.

[0044] As shown in FIG. 2A , the method 200 may further include the following steps.

[0045] In step 202, an image to be processed is obtained.

[0046] FIG3A shows a schematic diagram of an exemplary image 300 to be processed according to an embodiment of the present disclosure.

[0047] As shown in FIG3A , the image 300 may be an image to be processed. The user 112 may obtain the image 300 by taking an old photo or a used photo using the camera of the terminal device 104 , or may receive the image 300 from another device. The specific acquisition method is not limited.

[0048] In step 204, defects of the image to be processed are repaired to obtain a first image.

[0049] As shown in FIG3A , image 300 includes relatively large point-shaped scratches 302A, as well as long strip-shaped scratches 302B and 302C, and relatively small point-shaped noise (small dots scattered throughout image 300 in FIG3A ). Optionally, these scratches and noise can be repaired in this step. It is understood that the scratches are not limited to those caused by scratches on the photo, but can also include creases formed by folding, spots caused by dirt, and so on. As an optional embodiment, marks larger than a preset area can be determined as scratches, while marks smaller than the preset area can be determined as noise.

[0050] In some embodiments, as shown in FIG2B , step 204 of performing defect repair on the image to be processed to obtain the first image may further include the following steps.

[0051] In step 2042, scratch repair is performed on the image to be processed to obtain an intermediate image.

[0052] Since the scratches are usually large in area, in this step, the scratches can be repaired first.

[0053] In some embodiments, as shown in FIG. 2C , step 2042 of performing scratch repair on the image to be processed to obtain an intermediate image may further include the following steps.

[0054] In step 20422, scratch detection is performed on the image to be processed to obtain a scratch mask.

[0055] Alternatively, the scratch mask may be a mask used to block areas outside the scratched area of ​​the image to be processed, thereby exposing the scratched area. As shown in FIG3B , the hollowed-out area in the center of the scratch mask 310 corresponds to the scratched area, while the remaining areas serve as a mask to block non-scratched areas, thereby ensuring that the image quality in other areas is not affected when the scratch is repaired.

[0056] Optionally, a scratch detection model may be used to perform scratch detection on the image to be processed to obtain the scratch mask.

[0057] FIG4 shows a schematic diagram of a model architecture 400 provided by an embodiment of the present disclosure.

[0058] As shown in FIG4 , in some embodiments, the method 200 may be implemented using a model architecture 400, wherein the model architecture 400 may include a scratch processing model 402, which may further include a scratch detection model 4022 and a scratch repair model 4024. In this step, the image to be processed 300 is input into the scratch detection model 4022, and a scratch mask 310 may be output.

[0059] In step 20424, the scratched area in the image to be processed is repaired based on the scratch mask.

[0060] As described above, after obtaining the scratch mask 310 , the scratch mask 310 is used to mask the non-scratch area in the image to be processed 300 , and then the scratch area can be repaired, thereby ensuring the repair effect while avoiding affecting the image effect of the non-scratch area.

[0061] Optionally, as shown in FIG4 , a scratch repair model 4024 can be used to repair the scratched area in the image to be processed 300 based on the scratch mask 310. For example, the scratch mask 310 and the image to be processed 300 are input into the scratch repair model 4024, thereby outputting an intermediate image 320 after the scratches have been repaired, as shown in FIG3C .

[0062] In some embodiments, the scratch detection model 4022 and the scratch repair model 4024 may be obtained through pre-training using the first sample set.

[0063] The first sample set may further include multiple first samples and multiple second samples, wherein the first samples include images without scratches (e.g., images with higher definition), and the second samples include images with scratches. Thus, the scratch detection model 4022 and the scratch repair model 4024 are trained using both images with scratches and images without scratches, allowing the models to learn features of both images with scratches and images without scratches.

[0064] As an optional embodiment, the second sample may further include an image obtained by superimposing scratches on the first sample. In this way, if the number of images with scratches is insufficient, the second sample can be generated by adding scratches to an image without scratches, thereby expanding the second sample and facilitating better model training.

[0065] Optionally, the scratches superimposed in the second sample include scratches generated by a random walk method, so that the generated scratches are random and can better reflect the characteristics of naturally generated scratches.

[0066] As an optional embodiment, when generating scratches using a random walk approach, constraints can be added, such as limiting the scratch's horizontal coordinate difference range or vertical coordinate difference range. That is, the difference between the minimum horizontal coordinate point and the maximum horizontal coordinate point in the generated scratch must be within the horizontal coordinate difference range, or the difference between the minimum vertical coordinate point and the maximum vertical coordinate point in the generated scratch must be within the vertical coordinate difference range. In this way, the generated scratch can maintain a long strip shape when extending horizontally or vertically, which is closer to the scratch shape, such as an irregular strip scratch (e.g., scratches 302B and 302C in FIG. 3A ).

[0067] For another example, considering that old photos may have block-shaped marks due to dirt, the constraint can be to limit the area of ​​the scratches, for example, setting a minimum area value and a maximum area value, so that some block-shaped scratches can be generated (for example, scratch 302A in Figure 3A).

[0068] In some embodiments, the scratches superimposed in the second sample have a target color selected from the color of at least one scratch sample in the scratch sample set. It is understood that scratches in old photos are not necessarily all white, and scratches may appear in different colors due to different causes. Therefore, a scratch sample set can be pre-designed to include scratch samples of various colors. When superimposing scratches, in addition to superimposing scratches of corresponding shapes, a color (e.g., RGB value) can be randomly selected from the scratch sample set and added to the scratch, making the produced second sample more realistic.

[0069] As an optional embodiment, an image fusion algorithm may be used to add the generated scratches to the image, so that the added scratches can be better integrated with the original image and can better reflect the real scratch effect.

[0070] In some embodiments, the first sample set further includes a plurality of third samples, each of which includes a scratch mask generated based on the scratches superimposed on the second sample. For example, the scratch mask can be generated by binarizing the image after the scratches are superimposed, and used to train the scratch detection model 4022.

[0071] In step 2044, noise reduction processing is performed on the intermediate image to obtain the first image.

[0072] Optionally, as shown in FIG4 , the noise processing model 404 may be used to perform noise reduction processing on the intermediate image 320 to obtain a first image 330 , as shown in FIG3D .

[0073] Comparing FIG3C and FIG3D , it can be seen that noise processing model 404 is used to repair noise, thereby removing point noise in the image and further improving image clarity. As an optional embodiment, the noise processing model 404 can be obtained by pre-training using images containing noise and images without noise, which will not be described in detail here.

[0074] It can be understood that the aforementioned defect repair embodiments are merely exemplary. In some cases, other defect repairs may be performed on the first image, and these defect repair methods also fall within the scope of protection of the present disclosure.

[0075] As an optional embodiment, after defect repair is complete, a first image 330 can be output as the image processing result. Because first image 330 repairs defects such as scratches and noise, image 300 to be processed becomes a clear first image 330, effectively restoring the original image and improving the user experience.

[0076] Considering that the image after defect repair may still have the problem of image blur, in some embodiments, the first image 330 may be further processed.

[0077] Then, in step 206 , object detection may be performed on the first image 330 .

[0078] In this step, target detection can be performed on the first image 330 using a target detection algorithm or a target detection model to identify whether the first image 330 includes a target object. The target object can be a variety of objects, such as animals, plants, etc. The specific type of object to be identified may vary depending on the training sample, resulting in different recognition capabilities of the target detection model, which will not be further described here.

[0079] As shown in FIG3D , for example, first image 330 may include target object 332 and portion 334 outside target object 332. Target object 332 may further include face 332A and torso 332B. Optionally, the identified target object may be only face 332A or torso 332B. The specific object to be identified may vary depending on the training sample, resulting in different recognition capabilities of the target detection model.

[0080] In step 208 , in response to detecting the target object in the first image, the first image is enhanced to obtain a second image.

[0081] In this step, the image clarity can be improved by enhancing the image, thereby further improving the user experience.

[0082] In some embodiments, as shown in FIG2D , step 208 of performing enhancement processing on the first image may further include the following steps.

[0083] As shown in FIG. 4 , at step 2082 , a first enhancement process 4062 is performed on the target object 332 in the first image 330 .

[0084] In step 2084, a second enhancement process 4064 is performed on the portion 334 of the first image 330 excluding the target object 332. Optionally, the second enhancement process 4064 may be a background enhancement process, which mainly enhances details (richer details, clearer textures) while also taking into account noise suppression.

[0085] In this way, by performing different enhancement processing on the target object and the parts other than the target object, the difference between the target object and other parts can be highlighted, thereby improving the image effect.

[0086] Optionally, the degree of enhancement of the first enhancement process 4062 is greater than that of the second enhancement process 4064. For example, if the enhancement process is to improve clarity, the clarity of the target object 332 after the first enhancement process 4062 will be higher than the clarity of the other parts 334 after the second enhancement process 4064. In this way, after the enhancement process, the clarity of the target object 332 is higher than that of the other parts 334, thereby improving image quality and user experience.

[0087] In some embodiments, performing enhancement processing on the first image includes: performing the first enhancement processing 4062 and the second enhancement processing 4064 on the first image using the image enhancement model 406, as shown in FIG4 .

[0088] The image enhancement model 406 is obtained by pre-training using the second sample set. The second sample set includes a plurality of fourth samples and a plurality of fifth samples, the fourth samples include images with a clarity higher than a preset threshold (set according to actual conditions), and the fifth samples include images after the fourth samples are blurred and noised. In this way, the image enhancement model 406 is trained together with the blurred and noised images and the images with higher clarity, so that the model can learn the features of blurred images and noisy images, as well as the features of clear images. Optionally, the blurring process includes but is not limited to Gaussian blur, box blur, and double blur, and the noise processing includes but is not limited to Gaussian noise and salt and pepper noise.

[0089] In some embodiments, the fifth sample is obtained by performing a first blurring process and a first noise addition process on the target object in the fourth sample, and performing a second blurring process and a second noise addition process on the portion of the fourth sample excluding the target object. The blur level of the first blurring process is higher than the blur level of the second blurring process, and the noise level of the first noise addition process is higher than the noise level of the second noise addition process. The image enhancement model 406 trained with such samples, when inputted with the first image 330, can naturally output a second image 340 in which the target object 332 has been subjected to the first enhancement process 4062 and the remaining portion 334 has been subjected to the second enhancement process 4064.

[0090] As an optional embodiment, after the image is enhanced, a second image 340 can be output as the image processing result. Because second image 340 repairs defects such as scratches and noise and further enhances the image, processed image 300 becomes a clearer and more focused second image 340, effectively restoring the original image and improving the user experience.

[0091] Considering that old photos may be discolored or black and white due to their age, in some embodiments, the second image 340 may be further processed.

[0092] Therefore, in some embodiments, as shown in FIG2A , the method 200 may further include the following steps: Alternatively, in other embodiments, if the target object cannot be detected in the first image, the method may jump to the following steps to continue execution.

[0093] At step 210 , it is determined whether the second image is a decolorized photograph.

[0094] In this step, a decolorization detection is performed on the second image 340 to determine whether it is a decolorized photo, and different color processing is performed when it is a decolorized photo or not, thereby improving the image effect.

[0095] In some embodiments, as shown in FIG. 2E , step 210 of determining whether the second image is a decolorized photograph may further include the following steps.

[0096] In step 2102 , the mean and variance of the chromaticity values ​​of the second image in a target color space (eg, CIELUV color space) are calculated (eg, the mean and variance of the UV channel).

[0097] It can be understood that decolorized photos can be grayish-white photos, black-and-white photos, yellowed / old photos. Such photos usually show the problem of insufficient color saturation in the image. In this embodiment, based on the mean and variance of the chromaticity values, various types of decolorized photos can be better screened out for subsequent colorization processing, thereby improving the image restoration effect.

[0098] In step 2104, in response to the average value of the chromaticity values ​​being less than or equal to a preset average value or the variance of the chromaticity values ​​being less than or equal to a preset variance value, it is determined that the second image is a decolorized photograph.

[0099] In step 2106 , in response to the average value of the chromaticity values ​​being greater than a preset average value and the variance of the chromaticity values ​​being greater than a preset variance value, it is determined that the second image is a color photo.

[0100] Since the chromaticity value reflects the color of the image, in this embodiment, the images are divided into two categories, namely, decolorized photos and color photos, by statistically calculating the average value and variance of the chromaticity value and performing judgment based on the threshold.

[0101] In step 212, in response to the second image 340 being a decolorized photograph, a colorization process 4082 is performed on the second image 340 to obtain a third image 350. In this way, when the second image 340 is a black and white image, the colorization process converts it into a colored third image 350, resulting in a better image effect.

[0102] In step 214, in response to the second image 340 being a color photograph, color enhancement processing is performed on the second image 340 to obtain a fourth image 360. Because old photographs may have discoloration due to their age, color enhancement processing can, to some extent, eliminate the sense of age of the old photographs and improve the image quality.

[0103] In some embodiments, as shown in FIG. 4 , the model architecture 400 may further include a color processing model 408 , which may intelligently identify whether the second image 340 is a decolorized photo and further select colorization processing or color enhancement processing to obtain the third image 350 or the fourth image 360 ​​.

[0104] Optionally, the color processing model 408 can be pre-trained using color photos and decolorized photos after the color photos are decolorized, so that when the second image 340 is input into the pre-trained color processing model 408, the image after colorization or color enhancement processing can be directly output.

[0105] It is understood that step 210 and step 208 may not necessarily be performed sequentially, or step 208 may not necessarily occur before step 210. In some embodiments, the target object may not be detected in the first image, and therefore, enhancement processing may not be performed, and color processing may be performed directly. Alternatively, in other embodiments, although the target object may not be detected in the first image, unified enhancement processing may be performed on the first image, for example, performing a second enhancement processing on the first image and then performing color processing.

[0106] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0107] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0108] The present disclosure also provides a computer device for implementing the aforementioned method 200. FIG5 illustrates a schematic diagram of the hardware structure of an exemplary computer device 500 provided in the present disclosure. Computer device 500 can be used to implement server 106 in FIG1 , or terminal devices 102 and 104 in FIG1 . In some scenarios, computer device 500 can also be used to implement database server 108 in FIG1 .

[0109] 5 , a computer device 500 may include a processor 502, a memory 504, a network module 506, a peripheral interface 508, and a bus 510. The processor 502, the memory 504, the network module 506, and the peripheral interface 508 are connected to each other in communication within the computer device 500 via the bus 510.

[0110] The processor 502 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device (PLD), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or one or more integrated circuits. The processor 502 may be used to perform functions related to the technology described in this disclosure. In some embodiments, the processor 502 may also include multiple processors integrated into a single logical component. For example, as shown in FIG5 , the processor 502 may include multiple processors 502 a, 502 b, and 502 c.

[0111] The memory 504 can be configured to store data (e.g., instructions, computer codes, etc.). As shown in Figure 5, the data stored in the memory 504 can include program instructions (e.g., program instructions for implementing the method 200 of the embodiment of the present disclosure) and data to be processed (e.g., the memory can store configuration files of other modules, etc.). The processor 502 can also access the program instructions and data stored in the memory 504, and execute the program instructions to operate on the data to be processed. The memory 504 can include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 504 can include a random access memory (RAM), a read-only memory (ROM), an optical disc, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0112] The network interface 506 can be configured to provide the computer device 500 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It will be understood that the type of network is not limited to the specific examples above.

[0113] The peripheral interface 508 can be configured to connect the computer device 500 to one or more peripheral devices to enable information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, and various sensors, as well as output devices such as a display, a speaker, a vibrator, and an indicator light.

[0114] The bus 510 may be configured to transmit information between various components of the computer device 500 (e.g., the processor 502, the memory 504, the network interface 506, and the peripheral interface 508), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0115] It should be noted that although the architecture of the computer device 500 shown above only shows the processor 502, memory 504, network interface 506, peripheral interface 508, and bus 510, in a specific implementation, the architecture of the computer device 500 may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the architecture of the computer device 500 may also include only the components necessary to implement the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0116] The present disclosure also provides an image processing apparatus. FIG6 shows a schematic diagram of an exemplary apparatus 600 provided by the present disclosure. As shown in FIG6 , the apparatus 600 can be used to implement the method 200 and can further include the following modules.

[0117] The acquisition module 602 is configured to: acquire an image to be processed.

[0118] The repair module 604 is configured to perform defect repair on the image to be processed to obtain a first image.

[0119] The detection module 606 is configured to perform object detection on the first image.

[0120] The enhancement module 608 is configured to: in response to detecting the target object in the first image, perform enhancement processing on the first image to obtain a second image.

[0121] In some embodiments, the repair module 604 is configured to: perform scratch repair on the image to be processed to obtain an intermediate image; and perform noise reduction on the intermediate image to obtain the first image.

[0122] In some embodiments, the repair module 604 is configured to: perform scratch detection on the image to be processed to obtain a scratch mask; and repair the scratched area in the image to be processed based on the scratch mask.

[0123] In some embodiments, the repair module 604 is configured to: perform scratch detection on the image to be processed using a scratch detection model to obtain a scratch mask; and repair the scratch area in the image to be processed based on the scratch mask using a scratch repair model.

[0124] In some embodiments, the scratch detection model and the scratch repair model are obtained through pre-training using a first sample set; wherein the first sample set includes multiple first samples and multiple second samples, the first samples include images without scratches, the second samples include images with scratches superimposed on the first samples, and the scratches superimposed in the second samples include scratches generated using a random walk method.

[0125] In some embodiments, the superimposed scratch in the second sample has a target color selected from the color of at least one scratch sample in the set of scratch samples.

[0126] In some embodiments, the first sample set further includes a plurality of third samples, the third samples including scratch masks generated based on the scratches superimposed in the second samples.

[0127] In some embodiments, the enhancement module 608 is configured to: perform a first enhancement process on the target object in the first image; and perform a second enhancement process on the portion of the first image other than the target object.

[0128] In some embodiments, the enhancement module 608 is configured to: perform the first enhancement processing and the second enhancement processing on the first image using an image enhancement model; wherein the image enhancement model is obtained by pre-training using a second sample set, the second sample set includes multiple fourth samples and multiple fifth samples, the fourth samples include images with a clarity higher than a preset threshold, and the fifth samples include images after the fourth samples are blurred and denoised.

[0129] In some embodiments, the fifth sample is obtained by performing a first blurring process and a first noise adding process on the target object in the fourth sample and performing a second blurring process and a second noise adding process on the portion of the fourth sample other than the target object. The blurring degree of the first blurring process is higher than the blurring degree of the second blurring process, and the noise degree of the first noise adding process is higher than the noise degree of the second noise adding process.

[0130] In some embodiments, the device 600 also includes a color processing module, which is configured to: determine whether the second image is a decolorized photo; in response to the second image being a decolorized photo, perform color processing on the second image to obtain a third image; in response to the second image being a color photo, perform color enhancement processing on the second image to obtain a fourth image.

[0131] In some embodiments, the color processing module is configured to: calculate the average value and variance of the chromaticity values ​​of the second image in the target color space; determine that the second image is a decolorized photo in response to the average value of the chromaticity values ​​being less than or equal to a preset average value or the variance of the chromaticity values ​​being less than or equal to a preset variance value; and determine that the second image is a color photo in response to the average value of the chromaticity values ​​being greater than a preset average value and the variance of the chromaticity values ​​being greater than a preset variance value.

[0132] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0133] The apparatus of the above embodiment is used to implement the corresponding method 200 in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0134] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute method 200 as described in any of the above embodiments.

[0135] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0136] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method 200 described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0137] Based on the same inventive concept, corresponding to any of the above-described embodiments of method 200, the present disclosure further provides a computer program product comprising a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to perform method 200. For each step in each embodiment of method 200, the processor that executes the corresponding step may be a member of the corresponding execution entity.

[0138] The computer program product of the above embodiment is used to enable a processor to execute the method 200 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0139] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0140] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0141] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0142] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An image processing method, comprising: Get the image to be processed; Performing defect repair on the image to be processed to obtain a first image; performing object detection on the first image; In response to detecting the target object in the first image, the first image is enhanced to obtain a second image.

2. The method of claim 1, wherein: Performing defect repair on the image to be processed to obtain a first image includes: Performing scratch repair on the image to be processed to obtain an intermediate image; Perform noise reduction processing on the intermediate image to obtain the first image.

3. The method of claim 2, wherein: Repairing scratches on the image to be processed to obtain an intermediate image includes: Performing scratch detection on the image to be processed to obtain a scratch mask; Based on the scratch mask, the scratch area in the image to be processed is repaired.

4. The method of claim 3, wherein: Performing scratch detection on the image to be processed to obtain a scratch mask, including: performing scratch detection on the image to be processed using a scratch detection model to obtain a scratch mask; Repairing the scratched area in the image to be processed based on the scratch mask includes: using a scratch repair model to repair the scratched area in the image to be processed based on the scratch mask.

5. The method of claim 3, wherein: The scratch detection model and the scratch repair model are obtained by pre-training using a first sample set; wherein the first sample set includes multiple first samples and multiple second samples, the first sample includes an image without scratches, the second sample includes an image after scratches are superimposed on the first sample, and the scratches superimposed in the second sample include scratches generated by a random walk method.

6. The method of claim 5, wherein: The superimposed scratches in the second sample have a target color, and the target color is selected from the color of at least one scratch sample in the scratch sample set.

7. The method of claim 5, wherein: The first sample set also includes a plurality of third samples, the third samples including scratch masks generated based on the scratches superimposed in the second samples.

8. The method of claim 1, wherein: Performing enhancement processing on the first image includes: Performing a first enhancement process on the target object in the first image; A second enhancement process is performed on a portion of the first image except the target object.

9. The method of claim 8, wherein: Performing enhancement processing on the first image, comprising: performing the first enhancement processing and the second enhancement processing on the first image using an image enhancement model; The image enhancement model is obtained by pre-training using a second sample set, wherein the second sample set includes The method comprises a plurality of fourth samples and a plurality of fifth samples, wherein the fourth samples comprise images whose clarity is higher than a preset threshold, and the fifth samples comprise images after blurring and noise adding processing are performed on the fourth samples.

10. The method of claim 9, wherein: The fifth sample is obtained by performing a first blurring process and a first noise adding process on the target object in the fourth sample and performing a second blurring process and a second noise adding process on a part of the fourth sample other than the target object, wherein the blurring degree of the first blurring process is higher than the blurring degree of the second blurring process, and the noise degree of the first noise adding process is higher than the noise degree of the second noise adding process.

11. The method of claim 1, wherein: The method further comprises: determining whether the second image is a decolorized photograph; In response to the second image being a decolorized photograph, coloring the second image to obtain a third image; In response to the second image being a color photo, color enhancement processing is performed on the second image to obtain a fourth image.

12. The method of claim 11, wherein: Determining whether the second image is a decolorized photograph comprises: Calculating the mean and variance of the chromaticity values ​​of the second image in the target color space; In response to the average value of the chromaticity values ​​being less than or equal to a preset average value or the variance of the chromaticity values ​​being less than or equal to a preset variance value, determining that the second image is a decolorized photograph; In response to the average value of the chromaticity values ​​being greater than a preset average value and the variance of the chromaticity values ​​being greater than a preset variance value, it is determined that the second image is a color photo.

13. An image processing device, comprising: The acquisition module is configured to: acquire the image to be processed; A repair module is configured to: repair defects of the image to be processed to obtain a first image; A detection module is configured to: perform target detection on the first image; The enhancement module is configured to: in response to detecting a target object in the first image, perform enhancement processing on the first image to obtain a second image.

14. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1-12.

15. A non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to perform the method according to any one of claims 1 to 12.

16. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1-12.

Images