Image processing method and related equipment

Abstract

The invention provides an image processing method and related equipment. The method comprises the following steps: acquiring a to-be-processed image; performing defect repair on the to-be-processed image to obtain a first image; performing target detection on the first image; and in response to a target object detected in the first image, performing enhancement processing on the first image to obtain a second image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method and related equipment. Background Art

[0002] Photos preserved using early photography technology usually exist in physical form. Due to their own materials and improper preservation, such photos may have various problems after being preserved for a long time, such as blurred portraits, damaged backgrounds, scratches, poor details, discoloration, etc., which affect the visual perception. Summary of the invention

[0003] The present disclosure proposes an image processing method and related devices to solve or partially solve the above problems.

[0004] In a first aspect, the present disclosure provides an image processing method, comprising:

[0005] Get the image to be processed;

[0006] Performing defect repair on the image to be processed to obtain a first image;

[0007] performing object detection on the first image;

[0008] In response to detecting the target object in the first image, the first image is enhanced to obtain a second image.

[0009] In a second aspect of the present disclosure, there is provided an image processing device, comprising:

[0010] The acquisition module is configured to: acquire the image to be processed;

[0011] A repair module is configured to: repair defects of the image to be processed to obtain a first image;

[0012] A detection module is configured to: perform target detection on the first image;

[0013] The enhancement module is configured to: in response to detecting a target object in the first image, perform enhancement processing on the first image to obtain a second image.

[0014] In a third aspect of the present disclosure, a computer device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method described in the first aspect.

[0015] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0016] The image processing method and related devices provided by the embodiments of the present disclosure can achieve better image restoration effects and improve user experience by repairing defects in the image to be processed and then enhancing the image when a target object is detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram showing an exemplary system provided by an embodiment of the present disclosure is shown.

[0019] Figure 2A A flowchart of an exemplary method provided by an embodiment of the present disclosure is shown.

[0020] Figure 2B A schematic flow chart of an exemplary method for restoring a first image according to an embodiment of the present disclosure is shown.

[0021] Figure 2C A schematic flow chart of an exemplary method for repairing an intermediate image according to an embodiment of the present disclosure is shown.

[0022] Figure 2D A flowchart of an exemplary method for enhancement processing according to an embodiment of the present disclosure is shown.

[0023] Figure 2E A flowchart of an exemplary method for determining whether an image is a decolorized photograph according to an embodiment of the present disclosure is shown.

[0024] Figure 3A A schematic diagram of an exemplary image to be processed according to an embodiment of the present disclosure is shown.

[0025] Figure 3B A schematic diagram of an exemplary scratch mask according to an embodiment of the present disclosure is shown.

[0026] Figure 3C A schematic diagram of an exemplary intermediate image according to an embodiment of the present disclosure is shown.

[0027] Figure 3DA schematic diagram of an exemplary first image according to an embodiment of the present disclosure is shown.

[0028] Figure 4 A schematic diagram of an exemplary model architecture provided by an embodiment of the present disclosure is shown.

[0029] Figure 5 A schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of the present disclosure is shown.

[0030] Figure 6 A schematic diagram of an exemplary device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0032] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0033] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0034] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0035] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0036] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0037] Figure 1 A schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure is shown.

[0038] like Figure 1 As shown, the system 100 may include a terminal device 102, a terminal device 104, a server 106, and a database server 108. A medium (e.g., a network) providing a communication link may be included between the terminal device 102, the terminal device 104 and the server 106 and the database server 108. The network may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0039] Various applications (APPs) may be installed on the terminal device 104, such as image processing applications, video conferencing applications, reading applications, video applications, social applications, payment applications, web browsers, and instant messaging tools, etc. These applications may be used to display the delivered information.

[0040] The terminal device 102 and the terminal device 104 here can be hardware or software. When the terminal device 102 and the terminal device 104 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, MP3 players, laptop computers (Laptops) and desktop computers (PCs), etc. When the terminal device 102 and the terminal device 104 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0041] The server 106 may be a server that provides various services, such as a background server that provides support for various applications displayed on the terminal devices 102 and 104. The database server 108 may also be a database server that provides various services. It is understood that if the server 106 can implement the relevant functions of the database server 108, the database server 108 may not be set in the system 100.

[0042] Here, the server 106 and the database server 108 can also be hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

[0043] It should be noted that the image processing method provided in the embodiment of the present disclosure may be executed by the terminal device 102 and / or the terminal device 104. It should be understood that Figure 1 The number of terminal devices, users, servers and database servers in the embodiment is only for illustration. Any number of terminal devices, users, servers and database servers may be provided as required.

[0044] In one embodiment, an image processing application or software may be installed in the terminal device 104, and the user 112 may use the application or software to process the image, for example, to enlarge the image and display the enlarged image.

[0045] In some embodiments, the system 100 may be used to train a machine learning model for processing an image. For example, the user 110 may deploy a machine learning model to be trained in the server 106 and may use the terminal device 102 to design training samples, and then use the training samples to train the machine learning model, and during the training process, may use the terminal device 102 to adjust the parameters of the machine learning model until the machine learning model is trained.

[0046] Optionally, the pre-trained machine learning model can be deployed in the server 106. In this case, when the terminal device 104 processes an image, it can upload the image to the server 106 for processing, and the server 106 returns the processed image to the terminal device 104 for display. In some cases, if the pre-trained machine learning model can be lightweight, it can be deployed in the terminal device 104. When processing an image, the terminal device 104 can call the machine learning model stored locally to process the image without processing through the server 106.

[0047] As mentioned earlier, due to their age, many old photos (or old photos) have problems such as blurred portraits, damaged backgrounds, scratches, poor details, and discoloration, which affect the visual perception.

[0048] Therefore, how to restore these old photos (or old photos) through image processing technology to restore their old glory is of great significance for preserving and reproducing memories.

[0049] In view of this, an embodiment of the present disclosure provides an image processing method, which can achieve better image restoration effects and improve user experience by repairing defects in an image to be processed and then enhancing the image when a target object is detected.

[0050] Figure 2A FIG. 2 is a flow chart of an exemplary method 200 provided in an embodiment of the present disclosure. The method 200 may be used to process an image. Optionally, the method 200 may be performed by Figure 1 The terminal devices 102 and 104 are implemented separately, or can be implemented by Figure 1 The method 200 is implemented by the system 100. The following describes the method 200 by using the terminal device 104.

[0051] like Figure 2A As shown, the method 200 may further include the following steps.

[0052] In step 202, an image to be processed is obtained.

[0053] Figure 3A A schematic diagram of an exemplary image 300 to be processed according to an embodiment of the present disclosure is shown.

[0054] like Figure 3A As shown, the image 300 may be an image to be processed. The user 112 may obtain the image 300 by taking an old photo or old photo with the camera of the terminal device 104, or may receive the image 300 from other devices. The specific acquisition method is not limited.

[0055] In step 204, defects of the image to be processed are repaired to obtain a first image.

[0056] like Figure 3A As shown, the image 300 includes a relatively large dot-shaped scratch 302A, a long strip-shaped scratch 302B, 302C, and a relatively small dot-shaped noise ( Figure 3A 300). Optionally, in this step, the scratches and noise can be repaired. It can be understood that the scratches are not limited to the marks caused by scratches on the photo, but can also include creases formed by folding, spots caused by dirt, etc. As an optional embodiment, the marks with an area larger than the preset area can be determined as scratches, and the marks with an area smaller than the preset area can be determined as noise.

[0057] In some embodiments, Figure 2B As shown, step 204 of performing defect repair on the image to be processed to obtain the first image may further include the following steps.

[0058] In step 2042, scratch repair is performed on the image to be processed to obtain an intermediate image.

[0059] Since the scratches are usually large in area, in this step, the scratches may be repaired first.

[0060] In some embodiments, Figure 2C As shown, step 2042 of repairing scratches on the image to be processed to obtain an intermediate image may further include the following steps.

[0061] In step 20422, scratch detection is performed on the image to be processed to obtain a scratch mask.

[0062] Optionally, the scratch mask may refer to a mask used to cover the area outside the scratch area of ​​the image to be processed to expose the scratch area. Figure 3B As shown, the hollow area in the middle of the scratch mask 310 corresponds to the scratch area, and the other areas are used as a mask to shield the non-scratched area, so that the image effect of other areas will not be affected when the scratch is repaired.

[0063] Optionally, a scratch detection model may be used to perform scratch detection on the image to be processed to obtain the scratch mask.

[0064] Figure 4 A schematic diagram of a model architecture 400 provided by an embodiment of the present disclosure is shown.

[0065] like Figure 4 As shown, in some embodiments, the method 200 may be implemented using a model architecture 400, wherein the model architecture 400 may include a scratch processing model 402, and the scratch processing model 402 may further include a scratch detection model 4022 and a scratch repair model 4024. In this step, the image to be processed 300 is input into the scratch detection model 4022, and a scratch mask 310 may be output.

[0066] In step 20424, the scratched area in the image to be processed is repaired based on the scratch mask.

[0067] As described above, after obtaining the scratch mask 310, the scratch mask 310 is used to mask the non-scratch area in the image to be processed 300, and then the scratch area can be repaired, thereby ensuring the repair effect while avoiding affecting the image effect of the non-scratch area.

[0068] Alternatively, if Figure 4As shown, the scratch repair model 4024 can be used to repair the scratch area in the image to be processed 300 based on the scratch mask 310. For example, the scratch mask 310 and the image to be processed 300 are input into the scratch repair model 4024 together, so as to output the intermediate image 320 after the scratch is repaired, as shown in FIG. Figure 3C shown.

[0069] In some embodiments, the scratch detection model 4022 and the scratch repair model 4024 may be obtained by pre-training using the first sample set.

[0070] The first sample set may further include a plurality of first samples and a plurality of second samples, wherein the first samples include images without scratches (e.g., images with higher definition), and the second samples include images with scratches. In this way, the scratch detection model 4022 and the scratch repair model 4024 are trained together using images with scratches and images without scratches, so that the models can learn features of both images with scratches and images without scratches.

[0071] As an optional embodiment, the second sample may further include an image after scratches are superimposed on the first sample. In this way, when the sample quantity of the image with scratches is insufficient, the second sample can be generated by adding scratches to the image without scratches, so that the second sample can be expanded, which is conducive to better training the model.

[0072] Optionally, the scratches superimposed in the second sample include scratches generated by random walk, so that the generated scratches are random and can better reflect the characteristics of naturally generated scratches.

[0073] As an optional embodiment, when generating scratches by random walk, constraints may be added, for example, limiting the horizontal coordinate difference range or the vertical coordinate difference range of the scratches. That is, the difference between the minimum horizontal coordinate point and the maximum horizontal coordinate point in the generated scratches must be within the horizontal coordinate difference range, or the difference between the minimum vertical coordinate point and the maximum vertical coordinate point in the generated scratches must be within the vertical coordinate difference range. In this way, when the generated scratches extend in the horizontal direction or vertical direction, they can maintain a long strip shape, which is closer to the scratch shape. For example, irregular strip scratches (for example, Figure 3A Scratches 302B, 302C).

[0074] For another example, considering that old photos may have block-shaped marks due to dirt, the constraint may be to limit the area of ​​the scratches, for example, by setting a minimum area value and a maximum area value, so that some block-shaped scratches can be generated (for example, Figure 3A Scratches 302A).

[0075] In some embodiments, the scratches superimposed in the second sample have a target color, and the target color is selected from the color of at least one scratch sample in the scratch sample set. It can be understood that since the scratches on old photos are not necessarily all white, and considering the different reasons for the scratches, the scratches may present different colors. In view of this, a scratch sample set can be pre-designed, and the scratch sample set can include scratch samples of various colors. When superimposing scratches, in addition to superimposing scratches of corresponding shapes, a color (for example, RGB value) can be randomly selected from the scratch sample set to be added to the scratches, so that the produced second sample can be more realistic.

[0076] As an optional embodiment, an image fusion algorithm may be used to add the generated scratches to the image, so that the added scratches can be better integrated with the original image and can better reflect the real scratch effect.

[0077] In some embodiments, the first sample set further includes a plurality of third samples, wherein the third samples include scratch masks generated based on the scratches superimposed in the second sample. For example, the image after the scratches are superimposed can be binarized to obtain a corresponding scratch mask for training the scratch detection model 4022.

[0078] In step 2044, noise reduction is performed on the intermediate image to obtain the first image.

[0079] Alternatively, if Figure 4 As shown, the noise processing model 404 can be used to perform noise reduction processing on the intermediate image 320 to obtain the first image 330, as shown in FIG. Figure 3D shown.

[0080] contrast Figure 3C and Figure 3D It can be seen that the noise is repaired by the noise processing model 404, thereby removing the point noise in the image and further improving the clarity of the image. As an optional embodiment, the noise processing model 404 can be obtained by pre-training using images containing noise and images without noise, which will not be repeated here.

[0081] It can be understood that the aforementioned defect repair embodiments are merely exemplary, and in some cases, other defect repairs may be performed on the first image, and these defect repair methods also fall within the protection scope of the present disclosure.

[0082] As an optional embodiment, after the defect repair is completed, the first image 330 can be output as the image processing result. Since the first image 330 repairs the defects such as scratches and noise, the image to be processed 300 becomes a clear first image 330, which better restores the effect of the original image and improves the user experience.

[0083] Considering that the image after defect repair may still have the problem of image blur, in some embodiments, the first image 330 may be further processed.

[0084] Then, in step 206 , object detection may be performed on the first image 330 .

[0085] In this step, target detection may be performed on the first image 330 by using a target detection algorithm or a target detection model to identify whether a target object is included in the first image 330. The target object may be a variety of objects, such as animals, plants, etc. The specific object to be identified may vary according to the different training samples, so the recognition capability of the target detection model will not be described in detail here.

[0086] like Figure 3D As shown, illustratively, the first image 330 may include a target object 332 and a portion 334 outside the target object 332. The target object 332 may further include a face 332A and a torso 332B. Optionally, the identified target object may be only the face 332A or the torso 332B. The specific object to be identified may vary according to different training samples so that the recognition capability of the target detection model is different.

[0087] In step 208, in response to detecting the target object in the first image, the first image is enhanced to obtain a second image.

[0088] In this step, the image clarity can be improved by enhancing the image, thereby further improving the user experience.

[0089] In some embodiments, Figure 2D As shown, step 208 of performing enhancement processing on the first image may further include the following steps.

[0090] like Figure 4 As shown, in step 2082 , a first enhancement process 4062 is performed on the target object 332 in the first image 330 .

[0091] In step 2084, a second enhancement process 4064 is performed on the portion 334 of the first image 330 except the target object 332. Optionally, the second enhancement process 4064 may be a background enhancement process, which mainly enhances details (richer details, clearer textures) while taking into account noise suppression.

[0092] In this way, by performing different enhancement processing on the target object and the parts other than the target object, the difference between the target object and other parts can be highlighted, thereby improving the image effect.

[0093] Optionally, the enhancement degree of the first enhancement processing 4062 is stronger than that of the second enhancement processing 4064. For example, if the enhancement processing is to improve the clarity, the clarity of the target object 332 after the first enhancement processing 4062 will be higher than the clarity of the other parts 334 after the second enhancement processing 4064. In this way, after the enhancement processing, the clarity of the target object 332 is higher than that of the other parts 334, thereby improving the image effect and the user experience.

[0094] In some embodiments, performing enhancement processing on the first image includes: performing the first enhancement processing 4062 and the second enhancement processing 4064 on the first image using the image enhancement model 406, such as Figure 4 shown.

[0095] The image enhancement model 406 is obtained by pre-training using the second sample set. The second sample set includes a plurality of fourth samples and a plurality of fifth samples, the fourth sample includes an image with a clarity higher than a preset threshold (set according to actual conditions), and the fifth sample includes an image after blurring and noise processing the fourth sample. In this way, the image enhancement model 406 is trained together with the blurred and noise-processed images and the images with higher clarity, so that the model can learn the features of blurred images and noisy images, as well as the features of clear images. Optionally, blur processing includes but is not limited to Gaussian blur, box blur, double blur, and noise processing includes but is not limited to Gaussian noise and salt and pepper noise.

[0096] In some embodiments, the fifth sample is obtained by performing a first blurring process and a first noise adding process on the target object in the fourth sample and performing a second blurring process and a second noise adding process on the portion of the fourth sample other than the target object, wherein the blurring degree of the first blurring process is higher than the blurring degree of the second blurring process, and the noise degree of the first noise adding process is higher than the noise degree of the second noise adding process. The image enhancement model 406 obtained by training with such samples, after inputting the first image 330 therein, can naturally output the second image 340 in which the target object 332 is subjected to the first enhancement process 4062 and the other portion 334 is subjected to the second enhancement process 4064.

[0097] As an optional embodiment, after the image is enhanced, a second image 340 can be output as an image processing result. Since the second image 340 repairs defects such as scratches and noise and further enhances the image, the image to be processed 300 becomes a clearer and more focused second image 340, which better restores the effect of the original image and improves the user experience.

[0098] Considering that old photos may be discolored or black-and-white due to their age, in some embodiments, the second image 340 may be further processed.

[0099] Thus, in some embodiments, Figure 2A As shown, the method 200 may further include the following steps. Alternatively, in some other embodiments, if the target object cannot be detected in the first image, the method may jump to the following steps to continue execution.

[0100] In step 210, it is determined whether the second image is a decolorized photograph.

[0101] In this step, a decolorization detection is performed on the second image 340 to determine whether it is a decolorized photo, and different color processing is performed when it is a decolorized photo or not, so as to improve the image effect.

[0102] In some embodiments, Figure 2E As shown, step 210 of determining whether the second image is a decolorized photo may further include the following steps.

[0103] In step 2102, the mean and variance of the chromaticity values ​​of the second image in the target color space (eg, CIELUV color space) are calculated (eg, the mean and variance of the UV channel).

[0104] It can be understood that decolorized photos can be grayish-white photos, black-and-white photos, yellowed / old photos, and such photos usually show the problem of insufficient color saturation in the image. In this embodiment, based on the mean and variance of the chromaticity values, various types of decolorized photos can be better screened out to facilitate subsequent colorization processing and improve the image restoration effect.

[0105] In step 2104, in response to the average value of the chromaticity values ​​being less than or equal to a preset average value or the variance of the chromaticity values ​​being less than or equal to a preset variance value, it is determined that the second image is a decolorized photograph.

[0106] In step 2106, in response to the average value of the chromaticity values ​​being greater than a preset average value and the variance of the chromaticity values ​​being greater than a preset variance value, it is determined that the second image is a color photo.

[0107] Since the chromaticity value reflects the color of the image, in this embodiment, the images are divided into two categories, namely, decolorized photos and color photos, by statistically calculating the average value and variance of the chromaticity value and making judgments based on the threshold.

[0108] In step 212, in response to the second image 340 being a decolorized photo, the second image 340 is colorized 4082 to obtain a third image 350. In this way, when the second image 340 is a black and white image, it is converted into a colored third image 350 through the colorization process, so that the image effect is better.

[0109] In step 214, in response to the second image 340 being a color photo, the second image 340 is subjected to color enhancement processing to obtain a fourth image 360. Since old photos may have discoloration due to their age, color enhancement processing can eliminate the sense of age of the old photos to a certain extent and improve the image effect.

[0110] In some embodiments, Figure 4 As shown, the model architecture 400 may also include a color processing model 408 , which may intelligently identify whether the second image 340 is a decolorized photo and further select a colorization process or a color enhancement process to obtain a third image 350 or a fourth image 360 ​​.

[0111] Optionally, the color processing model 408 can be pre-trained using color photos and decolorized photos of color photos, so that when the second image 340 is input into the pre-trained color processing model 408, the image after colorization or color enhancement processing can be directly output.

[0112] It is understood that step 210 and step 208 may not necessarily be in a sequential relationship, or step 208 may not necessarily occur before step 210. In some embodiments, the target object may not be detected in the first image, so enhancement processing may not be performed, but color processing may be performed directly. Alternatively, in other embodiments, although the target object cannot be detected in the first image, a unified enhancement processing may be performed on the first image, for example, the first image may be subjected to a second enhancement processing and then subjected to color processing.

[0113] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0114] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0115] The embodiment of the present disclosure also provides a computer device for implementing the above method 200. Figure 5 FIG. 5 is a schematic diagram showing the hardware structure of an exemplary computer device 500 provided in an embodiment of the present disclosure. The computer device 500 can be used to implement Figure 1 The server 106 can also be used to implement Figure 1 In some scenarios, the computer device 500 can also be used to implement Figure 1 The database server 108.

[0116] like Figure 5 As shown, the computer device 500 may include: a processor 502, a memory 504, a network module 506, a peripheral interface 508 and a bus 510. The processor 502, the memory 504, the network module 506 and the peripheral interface 508 are connected to each other in communication within the computer device 500 through the bus 510.

[0117] Processor 502 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 502 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 502 may also include multiple processors integrated into a single logical component. For example, Figure 5 As shown, processor 502 may include a plurality of processors 502a, 502b, and 502c.

[0118] The memory 504 may be configured to store data (eg, instructions, computer code, etc.). Figure 5As shown, the data stored in the memory 504 may include program instructions (e.g., program instructions for implementing the method 200 of the embodiment of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 502 may also access the program instructions and data stored in the memory 504, and execute the program instructions to operate on the data to be processed. The memory 504 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 504 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0119] The network interface 506 can be configured to provide the computer device 500 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the above specific examples.

[0120] The peripheral interface 508 can be configured to connect the computer device 500 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0121] The bus 510 may be configured to transmit information between various components of the computer device 500 (e.g., the processor 502, the memory 504, the network interface 506, and the peripheral interface 508), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0122] It should be noted that, although the architecture of the above-mentioned computer device 500 only shows the processor 502, the memory 504, the network interface 506, the peripheral interface 508 and the bus 510, in the specific implementation process, the architecture of the computer device 500 may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the architecture of the above-mentioned computer device 500 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0123] The embodiment of the present disclosure also provides an image processing device. Figure 6 FIG. 6 is a schematic diagram of an exemplary device 600 provided by an embodiment of the present disclosure. Figure 6 As shown, the device 600 can be used to implement the method 200 and can further include the following modules.

[0124] The acquisition module 602 is configured to: acquire an image to be processed;

[0125] The repair module 604 is configured to: repair defects of the image to be processed to obtain a first image;

[0126] The detection module 606 is configured to: perform target detection on the first image;

[0127] The enhancement module 608 is configured to: in response to detecting a target object in the first image, perform enhancement processing on the first image to obtain a second image.

[0128] In some embodiments, the repair module 604 is configured to: perform scratch repair on the image to be processed to obtain an intermediate image; and perform noise reduction processing on the intermediate image to obtain the first image.

[0129] In some embodiments, the repair module 604 is configured to: perform scratch detection on the image to be processed to obtain a scratch mask; and repair the scratched area in the image to be processed based on the scratch mask.

[0130] In some embodiments, the repair module 604 is configured to: perform scratch detection on the image to be processed using a scratch detection model to obtain a scratch mask; and repair the scratch area in the image to be processed based on the scratch mask using a scratch repair model.

[0131] In some embodiments, the scratch detection model and the scratch repair model are obtained by pre-training using a first sample set; wherein the first sample set includes multiple first samples and multiple second samples, the first sample includes an image without scratches, the second sample includes an image with scratches superimposed on the first sample, and the scratches superimposed in the second sample include scratches generated by a random walk method.

[0132] In some embodiments, the superimposed scratch in the second sample has a target color, and the target color is selected from the color of at least one scratch sample in the scratch sample set.

[0133] In some embodiments, the first sample set further includes a plurality of third samples, the third samples including scratch masks generated based on the scratches superimposed in the second samples.

[0134] In some embodiments, the enhancement module 608 is configured to: perform a first enhancement process on the target object in the first image; and perform a second enhancement process on a portion of the first image other than the target object.

[0135] In some embodiments, the enhancement module 608 is configured to: perform the first enhancement processing and the second enhancement processing on the first image using an image enhancement model; wherein the image enhancement model is obtained by pre-training using a second sample set, the second sample set includes multiple fourth samples and multiple fifth samples, the fourth samples include images with clarity higher than a preset threshold, and the fifth samples include images after blurring and denoising the fourth samples.

[0136] In some embodiments, the fifth sample is obtained by performing a first blurring process and a first noise adding process on the target object in the fourth sample and performing a second blurring process and a second noise adding process on a portion of the fourth sample other than the target object, wherein the blurring degree of the first blurring process is higher than the blurring degree of the second blurring process, and the noise degree of the first noise adding process is higher than the noise degree of the second noise adding process.

[0137] In some embodiments, the device 600 also includes a color processing module configured to: determine whether the second image is a bleached photo; in response to the second image being a bleached photo, perform coloring on the second image to obtain a third image; in response to the second image being a color photo, perform color enhancement on the second image to obtain a fourth image.

[0138] In some embodiments, the color processing module is configured to: calculate the mean and variance of the chromaticity values ​​of the second image in the target color space; determine that the second image is a decolorized photo in response to the mean of the chromaticity values ​​being less than or equal to a preset mean or the variance of the chromaticity values ​​being less than or equal to a preset variance value; and determine that the second image is a color photo in response to the mean of the chromaticity values ​​being greater than a preset mean and the variance of the chromaticity values ​​being greater than a preset variance value.

[0139] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0140] The device of the above embodiment is used to implement the corresponding method 200 in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0141] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute method 200 described in any of the above embodiments.

[0142] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0143] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method 200 described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0144] Based on the same inventive concept, corresponding to the method 200 in any of the above embodiments, the present disclosure further provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors so that the processors execute the method 200. Corresponding to the execution subject corresponding to each step in each embodiment of the method 200, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0145] The computer program product of the above embodiment is used to enable the processor to execute the method 200 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0146] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0147] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0148] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0149] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An image processing method, comprising: Get the image to be processed; Performing defect repair on the image to be processed to obtain a first image; performing object detection on the first image; In response to detecting the target object in the first image, the first image is enhanced to obtain a second image.

2. The method of claim 1, wherein: Performing defect repair on the image to be processed to obtain a first image includes: Performing scratch repair on the image to be processed to obtain an intermediate image; Perform noise reduction processing on the intermediate image to obtain the first image.

3. The method of claim 2, wherein: Repairing scratches on the image to be processed to obtain an intermediate image includes: Performing scratch detection on the image to be processed to obtain a scratch mask; Based on the scratch mask, the scratch area in the image to be processed is repaired.

4. The method of claim 3, wherein: Performing scratch detection on the image to be processed to obtain a scratch mask, including: performing scratch detection on the image to be processed using a scratch detection model to obtain a scratch mask; Repairing the scratched area in the image to be processed based on the scratch mask includes: using a scratch repair model to repair the scratched area in the image to be processed based on the scratch mask.

5. The method of claim 3, wherein: The scratch detection model and the scratch repair model are obtained by pre-training using a first sample set; wherein the first sample set includes multiple first samples and multiple second samples, the first sample includes an image without scratches, the second sample includes an image after scratches are superimposed on the first sample, and the scratches superimposed in the second sample include scratches generated by a random walk method.

6. The method of claim 5, wherein: The superimposed scratches in the second sample have a target color, and the target color is selected from the color of at least one scratch sample in the scratch sample set.

7. The method of claim 5, wherein: The first sample set also includes a plurality of third samples, the third samples including scratch masks generated based on the scratches superimposed in the second samples.

8. The method of claim 1, wherein: Performing enhancement processing on the first image includes: Performing a first enhancement process on the target object in the first image; A second enhancement process is performed on a portion of the first image except the target object.

9. The method of claim 8, wherein: Performing enhancement processing on the first image, comprising: performing the first enhancement processing and the second enhancement processing on the first image using an image enhancement model; Among them, the image enhancement model is obtained through pre-training using a second sample set, the second sample set includes multiple fourth samples and multiple fifth samples, the fourth samples include images with clarity higher than a preset threshold, and the fifth samples include images after blurring and denoising the fourth samples.

10. The method of claim 9, wherein: The fifth sample is obtained by performing a first blurring process and a first noise adding process on the target object in the fourth sample and performing a second blurring process and a second noise adding process on a part of the fourth sample other than the target object, wherein the blurring degree of the first blurring process is higher than the blurring degree of the second blurring process, and the noise degree of the first noise adding process is higher than the noise degree of the second noise adding process.

11. The method of claim 1, wherein: The method further comprises: determining whether the second image is a decolorized photograph; In response to the second image being a decolorized photograph, coloring the second image to obtain a third image; In response to the second image being a color photo, color enhancement processing is performed on the second image to obtain a fourth image.

12. The method of claim 11, wherein: Determining whether the second image is a decolorized photograph comprises: Calculating the mean and variance of the chromaticity values ​​of the second image in the target color space; In response to the average value of the chromaticity values ​​being less than or equal to a preset average value or the variance of the chromaticity values ​​being less than or equal to a preset variance value, determining that the second image is a decolorized photograph; In response to the average value of the chromaticity values ​​being greater than a preset average value and the variance of the chromaticity values ​​being greater than a preset variance value, it is determined that the second image is a color photo.

13. An image processing device, comprising: The acquisition module is configured to: acquire the image to be processed; A repair module is configured to: repair defects of the image to be processed to obtain a first image; A detection module is configured to: perform target detection on the first image; The enhancement module is configured to: in response to detecting a target object in the first image, perform enhancement processing on the first image to obtain a second image.

14. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1-12.

15. A non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to perform the method according to any one of claims 1 to 12.

Images