Image processing method and related device

Abstract

The embodiments of the present application provide an image processing method and related devices, which are applied to the field of image processing technology. The method includes: obtaining an image to be processed; performing block processing on the image to be processed to obtain one or more block images; performing image processing on one or more block images according to the difficulty probability value of the block image to obtain one or more target block images, the target block image corresponds to the block image one-to-one, the difficulty probability value is used to describe the difficulty of image processing of the block image, and performing splicing processing on one or more target block images to obtain the target image. Images of different difficulty levels correspond to different processing methods, which can improve the processing effect of images with high difficulty levels and reduce the overfitting of images with low difficulty levels. Specifically, the clarity of the high-frequency area in the image can be improved, and the overfitting of the low-frequency area in the image can be reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and related devices. Background Art

[0002] During the process of image acquisition, transmission and processing, the image may be damaged to a certain extent due to noise, image distortion caused by the imaging system, imaging nonlinearity and motion blur, etc., and the image quality and clarity will be reduced, resulting in poor image visualization and information extraction capabilities.

[0003] In a possible design, the image is processed by image super resolution, image noise reduction, etc. to improve the clarity of the image.

[0004] However, the restoration effect of some or all areas in the processed image is not good, for example, the clarity is poor, pseudo texture appears, etc. Summary of the invention

[0005] The embodiments of the present application provide an image processing method and related devices, which are applied to the field of image processing technology. Performing image processing based on the probability value of the block image allows the image processing model to pay more attention to the block image corresponding to the high-frequency area, thereby improving the processing effect of the block image corresponding to the high-frequency area. The probability value of the difficult example can characterize the difficulty of image processing for the block image. Images of different difficulty levels correspond to different processing methods, which can improve the processing effect of images with high difficulty levels and reduce the overfitting of images with low difficulty levels. Specifically, the clarity of the high-frequency area in the image can be improved, and the overfitting of the low-frequency area in the image can be reduced.

[0006] In the first aspect, the embodiment of the present application proposes an image processing method. The method includes: obtaining an image to be processed; performing block processing on the image to be processed to obtain one or more block images; performing image processing on one or more block images according to the probability value of the hard example of the block image to obtain one or more target block images, and the target block images correspond to the block images one by one; performing splicing processing on one or more target block images to obtain a target image.

[0007] The probability value of a difficult example is used to describe the degree of difficulty of image processing of the segmented image. The probability value of a difficult example is related to one or more of the following parameters: changes in semantic information of the segmented image, changes in intensity of the segmented image, changes in grayscale of the segmented image, and changes in brightness of the segmented image. In the embodiment of the present application, image processing can be achieved through an image processing model or in any other manner, which is not limited here. The probability value of a difficult example of a segmented image can be achieved through a classification model or in any other manner (for example, a segmentation algorithm), which is not limited here. The clarity of the target segmented image is higher than that of the segmented image.

[0008] In this way, images of different difficulty levels have different processing methods, which can improve the processing effect of images with high difficulty levels and reduce the overfitting of images with low difficulty levels. In this way, different processing can be performed according to the difficulty level of the block images, which can improve the processing effect of images with high difficulty levels and reduce the overfitting of images with low difficulty levels.

[0009] Optionally, one or more block images are processed according to the hard example probability values ​​of the block images to obtain one or more target block images, including: inputting the block images into an image processing model to obtain processed block images; the image processing model is trained based on training images with different hard example probability values, and the training images with different hard example probability values ​​contribute differently to the image processing model.

[0010] In this way, training images with different difficulty probability values ​​contribute differently to the image processing model, which can increase the image processing device's attention to difficult images and improve the image processing device's processing effect on difficult images; correspondingly, the image processing device's contribution to the weight update of simple images can be reduced; the attention to simple images can be reduced, thereby reducing overfitting of simple images.

[0011] Optionally, the method also includes: obtaining a training image; inputting the training image into an image processing model to obtain an image processed by the image processing model and a first loss; optimizing the image processing model according to the first loss until the first loss is less than or equal to a first threshold; wherein the first loss is related to a hard example probability value of the training image, and the hard example probability value of the training image is related to one or more of the following parameters: a change in semantic information of the training image, a change in intensity of the training image, a change in grayscale of the training image, and a change in brightness of the training image.

[0012] The first loss may correspond to the hard example mining loss. The hard example probability value may be used to calculate the hard example loss of the training process, which may improve the learning effect of the image processing model for hard images.

[0013] In this way, by training the image processing device through the difficult example mining loss, the contribution of the image processing model to the weight update of difficult images can be improved, thereby increasing the image processing device's attention to difficult images and improving the image processing device's processing effect on difficult images; correspondingly, by training the image processing device through the difficult example mining loss, the contribution of the image processing device to the weight update of simple images can be reduced; the attention to simple images can be reduced, thereby reducing overfitting of simple images.

[0014] Optionally, the training images have label values ​​and the first loss satisfies: in, For the first loss, is the probability value of the hard example of the training image, GT is the label value of the training image, Net_out is the image processed by the image processing model (also called the processed training image), and γ is greater than 0.

[0015] In this way, the first loss is related to the probability value of the difficult example, so that images of different difficulty levels contribute differently to the weight update of the image processing model, thereby improving the processing effect of the image processing model on difficult images.

[0016] Optionally, one or more block images are processed according to the hard example probability values ​​of the block images to obtain one or more target block images, including: classifying each block image to obtain the hard example probability value of the block image; guiding the image processing model to extract features of the block images according to the hard example probability value to obtain image features; and restoring the image features to obtain the target block image.

[0017] In this way, the image processing model can process different block images according to the probability value of the difficult example, improve the processing effect of the images with high difficulty, and reduce the overfitting of the images with low difficulty. It is used for subsequent image restoration tasks. At the same time, the probability value of the difficult example will also be used to calculate the difficult example loss in the training process, so that the network can learn better for difficult samples.

[0018] Optionally, the image features are obtained by guiding the image processing model to extract features of the block images according to the hard example probability values, including: expanding the hard example probability values ​​into a feature map, and cascading the feature map with the first image feature to obtain the cascaded features; the first image feature is obtained by extracting features of the block images; and obtaining the image features by guiding the image processing model to extract features of the block images through the cascaded features.

[0019] In this way, the probability value of the hard example is associated with the feature of the image, so that subsequent images can pay more attention to the feature, improve the processing effect of the feature, and improve the clarity of the area corresponding to the feature in the processed image.

[0020] Optionally, the image processing model performs N feature extractions on the block image, where N is a positive integer; the image processing model is guided by the cascaded features to extract features from the block image to obtain image features, including: performing convolution and activation processing on the cascaded features to obtain attention features; after the Mth feature extraction, multiplying the attention features with the features extracted by the Mth feature extraction, and performing NM feature extractions on the multiplied features to obtain image features.

[0021] In this way, the attention feature can be introduced into image processing at an early stage, and the subsequent image processing can be affected by the attention feature, which has a better effect on improving the attention of block images with higher difficulty.

[0022] Optionally, the method further includes: inputting the divided images into a classification model to obtain a probability value of a difficult example, wherein the classification model is trained based on a plurality of training images, and the label values ​​corresponding to the plurality of training images are different.

[0023] In this way, the classification model is trained to make the classification processing of the classification model more accurate.

[0024] Optionally, the method also includes: inputting the training image into the classification model to obtain a hard example probability value and a second loss of the training image; optimizing the classification model according to the second loss until the second loss is less than or equal to a second threshold; wherein the second loss is related to the label value corresponding to the training image and the hard example probability value of the image output by the classification model.

[0025] The second loss may correspond to the classification loss.

[0026] In this way, the adjustment of the classification model parameters is achieved through the loss function.

[0027] Optionally, the second loss satisfies: is the second loss, X is the label value of the training image, and Y is the probability value of the hard example of the training image.

[0028] Optionally, the image processing includes: one or more of the following: image super-resolution processing, image denoising processing, image deblurring processing or image dark portion brightening processing.

[0029] In this way, it can be applied to a variety of image processing methods.

[0030] In a second aspect, an embodiment of the present application provides an image processing device, which may be an electronic device, or a chip or chip system in an electronic device. The image processing device may include a communication unit and a processing unit. When the image processing device is an electronic device, the communication unit is used to perform the step of acquiring the image to be processed, and the processing unit is used to implement the steps of segmenting the image to be processed, image processing, etc., so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing device is an electronic device, the processing unit may be a processor. The communication unit is used to perform the step of acquiring the image to be processed. The image processing device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit, so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit, so that the electronic device implements the image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (eg, a register, a cache, etc.), or a storage unit within the electronic device that is located outside the chip (eg, a read-only memory, a random access memory, etc.).

[0031] Exemplarily, a communication unit is used to obtain an image to be processed; a processing unit is used to perform block processing on the image to be processed to obtain one or more block images; the processing unit is also used to perform image processing on one or more block images according to the difficulty probability value of the block images to obtain one or more target block images, the target block images correspond to the block images one-to-one, the difficulty probability value is used to describe the difficulty of image processing of the block images, and the difficulty probability value is related to one or more of the following parameters: changes in semantic information of the block images, changes in intensity of the block images, changes in grayscale of the block images, and changes in brightness of the block images; the processing unit is also used to perform splicing processing on one or more target block images to obtain a target image.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the method described in the first aspect or any possible implementation of the first aspect.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0034] In a fifth aspect, an embodiment of the present application provides a computer program product including a computer program. When the computer program runs on a computer, the computer executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0035] In a sixth aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0036] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0037] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0039] Figure 2 A schematic diagram showing a comparison between an image before and after processing in a possible design;

[0040] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of the software architecture of a terminal device provided in an embodiment of the present application;

[0042] Figure 5 A flowchart of an image processing method provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of a flow chart of an image processing method for a segmented image provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of a block image processing flow provided in an embodiment of the present application;

[0045] Figure 8 A flow chart of a classification model training method provided in an embodiment of the present application;

[0046] Fig. 9 A flowchart of a method for training an image processing model provided in an embodiment of the present application;

[0047] Fig.10 A schematic diagram of a curve of weight items corresponding to different γ provided in an embodiment of the present application;

[0048] Fig.11 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0049] Fig.12 A flowchart of an image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0051] 1. Low level vision algorithms: They mainly focus on the basic characteristics of images, such as color, texture, and shape. Low level vision algorithms are usually used for edge detection, image enhancement, image super resolution, or image denoising. Low level vision algorithms are widely used in the field of digital image processing, such as image restoration or image beautification.

[0052] 2. Image super-resolution refers to the restoration of a high-resolution image from a low-resolution image or image sequence.

[0053] 3. Image noise reduction refers to the process of reducing noise in digital images. Image noise reduction can also be called image denoising.

[0054] 4. High-level visual algorithms: involve the understanding and interpretation of image content, such as object recognition, scene analysis, or semantic segmentation. High-level visual algorithms are applied to complex visual tasks, such as autonomous driving, virtual reality, or intelligent monitoring. When using high-level visual algorithms, they usually require low-level visual algorithm processing of the image first.

[0055] 5. Other terms

[0056] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0057] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0058] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0059] In the embodiment of the present application, "at..." can be the instant when a certain situation occurs, or can be a period of time after a certain situation occurs, and the embodiment of the present application does not specifically limit this. In addition, the interface of the terminal device provided in the embodiment of the present application is only an example, and the interface can also include more or less content.

[0060] 6. Terminal equipment

[0061] The terminal device of the embodiment of the present application may also be any form of electronic device. For example, the electronic device may include a handheld device with image processing function, a vehicle-mounted device, etc. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MID), wearable devices (e.g., smart watches, smart glasses, smart bracelets or smart jewelry, etc.), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, terminal devices in Internet of Things (IoT) systems, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), and wireless communication devices. assistant, PDA), handheld devices with wireless communication functions, computing devices or other processing devices connected to a wireless modem, vehicle-mounted devices, terminal devices in a 5G network, or terminal devices in a future evolved public land mobile communication network (public land mobile network, PLMN), etc., and the embodiments of the present application are not limited to this.

[0062] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0063] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0064] 7. Server

[0065] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, etc. The computer device mentioned in the embodiments of the present application can be located outside the blockchain network or inside the blockchain network, without limitation.

[0066] For example, Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, the scene includes: an electronic device 101. A user can use the electronic device 101 to capture an image. In response to the user's capture operation, the electronic device 101 can perform image super-resolution, image denoising, and other processing on the image captured by the camera to obtain and display the processed image. The clarity of the processed image is higher than the clarity of the image captured by the camera.

[0067] Above Figure 1 The scenes shown are only examples, and the image processing method provided in the embodiments of the present application can also be applied to other scenes that require image processing. For example, taking the editing function in the gallery application as an example, the user can perform image super-resolution, image denoising and other processing on the images in the gallery application to obtain and display the processed images. The clarity of the processed image is higher than the clarity of the image captured by the camera. For example, the image to be processed can also be input into the user interface displayed by the electronic device, and the electronic device performs image super-resolution, image denoising and other processing on the image to be processed to obtain and display the processed image. The embodiments of the present application do not specifically limit the application scenarios.

[0068] In the process of image processing, low-level visual algorithms (e.g., image super-resolution, image denoising, etc.) can usually be used to process the image to improve the clarity of the image. Exemplarily, the image is processed by a U-Net model to improve the clarity of the image.

[0069] In a possible design, the image processing method learned by the trained U-Net model is to perform the same processing on each block image, which may result in inaccurate details added to some areas of the processed image (for example, unrealistic textures appear in low-frequency areas) or insufficient clarity in some areas (for example, unclear high-frequency areas).

[0070] It should be noted that the image intensity (brightness / grayscale) corresponding to the low-frequency area in the image changes slowly, while the image intensity (brightness / grayscale) corresponding to the high-frequency area in the image changes dramatically. The difficulty of image processing corresponding to the low-frequency area and the high-frequency area is different. The image corresponding to the high-frequency area can be called a difficult image, and the image corresponding to the low-frequency area can be called a simple image.

[0071] In a possible design, the U-Net model does not distinguish the complexity of the training images when training the image processing algorithm (e.g., the image processing model), that is, the same loss function is used for training images of different complexity. In this way, each training image contributes equally to the training of the image processing algorithm, and then when the image processing algorithm (e.g., the U-Net model) is used for image processing later, the block images corresponding to the low-frequency area and the block images corresponding to the high-frequency area cannot be processed differently, resulting in poor clarity in some areas of the processed image.

[0072] For example, Figure 2 Figure 1 is a schematic diagram showing the comparison of an image before and after processing in a possible design. Figure 2 As shown, the image before processing includes high-frequency area 1a and low-frequency area 2a; the image after processing includes high-frequency area 1b and low-frequency area 2b. Compared with high-frequency area 1a, the clarity of high-frequency area 1b decreases; compared with low-frequency area 2a, the clarity of low-frequency area 2b increases. The clarity of some areas in the processed image decreases, and the user experience is poor.

[0073] In view of this, an embodiment of the present application provides an image processing method and related devices, and after the image is processed in blocks, the probability value of a difficult example of each block image can be obtained. Performing image processing according to the probability value of a difficult example of the block image allows the image processing model to pay more attention to the block images corresponding to the high-frequency area, thereby improving the processing effect of the block images corresponding to the high-frequency area. The probability value of a difficult example can characterize the difficulty of performing image processing on the block images. Image processing can be performed according to the probability value of a difficult example of the image, specifically, the clarity of the high-frequency area in the image can be improved, and the overfitting of the low-frequency area in the image can be reduced.

[0074] When training the image processing algorithm, the loss function is adjusted according to the difficulty of the training image, so that the loss function of the training image with a greater difficulty is larger, and then the training contribution of the training image with a greater difficulty is larger, thereby improving the processing effect of the image processing algorithm on the image with a greater complexity. Adaptively, the loss function of the training image with a less difficult degree is smaller, and then the training contribution of the training image with a less difficult degree is smaller, thereby reducing the overfitting of the image processing algorithm on the image with a less difficult degree.

[0075] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0076] For ease of understanding, the hardware and software architecture of the electronic device involved in the embodiments of the present application is described below. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3As shown, the electronic device includes: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0077] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0078] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. A memory may also be provided in the processor 110 for storing instructions and data.

[0079] In the embodiment of the present application, the electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0080] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. In some embodiments, the ISP can be set in the camera 193.

[0081] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In some embodiments, the electronic device can include 1 or N cameras 193, where N is a positive integer greater than 1.

[0082] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0083] For ease of understanding, the software system of the terminal device is described below. The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture, etc., which will not be described in detail here.

[0084] For example, Figure 4 A schematic diagram of the software architecture of a terminal device provided for an embodiment of the present application. Among them, the layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer (HAL), the kernel layer, and the hardware layer.

[0085] The application layer can include a series of application packages. Figure 4 As shown, the application package may include terminal camera applications, third-party camera applications, calendar applications, etc. Among them, the third-party camera application may include: and other applications.

[0086] like Figure 4 As shown, the application package may include applications such as camera, gallery, phone, map, etc.

[0087] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0088] like Figure 4 As shown, the application framework layer may include a window manager, a content provider, a resource manager, a view system, a camera access interface, etc.

[0089] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, touch the screen, drag the screen, capture the screen, etc.

[0090] Content providers are used to store and retrieve data (e.g., video, image, audio, etc.) and make the data accessible to applications.

[0091] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0092] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0093] The camera access interface allows applications to manage the camera and access the camera device, such as managing the camera to capture images.

[0094] The hardware abstraction layer may include multiple library modules, such as a camera library module. The Android system may load the corresponding library module for the device hardware, thereby achieving the purpose of the application framework layer accessing the device hardware. In the embodiment of the present application, the camera library module includes an algorithm module, and the algorithm model is used to realize the processing of the captured image.

[0095] Specifically, the algorithm module can process the image captured by the camera in blocks, and classify the processed images to obtain the probability value of a hard example. The hard example probability value is used to guide the image processing of the block images to obtain the target block images. The algorithm module can splice the target block images to obtain the target image.

[0096] Multiple modules in the HAL layer may comply with the hardware abstraction layer interface description language (HAL interface definition language, HIDL).

[0097] The kernel layer is the layer between hardware and software. Figure 4 As shown, the kernel layer may include one or more of the following: display driver, sensor driver, camera driver, etc.

[0098] like Figure 4 As shown, the hardware layer may include one or more of the following: external screen, internal screen, sensor, internal screen camera, external screen camera, etc.

[0099] Combine the following Figure 4 The photo-taking process of the terminal device in the embodiment of the present application is described.

[0100] Exemplarily, when the electronic device opens the camera application, the camera application transmits the camera opening command to the camera driver via the camera access interface in the application framework layer and the camera library module in the hardware abstraction layer. After the camera is turned on, the camera driver transmits the captured image to the algorithm module in the hardware abstraction layer. The algorithm module performs algorithm processing on the captured image to obtain the target image. The algorithm module transmits the target image to the camera application via the camera access interface for display. The specific algorithm processing can refer to the processing in the following image processing method, which will not be repeated here.

[0101] Combine the following Figures 5 to 8 The image processing method provided in the embodiment of the present application is described.

[0102] For example, Figure 5 A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the image processing method includes:

[0103] S501: Obtain an image to be processed.

[0104] In the embodiment of the present application, the image to be processed can be understood as an image that needs to improve the clarity. For example, an image with low resolution (for example, an image captured by a device with low resolution), an image with large noise, a damaged image, etc. A damaged image can be an image whose clarity is reduced due to noise, image distortion caused by the imaging system, imaging nonlinearity, motion blur, etc.

[0105] Exemplarily, the image to be processed may be an image taken by a user, an image downloaded by a user, an image of a video captured by screenshot, etc. The embodiment of the present application does not specifically limit the image to be processed.

[0106] S502 , performing block processing on the image to be processed to obtain a plurality of block images.

[0107] Block processing can be understood as dividing an image to be processed into several block images. The size of the block images is less than or equal to the size of the image to be processed.

[0108] It is understandable that dividing the image to be processed into multiple times for image processing can reduce the amount of single image processing operations and reduce the load on the electronic device.

[0109] In the embodiment of the present application, the electronic device randomly divides an image to be processed into a plurality of block images, or divides an image to be processed into a plurality of block images in sequence. For example, the image to be processed may be divided into a plurality of block images in sequence at a certain interval starting from the upper left corner of the image to be processed.

[0110] In the embodiment of the present application, several block images may partially overlap or may not overlap, which is not limited here.

[0111] In some embodiments, the computer device may divide an image to be processed according to a cut-off threshold. The cut-off threshold is used to indicate the maximum size of the switched image. When the cut-off threshold is larger, the number of divided images is smaller; when the cut-off threshold is smaller, the number of divided images is larger.

[0112] The interception threshold may be user-defined, or may be determined based on the performance of a computer device, or may be determined based on the content of an image, or in any other manner, which is not limited in the present embodiment.

[0113] S503: Classify the block images to obtain hard example probability values ​​of the block images.

[0114] In the embodiment of the present application, the block images can be classified according to the pre-trained classification model to obtain the probability value of the hard example of the block images; or the block images can be classified according to parameters such as the degree of change of image intensity, the degree of change of image grayscale, and the degree of change of image brightness to obtain the probability value of the hard example of the block images. This is not limited here.

[0115] For example, take the classification of block images by a classification model as an example, Figure 6 As shown, S503 includes: S601-S603.

[0116] S601: extract features from the block images to obtain features of the block images.

[0117] The features of the block images can be understood as generalized and easy-to-express features in the image, such as the texture, color, edge, and corner features of the image. The electronic device can extract features from the block images through a pre-trained feature extraction model, or through an algorithm such as a convolutional network, which is not specifically limited here.

[0118] Exemplarily, feature extraction may be performed on the block image through one or more convolutional networks.

[0119] S602: Perform global pooling processing on the features of the block image to obtain single-channel features.

[0120] Exemplarily, all pixel values ​​of each channel map in the feature of the block image may be averaged to obtain a new 1*1 channel map, that is, a single-channel feature.

[0121] S603, converting the single-channel features of the activation function into a hard example probability value.

[0122] Exemplarily, the features of a single channel are transformed by a sigmoid activation function to obtain a probability value of a hard example.

[0123] In this way, features can be extracted from the block images, and the difficulty of the block images can be determined based on the extracted features to obtain the corresponding probability value of the difficult example.

[0124] S504, expanding the probability value of the hard example into a feature map and concatenating it with the features extracted from the block image to obtain concatenated features.

[0125] Exemplarily, the hard example probability value may be expanded into a graph having the same size as the feature of the block image, and the expanded graph may be superimposed on the feature of the block image to obtain a cascaded feature.

[0126] The cascading method shown in S504 is only an example, and the cascading can be performed in any other way. For example, the hard example probability value can be multiplied with the feature of the block image to obtain the cascaded feature. The embodiment of the present application does not specifically limit the cascading method.

[0127] In this way, the probability value of the hard example is associated with the feature of the image, so that subsequent images can pay more attention to the feature, improve the processing effect of the feature, and improve the clarity of the area corresponding to the feature in the processed image.

[0128] S505: Obtain attention features based on the cascaded features.

[0129] In some embodiments, the cascaded features are processed by convolution and activation functions to obtain attention features. In this way, the image processing algorithm can focus on the difficult areas in the block image, further improve the processing effect of the difficult areas, and improve the clarity.

[0130] In some embodiments, the probability value of the difficult example may be expanded into a feature map as an attention feature. In this way, among the multiple block images, the block images with a high degree of difficulty correspond to a higher degree of attention, thereby improving the processing effect of the block images with a high degree of difficulty.

[0131] In some embodiments, the attention feature may be a feature map obtained by superimposing a probability value of a difficult example with a preset value and then performing an expansion process. In this way, a block image with a high degree of difficulty among multiple block images has a higher degree of attention, thereby improving the processing effect of the block image with a high degree of difficulty. In addition, the situation where the processing effect of the block image with a low degree of difficulty is poor due to low attention can be reduced.

[0132] S506: Perform image processing on the segmented images according to the attention features to obtain target segmented images.

[0133] In the embodiment of the present application, image processing can be performed on the block images according to a pre-trained image processing model or any other method.

[0134] For example, take the U-net model to process the image blocks as an example, Figure 6 As shown, S506 includes: S604-S607.

[0135] S604, performing a first feature extraction on the block image to obtain a first feature;

[0136] S605, multiplying the first feature by the attention feature map to obtain an attention allocation feature;

[0137] S606: Perform multiple feature extractions on the attention allocation features.

[0138] S607, performing multiple feature restoration on the features extracted multiple times to obtain processed block images.

[0139] Figure 6 In the model shown, the attention feature is introduced after the first feature extraction. In this way, the attention feature can be introduced into the image processing earlier, and the subsequent image processing can be affected by the attention feature, which has a better effect on improving the attention of the features with higher difficulty in the block image.

[0140] In some embodiments, attention features may be introduced after multiple feature extractions. Figure 6The attention feature is introduced once in the model shown; the attention feature can also be introduced multiple times, which is not specifically limited here.

[0141] S507: stitching the multiple processed block images to obtain a processed image.

[0142] Exemplarily, the multiple block images obtained in S502 above all have corresponding labels, and the labels correspond to the positions of the block images in the image to be processed. In S507, the positions of the processed block images can be determined according to the labels of the block images, and the multiple processed block images can be spliced ​​to obtain the processed image.

[0143] In this way, the image processing model can evaluate the difficulty of image processing in different areas of the image, and perform different processing according to the difficulty evaluation of each area. The image processing model pays different attention to block images of different difficulty levels, and increases the attention to the high-frequency areas of the image to improve the clarity of the high-frequency areas; at the same time, it reduces the overfitting of the low-frequency areas of the image, and reduces the appearance of unrealistic pseudo-textures in the low-frequency areas of the image.

[0144] For example, Figure 7 A schematic diagram of a processing flow of a block image provided in an embodiment of the present application is provided. Taking the designed classification network model to classify the block image, the U-net model is used to process the input block image as an example. Figure 7 As shown,

[0145] On the one hand, the block image is input into the classification model for classification processing to obtain the probability value of the hard example of the block image. On the other hand, the block image is input into the corresponding image processing model for feature extraction and feature restoration to obtain the target block image.

[0146] Exemplarily, after obtaining the block image, the classification model performs convolution processing and average pooling layer processing on the block image, and uses a sigmoid activation function to transform the features after the average pooling layer processing to obtain a difficult example probability value.

[0147] In some embodiments, the classification model can also perform expansion processing on the probability value of the hard example, and superimpose the expanded features with the convolution processed features to obtain cascaded features. The cascaded features are processed by convolution and sigmoid activation function to obtain the attention features of the block image.

[0148] After the image processing model of the block image is obtained, the image processing model of the block image performs multiple convolution processes and multiple deconvolution processes on the block image.

[0149] Taking the U-net model with 5 convolutions and 5 deconvolutions as an example, the U-net model performs convolution on the block image to obtain the first feature. The first feature is multiplied by the attention feature to obtain the attention feature. The attention feature is convolved again to obtain the second feature; the second feature is convolved again to obtain the third feature; the third feature is convolved again to obtain the fourth feature; the fourth feature is convolved again to obtain the fifth feature.

[0150] The fifth feature is deconvolved to obtain the sixth feature; the sixth feature and the fourth feature are deconvolved to obtain the seventh feature; the seventh feature and the third feature are deconvolved to obtain the eighth feature; the eighth feature and the second feature are deconvolved to obtain the ninth feature; the ninth feature and the first feature are deconvolved to obtain a processed block image.

[0151] In this way, the image processing model can evaluate the difficulty of image processing in different areas of the image, and perform different processing according to the difficulty evaluation of each area. The image processing model pays different attention to block images of different difficulty levels, increasing the attention to high-frequency areas in the image to improve the clarity of high-frequency areas; reducing the attention to low-frequency areas to reduce overfitting in low-frequency areas in the image and reduce the appearance of unrealistic textures in low-frequency areas in the image.

[0152] Based on the above embodiments, the embodiments of the present application can implement the classification and image processing of block images through the classification model and the image processing model. Figures 5 and 6 Before the image processing method shown, the classification model and image processing model involved in the method are also trained in advance to classify or process the image based on the trained model.

[0153] The following describes the classification model and image processing model training method involved in the embodiments of the present application in conjunction with the accompanying drawings.

[0154] For example, Figure 8 FIG. 1 is a flow chart of a training method for a classification model according to an embodiment of the present application. Figure 8 As shown, the training method includes:

[0155] S701, obtaining an image training set.

[0156] Each training image in the image training set is set with a label value. For example, taking the classification model to be trained as a binary classification model as an example, the label value of the training image can be set to 0 or 1. Specifically, the difficult image is set to 1; the simple image is set to 0.

[0157] It is understandable that the label value of the training image can be manually added by the user or automatically generated. Exemplarily, the label value of the training image can be added through image quality evaluation indicators. Image quality evaluation indicators include but are not limited to: peak signal-to-noise ratio (PSNR), structural similarity (SSIM) or learned perceptual image patch similarity (LPIPS), etc. This is not limited here.

[0158] Exemplarily, taking PSNR as an example, a training image is placed in a pre-trained image processing model, and the PSNR of the training image is calculated by the training image and the image output by the image processing model. When the PSNR of the training image is greater than or equal to the index threshold, the training image is a simple sample and the label value is 0. When the PSNR of the training image is less than the index threshold, the training image is a simple sample and the label value is 1. The index threshold can be 0.5, 0.1 or any value, which is not specifically limited here.

[0159] S702: Input the training images in the image training set into the pre-built classification model to obtain the hard example probability value of the training images output by the classification model.

[0160] S703: Obtain classification loss based on the probability value of the hard example of the block output by the classification model and the label value of the training image.

[0161] For example, the classification loss satisfy: Among them, X is the label value of the training image, and Y is the probability value of the hard example of the training image.

[0162] S704: Adjust the parameters of the classification model based on the classification loss, perform the next round of iterative training, and stop training when the training iteration stop condition is met to obtain a trained classification model. The training iteration stop condition includes: the classification loss is less than or equal to the first preset threshold.

[0163] In the embodiment of the present application, the adjustment of the parameters of the classification model can be achieved by the gradient descent method or any other method, which is not limited here. Exemplarily, based on the classification loss function as the objective function, the model parameters (for example, convolution kernel size, number of convolution kernels, step length, etc.) are adjusted using the gradient descent method, that is, the model parameters are adjusted to obtain the trained convolution kernel size, the trained number of convolution kernels, or the trained step length, etc.

[0164] For example, Figure 8Taking the classification model shown as an example, the classification loss adjusts the parameters in the convolution processing in the classification model.

[0165] In this way, training the classification model through classification loss can reduce the judgment error of the difficulty of the training images.

[0166] For example, Fig. 9 A flowchart of a training method for an image processing model provided in an embodiment of the present application is shown below. Fig. 9 As shown, the training method includes:

[0167] S801, obtaining an image training set, as well as an attention feature corresponding to each training image and a probability value of a difficult example for each training image.

[0168] In an embodiment of the present application, the image training set, the attention features corresponding to the training images, and the difficult example probability values ​​of the training images can be obtained separately or simultaneously. The order of acquisition is not limited here.

[0169] The attention features corresponding to the separated images and the probability values ​​of difficult examples can be input by the user or obtained through processing by the above-mentioned classification model. The embodiments of the present application do not specifically limit the source and acquisition process of the image training set, the attention features corresponding to the training images, and the probability values ​​of difficult examples of the training images.

[0170] S802: Input the training image and attention features into the image processing model to be trained to obtain a processed image.

[0171] S803: Obtain a difficult example mining loss according to the processed image, the label value of the training image, and the difficult example probability value of the training image.

[0172] Exemplarily, the hard example mining loss satisfies: in, is the probability value of the hard example of the training image, GT is the label value of the training image, and Net_out is the processed training image. γ is greater than 0.

[0173] It should be noted that when When it is small, it means that the current training image is a simple image, and the corresponding Smaller; when When it is larger, it means that the current training image is a difficult image, and the corresponding In this way, the contribution of difficult images to the training of the model can be increased, and the image processing model's attention to difficult images can be increased, thereby improving the clarity of difficult images; the contribution of simple images to the training of the model can be reduced, thereby reducing the overfitting of simple images.

[0174] It can be understood that γ is an adjustable parameter greater than 0. When γ is larger, the more contribution the difficult image makes to the training of the model, and the image processing model pays more attention to the difficult image; when γ is smaller, the less contribution the difficult image makes to the training of the model, and the image processing model pays less attention to the difficult image.

[0175] For example, as shown in Table 1 below, when γ is 2, When the probability value of the hard example is 0.5, the weight item of the training image is 0.25; when γ is 3, When the probability value of the hard example is 0.5, the weight term of the training image is 0.125; when γ is 4, When the probability value of the hard example is 0.5, the weight item of the training image is 0.0625.

[0176] Table 1 shows the ablation experiment of weight terms corresponding to different γ

[0177]

[0178] For example, Fig.10 A curve diagram of weight items corresponding to different γ provided in the embodiment of the present application. Fig.10 As shown, curve 901 is a weight curve when γ is 1; curve 902 is a weight curve when γ is 2; curve 903 is a weight curve when γ is 3; curve 904 is a weight curve when γ is 4; and curve 905 is a weight curve when γ is 5.

[0179] from Fig. 9 It can be seen that when γ is larger, the more contribution the difficult image makes to the training of the model, and the image processing model pays more attention to the difficult image; when γ is smaller, the less contribution the difficult image makes to the training of the model, and the image processing model pays less attention to the difficult image.

[0180] In some embodiments, γ can be confirmed by an ablation experiment. The ablation experiment can be understood as training the image processing model of the training images for different values ​​of γ under the same image training set, and determining the optimal value of γ according to the training results.

[0181] For example, taking γ as 1, 2, 3 or 4, when γ is 1, based on The image processing model of the training image is trained to obtain the trained image processing model A; when γ is 2, based on The image processing model of the training image is trained to obtain the trained image processing model B; when γ is 3, based on The image processing model of the training image is trained to obtain the trained image processing model C; when γ is 4, based on The image processing model of the training images is trained to obtain a trained image processing model D. The same set of images are input into the trained image processing models A and D, respectively, and the images output by the four models are compared. The γ value corresponding to the image processing model with the best output image clarity is selected from the images output by the four models.

[0182] S804, adjusting the parameters of the image processing model based on the hard example mining loss, performing the next round of iterative training, and stopping the training when the training iteration stop condition is met to obtain a trained image processing model. The training iteration stop condition includes: the hard example mining loss is less than or equal to the second preset threshold.

[0183] In some embodiments, the image processing model may not obtain attention features, that is, the image processing model is not modified and is trained according to the hard example mining loss. In this way, different training images contribute differently to the image processing model, which improves the processing effect of the image processing model on images with high difficulty and reduces overfitting on images with low difficulty.

[0184] For example, Fig. 9 Taking the image processing model shown as an example, the difficult example mining loss adjusts the parameters in multiple convolution processes and multiple deconvolution processes in the image processing model.

[0185] In this way, by training the image processing model through the difficult example mining loss, the contribution of the image processing model to the weight update of difficult images can be improved, thereby increasing the image processing model's attention to difficult images and improving the image processing model's processing effect on difficult images; correspondingly, by training the image processing model through the difficult example mining loss, the contribution of the image processing model to the weight update of simple images can be reduced; the attention to simple images can be reduced, thereby reducing overfitting of simple images.

[0186] The above embodiments improve the model training and model use. In the embodiments of the present application, only one of the two aspects may be improved. In some embodiments, the image processing model may not be improved, and the image processing model may be trained through the difficult example mining loss to improve the processing effect of the image processing model on the difficult images. Exemplarily, the electronic device may not execute S504-S506, the image processing model does not process the image according to the attention feature, and is trained according to the above-mentioned difficult example mining loss.

[0187] In some other embodiments, the image processing model can process the image according to the attention feature, and the loss function of the image processing model is independent of the probability value of the hard example. Figure 5 or Figure 6The method shown uses an image processing model whose loss function during training is independent of the hard example probability value.

[0188] In an embodiment of the present application, the classification model and the image processing model can be trained separately or combined together for end-to-end training simultaneously. The principle process of simultaneous training is similar to that of separate training and will not be repeated here.

[0189] The above embodiment uses the U-net model as an example to illustrate the image processing method or image processing model provided by the embodiment of the present application. The above U-net model can also be replaced by other models, for example, a strong baseline model based on Swin transform (SwinIR), a U-Net backbone network model based on a multilayer perceptron (MAXIM), a cyclic mirror transformation model (CycleISP), or a multi-stage progressive image restoration network (MPRNet) and other models. The embodiment of the present application does not specifically limit the model used for the image processing model.

[0190] The above embodiment uses an electronic device as an example to illustrate the image processing method. The image processing method provided in the embodiment of the present application can also be executed by a server. The specific forms of the electronic device and the server can refer to the above corresponding description, which will not be repeated here.

[0191] Alternatively, in some embodiments, the image processing method provided in the embodiments of the present application may also be jointly executed by a server and an electronic device; the electronic device and the server may be directly or indirectly connected via wired or wireless communication, and the present application does not impose any restrictions thereto.

[0192] Exemplarily, the electronic device may be responsible for acquiring the image to be processed, and performing block processing on the image to be processed to obtain a plurality of block images. The electronic device transmits the plurality of block images to the server. The server performs classification processing on each block image and image processing on the block image to obtain a plurality of processed block images. The server transmits the plurality of processed block images to the electronic device. The electronic device performs splicing processing on the block images and outputs the processed images.

[0193] Fig.11 A schematic diagram of an application scenario provided by an embodiment of the present application. Fig.11 As shown, the scene includes: electronic device 1001 and server 1002. Electronic device 1001 can obtain an image and transmit the image to server 1002. Server 1002 can process the image to obtain a processed image and transmit the processed image to electronic device 1001. Electronic device 1001 can display the processed image.

[0194] It should be understood that the two cases where the electronic device and the server jointly execute the above image processing method are merely illustrative and not exhaustive. The embodiment of the present application does not limit the specific implementation of the electronic device and the server jointly executing the above image processing method.

[0195] For example, Fig.12 A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the method includes:

[0196] S1101, obtaining an image to be processed.

[0197] S1102, performing block processing on the image to be processed to obtain one or more block images.

[0198] For details, please refer to the above corresponding instructions, which will not be repeated here.

[0199] S1103, performing image processing on one or more block images according to the hard example probability values ​​of the block images to obtain one or more target block images, wherein the target block images correspond to the block images one by one.

[0200] In the embodiment of the present application, the hard example probability value is used to describe the difficulty of performing image processing on the segmented image.

[0201] It can be understood that the hard example probability value is related to one or more of the following parameters: semantic information change of the block image, intensity change of the block image, grayscale change of the block image, and brightness change of the block image.

[0202] In the embodiments of the present application, image processing can be implemented through an image processing model or in any other manner, which is not limited here.

[0203] The hard example probability value of the block image can be realized by a classification model or any other method (eg, a block algorithm), which is not limited here. The definition of the target block image is higher than that of the block image.

[0204] It is understandable that the probability value of a difficult example is related to the difficulty of the block image. Different processing methods are used for images of different difficulty levels, which can improve the processing effect of images with high difficulty levels and reduce the overfitting of images with low difficulty levels. S1104: One or more target block images are spliced ​​to obtain a target image.

[0205] In this way, different processing can be performed on the block images according to the difficulty level, thereby improving the processing effect of images with high difficulty levels and reducing the overfitting of images with low difficulty levels.

[0206] Optionally, one or more block images are processed according to the hard example probability values ​​of the block images to obtain one or more target block images, including: inputting the block images into an image processing model to obtain processed block images; the image processing model is trained based on training images with different hard example probability values, and the training images with different hard example probability values ​​contribute differently to the image processing model.

[0207] In this way, training images with different difficulty probability values ​​contribute differently to the image processing model, which can increase the image processing device's attention to difficult images and improve the image processing device's processing effect on difficult images; correspondingly, the image processing device's contribution to the weight update of simple images can be reduced; the attention to simple images can be reduced, thereby reducing overfitting of simple images.

[0208] Optionally, the method also includes: obtaining a training image; inputting the training image into an image processing model to obtain an image processed by the image processing model and a first loss; optimizing the image processing model according to the first loss until the first loss is less than or equal to a first threshold; wherein the first loss is related to a hard example probability value of the training image, and the hard example probability value of the training image is related to one or more of the following parameters: a change in semantic information of the training image, a change in intensity of the training image, a change in grayscale of the training image, and a change in brightness of the training image.

[0209] The image processing model is used to perform image processing (eg, image super-resolution, image denoising, etc.). The image processing model may be a convolutional neural network model or any type of image processing model, which is not specifically limited here.

[0210] The first loss may correspond to the hard example mining loss. The hard example probability value may be used to calculate the hard example loss of the training process, which may improve the learning effect of the image processing model for hard images.

[0211] In this way, by training the image processing device through the difficult example mining loss, the contribution of the image processing model to the weight update of difficult images can be improved, thereby increasing the image processing device's attention to difficult images and improving the image processing device's processing effect on difficult images; correspondingly, by training the image processing device through the difficult example mining loss, the contribution of the image processing device to the weight update of simple images can be reduced; the attention to simple images can be reduced, thereby reducing overfitting of simple images.

[0212] Optionally, the training images have label values ​​and the first loss satisfies: in, For the first loss, is the probability value of the hard example of the training image, GT is the label value of the training image, Net_out is the image processed by the image processing model (also called the processed training image), and γ is greater than 0.

[0213] In this way, the first loss is related to the probability value of the difficult example, so that images of different difficulty levels contribute differently to the weight update of the image processing model, thereby improving the processing effect of the image processing model on difficult images.

[0214] Optionally, one or more block images are processed according to the hard example probability values ​​of the block images to obtain one or more target block images, including: classifying each block image to obtain the hard example probability value of the block image; guiding the image processing model to extract features of the block images according to the hard example probability value to obtain image features; and restoring the image features to obtain the target block image.

[0215] In this way, the image processing model can process different block images according to the probability value of the difficult example, improve the processing effect of the images with high difficulty, and reduce the overfitting of the images with low difficulty. It is used for subsequent image restoration tasks. At the same time, the probability value of the difficult example will also be used to calculate the difficult example loss in the training process, so that the network can learn better for difficult samples.

[0216] Optionally, the image features are obtained by guiding the image processing model to extract features of the block images according to the hard example probability value, including: cascading the hard example probability value with the first image feature to obtain the cascaded feature; the first image feature is obtained by extracting features of the block images; and obtaining the image features by guiding the image processing model to extract features of the block images through the cascaded feature.

[0217] In this way, the probability value of the hard example is associated with the feature of the image, so that subsequent images can pay more attention to the feature, improve the processing effect of the feature, and improve the clarity of the area corresponding to the feature in the processed image.

[0218] Optionally, the image processing model performs N feature extractions on the block image, where N is a positive integer; the image processing model is guided by the cascaded features to extract features from the block image to obtain image features, including: performing convolution and activation processing on the cascaded features to obtain attention features; after the Mth feature extraction, multiplying the attention features with the features extracted by the Mth feature extraction, and performing NM feature extractions on the multiplied features to obtain image features.

[0219] In this way, the attention feature can be introduced into image processing at an early stage, and the subsequent image processing can be affected by the attention feature, which has a better effect on improving the attention of block images with higher difficulty.

[0220] Optionally, the method further includes: inputting the divided images into a classification model to obtain a probability value of a difficult example, wherein the classification model is trained based on a plurality of training images, and the label values ​​corresponding to the plurality of training images are different.

[0221] In this way, the classification model is trained to make the classification processing of the classification model more accurate.

[0222] Optionally, the method also includes: inputting the training image into the classification model to obtain a hard example probability value and a second loss of the training image; optimizing the classification model according to the second loss until the second loss is less than or equal to a second threshold; wherein the second loss is related to the label value corresponding to the training image and the hard example probability value of the image output by the classification model.

[0223] The second loss may correspond to the classification loss.

[0224] In this way, the adjustment of the classification model parameters is achieved through the loss function.

[0225] Optionally, the second loss satisfies: is the second loss, X is the label value of the training image, and Y is the probability value of the hard example of the training image.

[0226] Optionally, the image processing includes: one or more of the following: image super-resolution processing, image denoising processing, image deblurring processing or image dark portion brightening processing.

[0227] In this way, it can be applied to a variety of image processing methods.

[0228] In the above embodiment, when the resolution of the input image meets the computer performance requirements, the computer device may not divide the image into blocks (for example, S1102 and S1104 are not executed), and the computer device performs image processing according to the hard example probability value of the image to be processed to obtain the target image. In this case, the processing method of the image to be processed can refer to the processing method of the above-mentioned divided image, and will not be described in detail here.

[0229] The method provided in the embodiment of the present application has been described above, and the device for executing the above method provided in the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided in the embodiment of the present application can execute the steps in the above method.

[0230] The image processing method provided in the embodiment of the present application can be applied to electronic devices with image processing functions. The electronic devices include terminal devices, and the specific device form of the terminal device can refer to the above related descriptions, which will not be repeated here.

[0231] An embodiment of the present application provides a terminal device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the above method.

[0232] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0233] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0234] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0235] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0236] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0237] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0238] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Performing block processing on the image to be processed to obtain one or more block images; Performing image processing on the one or more block images according to the probability value of the hard example of the block images to obtain one or more target block images, wherein the target block images correspond to the block images one by one, and the probability value of the hard example is used to describe the difficulty of performing image processing on the block images, and the probability value of the hard example is related to one or more of the following parameters: a change in semantic information of the block images, a change in intensity of the block images, a change in grayscale of the block images, and a change in brightness of the block images; Performing splicing processing on the one or more target block images to obtain a target image; The performing image processing on the one or more block images according to the hard example probability values ​​of the block images to obtain one or more target block images includes: Expanding the hard example probability value into a feature map, and cascading the feature map with a first image feature to obtain a cascaded feature; the first image feature is a feature obtained by extracting features from the block image; Extracting features of the block image by guiding the image processing model through the cascaded features to obtain the image features; The image features are restored to obtain the target block image.

2. The method according to claim 1, characterized in that The performing image processing on the one or more block images according to the hard example probability values ​​of the block images to obtain one or more target block images includes: Inputting the block images into an image processing model to obtain processed block images; The image processing model is trained based on training images with different hard example probability values, and the training images with different hard example probability values ​​contribute differently to the image processing model.

3. The method according to claim 2, characterized in that The method further comprises: Get training images; Inputting the training image into the image processing model to obtain an image processed by the image processing model and a first loss; Optimizing the image processing model according to the first loss until the first loss is less than or equal to a first threshold; Among them, the first loss is related to the hard example probability value of the training image, and the hard example probability value of the training image is related to one or more of the following parameters: semantic information change of the training image, intensity change of the training image, grayscale change of the training image, and brightness change of the training image.

4. The method according to claim 3, characterized in that The training images correspond to label values, and the first loss satisfies: Among them, the is the first loss, the is the probability value of a hard example of the training image, the GT is the label value of the training image, the Net_out is the image processed by the image processing model, and the γ is greater than 0.

5. The method according to any one of claims 1 to 4, characterized in that: Before expanding the hard example probability value into a feature map and cascading the feature map with the first image feature to obtain the cascaded feature, the method further includes: Classification processing is performed on each of the block images to obtain a hard example probability value of the block image.

6. The method according to claim 1, characterized in that The image processing model performs N feature extractions on the block image, where N is a positive integer; The cascaded features guide the image processing model to extract features of the block image to obtain image features, including: Perform convolution and activation processing on the cascaded features to obtain attention features; After the Mth feature extraction, the attention feature is multiplied by the feature extracted by the Mth feature extraction, and the multiplied feature is subjected to NM feature extractions to obtain the image feature.

7. The method according to claim 1, characterized in that The method further comprises: The block image is input into a classification model to obtain the hard example probability value, wherein the classification model is trained based on a plurality of training images, and the label values ​​corresponding to the plurality of training images are different.

8. The method according to claim 7, characterized in that The method further comprises: Inputting a training image into the classification model to obtain a hard example probability value and a second loss of the training image; Optimizing the classification model according to the second loss until the second loss is less than or equal to a second threshold; The second loss is related to the label value corresponding to the training image and the hard example probability value of the image output by the classification model.

9. The method according to claim 8, characterized in that The second loss satisfies: Said is the second loss, the X is the label value of the training image, and the Y is the hard example probability value of the training image.

10. The method according to any one of claims 1-4, 6-9, characterized in that: The image processing includes: one or more of the following: image super-resolution processing, image denoising processing, image deblurring processing or image dark portion brightening processing.

11. A chip system, characterized in that: The method comprises at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is used to run a computer program or instruction to execute the method according to any one of claims 1 to 10.

12. A computer device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the computer device performs the method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables a computer to execute the method according to any one of claims 1 to 10.

Images