Image processing method, electronic device, chip system and storage medium

Abstract

The application provides an image processing method, an electronic device, a chip system and a storage medium. The image processing method comprises: in response to a first instruction, obtaining an original image; if a preset region exists in the original image, performing image fusion processing on a first image and a second image to obtain a target image; the first image is an image obtained by processing the original image through a first image processing model, and the second image is an image obtained by processing the original image through a second image processing model; the image degradation degree of a second training sample of the second image processing model is higher than the image degradation degree of a first training sample of the first image processing model; the texture feature of the preset region in the second image is inconsistent with the texture feature in the original image; and the target image is output. The image quality of the second image is better, and the first image retains more texture features and color spaces. Therefore, while improving the definition of the target image, the occurrence of pseudo texture in the preset region can be inhibited.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, electronic device, chip system, and storage medium. Background Technology

[0002] After acquiring raw images from a camera, electronic devices can process these images using deep learning networks to obtain high-definition target images, thereby improving the user's photography experience. However, due to the limited learning capabilities of deep learning networks and the limited distribution range of training samples during training, pseudo-textures may appear in the generated target images. This means that areas without texture in the original image may appear with texture in the target image, reducing the quality of the resulting target image. Summary of the Invention

[0003] This application provides an image processing method, electronic device, chip system, and storage medium, which solves the problem of pseudo-texture in the target image obtained after processing the original image in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] In a first aspect, an image processing method is provided, executed in an electronic device, the method comprising:

[0006] In response to the first instruction, the original image captured by the camera of the electronic device is acquired;

[0007] If a preset region is determined to exist in the original image, image fusion processing is performed on the first image and the second image to obtain the target image; the first image is the image obtained after processing the original image by a first image processing model, and the second image is the image obtained after processing the original image by a second image processing model; the first image processing model is trained based on a first training sample, and the second image processing model is trained based on a second training sample, wherein the image degradation degree of the second training sample is higher than that of the first training sample; the texture features of the preset region in the second image are inconsistent with the texture features in the original image;

[0008] Output the target image.

[0009] In the above embodiments, since the image degradation degree of the second training sample used to train the second image processing model is higher than that of the first training sample used to train the first image processing model, the image output by the second image processing model has a higher image quality than the image input to the second image processing model. That is, the second image obtained using the second image processing model has better image quality and higher clarity than the original image. The first image obtained using the first image processing model has lower quality and weaker detail recovery capability than the second image, but retains more texture features and color space. In other words, the first image retains more texture features and color space in the preset region. Therefore, the target image obtained by fusing the first and second images can retain more texture features and color space in the preset region while maintaining clarity, thereby suppressing false textures in the preset region and improving the image quality of the target image.

[0010] In one embodiment, the step of performing image fusion processing on the first image and the second image to obtain a target image includes: generating a first image to be fused based on the location of the preset region and the first image, wherein the first image to be fused is used to characterize the pixel information of the preset region; generating a second image to be fused based on the location of the preset region and the second image, wherein the second image to be fused is used to characterize the pixel information of the region outside the preset region; and performing image fusion processing on the first image to be fused and the second image to be fused to obtain the target image.

[0011] Since the first image to be fused is generated based on the first image and is used to represent the pixel information of the preset region, and the second image to be fused is generated based on the second image and is used to represent the pixel information of the region outside the preset region, by performing image fusion processing on the first image to be fused and the second image to be fused, the preset region in the second image can be replaced by the preset region in the first image, and the region outside the preset region in the second image can be retained. Therefore, the clarity of the target image can be improved while suppressing the appearance of false textures in the preset region.

[0012] In one embodiment, before performing image fusion processing on the first image and the second image, the method further includes: determining whether the preset region exists in the original image based on the pixel differences of corresponding local regions in the second image and the original image. If the pixel differences of the local regions in the second image and the original image are large, it indicates that the corresponding local regions differ between the second image and the original image, thus indicating that the second image has lost more image details compared to the original image, and therefore it is determined that the preset region exists in the second image.

[0013] In one embodiment, determining whether the preset region exists in the original image based on the pixel differences of corresponding local regions in the second image and the original image includes: determining a difference image between the second image and the original image, the difference image representing the pixel differences of corresponding pixels in the second image and the original image; determining the variance of pixel values ​​of each local region in the difference image; if there is a local region with a variance greater than a threshold, it indicates that the corresponding local region has a large pixel difference in the original image and the second image, and it is determined that the preset region exists in the original image; if there is no local region with a variance greater than the threshold, it is determined that the preset region does not exist in the original image.

[0014] In one embodiment, the method further includes: if it is determined that there is no preset region in the original image, the second image is used as the target image, thereby obtaining a target image with higher clarity.

[0015] In one embodiment, before acquiring the original image captured by the camera of the electronic device, the method further includes: training a first classification model based on a first loss function and the first training samples to obtain a first image processing model. The first loss function is determined based on the first pixel difference between the first predicted image output by the first classification model and the corresponding first label image. The first pixel difference is the difference in pixel values ​​between corresponding pixels in the first predicted image and the first label image. Training the first classification model using the first loss function to obtain the first image processing model can improve the closeness between the image output by the first image processing model and the original image, allowing the image output by the first image processing model to retain more texture features of the original image.

[0016] In one embodiment, before acquiring the original image captured by the camera of the electronic device, the method further includes: training a second classification model based on a second loss function and the second training samples to obtain a second image processing model. The second loss function is determined based on the second pixel difference between the second predicted image and the corresponding second label image output by the second classification model, and the frequency domain difference of the preset region. The second pixel difference is the difference in pixel values ​​between corresponding pixels in the second predicted image and the second label image. The frequency domain difference of the preset region includes the difference in amplitude and phase between the first frequency domain image and the second frequency domain image. The first frequency domain image is the image obtained by frequency domain transformation of the preset region in the second predicted image, and the second frequency domain image is the image obtained by frequency domain transformation of the preset region in the second label image. Since the second loss function includes the frequency domain difference of the preset region, training the second classification model using the second loss function to obtain the second image processing model can improve the image restoration capability of the second image processing model in high-frequency texture regions.

[0017] In one embodiment, the method further includes:

[0018] A first standard deviation of pixel values ​​in each local region of the second predicted image is determined; a second standard deviation of pixel values ​​in each local region of the second labeled image is determined; and the location of the preset region is determined based on the difference between each first standard deviation and the corresponding second standard deviation. Specifically, for the same local region, if the difference between the first standard deviation and the second standard deviation is greater than a threshold, it is determined that the local region has pixel differences in the second labeled image and the second predicted image. The corresponding local region is the region with texture differences, and the region composed of all local regions is the preset region.

[0019] In a second aspect, an image processing apparatus is provided, comprising:

[0020] The acquisition module is used to acquire the original image captured by the camera of the electronic device in response to the first instruction;

[0021] A processing module is configured to perform image fusion processing on a first image and a second image to obtain a target image if a preset region is determined to exist in the original image; the first image is the image obtained after processing the original image by a first image processing model, and the second image is the image obtained after processing the original image by a second image processing model; the first image processing model is trained based on a first training sample, and the second image processing model is trained based on a second training sample, wherein the image degradation degree of the second training sample is higher than that of the first training sample; the texture features of the preset region in the second image are inconsistent with the texture features in the original image;

[0022] The output module is used to output the target image.

[0023] In one embodiment, the processing module is specifically used for:

[0024] A first image to be fused is generated based on the location of the preset region and the first image. The first image to be fused is used to characterize the pixel information of the preset region.

[0025] A second image to be fused is generated based on the location of the preset region and the second image. The second image to be fused is used to represent the pixel information of the region outside the preset region.

[0026] The first image to be fused and the second image to be fused are subjected to image fusion processing to obtain the target image.

[0027] In one embodiment, the processing module is further configured to:

[0028] The existence of the preset region in the original image is determined based on the pixel differences between the corresponding local regions in the second image and the original image.

[0029] In one embodiment, the processing module is further configured to:

[0030] Determine a difference image between the second image and the original image, wherein the difference image represents the pixel differences between corresponding pixels in the second image and the original image;

[0031] Determine the variance of pixel values ​​in each local region of the difference image;

[0032] If there is a local region with a variance greater than the threshold, it is determined that the preset region exists in the original image;

[0033] If there is no local region with a variance greater than the threshold, it is determined that the preset region does not exist in the original image.

[0034] In one embodiment, the processing module is further configured to:

[0035] If it is determined that the preset region does not exist in the original image, the second image is used as the target image.

[0036] In one embodiment, the processing module is further configured to:

[0037] The first classification model is trained based on the first loss function and the first training samples to obtain the first image processing model. The first loss function is determined based on the first pixel difference between the first predicted image and the corresponding first label image output by the first classification model. The first pixel difference is the difference between the pixel values ​​of the corresponding pixels in the first predicted image and the first label image.

[0038] In one embodiment, the processing module is further configured to:

[0039] The second classification model is trained using the second loss function and the second training samples to obtain the second image processing model. The second loss function is determined based on the second pixel difference between the second predicted image and the corresponding second label image output by the second classification model, as well as the frequency domain difference of the preset region. The second pixel difference is the difference in pixel values ​​between corresponding pixels in the second predicted image and the second label image. The frequency domain difference of the preset region includes the difference in amplitude and phase between the first frequency domain image and the second frequency domain image. The first frequency domain image is the image obtained by frequency domain transformation of the preset region in the second predicted image, and the second frequency domain image is the image obtained by frequency domain transformation of the preset region in the second label image.

[0040] In one embodiment, the processing module is further configured to:

[0041] Determine the first standard deviation of pixel values ​​for each local region in the second predicted image;

[0042] Determine the second standard deviation of pixel values ​​in each local region of the second labeled image;

[0043] The location of the preset region is determined based on the difference between each of the first standard deviations and the corresponding second standard deviations.

[0044] Thirdly, an electronic device is provided, including a processor for executing a computer program stored in a memory to implement the image processing method as described in the first aspect above.

[0045] Fourthly, a chip system is provided, including a processor coupled to a memory, the processor including a central processing unit and an image processor, the central processing unit or the image processor executing a computer program or instructions stored in the memory to implement the image processing method as described in the first aspect above.

[0046] The image processor can be an image signal processor, a digital signal processor, or a neural network processor, etc.

[0047] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method as described in the first aspect above.

[0048] In a sixth aspect, a computer program product is provided that, when the computer program product is run on an electronic device, causes the electronic device to perform the image processing method described in the first aspect above.

[0049] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0050] Figure 1 A software architecture diagram of an electronic device provided in an embodiment of this application;

[0051] Figure 2 This is a photographic scene diagram provided in one embodiment of this application;

[0052] Figure 3 A schematic diagram of an image containing textured regions provided in an embodiment of this application;

[0053] Figure 4 A schematic diagram of an image containing pseudo-texture regions provided in an embodiment of this application;

[0054] Figure 5 A schematic diagram of the process for training a first image processing model according to an embodiment of this application;

[0055] Figure 6 A schematic diagram of the process for training a second image processing model according to an embodiment of this application;

[0056] Figure 7 A schematic diagram illustrating a method for determining the position of a pseudo-texture region in a second predicted image according to an embodiment of this application;

[0057] Figure 8 A schematic diagram of a mask for a pseudo-texture region provided in an embodiment of this application;

[0058] Figure 9 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0059] Figure 10 A schematic diagram of an image without pseudo-texture regions provided in an embodiment of this application;

[0060] Figure 11 A schematic diagram of an image containing pseudo-texture regions provided in an embodiment of this application;

[0061] Figure 12 A schematic diagram of an image fusion method provided in an embodiment of this application;

[0062] Figure 13 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0063] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0064] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0065] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0066] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0067] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0068] Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0069] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0070] For example, the electronic device described in this application embodiment may be a mobile phone, tablet computer, handheld computer, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, media player, wearable device, or other device that can be held / operated with one hand. This application embodiment does not impose any special limitations on the specific form / type of the electronic device. The above-mentioned electronic device includes, but is not limited to, devices equipped with... Devices running Harmony OS or other operating systems.

[0071] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.

[0072] Figure 1This is a software structure block diagram of an electronic device according to an embodiment of the present invention.

[0073] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, the Hardware Abstraction Layer (HAL), and the kernel layer.

[0074] The application layer can include a series of application packages.

[0075] like Figure 1 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0076] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0077] like Figure 1 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0078] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0079] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0080] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0081] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0082] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0083] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0084] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0085] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0086] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0087] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0088] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0089] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0090] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0091] A 2D graphics engine is a graphics engine for 2D drawing.

[0092] HAL (Hardware Access Layer) provides a unified hardware access interface and resource management, enabling upper-layer applications to be developed and run independently of specific hardware platforms. HAL includes interfaces for cameras, sensors, audio, etc.

[0093] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0094] The following example, using a photography scenario, illustrates the workflow of electronic device software and hardware.

[0095] When the touch sensor receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a single touch operation as an example, where the corresponding control is the camera application icon, the camera application calls the application framework layer's interface to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through the camera.

[0096] The image processing method provided in this application is applied to photography scenarios.

[0097] For example, such as Figure 2 As shown in (a), in one application scenario, the electronic device responds to a user's instruction to open a camera application by displaying a preview of the camera application on the display interface. Upon detecting a user's click on the "Take Photo" control 21, the electronic device acquires the raw image captured by the camera, performs image processing on the raw image to obtain the target image, and displays it on the display interface as shown in (a). Figure 2 The target image is shown in (b). The target image is a high-resolution image, with higher image clarity and resolution than the original image.

[0098] In another application scenario, when the electronic device displays a preview of a camera application on its screen, if it detects a user's instruction to select a preset shooting mode (e.g., high-resolution shooting mode), it performs image processing on the raw image captured by the camera to obtain the target image, which is then displayed on the preview screen. Alternatively, if the electronic device's camera application is in a preset shooting mode, in response to a shooting instruction, it acquires the raw image captured by the camera, performs image processing on the raw image to obtain the target image, and displays the target image on the screen.

[0099] In other application scenarios, electronic devices can also respond to a user's instruction to open a camera application and directly display the target image on the camera application's preview screen. Alternatively, while displaying the camera application's preview screen, the electronic device can process each frame of the original video in response to the user's recording instruction to obtain a target video composed of multiple target images, and then display the target video on the screen.

[0100] In this process, electronic devices use an image processing model to process the original image to obtain a target image. The target image has higher resolution and sharpness than the original image. However, due to the limited image processing capabilities of the image processing model, it loses image details during the extraction of details from the original image, resulting in pseudo-texture regions in the target image.

[0101] In this context, the texture region of an image refers to an area composed of recurring texture units, which can be lines, dots, or patterns of specific shapes. The occurrence of these texture units can be regular, random, or semi-regular. Within the texture region, the image's pixels or brightness exhibit regular variations. For example, Figure 3 In region 31 of the image shown, the stripes of the two types of pixels appear repeatedly; region 31 is the texture region.

[0102] Pseudo-textured regions refer to areas in the original image that have no texture but are identified as having texture in the target image. For example, such as... Figure 4 In the original image shown in (a), region 41 has no pixel or brightness variation and is not a texture region. Figure 4 The original image shown in (a) is processed to obtain the image as shown in Figure 1. Figure 4 The target image shown in (b) has a texture (i.e., a region where pixels exhibit regular changes) in region 42, which corresponds to region 41. Region 42 is the pseudo-texture region, and region 41 in the original image corresponding to region 42 is also called the pseudo-texture region.

[0103] To address the issue of pseudo-textures in images, methods such as increasing training samples during the training phase of the image processing model or adjusting the loss function for regions prone to pseudo-textures can be used to suppress them. However, these methods are only applicable to scenarios processing images similar to the training samples. In real-world photography scenarios, the shooting environment, shooting distance, and subject vary greatly, resulting in diverse types of original images. Consequently, the regions prone to pseudo-textures also differ. Therefore, image processing models trained using these methods are not suitable for real-world photography scenarios.

[0104] This application provides an image processing method. After acquiring an original image, the electronic device first determines whether a preset region exists in the original image. The preset region is a pseudo-texture region, meaning that after the second image is processed by a second image model, the texture features in the second image are inconsistent with the texture features in the original image. In other words, the preset region does not have texture in the original image but has texture in the second image. If the existence of the preset region is determined, the electronic device performs image fusion processing on the first image and the second image to obtain a target image. The first image is the image obtained after the first image processing model processes the original image. Since the second training sample used to train the second image processing model has a higher degree of image degradation than the first training sample used to train the first image processing model, the second image obtained using the second image processing model has better image quality and higher clarity than the original image. The first image obtained using the first image processing model has lower quality and weaker detail recovery ability than the second image, but retains more texture features and color space. That is, the first image retains more texture features and color space of the preset region. Therefore, the target image obtained by fusing the first image and the second image can retain more texture features and color space of the preset area while ensuring clarity, thereby suppressing the appearance of false textures in the preset area and improving the image quality of the target image.

[0105] The image processing method provided in the embodiments of this application will be described in detail below.

[0106] The image processing method provided in this application requires processing the original image using a first image processing model and a second image processing model. The training methods for the first and second image processing models are described first.

[0107] The first image processing model is trained based on the first training samples. Specifically, the first training samples include multiple image groups, each image group consisting of two images. The two images contain the same content but have different qualities. Image quality reflects the sharpness of the image; higher image quality results in higher sharpness, and lower image quality results in lower sharpness. One of the two images can be the initial image, and the other image can be obtained by degrading the initial image. Image degradation refers to blurring the image to obtain an image with lower sharpness. The two images can also be images obtained by taking pictures of the same object using different shooting modes, resulting in images with different sharpness. Each image group is input into the first classification model, and the parameters of the first classification model are optimized to obtain the first image processing model. The first classification model can be a neural network model.

[0108] Specifically, such as Figure 5As shown, in each image group of the first training sample, the low-resolution image is used as the first input image, and the high-resolution image is used as the first label image. The first classification model is used to process the first input image to obtain the first predicted image. The parameters of the first classification model are optimized based on the difference between the first predicted image and the first input image.

[0109] In one embodiment, the first pixel difference between the first predicted image and the corresponding first label image is used as the first loss function, and the parameters of the first classification model are optimized based on the loss function.

[0110] Specifically, the first loss function is determined based on the first difference between each corresponding pixel in the first predicted image and the corresponding first label image.

[0111] In one embodiment, by determining the first difference between each corresponding pixel in the first predicted image and the corresponding first label image, multiple first differences can be obtained, and the mean of the squares of the first differences is the first loss function.

[0112] In other embodiments, the first loss function may be the sum of the first differences, the mean of the first differences, or the sum of the squares of the first differences, etc.

[0113] The electronic device optimizes the parameters of the first classification model based on the values ​​of the first loss function corresponding to each image group of the first training sample. The parameters corresponding to the minimum value of the first loss function are the optimal parameters, and the first classification model with the optimal parameters is the first image processing model.

[0114] By using the first pixel difference between the first predicted image and the corresponding first label image as the first loss function to optimize the first classification model, it is possible to ensure that the image output by the first image processing model is close to the input image, and to retain richer texture features and color space of the input image.

[0115] The second image processing model is trained based on a second set of training samples, where the image degradation level is higher than that of the first training samples. Specifically, the second training samples include multiple image groups, each containing two images with identical content but different image quality (i.e., different sharpness). Compared to the image groups in the first training samples, the difference in sharpness between the two images in the second training sample groups is greater. For example, for any initial image, a first image degradation process is performed to obtain a first sample image, and a second image degradation process is performed to obtain a second sample image. The degradation level of the second image degradation process is higher than that of the first image degradation process; that is, the image degradation level of the second sample image is higher than that of the first sample image. The initial image and the first sample image are used as one image group of the first training samples, and the initial image and the second sample image are used as another image group of the second training samples. Each image group is input into the second classification model, and the parameters of the second classification model are optimized to obtain the second image processing model. The second classification model can be a neural network model.

[0116] Specifically, in each image group, the low-resolution image is used as the second input image, and the high-resolution image is used as the second label image. The second classification model is used to process the second input image to obtain the second predicted image. The parameters of the second classification model are optimized based on the difference between the second predicted image and the second input image. The optimized second classification model is the second image processing model.

[0117] In the second training sample, the difference in sharpness between the two images is greater, meaning the difference in sharpness between the second input image and the second label image is greater. Correspondingly, the difference in sharpness between the second input image and the second predicted image is greater, which can easily lead to the appearance of pseudo-texture regions in the second predicted image when the second classification model generates the second predicted image based on the second input image.

[0118] In one embodiment, such as Figure 6 As shown, the electronic device determines the pseudo-texture region in the second predicted image based on the difference between the second predicted image and the corresponding second label, and then determines the second loss function based on the second pixel difference between the second predicted image and the corresponding second label image, as well as the frequency domain difference of the pseudo-texture region. The frequency domain difference of the pseudo-texture region includes the difference in amplitude and the difference in phase between the first frequency domain image and the second frequency domain image. The first frequency domain image is the image obtained by frequency domain transformation of the pseudo-texture region in the second predicted image, and the second frequency domain image is the image obtained by frequency domain transformation of the pseudo-texture region in the second label image.

[0119] Specifically, the electronic device inputs the second input image into the second classification model to obtain the corresponding second predicted image. A first standard deviation of the pixel values ​​of each local region in the second predicted image is determined, as well as a second standard deviation of the pixel values ​​of each local region in the second label image corresponding to the second input image. For any image, including multiple local regions with N*N pixels, according to Formula 1:

[0120]

[0121] Determine the standard deviation of pixel values ​​for each local region. Here, i represents the pixel in the i-th row, and j represents the pixel in the j-th column.

[0122] Indicates the first Arrive at the Between, the Listed to number The pixel values ​​of pixels between columns, where Var represents the standard deviation.

[0123] Each local region in the second predicted image corresponds one-to-one with the position of each local region in the second labeled image. For the second predicted image, the first standard deviation of each local region is calculated according to Formula 1 above. For the second labeled image, the second standard deviation of each local region is calculated according to Formula 1 above.

[0124] Subsequently, the electronic device determines the location of regions with texture differences in the second label image and the second predicted image based on the differences between each first standard deviation and the corresponding second standard deviation. Specifically, the electronic device determines whether each difference is greater than a threshold. If the difference is greater than the threshold, the corresponding local region has texture differences in the second predicted image and the second label image; if the difference is less than the threshold, the corresponding local region does not have texture differences in the second predicted image and the second label image.

[0125] In one embodiment, after obtaining each first standard deviation and the corresponding second standard deviation, the electronic device uses Formula 2: The differences between each first standard deviation and its corresponding second standard deviation are normalized to a range of 0 to 1. σ represents the normalized difference. pred σ represents the first standard deviation. lobel Let C represent the second characterization difference, and C represent a constant.

[0126] After obtaining the normalized difference according to Formula 2, the pseudo-texture region is determined based on the relationship between the normalized difference and the threshold, thereby reducing the amount of computation.

[0127] After obtaining the locations of regions with texture differences, the electronic device sets the pixel values ​​of regions with texture differences to 1 and the pixel values ​​of regions without texture differences to 0 in the second predicted image, resulting in a binary image. Then, a morphological algorithm is used to process the binary image to obtain pseudo-textured regions. The morphological algorithm is used to extract specified shapes or features from the image, facilitating target recognition. For example, the morphological algorithm could be an erosion operation, which uses a custom structuring element (e.g., a rectangle or circle) to slide across the image. The intersection of the pixels in the image and the pixels in the structuring element is the pixel count of the image after the erosion operation. Erosion can smooth the edges of the image and reduce the number of pixels. Alternatively, the morphological algorithm could be a dilation operation, which uses a custom structuring element (e.g., a rectangle or circle) to slide across the image. The union of the pixels in the image and the pixels in the structuring element is the pixel count of the image after the dilation operation. Dilation can smooth the edges of the image and increase the number of pixels. Morphological algorithms can be hole-filling operations, which are used to identify holes and fill them in an image using structuring elements, thereby removing interfering pixels from the image. Here, a hole is a background region surrounded by a boundary that connects to foreground pixels.

[0128] For example, according to such Figure 7 The second predicted image shown in (a) and as shown in Figure 7 The second label image shown in (b) can be accessed as follows: Figure 7 The image shown in (c) has a region with a pixel value of 1, which is a pseudo-texture region. The location of the pseudo-texture region in the second predicted image can then be determined.

[0129] After obtaining the location of the pseudo-texture region, the pseudo-texture region in the second predicted image is transformed in the frequency domain to obtain the first frequency domain image, and the pseudo-texture region in the second label image is transformed in the frequency domain to obtain the second frequency domain image.

[0130] Frequency domain transformation refers to performing a two-dimensional Fourier transform on an image to convert its gray-level distribution function into a frequency distribution function. The frequency of an image represents the degree of drastic change in gray levels. After frequency domain transformation, the amplitude and phase of the image can be obtained.

[0131] For example, for any image F(u, v), after frequency domain transformation, we get F(u, v) = R(u, v) + I(u, v)i, where R(u, v) represents the amplitude, I(u, v) represents the phase, and u and c represent the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the image, respectively.

[0132] After obtaining the first frequency domain image and the second frequency domain image, the frequency domain difference of the pseudo-texture region is determined based on the difference in amplitude and the difference in phase between the first frequency domain image and the second frequency domain image. Then, the loss function is determined based on the second pixel difference between the second predicted image and the corresponding second label image, as well as the frequency domain difference of the pseudo-texture region.

[0133] The frequency domain difference of the pseudo-texture region can be the sum of a first difference and a second difference. The first difference includes the difference in amplitude between corresponding values ​​in the first and second frequency domain images, and the second difference includes the difference in phase between corresponding values ​​in the first and second frequency domain images. The second pixel difference between the second predicted image and the second labeled image is determined based on the second difference of corresponding pixels in the second predicted image and the second labeled image. In one embodiment, the second pixel difference between the second predicted image and the second labeled image is the mean of the squares of each second difference. In other embodiments, the second pixel difference between the second predicted image and the second labeled image can also be the sum of each second difference, the mean of each second difference, or the sum of the squares of each second difference.

[0134] In one embodiment, the second loss function Loss is Loss = λ1L2Loss + λ2SSIM Loss + λ3Freq1Loss, where λ1, λ2, and λ3 are weights, L2Loss represents the second pixel difference between the second predicted image and the corresponding second label image, Freq1Loss represents the frequency domain difference between the pseudo-texture regions corresponding to the second predicted image and the second label image, and SSIM Loss is the structural loss function, representing the texture difference between the second predicted image and the corresponding second input image.

[0135] Wherein, according to the formula Determine SSIM Loss, μ x μ represents the mean pixel value of the second predicted image. y σ represents the mean pixel value of the second input image. xy σ represents the covariance between the pixel values ​​of the second predicted image and the pixel values ​​of the second input image. x σ represents the variance of the pixel values ​​in the second predicted image. y C1 and C2 represent the variance of the pixel values ​​of the second input image.

[0136] In another embodiment, the second loss function Loss is

[0137] Loss = λ3L2Loss + λ4SSIM Loss + λ5Freq2Loss + λ6L3Loss, where λ3, λ4, λ5, and λ6 are weights. Freq2Loss represents the frequency domain difference between the third frequency domain image obtained after frequency domain transformation of the second prediction image and the fourth frequency domain image obtained after frequency domain transformation of the second label image; that is, it represents the difference in amplitude and phase between the corresponding values ​​of the third and fourth frequency domain images. For example, Freq2Loss includes the third and fourth differences. The third difference includes the difference in amplitude between the corresponding values ​​of the third and fourth frequency domain images, and the fourth difference includes the difference in phase between the corresponding values ​​of the third and fourth frequency domain images. It can be understood that the first frequency domain image is a part of the third frequency domain image, and the second frequency domain image is a part of the fourth frequency domain image. L3Loss represents the pixel difference between the pseudo-texture regions in the second prediction image and the pseudo-texture regions in the second label image. For example, a difference image is calculated between the second predicted image and the second label image. Each pixel in the difference image corresponds to a position, and for any given pixel, there are pixels at the same position in both the second predicted image and the second label image. For any pixel in the difference image, the pixel value is the difference between the pixel values ​​of the corresponding pixel in the second predicted image and the pixel value of the corresponding pixel in the second label image. After obtaining the difference image, the product of the difference image and the mask of the pseudo-texture region is L3Loss. For example, the mask of the pseudo-texture region is determined based on the location of the pseudo-texture region. For instance, after obtaining the pseudo-texture region, setting the pixel value of the pseudo-texture region to 0 and the pixel value of the region outside the pseudo-texture region to 1 results in the following image: Figure 8 The mask for the pseudo-texture region shown.

[0138] After determining the second loss function, the parameters of the second classification model are optimized based on the values ​​of the second loss function corresponding to each image group of the second training samples. The parameters corresponding to the minimum value of the second loss function are the optimal parameters, and the second classification model with the optimal parameters is the second image processing model.

[0139] In the above embodiments, the second pixel difference between the second predicted image and the corresponding second label image is added to the second loss function to make the pixel value of the second predicted image close to the pixel value of the second label image. The texture difference between the second predicted image and the corresponding second input image is added to the second loss function to make the texture difference between the second predicted image and the second input image smaller. The frequency domain difference of the pseudo-texture region is added to the second loss function to make the amplitude and phase of the second predicted image close to the amplitude and phase of the second label image. This effectively improves the overall image restoration ability of the second image processing model and the image restoration ability in the pseudo-texture region, thereby improving the detail and clarity of the image output by the second image processing model.

[0140] In one embodiment, the first classification model and the second classification model have the same model structure. The first classification model can be trained first based on the first loss function and the first training samples to obtain the first image processing model. Then, the parameters of the first image processing model are used as the initial parameters of the second classification model, and the second classification model is trained based on the second loss function and the second training samples. Thus, the second image processing model can be obtained by adjusting the parameters based on the first image processing model, thereby improving the efficiency of training the second image processing model.

[0141] Because the second input image and the second label image have a greater difference in sharpness, the second image processing model can output a sharper image than the input image. Therefore, compared to the first image processing model, the image processed by the second image processing model has a higher sharpness. That is, both the first and second image processing models can perform image enhancement, but the first model's enhancement is weaker, while the second model's enhancement capability is stronger. The first image processing model produces a smoother image, although its sharpness is lower, it retains the rich texture features and color space of the input image and is less prone to pseudo-texture areas. The second image processing model can extract more refined features from the high-frequency texture details of the input image, resulting in a sharper image. However, while enhancing high-frequency texture details, it is more prone to producing pseudo-texture areas.

[0142] like Figure 9 As shown, an embodiment of this application provides an image processing method including:

[0143] S901: In response to the first instruction, acquire the raw image captured by the camera of the electronic device.

[0144] In one embodiment, such as Figure 2 As shown, the first instruction is a photo-taking instruction, and the original image is an unprocessed, low-resolution image captured by the camera. In other embodiments, the first instruction may also be a video recording instruction or an instruction to open a camera application.

[0145] S902: If it is determined that a preset region exists in the original image, image fusion processing is performed on the first image and the second image to obtain the target image; the first image is the image obtained after processing the original image by the first image processing model, and the second image is the image obtained after processing the original image by the second image processing model; the texture features of the preset region in the second image are inconsistent with the texture features in the original image.

[0146] Specifically, the original image is input into a first image processing model to obtain a first image output by the first image processing model, and the original image is input into a second image processing model to obtain a second image output by the second image processing model. The second image has higher clarity than the first image.

[0147] like Figure 3 and Figure 4 As shown, the location of the preset region in the original image is the same as the location of the preset region in the second image. The preset region is a pseudo-texture region. The electronic device determines whether there is a region with inconsistent texture features in the second image and the original image based on the pixel difference between the second image and the original image. This region is the preset region.

[0148] In one embodiment, the electronic device determines whether a preset region exists in the original image based on the pixel differences between corresponding local regions in the second image and the original image. The original image is composed of multiple local regions with a pixel count of N*N, where N is a positive integer and its value can be set according to actual needs. The local regions in the original image correspond one-to-one with the local regions in the second image.

[0149] In one embodiment, the specific method for determining whether a preset region exists in the original image is as follows. The electronic device first determines a difference image between the second image and the original image. The difference image represents the pixel difference between corresponding pixels in the second image and the original image. Specifically, the second image includes multiple pixels, each pixel corresponding to a position. There are two pixels with the same position in the original image and the second image. The difference between the pixel values ​​of the two pixels with the same position is the pixel value of the pixel at the same position in the difference image.

[0150] After obtaining the difference image, the electronic device determines the variance of the pixel values ​​in each local region of the difference image. For example, according to Formula 3: Determine the standard deviation of pixel values ​​in each local region of the difference image. Here, i represents the pixel in the i-th row, and j represents the pixel in the j-th column. Indicates the first Arrive at the Between, the Listed to number The pixel values ​​between columns are represented by Var, which indicates the standard deviation. The square of the standard deviation is the corresponding variance.

[0151] For each local region, if the variance is greater than a threshold, the texture features of that local region are determined to be inconsistent between the original image and the second image, i.e., a texture difference exists. If the variance is less than the threshold, the texture features of that local region are determined to be consistent between the original image and the second image, i.e., no texture difference exists. Local regions with inconsistent texture features between the original image and the second image are the local regions of pseudo-texture in the original image. For example, such as... Figure 10 (a) and Figure 10 In the image shown in (b), if each local region of the image is a flat region or a regular, clearly defined structure, and the variance of each local region in the corresponding difference image is less than a threshold, then there are no pseudo-texture regions. Figure 11 (a) and Figure 11 In the image shown in (b), if there is texture or unclear texture edges, and the variance of each local region of the corresponding difference image is greater than the threshold, then there is a pseudo-texture region.

[0152] Correspondingly, if there is a local region with a variance greater than the threshold, it is determined that the preset region exists in the original image; if there is no local region with a variance greater than the threshold, it is determined that the preset region does not exist in the original image.

[0153] If a preset region is determined to exist in the original image, the electronic device performs a fusion process on the first image and the second image to obtain the target image.

[0154] In one embodiment, fusing the first image and the second image to obtain a target image specifically includes: generating a first image to be fused based on the location of a preset region and the first image, the first image to be fused being used to characterize the pixel information of the preset region; generating a second image to be fused based on the location of the preset region and the second image, the second image to be fused being used to characterize the pixel information of the region outside the preset region; and performing image fusion processing on the first image to be fused and the second image to be fused to obtain the target image.

[0155] Specifically, after determining the variance corresponding to each layout region, in the difference image, the pixel values ​​of local regions with variance greater than a threshold are set to 1, and the pixel values ​​of local regions with variance less than a threshold are set to 0, resulting in a binary image. This binary image is then processed using a morphological algorithm to obtain the processed image. In the processed image, the pixel values ​​of the pseudo-texture regions are 1, and the pixel values ​​of regions outside the pseudo-texture regions are 0. Therefore, the processed image is also called a pseudo-texture region image. Inverting the pixel values ​​in the pseudo-texture region image—that is, setting the pixel value of a pixel with a value of 1 to 0 and the pixel value of a pixel with a value of 0 to 1—results in the mask for the pseudo-texture region.

[0156] like Figure 12As shown, image a represents the original image, image b represents the first image, image c represents the second image, image d represents the pseudo-texture region image, and image e represents the mask of the pseudo-texture region. Multiplying the first image and the pseudo-texture region image yields the first fused image. Since the pixel value of the pseudo-texture region in the pseudo-texture image is 1, and the pixel value of the region outside the pseudo-texture region is 0, the pixel value of the region outside the pseudo-texture region in the first fused image is 0, retaining only the pixel values ​​of the pseudo-texture region in the first image. Multiplying the second image and the mask of the pseudo-texture region yields the second fused image. Since the pixel value of the pseudo-texture region in the mask is 0, and the pixel value of the region outside the pseudo-texture region is 1, the pixel value of the pseudo-texture region in the second fused image is 0, retaining only the pixel values ​​of the region outside the pseudo-texture region in the second image. By fusing the first and second images, the complete target image f is obtained, and the preset region in the target image is obtained from the first image, retaining more texture features, thus eliminating the pseudo-texture region in the target image.

[0157] In other embodiments, after determining the location of the preset region, the preset region can be segmented from the first image, and the preset region in the second image can be replaced by the preset region segmented from the first image to obtain the target image.

[0158] S903: Output the target image.

[0159] Specifically, if the first instruction is a photo-taking instruction, the electronic device processes the original image and then displays the target image on the display interface. If the first instruction is a command to open a camera application, the electronic device opens the camera application and then displays the target image obtained after processing the original image on the preview interface.

[0160] In one embodiment, the image processing method flow is as follows: Figure 13 As shown, the electronic device inputs the original image into a second image processing model. Based on the pixel differences between the second and original images, it determines whether a preset region exists in the original image. If the preset region exists, the first and second images are fused to obtain the target image. If the preset region does not exist, the second image is used as the target image and output as the target image. This approach avoids false textures in the target image while still obtaining a high-resolution one when the preset region exists in the original image, and achieves an even higher-resolution target image when the preset region does not exist. Furthermore, using different image processing methods to obtain the target image based on the presence or absence of the preset region reduces the computational load of post-processing and improves the overall performance of the algorithm.

[0161] In the above embodiments, after acquiring the original image, the electronic device determines whether a preset region exists in the original image. If the preset region exists, the electronic device inputs the original image into a first image processing model to obtain a first image, and inputs the original image into a second image processing model to obtain a second image. The first and second images are then fused to obtain the target image. Since the second image has better image quality and higher clarity than the original image, and the first image retains more texture features and color space than the second image, the target image obtained by fusing the first and second images can retain more texture features and color space in the preset region while maintaining clarity. This can suppress the appearance of false textures in the preset region, thereby improving the image quality of the target image.

[0162] In one embodiment, the computer program implementing the above image processing method is stored in the HAL layer. After the data of the original image captured by the camera is uploaded to the HAL layer, the processor calls the computer program implementing the above image processing method to perform image processing on the data of the original image, and uploads the processed image to the application framework layer for further processing.

[0163] The processor can be a central processing unit (CPU) of an electronic device or an image signal processor (ISP).

[0164] In another embodiment, the computer program implementing the above image processing method may also be stored in the ISP of the electronic device or in a designated memory on the electronic device.

[0165] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0166] For example, Figure 14 A schematic diagram of an electronic device 100 is shown.

[0167] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0168] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0169] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0170] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0171] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0172] In some embodiments, processor 110 may include one or more interfaces.

[0173] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0174] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0175] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0176] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0177] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0178] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0179] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage functions.

[0180] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0181] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0182] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0183] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0184] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0185] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0186] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0188] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0189] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, executed in an electronic device, characterized in that, The method includes: In response to the first instruction, the original image captured by the camera of the electronic device is acquired; If a preset region is determined to exist in the original image, image fusion processing is performed on the first image and the second image to obtain the target image; the first image is the image obtained after processing the original image by a first image processing model, and the second image is the image obtained after processing the original image by a second image processing model; the first image processing model is trained based on a first training sample, and the second image processing model is trained based on a second training sample, wherein the image degradation degree of the second training sample is higher than that of the first training sample; the texture features of the preset region in the second image are inconsistent with the texture features in the original image; Outputting the target image; the step of performing image fusion processing on the first image and the second image to obtain the target image includes: A first image to be fused is generated based on the location of the preset region and the first image. The first image to be fused is used to characterize the pixel information of the preset region. A second image to be fused is generated based on the location of the preset region and the second image. The second image to be fused is used to represent the pixel information of the region outside the preset region. The first image to be fused and the second image to be fused are subjected to image fusion processing to obtain the target image.

2. The method according to claim 1, characterized in that, Before performing image fusion processing on the first image and the second image, the method further includes: The existence of the preset region in the original image is determined based on the pixel differences between the corresponding local regions in the second image and the original image.

3. The method according to claim 2, characterized in that, The step of determining whether the preset region exists in the original image based on the pixel differences of corresponding local regions in the second image and the original image includes: Determine a difference image between the second image and the original image, wherein the difference image represents the pixel differences between corresponding pixels in the second image and the original image; Determine the variance of pixel values ​​in each local region of the difference image; If there is a local region with a variance greater than the threshold, it is determined that the preset region exists in the original image; If there is no local region with a variance greater than the threshold, it is determined that the preset region does not exist in the original image.

4. The method according to claim 1, characterized in that, The method further includes: If it is determined that the preset region does not exist in the original image, the second image is used as the target image.

5. The method according to any one of claims 1 to 4, characterized in that, Before acquiring the raw image captured by the camera of the electronic device, the method further includes: The first classification model is trained based on the first loss function and the first training samples to obtain the first image processing model. The first loss function is determined based on the first pixel difference between the first predicted image and the corresponding first label image output by the first classification model. The first pixel difference is the difference between the pixel values ​​of the corresponding pixels in the first predicted image and the first label image.

6. The method according to any one of claims 1 to 4, characterized in that, Before acquiring the raw image captured by the camera of the electronic device, the method further includes: The second classification model is trained using the second loss function and the second training samples to obtain the second image processing model. The second loss function is determined based on the second pixel difference between the second predicted image and the corresponding second label image output by the second classification model, as well as the frequency domain difference of the preset region. The second pixel difference is the difference in pixel values ​​between corresponding pixels in the second predicted image and the second label image. The frequency domain difference of the preset region includes the difference in amplitude and phase between the first frequency domain image and the second frequency domain image. The first frequency domain image is the image obtained by frequency domain transformation of the preset region in the second predicted image, and the second frequency domain image is the image obtained by frequency domain transformation of the preset region in the second label image.

7. The method according to claim 6, characterized in that, The method further includes: Determine the first standard deviation of pixel values ​​for each local region in the second predicted image; Determine the second standard deviation of pixel values ​​in each local region of the second labeled image; The location of the preset region is determined based on the difference between each of the first standard deviations and the corresponding second standard deviations.

8. An electronic device, characterized in that, Includes a processor for executing a computer program stored in a memory to implement the method as claimed in any one of claims 1 to 7.

9. A chip system, characterized in that, The device includes a processor coupled to a memory, the processor comprising a central processing unit and an image processor, the central processing unit or the image processor executing a computer program or instructions stored in the memory to implement the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images