Image processing method and electronic device

Abstract

The application discloses an image processing method and an electronic device, relates to the field of image processing, and can remove the reflection part in an image and improve the definition of the image. The image processing method comprises the following steps: performing resolution reduction processing on an original resolution to-be-processed image to obtain a first image with a first resolution and a second image with a second resolution; inputting the first image into an image processing model to obtain a first reflection-removed image; inputting the first reflection-removed image, the first image and the second image into a first traction up-sampling model to obtain a second reflection-removed image; and inputting the second reflection-removed image, the second image and the to-be-processed image into a second traction up-sampling model to obtain a target image.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method and an electronic device. Background Technology

[0002] With the widespread use of camera functions on mobile phones and other electronic devices, the requirements for image clarity in different shooting scenarios are becoming increasingly higher.

[0003] Currently, when shooting in locations such as travel, museums, exhibition halls, zoos, or offices, landscapes, animals, cultural relics, and paintings are often photographed through glass. This can lead to issues such as blurriness, faded colors, ghosting, and artifacts caused by reflections, reducing image clarity. Summary of the Invention

[0004] This application provides an image processing method and an electronic device that can remove reflective parts from an image and improve image clarity.

[0005] To achieve the objective, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an image processing method is provided. This method downscales an image to be processed from its original resolution to obtain a first image with a first resolution and a second image with a second resolution. The first resolution is less than the original resolution, and the second resolution is less than the original resolution but greater than the first resolution. The image to be processed includes a reflective portion and a translucent portion. The first image includes a first reflective portion and a first translucent portion. The second image includes a second reflective portion and a second translucent portion. The first image is input into an image processing model to obtain a first de-reflected image. The first de-reflected image includes the first translucent portion but does not include the first reflective portion, and its resolution is the first resolution. The first de-reflected image, the first image, and the second image are input into a first traction upsampling model to obtain a second de-reflected image. The second de-reflected image includes the second translucent portion but does not include the second reflective portion, and its resolution is the second resolution. The second de-reflected image, the second image, and the image to be processed are input into a second traction upsampling model to obtain a target image. The target image includes the translucent portion but does not include the reflective portion, and its resolution is the original resolution.

[0007] The aforementioned image processing method reduces the resolution of the original image to obtain a first image at a first resolution and a second image at a second resolution. Then, the first image is input into an image processing model to obtain a first de-reflective image with the first reflective portion removed. Next, a first pull upsampling is performed on the first de-reflective image, the first image, and the second image to obtain a second de-reflective image at a second resolution with the second reflective portion removed. Finally, a second pull upsampling is performed on the second de-reflective image, the second image, and the image to be processed to obtain a target image at the original resolution with the reflective portion removed. This image processing method can remove reflective portions based on a reduced-resolution image, reducing memory usage, saving processing time, and effectively removing reflective portions from the image, thus improving image clarity.

[0008] In one possible implementation of the first aspect, inputting a first image into an image processing model to obtain a first de-reflected image includes: encoding the first image to obtain a first feature; the first feature being used to characterize a first reflective portion and a first translucent portion of the first image; splitting the first feature to obtain a first reflective feature and a first translucent feature; the first reflective feature being used to characterize the first reflective portion of the first image, and the first translucent feature being used to characterize the first translucent portion of the first image; decoding the first translucent feature to obtain a first de-reflected image, and decoding the first reflective feature to obtain a first reflective image.

[0009] In this implementation, a first image at a first resolution is first encoded to obtain the first feature of the first image. Then, the first feature is decomposed to obtain the first reflective feature and the first translucent feature of the first image. Finally, the first reflective feature and the first translucent feature are decoded respectively to obtain the corresponding first reflective image and first dereflected image. Dereflection processing can be performed on a low-resolution first image to obtain a low-resolution first dereflected image that excludes the first reflective portion but includes the first translucent portion. This not only reduces memory usage and saves processing time but also removes reflective parts from the image, improving image clarity.

[0010] In one possible implementation of the first aspect, the training process of the image processing model includes: acquiring an original reflective image and an original dereflective image; using the original reflective image to characterize the reflective portion of the original image, and using the original dereflective image to characterize the translucent portion of the original image; using the original image to obtain the image by merging the original reflective image and the original dereflective image; inputting the original image into the image processing model to obtain the dereflective image and the reflective image; and training the image processing model based on the reflective image and the original reflective image, as well as the dereflective image and the original dereflective image.

[0011] In this implementation, the image processing model is trained based on the reflected image and the original reflected image, as well as the dereflected image and the original dereflected image. This allows the reflected image to become closer and closer to the original reflected image, and the dereflected image to become closer and closer to the original dereflected image. Finally, the image processing model can output the original reflected image and the original dereflected image, which separate the reflected and transparent parts.

[0012] In one possible implementation of the first aspect, inputting a first de-reflected image, a first image, and a second image into a first traction upsampling model to obtain a second de-reflected image includes: encoding the first de-reflected image, the first image, and the second image respectively to obtain first de-reflected image features, first image features, and second image features; the first de-reflected image features are used to characterize a first reflective portion of the first image, the first image features are used to characterize a first reflective portion and a first translucent portion of the first image, and the second image features are used to characterize a second reflective portion and a second translucent portion of the second image; based on the similarity between the first de-reflected image features and the first image features, obtaining first marker information corresponding to each pixel in the first image; wherein the first marker information is used to indicate whether a pixel in the first image is translucent or reflective; based on the first marker information and the second image features, performing a first filtering process to obtain a first filtering feature; the first filtering feature is used to characterize a second translucent portion of the second image; decoding the first filtering feature to obtain a first decoding feature; and obtaining the second de-reflected image based on the first decoding feature.

[0013] In this implementation, based on the first de-reflective image features and the similarity of the first image features, first marker information corresponding to each pixel in the first image is obtained; the first marker information is used to indicate whether the pixels in the first image are translucent or reflective. Based on this first marker information, the second reflective portion in the second image features is removed, and finally, a second de-reflective image with the second reflective portion removed is obtained at a second resolution. This image processing method uses a first traction upsampling model, which can not only remove the reflective portion but also improve the image resolution.

[0014] In one possible implementation of the first aspect, obtaining first labeling information corresponding to each pixel in the first image based on the similarity between the first de-reflected image features and the first image features includes: calculating the first similarity between each pixel in the first image and the first de-reflected image using a first dot-multiplication matrix based on the first de-reflected image features and the first image features; and labeling each pixel in the first image based on the first similarity to obtain the first labeling information corresponding to each pixel in the first image.

[0015] In this implementation, since the first de-reflective image feature includes the first reflective portion of the first image but does not include the first translucent portion, and the first image feature includes both the first reflective and first translucent portions, the first dot product matrix is ​​used to calculate a first similarity measure of the similarity between the first de-reflective image and the first image in both the first reflective and first translucent portions. Therefore, by calculating the similarity between the first de-reflective image feature and the first image feature using the first dot product matrix, a first similarity can be obtained between each pixel of the first image and the first de-reflective image. Based on the first similarity, each pixel of the first image is labeled, resulting in first labeling information corresponding to each pixel in the first image, indicating whether a pixel in the first image is translucent or reflective.

[0016] In one possible implementation of the first aspect, a first filtering process is performed based on the first label information and the second image features to obtain the first filtering features, including: performing a first upsampling on the first image features based on the first label information to obtain the first image reconstruction features; the resolution of the first image reconstruction features is a second resolution; inputting the first image reconstruction features and the second image features into a first linear layer to obtain the first filtering features; the first linear layer is used to retain the second light-transmitting portion of the second image and remove the second reflective portion of the second image.

[0017] In this implementation, the resolution of the first image feature is a first resolution. The first labeling information is used to indicate whether each pixel in the first image is translucent or reflective. Based on the first labeling information, the first image feature can be first upsampled to obtain a first image reconstruction feature with a second resolution. Since the first image reconstruction feature is a high-resolution first image feature, the positions and proportions of translucent and reflective pixels in the first image reconstruction feature are exactly the same as those in the first image feature. Furthermore, the first linear layer is used to retain data with high similarity and remove data with low similarity. By inputting the first image reconstruction feature and the second image feature into the first linear layer for comparison and filtering, reflective pixels in the second image feature can be removed, resulting in a first filtered feature including translucent pixels. Therefore, the first image reconstruction feature can be obtained by first upsampling the first image feature based on the first labeling information using the first linear layer, and then the second image feature can be segmented based on the first image reconstruction feature to obtain a first filtered feature in the second image feature that only includes translucent pixels.

[0018] In one possible implementation of the first aspect, a second de-reflected image, a second image, and an image to be processed are input into a second traction upsampling model to obtain a target image. This includes: encoding the second de-reflected image, the second image, and the image to be processed to obtain second de-reflected image features, second image features, and image features to be processed; the second de-reflected image features characterize the second reflective portion of the second image, the second image features characterize the second reflective portion and the second translucent portion of the second image, and the image features to be processed characterize the reflective portion and translucent portion of the image to be processed; based on the similarity between the second de-reflected image features and the second image features, second labeling information corresponding to each pixel in the second image is obtained; the second labeling information indicates whether a pixel in the second image is translucent or reflective; based on the second labeling information and the image features to be processed, a second filtering process is performed to obtain second filtering features; the second filtering features characterize the translucent portion of the image to be processed; the second filtering features are decoded to obtain second decoding features; and based on the second decoding features, the target image is obtained.

[0019] In this implementation, based on the second de-reflective image features and the similarity of the second image features, second labeling information is obtained for each pixel in the second image. This second labeling information includes indicators to show whether a pixel in the second image is translucent or reflective. Based on this second labeling information, the reflective portion of the image features to be processed is removed, finally obtaining the target image with the reflective portion removed at the original resolution. This image processing method employs a second traction upsampling model, which can not only remove the reflective portion but also improve the image resolution, obtaining an image at the original resolution.

[0020] In one possible implementation of the first aspect, second labeling information corresponding to each pixel in the second image is obtained based on the similarity between the second de-reflected image features and the second image features, including: calculating the second similarity between each pixel in the second image and the second de-reflected image using a second dot-multiplication matrix based on the second de-reflected image features and the second image features; and labeling each pixel in the second image based on the second similarity to obtain the second labeling information corresponding to each pixel in the second image.

[0021] In this implementation, since the second de-reflective image feature includes the second reflective portion of the second image but does not include the second translucent portion, and the second image feature includes the second reflective and translucent portions, the second dot product matrix is ​​used to measure the similarity between the second de-reflective image and the second image in both the second reflective and translucent portions. Therefore, by calculating the similarity between the second de-reflective image feature and the second image feature using the second dot product matrix, a second similarity score between each pixel of the second image and the second de-reflective image can be obtained. Based on this second similarity score, each pixel of the second image is labeled, resulting in second labeling information corresponding to each pixel in the second image, indicating whether a pixel in the second image is translucent or reflective.

[0022] In one possible implementation of the first aspect, a second filtering process is performed based on the second labeling information and the features of the image to be processed to obtain a second filtering feature, including: performing a second upsampling on the second image features based on the second labeling information to obtain a second image reconstruction feature; the resolution of the second image reconstruction feature is the original resolution; inputting the second image reconstruction feature and the features of the image to be processed into a second linear layer to obtain the second filtering feature; the second linear layer is used to retain the light-transmitting part of the image to be processed and remove the reflective part of the image to be processed.

[0023] In this implementation, the resolution of the second image feature is a second resolution. The second labeling information is used to indicate whether each pixel in the second image is translucent or reflective. Based on the second labeling information, the second image feature can be upsampled a second time to obtain a second image reconstruction feature with the original resolution. Since the second image reconstruction feature is a high-resolution second image feature, the positions and proportions of translucent and reflective pixels in the second image reconstruction feature are exactly the same as those in the original image feature. Furthermore, the second linear layer is used to retain data with high similarity and remove data with low similarity. By inputting the second image reconstruction feature and the image feature to be processed into the second linear layer for comparison and filtering, reflective pixels in the image feature to be processed can be removed, resulting in a second filtered feature that includes translucent pixels. Therefore, the second image reconstruction feature can be obtained by first upsampling the second image feature based on the second labeling information using the second linear layer. Then, based on the second image reconstruction feature, the image feature to be processed can be segmented to obtain a second filtered feature that includes only translucent pixels in the image feature to be processed.

[0024] In a second aspect, an electronic device is provided, comprising: a memory and one or more processors, the memory being coupled to the processors; wherein the memory is used to store instructions executable by the one or more processors, the memory storing computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the image processing method of the first aspect and any embodiment thereof.

[0025] Thirdly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform an image processing method as described in the first aspect and any embodiment thereof.

[0026] Fourthly, a computer program product is provided that, when run on an electronic device, causes the electronic device to perform an image processing method as described in the first aspect and any embodiment thereof.

[0027] The technical effects of the design methods in the second, third, and fourth aspects can be found in the technical effects of the different design methods in the first aspect, and will not be repeated here. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a possible hardware structure of an electronic device provided in an embodiment of this application;

[0029] Figure 2 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0030] Figure 3 A schematic diagram showing a reflective portion outside the depth of field, provided as an embodiment of this application;

[0031] Figure 4 A schematic diagram of an image to be processed where the reflective portion is outside the depth of field, provided as an embodiment of this application;

[0032] Figure 5 A flowchart illustrating a method for training a de-reflection model within the depth of field, as provided in an embodiment of this application;

[0033] Figure 6 A schematic diagram of an image to be processed with the reflective portion within the depth of field and a schematic diagram of the reflective portion outside the depth of field, provided as an embodiment of this application.

[0034] Figure 7 A schematic diagram illustrating an image processing method provided in an embodiment of this application;

[0035] Figure 8This is a schematic diagram of the structure of an image processing model provided in an embodiment of this application;

[0036] Figure 9 A flowchart illustrating the training process of an image processing model provided in this application embodiment;

[0037] Figure 10 A schematic diagram of the structure of a first traction upsampling model provided in an embodiment of this application;

[0038] Figure 11 This is a schematic diagram of a first filtering process provided in an embodiment of this application;

[0039] Figure 12 A schematic diagram of the structure of a second traction upsampling model provided in an embodiment of this application;

[0040] Figure 13 This is a schematic diagram of a second filtering process provided in an embodiment of this application;

[0041] Figure 14 A schematic diagram of a target image provided in an embodiment of this application;

[0042] Figure 15 This is a schematic diagram of a possible software structure of an electronic device provided in an embodiment of this application;

[0043] Figure 16 This is a schematic diagram illustrating the interaction between the service side and the cloud side, provided as an embodiment of this application.

[0044] Figure 17 This is a flowchart illustrating a specific process of interaction between the business side and the cloud side, as provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the words "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding. The terms "coupling" and "connection" involved in the embodiments of this application should be interpreted broadly. For example, it can refer to a physical direct connection or an indirect connection implemented through electronic devices, such as a connection implemented through resistors, inductors, capacitors, or other electronic devices.

[0046] Removing reflections from images is an important image processing task with wide applications in fields such as photography, medical imaging, and autonomous driving.

[0047] Down-resolution processing is the process of converting a high-resolution image into a low-resolution image.

[0048] Traction upsampling is a signal processing technique used to increase the sampling rate or image resolution while preserving as much of the original data's features as possible. It primarily uses interpolation and signal reconstruction techniques to estimate a high-resolution signal from a low-resolution signal.

[0049] Image coding is the process of converting analog images into digital signal sequences to facilitate digital processing, transmission, and storage, improve the compression rate of image data, reduce storage and transmission costs, maintain image quality, and minimize information loss.

[0050] Feature decoding is the process of decoding compressed low-dimensional features back into a high-dimensional space for image reconstruction.

[0051] This application provides an electronic device with a display function. The electronic device can be mobile or fixed. It can be deployed on land (e.g., indoors or outdoors, handheld or vehicle-mounted), on water (e.g., on ships), or in the air (e.g., airplanes, balloons, and satellites). This electronic device can be referred to as user equipment (UE), access terminal, terminal unit, subscriber unit, terminal station, mobile station (MS), mobile station, terminal agent, or terminal device. For example, it can be a mobile phone, tablet computer, laptop computer, smart bracelet, smart screen, smartwatch, virtual reality (VR) device, augmented reality (AR) device, terminal in industrial control, terminal in self-driving, terminal in remote medical care, terminal in smart grid, terminal in transportation safety, terminal in smart city, terminal in smart home, etc. This application does not limit the specific type and structure of the electronic device. The following describes one possible structure of the electronic device.

[0052] Taking mobile phones as an example, the attached document... Figure 1A possible structure of an electronic device 100 is shown. The electronic device 100 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a power management module 240, a battery 241, a wireless charging coil 242, a mobile communication module 250, a wireless communication module 260, antennas 251 and 261, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. Optionally, in some embodiments, it may also include an audio digital signal processor (ADSP) 243.

[0053] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0054] Processor 210 may include one or more processing units, such as: a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a central processing unit (CPU), an application processor (AP), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, and a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. For example, processor 210 may be an application processor (AP). Alternatively, processor 210 may be integrated into a system-on-chip (SoC). Or, processor 210 may be integrated into an integrated circuit (IC) chip. The processor 210 may include an analog front end (AFE) and a micro-controller unit (MCU) in an IC chip.

[0055] The processor 210 may also include a memory for storing computer instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store computer instructions or data that the processor 210 has just used or that are being used repeatedly. If the processor 210 needs to use the same computer instructions or data again, it can retrieve them directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves system efficiency.

[0056] In some embodiments, the processor 210 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, etc.

[0057] In some embodiments, the processor may be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. The processor described above may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0058] The ADSP 243 can be coupled to the audio module 270 and the sensor module 280. The ADSP 243 can process audio signals and sensor data. Even when the processor 210 is in sleep mode, the ADSP 243 can remain operational, thereby reducing the power consumption of the electronic device 100.

[0059] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or a combination of multiple interface connection methods.

[0060] The external memory interface 220 can be used to connect an external memory card, thereby expanding the storage capacity of the electronic device 100. The external memory card communicates with the processor 210 through the external memory interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0061] Internal memory 221 can be used to store computer executable program code, which includes computer instructions. Processor 210 executes various functional applications and data processing of electronic device 100 by running the computer instructions stored in internal memory 221. In addition, internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0062] In this embodiment of the application, when computer instructions are executed by processor 210, electronic device 100 performs the image processing method of this embodiment of the application.

[0063] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0064] The electronic device 100 can implement its shooting function through an ISP, a camera 293, a video codec, a GPU, a display 294, and an application processor. The ISP is used to process data fed back from the camera 293. In some embodiments, the ISP can be located within the camera 293. The camera 293 is used to capture still images or videos. In some embodiments, the electronic device 100 may include one or N cameras 293, where N is a positive integer greater than 1.

[0065] Electronic device 100 can implement display functions through a GPU, display screen 294, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute computer instructions to generate or modify display information.

[0066] The display screen 294 is used to display images, applications, videos, etc. The display screen 294 includes a display panel. In some embodiments, the electronic device 100 may include one or more display screens 294.

[0067] In this embodiment of the application, the captured image is processed using the image processing method described in this embodiment of the application, and then the processed image is displayed on the display screen 294.

[0068] In related technologies, with the widespread adoption of 100% camera functions on mobile phones and other electronic devices, the requirements for image clarity in different shooting scenarios are becoming increasingly stringent. Currently, in shooting scenarios such as travel, museums, exhibition halls, zoos, or offices, landscapes, animals, cultural relics, and paintings are often photographed through glass. This can lead to issues such as blurriness, faded colors, ghosting, and artifacts caused by reflections, thus reducing image clarity.

[0069] Therefore, this application provides an image processing method that reduces the resolution of an image to be processed from its original resolution to obtain a first image at a first resolution and a second image at a second resolution. Then, the first image is input into an image processing model to obtain a first de-reflective image with the reflective parts removed. Next, the first de-reflective image, the first image, the second image, and the image to be processed are subjected to two rounds of upsampling to obtain a target image at its original resolution with the reflective parts removed. This image processing method can remove reflective parts from an image and improve image clarity.

[0070] The image processing method provided in this application embodiment can be executed by a processor as shown in the appendix. Figure 1 The processor 210 in the electronic device 100 shown can also be as attached. Figure 1The GPU in the processor 210 of the electronic device 100 shown.

[0071] This application uses an electronic device 100 including an image processing method as an example to specifically describe the image processing method of this application. For example, see the attached diagram. Figure 2 As shown, an image processing method provided in this application embodiment may include steps S201-S204:

[0072] Step S201: Reduce the resolution of the image to be processed from its original resolution to obtain a first image with a first resolution and a second image with a second resolution.

[0073] The first resolution is smaller than the original resolution, and the second resolution is smaller than the original resolution but larger than the first resolution.

[0074] The image to be processed can be acquired during the shooting process or obtained from an image library. In one possible implementation, the source of the image to be processed can be acquired during the shooting process of the camera in the electronic device, or it can be an image library in the electronic device or online network, etc. The image to be processed can also be an image downloaded from the network or received from other electronic devices. The source of the image to be processed is not limited in the embodiments of this application.

[0075] The image to be processed includes a reflective portion and a translucent portion. Since the first image at a first resolution and the second image at a second resolution are obtained by downscaling the original image to be processed, the first image, like the image to be processed, also includes a first reflective portion and a first translucent portion. The second image, like the image to be processed, also includes a second reflective portion and a second translucent portion.

[0076] Since the first reflective portion and the second reflective portion are the same as the reflective portion, the following embodiments of this application will use the reflective portion as an example to specifically describe the reflective portion of this application.

[0077] The reflective part includes two situations: 1) the reflective object is outside the depth of field; 2) the reflective object is within the depth of field.

[0078] In the first case: when the reflective object is outside the depth of field, for example, as shown in the attached... Figure 3 As shown, when a user photographs a person outside the glass within the depth of field through the glass, because the distance between the object on the user's side and the glass is greater than the distance of the back depth of field, the virtual imaging position of the reflected light from the object is outside the depth of field. This results in the corresponding object's position in the captured image appearing as a reflected portion, including blur, faded color, and ghosting. For example, see the attached image. Figure 4As shown, in the captured image to be processed, since the position of the human hair corresponds to the virtual imaging position of the object's reflected light, and the virtual imaging position of the object's reflected light is outside the depth of field, a reflected part including blur, light color and ghosting appears at the human hair.

[0079] In the second case: when the reflective object is within the depth of field, for example, as shown in the attached... Figure 5 As shown, when a user photographs a person outside the glass within the depth of field, because the distance between the object on the user's side and the glass is greater than or equal to the foreground depth of field distance and less than or equal to the background depth of field distance, the virtual imaging position of the reflected light from the object is within the depth of field. This results in a clear artifact appearing in the image where the corresponding object is not part of the shooting scene, i.e., the reflected portion. An example is shown in the attached image. Figure 6 As shown, in the captured image to be processed, since the position above the human body corresponds to the virtual imaging position of the light reflected from the lamp tube, and the virtual imaging position of the light reflected from the lamp tube is within the depth of field, a clear lamp tube appears above the human body. Since the user captured an outdoor landscape photo of people, and the lamp tube is indoors and not part of the shooting scene, a reflective part including artifacts appears in the captured image.

[0080] Based on the descriptions of the reflected parts in cases 1) and 2) above, it can be understood that the reflected parts include: blurriness, pale color, ghosting, and artifacts. This allows for better differentiation of the reflected parts, thus leading to better removal of them.

[0081] Since the first reflective part and the second reflective part are the same as the reflective part, the first reflective part and the second reflective part also include: blur, pale color, ghosting and artifact.

[0082] In one possible implementation, the down-resolution processing can be downsampling or multi-scale pyramid processing. The embodiments of this application do not limit the form of down-resolution processing.

[0083] In one possible implementation, the original resolution can be 3072*1024 or 1024*1024. This application does not limit the size of the original resolution.

[0084] In one possible implementation, the first resolution can be 1 / 4 or 1 / 8 of the original resolution. The embodiments of this application do not limit the size of the first resolution.

[0085] In one possible implementation, the second resolution can be 1 / 2 or 1 / 3 of the original resolution. The embodiments of this application do not limit the size of the second resolution.

[0086] For example, see attached Figure 7 As shown, the original image to be processed with a resolution of (768*4, 256*4) is reduced in resolution by a factor of 4 to obtain a first image with a first resolution of (768, 256). The original image to be processed with a resolution of (768*4, 256*4) is reduced in resolution by a factor of 2 to obtain a second image with a second resolution of (768*2, 256*2).

[0087] The image to be processed includes reflective and translucent portions, indicating that the image has been contaminated. In related technologies, directly removing reflections from a high-resolution image results in excessive memory usage and long latency. In this embodiment, the high-resolution image is first down-resolution processed, and then reflection removal is performed based on the down-resolution image, which reduces memory usage and saves processing time.

[0088] Step S202: Input the first image into the image processing model to obtain the first de-reflected image.

[0089] For example, see attached Figure 7 As shown, by inputting the first image with a first resolution of (768,256) into the image processing model, a first de-reflected image with a first resolution of (768,256) can be obtained.

[0090] For example, see attached Figure 8 As shown, the image processing model includes three modules: a detector, an image segmentation module, and a decoder. Inputting a first image into the image processing model to obtain a first de-reflected image involves: first, encoding the first image in the detector to obtain first features characterizing the first reflective portion and the first translucent portion of the first image; next, segmenting the first features in the image segmentation module to obtain first reflective features characterizing the first reflective portion and first translucent features characterizing the first translucent portion of the first image; and finally, decoding the first reflective features in the decoder to obtain the first reflective image, and decoding the first translucent features in the decoder to obtain the first de-reflected image.

[0091] In one possible implementation, the image processing model can be a U-Net convolutional neural network model, or an FCN or SegNet convolutional neural network model. This application does not limit the type of image processing model.

[0092] In one possible implementation, the image segmentation module can be a U-Net convolutional neural network module, or an FCN or SegNet convolutional neural network module. The type of image segmentation module is not limited in the embodiments of this application.

[0093] In one possible implementation, the detector can be a transformer detector or a centerNet detector; the type of detector is not limited in this application embodiment.

[0094] Since the first image is obtained by downscaling the original resolution image to be processed, and the image to be processed includes reflective and translucent parts, the first image, like the image to be processed, includes a first reflective part and a first translucent part, only with its resolution reduced to the first resolution. The first image is input into a detector for encoding to obtain first features characterizing the first reflective and first translucent parts of the first image. Therefore, the first features include features of the first reflective part and features of the first translucent part. Next, the first features are input into an image segmentation module for segmentation to obtain the features of the first reflective part (i.e., the first reflective feature) and the features of the first translucent part (i.e., the first translucent feature). Finally, the first reflective feature is input into a decoder for decoding to obtain a first reflective image that includes the first reflective part but does not include the first translucent part. The first translucent feature is input into a decoder for decoding to obtain a first dereflective image that includes the first translucent part but does not include the first reflective part.

[0095] In this embodiment, the image processing model is a U-Net convolutional neural network model, and the detector is a transformer detector.

[0096] For example, see attached Figure 9 As shown, the training process of the image processing model includes: acquiring the original reflected image and the original dereflected image; the original reflected image is used to characterize the reflected part of the original image, and the original dereflected image is used to characterize the light-transmitting part of the original image; the original image is obtained by merging the original reflected image and the original dereflected image; the original image is input into the image processing model to obtain the dereflected image and the reflected image; the image processing model is trained based on the reflected image and the original reflected image, as well as the dereflected image and the original dereflected image.

[0097] The original reflected image is a given reflected image, and the original dereflected image is a given dereflected image. According to step S201, the reflected components include: blurriness, pale color, ghosting, and artifacts. It can be understood that the original reflected image only includes the blurred, pale color, ghosting, and artifact-laden reflected components, and is a given image containing only the reflected components. The original dereflected image is a given image with the reflected components removed.

[0098] In one possible implementation, the original reflected image and the original dereflected image can be obtained from a gallery or downloaded from the network. This application embodiment does not limit the source of the original reflected image and the original dereflected image.

[0099] The original reflected image and the original de-reflected image are merged to obtain the original image. This original image is then input into an image processing model to obtain both a de-reflected image and a reflected image. Based on the original reflected image and the reflected image output by the image processing model, as well as the original de-reflected image and the de-reflected image output by the same model, the image processing model is trained. This process allows the reflected image output by the image processing model to increasingly resemble the original reflected image, and the de-reflected image output by the model to increasingly resemble the original de-reflected image. Consequently, the input image to the image processing model can output both realistic reflected and de-reflected images. Therefore, it is possible to better distinguish between reflective and translucent parts, ultimately allowing the image processing model to output separate de-reflected and reflected images.

[0100] The image processing method described in step S202 above involves inputting a first image at a first resolution (i.e., low resolution) into an image processing model, splitting the first image, removing the first reflective portion of the first image, and obtaining a first de-reflective image at a first resolution that includes the first translucent portion but does not include the first reflective portion. This not only reduces memory usage and saves time, but also removes reflective portions from the image, improving image clarity.

[0101] Step S203: Input the first de-reflected image, the first image, and the second image into the first traction upsampling model to obtain the second de-reflected image.

[0102] For example, see attached Figure 7 As shown, by inputting the first image with a first resolution of (768,256), the first de-reflected image with a first resolution of (768,256), and the second image with a second resolution of (768*2,256*2) into the first traction upsampling model, a second de-reflected image with a second resolution of (768*2,256*2) can be obtained.

[0103] Since the first de-reflected image has a first resolution and the second image has a second resolution, in order to obtain the image at the original resolution, the first de-reflected image with the first resolution needs to be upgraded to the second resolution.

[0104] In one possible implementation, the method for improving image resolution can be interpolation or upsampling. This application does not limit the type of method for improving image resolution.

[0105] In this embodiment, a traction upsampling method is used to improve the image resolution. Since the first de-reflected image at a first resolution is upsampled to a second resolution, according to the principle of the traction upsampling method, resolution is increased or decreased proportionally. Therefore, a second resolution image, which is a multiple of the first resolution, is needed for traction. In this embodiment, the first image has the first resolution. Therefore, it can be understood that the second resolution is twice the first resolution.

[0106] For example, see attached Figure 10 As shown, the first de-reflected image, the first image, and the second image are input into the first traction upsampling model to obtain the second de-reflected image. This includes: encoding the first de-reflected image, the first image, and the second image respectively to obtain first de-reflected image features, first image features, and second image features; the first de-reflected image features are used to characterize the first reflective portion of the first image, the first image features are used to characterize the first reflective portion and the first translucent portion of the first image, and the second image features are used to characterize the second reflective portion and the second translucent portion of the second image; based on the similarity between the first de-reflected image features and the first image features, first marker information corresponding to each pixel in the first image is obtained; wherein, the first marker information is used to indicate whether a pixel in the first image is translucent or reflective; based on the first marker and the second image features, a first filtering process is performed to obtain a first filtering feature; the first filtering feature is used to characterize the second translucent portion of the second image; the first filtering feature is decoded to obtain a first decoding feature; and based on the first decoding feature, the second de-reflected image is obtained.

[0107] In one possible implementation, the method for calculating similarity can be dot product matrix, Jaccard similarity, or cosine similarity, etc. The embodiments of this application do not limit the type of method for calculating similarity.

[0108] In one possible implementation, the filtering method can be a linear layer, a nonlinear neural network layer, or graph data filtering, etc. The embodiments of this application do not limit the type of filtering method.

[0109] In one possible implementation, based on the first de-reflected image features and the first image features, a first similarity between each pixel of the first image and the first de-reflected image is calculated using a first dot product matrix. Each pixel of the first image is then labeled based on the first similarity, resulting in first labeling information corresponding to each pixel in the first image.

[0110] In one possible implementation, based on the first label information corresponding to each pixel in the first image, the features of the first image are first upsampled to obtain the first image reconstruction features; the resolution of the first image reconstruction features is a second resolution. The first image reconstruction features and the second image features are input into a first linear layer to obtain the first filtering features; the first linear layer is used to retain the second translucent portion of the second image and remove the second reflective portion of the second image.

[0111] For example, see attached Figure 10 As shown, firstly, the first de-reflected image, the first image, and the second image are encoded to obtain first de-reflected image features, first image features, and second image features. The first de-reflected image features are used to characterize the first reflective portion of the first image, the first image features are used to characterize the first reflective portion and the first translucent portion of the first image, and the second image features are used to characterize the second reflective portion and the second translucent portion of the second image.

[0112] Secondly, based on the first de-reflective image features and the first image features at the same resolution (first resolution), the first similarity between each pixel of the first image and the first de-reflective image is calculated using the first dot-multiplication matrix. Since the first de-reflective image features do not include the first reflective portion but include the first translucent portion, and the first image features include the first reflective portion and the first translucent portion, in the first de-reflective image features and the first image features, the same portion is the first translucent portion, and the different portion is the first reflective portion. Since the pixels in the first image features that are similar to the first de-reflective image features are translucent pixels, and the pixels in the first image features that are unrelated to the first de-reflective image features are reflective pixels, each translucent pixel in the first image can be marked with a first value, and each reflective pixel in the first image can be marked with a second value. Therefore, each pixel of the first image can be marked based on the first similarity to obtain the first marking information corresponding to each pixel in the first image. Wherein, the first marking information is used to indicate whether a pixel in the first image is translucent or reflective. Each pixel corresponding to the first marking information of the first value is a translucent pixel, and each pixel corresponding to the first marking information of the second value is a reflective pixel. Therefore, based on the similarity between the first de-reflective image features and the first image features at the same resolution (first resolution), the first marking information used to indicate whether each pixel in the first image is translucent or reflective can be obtained. For example, the first similarity between each pixel of the first image and the first de-reflective image is calculated using the first dot product matrix. Since the first dot product matrix is ​​used to measure the similarity between the first de-reflective image and the first image in two directions, namely the first reflective portion and the first translucent portion, the translucent pixels in the first image features that are similar to the first de-reflective image features, and the reflective pixels in the first image features that are unrelated to the first de-reflective image features, can be obtained through the first dot product matrix.

[0113] Secondly, since the resolution of the first image feature is a first resolution, the first labeling information is used to indicate whether a pixel in the first image is translucent or reflective. The first labeling information for each translucent pixel is a first value, and the first labeling information for each reflective pixel is a second value. Based on the first labeling information corresponding to each pixel in the first image, the first image feature can be first upsampled to obtain a first image reconstruction feature with a second resolution. Since the first image reconstruction feature is a high-resolution first image feature, the positions and proportions of translucent and reflective pixels in the first image reconstruction feature are exactly the same as those in the first image feature. Furthermore, the first linear layer is used to retain data with high similarity and remove data with low similarity. By inputting the first image reconstruction feature and the second image feature into the first linear layer for comparison and filtering, reflective pixels in the second image feature can be removed, resulting in a first filtered feature in the second image that only includes translucent pixels. Therefore, a first image reconstruction feature can be obtained by first upsampling the first image features based on the first labeling information using a first linear layer. Then, based on the first image reconstruction feature, the second image features can be segmented to obtain a first filtering feature for each transparent pixel in the second image features, where the first labeling information is of the first value. For example, see attached... Figure 11 As shown, based on the first label information, the first image features at the first resolution are upsampled to obtain the first image reconstruction features at the second resolution. The first image reconstruction features and the second image features are input into the first linear layer, which can remove all reflective pixels in the second image features where the first label information is of the second value, and retain all translucent pixels where the first label information is of the first value. Therefore, the first linear layer is used to retain the second translucent part of the second image and remove the second reflective part of the second image. The second image features obtained after being filtered by the first linear layer are the second translucent part of the second image.

[0114] Finally, since the resolution of the second image corresponding to the second image feature is the second resolution, after removing the second reflective part of the second image and retaining the second translucent part of the second image, the second image feature is decoded to obtain the first decoded feature; the resolution of the second de-reflected image obtained based on the first decoded feature is the second resolution, and the second de-reflected image is an image in which the second reflective part of the second image has been removed, that is, it includes the second translucent part of the second image but does not include the second reflective part of the second image.

[0115] The image processing method described in step S203 above obtains first marker information corresponding to each pixel in the first image based on the similarity between the first de-reflective image features and the first image features. The first marker information indicates whether a pixel in the first image is translucent or reflective; each pixel corresponding to a first value of the first marker information is a translucent pixel, and each pixel corresponding to a second value of the first marker information is a reflective pixel. Based on this first marker information, the second reflective portion in the second image features is removed, finally obtaining a second de-reflective image with a second resolution after removing the second reflective portion from the second image. This image processing method employs a first traction upsampling model, which can not only remove the reflective portion but also improve the image resolution.

[0116] Step S204: Input the second de-reflected image, the second image, and the image to be processed into the second traction upsampling model to obtain the target image.

[0117] For example, see attached Figure 7 As shown, by inputting the second de-reflected image and the second image with the second resolution (768*2, 256*2) and the image to be processed with the original resolution (768*4, 256*4) into the second traction upsampling model, a target image with the original resolution (768*4, 256*4) can be obtained.

[0118] Since the resolution of the second de-reflected image is the second resolution, and the resolution of the image to be processed is the original resolution, in order to obtain an image at the original resolution, the second de-reflected image at the second resolution needs to be upgraded to the original resolution.

[0119] Similarly, increasing the image resolution is the same as step S203, and will not be repeated here.

[0120] Similarly, in this embodiment, a traction upsampling method is used to improve the image resolution. Since the second de-reflected image at a second resolution is being upsampled to the original resolution, according to the principle of the traction upsampling method, resolution is increased or decreased proportionally. Therefore, an image at an original resolution that is a multiple of the second resolution is needed for traction, and a second image at the second resolution is also required. Thus, it can be understood that the original resolution is twice the second resolution.

[0121] For example, see attached Figure 12As shown, the second de-reflected image, the second image, and the image to be processed are input into the second traction upsampling model to obtain the target image. This includes: encoding the second de-reflected image, the second image, and the image to be processed to obtain second de-reflected image features, second image features, and image to be processed features; the second de-reflected image features characterize the second reflective portion of the second image, the second image features characterize the second reflective portion and the second translucent portion of the second image, and the image to be processed features characterize the reflective and translucent portions of the image to be processed; based on the similarity between the second de-reflected image features and the second image features, second labeling information corresponding to each pixel in the second image is obtained; wherein, the second labeling information indicates whether a pixel in the second image is translucent or reflective; based on the second labeling information and the image to be processed features, filtering is performed to obtain second filtering features; the second filtering features characterize the translucent portion of the image to be processed; the second filtering features are decoded to obtain second decoding features; and based on the second decoding features, the target image is obtained.

[0122] Similarly, the methods for calculating similarity and filtering are the same as in step S203, and will not be repeated here.

[0123] In one possible implementation, based on the features of the second de-reflected image and the features of the second image, a second similarity between each pixel of the second image and the second de-reflected image is calculated using a second dot-multiplication matrix. Each pixel of the second image is then labeled based on the second similarity to obtain second labeling information.

[0124] In one possible implementation, based on the second label information corresponding to each pixel in the second image, the features of the second image are first upsampled to obtain the second image reconstruction features; the resolution of the second image reconstruction features is the original resolution. The second image reconstruction features and the features of the image to be processed are input into a second linear layer to obtain the second filtering features; the second linear layer is used to retain the translucent part of the image to be processed and remove the reflective part of the image to be processed.

[0125] For example, see attached Figure 12 As shown, firstly, the second de-reflected image, the second image, and the image to be processed are encoded to obtain the features of the second de-reflected image, the features of the second image, and the features of the image to be processed. The features of the second de-reflected image are used to characterize the second translucent portion of the second image, the features of the second image are used to characterize the second reflective portion and the second translucent portion of the second image, and the features of the image to be processed are used to characterize the reflective portion and the translucent portion of the image to be processed.

[0126] Secondly, based on the second de-reflective image features and the second image features at the same resolution (second resolution), the second similarity between each pixel of the second image and the second de-reflective image is calculated using the second dot-multiplication matrix. Since the second de-reflective image features do not include the second reflective feature but include the second translucent feature, and the second image features include both reflective and translucent features, in the second de-reflective image features and the second image features, the same features are the second translucent features, and different features are the second reflective features. Since pixels similar to the second de-reflective image features are translucent pixels, and pixels unrelated to the second de-reflective image features are reflective pixels, the translucent pixels in the second image can be marked as the third value, and the reflective pixels in the second image can be marked as the fourth value. Therefore, each pixel of the second image can be marked based on the second similarity to obtain the second marking information corresponding to each pixel in the second image. The second marking information indicates whether a pixel in the second image is translucent or reflective; pixels corresponding to the third value of the second marking information are translucent pixels, and pixels corresponding to the fourth value of the second marking information are reflective pixels. For example, the second similarity between each pixel of the second image and the second de-reflected image is calculated by the second dot product matrix. Since the second dot product matrix is ​​used to measure the similarity between the second de-reflected image and the second image in both the second reflective part and the second translucent part, the translucent pixels in the second image features that are similar to the features of the second de-reflected image, and the reflective pixels in the second image features that are not related to the features of the second de-reflected image, can be obtained by the second dot product matrix.

[0127] Secondly, since the resolution of the second image feature is the second resolution, the second labeling information is used to indicate whether the pixels in the second image are translucent or reflective. Each pixel corresponding to the third value of the second labeling information is a translucent pixel. Each pixel corresponding to the fourth value of the second labeling information is a reflective pixel. Based on the second labeling information corresponding to each pixel in the second image, the second image feature can be upsampled a second time to obtain a second image reconstruction feature with the original resolution. Since the second image reconstruction feature is a high-resolution second image feature, the positions and proportions of translucent and reflective pixels in the second image reconstruction feature are exactly the same as those in the original second image feature. Furthermore, the second linear layer is used to retain data with high similarity and remove data with low similarity. By inputting the second image reconstruction feature and the image feature to be processed into the second linear layer for comparison and filtering, reflective pixels in the image feature to be processed can be removed, resulting in a second filtered feature in the image to be processed that only includes translucent pixels. Therefore, a second image reconstruction feature can be obtained by performing a second upsampling on the second image features based on the second labeling information through a second linear layer. Then, based on the second image reconstruction feature, the image features to be processed are segmented to obtain the second filtering feature of each transparent pixel in the image features to be processed, where the second labeling information is all of the third value. For example, see attached... Figure 13 As shown, based on the second similarity, the second image features at the second resolution are upsampled to obtain the second image reconstruction features at the original resolution. The second image reconstruction features and the image features to be processed are input into the second linear layer, which can remove all reflective pixels in the image features to be processed whose second label information is all of the fourth value, and retain all translucent pixels whose second label information is all of the third value. Therefore, the second linear layer is used to retain the translucent part of the image to be processed and remove the reflective part of the image to be processed. The image features to be processed are filtered by the second linear layer to obtain the translucent part of the image to be processed.

[0128] Finally, since the resolution of the image to be processed corresponding to the image features to be processed is the original resolution, after removing the reflective part of the image to be processed and retaining the light-transmitting part of the image to be processed, the image features to be processed are decoded to obtain the second decoded features; the resolution of the target image obtained based on the second decoded features is the original resolution, and the target image is an image in which the reflective part of the image to be processed has been removed, that is, it includes the light-transmitting part of the image to be processed but does not include the reflective part of the image to be processed.

[0129] The image processing method described in step S204 above obtains second labeling information corresponding to each pixel in the second image based on the similarity between the second de-reflective image features and the second image features. The second labeling information indicates whether a pixel in the second image is translucent or reflective; pixels corresponding to the third value of the second labeling information are translucent pixels, and pixels corresponding to the fourth value of the second labeling information are reflective pixels. Based on this second labeling information, the reflective portion in the image features to be processed is removed, finally obtaining a target image with the reflective portion of the image to be processed removed at the original resolution. This image processing method employs a second traction upsampling model, which can not only remove the reflective portion but also improve the image resolution, obtaining an image at the original resolution.

[0130] The image processing method described in steps S201-S204 above reduces the resolution of the image to be processed from its original resolution to obtain a first image with a first resolution and a second image with a second resolution. Then, the first image is input into an image processing model to obtain a first de-reflected image with the first reflective portion removed. Next, a first pull upsampling is performed on the first de-reflected image, the first image, and the second image to obtain a second de-reflected image with the second reflective portion removed. Finally, a second pull upsampling is performed on the second de-reflected image, the second image, and the image to be processed to obtain a target image with the original resolution and the reflective portion of the image to be processed removed. This image processing method can remove reflective portions based on a reduced-resolution image to be processed, reducing memory usage, saving processing time, and improving image clarity by removing reflective portions.

[0131] For example, see attached Figure 14 As shown, based on the image processing method described in steps S201-S204 above, a target image can be obtained that has had its reflective parts removed and only includes the translucent parts of the image to be processed, and is clear, deep in color, uniform, and realistic. It can be understood that the translucent parts include: clear, deep in color, uniform, and realistic.

[0132] Since the first and second light-transmitting portions are the same as the light-transmitting portions, the first and second light-transmitting portions also include: clarity, deep color, uniformity, and realism.

[0133] For example, see attached Figure 15 As shown, taking the Android operating system running on the electronic device 100 as an example, the software architecture running on the processor 210 includes the application layer, framework layer, system runtime library layer, hardware abstraction layer (HAL), and kernel layer.

[0134] The kernel layer is the layer between hardware and software. For example, the kernel layer includes display drivers, camera drivers, etc. Display drivers are used to drive the display screen to show images or receive user touch operations, while camera drivers are used to drive the camera to capture image data.

[0135] In this embodiment, the camera driver is used to drive the camera 293 to acquire images to be processed.

[0136] The Hardware Abstraction Layer (HAL) is used to abstract hardware. The HAL hides the hardware interface details of a specific platform, providing the operating system with a virtual hardware platform and exhibiting hardware independence. For example, the HAL includes a display module and a camera module. The display module provides a virtual display screen, and the camera module provides a virtual camera.

[0137] The system runtime library layer includes C / C++ libraries and runtime libraries. Many core components and services of the Android operating system are built from native code and require C / C++ libraries. When an application is first installed, it is pre-compiled into machine code form as a runtime library; this process is called pre-compilation. This allows for faster startup and execution of the application by running the machine code.

[0138] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer includes predefined implementation methods. For example, it includes a window manager, content providers, a view system, a notification manager, and a camera service manager.

[0139] In this embodiment of the application, the camera service manager is used to respond to the call command of the camera application (app) and drive the camera 293 to acquire the image to be processed through the camera driver.

[0140] The application layer can include a series of application packages, such as the gallery, YOYO, camera, and other apps.

[0141] In one possible implementation, the image processing method described in this application embodiment can be integrated into a gallery or YOYO app, or into a camera app, or it can be a separate image processing app. This application embodiment does not limit the form of the image processing method app.

[0142] The following embodiments of this application illustrate the image processing method of this application by taking the example that the image processing method can be integrated into a gallery or the YOYO app.

[0143] Combined as attached Figure 15The software structure shown above, and the image processing method described in the embodiments of this application, can be implemented through interaction between the business side and the cloud side. For example, see the attached document. Figure 16 As shown, image processing of the image to be processed can be performed on the business side (i.e., the application side, such as the application layer) by clicking the image processing control in the control, interacting with the image processing plugin in the visual middleware algorithm, and then interacting with the cloud through the image processing microservice of the visual cloud service platform, and then interacting with the image processing inference service in the cloud-side model inference service.

[0144] For example, see attached Figure 17 As shown, specifically, the above is attached. Figure 16 The described interaction process includes the following stages:

[0145] Phase 1: Determine availability and initialize.

[0146] In the first phase, availability is determined. The application on the business side, either the image library or the YOYO app, sends an availability command to the image processing control. Upon receiving the availability command, the image processing control determines its availability and returns an availability result to the image library or the YOYO app. The image library or the YOYO app then terminates the first phase of the interaction process upon receiving this availability result. Specifically, if the image processing control is unavailable (e.g., due to a malfunction), it returns an unavailable result to the image library or the YOYO app; if the image processing control is available (e.g., functioning normally), it returns an available result. When the availability result is positive, the initialization interaction process begins.

[0147] Secondly, initialization is performed. The application library or YOYO app on the business side sends an initialization command to the image processing control. After receiving the initialization command, the image processing control continues to send a preparation command to the visual middleware algorithm. After receiving the preparation command, the visual middleware algorithm calls the function to create an image processing model object to create an image processing model object. This image processing model object is called when executing the image processing method in this embodiment, that is, when processing the image to be processed in the second stage, and returns the initialization result to the image processing control. After receiving the initialization result, the image processing control continues to return the initialization result to the library or YOYO app. After receiving the initialization result, the library or YOYO app ends the first stage of the interaction process.

[0148] The second stage: processing the image to be processed.

[0149] In the second phase, the application library or YOYO app on the business side sends processing instructions to the image processing control. Upon receiving the instructions, the image processing control sends them to the visual platform algorithm. The visual platform algorithm, upon receiving these instructions, calls the preprocessing function to preprocess the image and sends a request to the image processing microservice of the visual cloud service platform to upload the image to the cloud via the cloud-side API (framework layer). Upon receiving the request to upload the image to the cloud via the cloud-side API, the image processing microservice checks the number of times the image processing model object is called using a function to determine if the number of calls exceeds the limit, and calls the input parameter verification function to check the verification result. The number of times the image processing model object is called refers to the number of times the image processing model object created on the visual platform algorithm in the first phase is called. If the number of calls exceeds the limit, the image processing microservice returns an error code to the visual platform algorithm. If the input parameter verification fails, the image processing microservice returns an error code to the visual platform algorithm. If the number of calls does not exceed the limit and the input parameter verification succeeds, the image processing microservice sends an image processing model inference service instruction to the image processing inference service in the cloud-side model inference service. After receiving the image processing model inference service instruction, the image processing inference service performs inference on the image processing model and returns the image processing model algorithm result to the image processing microservice. The image processing microservice, based on the image processing model algorithm result, returns the image processing result to the visual middleware algorithm. Upon receiving the image processing result, the visual middleware algorithm calls the post-processing function to perform post-processing on the image and returns the image processing result to the image processing control on the business side. After receiving the image processing result returned by the visual middleware algorithm, the image processing control continues to return the image processing result returned by the visual middleware algorithm to the image library or the YOYO app. The image library or the YOYO app ends the second stage of the interaction process upon receiving the image processing result returned by the visual middleware algorithm.

[0150] Phase 3: Cancel the image processing model object and release resources.

[0151] In the third stage, firstly, the image processing model object is cancelled. It's understood that "cancelling the image processing model object" here refers to the image processing model object created in the first stage. The application on the business side, either the gallery or the YOYO app, sends a "cancel image processing model object" command to the image processing control. Upon receiving this command, the image processing control sends another "cancel image processing model object" command to the visual platform algorithm. The visual platform algorithm, upon receiving this command, calls the "cancel image processing model object" function to cancel the image processing model object and returns the cancellation result to the image processing control. Upon receiving this cancellation result, the image processing control returns the result to the gallery or the YOYO app. Finally, the gallery or the YOYO app ends the interaction process of cancelling the image processing model object.

[0152] In one possible implementation, the image processing model object can be cancelled after the second stage, or before or during the process of processing the image to be processed in the second stage. The embodiments of this application do not limit the time for cancelling the image processing model object.

[0153] In this embodiment, if the user clicks the cancel button during image processing, triggering the cancellation of the image processing model object, the process of canceling the image processing model object will be initiated. Upon triggering the cancellation of the image processing model object, the resource release interaction process will then commence.

[0154] Secondly, resource release is performed. This refers to releasing resources such as video memory or system memory. The application on the business side, either the image library or the YOYO app, sends a resource release command to the image processing control. Upon receiving this command, the image processing control sends another resource release command to the visual platform algorithm. The algorithm, upon receiving this command, calls the resource release function to release video memory or system memory and returns the result to the image processing control. The image processing control, upon receiving this result, returns it to the image library or the YOYO app. The image library or the YOYO app then concludes the third stage of the interaction process. In other words, this concludes the interaction between the business side and the cloud for this image processing method.

[0155] The image processing method and electronic device provided in this application reduce the resolution of the image to be processed from its original resolution to obtain a first image at a first resolution and a second image at a second resolution. Then, the first image is input into an image processing model to obtain a first de-reflective image with the reflective portion removed. Next, a first pull upsampling is performed on the first de-reflective image, the first image, and the second image to obtain a second de-reflective image at a second resolution with the reflective portion removed. Then, a second pull upsampling is performed on the second de-reflective image, the second image, and the image to be processed to obtain a target image at its original resolution with the reflective portion removed. This method, which removes reflective portions from the reduced-resolution image to be processed, reduces memory usage, saves processing time, and improves image clarity by removing reflective portions from the image.

[0156] It is understood that, in order to achieve the above functions, the electronic device includes hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application. This embodiment can divide the electronic device into functional modules according to the above method examples. For example, each function can be divided into separate functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0157] This application also provides a computer-readable storage medium storing computer program code. When the processor executes the computer program code, the electronic device executes the relevant method steps in the above method embodiments.

[0158] This application also provides a computer program product that, when run on a computer, causes the computer to execute the relevant method steps described in the above method embodiments.

[0159] The electronic devices, computer storage media, or computer program products provided in this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0160] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0162] The units described above as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The functions of the integrated unit can be implemented in hardware or as software functional units.

[0163] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the contributing parts, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: The original resolution of the image to be processed is reduced to obtain a first image with a first resolution and a second image with a second resolution; the first resolution is less than the original resolution, and the second resolution is less than the original resolution but greater than the first resolution; the image to be processed includes a reflective portion and a translucent portion; the first image includes a first reflective portion and a first translucent portion; the second image includes a second reflective portion and a second translucent portion; The first image is input into the image processing model to obtain a first de-reflected image; the first de-reflected image includes the first light-transmitting portion but does not include the first reflective portion, and the resolution of the first de-reflected image is the first resolution; The first de-reflected image, the first image, and the second image are input into the first traction upsampling model to obtain the second de-reflected image; the second de-reflected image includes the second light-transmitting portion but does not include the second reflective portion, and the resolution of the second de-reflected image is the second resolution; The second de-reflected image, the second image, and the image to be processed are input into the second traction upsampling model to obtain the target image; the target image includes the light-transmitting portion but does not include the reflective portion, and the resolution of the target image is the original resolution.

2. The image processing method according to claim 1, characterized in that, The step of inputting the first image into the image processing model to obtain the first de-reflected image includes: The first image is encoded to obtain a first feature; the first feature is used to characterize the first reflective portion and the first translucent portion of the first image. The first feature is split to obtain a first reflective feature and a first translucent feature; the first reflective feature is used to characterize the first reflective portion of the first image, and the first translucent feature is used to characterize the first translucent portion of the first image. The first light-transmitting feature is decoded to obtain the first de-reflective image, and the first reflective feature is decoded to obtain the first reflective image.

3. The image processing method according to claim 1 or 2, characterized in that, The training process of the image processing model includes: Obtain the original reflected image and the original dereflected image; the original reflected image is used to characterize the reflected portion of the original image, and the original dereflected image is used to characterize the translucent portion of the original image; the original image is obtained by merging the original reflected image and the original dereflected image. The original image is input into the image processing model to obtain a de-glare image and a glare image; The image processing model is trained based on the reflected image and the original reflected image, as well as the dereflected image and the original dereflected image.

4. The image processing method according to any one of claims 1-3, characterized in that, The step of inputting the first de-reflected image, the first image, and the second image into the first traction upsampling model to obtain the second de-reflected image includes: The first de-reflected image, the first image, and the second image are respectively encoded to obtain a first de-reflected image feature, a first image feature, and a second image feature; the first de-reflected image feature is used to characterize the first light-transmitting portion of the first image, the first image feature is used to characterize the first reflective portion and the first light-transmitting portion of the first image, and the second image feature is used to characterize the second reflective portion and the second light-transmitting portion of the second image. Based on the similarity between the first de-reflective image features and the first image features, first marker information corresponding to each pixel in the first image is obtained; wherein, the first marker information is used to indicate whether the pixel in the first image is translucent or reflective. Based on the first marker information and the second image features, a first filtering process is performed to obtain a first filtering feature; the first filtering feature is used to characterize the second light-transmitting portion of the second image. The first filtering feature is decoded to obtain the first decoded feature; based on the first decoded feature, the second de-reflected image is obtained.

5. The image processing method according to claim 4, characterized in that, The step of obtaining the first labeling information corresponding to each pixel in the first image based on the similarity between the first de-reflected image features and the first image features includes: Based on the first de-reflected image features and the first image features, the first similarity between each pixel of the first image and the first de-reflected image is calculated using the first dot product matrix. Each pixel of the first image is labeled based on the first similarity to obtain the first labeling information corresponding to each pixel in the first image.

6. The image processing method according to claim 4 or 5, characterized in that, The first filtering process based on the first marker information and the second image features to obtain the first filtering features includes: Based on the first labeling information, the first image features are upsampled to obtain the first image reconstruction features; the resolution of the first image reconstruction features is the second resolution. The first image reconstruction features and the second image features are input into the first linear layer to obtain the first filtering features; the first linear layer is used to retain the second light-transmitting part of the second image and remove the second reflective part of the second image.

7. The image processing method according to any one of claims 1-6, characterized in that, The step of inputting the second de-reflected image, the second image, and the image to be processed into the second traction upsampling model to obtain the target image includes: The second de-reflected image, the second image, and the image to be processed are respectively encoded to obtain the second de-reflected image features, the second image features, and the image to be processed features; the second de-reflected image features are used to characterize the second translucent portion of the second image, the second image features are used to characterize the second reflective portion and the second translucent portion of the second image, and the image to be processed features are used to characterize the reflective portion and the translucent portion of the image to be processed; Based on the similarity between the second de-reflective image features and the second image features, second labeling information corresponding to each pixel in the second image is obtained; wherein, the second labeling information is used to indicate whether the pixel in the second image is translucent or reflective; Based on the second labeling information and the features of the image to be processed, a second filtering process is performed to obtain a second filtering feature; the second filtering feature is used to characterize the light-transmitting portion of the image to be processed. The second filtered feature is decoded to obtain the second decoded feature; The target image is obtained based on the second decoding feature.

8. The image processing method according to claim 7, characterized in that, The step of obtaining the second labeling information corresponding to each pixel in the second image based on the similarity between the second de-reflected image features and the second image features includes: Based on the second de-reflected image features and the second image features, the second similarity between each pixel of the second image and the second de-reflected image is calculated using the second dot product matrix. Each pixel of the second image is labeled based on the second similarity to obtain the second labeling information corresponding to each pixel in the second image.

9. The image processing method according to claim 7 or 8, characterized in that, The second filtering process, based on the second labeling information and the features of the image to be processed, to obtain the second filtering features, includes: Based on the second labeling information, the second image features are upsampled a second time to obtain the second image reconstruction features; the resolution of the second image reconstruction features is the original resolution. The second image reconstruction features and the features of the image to be processed are input into the second linear layer to obtain the second filtering features; the second linear layer is used to retain the light-transmitting part of the image to be processed and remove the reflective part of the image to be processed.

10. An electronic device, characterized in that, include: A memory and one or more processors, the memory being coupled to the processors; wherein the memory is used to store instructions executable by the one or more processors, the memory storing computer program code including computer instructions that, when executed by the processor, cause the electronic device to perform the image processing method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method as described in any one of claims 1-9.

12. A computer program product, characterized in that, When the computer program product is run on a computer, the computer performs the image processing method as described in any one of claims 1-9.

Images