A method, device and storage medium for directional removal of rainbow-like glare in an under-screen RGB image

Abstract

A method for removing rainbow-like glare in an under-screen RGB image in a directional manner, comprising the following steps: step S1: obtaining a first RGB image, removing rainbow-like glare in an undirectional manner by using a deep neural network model to obtain a second RGB image; wherein the undirectional removal refers to removing both glare and light sources; step S2: fusing the light source and the uneven periphery of the light source in the first RGB image with the second RGB image to generate a third RGB image; and step S3: eliminating stepwise defects in the third RGB image by using a neural network model. The method first removes rainbow-like glare in an undirectional manner by using a deep neural network model, then fuses the images before and after processing to obtain a third RGB image with a light source, and then eliminates stepwise defects on the third RGB image to obtain a final RGB image, thereby effectively removing glare in an RGB image captured by an under-screen camera.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular, to a method and device for directional removal of rainbow-like glare in an under-screen RGB image and a storage medium. BACKGROUND

[0002] In recent years, with the rapid development of the concept of full-screen mobile phones, many mobile phone manufacturers are trying to place the front camera behind the display screen to achieve the effect of a truly full-screen. However, placing the front camera behind the display screen brings new problems, namely, the imaging quality of the under-screen camera is severely degraded. In particular, when a strong light source appears in the field of view, the RGB color image captured by the under-screen camera will have severe rainbow-like glare, which surrounds the light source in a circle and covers the background information, resulting in a poor user experience.

[0003] Currently, people have begun to study the quality problems caused by under-screen cameras, but there is currently no targeted solution to the problem of rainbow-like glare. Some people have also studied how to remove RGB image glare, but the images they targeted are not captured by under-screen cameras, and the glare patterns are very different from the rainbow-like glare in images captured by under-screen cameras.

[0004] The difficulty of this problem lies in how to remove the rainbow-like glare around the light source while preserving the light source, and at the same time ensure a natural transition between the light source edge and the surrounding background, so that the image can maintain a natural effect after the directional removal of glare.

[0005] Traditional glare is caused by the scattering and reflection of strong light under the action of optical elements in the sensor. The glare in the under-screen camera is not only caused by the characteristics of the optical elements in the sensor, but also caused by the change in the light path due to the influence of the display screen. The camera under the screen has a certain distance from the screen, and the terminal user may have a protective film, which further exacerbates the various optical changes of light before it enters the camera, making the final imaging different from traditional imaging.

[0006] In the under-screen image, the rainbow-like glare is more extensive than traditional glare, showing a closed and diffuse shape, and the light intensity also changes gradually, which is significantly different from traditional rainbow, so traditional solutions cannot effectively solve the problem of under-screen rainbow-like glare. SUMMARY

[0007] To this end, the present application first uses a deep neural network model to remove rainbow-like glare in an undirectional manner, then fuses the processed and unprocessed images to obtain a third RGB image with a light source, and then eliminates the step-like defects on the third RGB image to obtain a final RGB image, thereby effectively removing the glare in the RGB image captured by the under-screen camera.

[0008] In a first aspect, the present application provides a method for removing rainbow-like glare from an under-screen RGB image in a directional manner, comprising the following steps:

[0009] Step S1: Obtain a first RGB image, remove rainbow-like glare in an undirectional manner using a deep neural network model to obtain a second RGB image; wherein the undirectional manner refers to removing both glare and light sources;

[0010] Step S2: Fuse the light source and the uneven periphery of the light source in the first RGB image with the second RGB image to generate a third RGB image;

[0011] Step S3: Use a neural network model to eliminate the stepwise defects in the third RGB image.

[0012] Optionally, the method for removing rainbow-like glare from an under-screen RGB image in a directional manner comprises the following steps: in step S1, the deep neural network model is obtained by training a one-to-one corresponding data set of light source opening / closing.

[0013] Optionally, the method for removing rainbow-like glare from an under-screen RGB image in a directional manner comprises the following steps: the deep neural network model adopts an Encoder-BottleneckBlock-Decoder architecture, wherein the Encoder part uses a pre-trained VGGNet to extract image features, the BottleneckBlock part uses multiple residual modules stacked, and the residual modules are all provided with a channel attention mechanism, and the Decoder part up-samples the output of the BottleneckBlock part to the original image resolution.

[0014] Optionally, the method for removing rainbow-like glare from an under-screen RGB image in a directional manner comprises the following steps: the step S2 comprises:

[0015] Step S21: Segment the light source in the first RGB image to generate a mask;

[0016] Step S22: Dilate the mask, and set a weight for each dilated circle, so that the weight gradually decays from 1.0 to 0.0 in the process of spreading outward from the edge of the light source, thereby obtaining a single-channel weight alpha map;

[0017] Step S23: Use the obtained weight alpha map to fuse the light source and the periphery of the light source in the first RGB image with the second RGB image to generate a third RGB image.

[0018] Optionally, the method for removing rainbow-like glare of a screen-under RGB image in a directional manner has a feature that, in step S22, the speed of expansion in different directions is determined according to the angle of the light source.

[0019] Optionally, the method for removing rainbow-like glare of a screen-under RGB image in a directional manner has a feature that, in step S22, the weight wherein d is the shortest distance between a pixel point on the image and the edge of the light source, d max is the shortest distance between the edge of the periphery of the light source and the edge of the light source.

[0020] Optionally, the method for removing rainbow-like glare of a screen-under RGB image in a directional manner has a feature that, in step S3, the neural network model is a generative adversarial network model.

[0021] Optionally, the method for removing rainbow-like glare of a screen-under RGB image in a directional manner has a feature that the generative adversarial network includes a generator and a discriminator; in a training set, the label of a real image is true, and the label of a fused image is false, so as to cross-train the discriminator and the generator; finally, the trained generator is used to adjust the third RGB image, so that the details of the transition area are smoother.

[0022] In a second aspect, the present application provides a device for removing rainbow-like glare of a screen-under RGB image in a directional manner, which has the following features.

[0023] a processor;

[0024] a memory module, wherein executable instructions of the processor are stored;

[0025] The processor is configured to execute the steps of the method for removing rainbow-like glare of a screen-under RGB image in a directional manner by executing the executable instructions.

[0026] In a third aspect, the present application provides a computer-readable storage medium for storing a program, which has the following features.

[0027] Compared with the prior art, the present application has the following advantages:

[0028] The present application removes the glare first and then fuses, so that other objects outside the glare are clearer and more obvious in the image, and the influence of the glare is removed.

[0029] The present application further processes the fused image to eliminate the step-like defects in the screen-under glare, so that the image is more natural and real. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0031] Figure 1 This is a flowchart illustrating the steps of a method for targeted removal of rainbow-like glare in under-screen RGB images according to an embodiment of the present invention.

[0032] Figure 2 This is a first RGB image captured by an under-display camera in an embodiment of the present invention;

[0033] Figure 3 This is a second RGB image in an embodiment of the present invention;

[0034] Figure 4 This is a third RGB image in an embodiment of the present invention;

[0035] Figure 5 This is an enlarged third RGB image in an embodiment of the present invention;

[0036] Figure 6 This is a final image in an embodiment of the present invention;

[0037] Figure 7 This is a flowchart illustrating the steps for generating a third RGB image in an embodiment of the present invention;

[0038] Figure 8 This is a schematic diagram of the structure of a device for directional removal of rainbow-like glare from under-screen RGB images according to an embodiment of the present invention;

[0039] Figure 9 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0040] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0041] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, and above-mentioned drawings, if any, are used to distinguish between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, under appropriate circumstances, to describe the embodiments of the present application described herein, for example, can be practiced in other than the illustrated order if it is assumed that the embodiments of the present application appropriately described herein are not limited to the particular order illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a list of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or apparatuses.

[0042] The technical solutions of the present application will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in detail in some examples.

[0043] The method for removing rainbow-like glare of RGB image under screen provided by the embodiment of the present application aims to solve the problems in the prior art.

[0044] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in detail in some examples. The embodiments of the present application will be described below with reference to the drawings.

[0045] The present application first removes rainbow-like glare by using a deep neural network model, then fuses the images before and after processing to obtain a third RGB image with a light source, and then eliminates the step defects on the third RGB image to obtain the final RGB image, thereby effectively removing the glare in the RGB image captured by the camera under the screen.

[0046] Figure 1 The flow chart of the steps of the method for removing rainbow-like glare of RGB image under screen in the embodiment of the present application. As shown in Figure 1 The method for removing rainbow-like glare of RGB image under screen in the embodiment of the present application includes the following steps.

[0047] Step S1: Obtain a first RGB image, remove rainbow-like glare by using a deep neural network model, and obtain a second RGB image.

[0048] In this step, the non-directional refers to the removal of both glare and light source. Directional refers to the removal of glare only while retaining the light source. This embodiment achieves the effect of removing glare only while retaining the light source by first removing both glare and light source, and then processing the subsequent processing. The first RGB image is an image captured by the under-screen camera. As shown in Figure 2 , unlike the glare when the camera directly captures strong light, the image captured by the under-screen camera shows a larger range of continuous flare on the image due to the light passing through the screen and then entering the camera, and the color changes with the increase of the radius, similar to a rainbow. Further, in the peripheral area in Figure 2 , the glare shows a weaker form, and the information of other objects can be clearly seen, which is equivalent to adding a layer of mask to other objects, unlike the traditional glare where other objects cannot be seen. In this embodiment, the first RGB image is obtained by the camera or camera under the screen, and the screen is usually a glass screen. The camera or camera is at a certain distance from the glass screen, usually between 0.1mm-5mm, which is adjusted according to different application scenarios. For example, the distance in the front camera of a mobile phone is less than the corresponding distance in a robot.

[0049] The deep neural network model identifies the rainbow-shaped glare, so it can be removed. The deep neural network model not only needs to identify the area of the rainbow-shaped glare, but also needs to identify the intensity of the rainbow-shaped glare. As shown in Figure 3 , the obtained second image not only retains the information of other objects while removing the rainbow-shaped glare, but also removes the entire glare area.

[0050] In some embodiments, the deep neural network model is trained by a one-to-one corresponding dataset of light source on / off. During training, the training set is composed of a one-to-one corresponding data pair of light source on / off for each group. The number of samples in the training set can be any one of 10,000, 20,000, 30,000, 40,000, 50,000, 60,000, or any other value not listed. The data in the training set are all data samples taken under the screen. The data samples in the training set can be taken by different cameras or cameras at different distances from the glass screen, but all need to appear rainbow glare. Different cameras or cameras will have different parameters, and the size and range of the images taken will be different. The camera or camera at different distances from the glass screen will produce rainbow glare of different sizes and cause different images. When the distance between the light source and the glass screen is different, it will also cause the size of the rainbow glare on the image to be different. The training set is composed of image pairs with rainbow glare taken under the above conditions, which can increase the robustness of the deep neural network model. Each data pair is composed of an image with rainbow glare when the light source is on and a background image without the light source when the light source is off. The deep neural network model can be any type, including but not limited to convolutional neural networks, recurrent neural networks, generative adversarial networks, etc.

[0051] In some embodiments, the deep neural network model adopts an Encoder-BottleneckBlock-Decoder architecture. The Encoder is an encoder and can be any content such as a model, etc. In this embodiment, the Encoder part uses a pre-trained VGGNet to extract image features. The BottleneckBlock is used to construct the network and uses multiple residual modules stacked. In this embodiment, the residual modules all have a channel attention mechanism to further improve the recognition effect and computational efficiency of the rainbow glare. The Decoder is a decoder that upsamples the output of the BottleneckBlock part to the original image resolution. Since the rainbow glare exhibits a characteristic of emitting outward from the light source, the residual module used in this embodiment can quickly and effectively identify the rainbow glare, so it can not only identify the rainbow glare and is not affected by color changes, but also can quickly calculate and improve efficiency.

[0052] The expression of the residual module is y = F(x, {W i}) + x, where x and y are the input and output respectively, Wi is the parameter to be learned, and F is the operation function.

[0053] The expression of the channel attention mechanism is and s = σ(W2δ(W1z)), where H and W are the height and width of the feature map respectively, i and j are the coordinates of the feature map position, uc is the value of the corresponding position of the feature map, zc is the result of the global average pooling of the channel C, z is the stacked output of the output of each channel in the second row, W1 and W2 are parameters to be learned, δ uses the ReLu activation function, and σ is the sigmoid activation function.

[0054] Step S2: Fusing the light source and the uneven light source periphery in the first RGB image with the second RGB image to generate a third RGB image.

[0055] In this step, the light source and the light source periphery are smaller than the area identified in step S1, only the part where the light source is strong, and the external rainbow-like glare part is removed. In this step, various segmentation methods can be used to segment the light source and the light source periphery in the first RGB image, including but not limited to: threshold-based segmentation method, region-based segmentation method, edge-based segmentation method, and segmentation method based on specific theory, etc. The embodiment does not limit the segmentation method, as long as it can segment the light source and the light source periphery. As shown in FIG. 3B, in the third RGB image, the rainbow-like glare is removed, and the light source and the light source periphery are retained, so that the third RGB image is closer to the real scene. Figure 4

[0056] Step S3: Using a neural network model to eliminate the stepped defects in the third RGB image.

[0057] In this step, the third RGB image is further processed to eliminate the stepped defects in the third RGB image. As shown in FIG. 4B, in the transition area, there are one circle after another stepped defects, the grain feeling is strong, and there are still some rainbow-like glares. This step uses a deep learning model to learn and process the stepped defects, which can further remove the rainbow-like glares while removing the stepped defects. Figure 5

[0058] In some embodiments, the neural network model is a generative adversarial network model. The generative adversarial network is trained using backpropagation. During training or generating samples, no Markov chain or unfolded approximate inference network is needed. The generative adversarial network includes a generator and a discriminator; in the training set, the label of the real image is true, and the label of the fused image is false, so as to cross-train the discriminator and the generator; finally, the trained generator is used to adjust the third RGB image, so that the details of the transition area are smoother. Figure 6 For the image adjusted by the embodiment, it can be seen that the stepped defects have been eliminated, and the rainbow-like glares have been further reduced.

[0059] ​​The following is an optimization objective for training a generative adversarial network, where x and y are the fused image and the real image, respectively, D Y and G are the discriminator and the generator, respectively, and the goal of training a generative adversarial network model is to maximize L GAN

[0060]

[0061] Figure 7 A flowchart of a step of generating a third RGB image in an embodiment of the present application is shown in FIG. 21. As shown in FIG. 21, a method of generating a third RGB image in an embodiment of the present application includes the following steps. Figure 7

[0062] Step S21: Segment the light source in the first RGB image to generate a mask.

[0063] In this step, when the light source is obtained, a threshold segmentation method can be used to segment the area with higher brightness to obtain the light source area. In an RGB image, white is (255, 255, 255), that is, the R channel, the G channel, and the B channel are all 255. When performing threshold judgment, the R, G, and B channels are judged respectively, and when the values of the three channels are all greater than a preset threshold, the light source area is judged. The preset threshold is a fixed value, including but not limited to 255, 254, 253, 252, 251, and other arbitrary set values. The preset threshold is usually greater than 250. The thresholds of the R channel, the G channel, and the B channel can be different. The higher the threshold, the smaller the range of the obtained light source area. The lower the threshold, the larger the range of the obtained light source area. A mask is generated for the identified light source.

[0064] Step S22: Dilate the mask, and set a weight for each dilated circle, so that in the process of outward diffusion from the edge of the light source, the weight gradually decays from 1.0 to 0.0, thereby obtaining a single-channel weight alpha map.

[0065] In this step, the mask is dilated outward circle by circle, and a unique weight is set for each circle. The weight value of the edge of the light source is 1, and the weight value of the diffusion end is 0. The weights of different circles are different, and the farther away from the light source, the smaller the weight. The decay of the weight is nonlinear. According to the weight distribution, a single-channel weight alpha map representing the weight can be obtained, which can be used for image fusion calculation. The part formed by the strong light source on the first RGB image is defined from inside to outside as a light source, a light source periphery, and a glare part. The light source is a strong area with brightness close to white light, which is the area where the light source itself is located. The light source periphery is a natural area formed when the light source shines outward, rather than light formed by diffusion of lenses, glass sheets, etc. The glare part is the glare part formed by diffusion of lenses, glass sheets, etc., mainly in the form of rainbow-like glare. The three can be distinguished by brightness values. ​​

[0066] In some embodiments, the weight wherein d is the shortest distance between a pixel point on the image and the edge of the light source, d max is the shortest distance between the edge of the light source periphery and the edge of the light source. The value of d changes between [0, d max ] as the pixel point changes. When d = 0, a = 1. When d = d max , a = 0. The present embodiment can ensure that the change of the weight is more adaptive to the perception of light by the human eye, making the fusion effect more realistic.

[0067] In some embodiments, the speed of expansion in different directions is determined according to the angle of the light source. When the light source is perpendicular to the camera or camera, the expansion speed in each direction in this step is the same. When the light source is not perpendicular to the camera or camera, the light source is not circular on the image, but presents an ellipse or other shape, and the present embodiment can adjust the expansion speed in different directions according to the angle of the light source, so that the expansion is more uniform.

[0068] In some embodiments, the line width at different positions when expanding is determined according to the angle of the light source. When the light source is perpendicular to the camera or camera, the line width at different positions when expanding in each direction in this step is the same. When the light source is not perpendicular to the camera or camera, the light source is not circular on the image, but presents an ellipse or other shape, and the present embodiment can adjust the line width at different positions when expanding according to the angle of the light source, so that the expansion is more uniform.

[0069] In some embodiments, the line width at different positions when expanding is determined according to the distance between the center of the light source and the edge of the light source. When the light source is perpendicular to the camera or camera, the line width at different positions when expanding in each direction in this step is the same. When the center of the light source is closer to the edge of the light source, the line when expanding is thinner; when the center of the light source is farther away from the edge of the light source, the line when expanding is wider, so that each line reaches the edge of the light source periphery at the same time.

[0070] Step S23: using the obtained weight alpha map, the light source and the light source periphery in the first RGB image are fused with the second RGB image to generate a third RGB image.

[0071] In this step, the weight alpha map is multiplied by the light source and the light source periphery in the first RGB image, and then added to the second RGB image to obtain a third RGB image. The third RGB image not only retains the light source and the light source periphery, but also removes the rainbow-like glare, obtaining a more realistic image.

[0072] The embodiment divides the light source, obtains a curing film, gradually expands the curing film outward, obtains an alpha map, and then performs fusion, thereby retaining a region with a strong light source, removing rainbow-like glare, and being adjustable for different light source angles, and having wide applicability.

[0073] The embodiment of the present application also provides a device for removing rainbow-like glare of an under-screen RGB image in a directional manner, including a processor and a memory having executable instructions of the processor stored therein.

[0074] As described above, the present application first removes rainbow-like glare in a non-directional manner by using a deep neural network model, then fuses the images before and after processing to obtain a third RGB image with a light source, and then removes the step defects on the third RGB image to obtain a final RGB image, thereby effectively removing glare in the RGB image captured by an under-screen camera.

[0075] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "platform" here.

[0076] Figure 8 is a structural schematic diagram of a device for removing rainbow-like glare of an under-screen RGB image in a directional manner in the embodiment of the present application. The electronic device 600 according to this embodiment of the present application will be described below with reference to Figure 8 Figure 8 The display electronic device 600 is only an example and should not limit the functions and use range of the embodiment of the present application.

[0077] As shown in Figure 8 , the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include but are not limited to at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0078] The storage unit stores program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps described in the above directional removal of rainbow-like glare of an under-screen RGB image method according to various exemplary embodiments of the present application. For example, the processing unit 610 can perform the steps as shown in Figure 1 ​​

[0079] Storage 620 can include a readable medium in the form of volatile memory, such as random access memory (RAM) 6201 and / or cache memory 6202, and can further include non-volatile memory, such as read only memory (ROM) 6203.

[0080] Storage 620 can also include program / utility 6204 having a set of programs / modules 6205, including an operating system, one or more application programs, other program modules, and program data, each of which can implement aspects of a network environment, for example, as described in one or more of the examples above or some combination thereof.

[0081] Bus 630 can represent one or more of several types of bus structures, including a storage bus or bus controller, peripheral bus, graphics bus, processor or local bus using any of a variety of bus architectures.

[0082] Electronic device 600 can also communicate with one or more external devices 700 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices that enable a user to interact with electronic device 600; and / or one or more devices that enable electronic device 600 to communicate with one or more other computing devices. Such communication can be via input / output (I / O) interface 650. Similarly, such communication can be via network adapter 660, which can be any device or means Figure 8 Other hardware and / or software modules that can be used in conjunction with electronic device 600 can also be employed, such as microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0083] The embodiment of the present application also provides a computer readable storage medium for storing a program, the program being executed to implement the steps of the method for removing rainbow-like glare of an under-screen RGB image in a targeted manner. In some possible implementation manners, various aspects of the present application can also be implemented in the form of a program product, which includes program codes for causing terminal equipment to perform the steps described in the above method for removing rainbow-like glare of an under-screen RGB image in a targeted manner according to various exemplary embodiments of the present application when the program product is run on the terminal equipment.

[0084] As shown above, the program of the computer readable storage medium of this embodiment, when executed, first removes the rainbow-like glare in the unidirectional manner by using the deep neural network model, then fuses the images before and after the processing to obtain the third RGB image with the light source, and then eliminates the step defect on the third RGB image to obtain the final RGB image, thereby effectively removing the glare in the RGB image captured by the under-screen camera.

[0085] Figure 9 is a structural schematic diagram of the computer readable storage medium in the embodiment of the present application. Referring to Figure 9 As shown above, the program of the computer readable storage medium of this embodiment, when executed, first removes the rainbow-like glare in the unidirectional manner by using the deep neural network model, then fuses the images before and after the processing to obtain the third RGB image with the light source, and then eliminates the step defect on the third RGB image to obtain the final RGB image, thereby effectively removing the glare in the RGB image captured by the under-screen camera.

[0086] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or means, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0087] The computer readable storage medium can include a data signal propagating in the baseband or as part of a carrier wave propagating through the transmission medium, in which the readable program code is carried. Such a propagating data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than the readable storage medium, which can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, device or means. The program code contained on the readable storage medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0088] The program code may be executed by one or more programmable processing devices to perform a set of tasks on a computing device. The computing device may include a server, a personal computer, a laptop computer, a mobile device, a network router, switch or bridge, or any other suitable device. The computing device may include one or more processors, such as a processor from the Intel X86 family of processors, the Motorola family of processors, the Transmeta family of processors, the ARM family of processors, the AMD family of processors, or any other suitable processor family. The computing device may include one or more memory devices, such as a RAM, ROM, EEPROM, or flash memory. The computing device may include one or more storage devices, such as a hard disk drive, a floppy disk drive, a CD-ROM drive, or any other suitable storage device. The computing device may include one or more communication ports that allow the computing device to communicate with one or more other devices, such as a network interface, a wireless interface, a bus interface, or any other suitable communication port. The computing device may include one or more input / output devices, such as a keyboard, a mouse, a pen, a voice input device, a touch input device, a display, or any other suitable input / output device.

[0089] In the embodiments of the present application, first, the deep neural network model is used to remove the rainbow-like glare in an undirectional manner, then the image before and after processing is fused to obtain a third RGB image with a light source, and then the step defect on the third RGB image is removed to obtain the final RGB image, thereby effectively removing the glare in the RGB image captured by the under-screen camera.

[0090] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0091] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which do not affect the essential content of the present application.

Claims

1. A method for targeted removal of rainbow-like glare in under-screen RGB images, characterized in that, Includes the following steps: Step S1: Obtain a first RGB image, and use a deep neural network model to remove the rainbow-shaped glare in the first RGB image in a non-directional manner to obtain a second RGB image; wherein, the non-directional means removing both glare and light source simultaneously; the rainbow-shaped glare presents a closed diffusion shape, and the light intensity also changes gradually; the part formed by the strong light source on the first RGB image is defined from the inside out as the light source, the periphery of the light source, and the glare part; Step S2: The light source and the area surrounding the light source in the first RGB image are unevenly fused with the second RGB image to generate a third RGB image; Step S3: Eliminate the step defects in the third RGB image using a neural network model; Step S2 includes: Step S21: Segment the light source region in the first RGB image to generate a mask; Step S22: Expand the mask and set a weight for each expansion. As the mask expands outward from the edge of the light source to the edge of the light source, the weight gradually decreases from 1.0 to 0.0, thus obtaining a single-channel weight alpha map. Step S23: Using the obtained weighted alpha image, the light source and the area surrounding the light source in the first RGB image are fused with the second RGB image to generate a third RGB image; In step S22, weights Where d is the shortest distance between a pixel in the image and the edge of the light source. It is the shortest distance between the periphery of the light source and the edge of the light source.

2. The method for targeted removal of rainbow-like glare in under-screen RGB images according to claim 1, characterized in that, In step S1, the deep neural network model is trained using a dataset; the dataset consists of one-to-one data pairs obtained by shooting with the light source on / off.

3. The method for targeted removal of rainbow-like glare in under-screen RGB images according to claim 1, characterized in that, The deep neural network model adopts an Encoder-BottleneckBlock-Decoder architecture. The Encoder part uses a pre-trained VGGNet to extract image features, the BottleneckBlock part uses multiple residual modules stacked together, and each residual module has a channel attention mechanism. The Decoder part upsamples the output of the BottleneckBlock part to the original image resolution.

4. The method for targeted removal of rainbow-like glare in under-screen RGB images according to claim 3, characterized in that, In step S22, the velocities of the expansion in different directions are determined based on the angle of the light source.

5. A method for targeted removal of rainbow-like glare in under-screen RGB images according to claim 1, characterized in that, In step S3, the neural network model is a generative adversarial network model.

6. A method for targeted removal of rainbow-like glare in under-screen RGB images according to claim 5, characterized in that, The generative adversarial network includes a generator and a discriminator. In the training set, the real image is labeled as true and the fused image is labeled as false, thereby cross-training the discriminator and the generator. Finally, the trained generator is used to adjust the third RGB image to make the details in the transition region smoother.

7. A device for targeted removal of rainbow-like glare from under-screen RGB images, characterized in that, include: processor; A memory module that stores executable instructions of the processor; The processor is configured to perform the steps of the method for targeted removal of rainbow glare from under-screen RGB images according to any one of claims 1 to 6 by executing the executable instructions.

8. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, it implements the steps of the method for directional removal of rainbow glare in under-screen RGB images as described in any one of claims 1 to 6.

Images