Method, device and storage medium for removing rainbow-like glare from under-screen RGB image

Abstract

A method for removing rainbow-like glare in an under-screen RGB image, characterized by the following steps: step S1: obtaining a first RGB image, removing at least two connected light sources and their rainbow-like glare by using a deep neural network model to obtain a second RGB image; step S2: unevenly fusing the at least two connected light sources in the first RGB image with the second RGB image to generate a third RGB image; and step S3: eliminating stepwise defects in the third RGB image by using a neural network model. The method first removes at least two connected light sources and their rainbow-like glare by using a deep neural network model, then fuses the images before and after processing to obtain a third RGB image with light sources, and then eliminates stepwise defects on the third RGB image to obtain a final RGB image, thereby effectively removing the glare of multiple light sources in the RGB image captured by an under-screen camera.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular, to a method and device for removing rainbow glare in an RGB image under a screen and a storage medium. BACKGROUND

[0002] In recent years, with the rapid development of the concept of full screen for mobile phones, many mobile phone manufacturers are trying to place the front camera behind the display screen to achieve the effect of a truly full screen. However, placing the front camera behind the display screen brings new problems, namely, the imaging quality of the under-screen camera is severely degraded. In particular, when a strong light source appears in the field of view, the RGB color image captured by the under-screen camera will have serious rainbow glare, which surrounds the light source in a circle and covers the background information, resulting in a poor user experience.

[0003] Currently, people have begun to study the quality problems caused by under-screen cameras, but there is currently no targeted solution to the problem of rainbow glare. Some people have also studied how to remove the glare of RGB images, but the images they targeted are not captured by under-screen cameras, and the glare pattern is very different from the rainbow glare in images captured by under-screen cameras.

[0004] The difficulty of this problem lies in how to remove the rainbow glare around the light source while preserving the light source, and at the same time ensure a natural transition between the light source edge and the surrounding background, so that the overall image can maintain a natural effect after directional removal of glare.

[0005] Traditional glare is caused by the scattering and reflection of strong light under the action of optical elements in the sensor. The glare in the under-screen camera is not only caused by the characteristics of the optical elements in the sensor, but also caused by the change in the light path due to the influence of the display screen. The camera under the screen has a certain distance from the screen, and the terminal user may have a protective film, which further exacerbates the various optical changes of light before it enters the camera, making the final imaging of the camera significantly different from traditional imaging.

[0006] In the image under the screen, the range of rainbow glare is wider than that of traditional glare, showing a closed and diffuse shape, and the light intensity also changes gradually, which is significantly different from traditional rainbow glare, so traditional solutions cannot effectively solve the problem of rainbow glare under the screen.

[0007] In addition to the rainbow glare caused by a single light source, when there are multiple light sources in the field of view of the camera or the camera, especially when multiple light sources are connected or overlapped in the image, the overall characteristics are often very different from those of a single light source due to optical characteristics. SUMMARY

[0008] To this end, the present application first removes at least two connected light sources and their rainbow-like glare by using a deep neural network model, then fuses the images before and after processing to obtain a third RGB image with light sources, and finally removes the stepwise defects on the third RGB image to obtain the final RGB image, thereby effectively removing the glare of multiple light sources in the RGB image captured by the under-screen camera.

[0009] In a first aspect, the present application provides a method for removing rainbow-like glare of an under-screen RGB image, characterized in that it comprises the following steps:

[0010] Step S1: Obtain a first RGB image, remove at least two connected light sources and their rainbow-like glare by using a deep neural network model, and obtain a second RGB image;

[0011] Step S2: Unevenly fuse the at least two connected light sources in the first RGB image with the second RGB image to generate a third RGB image;

[0012] Step S3: Remove the stepwise defects in the third RGB image by using a neural network model.

[0013] Optionally, the method for removing rainbow-like glare of an under-screen RGB image is characterized in that, in step S1, after obtaining the first RGB image, the position and shape of the light source are determined according to the color of the image, and then the deep neural network model is used for processing.

[0014] Optionally, the method for removing rainbow-like glare of an under-screen RGB image is characterized in that the deep neural network model adopts an Encoder-BottleneckBlock-Decoder architecture, wherein the Encoder part uses a pre-trained VGGNet to extract image features, the BottleneckBlock part uses multiple residual modules stacked, and the residual modules all have channel attention mechanisms, and the Decoder part up-samples the output of the BottleneckBlock part to the original image resolution.

[0015] Optionally, the method for removing rainbow-like glare of an under-screen RGB image is characterized in that the deep neural network model is trained by fitting multiple light sources through data augmentation.

[0016] Optionally, the method for removing rainbow-like glare of an under-screen RGB image is characterized in that the step S2 comprises:

[0017] Step S21: Segment the light source in the first RGB image to generate a mask;

[0018] Step S22: uneven expansion is performed on the mask, and a weight is set for each expansion circle, so that the weight gradually decays from 1 to 0 in the process of diffusion from the light source edge to the effective edge, thereby obtaining a single-channel weight alpha map;

[0019] Step S23: the weight alpha map is multiplied by the light source edge diffusion to the effective edge in the first RGB image, and then fused with the second RGB image to generate a third RGB image.

[0020] Optionally, the method for removing rainbow-like glare in an under-screen RGB image has the characteristics that in step S22, the expansion speed in different directions is determined according to the shape of the light source.

[0021] Optionally, the method for removing rainbow-like glare in an under-screen RGB image has the characteristics that in step S22, the weight wherein d is the shortest distance between a pixel point on the image and the light source edge, and d max is the shortest distance between the effective edge and the light source edge.

[0022] Optionally, the method for removing rainbow-like glare in an under-screen RGB image has the characteristics that in step S3, the neural network model is a generative adversarial network model; the generative adversarial network includes a generator and a discriminator; in the training set, the label of the real image is true, and the label of the fused image is false, so as to cross-train the discriminator and the generator; finally, the trained generator is used to adjust the third RGB image, so that the details of the transition area are smoother.

[0023] In a second aspect, the present application provides a device for removing rainbow-like glare in an under-screen RGB image, which has the characteristics of comprising:

[0024] a processor;

[0025] a memory module having executable instructions of the processor stored therein;

[0026] The processor is configured to execute the steps of the method for removing rainbow-like glare in an under-screen RGB image according to any one of the above aspects by executing the executable instructions.

[0027] In a third aspect, the present application provides a computer-readable storage medium for storing a program, which has the characteristics that the program, when executed, implements the steps of the method for removing rainbow-like glare in an under-screen RGB image according to any one of the above aspects.

[0028] Compared with the prior art, the present application has the following beneficial effects:

[0029] The present application is directed to the characteristics of a larger screen under glare range and more colors, adopts the way of removing glare first and then fusing, so that other objects outside the glare are more clear and obvious in the image, and the influence of removing glare is removed.

[0030] The present application is directed to the case of at least two light sources connected, and the light sources are identified in combination with the characteristics between the light sources, so as to improve the identification effect.

[0031] The present application further processes the fused image, eliminates the stepwise defects in the screen under glare, and makes the image more natural and real. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings. Other features, objects and advantages of the present application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the following drawings:

[0033] Figure 1 The step flow chart of a method for removing rainbow-like glare of a screen under RGB image in an embodiment of the present application;

[0034] Figure 2 The first RGB image photographed by a screen under camera in an embodiment of the present application;

[0035] Figure 3 The schematic diagram of a light source connected in an embodiment of the present application;

[0036] Figure 4 The second RGB image in an embodiment of the present application;

[0037] Figure 5 The third RGB image in an embodiment of the present application;

[0038] Figure 6 The enlarged third RGB image in an embodiment of the present application;

[0039] Figure 7 The final image in an embodiment of the present application;

[0040] Figure 8 The step flow chart of a method for generating a third RGB image in an embodiment of the present application;

[0041] Figure 9A structure schematic diagram of an equipment for removing rainbow-like glare of an under-screen RGB image in an embodiment of the present application;

[0042] Figure 10 A computer readable storage medium schematic diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made. These all belong to the protection scope of the present application.

[0044] The terms "first", "second", "third", "fourth" and the like (if any) in the description, claims and above drawings of the present application are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0045] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0046] The embodiment of the present application provides a method for removing rainbow-like glare of an under-screen RGB image, aiming at solving the problems in the prior art.

[0047] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0048] The present application first removes at least two connected light sources and their rainbow-like glare by using a deep neural network model, then fuses the images before and after processing to obtain a third RGB image with light sources, and finally eliminates the stepwise defects on the third RGB image to obtain the final RGB image, thereby effectively removing the glare of multiple light sources in the RGB image shot by the under-screen camera.

[0049] Figure 1 A flow chart of steps of a method for removing rainbow-like glare from an RGB image under a screen in an embodiment of the present application. As shown in the figure, the method for removing rainbow-like glare from an RGB image under a screen in an embodiment of the present application includes the following steps. Figure 1

[0050] Step S1: Obtain a first RGB image, and remove at least two connected light sources and rainbow-like glare thereof using a deep neural network model to obtain a second RGB image.

[0051] In this step, the present embodiment removes glare and light sources at the same time, and then processes subsequent processing, so as to finally achieve the effect of removing only glare while retaining light sources. The first RGB image is an image captured by an under-screen camera. As shown in the figure, unlike the glare when a strong light is directly captured by a camera, the image captured by the under-screen camera shows a larger range of continuous reflection of the glare, and the color changes with the increase of the radius, similar to a rainbow. Further, in the peripheral area in the figure, the glare shows a weaker form, and the information of other objects can be clearly seen, which is equivalent to adding a mask to other objects, unlike the traditional glare in which other objects cannot be seen. In the present embodiment, the first RGB image is obtained by an under-screen camera or camera, and the screen is usually a glass screen. The camera or camera is at a certain distance from the glass screen, usually between 0.1 mm and 5 mm, which is adjusted according to different application scenarios. For example, the distance in the front camera of a mobile phone is less than the corresponding distance in a robot. Figure 2 Figure 2 As shown in the figure, when there are multiple light sources, the light sources can exist in multiple combinations.

[0052] As shown in the figure, when there are multiple light sources, the light sources can exist in multiple combinations. Figure 3 In the figure, a circle represents the effective edge of a light source in an image. It can be understood that although a circle is used to represent a light source in the figure, the light source in practice is not a circle, but shows an irregular shape, and the effective edge of the light source in the image is not a circle, but shows an irregular shape. Figure 3 Figure 3 Figure 3 ​​​​The effective edge is the edge of the light source that can be identified as a light source in the image and can cover other content. Within the effective edge, it is the effective area of the light source; outside the effective edge, it is the area not affected by the light source. Glare exists outside the effective edge. When the number of light sources is two, the light sources can be connected or have overlapping parts. When the number of light sources is three, there are various combinations of intersection and connection. When the sizes of the light sources are different, the order of the light sources will also affect the final effect, making the combination more diverse. When the three light sources are tangent, a hollow area appears in the middle; when the three light sources overlap, there can be no hollow area in the middle, but the entire middle becomes a light source part. On the image, the light source part is white. To make the effect of this specification more obvious, the subsequent part of this specification will use the effect of a single light source as an example.

[0053] In some embodiments, after obtaining the first RGB image, the position and shape of the light source are determined according to the color of the image, and then the deep neural network model is used for processing. Since the light source is white on the image, the approximate area of the light source can be determined according to the color of the image, that is, the position and shape of the light source are obtained. According to the position and shape of the light source, the number of light sources can be obtained, and then the deep neural network model is used for recognition, which can achieve faster and better results. At the same time, some white objects in the image may be selected in this step, and these white objects can be removed by the deep neural network model to identify the real light source.

[0054] The deep neural network model identifies the rainbow-shaped glare and can remove it. The deep neural network model not only needs to identify the area of the rainbow-shaped glare, but also needs to identify the intensity of the rainbow-shaped glare. As shown in Figure 4 The obtained second image not only removes the rainbow-shaped glare, but also retains the information of other objects, rather than removing the entire glare area.

[0055] In some embodiments, the deep neural network model is obtained by training a one-to-one corresponding data set of light source on / off. During training, each group of one-to-one corresponding data of light source on / off forms a training set. The number of samples in the training set can be any one of 10,000, 20,000, 30,000, 40,000, 50,000, 60,000, or any other value not listed. The data in the training set are all data samples taken under the screen. Each data pair is composed of an image with rainbow-shaped glare when the light source is on and a background image without light source when the light source is off. The deep neural network model can be any type, including but not limited to convolutional neural network, recurrent neural network, generative adversarial network, etc.

[0056] In some embodiments, the deep neural network model adopts an Encoder-BottleneckBlock-Decoder architecture. The Encoder is an encoder, which can be arbitrary content such as a model, etc. In this embodiment, the Encoder part extracts image features using a pre-trained VGGNet. The BottleneckBlock is used to construct the network, which adopts multiple residual modules stacked. In this embodiment, the residual modules are all provided with a channel attention mechanism to further improve the recognition effect and operation efficiency of the rainbow glare. The Decoder is a decoder that up-samples the output of the BottleneckBlock part to the original image resolution. Since the rainbow glare presents a characteristic of emitting outward from the light source, the residual module adopted in this embodiment can quickly and effectively identify the rainbow glare, so as to not only identify the rainbow glare without being affected by color changes, but also quickly calculate and improve efficiency. The essence of the attention mechanism is to locate the information of interest and suppress useless information, and the result is usually displayed in the form of a probability map or a probability feature vector, that is, to locate the light source information and weaken irrelevant information.

[0057] In some embodiments, the deep neural network is a convolutional neural network. For a CNN that inputs a 2-dimensional image, one dimension is the scale space of the image, i.e. length and width, and the other dimension is the channel. The channel attention mechanism divides a bypass branch after normal convolution operation. First, a Squeeze operation is performed to compress the spatial dimension, i.e. each two-dimensional feature map becomes a real number, which is equivalent to a pooling operation with a global receptive field, and the number of feature channels remains unchanged. Then, an Excitation operation is performed to generate a weight for each feature channel through a parameter w, which is learned to explicitly model the correlation between feature channels. After obtaining the weight of each feature channel, the weight is applied to each original feature channel, and based on a specific task, the importance of different channels can be learned.

[0058] In some embodiments, the deep neural network model, when trained, fits multiple light sources through data augmentation. In this embodiment, multiple light source images are obtained by copying, cutting, overlapping, and adding noise to the light source. When there is only single light source data in the training set, this embodiment can simulate the effect of multiple light sources, so as to obtain the image effect under multiple light sources and realize effective identification of multiple light sources with a smaller training set.

[0059] Step S2: unevenly fusing the at least two connected light sources in the first RGB image with the second RGB image to generate a third RGB image.

[0060] In this step, various segmentation methods can be used to segment at least two light sources in the first RGB image, including but not limited to threshold-based segmentation method, region-based segmentation method, edge-based segmentation method, and segmentation method based on specific theory, etc. The present embodiment does not limit the segmentation method, as long as the light source and the surrounding area of the light source can be segmented. As shown in FIG. 1C, in the third RGB image, the rainbow-like glare is removed, and the light source and the surrounding area of the light source are retained, so that the third RGB image is closer to the real scene. Figure 5

[0061] Step S3: eliminating the stepped defects in the third RGB image by using a neural network model.

[0062] In this step, the third RGB image is further processed to eliminate the stepped defects in the third RGB image. As shown in FIG. 1D, in the transition area, there are circles of stepped defects, and the granularity is strong, and there is still part of the rainbow-like glare. This step uses a deep learning model to process the stepped defects, which can further remove the rainbow-like glare while removing the stepped defects. Figure 6

[0063] In some embodiments, the neural network model is a generative adversarial network model. The generative adversarial network is trained by back propagation. During the training or generation of samples, any Markov chain or unfolded approximate inference network is not required. The generative adversarial network includes a generator and a discriminator; in the training set, the label of the real image is true, and the label of the fused image is false, so as to cross-train the discriminator and the generator; finally, the trained generator is used to adjust the third RGB image, so that the details of the transition area are smoother. Figure 7 For the image adjusted by the present embodiment, it can be seen that the stepped defects have been eliminated, and the rainbow-like glare has been further reduced.

[0064] The following is the optimization objective when training the generative adversarial network, where x and y are the fused image and the real image respectively, D Y and G are the discriminator and the generator respectively, and the goal of training the generative adversarial network model is to maximize L GAN .

[0065]

[0066] Figure 8 A step flow chart for generating a third RGB image in an embodiment of the present application is shown in FIG. 1E. Figure 8 As shown in FIG. 1E, a method for generating a third RGB image in an embodiment of the present application includes:

[0067] Step S21: segmenting the light source in the first RGB image to generate a mask.​​

[0068] In this step, when the light source is acquired, a threshold segmentation method can be used to segment the area with high brightness to obtain the light source area. In the RGB image, white is (255, 255, 255), that is, the R channel, the G channel and the B channel are all 255. When performing threshold judgment, the R, G and B channels are judged respectively, and when the values of the three channels are all greater than a preset threshold, the light source area is judged. The preset threshold is a fixed value, including but not limited to 255, 254, 253, 252, 251 and other multiple arbitrary set values. The preset threshold is usually greater than 250. The thresholds of the R channel, the G channel and the B channel can be different. The higher the threshold, the smaller the range of the obtained light source area. The lower the threshold, the larger the range of the obtained light source area. A mask is generated for the identified light source.

[0069] Step S22: Uneven expansion is performed on the mask, and a weight is set for each expansion. In the process of diffusion from the light source edge to the effective edge, the weight gradually decays from 1 to 0, so as to obtain a single-channel weight alpha map.

[0070] In this step, the mask is expanded outward by turns, and a unique weight is set for each turn. The weight value of the light source edge is 1, and the weight value of the diffusion end is 0. The weights of different turns are different, and the farther away from the light source, the smaller the weight. The decay of the weight is nonlinear. Due to the irregular shape of the light source, the expansion is also irregular and uneven. According to the weight distribution, a single-channel weight alpha map representing the weight can be obtained, which can be used for image fusion calculation. The part formed by the strong light source on the first RGB image is defined from inside to outside as a light source, a light source periphery and a glare part. The light source is a strong area with brightness close to white light, which is the area where the light source itself is located. The light source periphery is a natural area formed when the light source shines outward, rather than light formed by diffusion of lenses, glass sheets and the like. The glare part is the glare part formed by diffusion of lenses, glass sheets and the like, mainly in the form of rainbow-like glare. The three can be distinguished by brightness values.

[0071] In some embodiments, the weight where d is the shortest distance between a pixel point on the image and the light source edge, d max is the shortest distance between the effective edge and the light source edge. With the change of the pixel point, the value of d changes between [0, d max ]. When d = 0, a = 1. When d = d max , a = 0. This embodiment can ensure that the change of the weight is more suitable for the perception of light by the human eye, so that the fusion effect is more realistic.

[0072] In some embodiments, the expansion speed in different directions is determined according to the shape of the light source. When the light source is perpendicular to the camera or the camera head, the expansion speed in each direction in this step is the same. When the light source is not perpendicular to the camera or the camera head, the light source is not circular on the image, but presents an ellipse or other shapes, and this embodiment can adjust the expansion speed in different directions according to the angle of the light source, so that the expansion is more uniform.

[0073] In some embodiments, the line width at different positions during the expansion is determined according to the angle of the light source. When the light source is perpendicular to the camera or the camera head, the line width at different positions during the expansion in each direction in this step is the same. When the light source is not perpendicular to the camera or the camera head, the light source is not circular on the image, but presents an ellipse or other shapes, and this embodiment can adjust the line width at different positions during the expansion according to the angle of the light source, so that the expansion is more uniform.

[0074] In some embodiments, the line width at different positions during the expansion is determined according to the distance from the center of the light source to the edge of the light source. When the light source is perpendicular to the camera or the camera head, the line width at different positions during the expansion in each direction in this step is the same. When the center of the light source is closer to the edge of the light source, the line during the expansion is thinner; when the center of the light source is farther away from the edge of the light source, the line during the expansion is wider, so that each line reaches the edge of the light source at the same time.

[0075] Step S23: multiplying the weight alpha map and the light source edge diffusion to the effective edge in the first RGB image, and then fusing with the second RGB image to generate a third RGB image.

[0076] In this step, the weight alpha map is multiplied by the light source and the light source periphery in the first RGB image, and then added to the second RGB image to obtain a third RGB image. The third RGB image retains the light source and the light source periphery, and removes the rainbow-like glare, to obtain a more realistic image.

[0077] This embodiment can retain the strong area of the light source, remove the rainbow-like glare, and can be adjusted for different light source angles, and has wide applicability.

[0078] The application also provides an equipment for removing rainbow-like glare of an under-screen RGB image, which comprises a processor and a memory having executable instructions of the processor stored therein. The processor is configured to execute the steps of the method for removing rainbow-like glare of an under-screen RGB image.

[0079] As described above, the present application first removes the rainbow-like glare by using a deep neural network model, then fuses the images before and after processing to obtain a third RGB image with a light source, and then eliminates the step defect on the third RGB image to obtain the final RGB image, thereby effectively removing the glare in the RGB image captured by the under-screen camera.

[0080] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "platform" here.

[0081] Figure 9 is a structural schematic diagram of a device for removing rainbow-like glare of an under-screen RGB image in an embodiment of the present application. The electronic device 600 according to this embodiment of the present application will be described below with reference to Figure 9 Figure 9 The electronic device 600 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0082] As shown in Figure 9 , the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0083] The storage unit stores program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present application described in the above-mentioned method for removing rainbow-like glare of an under-screen RGB image in the present specification. For example, the processing unit 610 can perform the steps as shown in Figure 1

[0084] The storage unit 620 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 6201 and / or a cache memory unit 6202, and can further include a read-only memory (ROM) 6203.

[0085] The storage unit 620 can further include a program / utility 6204 having a set of program modules 6205, such as an operating system, one or more application programs, other program modules, and program data, each of which can include implementation of a network environment, alone or in combination with some of the examples. ​​

[0086] Bus 630 can be one of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, a graphics bus, a processor or local bus using any of a variety of bus structures, etc.

[0087] Electronic device 600 can also communicate with one or more external devices 700 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices associated with electronic device 600; and / or one or more devices that enable a user to interact with electronic device 600; and / or one or more devices that enable electronic device 600 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface 650. Still yet, electronic device 600 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network such as the Internet, via network adapter 660. As depicted, network adapter 660 can communicate with the other components of electronic device 600 via bus 630 although it is understood that many other architectures can be implemented to enable communication between network adapter 660 and other components of electronic device 600. Figure 9 It should be understood that although not shown, other hardware and / or software modules could be used in conjunction with electronic device 600. Such hardware would include, but is not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0088] The embodiment of the present application also provides a computer readable storage medium for storing a program, the program being executed to implement the steps of the method for removing rainbow-like glare of an under-screen RGB image. In some possible implementation manners, various aspects of the present application can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps described in the above method for removing rainbow-like glare of an under-screen RGB image according to various exemplary embodiments of the present application when the program product is run on the terminal device.

[0089] As shown above, the program of the computer readable storage medium of the embodiment, when executed, first removes at least two connected light sources and rainbow-like glare thereof by using a deep neural network model, then fuses the images before and after processing to obtain a third RGB image with light sources, and finally eliminates the stepwise defects on the third RGB image to obtain a final RGB image, thereby effectively removing the glare of multiple light sources in the RGB image captured by an under-screen camera.

[0090] Figure 10 is a structural schematic diagram of the computer readable storage medium in the embodiment of the present application. Referring to Figure 10As shown, a program product 800 for implementing the above-described method according to an embodiment of the present application is described, which can take the form of a portable compact disc read-only memory (CD-ROM) and includes a program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in the present document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0091] The program product can take any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0092] The computer readable storage medium can include a data signal transported, propagated or transmitted, in baseband or as part of a carrier wave, in which readable program code is embodied. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. The readable storage medium can also be any readable medium that is capable of storing, transmitting or transferring a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including, but not limited to, wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0093] The program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., or conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0094] In the embodiments of the present application, the at least two connected light sources and the rainbow-like glare thereof are removed by using a deep neural network model, then the images before and after the processing are fused to obtain a third RGB image with light sources, and then the step defects on the third RGB image are eliminated to obtain a final RGB image, thereby effectively removing the glare of multiple light sources in the RGB image captured by the under-screen camera.

[0095] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the various embodiments can be referred to each other. The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0096] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application.

Claims

1. A method for removing rainbow-like glare from under-screen RGB images, characterized in that, Includes the following steps: Step S1: Obtain a first RGB image, and use a deep neural network model to remove at least two connected light sources and their rainbow-shaped glare from the first RGB image to obtain a second RGB image; the rainbow-shaped glare presents a closed diffusion shape, and the light intensity also gradually changes; the part formed by the strong light source on the first RGB image is defined from the inside out as the light source, the light source periphery, and the glare part; Step S2: Non-uniformly fuse the at least two connected light source regions in the first RGB image with the second RGB image to generate a third RGB image; Step S3: Eliminate the step defects in the third RGB image using a neural network model; Step S2 includes: Step S21: Segment the light source region in the first RGB image to generate a mask; Step S22: The mask is expanded unevenly, and a weight is set for each expansion. As the expansion spreads from the edge of the light source to the effective edge, the weight gradually decreases from 1 to 0, thus obtaining a single-channel weight alpha map; the effective edge is the outer edge around the light source. Step S23: Multiply the weighted alpha image with the light source and the area surrounding the light source in the first RGB image, and then add it to the second RGB image to obtain the third RGB image; In step S22, weights Where d is the shortest distance between a pixel in the image and the edge of the light source. It is the shortest distance between the effective edge and the edge of the light source.

2. The method for removing rainbow-like glare from under-screen RGB images according to claim 1, characterized in that, In step S1, after acquiring the first RGB image, the position and shape of the light source are determined based on the color of the image, and then processed using the deep neural network model.

3. The method for removing rainbow-like glare from under-screen RGB images according to claim 1, characterized in that, The deep neural network model adopts an Encoder-BottleneckBlock-Decoder architecture. The Encoder part uses a pre-trained VGGNet to extract image features, the BottleneckBlock part uses multiple residual modules stacked together, and each residual module has a channel attention mechanism. The Decoder part upsamples the output of the BottleneckBlock part to the original image resolution.

4. The method for removing rainbow-like glare from under-screen RGB images according to claim 1, characterized in that, During training, the deep neural network model fits an image containing multiple light sources through data augmentation.

5. The method for removing rainbow-like glare from under-screen RGB images according to claim 1, characterized in that, In step S22, the expansion speed in different directions is determined according to the shape of the light source.

6. The method for removing rainbow-like glare from under-screen RGB images according to claim 1, characterized in that, In step S3, the neural network model is a generative adversarial network model; the generative adversarial network includes a generator and a discriminator; in the training set, the real image is labeled as true and the fused image is labeled as false, so as to cross-train the discriminator and the generator; finally, the trained generator is used to adjust the third RGB image to make the details in the transition region smoother.

7. A device for removing rainbow-like glare from under-screen RGB images, characterized in that, include: processor; A memory module that stores executable instructions of the processor; The processor is configured to perform the steps of the method for removing rainbow glare from under-screen RGB images according to any one of claims 1 to 6 by executing the executable instructions.

8. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, it implements the steps of the method for removing rainbow glare from under-screen RGB images as described in any one of claims 1 to 6.

Images