Image processing method and device, image processing model training method and device
By designing non-repetitively arranged light-emitting units and light-transmitting areas under the display screen, and combining them with an image processing model, the problem of poor display effect in under-display camera technology has been solved, and a significant improvement in image clarity has been achieved.
Patent Information
- Application Number
- CN202010179545.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-01-20
- Filing Date
- 2020-03-13
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-03-13
AI Technical Summary
In existing under-display camera technologies, the display effect of the screen is poor, and the clarity after image restoration is not good, which is difficult to improve effectively.
By designing non-repetitively arranged light-emitting units and light-transmitting areas, and combining them with an image processing model, the original diffraction image is restored, removing diffraction fringes and improving clarity.
It simplifies the image restoration process, significantly improves the clarity of the restored image, and enhances the display effect of the screen.
Smart Images

Figure CN113139911B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and device, and an image processing model training method and device. Background Art
[0002] With the development of mobile terminal technology and user needs, full-screen terminals have become an important development trend. In related technologies, a terminal device is provided with a front camera, and a slot or hole is generally provided at the position where the front camera is installed on the display screen of the terminal device, so that the front camera can capture external images. However, the slot or hole formed on the display screen of the terminal device reduces the screen-to-body ratio of the display screen.
[0003] For mobile terminals that use full-screen displays, under-screen cameras have gradually become the best solution to achieve full-screen displays. Under-screen cameras hide the front camera under the display without a hole in the display. When in use, the camera can frame and shoot through the light-transmitting area of the display.
[0004] However, the inventors have discovered through research that in existing under-screen camera solutions, the display effect of the display screen is poor. Summary of the invention
[0005] In view of this, an object of the present invention is to provide an image processing method and device, and an image processing model training method and device, which can effectively improve the quality of the restored image and enhance the clarity of the image.
[0006] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:
[0007] In a first aspect, an embodiment of the present invention provides an image processing method, which is applied to an electronic device, and comprises: acquiring an original diffraction image; inputting the original diffraction image into an image processing model; and restoring the original diffraction image through the image processing model to obtain a target standard image corresponding to the original diffraction image.
[0008] Furthermore, the step of restoring the original diffraction image through the image processing model to obtain a target standard image corresponding to the original diffraction image includes: detecting the brightness value of each pixel in the original diffraction image through the image processing model; determining a light spot area containing a target light source in the original diffraction image based on the detected brightness value; and restoring the original diffraction image based on the light spot area to obtain a target standard image corresponding to the original diffraction image.
[0009] Furthermore, the step of restoring the original diffraction image based on the light spot area includes: removing diffraction fringes from the light spot area to obtain an image to be restored corresponding to the original diffraction image; and performing clarity processing on the image to be restored to obtain a target standard image.
[0010] Furthermore, the step of determining the light spot area containing the target light source in the original diffraction image based on the detected brightness value includes: determining the brightness area on the original diffraction image according to the positions of the pixels whose detected brightness values are greater than a preset brightness threshold; judging whether the radius of the circumscribed circle of the brightness area is greater than a preset radius; and if so, determining the brightness area as the light spot area containing the target light source.
[0011] Furthermore, the step of obtaining the original diffraction image includes: collecting the original diffraction image through an under-screen camera of the electronic device.
[0012] Furthermore, the image processing model is trained based on image sample pairs, and the image sample pairs include a sample standard image of a specified scene photographed by an on-screen camera and a sample diffraction image corresponding to the sample standard image; wherein the sample diffraction image is an image obtained by simulating the under-screen camera photographing the specified scene based on the sample standard image, or is an image obtained by photographing the specified scene by the under-screen camera.
[0013] Furthermore, the electronic device includes a display screen, which includes multiple light-emitting units and multiple light-transmitting areas; wherein each of the light-emitting units includes a preset number of sub-pixels; the multiple sub-pixels in the multiple light-emitting units are arranged at intervals to form multiple light-transmitting areas between the sub-pixels, and the multiple light-transmitting areas include at least two non-repetitive first light-transmitting areas.
[0014] Furthermore, any sub-pixel in the plurality of light-emitting units is separated from any light-transmitting area in the plurality of light-transmitting areas.
[0015] Furthermore, the light-transmitting areas are arranged in one or more of the following ways:
[0016] At least two of the first light-transmitting areas have different size parameters, shape parameters, posture parameters, and position distribution parameters;
[0017] Each of the first light-transmitting areas has different size parameters, shape parameters, posture parameters, and position distribution parameters from other light-transmitting areas;
[0018] All the light-transmitting areas have different size parameters, shape parameters, posture parameters, and position distribution parameters.
[0019] Furthermore, the electronic device includes a display screen, which includes a plurality of light-emitting units and a plurality of light-transmitting areas; wherein each of the light-emitting units includes a preset number of sub-pixels; and the plurality of sub-pixels of at least two of the light-emitting units are distributed in a non-repetitive manner.
[0020] Furthermore, at least two non-repeatingly distributed light-transmitting areas are formed in the gaps between the non-repeatingly distributed sub-pixels.
[0021] Furthermore, the light emitting units are arranged in one or more of the following ways:
[0022] The multiple sub-pixels of at least two light-emitting units have different size parameters, shape parameters, posture parameters, and position distribution parameters;
[0023] The multiple sub-pixels of at least two light-emitting units have different size parameters, shape parameters, posture parameters, and position distribution parameters from the multiple sub-pixels of other light-emitting units;
[0024] The multiple sub-pixels of all light-emitting units have different size parameters, shape parameters, posture parameters, and position distribution parameters.
[0025] Furthermore, the electronic device is an electronic device with an under-screen camera.
[0026] In a second aspect, an embodiment of the present invention further provides a method for training an image processing model, the method comprising: inputting an image sample pair into the image processing model, wherein the image sample pair comprises a sample standard image and a sample diffraction image corresponding to the sample standard image; restoring the sample diffraction image through the image processing model to obtain a restored image of the sample diffraction image; determining a loss function value corresponding to the image processing model based on the restored image and the sample standard image; and iteratively updating the parameters of the image processing model based on the loss function value.
[0027] Furthermore, the parameters of the image processing model are iteratively updated based on the loss function value, including: determining whether the loss function value converges to a preset value, and / or whether the iterative update reaches a preset number of times; when the loss function value converges to a preset value, and / or the iterative update reaches a preset number of times, a trained image processing model is obtained.
[0028] Further, determining the loss function value corresponding to the image processing model based on the restored image and the sample standard image includes: calculating the similarity between the restored image and the sample standard image, and determining the loss function value corresponding to the image processing model based on the similarity.
[0029] Furthermore, the method for acquiring the image sample pair includes: photographing a specified scene through an on-screen camera to obtain the sample standard image; photographing a target light source in a dark background through a display screen through the on-screen camera to obtain a target light source image; and performing a convolution operation on the target light source image and the sample standard image to obtain the sample diffraction image.
[0030] Further, photographing the target light source in a dark background through the display screen by the on-screen camera to obtain a target light source image includes: photographing the target light source in a preset scheme through the display screen by the on-screen camera to obtain a candidate target light source image; wherein the preset scheme is a scheme for spatially arranging at least one target light source in a dark background, and in different preset schemes, the number of target light sources and / or the spatial arrangement of the target light sources are different, and the candidate target light source images corresponding to different preset schemes are different; and at least one of the candidate target light source images is determined as the target light source image.
[0031] Furthermore, before the step of performing a convolution operation on the target light source image and the sample standard image, the method further includes: performing a noise reduction process on the target light source image.
[0032] Furthermore, the display screen is the same display screen as the display screen of the electronic device in the above-mentioned image processing method.
[0033] Furthermore, the method for acquiring the image sample pair includes: photographing a designated scene at a preset shooting angle through an on-screen camera to obtain the sample standard image; photographing the designated scene at the shooting angle through an under-screen camera to obtain the sample diffraction image.
[0034] In a third aspect, an embodiment of the present invention provides an image processing device, which is applied to an electronic device, and includes: an image acquisition module, used to obtain an original diffraction image; an image input module, used to input the original diffraction image into an image processing model; and an image restoration module, used to restore the original diffraction image through the image processing model to obtain a target standard image corresponding to the original diffraction image.
[0035] In a fourth aspect, an embodiment of the present invention provides a training device for an image processing model, the device comprising: an input module for inputting an image sample pair into the image processing model, wherein the image sample pair comprises a sample standard image and a sample diffraction image corresponding to the sample standard image; a restoration module for restoring the sample diffraction image through the image processing model to obtain a restored image of the sample diffraction image; a calculation module for determining a loss function value corresponding to the image processing model based on the restored image and the sample standard image; and an update module for iteratively updating the parameters of the image processing model based on the loss function value.
[0036] In a fifth aspect, an embodiment of the present invention provides an image processing system, the system comprising a processor and a storage device; the storage device stores a computer program, and when the computer program is run by the processor, it executes the method as described in any one of the first aspect, or executes the method as described in any one of the second aspect.
[0037] In the sixth aspect, an embodiment of the present invention provides an electronic device, comprising a display screen and an under-screen camera, and also comprising an image processing system as described in the fifth aspect; the display screen comprises a plurality of light-emitting units and a plurality of light-transmitting areas; wherein each of the light-emitting units comprises a plurality of sub-pixels.
[0038] Furthermore, there are gaps between the multiple sub-pixels in the multiple light-emitting units, so as to form the multiple light-transmitting regions in the gaps, and the multiple light-transmitting regions include at least two non-repetitive first light-transmitting regions.
[0039] Furthermore, the multiple sub-pixels of at least two of the light-emitting units are distributed in a non-repetitive manner.
[0040] In the seventh aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any one of the first aspect above are executed, or the steps of the method described in any one of the second aspect are executed.
[0041] An embodiment of the present invention provides an electronic device, which includes a display screen and an under-screen camera; wherein the display screen includes multiple light-emitting units and multiple light-transmitting areas; each light-emitting unit includes a preset number of sub-pixels; the multiple light-transmitting areas are non-repetitively arranged between the sub-pixels of the multiple light-emitting units, so that the diffraction stripe image generated by the target light source through the display screen is a uniformly distributed stripe image; it is easy to accurately judge the regularity of the uniformly distributed stripe image, thereby reducing the difficulty of image restoration during image processing.
[0042] The embodiment of the present invention provides an image processing method and device, wherein the image processing method inputs the acquired original diffraction image into an image processing model, and restores the original diffraction image through the image processing model to obtain a target standard image corresponding to the original diffraction image. The above-mentioned image processing method provided by this embodiment can directly restore the original diffraction image using the image processing model, effectively simplifies the image restoration method and can improve the clarity of the restored target standard image, thereby effectively improving the display effect of the display screen on the image.
[0043] The embodiment of the present invention provides a training method and device for an image processing model. The training method inputs an image sample pair into the image processing model, performs restoration processing on the sample diffraction image through the image processing model, and obtains a restored image of the sample diffraction image; according to the restored image and the sample standard image, determines the loss function value corresponding to the image processing model, and iteratively updates the parameters of the image processing model. In the above training method provided by this embodiment, the sample standard image and the sample diffraction image with a corresponding relationship are used as training data, which can reduce the difference between the two images in the image sample pair, that is, improve the quality of the image sample pair. The image sample pair with higher quality helps to improve the training effect of the image processing model, and can improve the training efficiency and accuracy of the image processing model.
[0044] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.
[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown;
[0048] Figure 2 A schematic diagram showing the structure of a display screen provided by an embodiment of the present invention is shown;
[0049] Figure 3 A flowchart of an image processing method provided by an embodiment of the present invention is shown;
[0050] Figure 4 A schematic diagram of an original diffraction image provided by an embodiment of the present invention is shown;
[0051] Figure 5 A schematic diagram of a target standard image provided by an embodiment of the present invention is shown;
[0052] Figure 6 A flow chart of a training method for an image processing model provided by an embodiment of the present invention is shown;
[0053] Figure 7 A schematic diagram of a target light source image shooting scene provided by an embodiment of the present invention is shown;
[0054] Figure 8 A structural block diagram of an image processing device provided by an embodiment of the present invention is shown;
[0055] Fig. 9 A structural block diagram of a training device for an image processing model provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] Generally, the sub-pixels in the light-emitting units of the display screen are arranged repeatedly. For example, the sub-pixels in each of the multiple light-emitting units are arranged exactly the same, or the multiple light-emitting units are used as a pixel module, and the sub-pixels in each pixel module composed of the multiple light-emitting units are arranged exactly the same. However, for the under-screen camera, when the external target light source passes through the screen, the image collected by the under-screen camera will form "raindrop"-shaped diffraction stripes, which are non-uniformly attenuated from the center to the outside, resulting in non-uniformly distributed blurring in the image taken by the camera. In the image restoration process, since the non-uniform diffraction stripes are difficult to remove, the clarity of the restored image is very poor, which seriously affects the quality of the restored image.
[0058] In the solution of the under-screen camera, the inventors have found that the structure of a general display screen will form non-uniform diffraction stripes. Since the regularity of the diffraction stripes cannot be accurately judged, it is impossible to accurately identify and eliminate the diffraction stripes in the image, which makes it difficult to restore the image containing the target light source. The clarity of the restored image is very poor, which seriously affects the quality of the restored image. Based on this, in order to improve at least one of the above problems, the embodiments of the present invention provide an image processing method and device, and an image processing model training method and device, which can effectively improve the quality of the restored image and enhance the clarity of the image. The technology can be applied to various under-screen camera products, such as mobile phones, computers, cameras, and biomedical imaging equipment. For ease of understanding, the embodiments of the present invention are described in detail below.
[0059] Embodiment 1:
[0060] First, refer to Figure 1 An example electronic device 100 for implementing the image processing method and apparatus, and the image processing model training method and apparatus according to the embodiments of the present invention is described.
[0061] like Figure 1 The electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, an output device 108, and an image acquisition device 110. These components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are only exemplary and not restrictive. The electronic device may have Figure 1 Some of the components shown may also have Figure 1 Other components and structures are not shown.
[0062] The processor 102 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.
[0063] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.
[0064] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.
[0065] The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.
[0066] The image acquisition device 110 can capture images desired by the user (eg, photos, videos, etc.), and store the captured images in the storage device 104 for use by other components.
[0067] Exemplarily, an example electronic device for implementing an image processing method and apparatus, and an image processing model training method and apparatus according to an embodiment of the present invention can be implemented on an intelligent terminal such as a smart phone, a tablet computer, or a computer.
[0068] Embodiment 2:
[0069] This embodiment provides an image processing method, which can be applied to an electronic device. In order to better understand the technical solution of the present disclosure, the electronic device is first described based on the above embodiment.
[0070] In one possible structure, the electronic device may include a display screen, as shown in FIG. Figure 2 The structural schematic diagram of the display screen shown in FIG. 1 shows a display screen that may include multiple light-emitting units and multiple light-transmitting areas. Each light-emitting unit includes a preset number of sub-pixels; Figure 2In the example of the light-emitting unit enlarged on the left, each light-emitting unit may include three sub-pixels of R (red), G (green), and B (blue); of course, the light-emitting unit may also be in other composition forms, such as including four sub-pixels of R (red), G (green), B (blue), and W (white), which is not limited in this embodiment.
[0071] In practical applications, multiple light-emitting units can be arranged in a matrix, a herringbone shape, etc. This arrangement is the same as the arrangement of light-emitting units in an existing conventional display screen (i.e., a display screen without an under-screen camera), so that the display screen can be directly manufactured using existing technology to avoid possible technical difficulties. Moreover, the display screen in this embodiment can have a display effect similar to that of a display screen without an under-screen camera, which is conducive to providing users with a better visual experience. Of course, multiple light-emitting units can also be arranged in other regular or irregular ways, which is not limited in the embodiments of the present invention.
[0072] Considering that when an external target light source passes through the display screen, the repeatedly arranged light-transmitting areas on the existing display screen are likely to cause non-uniform diffraction stripes and affect the effect of photographing the image. Based on this, there are gaps between the multiple sub-pixels in the multiple light-emitting units to form multiple light-transmitting areas in the gaps, and the multiple light-transmitting areas include at least two non-repetitive first light-transmitting areas.
[0073] It can be understood that in some embodiments, the gap can be formed between multiple sub-pixels of the same light-emitting unit, or a gap can be formed between sub-pixels of two light-emitting units. Among the multiple sub-pixels within the same light-emitting unit, at least two sub-pixels may be connected and there is no gap. Among the multiple sub-pixels, there may be one or more light-transmitting parts in the gap between two adjacent sub-pixels, or there may be no light-transmitting part. By way of example, the multiple sub-pixels of the multiple light-emitting units are separated to form a gap in the separated area of the multiple sub-pixels. One or more light-transmitting parts are arranged in the gap between some sub-pixels of the multiple sub-pixels of the multiple first light-emitting units. By way of example, there is no gap between the multiple sub-pixels of at least one light-emitting unit, and the edges are connected to each other; or the multiple sub-pixels of different light-emitting units are not separated, and the edges are connected.
[0074] For example, the display screen is divided into a plurality of light-emitting units and non-light-emitting areas between the light-emitting units, and the plurality of light-transmitting areas are located in the non-light-emitting areas, and include at least two non-repetitive first light-transmitting areas. Specifically, the plurality of first light-transmitting areas may be non-repetitively divided, or the plurality of light-transmitting areas may be non-repetitively divided or non-uniformly divided relative to the plurality of pixel areas.
[0075] Regarding the non-repetitive distribution of the plurality of first light-transmitting regions, they may be arranged in one or more of the following ways:
[0076] In a first manner, at least two first light-transmitting areas have different size parameters, shape parameters, posture parameters, and position distribution parameters;
[0077] In the second mode, each first light-transmitting area has different size parameters, shape parameters, posture parameters, and position distribution parameters from other light-transmitting areas;
[0078] In a third manner, all light-transmitting areas have different size parameters, shape parameters, posture parameters, and position distribution parameters.
[0079] Among them, different size parameters refer to the size differences of the light-transmitting areas; different shape parameters refer to the different shapes of the light-transmitting areas, such as circle, rectangle, polygon, etc.; different posture parameters refer to the light-transmitting areas having different rotation angles; different position distribution parameters refer to the arrangement of the light-transmitting areas being non-aligned and having a certain misalignment deviation.
[0080] I will not list them one by one here. It can be seen that the relative position relationship between the light-transmitting areas is irregular, the relative position relationship between the light-transmitting areas and the sub-pixels in the corresponding light-emitting units is irregular, and the relative position relationship between sub-pixels of the same type (such as R sub-pixels) is irregular, that is, the arrangement of the light-transmitting areas is random and non-repetitive. The arrangement relationship between the light-transmitting areas and the sub-pixels described above means that at the visual level, the light-transmitting areas and the sub-pixels are both on the light-emitting surface of the display screen, so the light-transmitting areas and the sub-pixels can be regarded as being on the same two-dimensional plane, and the hierarchical structure of the cathode, anode, light-emitting material, etc. that constitute the sub-pixel is not limited.
[0081] In addition, it is understandable that in order to ensure the display effect of the display screen, the above-mentioned light-transmitting area and sub-pixel cannot overlap, that is, any sub-pixel in the multiple light-emitting units is separated from any light-transmitting area in the multiple light-transmitting areas.
[0082] In another embodiment, in order to avoid non-uniform diffraction fringes caused by an external target light source passing through the display screen, the display screen is configured such that a plurality of sub-pixels of at least two light-emitting units are distributed in a non-repetitive manner.
[0083] Regarding the non-repetitive distribution of multiple sub-pixels in the light-emitting unit, the arrangement may be performed in one or more of the following ways:
[0084] In the first mode, the multiple sub-pixels of at least two light-emitting units have different size parameters, shape parameters, posture parameters, and position distribution parameters;
[0085] In a second manner, the multiple sub-pixels of at least two light-emitting units have different size parameters, shape parameters, posture parameters, and position distribution parameters from the multiple sub-pixels of other light-emitting units;
[0086] In a third manner, multiple sub-pixels of all light-emitting units have different size parameters, shape parameters, posture parameters, and position distribution parameters.
[0087] Among them, different size parameters refer to the size differences of sub-pixels; different shape parameters refer to the different shapes of sub-pixels, such as circle, rectangle, polygon, etc.; different posture parameters refer to the different rotation angles of sub-pixels; different position distribution parameters refer to the arrangement of sub-pixels is not aligned and has a certain misalignment deviation.
[0088] Furthermore, the above two methods of avoiding non-uniform diffraction fringes may be combined to form at least two non-repetitively distributed light-transmitting areas in the gaps between the non-repetitively distributed sub-pixels.
[0089] By forming multiple non-repetitive first light-transmitting areas between sub-pixels, or distributing multiple sub-pixels of at least two light-emitting units in a non-repetitive manner, the brightness of the diffraction stripes formed by the external target light source through the light-transmitting port can be evenly distributed. In this case, the diffraction stripe image generated by the target light source through the display screen can be an evenly distributed stripe image, so that the image captured by the under-screen camera through the display screen is a blurred image showing an evenly distributed phenomenon; it is easy to accurately judge the regularity of the evenly distributed stripe image, thereby reducing the difficulty of image restoration during image processing, and the above-mentioned blurred image can be restored by a simple image processing method. It should be noted that the target light source can usually be a point light source, a linear light source, or other light source that is easy to produce diffraction.
[0090] Based on the display screen of the above structure, the electronic device in this embodiment can also be an electronic device with an under-screen camera. The display screen can be an OLED display screen, and the area where the under-screen camera is located is a transparent OLED display screen. When the area does not display the picture, it will be transparent, so that the external ambient light can pass through the transparent OLED display screen to reach the under-screen camera, thereby finally realizing imaging. Based on the positional relationship between the camera and the OLED display screen, it is equivalent to hiding the camera under the OLED display screen, so the camera can be called an under-screen camera. In addition, a circuit layer, a substrate layer and other structures may be provided between the OLED display screen and the under-screen camera.
[0091] In actual applications, the under-screen camera can be a camera inside an electronic device, that is, the electronic device, the display screen and the under-screen camera are an integrated structure; in addition, the under-screen camera can also be a camera independent of the electronic device, such as an independent camera structure or a camera in other devices, that is, the under-screen camera and the electronic device with a display screen are a combined structure.
[0092] According to the electronic device provided in the above embodiment, the embodiment of the present invention provides an image processing method using the electronic device. Figure 3 The image processing method flow chart shown in the figure, the method specifically includes the following steps S302 to S306:
[0093] Step S302, obtaining an original diffraction image. The original diffraction image may be an image captured by an under-screen camera of an electronic device in an actual shooting scene. Since the under-screen camera is disposed under the display screen, it can be considered that the under-screen camera is an original diffraction image captured through the display screen. The above shooting scene is any scene with light, such as a scene with a target light source. Taking the shooting scene with a target light source as an example, the following can be provided: Figure 4 The schematic diagram of the original diffraction image shown in the figure shows that there are obvious diffraction stripes in the target light source area in the original diffraction image, and the original diffraction image is relatively blurred as a whole. Of course, based on the physical meaning of diffraction, it can be understood that in a shooting scene where there is no target light source, diffraction stripes will also appear in the original diffraction image taken by the under-screen camera.
[0094] Step S304: input the original diffraction image into an image processing model, wherein the image processing model is a neural network model such as LeNet, R-CNN (Region-CNN) or Resnet.
[0095] In practical applications, the image processing model is pre-trained based on image sample pairs; the image sample pairs include a sample standard image and a sample diffraction image corresponding to the same scene. The sample standard image can be understood as an image obtained by shooting a specified scene with an on-screen camera; the on-screen camera should not be simply considered as a camera set above the display screen, it is just "on-screen" defined relative to the above-mentioned under-screen camera. Usually, the on-screen camera can be a conventional shooting device in production applications, such as a camera, a rear camera of a mobile phone, and so on. Since the sample standard image is an image shot by the on-screen camera, the sample standard image can also be called an on-screen image; the on-screen camera will not be adversely affected by the display screen on shooting. In this case, the sample standard image is a high-quality image with good clarity.
[0096] The sample diffraction image is an image obtained by simulating the under-screen camera to shoot a specified scene based on the sample standard image, or an image obtained by shooting a specified scene with the under-screen camera. Since the sample diffraction image is an image shot by the under-screen camera or a simulated under-screen camera, the sample diffraction image can also be called an under-screen image, and the sample diffraction image is generally a blurred image containing diffraction fringes.
[0097] It should be noted that the under-screen camera used to capture and obtain the sample diffraction image in this step is not necessarily the same under-screen camera as the under-screen camera used to capture the original diffraction image in step S302.
[0098] Step S306, restoring the original diffraction image through the image processing model to obtain a target standard image corresponding to the original diffraction image.
[0099] In a possible implementation, the diffraction fringes in the original diffraction image can be eliminated by an image processing model, and then the image after the diffraction fringes are eliminated can be restored to obtain a target standard image with higher definition. The obtained target standard image can be referred to Figure 5 As shown, it is the restored image corresponding to the original diffraction image, and the clarity is significantly improved.
[0100] The above-mentioned image processing method provided by the embodiment of the present invention can directly use the image processing model to restore the original diffraction image, effectively simplifying the image restoration method and improving the clarity of the restored target standard image, thereby effectively improving the display effect of the display screen on the image.
[0101] For ease of understanding, this embodiment describes the restoration method of the original diffraction image in step S306. The image processing model can restore the input original diffraction image based on a preset image restoration algorithm (such as Wiener filtering, regularized filtering, and blind area convolution, etc.) to obtain a target standard image.
[0102] Compared with the original diffraction image that does not contain the target light source, the phenomenon of diffraction fringes appearing in the original diffraction image that contains the target light source will be more obvious. In order to better improve the restoration effect of the original diffraction image that contains the target light source, this embodiment can also provide another method for restoring the original diffraction image, as shown in the following steps (1) to (3):
[0103] (1) The brightness value of each pixel in the original diffraction image is detected through the image processing model.
[0104] (2) Determine the light spot area containing the target light source in the original diffraction image based on the detected brightness value. In a specific implementation, the brightness area on the original diffraction image can be first determined based on the position of the pixel point whose detected brightness value is greater than the preset brightness threshold; then determine whether the radius of the circumscribed circle of the brightness area is greater than the preset radius. When the radius of the circumscribed circle of the brightness area is greater than the preset radius (such as r>2mm), it indicates that the brightness area has a high probability of being an area containing the target light source, and thus the brightness area is determined as the light spot area containing the target light source. If the radius of the circumscribed circle of the brightness area is not greater than the preset radius, it indicates that the brightness area may be caused by noise or other interfering light, and thus the brightness area is not determined as the light spot area.
[0105] (3) The original diffraction image is restored based on the spot area to obtain a target standard image corresponding to the original diffraction image.
[0106] In a possible implementation, restoration processing can be performed through the following specific process: First, the diffraction fringes are removed from the spot area to obtain the image to be restored corresponding to the original diffraction image. There may be at least one spot area in an original diffraction image, and the diffraction fringes in each spot area are removed to obtain the image to be restored corresponding to the original diffraction image. Since the light-transmitting area and the sub-pixels in the display screen are arranged non-repetitively, the diffraction fringes are stripes with uniform brightness distribution. In this case, the difficulty of removing the diffraction fringes can be effectively reduced.
[0107] Then, the image to be restored is processed for clarity to obtain the target standard image. In practical applications, the image to be restored can be processed for clarity based on a variety of methods such as Lucy-Richardson image restoration method, Wiener filtering or constrained least squares filtering to obtain a target standard image with good image quality and high clarity.
[0108] In summary of the above description, the present embodiment provides the above-mentioned image processing method, which can directly use the image processing model to restore the original diffraction image, effectively simplify the image restoration method and improve the clarity of the restored target standard image, thereby effectively improving the display effect of the display screen on the image. Further based on the applied electronic device, the display screen in the electronic device has a non-repetitive light-transmitting area, so that the original diffraction image that is easy to restore can be obtained, and then the original diffraction image is restored by the image processing model, which can not only improve the clarity of the restored image and improve the display effect of the display screen, but also can effectively improve the image restoration effect based on the original diffraction image that is easy to restore.
[0109] In order to enable the image processing model to be directly applied to the restoration of the original diffraction image and output a relatively clear target standard image, it is necessary to train the image processing model in advance to ultimately determine the parameters in the image processing model that can meet the requirements. Using the trained parameters, the restoration result of the original diffraction image by the image processing model can meet the expected image quality requirements. This embodiment provides a training method for an image processing model, referring to Figure 6 The training flow chart of the image processing model shown in FIG. 1 is a flowchart of the training flow chart of the image processing model shown in FIG. 1 . The method may specifically refer to the following steps S602 to S610:
[0110] Step S602, obtaining an image sample pair; wherein the image sample pair includes a sample standard image and a sample diffraction image corresponding to the sample standard image. In one embodiment, the image sample pair includes a sample standard image of a specified scene photographed by an on-screen camera and a sample diffraction image corresponding to the sample standard image, wherein the sample diffraction image is an image obtained by simulating the under-screen camera photographing the specified scene based on the sample standard image, or is an image obtained by photographing the specified scene by the under-screen camera; it can be understood that the specified scene corresponding to the sample standard image and the sample diffraction image is the same scene.
[0111] It should be noted that this step belongs to the preparation stage of the image processing model training, and the purpose of this step is to prepare the image sample pairs. If there are available image sample pairs, this step can be skipped and step S604 can be performed directly.
[0112] Step S604: input image sample pairs into the image processing model.
[0113] Referring to the above embodiment, the image sample pair includes a sample standard image and a sample diffraction image corresponding to the sample standard image. For example, the sample standard image and the sample diffraction image are an on-screen image and an off-screen image corresponding to the same scene.
[0114] Step S606, restoring the sample diffraction image through the image processing model to obtain a restored image of the sample diffraction image.
[0115] Step S608, determining the loss function value corresponding to the image processing model according to the restored image and the sample standard image.
[0116] Specifically, the similarity between the restored image and the sample standard image can be calculated, and the loss function value corresponding to the image processing model can be determined according to the similarity. In specific implementation, the similarity between the restored image and the sample standard image can be calculated using a variety of similarity algorithms such as a cosine similarity algorithm, a histogram algorithm, or a structural similarity measurement algorithm.
[0117] Step S610, iteratively updating the parameters of the image processing model according to the loss function value.
[0118] Because a single parameter update may not necessarily achieve the desired effect of the image processing model, it is necessary to perform iterative updates. Specifically, first determine whether the loss function value converges to a preset value, or whether the iterative update reaches a preset number of times. When the loss function value converges to a preset value, or the iterative update reaches a preset number of times, the training can be terminated to obtain the trained image processing model.
[0119] For example, first determine whether the loss function value converges to a preset value. If it has converged to the preset value, the training can be terminated to obtain the trained image processing model; if it has not converged to the preset value, the parameters of the image processing model are continuously iterated and updated. In addition, the number of iterations can also be set. When the preset number of iterations is reached and the loss function value is reduced to the preset value, the training is terminated.
[0120] In addition, the convergence of the loss function value and the number of iterations can be considered comprehensively. The training can only be ended when the loss function value has converged to the preset value and the iterative update has reached the preset number of times.
[0121] In the above-mentioned training method provided in the present embodiment, using the corresponding sample standard image and sample diffraction image as training data can reduce the difference between the two images in the image sample pair, that is, improve the quality of the image sample pair. The higher quality image sample pair is helpful to improve the training effect of the image processing model; at the same time, using the similarity as the loss function value reduces the calculation difficulty of the loss function, which can improve the training efficiency of the image processing model.
[0122] In the training process of the above-mentioned image processing model, it is necessary to rely on a large number of high-quality and diverse image sample pairs as training data. Based on this, this embodiment provides two methods for obtaining image sample pairs.
[0123] Acquisition method 1: Use the on-screen camera to shoot the specified scene at a preset shooting angle to obtain a sample standard image; and use the under-screen camera to shoot the specified scene at a shooting angle to obtain a sample diffraction image.
[0124] In the acquisition method of the image sample pair, the shooting angles and the designated scenes of the on-screen camera and the off-screen camera are the same, so the obtained sample standard image and sample diffraction image are basically the same, and can be used as training data for the image processing model. This acquisition method is simple and easy to operate, and has low requirements on the user's work ability.
[0125] Acquisition method 2: Considering that the sample standard image and the sample diffraction image may have deviations due to image content, shooting angle, etc., which may have an adverse effect on the training effect of the image processing model and lead to poor quality of the restored image in practical applications. In order to avoid the above problems, this embodiment can refer to the following method to obtain a sample standard image and a sample diffraction image with a good matching degree, including:
[0126] First, the specified scene is photographed through the on-screen camera to obtain a sample standard image. Then, the target light source in the dark background is photographed through the display screen by the on-screen camera to obtain a target light source image; in order to improve the simulation fidelity between the sample diffraction image and the real under-screen image photographed by the under-screen camera, the display screen is the same as the display screen of the electronic device. Finally, the target light source image is convolved with the sample standard image to obtain a sample diffraction image.
[0127] In the above method, a sample diffraction image that simulates the under-screen image is generated based on the sample standard image, which can avoid the deviation between the sample standard image and the sample diffraction image, so that the image processing model trained based on the image sample can have a better restoration effect and improve the clarity and image quality of the restored image.
[0128] In order to better understand the candidate target light source image, a method for obtaining the candidate target light source image is provided here. Figure 7 The target light source image shooting scene schematic diagram shown in the figure shows the on-screen camera, display screen and target light source arranged in sequence; the on-screen camera and display screen are equivalent to simulating the shooting method of the under-screen camera through the display screen. Figure 7 In the scene shown, the method of obtaining the candidate target light source image includes: photographing the target light source in the preset scheme through the display screen by the on-screen camera to obtain the candidate target light source image; wherein the preset scheme is a scheme for spatially arranging at least one target light source in a dark background, and in different preset schemes, the number of target light sources and / or the spatial arrangement of the target light sources are different. For example, preset scheme 1 is a target light source in a dark background, and the spatial arrangement of the target light source is a specified distance from the display screen; preset scheme 2 is three target light sources in a dark background, and the spatial arrangement of the three target light sources is arranged in a column, a row or a triangle at a certain distance; preset scheme 3 is n (n is any value greater than 1) target light sources in a dark background, and the spatial arrangement of the n target light sources may be multiple, such as arranged in multiple columns, randomly distributed, etc. A variety of preset schemes can be provided according to actual life scenes (such as office work scenes, family life scenes, outdoor scenes, etc.), and the candidate target light source images corresponding to each preset scheme are obtained, thereby improving the diversity of the candidate target light source images.
[0129] At least one of the candidate target light source images is determined as the target light source image. Based on the diversity of the candidate target light source images, there are multiple combinations of different candidate target light source images and sample standard images in different specified scenes, so that a large number of image sample pairs can be obtained quickly and easily, thereby increasing the number and diversity of image sample pairs. Based on the rich image sample pairs, the image restoration effect of the image processing model can be improved. In other embodiments, multiple candidate target light source images can also be determined as target light source images.
[0130] In practical applications, the target light source image may also be subjected to denoising, and the denoised target light source image may be convolved with the sample standard image to obtain a sample diffraction image with better quality.
[0131] According to the above description, the image sample pairs obtained in this embodiment are of high quality and diversity, which helps to better train the image processing model, thereby improving the image restoration effect of the image processing model in practical applications, and effectively improving the clarity and picture quality of the restored image.
[0132] Embodiment three:
[0133] Based on the image processing method provided in the above embodiment, this embodiment provides an image processing device. Figure 8 The structure block diagram of an image processing device shown in FIG. 1 is applied to an electronic device with an under-screen camera, and the device includes:
[0134] The image acquisition module 802 is used to acquire the original diffraction image.
[0135] The image input module 804 is used to input the original diffraction image into the image processing model.
[0136] The image restoration module 806 is used to restore the original diffraction image through the image processing model to obtain a target standard image corresponding to the original diffraction image.
[0137] The above-mentioned image processing device provided by the embodiment of the present invention can directly use the image processing model to restore the original diffraction image, effectively simplifying the image restoration method and improving the clarity of the restored target standard image, thereby effectively improving the display effect of the display screen on the image.
[0138] In some embodiments, the image restoration module 806 is further used to: detect the brightness value of each pixel in the original diffraction image through an image processing model; determine the light spot area of the original diffraction image containing the target light source based on the detected brightness value; and restore the original diffraction image based on the light spot area to obtain a target standard image corresponding to the original diffraction image.
[0139] In some embodiments, the image restoration module 806 is further used to: remove diffraction fringes from the light spot area to obtain an image to be restored corresponding to the original diffraction image; and perform clarity processing on the image to be restored to obtain a target standard image.
[0140] In some embodiments, the image restoration module 806 is further used to: determine the brightness area on the original diffraction image based on the position of the pixel point whose detected brightness value is greater than the preset brightness threshold; determine whether the radius of the circumscribed circle of the brightness area is greater than the preset radius; if so, determine the brightness area as the light spot area containing the target light source.
[0141] In some embodiments, the image acquisition module 802 is further used to: acquire the original diffraction image through an under-screen camera of the electronic device.
[0142] In some embodiments, the above-mentioned image processing model is trained based on image sample pairs, and the image sample pairs include a sample standard image of a specified scene photographed by an on-screen camera and a sample diffraction image corresponding to the sample standard image; wherein the sample diffraction image is an image obtained by simulating the under-screen camera photographing the specified scene based on the sample standard image, or is an image obtained by photographing the specified scene by the under-screen camera.
[0143] In some embodiments, the electronic device includes a display screen, which includes multiple light-emitting units and multiple light-transmitting areas; wherein each light-emitting unit includes a preset number of sub-pixels; the multiple light-transmitting areas are non-repetitively arranged between the sub-pixels of the multiple light-emitting units, so that the diffraction stripe image generated by the target light source through the display screen is a uniformly distributed stripe image.
[0144] In some embodiments, any sub-pixel in the plurality of light-emitting units is separated from any light-transmitting region in the plurality of light-transmitting regions.
[0145] In some embodiments, the electronic device is an electronic device with an under-screen camera.
[0146] The device provided in this embodiment has the same implementation principle and technical effects as the image processing method in the aforementioned embodiment 2. For the sake of brief description, for parts not mentioned in this embodiment, reference may be made to the corresponding contents in the aforementioned embodiment 2.
[0147] Embodiment 4:
[0148] Based on the training method of the image processing model provided in the above embodiment, this embodiment provides a training device for an image processing model, see Fig. 9 A structural block diagram of a training device for an image processing model shown in FIG. 1 , the device comprises:
[0149] An input module 904, configured to input an image sample pair into the image processing model, wherein the image sample pair includes a sample standard image and a sample diffraction image corresponding to the sample standard image;
[0150] A restoration module 906 is used to restore the sample diffraction image through an image processing model to obtain a restored image of the sample diffraction image;
[0151] A calculation module 908, configured to determine a loss function value corresponding to the image processing model according to the restored image and the sample standard image;
[0152] The updating module 910 is used to iteratively update the parameters of the image processing model according to the loss function value.
[0153] The training device for the above-mentioned image processing model provided in this embodiment uses the sample standard image and the sample diffraction image with a corresponding relationship as training data, which can reduce the difference between the two images in the image sample pair, that is, improve the quality of the image sample pair. The higher quality image sample pairs are helpful to improve the training effect of the image processing model; at the same time, using the similarity as the loss function value reduces the calculation difficulty of the loss function, which can improve the training efficiency of the image processing model.
[0154] In some embodiments, the training device may further include an acquisition module 902, which is used to: photograph a designated scene through an on-screen camera to obtain a sample standard image; determine at least one of the candidate target light source images as a target light source image; wherein the candidate target light source image is an image obtained by photographing the target light source in a dark background through a display screen by the on-screen camera; and perform a convolution operation on the target light source image and the sample standard image to obtain a sample diffraction image.
[0155] In some embodiments, the training data acquisition module 902 is further used to: photograph the target light source in a preset scheme through the display screen using an on-screen camera to obtain multiple candidate target light source images; wherein the preset scheme is a scheme for spatially arranging at least one target light source in a dark background, and in different preset schemes, the number of target light sources and / or the spatial arrangement of the target light sources are different, and different preset schemes correspond to different candidate target light source images.
[0156] In some implementations, the training data acquisition module 902 is further used to: perform noise reduction processing on the target light source image or the candidate target light source image.
[0157] In some implementations, the display screen is the same display screen as the display screen of the electronic device in the image processing method of the second embodiment.
[0158] In some embodiments, the training data acquisition module 902 is further used to: photograph a designated scene at a preset shooting angle using an on-screen camera to obtain a sample standard image; photograph a designated scene at a shooting angle using an under-screen camera to obtain a sample diffraction image.
[0159] The device provided in this embodiment has the same implementation principle and technical effects as the training method of the image processing model in the aforementioned embodiment 2. For the sake of brief description, for parts not mentioned in this embodiment, reference may be made to the corresponding contents in the aforementioned embodiment 2.
[0160] Embodiment five:
[0161] Based on the foregoing embodiments, this embodiment provides an image processing system, which includes: a processor and a storage device; wherein a computer program is stored on the storage device, and when the computer program is run by the processor, it executes any image processing method provided in Embodiment 2, or executes any image processing model training method provided in Embodiment 2.
[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0163] Embodiment six:
[0164] Based on the above embodiments, this embodiment provides an electronic device, which includes a display screen and an under-screen camera, and also includes an image processing system provided in the above embodiments. The display screen includes a plurality of light-emitting units and a plurality of light-transmitting areas, wherein each light-emitting unit includes a plurality of sub-pixels.
[0165] Furthermore, gaps exist between the multiple sub-pixels in the multiple light-emitting units to form multiple light-transmitting regions in the gaps, and the multiple light-transmitting regions include at least two non-repetitive first light-transmitting regions.
[0166] Furthermore, the multiple sub-pixels of at least two light-emitting units are distributed in a non-repetitive manner.
[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0168] Furthermore, this embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processing device, the computer program executes the steps of any one of the image processing methods provided in the above-mentioned embodiment 2, or executes the steps of any one of the image processing model training methods provided in embodiment 2.
[0169] An image processing method and device, an image processing model training method and device computer program product provided by the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.
[0170] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0171] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. An image processing method, characterized in that: The method is applied to an electronic device, wherein the electronic device is an electronic device with an under-screen camera, the electronic device includes a display screen, the display screen includes a plurality of light-emitting units and a plurality of light-transmitting areas, each of the light-emitting units includes a preset number of sub-pixels; the method includes: Acquire an original diffraction image; wherein the original diffraction image is a uniformly distributed fringe image acquired through the display screen by the under-screen camera of the electronic device; Inputting the original diffraction image into an image processing model; Restoring the original diffraction image by using the image processing model to obtain a target standard image corresponding to the original diffraction image; The step of restoring the original diffraction image by using the image processing model to obtain a target standard image corresponding to the original diffraction image comprises: Detecting the brightness value of each pixel in the original diffraction image through the image processing model; Determine the spot area containing the target light source in the original diffraction image based on the detected brightness value; The original diffraction image is restored based on the light spot area to obtain a target standard image corresponding to the original diffraction image.
2. The method according to claim 1, characterized in that The step of restoring the original diffraction image based on the light spot area comprises: Removing diffraction fringes from the light spot area to obtain an image to be restored corresponding to the original diffraction image; The image to be restored is processed for clarity to obtain a target standard image.
3. The method according to claim 1, characterized in that The step of determining the light spot area containing the target light source in the original diffraction image based on the detected brightness value comprises: Determining a brightness area on the original diffraction image according to the positions of pixels whose detected brightness values are greater than a preset brightness threshold; Determining whether the radius of the circumscribed circle of the brightness area is greater than a preset radius; If yes, the brightness area is determined as a light spot area containing the target light source.
4. The method according to claim 1, characterized in that: The image processing model is trained based on image sample pairs, and the image sample pairs include a sample standard image of a specified scene photographed by an on-screen camera and a sample diffraction image corresponding to the sample standard image; wherein the sample diffraction image is an image obtained by simulating the under-screen camera photographing the specified scene based on the sample standard image, and / or is an image obtained by photographing the specified scene by the under-screen camera.
5. The method according to claim 1, characterized in that There are gaps between the sub-pixels in the light emitting units, so that the plurality of light-transmitting regions are formed in the gaps, and the plurality of light-transmitting regions include at least two non-repetitive first light-transmitting regions.
6. The method according to claim 5, characterized in that Any sub-pixel in the plurality of light-emitting units is separated from any light-transmitting area in the plurality of light-transmitting areas.
7. The method according to claim 5, characterized in that The at least two non-repetitive first light-transmitting regions are one or more of the following first light-transmitting regions: At least two of the first light-transmitting areas have different size parameters, shape parameters, posture parameters, and position distribution parameters; Each of the first light-transmitting areas has different size parameters, shape parameters, posture parameters, and position distribution parameters from other light-transmitting areas; and All the light-transmitting areas have different size parameters, shape parameters, posture parameters, and position distribution parameters.
8. The method according to claim 1, characterized in that The sub-pixels of at least two of the light-emitting units are distributed in a non-repetitive manner.
9. The method according to claim 8, characterized in that At least two non-repeatingly distributed light-transmitting areas are formed in the gaps between the non-repeatingly distributed sub-pixels.
10. The method according to claim 8, characterized in that The light-emitting unit is one or more of the following light-emitting units: The multiple sub-pixels of at least two light-emitting units have different size parameters, shape parameters, posture parameters, and position distribution parameters; The multiple sub-pixels of at least two light-emitting units have different size parameters, shape parameters, posture parameters, and position distribution parameters from the multiple sub-pixels of other light-emitting units; as well as The multiple sub-pixels of all light-emitting units have different size parameters, shape parameters, posture parameters, and position distribution parameters.
11. A training method for an image processing model, characterized in that: The method comprises: Input an image sample pair to the image processing model, wherein the image sample pair includes a sample standard image and a sample diffraction image corresponding to the sample standard image; the sample standard image is an image obtained by photographing a specified scene through an on-screen camera; the sample diffraction image is an image obtained by simulating an under-screen camera photographing the specified scene through a display screen based on the sample standard image, or is an image obtained by photographing the specified scene through a display screen through a under-screen camera; the display screen is the same display screen as the display screen of the electronic device in the method according to any one of claims 1 to 10; Performing restoration processing on the sample diffraction image by using the image processing model to obtain a restored image of the sample diffraction image; Determining a loss function value corresponding to the image processing model according to the restored image and the sample standard image; The parameters of the image processing model are iteratively updated according to the loss function value.
12. The method according to claim 11, characterized in that The iterative updating of the parameters of the image processing model according to the loss function value comprises: Determine whether the loss function value converges to a preset value, and / or whether the iterative update reaches a preset number of times; When the loss function value converges to a preset value, and / or the iterative update reaches a preset number of times, a trained image processing model is obtained.
13. The method according to claim 11, characterized in that The step of determining the loss function value corresponding to the image processing model according to the restored image and the sample standard image includes: The similarity between the restored image and the sample standard image is calculated, and the loss function value corresponding to the image processing model is determined according to the similarity.
14. The method according to claim 11, characterized in that The method for acquiring the image sample pair comprises: The designated scene is photographed by the on-screen camera to obtain the sample standard image; The target light source in the dark background is photographed through the display screen by the on-screen camera to obtain an image of the target light source; The target light source image is convolved with the sample standard image to obtain the sample diffraction image.
15. The method according to claim 14, characterized in that The method of photographing the target light source in the dark background through the display screen by the on-screen camera to obtain the target light source image includes: The target light source in the preset scheme is photographed through the display screen by the on-screen camera to obtain a candidate target light source image; wherein the preset scheme is a scheme for spatially arranging at least one target light source in a dark background, and in different preset schemes, the number of target light sources and / or the spatial arrangement of the target light sources are different, and the candidate target light source images corresponding to different preset schemes are different; At least one candidate target light source image among the candidate target light source images is determined as the target light source image.
16. The method according to claim 14 or 15, characterized in that Before the step of performing a convolution operation on the target light source image and the sample standard image, the method further includes: Perform noise reduction processing on the target light source image.
17. The method according to claim 11, characterized in that The method for acquiring the image sample pair comprises: The designated scene is photographed by the on-screen camera at a preset shooting angle to obtain the sample standard image; The designated scene is photographed at the shooting angle by the under-screen camera to obtain the sample diffraction image.
18. An image processing device, characterized in that: The device is applied to an electronic device, wherein the electronic device is an electronic device with an under-screen camera, wherein the electronic device comprises a display screen, wherein the display screen comprises a plurality of light-emitting units and a plurality of light-transmitting areas, and each of the light-emitting units comprises a preset number of sub-pixels; the device comprises: An image acquisition module, used to acquire an original diffraction image; wherein the original diffraction image is a uniformly distributed fringe image acquired through the display screen by the under-screen camera of the electronic device; An image input module, used for inputting the original diffraction image into an image processing model; An image restoration module, used for restoring the original diffraction image through the image processing model to obtain a target standard image corresponding to the original diffraction image; Among them, the image restoration module is used to detect the brightness value of each pixel in the original diffraction image through the image processing model; determine the spot area of the target light source in the original diffraction image based on the detected brightness value; and restore the original diffraction image based on the spot area to obtain the target standard image corresponding to the original diffraction image.
19. A training device for an image processing model, characterized in that: The device comprises: An input module, used for inputting an image sample pair into the image processing model, wherein the image sample pair includes a sample standard image and a sample diffraction image corresponding to the sample standard image; the sample standard image is an image obtained by photographing a specified scene through an on-screen camera; the sample diffraction image is an image obtained by simulating an under-screen camera photographing the specified scene through a display screen based on the sample standard image, or is an image obtained by photographing the specified scene through a display screen through a under-screen camera; the display screen is the same display screen as the display screen of the electronic device in the device according to claim 18; A restoration module, used for performing restoration processing on the sample diffraction image through the image processing model to obtain a restored image of the sample diffraction image; A calculation module, used for determining a loss function value corresponding to the image processing model according to the restored image and the sample standard image; An updating module is used to iteratively update the parameters of the image processing model according to the loss function value.
20. An image processing system, characterized in that: The system includes a processor and a storage device; The storage device stores a computer program, and when the computer program is executed by the processor, the computer program executes the method according to any one of claims 1 to 10, or executes the method according to any one of claims 11 to 17.
21. An electronic device, characterized in that: The electronic device comprises a display screen and an under-screen camera, and also comprises the image processing system according to claim 20; The display screen includes a plurality of light-emitting units and a plurality of light-transmitting areas; wherein each of the light-emitting units includes a plurality of sub-pixels.
22. The electronic device according to claim 21, characterized in that: There are gaps between the sub-pixels in the light emitting units, so that the plurality of light-transmitting regions are formed in the gaps, and the plurality of light-transmitting regions include at least two non-repetitive first light-transmitting regions.
23. The electronic device according to claim 21 or 22, characterized in that: The sub-pixels of at least two of the light-emitting units are distributed in a non-repetitive manner.
24. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program executes the steps of the method according to any one of claims 1 to 10, or executes the steps of the method according to any one of claims 11 to 17.
