Image processing method, device, equipment and computer readable storage medium

Abstract

The embodiment of the present application discloses an image processing method, which is applied to a terminal, and the terminal includes a color image acquisition module and a base map image acquisition module. The method includes: using the color image acquisition module to acquire a color image of the target object; using the base map image acquisition module to acquire a base map image of the target object; extracting a first feature of the color image and a second feature of the base map image respectively; based on the first feature and the second feature, determining the noise area in the color image; restoring the contour information, grayscale information and color information of the noise area to obtain a composite image. The above method can eliminate the noise interference information in the captured image, and obtain a color image that can better reflect the real characteristics of the target object, thereby improving the image quality.

Description

Technical Field

[0001] The present application relates to the technical field of digital image processing, and in particular to an image processing method, apparatus, device and computer-readable storage medium. Background Art

[0002] Full-screen display technology is a relatively new technology. At present, it is mainly achieved by making the display screen low-pixel density (Pixels Per Inch, PPI) and transparent in a small area of ​​the screen, generally at the position of the front camera, so that the area is a transparent display area with light transmittance, and setting a camera under the area, thereby realizing the full-screen design.

[0003] In the related art, the gap is generally increased by reducing pixels or changing the sub-pixel arrangement, thereby increasing the light transmittance of the transparent display area corresponding to the camera position. However, these solutions all have the problem that the sub-pixel area is not light-transmissive, and a partial opaque area is formed in the transparent display area. Specifically, due to the existence of periodic sub-pixels in the transparent display area, the opaque area corresponding to the sub-pixels will present a periodic arrangement, similar to the structure of a transmission grating, which will cause a diffraction effect on the incident light entering the transparent display area, resulting in a decrease in the resolution of the camera head, affecting the imaging effect of the camera, and causing problems such as diffraction, interference and glare in the captured image. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device and computer-readable storage medium, which can eliminate noise interference information in a captured image and obtain a color image that can better reflect the true characteristics of the target object, thereby improving the image quality.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides an image processing method, which is applied to a terminal, wherein the terminal includes a color image acquisition module and a base map image acquisition module, and the method includes: using the color image acquisition module to acquire a color image of a target object; using the base map image acquisition module to acquire a base map image of the target object; respectively extracting a first feature of the color image and a second feature of the base map image; based on the first feature and the second feature, determining a noise area in the color image; and restoring contour information, grayscale information, and color information of the noise area to obtain a composite image.

[0007] An embodiment of the present application provides an image processing device, including: an extraction unit, used to extract a first feature of a color image and a second feature of a base image respectively; a determination unit, used to determine a noise area in the color image based on the first feature and the second feature; and a processing unit, used to restore contour information, grayscale information and color information of the noise area to obtain a composite image.

[0008] An embodiment of the present application provides an image acquisition device, comprising: a display screen and an image acquisition device arranged in sequence along an optical axis direction; the display screen comprises a transparent display area; the image acquisition device comprises a color image acquisition module and a base map image acquisition module; the color image acquisition module and the base map image acquisition module are both arranged at positions opposite to the transparent display area; the color image acquisition module is used to obtain a color image of the target object based on imaging of incident light that passes through the transparent display area and is incident on the color image acquisition module along the optical axis direction; the base map image acquisition module is used to obtain a color image of the target object through the transparent display area.

[0009] An embodiment of the present application provides a terminal, comprising: a display screen, a memory, a processor, and the above-mentioned color image acquisition module and base map image acquisition module; the display screen comprises a transparent display area; the color image acquisition module and the base map image acquisition module are both arranged at a position opposite to the transparent display area; the color image acquisition module is used to acquire a color image of the target object through the transparent display area; the base map image acquisition module is used to acquire a base map image of the target object through the transparent display area; the memory is used to store an executable computer program; the processor is used to implement the above-mentioned image processing method when executing the executable computer program stored in the memory.

[0010] An embodiment of the present application provides a computer-readable storage medium storing a computer program for causing a processor to execute the above-mentioned image processing method.

[0011] The image processing method provided in the embodiment of the present application, since the base image can reflect the real features of the target object, by comparing the color image with the base image, the area where shooting noise such as interference fringes, diffraction fringes and glare are located in the color image can be known, and by restoring the contour information, grayscale information and color information of the area where the shooting noise is located, a color image that can better reflect the real features of the target object can be obtained, thereby improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1Ais a schematic diagram of an arrangement of opaque sub-pixels on a transparent display area in an exemplary display screen provided in an embodiment of the present application;

[0013] Figure 1B is a schematic diagram of the light transmission effect of a transparent display area without wiring in an exemplary display screen provided in an embodiment of the present application under light;

[0014] Figure 2A is a schematic diagram of the effect of diffraction fringes formed by the diffraction of an exemplary light provided in an embodiment of the present application;

[0015] Figure 2B is a schematic diagram of the effect of interference fringes formed by the interference of exemplary light provided in an embodiment of the present application;

[0016] Figure 3A is a schematic diagram of a simulation effect of a point spread function of an exemplary color image collected through a transparent display area of ​​a display screen provided in an embodiment of the present application;

[0017] Figure 3B is a schematic diagram of a simulation effect of a point spread function of an exemplary image collected without passing through a transparent display area of ​​a display screen provided in an embodiment of the present application;

[0018] Figure 3C is a schematic diagram of an exemplary spectrum distribution effect of incident light passing through a transparent display area of ​​a display screen in a one-dimensional space provided by an embodiment of the present application;

[0019] Figure 4 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0020] Figure 5 is a schematic diagram of the principle of an exemplary structured light camera provided in an embodiment of the present application;

[0021] Figure 6 is a schematic diagram of the principle of an exemplary TOF camera provided in an embodiment of the present application;

[0022] Fig. 7A is an exemplary depth image provided in an embodiment of the present application;

[0023] Figure 7B is an exemplary depth image provided in an embodiment of the present application;

[0024] Figure 8 is a schematic diagram of an exemplary grayscale image provided in an embodiment of the present application;

[0025] Fig. 9 is a schematic diagram of the structure of an exemplary display screen provided in an embodiment of the present application;

[0026] Fig.10 is a partial cross-sectional schematic diagram of an exemplary terminal along the AB direction provided in an embodiment of the present application;

[0027] Fig.11 is another optional flowchart of the image processing method provided in the embodiment of the present application;

[0028] Fig.12 It is another optional flowchart of the image processing method provided in the embodiment of the present application;

[0029] Fig.13 This is another optional flowchart of the image processing method provided in the embodiment of the present application;

[0030] Fig.14 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0031] Fig.15 It is a schematic diagram of the structure of the terminal provided in the embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0033] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0035] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0037] 1) Color image: refers to an image in which each pixel is composed of R, G, and B components.

[0038] 2) Base image: in this application, it refers to images without color information, such as depth images and grayscale images.

[0039] 3) Depth image: refers to an image that uses the distance (depth) from the image collector to each point in the scene as the pixel value. It directly reflects the geometric shape of the visible surface of the scene. The depth image can be calculated into point cloud data after coordinate transformation.

[0040] 4) Grayscale image: An image with only one sampled color per pixel. This type of image is usually displayed as a grayscale from the darkest black to the brightest white, with many levels of color depth between black and white.

[0041] 5) Flare: It often appears in photos, especially when shooting against the light. Usually, the phenomenon of photos turning white and forming halos caused by strong light is called "flare".

[0042] 6) Interference of light: It is a phenomenon in which two or more light waves overlap in space and superimpose to form a new waveform; when interference occurs, some areas become brighter and some areas become darker, that is, interference fringes appear.

[0043] 7) Diffraction of light: When light encounters an obstacle or a small hole during its propagation, it will deviate from the straight-line propagation path and travel around behind the obstacle. The light and dark stripes or halos produced by diffraction are called diffraction patterns. Diffraction stripes refer to the stripes projected on the screen by the diffraction of light.

[0044] 8) Point spread function (PSF): For an optical system, the light field distribution of the output image when the input object is a point light source.

[0045] Currently, the technology of full-screen design mainly realizes the full-screen design by making the display screen low pixel density (Pixels Per Inch, PPI) and transparent in a small area of ​​the screen, generally at the position of the front camera, so that the area is a transparent display area with light transmittance, and setting a camera under the area (hereinafter referred to as "under-screen camera") . In the related art, the gap is generally increased by reducing pixels or changing the pixel arrangement, thereby increasing the light transmittance of the transparent display area of ​​the display screen corresponding to the position of the camera. However, in order to take into account the display function of the display screen and enable the light emitted by the sub-pixel to propagate outside the display screen, the metal anode located on the bottom surface of the sub-pixel in the transparent display area is usually designed to be opaque, thereby forming a metal anode reflective layer with a reflectivity close to 100%, that is, the sub-pixel area in the transparent display area is designed to be an opaque area. Exemplarily, Figure 1A is a schematic diagram of an arrangement of opaque sub-pixels on a transparent display area in an exemplary display screen provided in an embodiment of the present application; Figure 1B FIG. 1 is a schematic diagram of the light transmission effect of the unwired transparent display area in the exemplary display screen provided by the embodiment of the present application under light. Figure 1A and 1B As shown, the sub-pixels are evenly spaced, and the areas where the sub-pixels are located are all opaque areas, and the intervals between the sub-pixels (i.e., the areas without wiring) are transparent, allowing light to pass through. It is precisely because of this design that these solutions all have the periodic presence of sub-pixels on the transparent display area, so the opaque areas corresponding to the sub-pixels will be periodically arranged, forming a structure similar to a transmission grating, thereby causing a diffraction effect on the incident light entering the transparent display area, resulting in a decrease in the resolution of the camera head, affecting the imaging effect of the camera, and causing problems such as diffraction, interference, and glare in the captured image.

[0046] For example, Figure 2A is a schematic diagram of the effect of diffraction fringes formed by the diffraction of an exemplary light provided in an embodiment of the present application; Figure 2B Schematic diagram of the effect of interference fringes formed by interference of exemplary light provided in the embodiment of the present application. Figure 2A and 2B As shown, both diffraction and interference of light will produce fringe interference information.

[0047] For example, Figure 3A is a schematic diagram of a simulation effect of a point spread function of an exemplary color image collected through a transparent display area of ​​a display screen provided in an embodiment of the present application; Figure 3B It is a schematic diagram of the simulation effect of the point spread function of an exemplary image collected without passing through the transparent display area of ​​the display screen provided in an embodiment of the present application. Figure 3CIt is a schematic diagram of the spectral distribution effect of incident light passing through the transparent display area of ​​the display screen in one-dimensional space, provided in an exemplary embodiment of the present application. Figure 3A , 3B In 3C, the X-axis represents the spatial position and the Y-axis represents the logarithm of the light intensity (brightness). Figure 3C λ in R , G and λ B Represents incident light of different wavelengths; Figure 3A and 3B As shown, the light field distribution of the image captured through the transparent display area is different from the light field distribution of the image captured not through the transparent display area. Obviously, the transparent display area of ​​the display screen will affect the image acquisition effect of the under-screen camera.

[0048] Based on this, the embodiments of the present application provide an image processing method, apparatus, device and computer-readable storage medium, which can eliminate noise interference information in the captured image, thereby improving the quality of the image.

[0049] The following describes an exemplary application of the terminal provided in the embodiment of the present application. The terminal provided in the embodiment of the present application can be implemented as various types of terminals such as smart phones, tablet computers, and laptop computers with color image acquisition modules and base map image acquisition modules.

[0050] The image processing method provided in the embodiment of the present application will be described below in combination with the exemplary application and embodiments of the terminal provided in the embodiment of the present application.

[0051] Figure 4 is an optional flow chart of the image processing method provided in the embodiment of the present application, which will be combined with Figure 4 The steps shown are explained.

[0052] S101. Use a color image acquisition module to acquire a color image of a target object.

[0053] In an embodiment of the present application, the color image acquisition module may be an RGB camera; the terminal may acquire color images of target objects such as people and landscapes by controlling the three primary colors (Red Green Blue, RGB) camera.

[0054] S102: Using a base map image acquisition module, acquire a base map image of a target object.

[0055] In an embodiment of the present application, the base map image acquisition module may be a camera that acquires images that reflect the true features of the target object and do not contain color information, for example, it may be a black and white charge coupled device (CCD) camera. By controlling the base map image acquisition module, the terminal may acquire base map images that reflect the true features of the target objects such as people and scenery and do not contain color information.

[0056] In an embodiment of the present application, the terminal can control the color image acquisition module and the base map image acquisition module to simultaneously acquire images of the target object.

[0057] In an embodiment of the present application, the terminal can control the color image acquisition module and the base map image acquisition module to respectively acquire a color image and a base map image of the target object simultaneously.

[0058] S103, extracting first features of the color image and second features of the base image respectively.

[0059] In an embodiment of the present application, after acquiring the color image and the base map image, the terminal may extract feature information from the color image and extract feature information from the base map image.

[0060] In an embodiment of the present application, the terminal can extract one or more of the texture features, shape features, color features and spatial relationship features in the color image as the first feature; and extract one or more of the texture features, shape features, grayscale features and spatial relationship features of the base map image as the second feature.

[0061] In the embodiments of the present application, for texture features, the terminal can use geometric methods, statistical gray-level co-occurrence matrix and energy spectrum function methods, model methods and signal processing methods to extract texture features; for shape features, the terminal can use Fourier transform, geometric matrix and finite element method and other methods to extract shape features; for color features, the terminal can use color histogram, color aggregation vector and color correlation graph and other methods to extract color features; and, for spatial relationship features, the terminal can first automatically segment the image to divide the objects or color areas contained in the image, and then extract image features based on these areas and establish an index, or the terminal can evenly divide the image into several regular sub-blocks, and then extract features for each image sub-block, and establish an index, etc., so as to extract spatial relationship features. The corresponding feature extraction method can be selected according to actual needs, and the embodiments of the present application do not limit the feature extraction method here.

[0062] S104: Determine a noise area in the color image based on the first feature and the second feature.

[0063] In an embodiment of the present application, after extracting the first feature in the color image and the second feature in the base image, the terminal can compare the first feature with the second feature, and determine the noise area in the color image according to the comparison result. In an embodiment of the present application, the terminal can compare the first feature with the second feature by using a correlation comparison method, a histogram comparison method, or a time domain to frequency domain comparison method.

[0064] In an embodiment of the present application, the terminal may compare texture features extracted from a color image with texture features extracted from a base map image, compare shape features extracted from a color image with shape features extracted from a base map image, and compare spatial relationship features extracted from a color image with spatial relationship features extracted from a base map image. The embodiment of the present application does not limit the method of feature comparison.

[0065] In an embodiment of the present application, since the base image can reflect the real features of the target object, when certain features in the first feature are different from certain corresponding features in the second feature and the correlation is lower than a preset threshold, it indicates that these features are noise in the color image, and the area where these features are located is the noise area in the color image.

[0066] In an embodiment of the present application, the noise may include interference information caused by ambient light. Exemplarily, the interference information caused by ambient light includes at least one of diffraction fringes, interference fringes and glare.

[0067] S105 , restore the contour information, grayscale information and color information of the noise area to obtain a composite image.

[0068] In an embodiment of the present application, after the terminal determines the noise area in the color image, it can restore the contour information, grayscale information and color information of the noise area according to the features in the extracted base image to obtain a synthetic image that can better reflect the real characteristics of the target object.

[0069] In the embodiment of the present application, since the background image can reflect the true characteristics of the target object, by comparing the color image with the background image, the area where the shooting noise such as interference fringes, diffraction fringes and glare is located in the color image can be known, and by restoring the contour information, grayscale information and color information of the area where the shooting noise is located, a color image that can better reflect the true characteristics of the target object can be obtained, thereby improving the image quality.

[0070] In an embodiment of the present application, the base image acquisition module includes at least one of a depth image acquisition module and a grayscale image acquisition module; the above S102 can be implemented by S1021, S1022 or S1023, as follows:

[0071] S1021. Use a depth image acquisition module to acquire a depth image of the target object.

[0072] In an embodiment of the present application, the depth image acquisition module may be a structured light camera, an ultrasonic sensor or a time of flight (TOF) camera, and the terminal may acquire a depth image of the target object by controlling the structured light camera, the ultrasonic sensor or the TOF camera.

[0073] For example, Figure 5 is a schematic diagram of the principle of an exemplary structured light camera provided in an embodiment of the present application, such as Figure 5 As shown, the LED array continuously emits modulated light to the target object, and then the sensor module receives the reflected light reflected from the target object. By detecting the flight (round-trip) time of the light, the distance of the target object from the camera can be calculated to generate depth information, thereby obtaining a depth image containing the depth and contour of the target object.

[0074] The TOF camera can obtain the point cloud information of the image by emitting and receiving laser pulses. For example, Figure 6 is a schematic diagram of the principle of an exemplary TOF camera provided in an embodiment of the present application, such as Figure 6 As shown, a light transmitter continuously emits laser pulses to the object being photographed, and then a light receiver receives the laser pulses reflected from the object. The distance of the object being photographed from the camera is calculated by detecting the flight (round-trip) time of the light pulses to generate depth information, thereby obtaining a depth image containing the depth and outline of the object being photographed.

[0075] The ultrasonic sensor can generate depth information by sending ultrasonic signals to an object and receiving the sound wave signals reflected by the object, and calculating the distance of the object from the ultrasonic sensor by calculating the round-trip time, thereby obtaining a depth image including the depth and outline of the object.

[0076] For example, Fig. 7A and Figure 7B These are two exemplary depth images provided in the embodiments of the present application.

[0077] S1022: Use a grayscale image acquisition module to acquire a grayscale image of the target object.

[0078] In an embodiment of the present application, the grayscale image acquisition module may be an infrared camera, and the terminal may acquire a grayscale image of the target object by controlling the infrared camera.

[0079] For example, Figure 8 is a schematic diagram of an exemplary grayscale image provided in an embodiment of the present application; Figure 8 As shown, the grayscale image includes the contour information and grayscale information of the object.

[0080] S1023. Use a depth image acquisition module to acquire a depth image of the target object, and use a grayscale image acquisition module to acquire a grayscale image of the target object.

[0081] In the embodiment of the present application, since the infrared camera has no color information, it is less affected by diffraction. In addition, since the TOF camera and the structured light camera adopt a phased array transmitting module design in principle design, and the ultrasonic sensor adopts a method of transmitting and receiving ultrasonic waves, the influence of the opacity of sub-pixels in the transparent display area can be eliminated in advance; therefore, by using any one or more cameras or sensors among the infrared camera, TOF camera, ultrasonic sensor and structured light camera, an image that can reflect the true characteristics of the target object can be obtained.

[0082] In an embodiment of the present application, the terminal includes a display screen, which includes a transparent display area and a non-transparent display area; the color image acquisition module and the base map image acquisition module are both arranged at a position opposite to the transparent display area, and are respectively used to acquire the color image and the base map image of the target object through the transparent display area.

[0083] In an embodiment of the present application, a color image acquisition module and a base map image acquisition module are built into the terminal and are arranged at a position corresponding to the transparent display area. When performing image acquisition, the color image acquisition module acquires a color image of the target object by acquiring incident light that enters the terminal through the transparent display area, and the base map image acquisition module acquires a base map image of the target object by emitting and receiving light waves that can pass through the transparent display area.

[0084] For example, Fig. 9 is a schematic diagram of the structure of an exemplary display screen provided in an embodiment of the present application; Fig.10 FIG. 1 is a partial cross-sectional schematic diagram of an exemplary terminal along the AB direction provided in an embodiment of the present application. Fig. 9 and Fig.10 As shown, the terminal includes a display screen 10, a color image acquisition module 20 and a base map image acquisition module 30. The display screen 10 includes a transparent display area 11 and a non-transparent display area 12. The color image acquisition module 20 and the base map image acquisition module 30 are both arranged at positions corresponding to the transparent display area 11.

[0085] Fig.11is an optional flow chart of the image processing method provided in the embodiment of the present application. After the above S103, S201 is also included. The above S105 can be implemented by S1051-S1052. Taking the example of including S201 after S104, Fig.11 The steps shown are explained.

[0086] S201. Determine a noise reference feature based on a first feature and a second feature; the noise reference feature is a feature of the first feature used to optimize the noise area.

[0087] In an embodiment of the present application, when the terminal compares the first feature and the second feature, when it is determined that the correlation between certain features in the first feature and certain corresponding features in the second feature is lower than a preset threshold, it indicates that these features are noise in the color image, and the area where these features are located is the noise area in the color image. At the same time, these corresponding features in the second feature are noise reference features used to optimize the noise area.

[0088] S1051, denoising the noise area.

[0089] In an embodiment of the present application, after determining the noise region in the color image, the terminal may use methods such as wavelet transform, frequency domain processing or Laplace transform to remove the noise in the noise region.

[0090] S1052: restore the contour information, grayscale information and color information of the denoised area according to the noise reference feature.

[0091] In an embodiment of the present application, after the terminal eliminates the noise in the noise area, the terminal can restore the contour information, grayscale information and color information of the area after the noise is eliminated based on the noise reference feature in the determined first feature. For example, when the noise area is a diffraction fringe area, after the terminal eliminates the diffraction fringe in the color image, the contour information, grayscale information and color information of the area after the diffraction fringe is eliminated in the color image can be restored based on one or more of the texture features, grayscale features, shape features and spatial relationship features of the image area in the base image that is at the same position as the area where the diffraction fringe is located.

[0092] In the embodiment of the present application, since the base image can reflect the true characteristics of the target object, by comparing the color image with the base image, the area where shooting noise such as interference fringes, diffraction fringes and glare are located in the color image can be known. By eliminating the shooting noise and restoring the contour information, grayscale information and color information of the area after the shooting noise is eliminated, a color image that can better reflect the true characteristics of the target object can be obtained, thereby improving the image quality.

[0093] In some other embodiments of the present application, S201 may also be executed simultaneously with S104.

[0094] Fig.12 is an optional flow chart of the image processing method provided in the embodiment of the present application, which may include S301-S302 after S103 and before S105. Figure 4 For example, S103 and S301-S302 are included in the Fig.12 The steps shown are explained.

[0095] S301. Determine a blurred area and an optimized reference feature in a color image based on a first feature and a second feature; the optimized reference feature is a feature in the first feature used to optimize the blurred area.

[0096] In an embodiment of the present application, when the terminal determines that the correlation between certain features in the first feature and certain corresponding features in the second feature is lower than another preset threshold, it indicates that these features are blurred areas in the color image, then the area where these features are located is the blurred area in the color image, and at the same time, these corresponding features in the second feature are optimized reference features for optimizing the blurred area.

[0097] S302: Correct the contour information, grayscale information and color information of the blurred area according to the optimized reference features.

[0098] In an embodiment of the present application, after the terminal determines the blurred image in the color image, the terminal can restore the contour information, grayscale information and color information of the blurred area based on the optimized reference feature in the determined first feature. For example, the terminal can correct the contour information, grayscale information and color information of the blurred area in the color image based on one or more of the texture features, color features, shape features and spatial relationship features of the image area in the base image that is at the same position as the blurred area, so that the blurred area becomes clear.

[0099] In other embodiments of the present application, S301 - S302 may also be executed synchronously with S104 , which is not limited in the embodiments of the present application.

[0100] In the embodiment of the present application, since the base map image acquisition module including one or more of an infrared camera, a TOF camera, a structured light camera and an ultrasonic sensor has better adaptability to dark light environments, the color image captured by the color image acquisition module in a dark light environment is corrected based on the features in the base map image captured by the base map image acquisition module, which can compensate the color image and thus improve the imaging quality of the terminal in a dark light environment.

[0101] Fig.13 is an optional flow chart of the image processing method provided in the embodiment of the present application. After S105, S106 may also be included. Figure 4 For example, S106 is included after S105 in the example, and the combination Fig.13 The steps shown are explained.

[0102] S106: Use a deconvolution model to perform smoothing compensation processing on the processed color image.

[0103] In an embodiment of the present application, after obtaining the color image after correction and restoration processing, the terminal can input the obtained color image into a deconvolution model, and use the deconvolution model to perform smoothing compensation processing on the image area in the color image that has been corrected and restored, so as to obtain a smooth image with noise interference information eliminated.

[0104] In the embodiment of the present application, before S106, S401-S403 are also included, which are as follows:

[0105] S401, construct training set, validation set and test set.

[0106] In an embodiment of the present application, a sample set may be constructed based on the color image after correction processing and restoration processing, and the sample set may be divided into a training set, a verification set, and a test set.

[0107] S402: Using the training set and the validation set, train the preset initial deconvolution neural network model.

[0108] In an embodiment of the present application, a training set can be used to train a preset initial deconvolution neural network model, and a corresponding loss value can be calculated through a preset loss function, and the model parameters of the initial deconvolution neural network model can be adjusted. After the loss value reaches a preset condition, the training is completed and the trained initial deconvolution neural network model is output; the hyperparameters of the initial deconvolution neural network model are adjusted during the training process through a validation set.

[0109] S403: Using the test set, perform a smoothing compensation test on the trained initial deconvolution neural network model, and use the initial deconvolution neural network that meets the preset smoothing compensation conditions as the deconvolution model.

[0110] In an embodiment of the present application, a test set can be used to perform a smoothing compensation test on the trained initial deconvolution neural network model, and determine whether the smoothing compensation effect reaches a preset smoothing compensation effect. If the preset smoothing compensation effect is not reached, the training set and the validation set are used to continue training and parameter optimization of the initial deconvolution neural network model, and then the test set is used to continue testing until the smoothing compensation effect of the initial deconvolution neural network model obtained by the test reaches the preset smoothing compensation effect. At this time, the initial deconvolution neural network model that reaches the preset smoothing compensation effect is used as the deconvolution model.

[0111] In the embodiment of the present application, through training and testing, a deconvolution model with better smoothing compensation effect can be obtained, thereby obtaining a good image processing effect.

[0112] The present application also provides an image processing device, Fig.14 is a schematic diagram of the structure of the image processing device provided in the embodiment of the present application; Fig.14 As shown, the image processing device 2 includes: an extraction unit 21, used to extract the first feature of the color image and the second feature of the base image respectively; a determination unit 22, used to determine the noise area in the color image based on the first feature and the second feature; a processing unit 23, used to restore the contour information, grayscale information and color information of the noise area to obtain a composite image.

[0113] In some embodiments of the present application, the determination unit 22 is further used to determine a noise reference feature based on the first feature and the second feature; the noise reference feature is a feature of the first feature used to optimize the noise area; denoising the noise area; and restoring the contour information, grayscale information and color information of the denoised area according to the noise reference feature.

[0114] In some embodiments of the present application, the noise includes interference information caused by ambient light.

[0115] In some embodiments of the present application, the interference information caused by the ambient light includes at least one of the following: diffraction fringes, interference fringes, and glare.

[0116] In some embodiments of the present application, after respectively extracting the first feature of the color image and the second feature of the base image, and before obtaining the composite image, the determination unit 22 is further used to determine the blurred area and the optimized reference feature in the color image based on the first feature and the second feature; the optimized reference feature is a feature of the first feature used to optimize the blurred area; the processing unit 23 is further used to correct the contour information, grayscale information and color information of the blurred area according to the optimized reference feature.

[0117] In some embodiments of the present application, after the color image is restored and before the synthesized image is obtained, the processing unit 23 is further used to perform smoothing compensation processing on the processed color image using a deconvolution model.

[0118] An embodiment of the present application provides an image acquisition device, comprising: a display screen and an image acquisition device arranged in sequence along an optical axis direction; the display screen comprises a transparent display area; the image acquisition device comprises a color image acquisition module and a base map image acquisition module; the color image acquisition module and the base map image acquisition module are both arranged at positions opposite to the transparent display area; the color image acquisition module is used to obtain a color image of the target object based on imaging of incident light that passes through the transparent display area and is incident on the color image acquisition module along the optical axis direction; the base map image acquisition module is used to obtain a color image of the target object through the transparent display area.

[0119] The embodiment of the present application provides a terminal, Fig.15 is a schematic diagram of the structure of the terminal provided in the embodiment of the present application, such as Fig.15 As shown, the terminal 1 includes: a display screen 31, a memory 32, a processor 33, and the above-mentioned color image acquisition module 34 and base map image acquisition module 35; and the display screen 31, the memory 32, the processor 33, and the color image acquisition module 34 and the base map image acquisition module 35 are all connected through a bus 36. The display screen 31 includes a transparent display area; the color image acquisition module 34 and the base map image acquisition module 35 are both arranged at a position opposite to the transparent display area, the color image acquisition module 34 is used to acquire the color image of the target object through the transparent display area; the base map image acquisition module 35 is used to acquire the base map image of the target object through the transparent display area; the memory 32 is used to store an executable computer program; the processor 33 is used to implement the method provided in the embodiment of the present application when executing the executable computer program stored in the memory 32. For example, the image processing method provided in the embodiment of the present application.

[0120] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, in which a computer program is stored. When the computer program is executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, the image processing method provided by the embodiment of the present application.

[0121] In some embodiments of the present application, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0122] In some embodiments of the present application, executable instructions may be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0123] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0124] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0125] In summary, the present technical implementation scheme is adopted. Since the infrared camera or TOF structured light camera has better adaptability to dark light environments and is less affected by the opaque sub-pixels of the transparent display area, the color image collected by the RGB camera and containing noise such as interference fringes, diffraction fringes and glare and blurred areas is restored and corrected based on the features in the grayscale image or depth image captured by the infrared camera or TOF structured light camera. A color image with noise information such as interference fringes, diffraction fringes and glare eliminated and the blur problem improved can be obtained, thereby obtaining a color image that can better reflect the real characteristics of the target object and improve the quality of the obtained color image.

[0126] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to a terminal, the terminal includes a color image acquisition module and a base map image acquisition module, and the method includes: Using the color image acquisition module to acquire a color image of a target object; Using the base map image acquisition module to acquire a grayscale image of the target object; Extracting a first feature of the color image and a second feature of the grayscale image respectively; wherein the first feature is one or more of a texture feature, a shape feature, a color feature and a spatial relationship feature in the color image; and the second feature is one or more of a texture feature, a shape feature, a grayscale feature and a spatial relationship feature in the grayscale image; comparing the same type of features in the first feature and the second feature to determine a noise area in the color image; Restoring the contour information, grayscale information and color information of the noise area to obtain a composite image; Among them, the texture features are extracted by one of the following methods: geometric method, gray-level co-occurrence matrix and energy spectrum function method based on statistics, model method and signal processing method; the shape features are extracted by one of the following methods: Fourier transform, geometric matrix and finite element method; the color features are extracted by one of the following methods: color histogram, color aggregation vector and color correlation map; and the spatial relationship features are extracted by: automatically segmenting the image, dividing the objects or color areas contained in the image, extracting image features according to the objects or color areas, and establishing an index; or, evenly dividing the image into at least one regular sub-block, extracting features for each of the regular sub-blocks, and establishing an index, thereby realizing the extraction of the spatial relationship features.

2. The method according to claim 1, characterized in that The terminal includes a display screen, and the display screen includes a transparent display area; the color image acquisition module and the base map image acquisition module are both arranged at positions opposite to the transparent display area; The color image acquisition module acquires a color image of the target object through the transparent display area; The base image acquisition module acquires a grayscale image of the target object through the transparent display area.

3. The method according to claim 1, characterized in that: After respectively extracting the first feature of the color image and the second feature of the grayscale image, the method further includes: Compare the same type of features in the first feature and the second feature to determine a noise reference feature; the noise reference feature is a feature in the first feature used to optimize the noise area; The restoring process of the contour information, grayscale information and color information of the noise area to obtain a synthetic image includes: Performing denoising on the noise area; According to the noise reference features, the contour information, grayscale information and color information of the denoised area are restored.

4. The method according to claim 3, characterized in that The noise includes interference information caused by ambient light.

5. The method according to claim 4, characterized in that The interference information caused by ambient light includes at least one of the following: Diffraction fringes, interference fringes and glare.

6. The method according to claim 1 or 4, characterized in that: After respectively extracting the first feature of the color image and the second feature of the grayscale image, and before obtaining the composite image, the method further includes: Compare the same type of features in the first feature and the second feature to determine a blurred area and an optimized reference feature in the color image; the optimized reference feature is a feature in the first feature used to optimize the blurred area; According to the optimized reference features, the contour information, grayscale information and color information of the blurred area are corrected.

7. The method according to claim 1, characterized in that After the color image is restored and before the synthesized image is obtained, the method further includes: The deconvolution model is used to perform smoothing compensation on the processed color image.

8. The method according to claim 1, characterized in that: The color image acquisition module includes a three-primary-color RGB camera; the base-map image acquisition module includes an infrared camera.

9. An image processing device, characterized in that: include: An extraction unit, used to extract a first feature of the color image and a second feature of the grayscale image respectively; A determination unit, used for comparing the same type of features in the first feature and the second feature to determine the noise area in the color image; wherein the first feature is one or more of the texture feature, shape feature, color feature and spatial relationship feature in the color image; the second feature is one or more of the texture feature, shape feature, grayscale feature and spatial relationship feature of the grayscale image; the texture feature is extracted by one of the geometric method, the gray level co-occurrence matrix and energy spectrum function method based on statistics, the model method and the signal processing method; the shape feature is extracted by one of the Fourier transform, the geometric matrix and the finite element method; the color feature is extracted by one of the color histogram, the color aggregation vector and the color correlation map; and the spatial relationship feature is extracted by: automatically segmenting the image to divide the objects or color areas contained in the image, extracting image features according to the objects or color areas, and establishing an index; or, evenly dividing the image into at least one regular sub-block, extracting features for each of the regular sub-blocks, and establishing an index, thereby realizing the extraction of the spatial relationship feature; The processing unit is used to restore the contour information, grayscale information and color information of the noise area to obtain a composite image.

10. An image acquisition device, characterized in that: include: A display screen and an image acquisition device are sequentially arranged along the optical axis direction; The display screen includes a transparent display area; The image acquisition device includes a color image acquisition module and a base map image acquisition module; The color image acquisition module and the base image acquisition module are both arranged at positions opposite to the transparent display area; The color image acquisition module is used to obtain a color image of the target object based on imaging of incident light that passes through the transparent display area along the optical axis and is incident on the color image acquisition module; The base image acquisition module is used to obtain a grayscale image of the target object through the transparent display area; The color image and the grayscale image are used to extract a first feature of the color image and a second feature of the grayscale image respectively; the first feature is one or more of a texture feature, a shape feature, a color feature and a spatial relationship feature in the color image; the second feature is one or more of a texture feature, a shape feature, a grayscale feature and a spatial relationship feature in the grayscale image; the same type of features in the first feature and the second feature are compared to determine a noise area in the color image; the contour information, grayscale information and color information of the noise area are restored to obtain a composite image; Among them, the texture features are extracted by one of the following methods: geometric method, gray-level co-occurrence matrix and energy spectrum function method based on statistics, model method and signal processing method; the shape features are extracted by one of the following methods: Fourier transform, geometric matrix and finite element method; the color features are extracted by one of the following methods: color histogram, color aggregation vector and color correlation map; and the spatial relationship features are extracted by: automatically segmenting the image, dividing the objects or color areas contained in the image, extracting image features according to the objects or color areas, and establishing an index; or, evenly dividing the image into at least one regular sub-block, extracting features for each of the regular sub-blocks, and establishing an index, thereby realizing the extraction of the spatial relationship features.

11. A terminal, characterized in that: include: A display screen, a memory, a processor, and the color image acquisition module and the base map image acquisition module as described in claim 1; The display screen includes a transparent display area; The color image acquisition module and the base image acquisition module are both arranged at positions opposite to the transparent display area; The color image acquisition module is used to acquire a color image of the target object through the transparent display area; The base image acquisition module is used to acquire a grayscale image of the target object through the transparent display area; The memory is used to store an executable computer program; The processor is configured to implement the method according to any one of claims 1 to 8 when executing the executable computer program stored in the memory.

12. A computer-readable storage medium, characterized in that: A computer program is stored, which is used to cause a processor to implement the method according to any one of claims 1 to 8 when the program is executed.

Images