Moire simulation method and device

Abstract

The invention belongs to the technical field of image simulation, and particularly discloses a moire simulation method and device, and the method comprises the steps: carrying out the zooming processing adaptive to the resolution of a display screen on an input image to be displayed on the display screen, and simulating a zoomed image; on the basis of the custom unevenness tolerance of the surface of the display screen, the vertical coordinates of the pixel units in the scaled image are disturbed randomly, and a display image containing unevenness information is simulated; determining point spread function information of the display image containing the unevenness information under a set view field height, performing convolution operation on the display image based on the point spread function information, and simulating an intermediate image carrying aberration; and performing detector sampling simulation processing on the intermediate image to simulate a final image containing moire patterns. In the simulation process, the process of displaying the image by the display screen and the process of shooting the display screen by the image capturing device are both considered, and the simulated moire pattern image is highly consistent with the actually shot image.

Description

Technical Field

[0001] The present application belongs to the field of image simulation technology, and more specifically, to a moiré simulation method and device. Background Art

[0002] Moiré is a high-frequency interference stripe that appears on the photosensitive element of digital cameras, scanners and other devices. These stripes make the image appear colorful and have no obvious shape pattern. The principle of its generation mainly involves the relationship between the spatial frequency of the pixels of the photosensitive element and the spatial frequency of the stripes in the image. When the two are close or the thin strip structure of the pattern crosses the structure of the sensor at a small angle, moiré may occur.

[0003] In industrial scenarios, display screens are photographed using imaging devices, and the images obtained are used to evaluate the performance of the display screens. Irregular moiré patterns, also known as screen moiré, often appear in the display images obtained by imaging devices, affecting the accuracy and efficiency of subsequent image-based display screen performance evaluations. For example, in the scenario of display screen defect detection, defects and moiré are easily confused, causing interference to defect detection. By simulating moiré to obtain moiré feature data, it can be used for parameter optimization of defect detection algorithms to improve the accuracy of defect detection.

[0004] In the related art, the moiré simulation method generally describes the spatial frequency aliasing between the basic light-emitting units and the sensor structure inside the display screen from the perspective of image processing. This ignores the optical characteristics of the imaging system, and the simulated screen moiré is poorly interpretable from the perspective of optical principles, and has low consistency with the moiré pattern obtained from actual photography, which limits the effectiveness and practical application scope of the moiré feature data for subsequent image processing. Summary of the invention

[0005] In view of the defects existing in the related art, the embodiments of the present application provide a moiré simulation method and device, aiming to solve the problem in the related art that the moiré simulation results are less consistent with the actually photographed moiré patterns.

[0006] In a first aspect, an embodiment of the present application provides a moiré simulation method, comprising:

[0007] Scaling the input image to be displayed on the display screen to adapt to the resolution of the display screen, and simulating a scaled image;

[0008] Based on the customized roughness tolerance of the display screen surface, the vertical coordinates of the pixel units in the scaled image are randomly disturbed to simulate a display image containing roughness information;

[0009] Determine the point spread function information of the display image containing the roughness information at the set field of view height, perform a convolution operation on the display image based on the point spread function information, and simulate an intermediate image carrying the aberration;

[0010] The detector sampling simulation is performed on the intermediate image to simulate a final image containing moiré patterns.

[0011] In a second aspect, the present application also provides a moiré simulation device, including:

[0012] A scaling simulation module is used to scale the input image to be displayed on the display screen to adapt to the resolution of the display screen, and simulate a scaled image;

[0013] An unevenness simulation module is used to randomly perturb the vertical coordinates of the pixel units in the scaled image based on the customized unevenness tolerance of the display surface, so as to simulate a display image containing unevenness information;

[0014] An aberration simulation module is used to determine the point spread function information of the display image containing the roughness information at a set field of view height, perform a convolution operation on the display image based on the point spread function information, and simulate an intermediate image carrying the aberration;

[0015] The detector sampling simulation module is used to perform detector sampling simulation processing on the intermediate image to simulate a final image containing moiré patterns.

[0016] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0018] In a fifth aspect, an embodiment of the present application further provides a computer program product, which, when executed on a processor, enables the processor to execute the method described in the first aspect or any possible implementation of the first aspect.

[0019] The moiré simulation method and device provided in the embodiments of the present application take into account both the process of a display screen displaying an image and the process of an imaging device photographing the display screen during the simulation process. The unevenness of the display screen surface is taken into account during the simulation process of the display screen displaying an image. After the original input image is scaled to adapt to the resolution of the display screen, the vertical coordinates of the pixel units in the image are randomly perturbed by a customized display screen unevenness tolerance to obtain a display image containing unevenness information for subsequent point spread function solution. In the process of modeling the process of an imaging device photographing the display screen, the aberration and other characteristics of the imaging system are taken into account, an intermediate image carrying aberrations is obtained through the point spread function information of the display image, and the intermediate image is further subjected to detector sampling simulation processing, thereby simulating a final image containing moiré that is highly consistent with the actual photographed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 This is one of the flow charts of the moiré simulation method provided in the embodiment of the present application;

[0022] Figure 2 is a schematic diagram of the principle of resampling a scaled image provided by an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of the principle of color filtering processing provided by an embodiment of the present application;

[0024] Figure 4 This is the second flow chart of the moiré simulation method provided in the embodiment of the present application;

[0025] Figure 5 is a schematic diagram of the structure of a moiré simulation device provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] The display screen in the embodiment of the present application is a device for displaying images and their colors, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display screen, or other pixel arrangement displays.

[0029] Figure 1 is one of the flow charts of the moiré simulation method provided in the embodiment of the present application, such as Figure 1 As shown, the method at least includes the following steps:

[0030] S101 , scaling an input image to be displayed on a display screen to adapt to the resolution of the display screen, and simulating a scaled image.

[0031] S102 . Based on the customized unevenness tolerance of the display screen surface, randomly perturb the vertical coordinates of the pixel units in the scaled image to simulate a display image containing unevenness information.

[0032] S103, determining point spread function information of a display image containing unevenness information at a set field of view height, performing a convolution operation on the display image based on the point spread function information, and simulating an intermediate image carrying aberrations.

[0033] S104 , performing detector sampling simulation processing on the intermediate image to simulate a final image containing moiré patterns.

[0034] The moiré simulation method provided in the embodiment of the present application takes into account both the process of the display screen displaying an image and the process of the imaging device photographing the display screen during the simulation process. The unevenness of the display screen surface is taken into account during the simulation process of the display screen displaying an image. After the original input image is scaled to adapt to the resolution of the display screen, the vertical coordinates of the pixel units in the image are randomly perturbed by the customized display screen unevenness tolerance to obtain a display image containing unevenness information for subsequent point spread function solution. In the process of modeling the process of the imaging device photographing the display screen, the aberration and other characteristics of the imaging system are taken into account, and an intermediate image carrying aberrations is obtained through the point spread function information of the display image, and the intermediate image is further subjected to detector sampling simulation processing, thereby simulating a final image containing moiré that has high consistency with the actual photographed image.

[0035] Specifically, during the simulation process, the process of displaying an image on the display screen is first simulated to improve the consistency between the simulated image and the actual displayed image.

[0036] In S101, based on the resolution of the display screen, the pixel distribution of the input image p1 to be displayed on the display screen is scaled according to the scaling ratio required by the display screen, thereby simulating a scaled image p2 displayed on the display screen. Optionally, the scaling of the input image p1 is implemented by an interpolation method. The interpolation method generates new pixel points by estimating the pixel values ​​at non-integer pixel positions, thereby changing the size of the image. The interpolation method specifically includes but is not limited to bilinear interpolation, nearest neighbor interpolation, bicubic interpolation, etc.

[0037] Optionally, after the input image p1 is scaled, further resampling is performed to ensure the visual effect and data accuracy of the input image p1 after the scaling. Specifically, the scaled image p2 is resampled according to the spatial distribution of the basic light-emitting units of the display screen. Figure 2 is a schematic diagram of the principle of resampling a scaled image provided by an embodiment of the present application, such as Figure 2 As shown, the pixel unit of the RGB scaled image is remodeled into a 3×3 sub-pixel array, and the pixel arrangement and distance of the display screen are modeled according to the following arrangement:

[0038]

[0039] The value of each element of the sub-pixel array is assigned according to the RGB three-channel value of the corresponding pixel unit of the original RGB scaled image to obtain a scaled resampled image.

[0040] In S102, due to the actual manufacturing process of the display screen, tiny protrusions and depressions may appear on the surface of the display screen, and these unevenness may cause the coordinates of the pixel units to shift. The unevenness tolerance refers to the deviation range of the surface of the optical element from the ideal state, that is, the maximum allowable variation of the degree of unevenness of the surface shape. The unevenness tolerance of the display screen surface can be customized.

[0041] In order to simulate the influence of the unevenness of the display screen surface on the imaging process, the vertical coordinates of each pixel unit in the scaled image p2 are randomly perturbed based on the unevenness tolerance of the display screen surface to obtain a display image p3 that is more consistent with the actual display image, and the display image p3 contains unevenness information.

[0042] In some embodiments, randomly perturbing the vertical coordinates of the pixel units in the scaled image in S102 specifically includes:

[0043] Generate a random disturbance value data set; the random disturbance value data set obeys a normal distribution, and the standard deviation is the unevenness tolerance of the display surface;

[0044] The vertical coordinates of the pixel cells in the scaled image are randomly perturbed based on a dataset of random perturbation values.

[0045] Specifically, the unevenness tolerance in the embodiment of the present application mainly refers to the flatness tolerance of the display screen surface, which is used to characterize the unevenness of the display screen surface within the plane where the display screen is located. Therefore, random perturbations are mainly performed on the vertical coordinates of the scaled image p2.

[0046] In order to better simulate the actual situation of the display screen displaying images, a random perturbation value data set is generated. The random perturbation value data set obeys the normal distribution, and the standard deviation is the unevenness tolerance of the display screen surface. Based on the random perturbation value data set, the vertical coordinates of the pixel units in the scaled image p2 are randomly perturbed to obtain a display image p3 containing unevenness information. It is also possible to consider generating a random perturbation value data set through uniform distribution.

[0047] Specifically, the display screen surface is divided into dot matrix, and each dot corresponds to a point light source, that is, a sampling coordinate, that is, a pixel unit. The XOY plane coordinate is constructed with the plane where the display screen is located, and the Z axis is constructed with the direction perpendicular to the display screen. Then the initial coordinate of the i-th pixel unit is (x i ,y i ,z i ),

[0048] The z coordinate of each pixel unit is randomly disturbed, and the random disturbance value is Δz generated according to the normal distribution, and the standard deviation is the unevenness tolerance of the display surface. Then the corrected coordinate of the i-th pixel unit is (x i ,y i ,z i +Δz i ), thereby obtaining point light sources that are not in the same plane for subsequent calculation of the point spread function.

[0049] The moiré simulation method provided in the embodiment of the present application obtains a display image containing unevenness information by simulating the unevenness of the surface of the display screen during the simulation of the image displayed on the display screen, thereby improving the accuracy of subsequent point spread function calculations and further improving the consistency between the moiré simulation image and the actual image.

[0050] In some embodiments, after S101, the moiré simulation method further includes:

[0051] Based on the angle between the lens of the imaging device and the display screen, the zoomed image or the displayed image containing uneven information is rotated and transformed.

[0052] Specifically, ideally, when an imaging device is used to capture an image of a display screen, the display screen and the imaging device lens should be placed in parallel; however, in the actual imaging process, the imaging device lens and the display screen are not absolutely parallel, and there may be a certain angle deviation; or the imaging device lens and the display screen are set at a certain angle.

[0053] Therefore, it is necessary to consider the angle between the lens of the imaging device and the display screen, and perform a rotation transformation on the scaled image p2 obtained in S101 (first perform a rotation transformation, then perform an unevenness simulation), or perform a rotation transformation on the display image p3 obtained in S102 (first perform an unevenness simulation, then perform a rotation transformation), so as to obtain a scene image p4 that is more consistent with the actual shooting.

[0054] The rotation transformation matrix satisfies:

[0055]

[0056] Among them, α represents the angle of rotation about the y-axis, β represents the angle of rotation about the x-axis, and γ represents the angle of rotation about the z-axis, with clockwise rotation being positive.

[0057] It is conceivable that, if the image is rotated and transformed during the process of displaying the image on the analog display screen, the displayed image in the related operations during the subsequent analog imaging device imaging process refers to the displayed image after the rotation transformation.

[0058] In S103, the point spread function (PSF) is an important parameter that describes the response of the imaging system to a point light source or a point object, and describes how the imaging system distributes the energy of a point light source or a point object to various positions on the imaging surface. Ideally, a point light source should be imaged as a perfect point, but in an actual system, due to factors such as diffraction and aberration, the point light source will be diffused into a bright spot of finite size (diffuse spot), and the shape and size of this bright spot constitute the PSF.

[0059] In order to simulate the image blur phenomenon caused by the aberration of the optical system, the point spread function information of the display image p3 containing the roughness information at the set field of view height is calculated, and the point spread function information is used to perform a convolution operation on the display image p3, so as to simulate the intermediate image carrying the aberration taken by the imaging device.

[0060] In some embodiments, determining the point spread function information of the display image containing the roughness information at the set field of view height in S103 specifically includes:

[0061] The point light sources corresponding to the pixel units in the display image p3 are sampled, and the point spread function of each sampled point light source at the set field of view height is determined based on ray tracing as the point spread function information of the display image p3.

[0062] Specifically, optical tracing predicts and evaluates the performance of the imaging system by simulating the propagation path of light in the optical system. The optical tracing process takes into account the effects of light reflection, refraction, and the influence of various optical parameters of the imaging system on the propagation of light, including the radius of curvature, thickness, and refractive index of the lens. Through ray tracing, we can obtain the specific position, direction, and intensity information of the light inside the imaging system and on the image plane, and further obtain the aberration characteristics of the imaging system.

[0063] Considering the geometric aberration of the image p3 displayed on the display screen after being captured by the imaging optical system, the diffuse spot distribution corresponding to the point light source on the image plane at the set field of view height is obtained based on optical tracing. The point spread function of the point light source at the set field of view height is obtained by discretizing it according to the size and resolution of the image plane pixels, which specifically satisfies:

[0064]

[0065] Among them, PSF p,q It represents the PSF corresponding to the pixel unit (p, q) in the diffuse speckle distribution map composed of discrete pixel units, a and b represent the two-dimensional coordinates corresponding to the center of the pixel unit, i represents the number of the object-side sampling light, m ​​represents the number of the object-side sampling light, n represents the number of sampling light lines detected on the image plane, and the round function performs rounding operation.

[0066] In actual optical systems, PSF is related to the field of view. Since the PSF distribution of adjacent pixel units is similar, sampling is used to reduce the amount of calculation. The point light sources corresponding to the pixel units in the display image p3 are sampled, and the point spread function of each sampled point light source is obtained using the above formula as the point spread function information of the display image p3.

[0067] In some embodiments, simulating an intermediate image with aberration in S103 specifically includes:

[0068] Perform convolution operations on the displayed image p3 and the point spread function of each sampling point light source to obtain an image set under a single field of view;

[0069] The image sets are fused through two-dimensional SINC interpolation to obtain intermediate images under different fields of view.

[0070] Specifically, the point spread function information of the display image p3 obtained through the above steps is used to perform convolution operations on the display image p3 and the point spread functions of each sampling point light source to obtain an image set under a single field of view; then the image set is fused by interpolation to obtain an overall image under different fields of view of the imaging optical system, that is, an intermediate image. In image fusion, interpolation can be used to smoothly transition the boundaries between different images and reduce artifacts after fusion.

[0071] Optionally, the interpolation method uses two-dimensional SINC interpolation, which specifically satisfies:

[0072]

[0073] Among them, I A represents the intermediate image considering the influence of aberration, X and Y represent the sampling point light source on the displayed image p3, S a,b Represents the value of the two-dimensional SINC function with the width of the field of view block corresponding to the sampling point light source (a, b) as the unit length, PSF a,b Indicates the PSF corresponding to the sampling point light source (a, b), I s Indicates that image p3 is displayed.

[0074] In some embodiments, in S104, the intermediate image is further subjected to detector sampling simulation processing, which may include: distortion processing, color filtering processing, noise addition processing, de-mosaicing, etc., to simulate the final image containing moiré patterns taken by the imaging device.

[0075] In some embodiments, the detector sampling simulation processing in S104 includes distortion processing, and S104 specifically includes:

[0076] Gridding the intermediate image based on the arrangement of pixel units to determine the distribution change of grid points after the distortion of the imaging optical system;

[0077] For non-integer grid points after distortion, nearest neighbor interpolation is used to assign values ​​to pixel units to obtain a distorted image.

[0078] Specifically, the intermediate image obtained by convolving the displayed image with the PSF only reflects the image blur caused by the aberration of the imaging optical system. For the image deformation effect caused by distortion, image remapping is required to obtain a distorted image that is closer to the actual captured image.

[0079] The process of image remapping is as follows: gridding the intermediate image based on the arrangement of pixel units, setting the same number of grid points in the object space as the arrangement of pixel units in the intermediate image, and calculating the distribution change of the grid points after the distortion of the imaging optical system; for non-integer grid points after distortion, the nearest neighbor interpolation is used to assign values ​​to the pixel units to obtain the distorted image. The distorted image is the initial image projected onto the image sensor after the imaging device captures the image.

[0080] In some embodiments, the detector sampling simulation process in S104 includes color filtering process, and S104 specifically includes:

[0081] The distorted image is resampled according to the color distribution of the color filter array to obtain a color filter array image.

[0082] Specifically, the color filter array (CFA) is a mosaic layer on top of the image sensor that is used to capture color information. CFA filters the light wavelengths so that light of a specific wavelength can pass through, allowing the image sensor to distinguish between light of different wavelengths and obtain the color information of the pixel.

[0083] Considering that in practical applications, the input image p1 is generally a color image. In order to better simulate the image acquisition process, the distorted image is subjected to color filtering: the distorted image is resampled according to the color distribution of CFA to obtain the CFA image. The values ​​of the corresponding channels in the distorted image are assigned to the grids of the corresponding colors in the color filter array. After resampling, the image resolution of each RGB channel decreases, and the image presents a mosaic effect.

[0084] Optionally, CFA is a Bayer filter array, including a red light filter, a blue light filter and two green light filters (the human eye is more sensitive to green light). This structure allows every four pixels to share a color value. Compared with the equivalent processing of three RGB colors, the restored image appears to the human eye with lower noise and clearer details. Figure 3 Schematic diagram of the principle of color filtering processing provided by the embodiment of the present application. Figure 3 As shown, after the incident light passes through the color filter covering the image sensor, only the light of the corresponding color can pass through the filters of different colors.

[0085] If the image displayed on the display screen is a single color, black and white image, or a grayscale image, etc., color filtering processing is not considered in the detector sampling simulation processing.

[0086] In some embodiments, the detector sampling simulation process in S104 includes a noise addition process. S104 specifically includes:

[0087] The color filter array image is subjected to noise addition processing to obtain a noise image including the noise of the image sensor inside the imaging device.

[0088] Specifically, noise processing refers to simulating image noise interference in a real environment. Through noise processing, a noise image containing the noise of the image sensor inside the imaging device is simulated, which is closer to the actual captured image.

[0089] The general process of noise addition is: determine the noise type, such as salt and pepper noise, Poisson noise, Gaussian noise, etc.; calculate the amount of noise that needs to be added based on the noise type and the required signal-to-noise ratio; randomly generate noise data that conforms to the noise distribution; superimpose the noise data on the color filter array image to generate a noise image after noise addition.

[0090] In some embodiments, the detector sampling simulation processing in S104 includes demosaicing, and S104 specifically includes:

[0091] The color filter array image is demosaiced by bilinear interpolation to obtain a final image containing moiré patterns.

[0092] Specifically, demosaicing analyzes the color information of surrounding pixels to guess and supplement the missing color information for each pixel. This process involves interpolation calculations to ensure that the generated color image is as close to the real scene as possible in color and detail.

[0093] Optionally, the color filter array image is demosaiced using bilinear interpolation, which uses the color values ​​of adjacent pixels for linear interpolation to obtain a moiré image containing full-color information. At the same time, since the color filtering process reduces the resolution of the image, demosaicing can restore the resolution of the image.

[0094] Color filtering captures color images through a specific color filter arrangement, while demosaicing reconstructs incomplete color sampling into a complete color image through interpolation calculation, thereby obtaining high-quality moiré simulation images.

[0095] The technical solution provided in the embodiment of the present application is further illustrated below by using a specific example.

[0096] Figure 4 FIG. 2 is a second flow chart of the moiré simulation method provided in an embodiment of the present application. Figure 4As shown in the figure, ① is the input image, ② is the LCD display, ③ is the imaging optical system, ④ is the color filter, and ⑤ is the image sensor of the imaging device. The resolution of the input image ① is 500×375; the screen resolution of the LCD display ② is 1500×1125, and the physical size is 160mm×120mm; the working distance between the imaging optical system ③ and the LCD display ② is 300mm, and the lens structure parameters of the imaging optical system ③ are shown in Table 1. The array arrangement of the color filter ④ is a Bayer filter array, the resolution of the image sensor ⑤ is 4500×3375, and the noise distribution is set to Gaussian.

[0097] Table 1

[0098]

[0099]

[0100] Combination Figure 4 In one embodiment of the present application, the moiré simulation method specifically includes the following process:

[0101] Step 1: Scale and resample the input image ① to form a scaled image that adapts to the screen resolution of the display screen ②. According to the resolution of the input image ① and the LCD display screen ②, the scaling ratio is 1:3; then use the bilinear interpolation method to obtain the pixel values ​​corresponding to the newly added pixels to obtain the RGB scaled image for projection display on the LCD display screen ②.

[0102] According to the spatial distribution of the basic light-emitting units of the display screen, the RGB scaled image is resampled. Figure 2 As shown, the pixel unit of the RGB scaled image is remodeled into a 3×3 sub-pixel array, and the pixel arrangement and distance of the LCD display ② are used for modeling. The specific arrangement is as follows:

[0103]

[0104] The value of each element of the sub-pixel array is assigned according to the RGB three-channel value of the corresponding pixel unit of the original RGB scaled image, and the resolution of the scaled resampled image is 4500×3375.

[0105] Step 2: Customize the roughness tolerance according to the surface roughness of the LCD display screen ②, correct the vertical coordinate of the scaled image obtained in step 1, and generate a display image containing roughness information for subsequent point spread function calculation.

[0106] Generate 4500×3375 initialization dot matrix coordinates, with an initial z coordinate of -300mm. Set the surface unevenness tolerance of LCD display screen ② to ±0.05mm, and assume that the surface unevenness obeys a normal distribution, that is, most deviations are concentrated within the tolerance range, and the middle value has the largest deviation. Generate a random perturbation Δz and apply it to the dot matrix coordinates, so that the z coordinate value of the dot matrix fluctuates within the range of -300.05mm to -299.95mm, that is, adjust the pixel position in the scaled image to reflect the surface unevenness of the actual LCD display screen ②. Using the corrected coordinates, point light sources that are not in the same plane can be generated, and these point light sources will be used for the subsequent calculation of the point spread function.

[0107] Step 3: Set the angle between the lens in the imaging optical system ③ and the LCD display screen ②, and rotate the RGB display image obtained in step 2 to obtain a scene image that is more consistent with the actual shooting.

[0108] Assuming that the angle between the lens of the imaging device and the display screen is 5°, the rotation angle is set to 5° around the x-axis, and the y-axis and z-axis directions remain unchanged. The corresponding rotation transformation matrix refers to the following formula:

[0109]

[0110] Then the overall rotation transformation matrix acting on the RGB display image is:

[0111]

[0112] Applying this matrix to the RGB display image will result in a scene image that is more consistent with the actual shooting. The resolution of the scene image is the same as that of the display image, both of which are 4500×3375.

[0113] Step 4: Determine the point spread function information of the imaging optical system ③ based on ray tracing, convolve it with the scene image obtained in step 3, and perform distortion processing to obtain the image after passing through the imaging optical system and before reaching the color filter ④.

[0114] The number of sampling field points on the scene image is set to 5×5, that is, 25 evenly distributed point light sources in the object space. The diffuse spot distribution of each point light source after passing through the optical system is obtained based on traditional ray tracing and discretized. Here, the number of pixels of a single PSF image is set to 64×64. The 25 PSFs are convolved with the scene image to form an image set under each single field of view. The image sets are fused through two-dimensional SINC interpolation to obtain the overall image under different fields of view of the imaging optical system:

[0115]

[0116] Furthermore, 4500×3375 grid points are set in the object space of the imaging optical system ③, and the distribution change of the grid points after the distortion of the imaging optical system ③ is calculated. For the non-integer grid points after distortion, the nearest neighbor interpolation method is used to assign values ​​to the pixel points of the image plane to obtain the overall image after considering the distortion effect. The resolution of this image is the same as that of the scene image in step 3.

[0117] Step 5: resample the image obtained in step 4 according to the color distribution of the Bayer color filter array. The color distribution of the Bayer CFA is as follows: Figure 3 As shown, the array size is consistent with the resolution of the image sensor ⑤, which is 4500×3375. Therefore, the RGB image obtained in step 4 can assign the grayscale value of the corresponding color channel to the detector pixel according to the color distribution in the color filter array to simulate the original reading of the image sensor ⑤ in the imaging device.

[0118] Furthermore, Gaussian or other types of distributed noise are added to the resampled CFA data to simulate the noise of each pixel during the working process of the image sensor⑤. The process of adding Gaussian noise distribution can be expressed as:

[0119] n=n 0 +p

[0120] N=O+n

[0121] In the formula, p represents a normal distribution with a mean of 0 and a standard deviation of σ, and n o It represents an image with the same dimension as the image O to be processed by the Bayer filter array and the same pixel value as the mean of the normal distribution. Here, the standard deviation σ is set to 0.1.

[0122] Step 6: De-mosaic the original readings of the RGB channels of the image sensor ⑤ using a bilinear interpolation method to restore the resolution of the image and obtain a full-resolution moiré image containing full-color RGB information.

[0123] Through the moiré simulation method provided in the embodiment of the present application, the final RGB image has colored curved moiré, and the distribution range of the moiré is concentrated within a certain range, which is consistent with the real shooting phenomenon in daily life. Compared with the existing moiré simulation process, considering the unevenness of the display surface imaging and the optical characteristics such as aberration caused by the imaging process, the final simulated moiré image is smoother and more natural.

[0124] Figure 5 is a schematic diagram of the structure of the moiré simulation device provided in the embodiment of the present application, such as Figure 5 As shown, the device at least includes:

[0125] A scaling simulation module 501 is used to scale the input image to be displayed on the display screen to adapt to the resolution of the display screen, and simulate a scaled image;

[0126] The roughness simulation module 502 is used to randomly perturb the vertical coordinates of the pixel units in the zoomed image based on the customized roughness tolerance of the display screen surface, so as to simulate a display image containing roughness information;

[0127] The aberration simulation module 503 is used to determine the point spread function information of the display image containing the roughness information at the set field of view height, perform a convolution operation on the display image based on the point spread function information, and simulate an intermediate image carrying the aberration;

[0128] The detector sampling simulation module 504 is used to perform detector sampling simulation processing on the intermediate image to simulate a final image containing moiré patterns.

[0129] In some embodiments, the roughness simulation module 502 is specifically used to:

[0130] Generate a random disturbance value data set, the random disturbance value data set obeys the normal distribution, and the standard deviation is the roughness tolerance;

[0131] The vertical coordinates of the pixel cells in the scaled image are randomly perturbed based on a dataset of random perturbation values.

[0132] In some embodiments, the aberration simulation module 503 is specifically used for:

[0133] The point light sources corresponding to the pixel units in the displayed image are sampled, and the point spread function of each sampled point light source at a set field of view height is determined based on ray tracing as the point spread function information of the displayed image.

[0134] In some embodiments, the aberration simulation module 503 is specifically used for:

[0135] Convolution operations are performed on the displayed image and the point spread function of each sampling point light source to obtain an image set under a single field of view;

[0136] The image sets are fused through two-dimensional SINC interpolation to obtain intermediate images under different fields of view.

[0137] In some embodiments, the detector sampling simulation process includes one or more of the following:

[0138] Distortion processing;

[0139] Noise processing;

[0140] Color filtering processing;

[0141] De-mosaicing.

[0142] In some embodiments, the detector sampling simulation processing includes distortion processing, and the detector sampling simulation module 504 is specifically used to:

[0143] The intermediate image is gridded based on the arrangement of pixel units to determine the distribution change of the grid points after the distortion of the imaging optical system;

[0144] For non-integer grid points after distortion, nearest neighbor interpolation is used to assign values ​​to pixel units to obtain a distorted image.

[0145] In some embodiments, the detector sampling simulation process further includes color filtering process, and the detector sampling simulation module 504 is specifically used for:

[0146] Resample the distorted image according to the color distribution of the color filter array to obtain a color filter array image;

[0147] The color filter array is a Bayer filter array.

[0148] In some embodiments, the detector sampling simulation processing further includes demosaicing, and the detector sampling simulation module 504 is specifically used for:

[0149] The color filter array image is demosaiced by bilinear interpolation to obtain a final image containing moiré patterns.

[0150] In some embodiments, the apparatus further comprises:

[0151] The rotation transformation module is used to perform rotation transformation on the scaled image or the displayed image based on the angle between the lens of the imaging device and the display screen.

[0152] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0153] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0154] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device. The device may include: at least one memory for storing programs and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the above embodiment.

[0155] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6As shown, the electronic device may include: a processor (processor) 601, a communication interface (Communications Interface) 602, a memory (memory) 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 can call the software instructions in the memory 603 to execute the method described in the above embodiment.

[0156] In addition, the logic instructions in the above-mentioned memory 603 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0157] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0158] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0159] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0160] The method steps in the embodiments of the present application can be implemented by hardware, or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (Random Access Memory, RAM), flash memory, read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0162] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0163] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A moiré simulation method, characterized in that: include: Scaling the input image to be displayed on the display screen to adapt to the resolution of the display screen, and simulating a scaled image; Based on the customized roughness tolerance of the display screen surface, randomly perturbing the vertical coordinates of the pixel units in the zoomed image to simulate a display image containing roughness information; Determine the point spread function information of the display image containing the roughness information at a set field of view height, perform a convolution operation on the display image based on the point spread function information, and simulate an intermediate image carrying aberrations; The intermediate image is subjected to a detector sampling simulation process to simulate a final image containing moiré patterns.

2. The moiré simulation method according to claim 1, characterized in that: The randomly perturbing the vertical coordinates of the pixel units in the zoomed image comprises: Generate a random disturbance value data set, wherein the random disturbance value data set obeys a normal distribution and a standard deviation is the unevenness tolerance; The vertical coordinates of the pixel units in the scaled image are randomly perturbed based on the random perturbation value data set.

3. The moiré simulation method according to claim 1, characterized in that: The step of determining the point spread function information of the display image containing the roughness information at a set viewing field height includes: The point light sources corresponding to the pixel units in the displayed image are sampled, and the point spread function of each sampled point light source at the set field of view height is determined based on ray tracing as the point spread function information of the displayed image.

4. The moiré simulation method according to claim 3, characterized in that: The simulating of an intermediate image carrying aberrations comprises: Perform convolution operations on the displayed image and the point spread function of each sampling point light source to obtain an image set under a single field of view; The image set is fused through two-dimensional SINC interpolation to obtain the intermediate image under different fields of view.

5. The moiré simulation method according to claim 1, characterized in that: The detector sampling simulation process includes one or more of the following: Distortion processing; Noise processing; Color filtering processing; De-mosaicing.

6. The moiré simulation method according to claim 1, characterized in that: The detector sampling simulation processing includes distortion processing, and the detector sampling simulation processing on the intermediate image includes: Performing grid processing on the intermediate image based on the arrangement of pixel units to determine the distribution change of grid points after being distorted by the imaging optical system; For non-integer grid points after distortion, nearest neighbor interpolation is used to assign values ​​to pixel units to obtain a distorted image.

7. The moiré simulation method according to claim 6, characterized in that: The detector sampling simulation processing further includes color filtering processing, and the detector sampling simulation processing on the intermediate image includes: Resampling the distorted image according to the color distribution of the color filter array to obtain a color filter array image; Wherein, the color filter array is a Bayer filter array.

8. The moiré simulation method according to claim 7, characterized in that: The detector sampling simulation processing further includes demosaicing, and the detector sampling simulation processing on the intermediate image includes: The color filter array image is demosaiced by bilinear interpolation to obtain a final image containing moiré patterns.

9. The moiré simulation method according to claim 1, characterized in that: The method further comprises: The zoomed image or the displayed image is rotationally transformed based on the angle between the lens of the imaging device and the display screen.

10. A moiré simulation device, characterized in that: include: A scaling simulation module is used to scale the input image to be displayed on the display screen to adapt to the resolution of the display screen, and simulate a scaled image; An unevenness simulation module, configured to randomly perturb the vertical coordinates of the pixel units in the zoomed image based on a user-defined unevenness tolerance of the display screen surface, so as to simulate a display image containing unevenness information; an aberration simulation module, used to determine the point spread function information of the display image containing the roughness information at a set field of view height, and perform a convolution operation on the display image based on the point spread function information to simulate an intermediate image carrying the aberration; The detector sampling simulation module is used to perform detector sampling simulation processing on the intermediate image to simulate a final image containing moiré patterns.

Images