Image moire removal processing method and device based on multiple lenses, equipment and medium

Abstract

The invention relates to the field of multi-lens-based image moire removal processing, and provides a multi-lens-based image moire removal processing method and device, equipment and a medium, and the method comprises the steps: obtaining a shooting mode in response to a shooting instruction; when the shooting mode is a moire pattern removing mode, controlling a lens to shoot a target object at a plurality of different focal distances to obtain a reference image and a plurality of to-be-processed images; adjusting the plurality of to-be-processed images based on the reference image to obtain a plurality of to-be-fused images; and fusing the reference image and the plurality of to-be-fused images based on a preset artificial intelligence image fusion algorithm to obtain a target image. The technical problems that an existing moire pattern removing mode is complex and poor in effect are solved, pictures with different focal distances are obtained through the lens, multiple pictures without moire patterns or with moire pattern positions different are obtained and fused to obtain the moire pattern removed picture, and the moire pattern removing efficiency and effect of the image are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-lens based image moiré removal processing method, device, equipment and medium. Background Art

[0002] Generally, moiré is a visual phenomenon, where interference fringes occur on objects at certain intervals. When actually shooting, it is like rainbow-colored fringes appear between objects in the picture. This phenomenon is called moiré. For example, when sunny weather reflects water-wave-like fringes on mosquito nets or curtains, rainbow-colored fringes will appear when taking photos after wearing striped clothes. In other words, when the pixel spatial frequency of the photographic equipment is close to the spatial frequency value of the fringes in the picture, moiré will appear in the picture, such as Figure 1a The moiré phenomenon shown.

[0003] In the related art, deep learning methods are used to remove moiré interference. This method requires the construction of a data set, a large amount of image data collection, alignment, and training, and cannot reduce the image blur caused by various reasons. In addition, most of the existing deep learning-based moiré removal network models are deployed on the server side, with complex network structures, huge training data, and high computing resource configuration, which makes it difficult to directly deploy them on mobile terminal devices with limited computing power. Summary of the invention

[0004] In order to solve the above technical problem or at least partially solve the above technical problem, the present application provides a multi-lens based image de-moiré processing method, device, equipment and medium.

[0005] The present application provides a multi-lens image de-moiré processing method, the method comprising:

[0006] In response to a shooting instruction, acquiring a shooting mode;

[0007] When the shooting mode is a moiré removal mode, controlling the lens to shoot the target object at a plurality of different focal lengths to obtain a reference image and a plurality of images to be processed;

[0008] Adjusting the multiple images to be processed based on the reference image to obtain multiple images to be fused;

[0009] The reference image and the multiple images to be fused are fused based on a preset artificial intelligence image fusion algorithm to obtain a target image.

[0010] In one embodiment, the control lens is used to shoot the target object at multiple different focal lengths to obtain a reference image and multiple images to be processed, including:

[0011] Controlling the lens to shoot the target object at a reference focal length to obtain the reference image;

[0012] Controlling the lens to shoot the target object at a first focal length to obtain a first image to be processed; wherein the first focal length is less than the reference focal length; wherein when the first focal length is a plurality of different focal lengths, the number of the first images to be processed is a plurality;

[0013] The lens is controlled to shoot the target object at a second focal length to obtain a second image to be processed; wherein the second focal length is greater than the reference focal length; wherein when the second focal length is a plurality of different focal lengths, the number of the second images to be processed is multiple.

[0014] In one embodiment, the adjusting the multiple images to be processed based on the reference image to obtain multiple images to be fused includes:

[0015] Acquire the field angle of the reference image;

[0016] Cropping the first image to be processed based on the field of view angle to obtain a first image to be fused;

[0017] Acquire a downsampling coefficient between the reference image and the second image to be processed;

[0018] Downsampling is performed on the second image to be processed based on the downsampling coefficient to obtain a second image to be fused.

[0019] In one embodiment, the method of fusing the reference image and the plurality of images to be fused based on a preset artificial intelligence image fusion algorithm to obtain a target image includes:

[0020] Based on the artificial intelligence image fusion algorithm, feature extraction is performed on the reference image and the multiple images to be fused to obtain features of the reference image, features of the multiple images to be fused, and motion information between images;

[0021] Determining a target image to be fused from the plurality of images to be fused based on the inter-image motion information;

[0022] The reference image and the target image to be fused are aligned based on the reference image features and the image features to be fused corresponding to the target image to be fused, and a weighted average fusion process is performed on the noise of the aligned images to obtain the target image.

[0023] In one embodiment, the multi-lens based image de-moiré processing method further includes:

[0024] Display the shooting mode setting interface;

[0025] receiving a shooting mode setting instruction input by a user based on the shooting mode setting interface;

[0026] The shooting mode is set based on the shooting mode setting instruction.

[0027] The present application provides a multi-lens image de-moiré processing device, the device comprising:

[0028] A response acquisition module, used for acquiring a shooting mode in response to a shooting instruction;

[0029] A control module, used for controlling the lens to shoot the target object at a plurality of different focal lengths to obtain a reference image and a plurality of images to be processed when the shooting mode is a moiré removal mode;

[0030] An adjustment module, used for adjusting the multiple images to be processed based on the reference image to obtain multiple images to be fused;

[0031] The processing module is used to fuse the reference image and the multiple images to be fused based on a preset artificial intelligence image fusion algorithm to obtain a target image.

[0032] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the multi-lens image de-moiré processing method provided in any embodiment of the present application are implemented.

[0033] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multi-lens image de-moiré processing method provided in any embodiment of the present application are implemented.

[0034] The multi-lens image de-moiré processing method, device, equipment and medium provided in the embodiments of the present application respond to shooting instructions to obtain shooting mode; when the shooting mode is the de-moiré mode, the lens is controlled to shoot the target object at multiple different focal lengths to obtain a reference image and multiple images to be processed; the multiple images to be processed are adjusted based on the reference image to obtain multiple images to be fused; the reference image and the multiple images to be fused are fused based on a preset artificial intelligence image fusion algorithm to obtain a target image. The technical problem that the existing de-moiré methods are relatively complex and have relatively poor effects is solved. Pictures with different focal lengths are obtained through the lens to obtain multiple pictures without moiré or with different moiré positions, and then the pictures are fused to obtain pictures without moiré, thereby improving the efficiency and effect of image de-moiré. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1a An example diagram of the moiré phenomenon in the related art;

[0038] Figure 1b This is a diagram of an application scenario of a multi-lens image de-moiré processing method in one embodiment;

[0039] Figure 1c A schematic diagram of a multi-lens image de-moiré processing in a related art;

[0040] Figure 2 is a schematic diagram of a process flow of a multi-lens image de-moiré processing method in one embodiment;

[0041] Figure 3 is a schematic flow chart of a multi-lens image de-moiré processing method in another embodiment;

[0042] Figure 4 A schematic diagram of a process flow of a multi-lens image de-moiré processing method in yet another embodiment;

[0043] Figure 5 Schematic diagram of a process flow of a multi-lens image de-moiré processing method in another embodiment;

[0044] Figure 6 An example diagram of image de-moiré processing based on multiple lenses in one embodiment;

[0045] Figure 7 is a structural block diagram of a multi-lens image de-moiré processing device in one embodiment;

[0046] Figure 8 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0047] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0049] The multi-lens image de-moiré processing method of the present application can be applied to Figure 1b In the application environment shown. The multi-lens image de-moiré processing method is applied to a multi-lens image de-moiré processing system. The multi-lens image de-moiré processing system includes a terminal device 102 and an image acquisition device 104. The terminal device 102 communicates with the image acquisition device 104 through a network. In response to a shooting instruction, a shooting mode is obtained; when the shooting mode is a de-moiré mode, the lens is controlled to shoot the target object at multiple different focal lengths to obtain a reference image and multiple images to be processed; the multiple images to be processed are adjusted based on the reference image to obtain multiple images to be fused; the reference image and the multiple images to be fused are fused based on a preset artificial intelligence image fusion algorithm to obtain a target image. The terminal device 102 may be, but not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the image acquisition device 104 may be, but not limited to, various cameras, cameras on the terminal device 102, and the image acquisition device 104 may be set on the terminal device 102, or may be an independent device with image acquisition function.

[0050] Specifically, if the spatial frequency of the CCD (Charge-coupled Device) / CMOS (Complementary Metal Oxide Semiconductor) pixel of the photosensitive element is close to the spatial frequency of the stripes in the image, moiré will be produced. Usually, in order to eliminate moiré, the lens resolution should be much smaller than the spatial frequency of the photosensitive element. When this condition is met, stripes close to the photosensitive element will not appear in the image, and moiré will not be produced. In order to reduce moiré, a low-pass filter is installed in the digital camera to filter out the higher spatial frequency part of the image, which will of course reduce the sharpness of the image. If the pixel density of future digital cameras can be greatly improved and far exceeds the lens resolution, the annoying moiré will not appear.

[0051] Specifically, changing the camera angle. Since the angle between the camera and the object can cause moiré, slightly changing the camera angle (by rotating the camera) can eliminate or change any existing moiré, or changing the camera position; changing the focus. Too sharp a focus and high detail on a delicate pattern may cause moiré. Slightly changing the focus can change the clarity and help eliminate moiré; changing the focal length of the lens. Different lenses or focal length settings can be used to change or eliminate moiré; using software processing, such as Nikon Capture or Photoshop plug-ins, can eliminate any moiré that appears in the final image.

[0052] For example, currently, a typical method is to use deep learning to remove moiré interference, such as Figure 1c As shown in the figure, for the input image, the network does not use the pooling layer, and the convolution kernel with a step size of 2 is used to replace the pooling operation. The downsampling layer uses a large number of 3*3 convolution kernels with a step size of 1 during the downsampling process. The upsampling layer increases the image resolution during the deconvolution process and generates a 3-channel pixel map. Finally, the output is completed by superposition. The coarse-scale and fine-scale output images (Output) are processed by the neural network to remove the moiré of low-frequency and high-frequency components.

[0053] Among them, the Back Prop algorithm is a gradient descent method widely used in neural network training, which is used to calculate the update of network parameters. It efficiently calculates the gradient of the loss function relative to the network weights by applying the chain rule. The back propagation algorithm is not limited to the back propagation of errors, but also includes the calculation of gradients, so it is a gradient estimation method; that is, in a multi-layer feedforward neural network, the back propagation algorithm first calculates the output of the network, and then calculates the error of the output layer according to the loss function, then it backpropagates the error to the hidden layer and calculates the gradient of each weight, which are then used to update the network weights to minimize the loss function. The efficiency of the back propagation algorithm comes from the way it calculates gradients layer by layer, avoiding redundant calculations and obtaining a target image (Target) without moiré.

[0054] The above processing method requires the construction of a data set, a large amount of image data collection, alignment, and training, which cannot reduce the image blur caused by various reasons. Most of the existing deep learning-based moiré removal network models are deployed on the server side. Their network structure is complex, the training data is huge, and the computing resource configuration is high. It is difficult to directly deploy them on mobile terminal devices with limited computing power.

[0055] The present application obtains pictures with different focal lengths through a lens to obtain multiple pictures without moiré or with different moiré positions, and then fuses them to obtain pictures with moiré eliminated, thereby improving the efficiency and effect of image moiré removal.

[0056] In one embodiment, Figure 2 As shown, a multi-lens image de-moiré processing method is provided. This embodiment mainly applies the method to Figure 1b The terminal device 102 in FIG. 1 is used as an example. In this embodiment, the method includes the following steps:

[0057] Step 202: Acquire a shooting mode in response to a shooting instruction.

[0058] In the embodiment of the present application, any scene can be photographed by the image acquisition device 104 (such as a camera) on the terminal device 102, and a shooting mode, such as a moiré removal mode, can be selected through the terminal device 102 display interface, and a shooting instruction can be triggered by operations such as triggering a related control on the terminal device 102 or clicking an application logo corresponding to the image acquisition device 104, thereby obtaining a shooting mode. As an example, a shooting mode setting interface is displayed, a shooting mode setting instruction input by a user based on the shooting mode setting interface is received, and a shooting mode is set based on the shooting mode setting instruction.

[0059] In an embodiment of the present application, the terminal device 102 receives a shooting instruction and obtains the currently selected shooting mode. Specifically, the shooting mode setting parameters of the current shooting scene are obtained, and the shooting mode is determined according to the shooting mode setting parameters of the current shooting scene, such as the moiré removal mode.

[0060] Step 204 , when the shooting mode is the moiré removal mode, the lens is controlled to shoot the target object at a plurality of different focal lengths to obtain a reference image and a plurality of images to be processed.

[0061] In the embodiments of the present disclosure, different focal lengths refer to the focal length of the lens, and the shooting angle of view is changed by adjusting the focal length of the lens. The baseline image refers to the image shot at the baseline focal length, and the multiple images to be processed refer to the images shot at multiple focal lengths other than the baseline focal length. The target object can be selected and set according to the actual application scenario, such as clothes, animals, etc.

[0062] Specifically, when the shooting mode is the moiré removal mode, the lens is controlled to shoot the target object at multiple different focal lengths to obtain a reference image and multiple images to be processed. In some embodiments, the lens is controlled to shoot the target object at a reference focal length to obtain a reference image, and the lens is controlled to shoot the target object at multiple focal lengths other than the reference focal length to obtain multiple images to be processed.

[0063] Step 206: Adjust the multiple images to be processed based on the reference image to obtain multiple images to be fused.

[0064] In the embodiment of the present application, multiple images to be processed are adjusted based on a reference image to obtain multiple images to be fused.

[0065] Exemplarily, the image to be processed is cropped according to the reference image to obtain the image to be fused, and the image to be processed is downsampled according to the reference image to obtain the image to be fused; but the present application is not limited to this and does not specifically limit this.

[0066] Step 208: fuse the reference image and the multiple images to be fused based on a preset artificial intelligence image fusion algorithm to obtain a target image.

[0067] Specifically, a reference image and a plurality of images to be fused are fused to obtain a target image and output it.

[0068] Exemplarily, a neural network such as a correlation convolution is used to extract features of a reference image and multiple images to be fused and predict motion between images, and the reference image and multiple images to be fused are aligned and fused to obtain a target image. That is, the principle of multi-frame image fusion is used to remove the moiré pattern by removing the changed parts of the image and performing weighted averaging through the correlation of pixels in different frames, but this is not limited to this and the present application does not specifically limit this.

[0069] In the above-mentioned multi-lens image de-moiré processing method, the shooting mode is obtained by responding to the shooting instruction; when the shooting mode is the de-moiré mode, the lens is controlled to shoot the target object at multiple different focal lengths to obtain a reference image and multiple images to be processed; the multiple images to be processed are adjusted based on the reference image to obtain multiple images to be fused; the reference image and the multiple images to be fused are fused based on a preset artificial intelligence image fusion algorithm to obtain a target image. The technical problem that the existing de-moiré methods are relatively complex and have relatively poor effects is solved. Images with different focal lengths are obtained through the lens to obtain multiple images without moiré or with different moiré positions, and then the images are fused to obtain images without moiré, thereby improving the efficiency and effect of image de-moiré.

[0070] Figure 3 is a flowchart of image de-moiré processing based on multiple lenses in another embodiment, Figure 3 is Figure 2 Based on the illustrated embodiment, a possible implementation of S204 is further described.

[0071] Step 204a: when the shooting mode is the moiré removal mode, the lens is controlled to shoot the target object at a reference focal length to obtain a reference image.

[0072] Step 204b, controlling the lens to shoot the target object at a first focal length to obtain a first image to be processed; wherein the first focal length is smaller than a reference focal length; wherein when the first focal length is a plurality of different focal lengths, the number of first images to be processed is multiple.

[0073] Step 204c, controlling the lens to shoot the target object at a second focal length to obtain a second image to be processed; wherein the second focal length is greater than the reference focal length; wherein when the second focal length is a plurality of different focal lengths, the second image to be processed is a plurality of.

[0074] Exemplarily, the lens is controlled to take pictures at 1X, that is, to shoot the target object at a reference focal length to obtain a reference image, and then the lens is controlled to take pictures at 0.6X, that is, to shoot at the first focal length to obtain a first image to be processed, and then the lens is controlled to take pictures at 0.8X, that is, to shoot at the first focal length to obtain a first image to be processed, and then the lens is controlled to take pictures at 2X, that is, to shoot at the second focal length to obtain a second image to be processed, and then the lens is controlled to take pictures at 3X, that is, to shoot at the second focal length to obtain a second image to be processed.

[0075] In this way, the reference image and multiple images to be processed can be acquired quickly and accurately, thereby improving processing efficiency.

[0076] Figure 4 This is a flow chart of a method for removing moiré from an image based on multiple lenses in another embodiment. Figure 4 is Figure 2 Based on the embodiment shown, a possible implementation of S204 is further described as follows: Figure 4 As shown:

[0077] Step 206a, obtaining the field angle of the reference image, and cropping the first image to be processed based on the field angle to obtain the first image to be fused.

[0078] Step 206b: Obtain a downsampling coefficient between the reference image and the second image to be processed, and perform downsampling processing on the second image to be processed based on the downsampling coefficient to obtain a second image to be fused.

[0079] In some embodiments, different images to be processed are adjusted in different ways. For example, for a first image to be processed, the first image to be processed is cropped based on the field of view angle of a reference image to obtain a first image to be fused. For another example, for a second image to be processed, the second image to be processed is downsampled according to a downsampling coefficient between the reference image and the second image to be processed to obtain a second image to be fused.

[0080] Among them, the field of view of the reference image refers to the angle range in which the lens can receive the image in the imaging scene of the reference image. The first image to be processed obtained with a focal length less than the reference focal length has a larger field of view than the reference image. Therefore, the first image to be processed needs to be cropped according to the field of view size of the reference image to obtain the first image to be fused.

[0081] Among them, downsampling refers to reducing the resolution or size of the image, that is, reducing the number of pixels in the image. The second image to be processed obtained when the focal length is greater than the reference focal length has more pixels than the reference image. Therefore, it is necessary to downsample the second image to be processed to obtain a second image to be fused. Different second images to be processed have different degrees of downsampling, which can be represented by downsampling parameters. Different second images to be processed can be downsampled using different downsampling parameters.

[0082] Exemplarily, taking 0.6X image, 1X image, 2X image and 3X image as examples, the 0.6X image and the 1X image are compared using the existing alignment and cropping operation and then cropped to the same field of view size; the 2X image and the 3X image are downsampled to (h / 2, w / 3) and (h / 3, w / 3) sizes respectively.

[0083] In this way, an image to be fused that matches the reference image can be obtained, thereby improving the subsequent fusion effect.

[0084] Figure 5 This is a flow chart of a method for removing moiré from an image based on multiple lenses in another embodiment. Figure 5 is Figure 2 Based on the embodiment shown, a possible implementation of S208 is further described as follows: Figure 5 As shown:

[0085] Step 208a: extract features from the reference image and the multiple images to be fused based on an artificial intelligence image fusion algorithm to obtain features of the reference image, features of the multiple images to be fused, and motion information between images.

[0086] Step 208b: determining a target image to be fused from a plurality of images to be fused based on the inter-image motion information.

[0087] Step 208c, aligning the reference image and the target image to be fused based on the reference image features and the image features to be fused corresponding to the target image to be fused, and performing weighted average fusion processing on the noise of the aligned images to obtain the target image.

[0088] Among them, the artificial intelligence image fusion algorithm is a technology that uses artificial intelligence technology to fuse two or more photos into a new image. This technology analyzes the character features, light, background and other information in the photos through a deep learning algorithm, and then automatically matches and fuses them to generate a natural and harmonious image.

[0089] Specifically, by taking 1X as the reference image, using convolutional neural networks or other deep models to extract the image features of the reference image and multiple images to be fused and predict the motion information between images, such as two-stream networks. Multiple images to be fused are screened according to the motion information between images to obtain the target image to be fused, and then the reference image and the target image to be fused are aligned and fused. Using the principle of multi-frame fusion, the parts with changes in the image are eliminated and the weighted average is performed through the correlation of pixels of different images to remove the moiré.

[0090] Specifically, a weighted average is performed on the correlation of pixels of different images, for example, two images are added: S = S1 + S2; S represents the signal; σ t represents the time domain noise; the signal-to-noise ratio is When S1=S2, σ t1 =σ t2 ; The signal-to-noise ratio is S / σ t =2 0.5 (S1 / σ t1 ); when S1=S2=...=Sn; S / σ t =n 0.5 (S1 / σ t1 ).

[0091] From this, we can see that n frames of image will have n times the temporal noise on average. 0.5 times, so the method of removing time domain noise in signal processing is multi-frame averaging plus motion detection. If there is a change in the image, it will not be accumulated. If the image does not change, it will be accumulated and averaged.

[0092] For example, for a 1X reference image, a 0.6X image, and a 2X image, first adjust 0.6X and 2X to be the same as 1X, and then fuse the three frames of images. Specifically, extract the image features of the 1X image, the image features of the 0.6X image, the image features of the 2X image, and the inter-frame motion information between the two images. If there is a change between the 1X reference image and the 0.6X image, it is not necessary to fuse the 0.6X image, and only the 1X image and the 2X image need to be fused. That is to say, align the 1X image and the 2X image according to the image features and perform weighted averaging according to the above formula to obtain a fused target image.

[0093] Therefore, based on the artificial intelligence image fusion algorithm, the reference image and multiple images to be fused are fused to obtain the target image, thereby improving the efficiency and effect of image moiré removal.

[0094] As an example, Figure 6 As shown in the figure, the user selects the moiré removal mode in the camera interface. After clicking to take a photo, the camera automatically calls different zoom modes to take photos, and then obtains raw (unprocessed) format pictures taken at different focal lengths. After being sent to the algorithm for processing, it goes through the hardware ISP (Image Signal Processor), that is, image processing. Its main function is to post-process the signal output by the front-end image sensor and then display an image with moiré removed.

[0095] Specifically, by using different focal length lenses to obtain images with different focal lengths, we can obtain multiple images without moiré or with different moiré positions, and then use artificial intelligence multi-image fusion technology to obtain an image with moiré removed. By using the moiré principle and the characteristics of different focal length lenses showing different or no moiré, we can remove moiré, which solves the problem that most of the existing deep learning-based moiré removal network models are deployed on the server side, with complex network structures, huge training data, and high computing resource configuration, making it difficult to directly deploy them on mobile terminal devices with limited computing power.

[0096] Specifically, after the artificial intelligence image fusion algorithm obtains the 1X, 0.6X, 2X, and 3X unprocessed images, the 0.6X image is compared with the 1X image using the existing alignment and cropping operation and then cropped to the same field of view size; the 2X and 3X images are downsampled to (h / 2, w / 3) and (h / 3, w / 3) sizes respectively; the 1X, 0.6X_a, 2X_a, and 3X_a images are input into the algorithm, and the 1X image is used as the reference image. The multi-frame algorithm and the artificial intelligence image fusion algorithm are used to retain the 1X original signal part, and the inconsistent part of the moiré is eliminated, or the moiré part is replaced; finally, the image processed by the algorithm is sent back to the ISP for normal processing and then echoed.

[0097] The disclosed embodiment obtains multiple images without moiré or with different moiré positions by using lenses of different focal lengths to obtain images of different focal lengths, and then uses an artificial intelligence image fusion algorithm to obtain an image with the moiré eliminated. This solves the problem that most of the existing deep learning-based moiré removal network models are deployed on the server side, have a complex network structure, require huge amounts of training data, and require high computing resource configuration, making them difficult to be directly deployed on mobile terminal devices with limited computing power. There is no need to train a large model, and only needs to use existing mobile devices to collect images of different focal lengths in real time as input images for the artificial intelligence image fusion algorithm to achieve the purpose of removing moiré from photo images. This allows the use of existing telescopic lenses and artificial intelligence on existing devices, or the acquisition of photos with more focal lengths to achieve more accurate and excellent removal effects.

[0098] It should be understood that, although each step in the flow chart of the above-described embodiment is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flow chart of the above-described embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0099] In one embodiment, Figure 7 As shown, a multi-lens image de-moiré processing device is provided, comprising: a response acquisition module 702, a control module 704, an adjustment module 706 and a processing module 708, wherein:

[0100] A response acquisition module 702, used to acquire a shooting mode in response to a shooting instruction;

[0101] A control module 704 is used for controlling the lens to shoot the target object at a plurality of different focal lengths to obtain a reference image and a plurality of images to be processed when the shooting mode is a moiré removal mode;

[0102] An adjustment module 706, configured to adjust the plurality of images to be processed based on the reference image to obtain a plurality of images to be fused;

[0103] The processing module 708 is used to fuse the reference image and the multiple images to be fused based on a preset artificial intelligence image fusion algorithm to obtain a target image.

[0104] In the first embodiment of the present application, the control module 704 is specifically used for:

[0105] Controlling the lens to shoot the target object at a reference focal length to obtain the reference image;

[0106] Controlling the lens to shoot the target object at a first focal length to obtain a first image to be processed; wherein the first focal length is less than the reference focal length; wherein when the first focal length is a plurality of different focal lengths, the number of the first images to be processed is a plurality;

[0107] The lens is controlled to shoot the target object at a second focal length to obtain a second image to be processed; wherein the second focal length is greater than the reference focal length; wherein when the second focal length is a plurality of different focal lengths, the number of the second images to be processed is multiple.

[0108] In the first embodiment of the present application, the adjustment module 706 is specifically used to:

[0109] Acquire the field angle of the reference image;

[0110] Cropping the first image to be processed based on the field of view angle to obtain a first image to be fused;

[0111] Acquire a downsampling coefficient between the reference image and the second image to be processed;

[0112] Downsampling is performed on the second image to be processed based on the downsampling coefficient to obtain a second image to be fused.

[0113] In the first embodiment of the present application, the processing module 708 is specifically used for:

[0114] Based on the artificial intelligence image fusion algorithm, feature extraction is performed on the reference image and the multiple images to be fused to obtain features of the reference image, features of the multiple images to be fused, and motion information between images;

[0115] Determining a target image to be fused from the plurality of images to be fused based on the inter-image motion information;

[0116] The reference image and the target image to be fused are aligned based on the reference image features and the image features to be fused corresponding to the target image to be fused, and a weighted average fusion process is performed on the noise of the aligned images to obtain the target image.

[0117] In an implementation manner of the first embodiment of the present application, the multi-lens based image moiré removal processing device further includes:

[0118] A display module, used for displaying a shooting mode setting interface;

[0119] A receiving module, used for receiving a shooting mode setting instruction input by a user based on the shooting mode setting interface;

[0120] A setting module is used to set the shooting mode based on the shooting mode setting instruction.

[0121] The multi-lens image de-moiré processing device provided in the embodiment of the present application obtains the shooting mode by responding to the shooting instruction; when the shooting mode is the de-moiré mode, the lens is controlled to shoot the target object at multiple different focal lengths to obtain a reference image and multiple images to be processed; multiple images to be processed are adjusted based on the reference image to obtain multiple images to be fused; the reference image and multiple images to be fused are fused based on a preset artificial intelligence image fusion algorithm to obtain a target image. The technical problem that the existing de-moiré methods are relatively complex and the effect is relatively poor is solved. Images with different focal lengths are obtained through the lens to obtain multiple images without moiré or with different moiré positions to be fused to obtain images with moiré eliminated, thereby improving the efficiency and effect of image de-moiré.

[0122] For the specific definition of the multi-lens based image de-moiré processing device, please refer to the definition of the multi-lens based image de-moiré processing method above, which will not be repeated here. Each module in the above-mentioned multi-lens based image de-moiré processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0123] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for removing moiré patterns from an image based on a multi-lens is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0124] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In one embodiment, the multi-lens image de-moiré processing device provided in the present application can be implemented in the form of a computer program. The computer program can be Figure 8 The memory of the electronic device may store various program modules constituting the multi-lens image de-moiré processing device, for example, Figure 7 The response acquisition module 702, control module 704, adjustment module 706 and processing module 708 are shown. The computer program composed of various program modules enables the processor to execute the steps of the multi-lens image de-moiré processing method of various embodiments of the present application described in this specification.

[0126] For example, Figure 8 The electronic device shown can be Figure 7 The response acquisition module 702 in the multi-lens image de-moiré processing device shown in the figure performs step 202. The electronic device can perform step 204 through the control module 704. The electronic device can perform step 206 through the adjustment module 706. The electronic device can perform step 208 through the processing module 708.

[0127] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: step 202, in response to a shooting instruction, obtaining a shooting mode; step 204, when the shooting mode is a de-moiré mode, controlling the lens to shoot a target object at multiple different focal lengths to obtain a reference image and multiple images to be processed; step 206, adjusting the multiple images to be processed based on the reference image to obtain multiple images to be fused; step 208, fusing the reference image and the multiple images to be fused based on a preset artificial intelligence image fusion algorithm to obtain a target image.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Step 202, in response to a shooting instruction, obtaining a shooting mode; Step 204, when the shooting mode is a de-moiré mode, controlling the lens to shoot a target object at multiple focal lengths to obtain a reference image and multiple images to be processed; Step 206, adjusting the multiple images to be processed based on the reference image to obtain multiple images to be fused; Step 208, fusing the reference image and the multiple images to be fused based on a preset artificial intelligence image fusion algorithm to obtain a target image.

[0129] In one embodiment, the control lens is used to shoot the target object at multiple different focal lengths to obtain a reference image and multiple images to be processed, including:

[0130] Controlling the lens to shoot the target object at a reference focal length to obtain the reference image;

[0131] Controlling the lens to shoot the target object at a first focal length to obtain a first image to be processed; wherein the first focal length is less than the reference focal length; wherein when the first focal length is a plurality of different focal lengths, the number of the first images to be processed is a plurality;

[0132] The lens is controlled to shoot the target object at a second focal length to obtain a second image to be processed; wherein the second focal length is greater than the reference focal length; wherein when the second focal length is a plurality of different focal lengths, the number of the second images to be processed is multiple.

[0133] In one embodiment, the adjusting the multiple images to be processed based on the reference image to obtain multiple images to be fused includes:

[0134] Acquire the field angle of the reference image;

[0135] Cropping the first image to be processed based on the field of view angle to obtain a first image to be fused;

[0136] Acquire a downsampling coefficient between the reference image and the second image to be processed;

[0137] Downsampling is performed on the second image to be processed based on the downsampling coefficient to obtain a second image to be fused.

[0138] In one embodiment, the method of fusing the reference image and the plurality of images to be fused based on a preset artificial intelligence image fusion algorithm to obtain a target image includes:

[0139] Based on the artificial intelligence image fusion algorithm, feature extraction is performed on the reference image and the multiple images to be fused to obtain features of the reference image, features of the multiple images to be fused, and motion information between images;

[0140] Determining a target image to be fused from the plurality of images to be fused based on the inter-image motion information;

[0141] The reference image and the target image to be fused are aligned based on the reference image features and the image features to be fused corresponding to the target image to be fused, and a weighted average fusion process is performed on the noise of the aligned images to obtain the target image.

[0142] In one embodiment, the multi-lens based image de-moiré processing method further includes:

[0143] Display the shooting mode setting interface;

[0144] receiving a shooting mode setting instruction input by a user based on the shooting mode setting interface;

[0145] The shooting mode is set based on the shooting mode setting instruction.

[0146] By responding to the shooting instruction, the shooting mode is obtained; when the shooting mode is the moiré removal mode, the lens is controlled to shoot the target object at multiple different focal lengths to obtain a reference image and multiple images to be processed; the multiple images to be processed are adjusted based on the reference image to obtain multiple images to be fused; the reference image and the multiple images to be fused are fused based on a preset artificial intelligence image fusion algorithm to obtain a target image. The technical problem that the existing moiré removal method is relatively complex and has relatively poor effect is solved. Images with different focal lengths are obtained through the lens to obtain multiple images without moiré or with different moiré positions, and then the images are fused to obtain images without moiré, thereby improving the efficiency and effect of image moiré removal.

[0147] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multi-lens image de-moiré processing method, characterized in that: include: In response to a shooting instruction, acquiring a shooting mode; When the shooting mode is a moiré removal mode, controlling the lens to shoot the target object at a plurality of different focal lengths to obtain a reference image and a plurality of images to be processed; Adjusting the multiple images to be processed based on the reference image to obtain multiple images to be fused; The reference image and the multiple images to be fused are fused based on a preset artificial intelligence image fusion algorithm to obtain a target image.

2. The multi-lens image de-moiré processing method according to claim 1, characterized in that: The control lens is used to shoot the target object at multiple different focal lengths to obtain a reference image and multiple images to be processed, including: Controlling the lens to shoot the target object at a reference focal length to obtain the reference image; Controlling the lens to shoot the target object at a first focal length to obtain a first image to be processed; wherein the first focal length is less than the reference focal length; wherein when the first focal length is a plurality of different focal lengths, the number of the first images to be processed is a plurality; The lens is controlled to shoot the target object at a second focal length to obtain a second image to be processed; wherein the second focal length is greater than the reference focal length; wherein when the second focal length is a plurality of different focal lengths, the number of the second images to be processed is multiple.

3. The multi-lens image de-moiré processing method according to claim 2, characterized in that: The step of adjusting the plurality of images to be processed based on the reference image to obtain a plurality of images to be fused includes: Acquire the field angle of the reference image; Cropping the first image to be processed based on the field of view angle to obtain a first image to be fused; Acquire a downsampling coefficient between the reference image and the second image to be processed; Downsampling is performed on the second image to be processed based on the downsampling coefficient to obtain a second image to be fused.

4. The multi-lens image de-moiré processing method according to claim 1, characterized in that: The method of fusing the reference image and the plurality of images to be fused based on a preset artificial intelligence image fusion algorithm to obtain a target image includes: Based on the artificial intelligence image fusion algorithm, feature extraction is performed on the reference image and the multiple images to be fused to obtain features of the reference image, features of the multiple images to be fused, and motion information between images; Determining a target image to be fused from the plurality of images to be fused based on the inter-image motion information; The reference image and the target image to be fused are aligned based on the reference image features and the image features to be fused corresponding to the target image to be fused, and a weighted average fusion process is performed on the noise of the aligned images to obtain the target image.

5. The multi-lens image de-moiré processing method according to claim 1, characterized in that: Also includes: Display the shooting mode setting interface; receiving a shooting mode setting instruction input by a user based on the shooting mode setting interface; The shooting mode is set based on the shooting mode setting instruction.

6. A multi-lens image de-moiré processing device, characterized in that: The device comprises: A response acquisition module, used for acquiring a shooting mode in response to a shooting instruction; A control module, used for controlling the lens to shoot the target object at a plurality of different focal lengths to obtain a reference image and a plurality of images to be processed when the shooting mode is a moiré removal mode; An adjustment module, used for adjusting the multiple images to be processed based on the reference image to obtain multiple images to be fused; The processing module is used to fuse the reference image and the multiple images to be fused based on a preset artificial intelligence image fusion algorithm to obtain a target image.

7. The multi-lens image de-moiré processing device according to claim 6, characterized in that: The control module is specifically used for: Controlling the lens to shoot the target object at a reference focal length to obtain the reference image; Controlling the lens to shoot the target object at a first focal length to obtain a first image to be processed; wherein the first focal length is less than the reference focal length; wherein when the first focal length is a plurality of different focal lengths, the number of the first images to be processed is a plurality; The lens is controlled to shoot the target object at a second focal length to obtain a second image to be processed; wherein the second focal length is greater than the reference focal length; wherein when the second focal length is a plurality of different focal lengths, the number of the second images to be processed is multiple.

8. The multi-lens image de-moiré processing device according to claim 7, characterized in that: The adjustment module is specifically used for: Acquire the field angle of the reference image; Cropping the first image to be processed based on the field of view angle to obtain a first image to be fused; Acquire a downsampling coefficient between the reference image and the second image to be processed; Downsampling is performed on the second image to be processed based on the downsampling coefficient to obtain a second image to be fused.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images