Moire elimination method, electronic device and readable storage medium

Abstract

The present application discloses a method for eliminating moiré, an electronic device, and a readable storage medium, and belongs to the field of terminal technology. Applied in an electronic device, the method comprises: when the size of the first image is larger than the first target size, determining at least one tile image of the first image, the first image is an image with moiré, the target elimination model is a pre-trained neural network model for eliminating moiré, and the network structure of the target elimination model is an encoding-decoding structure; eliminating the moiré carried by each tile image in at least one tile image through the target elimination model, and obtaining at least one tile image with moiré eliminated; determining a target image with moiré eliminated based on at least one tile image with moiré eliminated. While ensuring the elimination of moiré in images of various sizes, the present application reduces the memory occupied by the target elimination model, thereby improving the efficiency of the operation of the electronic device.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method for eliminating moiré, an electronic device, and a readable storage medium. Background Art

[0002] With the development of terminal technology, electronic devices have been applied in all aspects of user life. When using electronic devices, users may need to use the camera of the electronic device to shoot the content displayed on the display screen of other devices. If the sampling frequency of the camera of the electronic device is lower than the frequency of stripe changes on the display screen, moiré will be generated in the image captured by the camera of the electronic device. Since moiré will affect the display quality of the image, it is usually necessary to eliminate the moiré in the image.

[0003] At present, when it is necessary to eliminate moiré in an image, the image with moiré can be input into a pre-trained neural network model for eliminating moiré. For example, the neural network model for eliminating moiré can be an Open Neural Network Exchange (ONN) model, and the moiré in the image can be eliminated through the ONNx model.

[0004] However, in order to ensure that moiré patterns can be eliminated in images of various sizes, the neural network model for eliminating moiré patterns is usually a neural network model with a pyramid structure. However, the neural network model with a pyramid structure occupies more memory, which reduces the efficiency of the operation of the electronic device. Summary of the invention

[0005] The present application provides a moiré elimination method, an electronic device, and a readable storage medium, which can be used to solve the problem in the related art that the neural network model occupies a large amount of memory, resulting in low operating efficiency of the electronic device. The technical solution is as follows:

[0006] In a first aspect, a method for eliminating moiré patterns is provided, which is applied to an electronic device, and the method comprises:

[0007] In a case where the size of the first image is larger than the first target size, at least one tile image of the first image is determined, the first image is an image with moiré patterns, the size of each tile image in the at least one tile image is the first target size, the first target size is the size of an image required to be input by a target elimination model, the target elimination model is a pre-trained neural network model for eliminating moiré patterns, and the network structure of the target elimination model is an encoding-decoding structure;

[0008] Eliminating the moiré pattern in each of the at least one tile image by using the target elimination model, thereby obtaining at least one tile image from which the moiré pattern is eliminated;

[0009] Based on the at least one moiré-eliminated tile image, a moiré-eliminated target image is determined.

[0010] In this way, when the size of the first image is large and does not meet the size requirements of the pre-trained target elimination model, at least one tile image of the first image can be determined. Since the size of at least one tile image meets the size requirements of the target elimination model, the moiré pattern in at least one tile image can be eliminated through the target elimination model. Since it is possible to eliminate the moiré pattern in the first image by determining at least one tile image of the first image and eliminating the moiré pattern of each tile image in at least one tile image when the size of the first image is large, it is ensured that the moiré pattern in images of various sizes can be eliminated. In addition, since the network structure of the target elimination model is an encoding-decoding structure, this network structure is relatively simple and occupies less memory, thereby improving the efficiency of the operation of the electronic device.

[0011] As an example of the present application, when a size of the first image is larger than a first target size, determining at least one tile image of the first image includes:

[0012] When the size of the first image is larger than the first target size, determining the number of slices for slicing the first image according to the size of the first image, the first target size, and a preset overlap size;

[0013] When the number of slices is greater than 1, a cutting operation is performed on the first image according to the first target size, the preset overlapping size and the number of slices to obtain a plurality of slice images;

[0014] When the number of slices is 1, the first image is resampled to obtain a first reference image, the size of the first reference image is the first target size, and the first reference image is a sliced ​​image of the first image.

[0015] In this way, by determining the number of slices for slicing the first image, the processing method for the first image can be accurately selected, thereby improving the accuracy of processing the first image.

[0016] As an example of the present application, when the number of slices is greater than 1, the first image is cut according to the first target size, the preset overlap size and the number of slices to obtain a plurality of slice images, including:

[0017] When the number of slices is greater than 1, determining a second target size according to the first target size, the preset overlap size, and the number of slices, wherein the second target size is a size that does not require resampling and is greater than the first target size;

[0018] When the size of the first image is not the second target size, resampling the first image to obtain a second reference image, wherein the size of the second reference image is the second target size;

[0019] According to the first target size, the preset overlapping size and the number of slices, a cutting operation is performed on the second reference image to obtain the multiple slice images.

[0020] In this way, when the size of the first image is not the second target size, the first image can be resampled to ensure that the first image is completely cut, thereby avoiding the loss of image information of the first image to the greatest extent possible.

[0021] As an example of the present application, when a size of the first image is larger than a first target size, determining at least one tile image of the first image includes:

[0022] When the size of the first image is larger than the first target size, a cutting operation is performed on the first image according to the first target size and a preset overlapping size to obtain at least one third reference image, wherein the size of the at least one third reference image is not larger than the first target size;

[0023] In the case that there is a third reference image whose size is smaller than the first target size in the at least one third reference image, resampling the third reference image whose size is smaller than the first target size so that the size of the resampled third reference image is the first target size;

[0024] The un-resampled third reference image and the resampled third reference image are both determined as tile images of the first image to obtain the at least one tile image.

[0025] In this way, by directly cutting the first image according to the first target size and the preset overlap size, the computational complexity is reduced and the cutting efficiency is improved. In addition, by resampling the third reference image whose size is smaller than the first target size, it is ensured that the first image can maintain complete image information after being cut, avoiding the loss of image information and improving the accuracy of image cutting.

[0026] As an example of the present application, determining a target image for removing moiré based on the at least one moiré-removed tile image includes:

[0027] When the number of the at least one moiré-eliminating tile image is 1, determining the moiré-eliminating tile image as the target image;

[0028] In the case that the number of the at least one moiré-eliminating tile image is multiple, the multiple moiré-eliminating tile images are fused to obtain the target image.

[0029] In this way, when the number of at least one moiré-eliminating tile image is different, different methods are used to determine the target image, thereby improving the reliability of determining the target image.

[0030] As an example of the present application, when the number of the at least one moiré-eliminating tile image is multiple, fusing multiple moiré-eliminating tile images to obtain the target image includes:

[0031] In a case where the number of the at least one tile image for eliminating moiré is plural, determining a cutting position of each tile image in the plurality of tile images;

[0032] Determine a fusion matrix corresponding to each of the slice images according to the cutting position corresponding to each of the slice images, wherein the fusion matrix is ​​used to make the overlapping parts of the corresponding slice images undergo color gradient change;

[0033] Multiplying each of the tile images with the corresponding fusion matrix respectively to obtain a plurality of tile images with color gradients;

[0034] According to the cutting position of each of the segmented images, the plurality of segmented images with color gradients are merged to obtain the target image.

[0035] In this way, at least one moiré-eliminating tile image is fused through a fusion matrix, thereby avoiding an obvious boundary sense in the fused image and improving the image display quality.

[0036] As an example of the present application, the method further includes:

[0037] When the size of the first image is smaller than the first target size, resampling the first image to obtain a fourth reference image, where the size of the fourth reference image is the first target size;

[0038] The target image is obtained by eliminating the moiré patterns of the fourth reference image using the target elimination model.

[0039] In this way, when the size of the first image is smaller than the first target size, the first image is resampled so that the size of the obtained fourth reference image meets the size requirement of the target elimination model, thereby ensuring the elimination of moiré patterns for images of various sizes and improving the reliability of image moiré elimination.

[0040] As an example of the present application, the method further includes:

[0041] When the size of the first image is not the first target size, before determining at least one tile image of the first image, filtering processing is performed on the first image.

[0042] In this way, by performing low-pass filtering on the first image, the frequency aliasing in the first image is avoided from being aggravated when the first image is subsequently resampled, and after the moiré in the first image is eliminated by the target elimination model, a better moiré removal effect can be achieved.

[0043] As an example of the present application, before eliminating the moiré patterns in each of the at least one tile image by using the target elimination model to obtain at least one tile image with the moiré patterns eliminated, the method further includes:

[0044] Acquire training data, the training data comprising a plurality of negative sample images and a plurality of positive sample images, each of the plurality of negative sample images carries moiré patterns, each of the plurality of positive sample images does not carry moiré patterns, and the plurality of negative sample images correspond one-to-one to the plurality of positive sample images;

[0045] The initial elimination model is iteratively trained based on the training data to obtain the target elimination model, and the network structure of the initial elimination model is the encoding-decoding structure.

[0046] In this way, since the target elimination model has a simple structure with a coding-decoding network structure, the target elimination model occupies less memory and can be deployed in electronic devices, and the operating efficiency of the electronic devices is guaranteed.

[0047] As an example of the present application, the obtaining of training data includes:

[0048] Acquire the multiple negative sample images, where each negative sample image is obtained by photographing the display content of the corresponding display screen;

[0049] For each negative sample image among the multiple negative sample images, obtaining a screenshot image of display content of the display screen corresponding to each negative sample image;

[0050] Performing a topological transformation on the screenshot image corresponding to each negative sample image to obtain a topologically transformed screenshot image corresponding to each negative sample image, wherein data features of the topologically transformed screenshot image are the same as data features of the corresponding negative sample image;

[0051] The color features of each negative sample image are transferred to the corresponding topological transformation screenshot image to obtain the positive sample image corresponding to each negative sample image.

[0052] In this way, by unifying the color of each negative sample image with the corresponding positive sample image, the pressure on image brightness learning during model training is reduced, allowing the model to focus on learning the removal of moiré patterns, thereby avoiding changes in image color during the subsequent moiré elimination process and reducing damage to image information.

[0053] As an example of the present application, when a size of the first image is larger than a first target size, determining at least one tile image of the first image includes:

[0054] displaying the first image;

[0055] receiving a first user operation on the first image;

[0056] In response to the first user operation, when a size of the first image is larger than the first target size, at least one tile image of the first image is determined.

[0057] In this way, the moiré removal operation in the first image is performed in response to the first user operation triggered by the user, thereby improving interactivity with the user.

[0058] In a second aspect, an electronic device is provided, wherein the structure of the electronic device includes a processor and a memory, wherein the memory is used to store a program that supports the electronic device to execute the moiré elimination method provided in the first aspect, and to store data involved in implementing the moiré elimination method described in the first aspect. The processor is configured to execute the program stored in the memory. The electronic device may also include a communication bus, wherein the communication bus is used to establish a connection between the processor and the memory.

[0059] In a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the moiré elimination method described in the first aspect.

[0060] In a fourth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the moiré elimination method described in the first aspect.

[0061] The technical effects obtained by the above-mentioned second aspect, third aspect and fourth aspect are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of a display screen provided in an embodiment of the present application that displays content through discrete display pixels;

[0063] Figure 2 It is a schematic diagram of a camera provided in an embodiment of the present application capturing an image through a discrete acquisition method;

[0064] Figure 3 is a schematic diagram of an image with moiré patterns provided in an embodiment of the present application;

[0065] Figure 4 is a schematic diagram of a neural network model with a pyramid structure provided in an embodiment of the present application;

[0066] Figure 5 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0067] Figure 6 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0068] Figure 7 is a block diagram of a software system of an electronic device provided in an embodiment of the present application;

[0069] Figure 8 It is a schematic flow chart of a method for eliminating moiré provided in an embodiment of the present application;

[0070] Fig. 9 is a schematic diagram of a preset overlap size provided in an embodiment of the present application;

[0071] Fig.10 It is a schematic diagram of representing different fusion matrices by different binary images provided in an embodiment of the present application;

[0072] Fig.11 is a comparative schematic diagram of fusing slice images in different ways provided in an embodiment of the present application;

[0073] Fig.12 is a schematic diagram comparing the effects of low-pass filtering on moiré patterns provided in an embodiment of the present application;

[0074] Fig.13 It is a schematic flow chart of another method for eliminating moiré provided in an embodiment of the present application;

[0075] Fig.14It is a flowchart of a model training method provided in an embodiment of the present application;

[0076] Fig.15 is a schematic diagram of a negative sample image with different depth information provided by an embodiment of the present application;

[0077] Fig.16 is a schematic diagram of a processing flow of a screenshot image corresponding to each negative sample image provided by an embodiment of the present application;

[0078] Fig.17 This is an effect diagram of image information affected during the moiré removal process provided by an embodiment of the present application;

[0079] Fig.18 It is a schematic flow chart of another method for eliminating moiré provided in an embodiment of the present application;

[0080] Fig.19 It is a schematic flow chart of another method for eliminating moiré provided in an embodiment of the present application;

[0081] Fig. 20 It is a flowchart of another model training method provided in an embodiment of the present application;

[0082] Fig.21 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0084] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0085] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0086] In one scenario, when a user is using an electronic device such as a mobile phone, he or she may need to use the camera of the electronic device to shoot content displayed on other display screens. Since the display screen displays content through discrete display pixels (for example, Figure 1 As shown), and the camera also collects images by discrete sampling, for example, see Figure 2 Figure (a) or Figure 2 In the figure (b), after the camera captures the scene, a discrete image is obtained. In this way, if the sampling frequency of the camera of the electronic device is lower than the frequency of the stripe changes of the captured display screen, moiré patterns will be generated in the image captured by the camera of the electronic device (for example, Figure 3 As shown in FIG. 1 , the moiré pattern will affect the display quality of the image, and therefore, it is usually necessary to eliminate the moiré pattern in the image.

[0087] At present, when it is necessary to eliminate moiré in an image, the image with moiré can be input into a pre-trained neural network model for eliminating moiré. For example, the neural network model for eliminating moiré can be an onnx model, through which the moiré in the image can be eliminated.

[0088] However, in order to ensure that moiré patterns of various sizes can be eliminated, the neural network model for eliminating moiré patterns is usually a pyramid structure (for example, Figure 4 The neural network model of the pyramid structure occupies more memory, which reduces the efficiency of the electronic device. If the neural network model of the pyramid structure is placed in the cloud, although the problem of low operating efficiency is solved, it needs to be connected to the Internet during use, which is inconvenient to use and may cause the leakage of user privacy.

[0089] In order to ensure that moiré patterns in images of various sizes can be eliminated, and to reduce the memory occupied by the neural network model to improve the operating efficiency of the electronic device, an embodiment of the present application provides a method for eliminating moiré patterns. In this method, when the size of the first image is large and does not meet the size requirements of the pre-trained target elimination model, at least one slice image of the first image can be determined, and the size of the at least one slice image meets the size requirements of the target elimination model. The network structure of the target elimination model is a coding-decoding structure, and the moiré patterns in the at least one slice image can be eliminated by the target elimination model. Since the moiré patterns in the first image can be eliminated by determining at least one slice image of the first image and eliminating the moiré patterns of each slice image in the at least one slice image when the size of the first image is large, it is ensured that the moiré patterns in images of various sizes can be eliminated. Since the network structure of the target elimination model is a coding-decoding structure, this network structure is relatively simple and occupies less memory, thereby improving the operating efficiency of the electronic device. And because the target elimination model has a simple structure, it does not need to be deployed in the cloud, avoiding the leakage of user privacy.

[0090] For ease of understanding, before a detailed introduction to the moiré elimination method provided in an embodiment of the present application, the application scenarios involved in the embodiment of the present application are introduced by taking a mobile phone as an example of an electronic device.

[0091] Please refer to Figure 5 , Figure 5 is a schematic diagram of an application scenario according to an exemplary embodiment. In a possible scenario, when a user is using a mobile phone, he or she may need to take a photo of the content displayed on the display screen of other electronic devices, for example, take a photo of the text displayed on the display screen. In this case, see Figure 5 In Figure (a), the user can use the mobile phone camera to collect the content displayed on the display. If the sampling frequency of the mobile phone camera is less than the fringe change rate of the display, then see Figure 5 As shown in Figure (b) in the figure, the captured image displayed on the mobile phone display will have moiré (moiré is represented by a striped pattern in the drawings of the embodiments of the present application). In order to eliminate the moiré in the captured image, the user can click the "moiré elimination" button displayed on the current interface; in response to the click operation on the "moiré elimination" button, the mobile phone can resample and / or cut the captured image when the size of the captured image is larger than the first target size to determine at least one tile image of the captured image; the moiré in each tile image of at least one tile image is eliminated through the pre-trained target elimination model to obtain at least one tile image with the moiré eliminated; thereafter, based on the at least one tile image with the moiré eliminated, a display is shown as follows Figure 5The captured image with moiré removed is shown in Figure (c).

[0092] It should be noted that the first target size is the size of the image required to be input by the target elimination model, and the first target size can be set in advance according to demand. The first target size can be 1024 (number of pixels per row) * 1024 (number of pixels per column), 2048*1024 or 2048*2048, etc. The embodiment of the present application takes the first target size of 1024*1024 as an example for explanation.

[0093] In another application scenario, after obtaining the photographed image with the moiré removed, the mobile phone can store both the photographed image with the moiré and the photographed image with the moiré removed.

[0094] In another application scenario, when the mobile phone collects a captured image, it can also automatically identify whether the captured image carries moiré patterns, and when the captured image carries moiré patterns, the mobile phone can automatically perform the above-mentioned operations without the need for manual triggering by the user. Figure 5 The method described in the scenario eliminates the moiré in the captured image through the target elimination model to obtain a captured image with the moiré eliminated.

[0095] Please refer to Figure 6 , Figure 6 is a schematic diagram of an application scenario according to an exemplary embodiment. In a possible scenario, after a user obtains multiple images with moiré patterns through a mobile phone, the moiré patterns in the multiple images can be removed in batches. For example, see Figure 6 In (a), when the mobile phone displays the picture selection interface, the user can select the image that needs to remove the moiré pattern; in response to the user's selection operation, see Figure 6 In Figure (b), the selected mark corresponding to the image selected by the user is highlighted (in the drawings of the embodiments of the present application, the selected mark is a circle, and the circle is blackened to indicate the highlighting of the selected mark). After the user has completed the selection, the "moiré removal" button can be clicked; in response to the click operation on the "moiré removal" button, the mobile phone can sequentially eliminate the moiré in the selected image through the target elimination model. Among them, for each selected image, when the size of the selected image is larger than the first target size, the selected image can be resampled and / or cut to determine at least one tile image of the selected image; then the moiré in each tile image of the at least one tile image is eliminated through the target elimination model to obtain at least one tile image with the moiré removed; thereafter, based on the at least one tile image with the moiré removed, a display as shown is displayed. Figure 6 Multiple images with moiré removed as shown in Figure (c).

[0096] It should be noted that, in the embodiments of the present application, only the above Figure 5 or Figure 6 The application scenario shown is used as an example for illustration and does not constitute a limitation on the embodiments of the present application.

[0097] Next, the software system of the electronic device 100 will be described.

[0098] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software system of the electronic device 100.

[0099] Figure 7 1 is a block diagram of a software system of an electronic device 100 provided in an embodiment of the present application. Figure 7 The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system layer, and the kernel layer.

[0100] The application layer can include a series of application packages. Figure 7 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0101] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 7As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make these data accessible to applications. These data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of the application, and the display interface can be composed of one or more views, for example, including a view for displaying a text message notification icon, including a view for displaying text, and including a view for displaying pictures. The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of an icon or scroll bar text, such as a notification of an application running in the background. The notification manager can also be a notification that appears on the screen in the form of a dialog window, such as a text message in the status bar, a reminder sound, an electronic device vibrating, an indicator light flashing, etc.

[0102] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0103] The system library can include multiple functional modules, such as: surface manager, media library, 3D graphics processing library (such as: OpenGL ES), 2D graphics engine (such as: SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0104] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0105] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0106] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the raw input event. For example, if the touch operation is a single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer, starts the camera application, and then calls the kernel layer to start the camera driver to capture static images or videos through the camera 193.

[0107] Based on the execution subjects and application scenarios provided in the above embodiments, the moiré elimination method provided in the embodiments of the present application is introduced below. Figure 8 , Figure 8 FIG. 1 is a flow chart of a method for eliminating moiré patterns according to an exemplary embodiment. As an example but not a limitation, the electronic device may include not only the above Figure 7 The modules shown may also include other modules, for example, a camera, a moiré removal module and a multimedia database. Figure 8 Taking the method applied in an electronic device and the interactive implementation of a camera, a gallery, a moire removal module and a multimedia database included in the electronic device as an example for explanation, the method may include some or all of the following contents:

[0108] Step 801: The camera receives a start operation triggered by a user.

[0109] When using an electronic device, a user may need to use the camera of the electronic device to collect the content displayed on the display screen of other devices, such as the above Figure 5 In the scenario shown in Figure (a), in this case, the user can trigger the start operation of the camera, so that the camera can receive the start operation triggered by the user.

[0110] For example, the user's operation of opening the camera application can be the start operation received by the camera. Alternatively, when using a social application, the user may need to use the camera of the electronic device to capture the content displayed on the display screen of other devices. In this case, the user can trigger the photo function in the display interface of the social application, and the user's triggering operation of the photo function is the start operation that can be received by the camera.

[0111] Step 802: In response to the start-up operation, the camera is started.

[0112] Step 803: The camera receives a collection operation.

[0113] After the camera is started, the user can trigger a collection operation, so that the camera can receive the collection operation.

[0114] Step 804: the camera captures the first image in response to the capture operation, and sends the first image to the gallery.

[0115] Step 805: The gallery receives the first image sent by the camera and displays the first image.

[0116] Step 806: The gallery receives a moiré removal operation triggered by the user.

[0117] Since the camera shoots the content displayed on other display screens, the captured first image is likely to have moiré patterns. If the first image has moiré patterns, the user can trigger the moiré pattern removal operation so that the gallery can receive the moiré pattern removal operation.

[0118] Step 807: In response to the moiré removal operation, the image library sends the first image to the moiré removal module.

[0119] It should be noted that the moiré removal module includes a target removal model, which is a pre-trained neural network model for removing moiré. Therefore, in order to remove the moiré in the first image, the gallery can send the first image to the moiré removal module.

[0120] As an example, the network structure of the target elimination model is an encoding-decoding structure.

[0121] It is worth noting that the target elimination model has a simple structure and occupies little memory, thus ensuring the operating efficiency of the electronic device.

[0122] It should be noted that the training process of the target elimination model can refer to the following Fig.14 The method shown in the present application will not be described in detail in the embodiments.

[0123] Step 808: The moiré removal module receives the first image.

[0124] Step 809: The moiré removal module determines the size of the first image.

[0125] Since the target removal model has certain requirements on the size of the input image, if the size of the first image does not meet the size required by the target removal model, the moiré removal module needs to perform some processing on the first image. If the size of the first image meets the size required by the target removal model, the moiré in the first image can be directly removed by the target removal model. Therefore, in order to determine the operation of the moiré removal module, the moiré removal module needs to determine the size of the first image.

[0126] Step 810: When the size of the first image is larger than the first target size, the moiré removal module determines at least one tile image of the first image.

[0127] It should be noted that the size of each tile image in at least one tile image is a first target size, and the first target size is the size of the image required to be input by the target elimination model, and the first target size can be pre-set according to requirements.

[0128] As an example, when the size of the first image is larger than the first target size, the moiré removal module can determine at least one tile image of the first image in different ways. The following two ways are used as examples for explanation.

[0129] In a possible implementation, the operation of the moiré removal module to determine at least one sliced ​​image of the first image when the size of the first image is larger than the first target size includes: when the size of the first image is larger than the first target size, determining the number of slices for slicing the first image according to the size of the first image, the first target size, and a preset overlapping size; when the number of slices is greater than 1, performing a cutting operation on the first image to obtain multiple sliced ​​images according to the first target size, the preset overlapping size, and the number of slices; when the number of slices is 1, resampling the first image to obtain a first reference image, the size of the first reference image is the first target size, and the first reference image is a sliced ​​image of the first image.

[0130] In the case where the size of the first image is larger than the first target size, the size of the first image may be slightly larger than the first target size. In this case, the size of the first image is not enough to support the first image being cut, and the first image needs to be resampled; of course, the size of the first image may also be larger and can support the first image being cut. Therefore, in order to determine the processing method for the first image, when the size of the first image is larger than the first target size, the moiré removal module may determine the number of slices for slicing the first image according to the size of the first image, the first target size, and the preset overlap size.

[0131] It should be noted that the preset overlap size can be pre-set according to demand, and the preset overlap size is related to the first target size, that is, in the case of overlap in the horizontal direction (or horizontal direction), the preset overlap size is the same as the longitudinal size of the first target size in the longitudinal direction (or vertical direction). In this case, the preset overlap size can be 1024 (the same as the longitudinal size of the first target size) * 128 (lateral overlap width) or 1024 * 256, etc.; in the case of overlap in the longitudinal direction, the preset overlap size is the same as the transverse size of the first target size in the transverse direction. In this case, the preset overlap size can be 128 (longitudinal overlap width) * 1024 (the same as the transverse size of the first target size) or 256 * 1024, etc. By way of example, in order to facilitate understanding of the setting of the preset overlap size when overlap occurs in different directions, an embodiment of the present application provides a schematic diagram of the preset overlap size, please refer to Fig. 9 .

[0132] Since the horizontal length and the vertical length of the first image are not necessarily the same, the number of times the moiré elimination module cuts the first image horizontally and the number of times it cuts it vertically are not necessarily the same, that is, the number of horizontal cuts and the number of vertical cuts of the first image are not necessarily the same. Therefore, as an example, the moiré elimination module can determine the number of horizontal cuts and the number of vertical cuts of the first image according to the size of the first image, the first target size and the preset overlapping size, respectively; and multiply the number of horizontal cuts by the number of vertical cuts to obtain the number of slices for slicing the first image.

[0133] In some embodiments, the moiré removal module determines the number of horizontal cuts of the first image by the following first formula according to the horizontal length of the first image, the horizontal length in the first target size, and the horizontal overlap length in the preset overlap size.

[0134]

[0135] It should be noted that in the above first formula (1), is the number of transverse cuts, N row0is the horizontal length of the first image (expressed by the number of horizontal pixels), overlap0 is the horizontal overlap length in the preset overlap size (expressed by the number of horizontal pixels), and patchsize0 is the horizontal length in the first target size (expressed by the number of horizontal pixels).

[0136] As an example, the moiré removal module can also determine the number of vertical cuts of the first image through the above first formula. The moiré removal module can determine the number of vertical cuts of the first image through the above first formula according to the vertical length of the first image, the vertical overlap length in the preset overlap size, and the vertical length in the first target size. When the number of vertical cuts of the first image is determined through the first formula, each parameter in the first formula is a vertically related parameter.

[0137] In some embodiments, when the first size is greater than the first target size and the first size is greater than the size threshold, the moiré removal module can determine the number of horizontal cuts of the first image according to the above-mentioned first formula, or can determine the number of horizontal cuts of the first image according to the following second formula. The embodiments of the present application do not impose specific restrictions on this.

[0138]

[0139] It should be noted that in the above second formula (2), is the number of transverse cuts, N row0 is the horizontal length of the first image, overlap0 is the horizontal overlap length in the preset overlap size, and patchsize0 is the horizontal length in the first target size.

[0140] Similarly, the moiré removal module can also determine the number of vertical cuts of the first image by using the second formula. When the number of vertical cuts of the first image is determined by using the second formula, each parameter in the second formula is a vertically related parameter.

[0141] It should be noted that the size threshold can be pre-set according to requirements. For example, the size threshold can be 3000 pixels, 4000 pixels, or 5000 pixels.

[0142] As an example, in the second formula (2), the moiré removal module performs “N row0 The operation condition of / 2" may refer to that the moiré removal module samples the horizontal size of the first image to be N row0 / 2. Of course, it can also be a simple calculation.

[0143] In some embodiments, when the number of slices is greater than 1, a cutting operation is performed on the first image according to the first target size, the preset overlapping size and the number of slices to obtain multiple sliced ​​images, including: when the number of slices is greater than 1, a second target size is determined according to the first target size, the preset overlapping size and the number of slices, the second target size is a size that does not require resampling, and the second target size is greater than the first target size; when the size of the first image is not the second target size, the first image is resampled to obtain a second reference image, the size of the second reference image is the second target size; according to the first target size, the preset overlapping size and the number of slices, a cutting operation is performed on the second reference image to obtain multiple sliced ​​images.

[0144] Since when the number of slices is greater than 1, the size of the first image may not be able to be cut into an integer number of slice images according to the first target size and the preset overlap size. In this case, the first image needs to be resampled. Therefore, in order to determine whether the first image needs to be resampled, the moiré elimination module can determine the second target size based on the first target size, the preset overlap size and the number of slices.

[0145] As an example, the moiré elimination module can determine the horizontal length of the second target size based on the horizontal length of the first target size, the horizontal overlap length of the preset overlap size, and the number of horizontal slices; and determine the vertical length of the second target size based on the vertical length of the first target size, the vertical overlap length of the preset overlap size, and the number of vertical slices.

[0146] As an example, the moiré removal module may determine the lateral length in the second target size by using the following third formula according to the lateral length of the first target size, the lateral overlap length of the preset overlap size, and the number of lateral slices.

[0147]

[0148] It should be noted that in the third formula (3), N row1 is the horizontal length of the second target size, is the number of horizontal slices, patchsize0 is the horizontal length of the first target size, and overlap0 is the horizontal overlap width of the preset overlap size.

[0149] As an example, the moiré removal module can also determine the longitudinal length of the second target size by using the third formula. When the longitudinal length of the second target size is determined by using the third formula, each parameter in the third formula is a longitudinally related parameter.

[0150] In some embodiments, when the size of the first image is not the second target size, the moiré removal module resamples the first image, including: down-sampling the first image when the size of the first image is larger than the second target size; and up-sampling the first image when the size of the first image is smaller than the second target size.

[0151] In another possible implementation, when the size of the first image is larger than the first target size, the operation of determining at least one tile image of the first image by the moiré elimination module includes: when the size of the first image is larger than the first target size, cutting the first image according to the first target size and a preset overlap size to obtain at least one third reference image, wherein the size of the at least one third reference image is no larger than the first target size; when there is a third reference image whose size is smaller than the first target size in at least one third reference image, resampling the third reference image whose size is smaller than the first target size so that the size of the resampled third reference image is the first target size; and determining both the un-resampled third reference image and the resampled third reference image as tile images of the first image to obtain at least one tile image.

[0152] When segmenting the first image, it is possible that the first image can be segmented into an integer number of slice images according to the first target size and the preset overlap size, or it is possible that after segmenting the first image according to the first target size and the preset overlap size, the first image cannot be cut into an integer number of slice images according to the first target size. In this case, at least one third reference image will have a third reference image whose size is smaller than the first target size. Therefore, when at least one third reference image has a third reference image whose size is smaller than the first target size, in order to avoid losing image information, the moiré removal module can resample the third reference image whose size is smaller than the first target size.

[0153] It should be noted that resampling the third reference image having a size smaller than the first target size refers to upsampling the third reference image having a size smaller than the first target size.

[0154] It is worth noting that by resampling the third reference image whose size is smaller than the first target size, it is ensured that the first image can maintain complete image information after being cut, thereby avoiding the loss of image information and improving the accuracy of image cutting.

[0155] As an example, when there is no third reference image with a size smaller than the first target size in at least one third reference image, that is, when the size of at least one third reference image is the first target size, at least one third reference image is determined as at least one slice image of the first image.

[0156] Step 811: The moiré removal module removes the moiré in each tile image in at least one tile image by using a target removal model to obtain at least one tile image with the moiré removed.

[0157] As an example, the moiré removal module can input each tile image in at least one tile image into the target removal model in sequence to remove the moiré in each tile image.

[0158] Since the target elimination model is pre-trained, the moiré pattern in each tile image can be accurately eliminated through the target elimination model, thereby improving the display quality of each tile image.

[0159] Step 812: The moiré removal module determines a target image for removing moiré based on at least one moiré-removed tile image.

[0160] From the above, it can be seen that the number of at least one tile image of the first image may be 1 or more. Therefore, the number of at least one tile image for eliminating moiré may also be 1 or more. Depending on the number of tile images for eliminating moiré, the moiré elimination module determines the target image for eliminating moiré in different ways.

[0161] In some embodiments, when the number of at least one moiré-eliminating tile image is 1, the moiré-eliminating tile image is determined to be the target image; when the number of at least one moiré-eliminating tile image is multiple, multiple moiré-eliminating tile images are fused to obtain the target image.

[0162] When there are multiple moiré-eliminating slice images, it means that the first image is cut before the moiré-eliminating process. Therefore, in order to obtain a complete target image, the moiré-eliminating module needs to fuse multiple moiré-eliminating slice images.

[0163] As an example, in the case where there are multiple tile images for at least one moiré-eliminating slice image, the operation of the moiré-eliminating module fusing multiple tile images for at least one moiré-eliminating slice image to obtain a target image includes: in the case where there are multiple tile images for at least one moiré-eliminating slice image, determining a cutting position of each tile image in the multiple tile images; determining a fusion matrix corresponding to each tile image according to the cutting position corresponding to each tile image, the fusion matrix being used to cause color gradient in the overlapping parts of the corresponding tile images; multiplying each tile image by the corresponding fusion matrix respectively to obtain multiple tile images with color gradients; and fusing the multiple tile images with color gradients according to the cutting position of each tile image to obtain a target image.

[0164] Since there is an overlapping part between two adjacent slice images when the first image is cut, in order to make the multiple slice images display more smoothly after fusion, the moiré removal module can determine each slice image and the corresponding fusion matrix according to the cutting position corresponding to each slice image.

[0165] It should be noted that the fusion matrix is ​​a preset matrix describing the color gradient of the binary image, that is, the fusion matrix is ​​a matrix that changes between 0-1 or 1-0, and the color gradient part in the binary image is used to indicate the overlapping part in the image. Fig.10 A schematic diagram of representing different fusion matrices through different binary images provided in the embodiment of the present application. Fig.10 Figure (a) shows a binary image A and a binary image B, wherein the pixel values ​​of the right part of the binary image A gradually change between 1-0 from left to right, and the pixel values ​​of the left part of the binary image B gradually change between 0-1 from left to right. Fig.10 (b) in the figure shows a binary image C and a binary image D. The pixel values ​​of the pixels in the lower part of the binary image C gradually change from 1 to 0 from top to bottom, and the pixel values ​​of the pixels in the upper part of the binary image D gradually change from 0 to 1 from top to bottom.

[0166] In some embodiments, the operation of the moiré removal module determining the fusion matrix corresponding to each tile image according to the cutting position corresponding to each tile image includes: determining the number and position of overlaps between the border of each tile image and the border of the first image according to the cutting position corresponding to each tile image; determining the fusion matrix corresponding to each tile image according to the number and position of overlaps corresponding to each tile image.

[0167] It should be noted that the overlap between the frame of the tile image and the frame of the first image means that the frame of the tile image and the frame of the first image partially overlap.

[0168] As an example, when the number of overlaps is 4, and the overlap positions are the left border, the right border, the upper border, and the lower border, respectively, it is determined that there is no corresponding fusion matrix.

[0169] Since the number of overlaps is 4 and the overlap positions are the left border, the right border, the upper border and the lower border, respectively, it means that the first image has not been cut. In this case, there is no need to fuse the sliced ​​images, so there is no corresponding fusion matrix. In addition, in this case, since no cutting is performed, the moiré removal module may not perform the determination operation of the fusion matrix. The embodiment of the present application does not make any specific restrictions on this.

[0170] As an example, when the number of overlaps is 3, and the overlapping positions are the left border, the upper border, and the lower border, respectively, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the right part undergo a color gradient; when the number of overlaps is 3, and the overlapping positions are the right border, the upper border, and the lower border, respectively, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the left part undergo a color gradient; when the number of overlaps is 3, and the overlapping positions are the left border, the upper border, and the right border, respectively, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the lower part undergo a color gradient; when the number of overlaps is 3, and the overlapping positions are the left border, the lower border, and the right border, respectively, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the upper part undergo a color gradient. Exemplarily, the various fusion matrices can refer to the above Fig.10 The fusion matrix indicated by each binary image shown in .

[0171] As an example, when the number of overlaps is 2 and the overlapping positions are the left border and the upper border respectively, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the right part undergo a color gradient, and the fusion matrix in which the pixel values ​​of the pixels in the lower part undergo a color gradient; when the number of overlaps is 2 and the overlapping positions are the left border and the lower border respectively, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the right part undergo a color gradient, and the fusion matrix in which the pixel values ​​of the pixels in the upper part undergo a color gradient; when the number of overlaps is 2 and the overlapping positions are the right border and the upper border respectively, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the left part undergo a color gradient, and the fusion matrix in which the pixel values ​​of the pixels in the lower part undergo a color gradient; when the number of overlaps is 2 and the overlapping positions are the right border and the lower border respectively, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the left part undergo a color gradient, and the fusion matrix in which the pixel values ​​of the pixels in the upper part undergo a color gradient.

[0172] As an example, when the number of overlaps is 1 and the overlapping positions are the left borders, the fusion matrix corresponding to the tiled image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the right part undergo a color gradient, a fusion matrix in which the pixel values ​​of the pixels in the upper part undergo a color gradient, and a fusion matrix in which the pixel values ​​of the pixels in the lower part undergo a color gradient; when the number of overlaps is 1 and the overlapping positions are the right borders, the fusion matrix corresponding to the tiled image is determined to be a fusion matrix in which the pixel values ​​of the pixels in the left part undergo a color gradient, a fusion matrix in which the pixel values ​​of the pixels in the upper part undergo a color gradient, and a fusion matrix in which the pixel values ​​of the pixels in the lower part undergo a color gradient. a fusion matrix; when the number of overlaps is 1 and the overlap positions are the upper borders, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the left part of the pixels undergo color gradient, a fusion matrix in which the pixel values ​​of the right part of the pixels undergo color gradient, and a fusion matrix in which the pixel values ​​of the lower part of the pixels undergo color gradient; when the number of overlaps is 1 and the overlap positions are the lower borders, the fusion matrix corresponding to the tile image is determined to be a fusion matrix in which the pixel values ​​of the left part of the pixels undergo color gradient, a fusion matrix in which the pixel values ​​of the right part of the pixels undergo color gradient, and a fusion matrix in which the pixel values ​​of the upper part of the pixels undergo color gradient.

[0173] As an example, when the number of overlaps is 0, the fusion matrix corresponding to the segmented image is determined to be a fusion matrix in which the pixel values ​​of the left part have a color gradient, a fusion matrix in which the pixel values ​​of the right part have a color gradient, a fusion matrix in which the pixel values ​​of the upper part have a color gradient, and a fusion matrix in which the pixel values ​​of the lower part have a color gradient.

[0174] As an example, the moiré removal module may compare the coordinate positions of the border pixels of each tile image in a preset coordinate system with the coordinate positions of the border pixels of the first image in a preset coordinate system to determine the overlapping borders and the number of overlaps corresponding to each tile image. Exemplarily, when the ordinate of any pixel in the upper border of the tile image is the same as the ordinate of any pixel in the upper border of the first image, it is determined that the upper border of the tile image overlaps the upper border of the first image.

[0175] It should be noted that the coordinate system of the sliced ​​image and the first image is the same coordinate system. The preset coordinate system can be a coordinate system established with the upper left vertex of the first image as the origin, the straight line where the upper border is located as the X-axis (horizontal coordinate), and the straight line where the left border is located as the Y-axis (vertical coordinate). It can also be a coordinate system established in other ways. The embodiments of the present application do not impose specific restrictions on this.

[0176] As an example, the moiré removal module can determine the corresponding fusion matrix based on the number and position of overlaps between the borders of each segmented image and the borders of the first image, or it can be determined in other ways. For example, the positions of the four vertices of the segmented image relative to the first image can also be determined. The determination method can also be based on the coordinates of the four vertices in a preset coordinate system and the positions of the four vertices and / or four borders of the first image in the preset coordinate system. The embodiments of the present application will not go into details one by one.

[0177] It should be noted that, since the color gradient portion indicated in the fusion matrix is ​​the overlapping portion of the image, the color gradient portion of each tile image in the plurality of color gradient tile images is the overlapping portion of each tile image.

[0178] In some embodiments, the moiré removal module fuses multiple color gradient tile images according to the cutting position of each tile image to obtain the target image, including: sorting the multiple color gradient tile images in a cutting order according to the cutting position of each tile image, and the cutting order is used to indicate the corresponding positions of the multiple color gradient tile images in the first image; fusing two adjacent color gradient tile images in the sorted multiple color gradient tile images to obtain the target image.

[0179] As an example, the moiré removal module may merge the overlapping parts of two adjacent color gradient segmented images. The merging method may be to add the overlapping parts of two adjacent color gradient segmented images. Exemplarily, the scenario in which the moiré removal module merges the overlapping parts of two adjacent color gradient segmented images may refer to Fig.10 The scenes shown in (a) and (b) in Figure 3.

[0180] In some embodiments, the moiré removal module can also fuse multiple moiré-eliminating segmented images by other means. For example, the moiré removal module can also fuse multiple moiré-eliminating segmented images by Poisson fusion, multi-band fusion, and other methods. The embodiments of the present application do not impose specific restrictions on this.

[0181] In some embodiments, when the moiré removal module cuts the first image, there may be no overlapping part between two adjacent slice images, that is, no preset overlapping size is set in the process of cutting the first image. After obtaining multiple slice images with moiré removed, the multiple slice images with moiré removed can be directly spliced. This process is not a fusion operation, which results in an obvious sense of boundary in the spliced ​​target image, such as Fig.11As shown in Figure (a) in FIG. 1 , there is an obvious sense of boundary in the spliced ​​target image, resulting in unsatisfactory image display quality. Therefore, in order to improve the image display quality, the moiré removal module can cut the first image according to the operation of step 810 above. Since the overlapping parts between two adjacent slice images can be fused, it is ensured that there is no obvious sense of boundary after the image fusion, thereby improving the image display quality. Fig.11 As shown in Figure (b).

[0182] In some embodiments, the size of the first image may be larger than the first target size. Of course, the size of the first image may also be smaller than or equal to the first target size. When the size of the first image is smaller than or equal to the first target size, the moiré elimination module may eliminate the moiré in the first image by other means.

[0183] As an example, when the size of the first image is smaller than the first target size, the first image can be resampled to obtain a fourth reference image, the size of which is the first target size; the moiré pattern of the fourth reference image is eliminated by the target elimination model to obtain the target image. When the size of the first image is equal to the first target size, the moiré pattern elimination module can directly eliminate the moiré pattern of the first image by the target elimination model to obtain the target image.

[0184] It is worth noting that when the size of the first image is smaller than the first target size, the first image is resampled so that the size of the obtained fourth reference image meets the size requirement of the target elimination model, thereby ensuring the elimination of moiré patterns for images of various sizes and improving the reliability of image moiré elimination.

[0185] In some embodiments, when the size of the first image is not the first target size, the moiré removal module may further perform filtering processing on the first image before determining at least one tile image of the first image.

[0186] Since the first image contains moiré, when the size of the first image is not the first target size, the first image needs to be cut and / or resampled. Resampling the first image may aggravate the moiré in the first image. Therefore, in order to avoid aggravating the moiré in the first image due to resampling, the moiré elimination module may further perform low-pass filtering on the first image before determining at least one tile image of the first image.

[0187] For example, in order to intuitively understand the impact of low-pass filtering on resampling, the present application embodiment uses Fig.12 A comparison diagram of how low-pass filtering can reduce the emphasis of moiré patterns is shown in Figure 2. Fig.12After acquiring the content displayed on the display screen to obtain the first image, the display frequency of the first image is significantly changed compared with the frequency of the content displayed on the display screen, and the display frequency in the first image is aliased, thereby generating moiré patterns. If resampling is performed on this basis, the frequency aliasing is significantly aggravated, and the moiré patterns in the resampled first image are aggravated. If the first image is low-pass filtered before resampling, the frequency aliasing can be significantly reduced, so that after resampling the filtered first image, the frequency aliasing is not aggravated, and the moiré patterns in the first image are not aggravated. Then, after the moiré patterns in the first image are eliminated by the target elimination model, a better de-moiré effect can be achieved.

[0188] Step 813: The moiré removal module sends the target image to the multimedia database.

[0189] Step 814: The multimedia database receives the target image.

[0190] Step 815: The multimedia database stores the target image and sends the target image to the gallery.

[0191] It should be noted that the multimedia database can store the target image and the first image at the same time, thereby avoiding the loss of image information.

[0192] Step 816: The gallery receives the target image and displays the target image.

[0193] In this embodiment of the present application, when the size of the first image is large and does not meet the size requirements of the pre-trained target elimination model, at least one tile image of the first image can be determined. Since the size of at least one tile image meets the size requirements of the target elimination model, the moiré pattern in the at least one tile image can be eliminated through the target elimination model. Since it is possible to eliminate the moiré pattern in the first image by determining at least one tile image of the first image and eliminating the moiré pattern of each tile image in the at least one tile image when the size of the first image is large, it is ensured that the moiré pattern in images of various sizes can be eliminated. In addition, since the network structure of the target elimination model is an encoding-decoding structure, this network structure is relatively simple and occupies less memory, thereby improving the efficiency of the operation of the electronic device.

[0194] Since the electronic device may include not only a camera, a gallery, a moiré removal module and a multimedia database, but also other modules, for example, the electronic device may also include a file scanning module, the electronic device may also eliminate moiré through the camera, the file scanning module and the moiré removal module. Fig.13 , Fig.13This is a flow chart of a method for removing moire patterns according to another exemplary embodiment. As an example but not a limitation, the method is applied to an electronic device, and the interactive implementation of a camera, a file scanning module, and a moire pattern removal module included in the electronic device is used as an example for explanation. The method may include some or all of the following contents:

[0195] Step 1301: The file scanning module receives a file scanning operation triggered by a user.

[0196] Step 1302: The file scanning module sends a start message to the camera in response to the file scanning operation.

[0197] Step 1303: The camera receives the startup message and starts up.

[0198] Step 1304: The camera captures a first image and sends the first image to the file scanning module.

[0199] Step 1305: The document scanning module receives the first image and displays the first image.

[0200] Step 1306: The document scanning module receives a moiré removal operation triggered by a user.

[0201] Step 1307: The document scanning module sends the first image to the moiré removal module.

[0202] The operations of step 1308 to step 1312 may refer to the operations of the above-mentioned step 808 to step 812, and the embodiments of the present application will not describe them one by one.

[0203] In this embodiment of the present application, when the size of the first image is large and does not meet the size requirements of the pre-trained target elimination model, at least one tile image of the first image can be determined. Since the size of at least one tile image meets the size requirements of the target elimination model, the moiré pattern in the at least one tile image can be eliminated through the target elimination model. Since it is possible to eliminate the moiré pattern in the first image by determining at least one tile image of the first image and eliminating the moiré pattern of each tile image in the at least one tile image when the size of the first image is large, it is ensured that the moiré pattern in images of various sizes can be eliminated. In addition, since the network structure of the target elimination model is an encoding-decoding structure, this network structure is relatively simple and occupies less memory, thereby improving the efficiency of the operation of the electronic device.

[0204] Next, the present application embodiment explains the method flow of training an electronic device to obtain a target elimination model. Fig.14 , Fig.14This is a flow chart of a model training method provided in an embodiment of the present application. As an example but not a limitation, the method is applied to an electronic device as an example for explanation. The method may include some or all of the following contents:

[0205] Step 1401: The electronic device obtains a plurality of negative sample images.

[0206] It should be noted that each of the multiple negative sample images is obtained by photographing the display content of the corresponding display screen, and each of the multiple negative sample images carries moiré patterns.

[0207] In some embodiments, the multiple negative sample images may be acquired by the electronic device through its own camera, or may be captured by other devices and then sent to the electronic device.

[0208] It should be noted that the multiple negative sample images may be multiple negative sample images of different sizes or multiple negative sample images of the same size, and when the multiple negative sample images are captured, the distance between the camera and the corresponding display screen may be the same or different, that is, the depth information of the multiple negative sample images may be the same or different, and the embodiment of the present application does not impose any specific restrictions on this. Of course, the severity of the moiré pattern generated in the captured image may be different when the distance between the camera and the content displayed on the display screen is different during shooting. For example, see Fig.15 From the human eye, the moiré situation in the image collected at a distance of 20 cm from the display screen is relatively light, and the moiré situation in the image collected at a distance of 10 cm from the display screen is relatively serious. Therefore, in general, in order to increase the diversity of training data, the depth information of the multiple negative sample images is different.

[0209] Step 1402: For each negative sample image among the multiple negative sample images, the electronic device obtains a screenshot image of the display screen display content corresponding to each negative sample image.

[0210] Since the positive sample image is required to have no moiré, and the screenshot image obtained by directly taking a screenshot of the content displayed on the display screen will not have moiré, the electronic device can obtain a screenshot image of the content displayed on the display screen corresponding to each negative sample image.

[0211] In some embodiments, the electronic device may directly determine the obtained screenshot image as a positive sample image, but in order to reduce the pressure of model training, the electronic device may also perform some processing on the screenshot image to obtain a positive sample image. The processing of the screenshot image corresponding to each negative sample image by the electronic device may refer to the following steps 1403-1404.

[0212] Step 1403: Perform a topological transformation on the screenshot image corresponding to each negative sample image to obtain a topologically transformed screenshot image corresponding to each negative sample image.

[0213] It should be noted that the data features of the topological transformation screenshot image are the same as the data features of the corresponding negative sample image.

[0214] Since the camera may not be facing the display screen when capturing the display screen content corresponding to each negative sample image, and other background patterns may exist in addition to the display screen content in the negative sample image, in order to ensure the consistency of data features between the screenshot image and the corresponding negative sample image, the electronic device can perform a topological transformation on the screenshot image corresponding to each negative sample image.

[0215] As an example, the electronic device can respectively extract the feature points of each negative sample image and the feature points of the corresponding screenshot image, and perform a topological transformation on the screenshot image corresponding to each negative sample image based on the feature points of each negative sample image and the feature points of the corresponding screenshot image to obtain a topologically transformed screenshot image corresponding to each negative sample image.

[0216] For example, see Fig.16 , one of the negative sample images A and the corresponding screenshot image A obtained by the electronic device can be as follows Fig.16 As shown, after the electronic device performs topological transformation on the screenshot image A, the data features of the screenshot image A after the topological transformation are consistent with the data features of the negative sample image A.

[0217] Step 1404: Migrate the color features of each negative sample image to the corresponding topological transformation screenshot image to obtain a positive sample image corresponding to each negative sample image.

[0218] It should be noted that the positive sample image corresponding to each negative sample image does not carry moiré patterns, multiple negative sample images correspond one-to-one to multiple positive sample images, and multiple negative sample images and multiple positive sample images constitute training data for model training.

[0219] Generally, the electronic device can directly determine the topological transformation screenshot image as a positive sample image. In this case, the electronic device can perform model training according to the operation of the following step 1405. The obtained elimination model can remove the moiré pattern in the image, but also damage the original information of the image, or make the character image unsightly. For example, see Fig.17For character images, after the moiré is eliminated by the elimination model, the color of the character in the image may be too dark, affecting the appearance of the character; for images that were originally darker in color, after the moiré is eliminated by the elimination model, the image may become darker, which in turn causes the user to be unable to see the information in the image clearly; for some images, after the moiré is eliminated by the elimination model, part of the image may be too bright or the image information may be lost, which also causes the user to be unable to see the information in the image clearly. Therefore, in order to avoid the color of any image changing in the subsequent process of eliminating the moiré in any image through the trained elimination model, the electronic device can unify the color of each negative sample image and the corresponding topological transformation screenshot image. Typically, the electronic device can transfer the color features of each negative sample image to the corresponding topological transformation screenshot image, thereby obtaining a positive sample image corresponding to each negative sample image.

[0220] In some embodiments, the operation of the electronic device transferring the color features of each negative sample image to the corresponding topological transformation screenshot image can refer to related technologies, for example, see Fig.16 The electronic device can determine the MASK area in the topological transformation screenshot image according to the color of the data features of the topological transformation screenshot image, and perform an opening operation on the MASK area to ensure that the colors of the pixels in the MASK area are all black. Then, the color migration is completed based on the MASK area and the Reinhard criterion.

[0221] As an example, the electronic device may perform color migration based on the MASK area and the Reinhard criterion by using the following fourth formula.

[0222]

[0223] It should be noted that in the fourth formula above, I 原lab is the pixel value of the MASK area in the topological transformation screenshot in the LAB (Lab color space) space, I 参考lab It is the pixel value of the MASK area in the lab space corresponding to the negative sample image, mean() means taking the mean, and var() means taking the variance.

[0224] Since the data features of the topological transformation screenshot area are the same as the data features of the corresponding negative sample image, the MASK area in the topological transformation screenshot image is the same as the MASK area of ​​the corresponding negative sample image.

[0225] Step 1405: Iteratively train the initial elimination model based on the training data to obtain a target elimination model.

[0226] It should be noted that the network structure of the initial elimination model is an encoding-decoding structure. Exemplarily, the initial elimination model may be a U-net model.

[0227] In some embodiments, the model may be provided with a loss function, and the loss function is used in the model training stage. After each batch of training data is input into the initial elimination model, the predicted value is output through forward propagation, and the difference value (also called loss value) between the predicted value and the true value can be calculated through the loss function. After obtaining the loss value, the initial elimination model updates each parameter through back propagation to reduce the loss between the true value and the predicted value, so that the generated predicted value is close to the true value, thereby achieving the purpose of training. The loss function of the target elimination model can be as follows.

[0228]

[0229] It should be noted that in the above loss function (5), I GT The data of the screenshot image for topological transformation, is the output data of the target elimination model, L is the loss value of the loss function, is Charbonnier Loss, is Perceptual Loss, is the gradient loss, is SSIM loss (image quality loss function).

[0230] In some embodiments, the loss function can be as shown above, or it can be other loss functions. For example, the loss function is a loss function composed of at least one of Charbonnier Loss, Perceptual Loss, gradient loss and SSIM loss. The embodiments of the present application do not impose specific restrictions on this.

[0231] In some embodiments, the electronic device may determine that iterative training is completed to obtain a target elimination model when the loss value of the loss function is less than a preset value, and the preset value may be set in advance according to demand.

[0232] In an embodiment of the present application, by unifying the color of each negative sample image with the corresponding positive sample image, the pressure on image brightness learning during model training is reduced, allowing the model to focus on learning the removal of moiré patterns, thereby avoiding changes in image color during the subsequent elimination of moiré patterns and reducing damage to image information.

[0233] Next, refer to Fig.18 , Fig.181 is a flow chart of a method for eliminating moiré provided in an embodiment of the present application. As an example but not a limitation, the method is described here by taking the application of the method in an electronic device as an example. The method may include some or all of the following contents:

[0234] Step 1801: Display a first image.

[0235] It should be noted that the first image is an image with moiré. The first image may be obtained by an electronic device shooting the content displayed on a display screen through a camera, or may be sent to the electronic device by other devices, and the present application embodiment does not impose specific restrictions on this. For example, see Fig.19 , the electronic device displays a first image, and the first image is an image with moiré patterns, that is, the electronic device obtains a moiré pattern image.

[0236] Step 1802: Receive a first user operation on a first image.

[0237] Since the moiré pattern in the first image will affect the display quality of the image, in order to improve the display quality of the first image, the user can perform some operations to enable the electronic device to eliminate the moiré pattern in the first image. Figure 5 or Figure 6 The application scenario shown.

[0238] Step 1803: In response to a first user operation, when a size of the first image is larger than a first target size, determining at least one tile image of the first image.

[0239] It should be noted that the size of each tile image in the at least one tile image is the first target size.

[0240] It is worth noting that, by performing the operation of removing the moiré in the first image in response to the first user operation triggered by the user, the interactivity with the user is improved.

[0241] In some embodiments, the electronic device may determine at least one tile image of the first image in response to the first user operation when the size of the first image is larger than the first target size. At least one tile image of the first image may also be determined in other cases when the size of the first image is larger than the first target size. For example, the electronic device may automatically identify whether the first image contains moiré patterns, and determine at least one tile image of the first image when it is identified that the first image contains moiré patterns and the size of the first image is larger than the first target size.

[0242] Since the electronic device uses different methods to eliminate moiré patterns in first images of different sizes, the electronic device needs to determine the size of the first image and, when the size of the first image is larger than the first target size, determine at least one slice image of the first image.

[0243] As an example, when the size of the first image is larger than the first target size, the electronic device can determine at least one slice image of the first image in different ways, which are explained below using the following two ways as examples.

[0244] In a possible implementation, when the size of the first image is larger than the first target size, the operation of determining at least one slice image of the first image includes: when the size of the first image is larger than the first target size, determining the number of slices for slicing the first image according to the size of the first image, the first target size, and a preset overlapping size; when the number of slices is larger than 1, performing a cutting operation on the first image according to the first target size, the preset overlapping size, and the number of slices to obtain a plurality of slice images; when the number of slices is 1, resampling the first image to obtain a first reference image, the size of the first reference image being the first target size, and the first reference image being a slice image of the first image.

[0245] It should be noted that, when the size of the first image is larger than the first target size, the specific method for the electronic device to determine at least one slice image of the first image can refer to the operation of the moiré elimination module to determine at least one slice image of the first image in the above step 810, and the embodiments of the present application will not elaborate on this one by one.

[0246] It is worth noting that, by determining the number of slices for slicing the first image, the processing method for the first image can be accurately selected, thereby improving the accuracy of processing the first image.

[0247] In some embodiments, when the number of slices is greater than 1, a cutting operation is performed on the first image according to the first target size, the preset overlapping size and the number of slices to obtain multiple sliced ​​images, including: when the number of slices is greater than 1, a second target size is determined according to the first target size, the preset overlapping size and the number of slices, the second target size is a size that does not require resampling, and the second target size is greater than the first target size; when the size of the first image is not the second target size, the first image is resampled to obtain a second reference image, the size of the second reference image is the second target size; according to the first target size, the preset overlapping size and the number of slices, a cutting operation is performed on the second reference image to obtain multiple sliced ​​images.

[0248] It should be noted that, when the number of slices is greater than 1, the electronic device performs a cutting operation on the first image according to the first target size, the preset overlapping size and the number of slices to obtain a specific operation of multiple sliced ​​images. The moiré elimination module in the above step 810 performs a cutting operation on the first image according to the first target size, the preset overlapping size and the number of slices to obtain a specific operation of multiple sliced ​​images, which will not be described one by one in the embodiments of the present application.

[0249] It is worth noting that, when the size of the first image is not the second target size, by resampling the first image, it is possible to ensure that the first image is completely cut, thereby avoiding the loss of image information of the first image to the greatest extent possible.

[0250] In some embodiments, when the size of the first image is not the second target size, the electronic device resamples the first image including: down-sampling the first image when the size of the first image is larger than the second target size; and up-sampling the first image when the size of the first image is smaller than the second target size.

[0251] In another possible implementation, when the size of the first image is larger than the first target size, the operation of determining at least one tile image of the first image includes: when the size of the first image is larger than the first target size, according to the first target size and a preset overlapping size, cutting the first image to obtain at least one third reference image, wherein the size of at least one third reference image is not larger than the first target size; when there is a third reference image whose size is smaller than the first target size in at least one third reference image, resampling the third reference image whose size is smaller than the first target size so that the size of the resampled third reference image is the first target size; and determining both the unresampled third reference image and the resampled third reference image as tile images of the first image to obtain at least one tile image.

[0252] It should be noted that, when the size of the first image is larger than the first target size, the operation of the electronic device to determine at least one slice image of the first image can refer to the operation of the moiré elimination module in the above step 810 to determine at least one slice image of the first image in another possible implementation method when the size of the first image is larger than the first target size, and the embodiments of the present application will not elaborate on this one by one.

[0253] It is worth noting that by directly cutting the first image according to the first target size and the preset overlap size, the computational complexity is reduced and the cutting efficiency is improved. In addition, by resampling the third reference image whose size is smaller than the first target size, it is ensured that the first image can maintain complete image information after being cut, avoiding the loss of image information and improving the accuracy of image cutting.

[0254] Step 1804: removing the moiré pattern in each of the at least one tile image by using the target removal model, and obtaining at least one tile image with the moiré pattern removed.

[0255] It should be noted that the target elimination model is a pre-trained neural network model for eliminating moiré patterns, and the network structure of the target elimination model is an encoding-decoding structure.

[0256] In some embodiments, the size of the first image may be larger than the first target size. Of course, the size of the first image may also be smaller than or equal to the first target size. When the size of the first image is smaller than or equal to the first target size, the electronic device may eliminate the moiré pattern in the first image by other means.

[0257] As an example, when the size of the first image is smaller than the first target size, the first image can be resampled to obtain a fourth reference image, the size of which is the first target size; the moiré pattern of the fourth reference image is eliminated by the target elimination model to obtain the target image. When the size of the first image is equal to the first target size, the electronic device can directly eliminate the moiré pattern of the first image by the target elimination model to obtain the target image.

[0258] It is worth noting that when the size of the first image is smaller than the first target size, the first image is resampled so that the size of the obtained fourth reference image meets the size requirement of the target elimination model, thereby ensuring the elimination of moiré patterns for images of various sizes and improving the reliability of image moiré elimination.

[0259] In some embodiments, when the size of the first image is not the first target size, the moiré removal module may further perform filtering processing on the first image before determining at least one tile image of the first image.

[0260] Since the first image contains moiré, when the size of the first image is not the first target size, the first image needs to be cut and / or resampled, and resampling the first image may aggravate the moiré in the first image. Therefore, in order to avoid aggravating the moiré in the first image due to resampling, the electronic device may also perform low-pass filtering on the first image before determining at least one tile image of the first image. That is, see Fig.19 After the electronic device obtains the moiré image (that is, the first image), since the moiré image may need to be resampled, the electronic device may also perform low-pass filtering on the moiré image, and then perform image segmentation and / or image resampling. Fig.19 Image cutting is used as an example to illustrate.

[0261] It is worth noting that by performing low-pass filtering on the first image, the frequency aliasing in the first image is avoided from being aggravated when the first image is subsequently resampled, and after the moiré in the first image is eliminated by the target elimination model, a better de-moiré effect can be achieved.

[0262] In some embodiments, before eliminating the moiré carried by each tile image in at least one tile image through a target elimination model to obtain at least one tile image with the moiré eliminated, the electronic device can also obtain training data, the training data including multiple negative sample images and multiple positive sample images, each of the multiple negative sample images carries moiré, each of the multiple positive sample images does not carry moiré, and the multiple negative sample images correspond one-to-one to the multiple positive sample images; the initial elimination model is iteratively trained based on the training data to obtain the target elimination model, and the network structure of the initial elimination model is an encoding-decoding structure.

[0263] For example, the process of training the initial elimination model can refer to Fig. 20 As shown in the schematic diagram of the training process, after obtaining the training data, the training data can be input into the initial elimination model in pairs. During the training process, under the constraints of multiple positive sample images and the loss function, a moiré-free image corresponding to each negative sample image can be output.

[0264] It is worth noting that since the target elimination model has a simple structure with a coding-decoding network structure, the target elimination model occupies less memory, can be deployed in electronic devices, and ensures the operating efficiency of the electronic devices.

[0265] In some embodiments, the target elimination model can be deployed in an electronic device or in the cloud, and the embodiments of the present application do not impose specific limitations on this.

[0266] In some embodiments, the operation of an electronic device acquiring training data includes: acquiring multiple negative sample images, each of the multiple negative sample images is obtained by photographing the corresponding display screen display content; for each negative sample image in the multiple negative sample images, acquiring a screenshot image of the display screen display content corresponding to each negative sample image; performing a topological transformation on the screenshot image corresponding to each negative sample image to obtain a topologically transformed screenshot image corresponding to each negative sample image, the data features of the topologically transformed screenshot image being the same as the data features of the corresponding negative sample image; and migrating the color features of each negative sample image to the corresponding topologically transformed screenshot image to obtain a positive sample image corresponding to each negative sample image.

[0267] It should be noted that the specific operations of the electronic device to obtain training data can refer to the operations of the above steps 1401 to 1404, and the electronic device in this embodiment of the application will not be described one by one.

[0268] It is worth noting that by unifying the color of each negative sample image with the corresponding positive sample image, the pressure on image brightness learning during model training is reduced, allowing the model to focus on learning the removal of moiré patterns, thereby avoiding changes in image color during the subsequent moiré elimination process and reducing damage to image information.

[0269] Step 1805: Determine a target image from which moiré is to be eliminated based on at least one moiré-eliminated tile image.

[0270] In some embodiments, the electronic device determines the target image with moiré removed based on at least one moiré-removed tile image, including: when the number of at least one moiré-removed tile image is 1, determining the moiré-removed tile image as the target image; when the number of at least one moiré-removed tile image is multiple, fusing multiple moiré-removed tile images to obtain the target image. For example, see Fig.19 After the electronic device performs a fusion operation on the segmented images (at least one segmented image with moiré eliminated is fused), a moiré-free image (ie, a target image) can be obtained.

[0271] It is worth noting that, when the number of at least one moiré-eliminating tile image is different, different methods are used to determine the target image, thereby improving the reliability of determining the target image.

[0272] In some embodiments, when there are multiple moiré-eliminating tile images, the electronic device fuses multiple moiré-eliminating tile images to obtain a target image, including: when there are multiple moiré-eliminating tile images, determining a cutting position of each tile image in the multiple tile images; determining a fusion matrix corresponding to each tile image according to the cutting position corresponding to each tile image, the fusion matrix being used to cause color gradient in overlapping parts of the corresponding tile images; multiplying each tile image by the corresponding fusion matrix respectively to obtain multiple color gradient tile images; and fusing the multiple color gradient tile images according to the cutting position of each tile image to obtain a target image.

[0273] It should be noted that, in the case where there are multiple moiré-eliminating segmented images, the specific operation of the electronic device fusing the multiple moiré-eliminating segmented images to obtain the target image can refer to the operation of the moiré elimination module fusing the multiple moiré-eliminating segmented images to obtain the target image in the above step 812, and the embodiments of the present application will not elaborate on this one by one.

[0274] It is worth noting that by fusing at least one moiré-eliminating tile image through a fusion matrix, an obvious sense of boundary in the fused image is avoided, thereby improving the image display quality.

[0275] In an embodiment of the present application, when the size of the first image is large and does not meet the size requirements of the pre-trained target elimination model, at least one tile image of the first image can be determined. Since the size of at least one tile image meets the size requirements of the target elimination model, the moiré pattern in the at least one tile image can be eliminated through the target elimination model. Since it is possible to eliminate the moiré pattern in the first image by determining at least one tile image of the first image and eliminating the moiré pattern of each tile image in the at least one tile image when the size of the first image is large, it is ensured that the moiré pattern in images of various sizes can be eliminated. In addition, since the network structure of the target elimination model is an encoding-decoding structure, this network structure is relatively simple and occupies less memory, thereby improving the efficiency of the operation of the electronic device.

[0276] Next, the electronic device involved in the embodiment of the present application is described.

[0277] Fig.21 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.21The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0278] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0279] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0280] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0281] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0282] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0283] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple sets of I2C interfaces. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C interface, thereby realizing the touch function of the electronic device 100.

[0284] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple groups of I2S interfaces. The processor 110 can be coupled to the audio module 170 via the I2S interface to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0285] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls through a Bluetooth headset.

[0286] The UART interface is a universal serial data bus for asynchronous communication. The UART interface can be a bidirectional communication bus. The UART interface can convert the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example: the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0287] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0288] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0289] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The USB interface 130 can also be used to connect other electronic devices, such as AR devices, etc.

[0290] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0291] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device 100 through the power management module 141.

[0292] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193 and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0293] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0294] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0295] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is an integer greater than 1.

[0296] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0297] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0298] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is an integer greater than 1.

[0299] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0300] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0301] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0302] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0303] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the electronic device 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0304] The electronic device 100 can implement audio functions, such as music playback, recording, etc., through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D and the application processor.

[0305] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than a pressure threshold acts on a short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

[0306] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0307] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E can also be used to identify the posture of the electronic device 100, and is applied to applications such as horizontal and vertical screen switching, pedometers, etc.

[0308] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, in a shooting scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0309] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0310] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0311] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0312] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. A method for eliminating moiré, characterized in that: Applied in an electronic device, the method comprises: In a case where the size of the first image is larger than the first target size, at least one tile image of the first image is determined, the first image is an image with moiré patterns, and the first image is obtained by the electronic device by photographing content displayed on a display screen through a camera, or the first image is sent to the electronic device by other devices, the size of each tile image in the at least one tile image is the first target size, the first target size is the size of an image required to be input by a target elimination model, the target elimination model is a pre-trained neural network model for eliminating moiré patterns, and the network structure of the target elimination model is an encoding-decoding structure; Eliminating the moiré pattern in each of the at least one tile image by using the target elimination model, thereby obtaining at least one tile image from which the moiré pattern is eliminated; When the number of the at least one moiré-eliminating tile image is 1, determining the moiré-eliminating tile image as a moiré-eliminating target image; In the case that the number of the at least one tile image for eliminating moiré is multiple, the cutting position of each tile image in the multiple tile images is determined; according to the cutting position corresponding to each tile image, the fusion matrix corresponding to each tile image is determined, the fusion matrix is ​​used to make the overlapping parts of the corresponding tile images have color gradient, and the fusion matrix is ​​a preset matrix describing the color gradient of the binary image; each tile image is multiplied by the corresponding fusion matrix respectively to obtain multiple tile images with color gradient; according to the cutting position of each tile image, the multiple tile images with color gradient are fused to obtain the target image; When the size of the first image is smaller than the first target size, resampling the first image to obtain a fourth reference image, where the size of the fourth reference image is the first target size; Eliminating the moiré pattern of the fourth reference image by using the target elimination model to obtain the target image; Wherein, when the size of the first image is larger than the first target size, determining at least one tile image of the first image includes: When the size of the first image is larger than the first target size, determining the number of slices for slicing the first image according to the size of the first image, the first target size, and a preset overlap size; When the number of slices is greater than 1, a cutting operation is performed on the first image according to the first target size, the preset overlapping size and the number of slices to obtain a plurality of slice images; When the number of slices is 1, the first image is resampled to obtain a first reference image, the size of the first reference image is the first target size, and the first reference image is a sliced ​​image of the first image.

2. The method according to claim 1, characterized in that When the number of slices is greater than 1, the first image is cut according to the first target size, the preset overlap size and the number of slices to obtain a plurality of slice images, including: When the number of slices is greater than 1, determining a second target size according to the first target size, the preset overlap size, and the number of slices, wherein the second target size is a size that does not require resampling and is greater than the first target size; When the size of the first image is not the second target size, resampling the first image to obtain a second reference image, wherein the size of the second reference image is the second target size; According to the first target size, the preset overlapping size and the number of slices, a cutting operation is performed on the second reference image to obtain the multiple slice images.

3. The method according to claim 1, characterized in that The step of determining the fusion matrix corresponding to each of the slice images according to the cutting position corresponding to each of the slice images includes: Determining, according to the cutting position corresponding to each of the slice images, the number and position of overlaps between the frame of each of the slice images and the frame of the first image; According to the overlap quantity and overlap position corresponding to each tile image, a fusion matrix corresponding to each tile image is determined.

4. The method according to claim 1, characterized in that The step of fusing the plurality of color gradient segment images according to the cutting position of each segment image to obtain the target image comprises: According to the cutting position of each of the tile images, the plurality of tile images with color gradients are sorted in a cutting order, wherein the cutting order is used to indicate corresponding positions of the plurality of tile images with color gradients in the first image; Two adjacent color gradient segmented images among the sorted plurality of color gradient segmented images are merged to obtain the target image.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: When the size of the first image is not the first target size, before determining at least one tile image of the first image, filtering processing is performed on the first image.

6. The method according to claim 1, characterized in that Before eliminating the moiré patterns in each of the at least one tile image by using the target elimination model to obtain at least one tile image with the moiré patterns eliminated, the method further includes: Acquire training data, the training data comprising a plurality of negative sample images and a plurality of positive sample images, each of the plurality of negative sample images carries moiré patterns, each of the plurality of positive sample images does not carry moiré patterns, and the plurality of negative sample images correspond one-to-one to the plurality of positive sample images; The initial elimination model is iteratively trained based on the training data to obtain the target elimination model, and the network structure of the initial elimination model is the encoding-decoding structure.

7. The method according to claim 6, characterized in that The obtaining of training data comprises: Acquire the multiple negative sample images, where each negative sample image is obtained by photographing the display content of the corresponding display screen; For each negative sample image among the multiple negative sample images, obtaining a screenshot image of display content of the display screen corresponding to each negative sample image; Performing a topological transformation on the screenshot image corresponding to each negative sample image to obtain a topologically transformed screenshot image corresponding to each negative sample image, wherein data features of the topologically transformed screenshot image are the same as data features of the corresponding negative sample image; The color features of each negative sample image are transferred to the corresponding topological transformation screenshot image to obtain the positive sample image corresponding to each negative sample image.

8. The method according to any one of claims 1 to 4, 6 to 7, characterized in that: When the size of the first image is larger than the first target size, determining at least one tile image of the first image includes: displaying the first image; receiving a first user operation on the first image; In response to the first user operation, when a size of the first image is larger than the first target size, at least one tile image of the first image is determined.

9. An electronic device, characterized in that: The structure of the electronic device includes a processor and a memory; The memory is used to store a program that supports the electronic device to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 8.

Images