Screen line removing model training method and device, equipment, storage medium and product

Abstract

The embodiment of the invention discloses a method, a device and equipment for training a screenline removing model, a storage medium and a product, and belongs to the technical field of artificial intelligence. Comprising the steps that a first image and a second image displayed on a display screen are shot respectively to obtain a third image and a fourth image, the first image is a positioning plate image without screen stripes, the second image is an image obtained by superposing a first target image without screen stripes in the first image, and the third image and the fourth image are images with screen stripes; determining a target mapping matrix based on the first image, the second image, the third image and the fourth image; performing screenshot in the third image and the fourth image based on the target mapping matrix to obtain a fifth image and a sixth image; performing screenshot on the fifth image based on the fifth image and the sixth image to obtain a first sample image, wherein the first sample image is a first target image with screen lines; and performing model training based on the first target image and the first sample image to obtain a screen line removing model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device, storage medium, and product for removing screen patterns. Background Technology

[0002] Images, as a crucial visual foundation for human cognition, have become a key means of information recording, expression, storage, and dissemination. With the widespread use of optical lens devices such as digital cameras, tablets, and smartphones, people can easily capture various meaningful moments. However, in daily life, when using optical lens devices to photograph electronic screens or high-frequency repetitive textures, moiré patterns and other screen artifacts appear in the captured images, severely affecting image quality and aesthetics. Therefore, it is necessary to perform screen artifact removal processing on images using screen artifact removal models to improve image quality and aesthetics. Summary of the Invention

[0003] This application provides a training method, apparatus, device, storage medium, and product for removing screen artifacts from a model. The technical solution is as follows:

[0004] On one hand, embodiments of this application provide a training method for a screen pattern removal model, the method comprising:

[0005] The first image and the second image displayed on the screen are captured respectively to obtain the third image and the fourth image. The first image is a positioning plate image without screen patterns. The second image is an image obtained by superimposing the first target image without screen patterns on the first image. The third image and the fourth image are images with screen patterns.

[0006] Based on the first image, the second image, the third image, and the fourth image, a target mapping matrix is ​​determined, which is used to represent the mapping relationship between the pixels of the first image and the pixels of the third image;

[0007] Based on the target mapping matrix, a fifth image and a sixth image are obtained by taking screenshots from the third image and the fourth image, respectively. The fifth image and the sixth image are actual screenshot images of the first image and the second image, respectively.

[0008] Based on the fifth image and the sixth image, a first sample image is obtained by taking a screenshot from the fifth image. The first sample image is a first target image with screen patterns.

[0009] The screen pattern removal model is obtained by training the model based on the first target image and the first sample image.

[0010] On the other hand, embodiments of this application provide a training device for a screen pattern removal model, the device comprising:

[0011] The shooting module is used to capture a first image and a second image displayed on the display screen respectively to obtain a third image and a fourth image. The first image is a positioning plate image without screen patterns, and the second image is an image obtained by superimposing the first target image without screen patterns on the first image. The third image and the fourth image are images with screen patterns.

[0012] The determining module is used to determine a target mapping matrix based on the first image, the second image, the third image, and the fourth image, wherein the target mapping matrix is ​​used to represent the mapping relationship between the pixels of the first image and the pixels of the third image;

[0013] The first screenshot module is used to capture the fifth image and the sixth image respectively from the third image and the fourth image based on the target mapping matrix. The fifth image and the sixth image are real-shot screen images of the first image and the second image, respectively.

[0014] The second screenshot module is used to take a screenshot from the fifth image based on the fifth image and the sixth image to obtain a first sample image, wherein the first sample image is a first target image with screen patterns;

[0015] The training module is used to train the model based on the first target image and the first sample image to obtain the screen pattern removal model.

[0016] On the other hand, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores at least one computer instruction, which is loaded and executed by the processor to implement the training method for the screen pattern removal model as described above.

[0017] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one computer instruction, which is executed by a processor to implement the training method for the screen pattern removal model as described above.

[0018] On the other hand, embodiments of this application provide a computer program product, the computer program product including computer instructions stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the training method for the screen pattern removal model as described above.

[0019] In this embodiment, the second image is an image obtained by superimposing a first target image without screen patterns onto a positioning plate image. By capturing the second image displayed on the screen, a third image with screen patterns can be obtained. With the help of the positioning plate image, a sample image with screen patterns can be extracted from the third image with screen patterns, thereby automatically generating a sample image with screen patterns. Compared with a synthesized sample image with screen patterns, the generated sample image with screen patterns is more realistic, which improves the quality of the generated sample image and thus improves the inference ability of the screen pattern removal model trained based on the sample image. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the implementation environment of the training method for the screen pattern removal model provided in an exemplary embodiment of this application;

[0021] Figure 2 This is a flowchart of a training method for a screen-texture removal model provided in an exemplary embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating a training method for a screen pattern removal model, as shown in an exemplary embodiment of this application;

[0023] Figure 4 This is a schematic diagram of a first image illustrating an exemplary embodiment of this application;

[0024] Figure 5 This is a schematic diagram illustrating a training method for a screen pattern removal model, as shown in an exemplary embodiment of this application;

[0025] Figure 6 This is a flowchart illustrating a training method for a screen pattern removal model, as shown in an exemplary embodiment of this application.

[0026] Figure 7 This is a schematic diagram illustrating a training method for a screen pattern removal model, as shown in an exemplary embodiment of this application;

[0027] Figure 8 This is a flowchart illustrating a training method for a screen pattern removal model, as shown in an exemplary embodiment of this application.

[0028] Figure 9 This is a schematic diagram illustrating a training method for a screen pattern removal model, as shown in an exemplary embodiment of this application;

[0029] Figure 10 This invention provides a structural block diagram of a training apparatus for a desmearing model, as illustrated in another exemplary embodiment of this application.

[0030] Figure 11 A structural block diagram of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0032] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0033] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the first target image, the first image, and the second image involved in this application were obtained with full authorization.

[0034] The method for training a destigmatism model provided in this application can be executed by a computer device with computing, processing, and storage functions. This computer device can be a smartphone, tablet, smart wearable device, personal computer, etc. In the following embodiments, the computer device is used as the execution subject of the destigmatism model training method for example. Please refer to... Figure 1 This illustration shows a schematic diagram of the implementation environment for a training method of a screen-removal model provided in an exemplary embodiment of this application. The implementation environment includes a computer device that trains the screen-removal model. In some embodiments, the computer device includes three core modules: a fully automatic data acquisition and labeling module, a deep learning model (Transformer-Boosted Encoder Decoder, T-BED), and a frequency domain-assisted supervision module. The fully automatic data acquisition and labeling module is used to automatically acquire a first target image without screen patterns, then automatically generate a first sample image with screen patterns based on the first target image, and then label the first sample image; wherein, the screen pattern can be any type of screen pattern; for example, the screen pattern can be a moiré pattern. In this embodiment of the application, the fully automatic data acquisition and labeling module automatically acquires and labels the first sample image with screen patterns, thereby enabling agile and automated iteration and management.

[0035] The T-BED model includes a Transformer, which generates multiple prediction images based on a first sample image. These prediction images are the first sample image after destigmatization, and each prediction image has a different size. In this embodiment, the T-BED model leverages the powerful knowledge compression and mapping learning capabilities of the Transformer, enabling the trained destigmatization model to be easily scaled with the data size, allowing for the development and optimization of different parameter models.

[0036] The frequency domain-assisted supervision module is used for supervised training based on multiple predicted images to obtain a screen removal model. In this embodiment, the frequency domain-assisted supervision module, based on prior knowledge of screen pattern characteristics, can fully and reasonably set training objectives, achieving decoupling of screen removal while preserving the original image distortion effect. Since users compare the effects before and after screen removal when using the screen removal model, if distortion processing is applied during screen removal, the difference between the before and after images becomes significant, potentially misleading users. Therefore, decoupling screen removal and distortion processing makes the trained screen removal model more focused, efficient, and stable.

[0037] Please refer to Figure 2 This document illustrates a flowchart of a method for training a screen artifact removal model according to an exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example for illustration, and the method may include the following steps:

[0038] Step 201: The computer device captures the first image and the second image displayed on the screen respectively to obtain the third image and the fourth image. The first image is a positioning plate image without screen patterns. The second image is an image obtained by superimposing the first target image without screen patterns on the first image. The third image and the fourth image are images with screen patterns.

[0039] The first target image is a real-world image without screen patterns. In this embodiment, a first sample image with screen patterns is generated based on the first target image, and then a screen pattern removal model is trained based on the first sample image. Since training the screen pattern removal model requires multiple first sample images, multiple first target images are needed in this step. For example, please refer to... Figure 3 The computer device acquires an image set, which includes multiple first target images. For any one of the first target images, steps 201 to 205 are executed. Furthermore, while capturing the first and second images, the computer device continuously captures multiple third and fourth images, and the most recently captured third and fourth images are grouped together for subsequent processing. That is, multiple groups of images are acquired in this step, and each group of images includes a third and a fourth image.

[0040] In some embodiments, the computer device determines the requirement information for a screen texture removal model, which represents the requirements for the screen texture removal model; based on the requirement information, it determines a sampling scenario, and based on the sampling scenario, it determines a first target image and a display screen. The sampling scenario represents the type of display screen and the image content of the first target image; for example, the type of display screen can be a mobile phone screen, a computer screen, or a television screen. The image content can be text, pictures, or a combination of text and images. For example, please refer to [link / reference]. Figure 3 The computer equipment analyzes the demand information to determine the data collection scenario, prepares the data collection scenario, and then performs image acquisition operations.

[0041] In some embodiments, before this step, it is necessary to generate a first image and a second image, and then display the first image and the second image on the screen; correspondingly, before this step, the following steps (1) to (3) are also included:

[0042] (1) The computer device generates a first image based on the image size of the first target image and the screen size of the display screen. The first image includes multiple rows and columns of marker detection points.

[0043] For example, the screen size is 16:9, the image size of the first image is larger than the image size of the first target image but smaller than the screen size of the display. The first image can be ChArUcoBoard (a fusion image of the chessboard image and the marker detection image), ChessBoard (a chessboard image), or ArUco (a marker detection image); the advantages of ArUco are anti-occlusion, fast detection, and easy counting; the disadvantage is that even after applying subpixel thinning, the accuracy of its corner positions is not very high; conversely, ChessBoard can obtain more precise and refined corner points, but finding chessboard patterns is not as universal as finding ArUco chessboards (it must be fully visible and occlusion is not allowed). Therefore, in this embodiment, ChArUcoBoard obtained based on ChessBoard and ArUco is used as the positioning board image, which is not only easy to count but also has high corner point accuracy. Accordingly, the step of generating the first image by the computer device based on the image size of the first target image and the screen size of the display can be implemented by the following steps (1-1) and (1-2), including:

[0044] (1-1) The computer device determines the chessboard image and the marker detection image based on the image size of the first target image and the screen size of the display.

[0045] The chessboard image includes alternating black and white first and second cells, and the marker detection image includes multiple third cells displaying marker detection points (or marker detection patterns). The marker detection points (or marker detection patterns) displayed in each third cell may be the same or different. In this embodiment, the example of different marker detection points (or marker detection patterns) displayed in each third cell is used for illustration. The first, second, and third cells can all be the same size. The image size of the chessboard image and the image size of the marker detection image are larger than the image size of the first target image and smaller than the screen size of the display screen, and the image size of the chessboard image and the image size of the marker detection image are the same.

[0046] (1-2) The computer device merges the third cell in the marker detection image into the second cell in the chessboard image to obtain the first image.

[0047] The process iterates through the second cells of the chessboard image. For each second cell, the computer determines a third cell with the same coordinates from the marker detection image. The marker detection points (or marker detection patterns) in the third cell are either directly merged into the second cell, or compressed and then merged into the second cell. After traversing all the second cells in the chessboard image, the first image is generated. This process can be referenced [reference needed]. Figure 4 .exist Figure 4 The image also shows the normal, unoccluded result of the first image and the result after occlusion.

[0048] (2) The computer device overlays the first target image onto the first image to obtain the second image, and at least one row and one column of marker detection points are reserved around the first target image in the second image.

[0049] The computer device overlays the first target image onto the center of the first image to obtain a second image. In the second image, at least one row and one column of marker detection points are retained around the perimeter of the first target image; for example, at least one row of marker detection points is retained at the top and bottom of the first target image, and at least one column of marker detection points is retained on the left and right sides of the first target image, to ensure sufficient key points for subsequent consistency verification. For example, please refer to... Figure 5 The computer device generates a first image (ChArUcoBoard) and superimposes the first target image (Target, T) onto the center of the first image to obtain a second image (Target_with_Board). At least one row and one column of marker detection points are reserved around the first target image.

[0050] (3) The computer equipment displays the first image and the second image on the screen.

[0051] The computer device displays the first and second images in full screen on the display screen. Before capturing the first and second images, the computer device fixes the camera parameters and position (using a tripod or gimbal), and then captures the images to obtain the first image (Moire_1st_Photo, M1P) and the second image (Moire_2nd_Photo, M2P).

[0052] Step 202: The computer device determines a target mapping matrix based on the first image, the second image, the third image, and the fourth image. The target mapping matrix is ​​used to represent the mapping relationship between the pixels of the first image and the pixels of the third image.

[0053] The target mapping matrix can also represent the mapping relationship between the pixels of the second image and the pixels of the third image. Accordingly, this step can be implemented by the following steps (1) to (3), including:

[0054] (1) The computer device determines a first mapping matrix based on the first image and the third image. The first mapping matrix is ​​used to represent the mapping relationship between the pixels of the first image and the pixels of the third image.

[0055] (2) The computer device determines a second mapping matrix based on the second image and the fourth image. The second mapping matrix is ​​used to represent the mapping relationship between the pixels of the second image and the pixels of the fourth image.

[0056] Since multiple sets of images were captured in step 201, each set of images includes a third image and a fourth image; therefore, the computer device traverses multiple sets of images. For each set of images traversed, the computer device determines a first mapping matrix based on the first image and the third image included in that set of images, and determines a second mapping matrix based on the second image and the fourth image included in that set of images.

[0057] (3) The computer device determines the target mapping matrix from the first mapping matrix and the second mapping matrix.

[0058] The computer device determines the first reprojection error of the first mapping matrix and the second reprojection error of the second mapping matrix; based on the first and second reprojection errors, it determines the target mapping matrix from the first and second mapping matrices, and the target mapping matrix is ​​the mapping matrix with the smallest reprojection error among the first and second mapping matrices.

[0059] Since multiple sets of images were captured in step 201, each set of images includes a third image and a fourth image; therefore, the computer device traverses multiple sets of images, and for each set of images traversed, a first mapping matrix and a second mapping matrix can be generated according to steps (1) and (2); it is determined whether the first reprojection error of the first mapping matrix and the second reprojection error of the second mapping matrix are greater than a preset error; if both the first reprojection error of the first mapping matrix and the second reprojection error of the second mapping matrix are greater than the preset error, the set of images is discarded, and the next set of images is traversed, and then steps (1)-(3) are executed based on the next set of images obtained from the traversal; if the first reprojection error of the first mapping matrix is ​​greater than the preset error and the second reprojection error of the second mapping matrix is ​​not greater than the preset error, the target mapping matrix is ​​determined to be the second mapping matrix; if the first reprojection error of the first mapping matrix is ​​not greater than the preset error and the second reprojection error of the second mapping matrix is ​​greater than the preset error, the target mapping matrix is ​​determined to be the mapping matrix with the smallest reprojection error among the first mapping matrix and the second mapping matrix.

[0060] In some embodiments, if the first reprojection error of the first mapping matrix is ​​greater than a preset error and the second reprojection error of the second mapping matrix is ​​not greater than a preset error, the first mapping matrix is ​​determined as a candidate mapping matrix; if the first reprojection error of the first mapping matrix is ​​not greater than a preset error and the second reprojection error of the second mapping matrix is ​​greater than a preset error, the first mapping matrix is ​​determined as a candidate mapping matrix; if the first reprojection error of the first mapping matrix is ​​not greater than a preset error and the second reprojection error of the second mapping matrix is ​​not greater than a preset error, the mapping matrix with the smallest reprojection error among the first and second mapping matrices is determined as the candidate mapping matrix. Since multiple sets of images are captured in step 201, at least one candidate mapping matrix can be obtained here; the computer device can determine the mapping matrix with the smallest reprojection error among multiple candidate mapping matrices as the target mapping matrix.

[0061] In other embodiments, the computer device may also determine a first target mapping matrix with the smallest first projection error from a plurality of first mapping matrices, and a second target mapping matrix with the smallest second projection error from a plurality of second mapping matrices.

[0062] Step 203: Based on the target mapping matrix, the computer device takes screenshots from the third and fourth images respectively to obtain the fifth and sixth images. The fifth and sixth images are the actual screen images of the first and second images, respectively.

[0063] A real-world screenshot image refers to an image of the same size as the captured image, with no extra pixels around its edges. Since the target mapping matrix represents the mapping relationship between the pixels of the first image and the pixels of the third image (i.e., multiplying the pixels of the first image by the target mapping matrix yields the pixels of the third image), in this step, the computer device determines the ratio of the pixels of the third image to the target mapping matrix to obtain the fifth image. Similarly, the target mapping matrix represents the mapping relationship between the pixels of the second image and the pixels of the fourth image (i.e., multiplying the pixels of the second image by the target mapping matrix yields the pixels of the fourth image), in this step, the computer device determines the ratio of the pixels of the fourth image to the target mapping matrix to obtain the sixth image. For example, please refer to [link to example]. Figure 5 The computer device uses the target mapping matrix T* to align the screen area and crop the real-shot image to obtain the fifth image (Moire_1st_Screen, M1S) and the sixth image (Moire_2nd_Screen, M2S).

[0064] In other embodiments, the computer device may further determine a first target mapping matrix with the smallest first projection error from a plurality of first mapping matrices, and a second target mapping matrix with the smallest second projection error from a plurality of second mapping matrices. Accordingly, in this step, the computer device captures a fifth image from a third image based on the first target mapping matrix; and captures a sixth image from a fourth image based on the second target mapping matrix. The fifth and sixth images are actual screenshots of the first and second images, respectively.

[0065] Step 204: The computer device takes a screenshot from the fifth image based on the fifth image and the sixth image to obtain the first sample image, which is the first target image with screen patterns.

[0066] The computer device performs a difference operation on the fifth and sixth images to obtain the seventh image; the seventh image is binarized to obtain the eighth image, which is a mask image of the first target image; based on the eighth image, the first target image is extracted from the fifth image to obtain the first sample image.

[0067] In some embodiments, after the computer device determines the fifth and sixth images, it can directly perform a difference operation on the fifth and sixth images to obtain the seventh image, thereby improving computational efficiency. In other embodiments, after the computer device determines the fifth and sixth images, it first performs a consistency check on the fifth and sixth images; only if the consistency check passes, does it perform a difference operation on the fifth and sixth images. Accordingly, the step of the computer device performing a difference operation on the fifth and sixth images to obtain the seventh image can be implemented by the following steps (1) to (3), including:

[0068] (1) The computer equipment determines multiple sets of marker detection points based on the fifth and sixth images.

[0069] A set of marker detection points includes marker detection points with the same coordinates in the fifth and sixth images; the marker detection points can be ArUco marker points.

[0070] (2) The computer equipment performs consistency verification on the fifth and sixth images based on multiple sets of marker detection points.

[0071] For each set of marker detection points, the computer device determines the distance between the two marker detection points included in each set of marker detection points; since there are multiple sets of marker detection points, multiple distances will be obtained in this step; the computer device determines the maximum distance from the multiple distances; determines whether the maximum distance is greater than the preset distance; if the maximum distance is greater than the preset distance, it is determined that the consistency check of the fifth image and the sixth image fails, and the fifth image and the sixth image are discarded; if the maximum distance is not greater than the preset distance, it is determined that the consistency check of the fifth image and the sixth image passes, and step (3) is executed.

[0072] (3) If the consistency check passes, the computer device performs a difference operation on the fifth and sixth images to obtain the seventh image.

[0073] For example, please continue to refer to Figure 5 To verify the consistency between M1S and M2S, the computer equipment performs ArUco marker detection and registration on M1S and M2S respectively, and determines whether the maximum distance between these marker pairs is greater than a preset distance. If the maximum distance is not greater than the preset distance, the computer equipment performs difference operations and binarization on M1S and M2S to obtain a seventh image (Mask). Based on the Mask, M1S can be cropped to obtain the final first sample image (M). In this embodiment, the generation of the first sample image with screen patterns is completed by an automated script; therefore, it is possible to automatically generate the first sample image with screen patterns, which not only saves labor costs but also improves the efficiency of generating the first sample image.

[0074] In some embodiments, after the computer device determines the first sample image, it can directly execute step 205; alternatively, it can manually assist in verifying the quality and consistency of the first target image and the first sample image. After the quality and consistency verification of the first target image and the first sample image passes, step 205 is executed. The step of manually assisting in verifying the quality and consistency of the first target image and the first sample image can be as follows: the computer device displays a first feedback interface, which is used for the user to input or select feedback on the quality and consistency of the first target image and the first sample image; the computer device obtains the quality and consistency feedback of the first target image and the first sample image; if the image quality and consistency feedback indicates that the quality meets the conditions and is consistent, the computer device determines that the quality and consistency verification of the first target image and the first sample image has passed; if the image quality and consistency feedback indicates that the quality does not meet the conditions or is inconsistent, the computer device determines that the quality and consistency verification of the first target image and the first sample image has failed.

[0075] In some embodiments, the computer device may also generate the first sample image using a fully manual labeling process. This process may involve the user manually aligning and labeling the four corners or edges of the first target image and the first sample image, thereby increasing reliability. In other embodiments, the computer device may also generate the first sample image based on a labeling process for synthetic data. This process may involve the computer device using computer graphics technology to generate a synthetic first sample image, replacing the actually captured first sample image, thereby significantly reducing data acquisition costs.

[0076] Step 205: The computer device trains the model based on the first target image and the first sample image to obtain the screen pattern removal model.

[0077] The computer device determines multiple prediction images based on a first sample image using a destigmatization model. These multiple prediction images are the first sample image after destigmatization, and each prediction image has a different size. The computer device performs supervised training based on the first target image and the multiple prediction images to obtain the destigmatization model. This process is described in detail in subsequent embodiments. In some embodiments, after the computer device trains the destigmatization model, it can be deployed online. Furthermore, after the destigmatization model is deployed, the computer device will collect user feedback and iterate the model training continuously to ensure that users obtain the best destigmatization effect in actual use. Correspondingly, the method also includes: obtaining a ninth image; inputting the ninth image into the destigmatization model and outputting a tenth image, where the tenth image is the image obtained by destigmatizing the ninth image using the destigmatization model; and updating the destigmatization model based on the destigmatization feedback of the tenth image.

[0078] The computer device displays a second feedback interface, which is used by the user to input or select descratching feedback for the tenth image. The computer device acquires the descratching feedback for the tenth image. If the descratching feedback for the tenth image is satisfactory, the process ends; if the descratching feedback is unsatisfactory, the descratching model is updated. The process of updating the descratching model by the computer device can be as follows: the computer device adds the first target image to the failed sample set, then analyzes the first target image in the failed sample set, updates the acquisition scene, and re-executes steps 201-205 based on the updated acquisition scene.

[0079] For example, please continue to refer to Figure 3 After the computer device goes online with the destigma model, it acquires the ninth image, which is a real-world photograph. The destigma model is used to process the ninth image to obtain the tenth image, which is then output as the result. If the user is satisfied, the process ends. If the user is not satisfied, the first target image is added to the failure sample set. Then, the first target image in the failure sample set is analyzed to update the acquisition scene.

[0080] In some embodiments, the computer device can test the screen-removed model before training it and deploying it online; the screen-removed model is deployed only if it passes the test; if it fails the test, the screen-removed model is updated; the process of updating the screen-removed model is similar to the update process described above, and will not be repeated here.

[0081] The process of testing the desmearing model using computer equipment can be as follows: The computer equipment acquires a set of test images, which includes multiple twelfth images, each a real-world photograph; the twelfth image is input into the desmearing model and output as the twelfth image, displaying a third feedback interface. This third feedback interface allows the user to input or select desmearing feedback for the twelfth image. The computer equipment acquires the desmearing feedback for the twelfth image. If the desmearing feedback for the twelfth image is satisfactory, the desmearing model is deemed to have passed the test; if the desmearing feedback for the twelfth image is unsatisfactory, the desmearing model is updated. For example, please refer to [reference needed]. Figure 3 The desmearing feedback of the twelfth image can be used to determine whether the subjective effect of the twelfth image meets the standard.

[0082] In this embodiment, a fully automatic data acquisition and labeling algorithm is adopted to automatically determine and label the first sample image with screen patterns based on the first target image without screen patterns in the image set. This improves the iteration efficiency and management automation level of the image set, reduces manual intervention, ensures the high quality and consistency of the first sample image acquisition, and provides a solid foundation for the training of the screen pattern removal model.

[0083] In this embodiment, the second image is an image obtained by superimposing a first target image without screen patterns onto a positioning plate image. By capturing the second image displayed on the screen, a third image with screen patterns can be obtained. With the help of the positioning plate image, a sample image with screen patterns can be extracted from the third image with screen patterns, thereby automatically generating a sample image with screen patterns. Compared with a synthesized sample image with screen patterns, the generated sample image with screen patterns is more realistic, which improves the quality of the generated sample image and thus improves the inference ability of the screen pattern removal model trained based on the sample image.

[0084] Please refer to Figure 6 The diagram illustrates a flowchart of a method for training a screen pattern removal model according to an exemplary embodiment of this application. The method may include the following steps:

[0085] Step 601: The computer device captures the first image and the second image displayed on the screen respectively to obtain the third image and the fourth image. The first image is a positioning plate image without screen patterns. The second image is an image obtained by superimposing the first target image without screen patterns on the first image. The third image and the fourth image are images with screen patterns.

[0086] In some embodiments, this step is the same as step 201, and will not be described again here.

[0087] Step 602: The computer device determines a target mapping matrix based on the first image, the second image, the third image, and the fourth image. The target mapping matrix is ​​used to represent the mapping relationship between the pixels of the first image and the pixels of the third image.

[0088] In some embodiments, this step is the same as step 202, and will not be described again here.

[0089] Step 603: Based on the target mapping matrix, the computer device takes screenshots from the third and fourth images respectively to obtain the fifth and sixth images. The fifth and sixth images are actual screenshots of the first and second images, respectively.

[0090] In some embodiments, this step is the same as step 203, and will not be described again here.

[0091] Step 604: The computer device takes a screenshot from the fifth image based on the fifth image and the sixth image to obtain a first sample image, which is a first target image with screen patterns.

[0092] In some embodiments, this step is the same as step 204, and will not be described again here.

[0093] Step 605: The computer device determines the first image feature of the first sample image. The first image feature includes a first spatial feature, a first channel feature, and other features. The first spatial feature and the first channel feature are used to represent the feature information of the spatial dimension and the feature information of the channel dimension of the first sample image, respectively. The other features are the features in the image feature other than the first spatial feature and the first channel feature.

[0094] In this embodiment, the computer device iteratively trains an initial destigmatization model to obtain a final destigmatization model. The initial destigmatization model can be a T-BED model, which includes an encoder, a Transformer, and a decoder. In this step, the encoder determines the first image features of the first sample image. The encoder can be a downsampling encoder, and the decoder can be an upsampling decoder. Correspondingly, the computer device inputs the first sample image into the destigmatization model, and through processing by the encoder and decoder, finally outputs multiple predicted images. These multiple predicted images are the destigmatized first sample images, and the sizes of the multiple predicted images are different. This process can be referenced from [reference needed]. Figure 7 .

[0095] In this embodiment, the T-BED model is used for end-to-end moiré pattern removal and image restoration at high resolutions (e.g., 2K). The T-BED model comprises a symmetrical downsampling encoder and upsampling decoder, characterized by the use of a scalable and uniformly applicable basic unit (Transformer-Boosted Block, TBB) suitable for upsampling / downsampling / isosampling in the encoder / decoder. Based on the traditional encoder / decoder, the TBB module uses a Transformer to extract spatial attention (SA) and channel attention (CA) features in parallel using the spatial and channel information features of the first sample image as input. After multiple TBB encoding / decoding operations, the moiré pattern removal model generates multi-scale prediction images P_3, P_2, and P_1 after moiré pattern removal, thereby achieving multi-scale supervised learning and test inference.

[0096] Step 606: The computer device performs predictive processing on the first spatial features to obtain the second spatial features, and performs predictive processing on the first channel features to obtain the second channel features.

[0097] In some embodiments, the step of the computer device performing predictive processing on the first spatial feature to obtain the second spatial feature may be as follows: the computer device performs dimensionality reduction processing on the first spatial feature to obtain a third spatial feature, the dimension of the third spatial feature being lower than the dimension of the first spatial feature; stretching the third spatial feature to obtain a fourth spatial feature, the size of the fourth feature matching the size of the descratching model; inputting the fourth spatial feature into the descratching model to output a fifth spatial feature; restoring the fifth spatial feature to obtain a sixth spatial feature, the restoration processing being the inverse of the stretching processing; and upsampling the sixth spatial feature to obtain the second spatial feature.

[0098] The steps for a computer device to input fourth-space features into a desmearing model and output fifth-space features can be as follows: The computer device inputs the fourth-space features into the transformer in the desmearing model and outputs the fifth-space features. For example, please refer to [link to relevant documentation]. Figure 7 The first spatial feature of the computer device is first reduced in dimensionality by the PixelShuffleDown module. Then, the third spatial feature obtained by dimensionality reduction is stretched into SpatialTokens by SpatialFlatten+Transpose, so that it can be fed into the Transformer module for spatial attention processing and equal-length sequence output. The processed SpatialTokens (fourth spatial feature) are restored to their original shape by SpatialUnFlatten+Transpose (fifth spatial feature), and then spatially upsampled to the original resolution by the PixelShuffleUp module, thus obtaining the second spatial feature SA (Spatial Attention).

[0099] In other embodiments, the step of the computer device performing prediction processing on the first channel feature to obtain the second channel feature can be as follows: the computer device performs dimensionality reduction processing on the first channel feature to obtain a third channel feature, the dimension of the third channel feature being lower than the dimension of the first channel feature; the third channel feature is stretched to obtain a fourth channel feature, the size of the fourth feature matching the size of the descratching model; the fourth channel feature is input into the descratching model to output a fifth channel feature; the fifth channel feature is restored to obtain a sixth channel feature, the restoration processing being the inverse of the stretching processing; and the sixth channel feature is upsampled to obtain the second channel feature.

[0100] For example, please continue to refer to Figure 7The computer device performs channel dimensionality reduction on the first channel features using a 1x1 convolution. Then, it uses SpatialFlatten to convert the dimensionality-reduced third channel features into ChannelTokens, which are then fed into the Transformer module for channel attention processing and equal-length sequence output. The processed SpatialTokens (fourth channel features) are restored to their original shape (fifth channel features) using SpatialUnFlatten, and then the channel dimensionality is increased back to the original number using a 1x1 convolution, resulting in the second channel features, CA (Channel Attention).

[0101] In this embodiment, the T-BED model leverages the powerful capabilities of the Transformer module, enabling flexible expansion to adapt to different parameters and model requirements, ensuring both efficiency and scalability. Through multi-scale prediction and supervision, the T-BED model significantly improves the demoiring effect while preserving image details and quality.

[0102] In some embodiments, the computer device may also use a conventional convolutional neural network architecture to enhance spatial and channel features, replacing some devices that do not support Transformer inference deployment.

[0103] Step 607: The computer device fuses the second spatial features, the second channel features, and other features to obtain the second image features of the first sample image.

[0104] The computer device fuses the second spatial features, the second channel features, and other features through the residual network in the descratching model to obtain the second image features of the first sample image.

[0105] Step 608: The computer device performs multi-scale sampling and decoding of the second image features to obtain multiple predicted images.

[0106] The computer device performs upsampling and downsampling processing on the second image features to obtain the third and fourth image features, respectively. The computer device then decodes the second, third, and fourth image features to obtain multiple predicted images. Specifically, the computer device uses the PixelShuffleUp module to upsample the second image features to obtain the third image features, and uses the PixelShuffleDown module to downsample the second image features to obtain the fourth image features.

[0107] In some embodiments, the computer device can also add a multi-scale feature fusion module to the T-BED model. By fusing feature maps of different scales, the removal effect of the screen pattern removal model and the image restoration capability can be improved. Accordingly, the step of the computer device decoding the second image features, the third image features, and the fourth image features to obtain multiple predicted images can be replaced by: the computer device fusing the second image features, the third image features, and the fourth image features to obtain fused image features, and decoding the fused image features to obtain the predicted image.

[0108] In other embodiments, the computer device can also perform multi-scale supervised training and inference testing based on the Laplacian pyramid to achieve better multi-scale frequency band separation; correspondingly, the step of the computer device performing multi-scale sampling and decoding processing on the second image features to obtain multiple predicted images can be as follows: the computer device determines the third image features and the fourth image features through the Laplacian pyramid; the computer device performs decoding processing on the second image features, the third image features and the fourth image features respectively to obtain multiple predicted images.

[0109] Step 609: The computer device performs supervised training based on the first target image and multiple predicted images to obtain the screen pattern removal model.

[0110] The computer device iteratively trains the desmearing model based on the first target image and multiple predicted images using any supervised training algorithm to obtain the final trained desmearing model.

[0111] In this embodiment, the T-BED model leverages the powerful capabilities of the Transformer module, enabling flexible expansion to adapt to different parameters and model requirements, ensuring both efficiency and scalability. Through multi-scale prediction and supervision, the T-BED model significantly improves the demoiring effect while preserving image details and quality.

[0112] Please refer to Figure 8 The diagram illustrates a flowchart of a method for training a screen pattern removal model according to an exemplary embodiment of this application. The method may include the following steps:

[0113] Step 801: The computer device captures the first image and the second image displayed on the screen respectively to obtain the third image and the fourth image. The first image is a positioning plate image without screen patterns. The second image is an image obtained by superimposing the first target image without screen patterns on the first image. The third image and the fourth image are images with screen patterns.

[0114] In some embodiments, this step is the same as step 201, and will not be described again here.

[0115] Step 802: The computer device determines a target mapping matrix based on the first image, the second image, the third image, and the fourth image. The target mapping matrix is ​​used to represent the mapping relationship between the pixels of the first image and the pixels of the third image.

[0116] In some embodiments, this step is the same as step 202, and will not be described again here.

[0117] Step 803: Based on the target mapping matrix, the computer device takes screenshots from the third and fourth images respectively to obtain the fifth and sixth images. The fifth and sixth images are actual screenshots of the first and second images, respectively.

[0118] In some embodiments, this step is the same as step 203, and will not be described again here.

[0119] Step 804: The computer device takes a screenshot from the fifth image based on the fifth image and the sixth image to obtain a first sample image, which is a first target image with screen patterns.

[0120] In some embodiments, this step is the same as step 204, and will not be described again here.

[0121] Step 805: Based on the first sample image, the computer device determines multiple prediction images through a destigmatization model. The multiple prediction images are the first sample image after destigmatization, and the multiple prediction images have different sizes.

[0122] In some embodiments, this step is the same as step 205; or this step can be implemented by the above steps 605-608, which will not be described again here.

[0123] Step 806: The computer device determines a first loss function based on the differences between the multiple predicted images and the first target image, wherein the first loss function is used to characterize the sum of the differences between the multiple predicted images and the first target image.

[0124] The computer device scales a first target image based on the sizes of multiple predicted images to obtain multiple second target images, each with the same size as the predicted images. For example, the computer device performs downsampling on the first target image to obtain two second target images, and then combines the first target image and the two second target images to form multiple second target images. For any predicted image, the computer device determines the difference between the predicted image and the second target images of the same size. Since there are multiple predicted images, multiple differences are obtained, and these differences are summed to obtain a first loss function. For example, the first loss function is shown in Formula 1:

[0125] Formula 1:

[0126] Where L1 represents the first loss function, i represents the index of the predicted image, 3 represents the number of predicted images (also known as the number of second target images), and T i Let P represent the i-th second target image. i Let i represent the i-th predicted image.

[0127] Step 807: The computer device determines a second loss function based on the differences between the image features of the multiple predicted images and the image features of the first target image. The second loss function is used to characterize the sum of the differences between the image features of the multiple predicted images and the image features of the first target image.

[0128] The computer device scales a first target image based on the sizes of multiple predicted images to obtain multiple second target images, each with the same size as the predicted images. For example, the computer device performs downsampling on the first target image to obtain two second target images, and combines the first target image and the two second target images to form multiple second target images. Image features of the multiple second target images are determined. For any predicted image, the computer device determines the difference between the image features of the predicted image and the image features of a second target image of the same size. Since there are multiple predicted images, multiple differences are obtained, and these differences are summed to obtain a second loss function. For example, the second loss function is shown in Formula 2.

[0129] Formula 2:

[0130] Among them, L p VGG(T) represents the second loss function, i represents the index of the predicted image, 3 represents the number of predicted images (also known as the number of second target images), and VGG(T) represents the second loss function. i VGG(P) represents the image features of the i-th second target image. i ) represents the image features of the i-th predicted image.

[0131] Since screen pattern removal and lens distortion ignoring can be supervised at high and low frequencies respectively: for screen pattern removal, the predicted image output by the screen pattern removal model needs to learn from the first target image, because screen pattern features affect high-frequency components; for lens distortion ignoring, the predicted image output by the screen pattern removal model needs to learn from the first sample image, because lens distortion is mainly manifested in low-frequency components. Therefore, for T-BED multi-scale output, in this embodiment, in addition to multi-scale conventional spatial domain supervision achieved by the first and second loss functions, multi-scale frequency domain auxiliary supervision is still required through steps 808 and 809, namely differential frequency domain low-pass supervision (third loss function) and differential frequency domain high-pass supervision (fourth loss function).

[0132] In this embodiment, based on prior knowledge of screen ripple frequency domain characteristics, a supervision mechanism separating high and low frequencies is designed to decouple moiré removal from image distortion preservation. High-pass / low-pass filtering attention in the frequency domain precisely supervises both moiré and lens distortion, ensuring focused, efficient, and stable training. These innovations collectively constitute an efficient, accurate, and scalable moiré removal solution, significantly improving image processing quality and user experience.

[0133] Step 808: The computer device determines a third loss function based on the difference between the frequency domain features of the multiple predicted images and the frequency domain features of the first sample image. The third loss function is used to characterize the sum of the differences between the frequency domain features of the multiple predicted images and the frequency domain features of the first sample image.

[0134] (1) The computer equipment determines the difference between the spatial features of multiple predicted images and the spatial features of the first sample image, and obtains multiple first spatial interpolation maps.

[0135] The computer device scales a first sample image based on the dimensions of multiple predicted images to obtain multiple second sample images, each with the same dimensions as the predicted images. For example, the computer device performs two-scale sampling on the first sample image to obtain two second sample images, and combines the first sample image and the two second sample images to form multiple second sample images. The spatial features of the multiple second sample images are determined. For any predicted image, the computer device determines the difference between the spatial features of the predicted image and the spatial features of the second sample images of the same size as the predicted image to obtain a first spatial interpolation map. Since there are multiple predicted images, multiple first spatial interpolation maps are obtained.

[0136] (2) The computer equipment performs Fourier transform on the multiple first spatial interpolation maps respectively to obtain multiple first frequency domain vector spectra.

[0137] For any first spatial interpolation map, the computer device determines the Fourier transform of the first spatial interpolation map using the following formula three to obtain the first frequency domain vector spectrum; since there are multiple first spatial interpolation maps, multiple first frequency domain vector spectra will be obtained.

[0138] Formula 3:

[0139] Where F(u, v) represents the frequency domain vector value with coordinates (u, v), x represents the x-coordinate of a pixel in the first spatial interpolation map, y represents the y-coordinate of a pixel in the first spatial interpolation map, M represents the number of pixels included in the length of the first spatial interpolation map, N represents the number of pixels included in the width of the first spatial interpolation map, i represents the index of the first frequency domain vector spectrum, u and v represent the horizontal component (or x-coordinate), and v represents the vertical component (or y-coordinate).

[0140] In this embodiment, the computer device performs a Fourier transform of the complex representation using FFL, thereby simultaneously considering the equally important amplitude and phase diagram monitoring.

[0141] In some embodiments, the computer device can also perform frequency domain supervision based on wavelet transform, that is, use wavelet transform instead of Fourier transform for frequency domain supervision. Wavelet transform can capture the time and frequency information of the image simultaneously, but its computational complexity is high, and it is needed in challenging scenarios with higher accuracy requirements. Accordingly, step (2) can be replaced by: the computer device performing wavelet transform on multiple first spatial interpolation maps respectively to obtain multiple first frequency domain vector spectra.

[0142] (3) The computer device filters multiple first frequency domain vector spectra through a first filter to obtain multiple first attention weights. The first filter is used to filter out signals in the multiple first frequency domain vector spectra whose frequencies are higher than a first preset frequency.

[0143] (4) The computer equipment determines multiple fifth loss functions in the spectral domain for the spatial features of multiple predicted images and the spatial features of the first sample image.

[0144] The computer device scales a first sample image based on the dimensions of multiple prediction images to obtain multiple second sample images, each with the same dimensions as the prediction images. For example, the computer device performs downsampling on the first sample image by two scales to obtain two second sample images. These first and second sample images are then combined to form multiple second sample images. For any prediction image, the computer device determines the difference between the spatial features of the prediction image and the spatial features of the second sample images of the same size, thus obtaining a fifth loss function. Because there are multiple prediction images, multiple fifth loss functions are obtained. For example, please refer to [reference needed]. Figure 9 The computer device downsamples the first sample image M1 to obtain the second sample image M2, and downsamples the second sample image M2 to obtain the second sample image M3. The third loss function is determined by the encoder and decoder in the T-BED model based on M1, M2 and M3, as well as multiple predicted images P1, P2 and P3.

[0145] (5) The computer device uses multiple first attention weights to weight multiple fifth loss functions to obtain the third loss function.

[0146] Steps (1) to (5) can be achieved using Formula 4:

[0147] Formula 4:

[0148] Among them, L M-P This represents the third loss function, where i represents the index of the predicted image, 3 represents the number of predicted images, and H represents the number of predicted images. LowPass Represents a low-pass filter; F(|M i -P i |) represents the first attention weight corresponding to the i-th predicted image, |F(M) i )-F(P i )| 2 Let represent the fifth loss function corresponding to the i-th predicted image.

[0149] Step 809: The computer device determines a fourth loss function based on the difference between the frequency domain features of the multiple predicted images and the frequency domain features of the first target image. The fourth loss function is used to characterize the sum of the differences between the frequency domain features of the multiple predicted images and the frequency domain features of the first target image.

[0150] (1) The computer equipment determines the difference between the spatial features of multiple predicted images and the spatial features of the first target image, and obtains multiple second spatial interpolation maps.

[0151] The computer device scales a first target image based on the sizes of multiple predicted images to obtain multiple second target images, each with the same size as the predicted images. For example, the computer device performs two-scale sampling on the first target image to obtain two second target images, and combines the first target image and the two second target images to form multiple second target images. The spatial features of the multiple second target images are determined. For any predicted image, the computer device determines the difference between the spatial features of the predicted image and the spatial features of the second target images of the same size to obtain a second spatial interpolation map. Since there are multiple predicted images, multiple second spatial interpolation maps are obtained.

[0152] (2) The computer equipment performs Fourier transform on multiple second spatial interpolation maps to obtain multiple second frequency domain vector spectra.

[0153] This step is similar to step (2) in step 808, and will not be repeated here.

[0154] (3) The computer device filters multiple second frequency domain vector spectra through a second filter to obtain multiple second attention weights. The second filter is used to filter out signals with frequencies lower than a second preset frequency in the multiple second frequency domain vector spectra.

[0155] (4) The computer equipment determines the sixth loss function in the spectral domain of the spatial features of multiple predicted images and the spatial features of the first target image.

[0156] The computer device scales a first target image based on the sizes of multiple predicted images to obtain multiple second target images, each with the same size as the predicted images. For example, the computer device performs two-scale sampling on the first target image to obtain two second target images, and combines the first target image and the two second target images to form multiple second target images. For any predicted image, the computer device determines the difference between the spatial features of the predicted image and the spatial features of the second target images of the same size, thus obtaining a sixth loss function. Since there are multiple predicted images, multiple sixth loss functions are obtained.

[0157] (5) The computer device uses multiple second attention weights to weight multiple sixth loss functions to obtain a fourth loss function.

[0158] Steps (1) to (5) can be achieved using Formula 5:

[0159] Formula 5:

[0160] Among them, L T-P This represents the fourth loss function, where i represents the index of the predicted image, 3 represents the number of predicted images, and H... HighPass F(|T) represents a high-pass filter; i -P i |) represents the second attention weight corresponding to the i-th predicted image, |F(T) i )-F(P i )| 2 Let represent the sixth loss function corresponding to the i-th predicted image.

[0161] For example, please continue to refer to Figure 9 The computer device downsamples the first target image T1 to obtain the second target image T2, and downsamples the second target image T2 to obtain the second target image T3. The fourth loss function is determined by the encoder and decoder in the T-BED model based on T1, T2 and T3, as well as multiple predicted images P1, P2 and P3.

[0162] Step 810: The computer device iteratively updates the screen pattern removal model based on the first loss function, the second loss function, the third loss function, and the fourth loss function to obtain the updated screen pattern removal model.

[0163] The computer device updates the screen pattern removal model based on the first loss function, the second loss function, the third loss function, and the fourth loss function. Since the image set includes multiple first target images, multiple first sample images can be generated through steps 801-804. These multiple first target images and multiple first sample images are stored in pairs in the training set and the validation set. It is determined whether the paired images in the training set have completed a first preset number of iterations. If the paired images in the training set have completed the first preset number of iterations, the paired images in the validation set are traversed. Based on the paired images, it is determined whether the descratching effect of the updated descratching model has not improved. If the descratching effect has improved, the iteration count is reset and the stored optimal parameter weights are updated. If the descratching effect of the updated descratching model has not improved, it is determined whether the consecutive number of times the descratching effect has not improved exceeds a preset number. If it has not exceeded the preset number, the iteration is reset; at this time, there is no need to update the stored optimal parameter weights. If the consecutive number of times the descratching effect has not improved exceeds the preset number, the iteration update ends, the stored optimal parameter weights are loaded, and based on the stored optimal parameter weights, the weight parameters of the descratching model are set to the stored optimal parameter weights to obtain the final descratching model. If the paired images in the training set have not completed the first preset number of iterations, the computer device randomly samples images from the training set without replacement. Based on the sampled image pairs, predictions are made using the destigmatization model, and the destigmatization model is updated. Please refer to [link to documentation] for this process. Figure 3 .

[0164] In this embodiment, based on the first, second, third, and fourth loss functions, the screen removal model is iteratively updated to obtain an updated screen removal model that can remove screen patterns and ignore lens distortion. This decouples screen pattern removal and lens distortion processing, allowing the screen removal model to focus on screen pattern removal while ignoring lens distortion, making the training process more focused, efficient, and stable. The overall solution not only improves the screen pattern removal effect but also ensures the original image quality and user experience, demonstrating significant practical value and technical advantages. For example, please refer to... Figure 9The computer equipment iteratively trains the screen-removed image based on the first and second loss functions, combined with the third and fourth loss functions. Furthermore, to remove screen patterns, the high-frequency features of multiple predicted images need to be different from those of multiple second sample images, and the high-frequency features of multiple predicted images need to learn from multiple second target images. To ignore lens distortion, the low-frequency features of multiple predicted images need to learn from multiple second target images, and the high-frequency features of multiple predicted images need to be different from those of multiple second target images.

[0165] In some embodiments, when training the screen pattern removal model, the model is trained based on the first sample image and the first target image as sample pairs, that is, positive sample pairs are constructed; the computer device can also obtain negative sample pairs, which include the third target image and the third sample image, both of which are screen pattern images or screen pattern-free images; based on the positive sample pairs and the negative sample pairs, the model is trained through contrastive learning to obtain the screen pattern removal model, thereby improving the generalization ability of the screen pattern removal model through contrastive learning.

[0166] Please refer to Figure 10 This illustration shows a structural block diagram of a training apparatus for a desmearing model provided in an exemplary embodiment of this application. The apparatus includes:

[0167] The imaging module 1001 is used to capture a first image and a second image displayed on the display screen respectively, to obtain a third image and a fourth image. The first image is a positioning plate image without screen patterns, and the second image is obtained by superimposing a first target image without screen patterns onto the first image. The third image and the fourth image are images with screen patterns. The determining module 1002 is used to determine a target mapping matrix based on the first image, the second image, the third image, and the fourth image. The target mapping matrix is ​​used to represent the mapping relationship between the pixels of the first image and the pixels of the third image. The first screenshot module 1003 is used to take screenshots from the third image and the fourth image respectively based on the target mapping matrix to obtain a fifth image and a sixth image. The fifth image and the sixth image are actual screen images of the first image and the second image, respectively. The second screenshot module 1004 is used to take screenshots from the fifth image based on the fifth image and the sixth image to obtain a first sample image. The first sample image is a first target image with screen patterns. The training module 1005 is used to train a model based on the first target image and the first sample image to obtain a screen pattern removal model.

[0168] In some embodiments, the determining module 1002 is configured to: determine a first mapping matrix based on the first image and the third image, wherein the first mapping matrix represents the mapping relationship between pixels in the first image and pixels in the third image; determine a second mapping matrix based on the second image and the fourth image, wherein the second mapping matrix represents the mapping relationship between pixels in the second image and pixels in the fourth image; and determine the target mapping matrix from the first mapping matrix and the second mapping matrix.

[0169] In some embodiments, the determining module 1002 is configured to determine a first reprojection error of the first mapping matrix and a second reprojection error of the second mapping matrix; and based on the first reprojection error and the second reprojection error, to determine a target mapping matrix from the first mapping matrix and the second mapping matrix, wherein the target mapping matrix is ​​the mapping matrix with the smallest reprojection error among the first mapping matrix and the second mapping matrix.

[0170] In some embodiments, the second screenshot module 1004 is used to perform a difference operation on the fifth image and the sixth image to obtain a seventh image; to perform binarization processing on the seventh image to obtain an eighth image, the eighth image being a mask image of the first target image; and to extract the first target image from the fifth image based on the eighth image to obtain the first sample image.

[0171] In some embodiments, the second screenshot module 1004 is configured to determine multiple sets of marker detection points based on the fifth image and the sixth image, wherein each set of marker detection points includes marker detection points with the same coordinates in the fifth image and the sixth image; perform consistency verification on the fifth image and the sixth image based on the multiple sets of marker detection points; and perform a difference operation on the fifth image and the sixth image to obtain a seventh image if the consistency verification passes.

[0172] In some embodiments, the apparatus further includes: a generation module, configured to generate the first image based on the image size of the first target image and the screen size of the display screen, the first image including multiple rows and columns of marker detection points; an overlay module, configured to overlay the first target image onto the first image to obtain a second image, wherein at least one row and one column of marker detection points are reserved around the first target image in the second image; and a display module, configured to display the first image and the second image on the display screen.

[0173] In some embodiments, the generation module is configured to determine a chessboard image and a marker detection image based on the image size of the first target image and the screen size of the display screen. The chessboard image includes alternating black and white first and second cells, and the marker detection image includes multiple third cells displaying marker detection points. The third cells in the marker detection image are then merged into the second cells in the chessboard image to obtain the first image.

[0174] In some embodiments, the training module 1005 is used to determine multiple prediction images based on the first sample image using a destigmatization model, wherein the multiple prediction images are the first sample image after destigmatization, and the multiple prediction images have different sizes; and to perform supervised training based on the first target image and the multiple prediction images to obtain the destigmatization model.

[0175] In some embodiments, the training module 1005 is configured to determine a first image feature of the first sample image, the first image feature including a first spatial feature, a first channel feature, and other features, wherein the first spatial feature and the first channel feature are respectively used to represent feature information in the spatial dimension and feature information in the channel dimension of the first sample image, and the other features are features other than the first spatial feature and the first channel feature in the image feature; perform prediction processing on the first spatial feature to obtain a second spatial feature, and perform prediction processing on the first channel feature to obtain a second channel feature; fuse the second spatial feature, the second channel feature, and the other features to obtain a second image feature of the first sample image; and perform multi-scale sampling and decoding processing on the second image feature to obtain the plurality of predicted images.

[0176] In some embodiments, the training module 1005 is configured to: perform dimensionality reduction processing on the first spatial feature to obtain a third spatial feature, wherein the dimension of the third spatial feature is lower than that of the first spatial feature; perform stretching processing on the third spatial feature to obtain a fourth spatial feature, wherein the size of the fourth feature matches the size of the descratching model; input the fourth spatial feature into the descratching model to output a fifth spatial feature; perform restoration processing on the fifth spatial feature to obtain a sixth spatial feature, wherein the restoration processing is the inverse of the stretching processing; and upsample the sixth spatial feature to obtain a second spatial feature.

[0177] In some embodiments, the training module 1005 is configured to: perform dimensionality reduction processing on the first channel feature to obtain a third channel feature, wherein the dimension of the third channel feature is lower than that of the first channel feature; perform stretching processing on the third channel feature to obtain a fourth channel feature, wherein the size of the fourth feature matches the size of the descratching model; input the fourth channel feature into the descratching model to output a fifth channel feature; perform restoration processing on the fifth channel feature to obtain a sixth channel feature, wherein the restoration processing is the inverse of the stretching processing; and upsample the sixth channel feature to obtain a second channel feature.

[0178] In some embodiments, the training module 1005 is used to perform upsampling and downsampling processing on the second image features to obtain third image features and fourth image features; and to perform decoding processing on the second image features, the third image features and the fourth image features to obtain the plurality of predicted images.

[0179] In some embodiments, the training module 1005 is configured to: determine a first loss function based on the differences between the plurality of predicted images and the first target image, wherein the first loss function represents the sum of the differences between the plurality of predicted images and the first target image; determine a second loss function based on the differences between the image features of the plurality of predicted images and the image features of the first target image, wherein the second loss function represents the sum of the differences between the image features of the plurality of predicted images and the image features of the first target image; determine a third loss function based on the differences between the frequency domain features of the plurality of predicted images and the frequency domain features of the first sample image, wherein the third loss function represents the sum of the differences between the frequency domain features of the plurality of predicted images and the frequency domain features of the first sample image; determine a fourth loss function based on the differences between the frequency domain features of the plurality of predicted images and the frequency domain features of the first target image, wherein the fourth loss function represents the sum of the differences between the frequency domain features of the plurality of predicted images and the frequency domain features of the first target image; and iteratively update the descratching model based on the first loss function, the second loss function, the third loss function, and the fourth loss function to obtain an updated descratching model.

[0180] In some embodiments, the training module 1005 is configured to: determine the differences between the spatial features of the plurality of predicted images and the spatial features of the first sample image to obtain a plurality of first spatial interpolation maps; perform Fourier transform on the plurality of first spatial interpolation maps to obtain a plurality of first frequency domain vector spectra; filter the plurality of first frequency domain vector spectra using a first filter to obtain a plurality of first attention weights, wherein the first filter is used to filter out signals in the plurality of first frequency domain vector spectra whose frequencies are higher than a first preset frequency; determine a plurality of fifth loss functions in the spectral domain for the spatial features of the plurality of predicted images and the spatial features of the first sample image; and weight the plurality of fifth loss functions based on the plurality of first attention weights to obtain the third loss function.

[0181] In some embodiments, the training module 1005 is configured to: determine the differences between the spatial features of the plurality of predicted images and the spatial features of the first target image to obtain a plurality of second spatial interpolation maps; perform Fourier transform on the plurality of second spatial interpolation maps to obtain a plurality of second frequency domain vector spectra; filter the plurality of second frequency domain vector spectra using a second filter to obtain a plurality of second attention weights, wherein the second filter is used to filter out signals in the plurality of second frequency domain vector spectra whose frequencies are lower than a second preset frequency; determine a sixth loss function in the spectral domain for the spatial features of the plurality of predicted images and the spatial features of the first target image; and weight the plurality of sixth loss functions based on the plurality of second attention weights to obtain a fourth loss function.

[0182] In some embodiments, the apparatus further includes: an acquisition module for acquiring a ninth image; an input module for inputting the ninth image into the destigmatization model and outputting a tenth image, wherein the tenth image is an image obtained by destigmatizing the ninth image through the destigmatization model; and an update module for updating the destigmatization model based on the destigmatization feedback of the tenth image.

[0183] In this embodiment, the second image is an image obtained by superimposing a first target image without screen patterns onto a positioning plate image. By capturing the second image displayed on the screen, a third image with screen patterns can be obtained. With the help of the positioning plate image, a sample image with screen patterns can be extracted from the third image with screen patterns, thereby automatically generating a sample image with screen patterns. Compared with a synthesized sample image with screen patterns, the generated sample image with screen patterns is more realistic, which improves the quality of the generated sample image and thus improves the inference ability of the screen pattern removal model trained based on the sample image.

[0184] It should be noted that the screen-texture removal model training device provided in the above embodiments is only illustrated by the division of the above functional modules when training the screen-texture removal model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the screen-texture removal model training device and the screen-texture removal model training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0185] See Figure 11 , Figure 11 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. The computer device may further include one or more of the following components: processor 1110, memory 1120, and display screen 1130.

[0186] The processor 1110 connects to various parts of the computer device 1100 via various interfaces and lines. It performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1120, and by calling data stored in the memory 1120. Optionally, the processor 1110 can be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1110 can integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), and modem. Specifically, the CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required to be displayed on the display screen 1130; the NPU is used to implement Artificial Intelligence (AI) functions; and the modem is used to handle wireless communication. It is understandable that the aforementioned modem may not be integrated into the processor 1110, but may be implemented separately through a computer program product.

[0187] The memory 1120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1120 may include a non-transitory computer-readable storage medium. The memory 1120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc.; the data storage area may store data created according to the use of the computer device 1100 (such as audio data, telephone book, etc.).

[0188] Display screen 1130 is a display component used to display a user interface. Optionally, the display screen 1130 is a touch-enabled display screen, through which users can use their fingers, styluses, or any suitable object to perform touch operations on the display screen 1130. Display screen 1130 is typically located on the front panel of computer device 1100. Display screen 1130 can be designed as a full-screen, curved screen, irregularly shaped screen, dual-sided screen, or foldable screen. Display screen 1130 can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen, etc., which are not limited in this embodiment.

[0189] In addition, those skilled in the art will understand that the structure of the computer device 1100 shown in the above figures does not constitute a limitation on the computer device 1100. The computer device 1100 may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the computer device 1100 may also include a Wi-Fi module, an audio acquisition device, a speaker, a radio frequency circuit, an input unit, a sensor, an audio circuit, a Bluetooth module, a power supply, and other components, which will not be described in detail here.

[0190] This application provides a computer-readable storage medium storing at least one computer instruction, which is executed by a processor to implement the screen pattern removal model training method described in the above embodiments. Alternatively, this application provides a computer program product including computer instructions stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the screen pattern removal model training method described in the above embodiments.

[0191] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. The above descriptions are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a screen pattern removal model, characterized in that, The method includes: The first image and the second image displayed on the screen are captured respectively to obtain the third image and the fourth image. The first image is a positioning plate image without screen patterns. The second image is an image obtained by superimposing the first target image without screen patterns on the first image. The third image and the fourth image are images with screen patterns. Based on the first image, the second image, the third image, and the fourth image, a target mapping matrix is ​​determined, which is used to represent the mapping relationship between the pixels of the first image and the pixels of the third image; Based on the target mapping matrix, a fifth image and a sixth image are obtained by taking screenshots from the third image and the fourth image, respectively. The fifth image and the sixth image are actual screenshot images of the first image and the second image, respectively. Based on the fifth image and the sixth image, a first sample image is obtained by taking a screenshot from the fifth image. The first sample image is a first target image with screen patterns. The screen pattern removal model is obtained by training the model based on the first target image and the first sample image.

2. The method according to claim 1, characterized in that, The step of determining the target mapping matrix based on the first image, the second image, the third image, and the fourth image includes: Based on the first image and the third image, a first mapping matrix is ​​determined, which is used to represent the mapping relationship between the pixels of the first image and the pixels of the third image. Based on the second image and the fourth image, a second mapping matrix is ​​determined, which is used to represent the mapping relationship between the pixels of the second image and the pixels of the fourth image; The target mapping matrix is ​​determined from the first mapping matrix and the second mapping matrix.

3. The method according to claim 2, characterized in that, Determining the target mapping matrix from the first mapping matrix and the second mapping matrix includes: Determine the first projection error of the first mapping matrix and the second projection error of the second mapping matrix; Based on the first reprojection error and the second reprojection error, the target mapping matrix is ​​determined from the first mapping matrix and the second mapping matrix. The target mapping matrix is ​​the mapping matrix with the smallest reprojection error among the first mapping matrix and the second mapping matrix.

4. The method according to claim 1, characterized in that, The step of obtaining the first sample image by cropping from the fifth image based on the fifth image and the sixth image includes: The seventh image is obtained by performing a difference operation on the fifth image and the sixth image; The seventh image is binarized to obtain the eighth image, which is a mask image of the first target image; Based on the eighth image, the first target image is extracted from the fifth image to obtain the first sample image.

5. The method according to claim 4, characterized in that, The step of performing a difference operation on the fifth image and the sixth image to obtain the seventh image includes: Based on the fifth image and the sixth image, multiple sets of marker detection points are determined, and each set of marker detection points includes marker detection points with the same coordinates in the fifth image and the sixth image; Based on the multiple sets of marker detection points, a consistency check is performed on the fifth image and the sixth image; If the consistency check passes, a difference operation is performed on the fifth image and the sixth image to obtain the seventh image.

6. The method according to claim 1, characterized in that, Before capturing the first and second images displayed on the screen respectively, and obtaining the third and fourth images, the method further includes: The first image is generated based on the image size of the first target image and the screen size of the display screen. The first image includes multiple rows and columns of marker detection points. The first target image is superimposed on the first image to obtain the second image, wherein at least one row and one column of marker detection points are retained around the first target image in the second image; The first image and the second image are displayed on the screen.

7. The method according to claim 6, characterized in that, Generating the first image based on the image size of the first target image and the screen size of the display screen includes: Based on the image size of the first target image and the screen size of the display screen, a chessboard image and a marker detection image are determined. The chessboard image includes a first cell and a second cell with alternating black and white colors, and the marker detection image includes a plurality of third cells displaying marker detection points. The third cell in the marker detection image is merged into the second cell in the chessboard image to obtain the first image.

8. The method according to claim 1, characterized in that, The step of training a model based on the first target image and the first sample image to obtain a screen pattern removal model includes: Based on the first sample image, multiple prediction images are determined by a screen removal model. The multiple prediction images are the first sample image after screen removal, and the multiple prediction images have different sizes. The screen descratching model is obtained by supervised training based on the first target image and the multiple predicted images.

9. The method according to claim 8, characterized in that, The step of determining multiple prediction images based on the first sample image using a destigmatism model includes: A first image feature is determined for the first sample image. The first image feature includes a first spatial feature, a first channel feature, and other features. The first spatial feature and the first channel feature are used to represent the feature information of the spatial dimension and the feature information of the channel dimension of the first sample image, respectively. The other features are features other than the first spatial feature and the first channel feature in the image features. The first spatial feature is predicted to obtain the second spatial feature, and the first channel feature is predicted to obtain the second channel feature. The second spatial feature, the second channel feature, and the other features are fused to obtain the second image feature of the first sample image; The second image features are sampled at multiple scales and then decoded to obtain the multiple predicted images.

10. The method according to claim 9, characterized in that, The step of predicting the first spatial features to obtain the second spatial features includes: The first spatial feature is subjected to dimensionality reduction processing to obtain a third spatial feature, wherein the dimension of the third spatial feature is lower than that of the first spatial feature; The third spatial feature is stretched to obtain a fourth spatial feature, the size of which matches the size of the screen texture removal model. The fourth spatial feature is input into the screen pattern removal model, and the fifth spatial feature is output. The fifth spatial feature is restored to obtain the sixth spatial feature, and the restoration process is the inverse of the stretching process. The second spatial feature is obtained by upsampling the sixth spatial feature.

11. The method according to claim 9, characterized in that, The process of predicting the features of the first channel to obtain the features of the second channel includes: The first channel feature is subjected to dimensionality reduction processing to obtain the third channel feature, and the dimension of the third channel feature is lower than that of the first channel feature; The third channel feature is stretched to obtain the fourth channel feature, and the size of the fourth feature matches the size of the screen texture removal model. The fourth channel feature is input into the screen pattern removal model to output the fifth channel feature; The fifth channel feature is restored to obtain the sixth channel feature. The restoration process is the inverse of the stretching process. The second channel feature is obtained by upsampling the sixth channel feature.

12. The method according to claim 9, characterized in that, The process of multi-scale sampling and decoding the second image features to obtain the multiple predicted images includes: The second image features are subjected to upsampling and downsampling processes respectively to obtain the third and fourth image features; The second image feature, the third image feature, and the fourth image feature are decoded respectively to obtain the plurality of predicted images.

13. The method according to claim 8, characterized in that, The step of supervising training based on the first target image and the plurality of predicted images to obtain the screen pattern removal model includes: Based on the differences between the plurality of predicted images and the first target image, a first loss function is determined, which is used to characterize the sum of the differences between the plurality of predicted images and the first target image. Based on the differences between the image features of the plurality of predicted images and the image features of the first target image, a second loss function is determined. The second loss function is used to characterize the sum of the differences between the image features of the plurality of predicted images and the image features of the first target image, respectively. Based on the difference between the frequency domain features of the multiple predicted images and the frequency domain features of the first sample image, a third loss function is determined. The third loss function is used to characterize the sum of the differences between the frequency domain features of the multiple predicted images and the frequency domain features of the first sample image, respectively. Based on the difference between the frequency domain features of the multiple predicted images and the frequency domain features of the first target image, a fourth loss function is determined. The fourth loss function is used to characterize the sum of the differences between the frequency domain features of the multiple predicted images and the frequency domain features of the first target image, respectively. Based on the first loss function, the second loss function, the third loss function, and the fourth loss function, the screen pattern removal model is iteratively updated to obtain the updated screen pattern removal model.

14. The method according to claim 13, characterized in that, The step of determining a third loss function based on the difference between the frequency domain features of the multiple predicted images and the frequency domain features of the first sample image includes: The spatial features of the multiple predicted images and the spatial features of the first sample image are determined respectively to obtain multiple first spatial interpolation maps; Perform Fourier transforms on the plurality of first spatial interpolation maps respectively to obtain a plurality of first frequency domain vector spectra; A first filter is used to filter the plurality of first frequency domain vector spectra to obtain a plurality of first attention weights. The first filter is used to filter out signals in the plurality of first frequency domain vector spectra whose frequencies are higher than a first preset frequency. Determine multiple fifth loss functions in the spectral domain for the spatial features of the multiple predicted images and the spatial features of the first sample image, respectively; The third loss function is obtained by weighting the multiple first attention weights and the multiple fifth loss functions respectively.

15. The method according to claim 13, characterized in that, The step of determining a fourth loss function based on the difference between the frequency domain features of the multiple predicted images and the frequency domain features of the first target image includes: The spatial features of the multiple predicted images and the spatial features of the first target image are determined respectively to obtain multiple second spatial interpolation maps; Perform Fourier transforms on the plurality of second spatial interpolation maps respectively to obtain a plurality of second frequency domain vector spectra; The plurality of second frequency domain vector spectra are filtered by a second filter to obtain a plurality of second attention weights. The second filter is used to filter out signals in the plurality of second frequency domain vector spectra whose frequencies are lower than a second preset frequency. The spatial features of the plurality of predicted images and the spatial features of the first target image are respectively determined in the spectral domain by a sixth loss function; The fourth loss function is obtained by weighting the multiple second attention weights and the multiple sixth loss functions respectively.

16. The method according to claim 1, characterized in that, The method further includes: Obtain the ninth image; The ninth image is input into the screen removal model, and the tenth image is output. The tenth image is the image obtained by removing screen patterns from the ninth image through the screen removal model. Based on the descratization feedback of the tenth image, the descratization model is updated.

17. A training device for a destigmatism model, characterized in that, The device includes: The shooting module is used to capture a first image and a second image displayed on the display screen respectively to obtain a third image and a fourth image. The first image is a positioning plate image without screen patterns, and the second image is an image obtained by superimposing the first target image without screen patterns on the first image. The third image and the fourth image are images with screen patterns. The determining module is used to determine a target mapping matrix based on the first image, the second image, the third image, and the fourth image, wherein the target mapping matrix is ​​used to represent the mapping relationship between the pixels of the first image and the pixels of the third image; The first screenshot module is used to capture the fifth image and the sixth image respectively from the third image and the fourth image based on the target mapping matrix. The fifth image and the sixth image are real-shot screen images of the first image and the second image, respectively. The second screenshot module is used to take a screenshot from the fifth image based on the fifth image and the sixth image to obtain a first sample image, wherein the first sample image is a first target image with screen patterns; The training module is used to train the model based on the first target image and the first sample image to obtain the screen pattern removal model.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the training method for the descratching model as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is executed by a processor to implement the training method for the screen-texture removal model as described in any one of claims 1 to 16.

20. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the training method for the screen pattern removal model as described in any one of claims 1 to 16.

Images