Image processing method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present invention relate to the field of computer vision. Disclosed are an image processing method and apparatus. The method comprises: obtaining an image to be processed, and performing position detection and image segmentation on said image to obtain a segmented image of the image to be processed, wherein the segmented image is used for indicating an area to be processed in the image to be processed, and said area comprises a redundant mark to be removed; performing image processing on the segmented image to obtain a mask image, wherein an area to be processed in the mask image corresponds to the area to be processed in the segmented image; using a deep learning algorithm and an attention mechanism to perform feature extraction on the segmented image and the mask image to obtain an output image; and performing image fusion on the output image and the image to be processed to obtain a resultant image in which the redundant mark has been removed. The present invention solves the problems in the prior art of complex operation and poor image processing effect.

Description

Image processing method, device, electronic device and storage medium Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Nowadays, many documents require digital feature extraction or reconstruction, such as document OCR text recognition, table reconstruction, etc. Many of these document images contain content such as seals, which will seriously affect OCR recognition, digital reconstruction and other operations, and seriously interfere with feature extraction, greatly reducing the recognition rate. Therefore, it is very important to remove the seals before operating on the image.

[0003] Image stamp removal is an image processing technique used to eliminate or reduce unwanted embedded objects, such as stamps, from images. It is very common in computer vision and has broad applications in image editing, document processing, privacy protection, and data analysis. Traditional image stamp removal methods primarily rely on techniques such as pixel replacement, filling, and texture synthesis. These methods often require manual labeling or additional information from the user to guide the stamp removal process.

[0004] In recent years, deep learning methods have been able to learn seal features from large amounts of labeled data and automatically remove them, reducing manual user intervention. However, deep learning methods also face challenges such as large data requirements and complex model training. Existing algorithms still have many problems, such as high dataset requirements, the need to annotate seal locations, and poor results obtained with large-scale images as input. Furthermore, methods such as generative adversarial networks (GANs) also suffer from unstable training and cumbersome parameter adjustment.

[0005] Therefore, there is an urgent need for an image processing method that is simple to operate and has good image processing effects.

[0006] Summary of the Invention

[0007] The embodiments of the present invention provide an image processing method, apparatus, electronic device, and storage medium to address the problems of complex operations and poor image processing effects in related technologies. The technical solutions are as follows:

[0008] According to one aspect of the present invention, an image processing method includes: acquiring an image to be processed, performing position detection and image segmentation on the image to be processed, and obtaining a segmented image of the image to be processed; the segmented image is used to indicate the area to be processed in the image to be processed; the area to be processed contains redundant marks to be removed; performing image processing on the segmented image to obtain a mask image; the area to be processed in the mask image corresponds to the area to be processed in the segmented image; using a deep learning algorithm and an attention mechanism to extract features from the segmented image and the mask image to obtain an output image; performing image fusion on the output image and the image to be processed to obtain a result image in which the redundant marks are removed.

[0009] In one embodiment, position detection and image segmentation are performed on the image to be processed to obtain a segmented image of the image to be processed by the following steps: using a trained detection model, the position of the area to be processed in the image to be processed is detected to obtain position information of the area to be processed in the image to be processed; and image segmentation is performed on the area to be processed in the image to be processed according to the position information to obtain a segmented image of the image to be processed.

[0010] In one embodiment, image processing is performed on the segmented image to obtain a mask image by the following steps: the contour of the area to be processed is screened in the segmented image to obtain the position of the contour of the area to be processed in the segmented image; and image inversion processing is performed on the image to be processed according to the position of the contour in the segmented image to obtain the mask image.

[0011] In one embodiment, the segmented image and the mask image are subjected to feature extraction using a deep learning algorithm and an attention mechanism to obtain an output image by the following steps: multi-level feature extraction is performed on the segmented image using the deep learning algorithm to obtain an intermediate feature map; multi-level feature fusion is performed on the mask image and the intermediate feature map using the attention mechanism to obtain a fused feature map; multi-level feature extraction is performed on the fused feature map using the deep learning algorithm to obtain an output feature map; and regional position prediction is performed on the output feature map to obtain the output image.

[0012] In one embodiment, the mask image and the intermediate feature map are subjected to multi-level feature fusion through the attention mechanism to obtain a fused feature map by the following steps: the intermediate feature map of the current level is used as the key value K, and the mask image is used as the query value Q; attention calculation is performed on the mask image and the intermediate feature map according to the key value K and the query value Q to obtain a first feature map; the intermediate feature map is calculated to obtain a weight V, and the first feature map is weighted according to the weight V to obtain a second feature map; the first feature map and the second feature map are fused to obtain a next-level intermediate feature map; the next-level intermediate feature map and the mask image are continued to be feature-fused until the weight reaches a set threshold to obtain the fused feature map.

[0013] In one embodiment, the image processing process for the image to be processed is implemented by calling an image processing model, and the image processing model is a trained machine learning model with the ability to perform image processing on the image to be processed.

[0014] In one embodiment, an image processing model includes a detection model, a feature extraction module, and an attention module; the training process of the image processing model is implemented by the following steps: obtaining a training set; the training set includes a plurality of training images, and the training images are images containing different redundant labels;

[0015] The current training image in the training set is input into the machine learning model for training to obtain a loss value; if the loss value meets the convergence condition, the training is completed to obtain the image processing model; otherwise, the model parameters of the machine learning model are updated, and other training images in the training set are obtained and input into the machine learning model to continue training until the loss value meets the convergence condition.

[0016] According to one aspect of the present invention, an image processing device includes: an image segmentation module for acquiring an image to be processed, performing position detection and image segmentation on the image to be processed, and obtaining a segmented image of the image to be processed; the segmented image is used to indicate a region to be processed in the image to be processed; the region to be processed includes redundant marks to be removed; a mask acquisition module for performing image processing on the segmented image to obtain a mask image; the region to be processed in the mask image corresponds to the region to be processed in the segmented image; and an image processing module for performing feature extraction on the segmented image and the mask image using a deep learning algorithm and an attention mechanism to obtain an output image;

[0017] The result acquisition module is used to perform image fusion on the output image and the image to be processed to obtain a result image after the redundant marks are removed.

[0018] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the image processing method described above.

[0019] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, wherein the computer-readable instructions are executed by one or more processors to implement the image processing method described above.

[0020] According to one aspect of the present invention, a computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the image processing method described above.

[0021] The beneficial effects brought about by the technical solution provided by the present invention are:

[0022] In the above technical solution, the image to be processed is first obtained, and then the position detection and image segmentation of the image to be processed are performed to obtain a segmented image of the image to be processed, and then the segmented image is processed to obtain a mask image, and then the deep learning algorithm and the attention mechanism are used to extract features of the segmented image and the mask image. Then, a high-quality output image can be obtained according to the position of the seal through the mask image and deep learning. At the same time, the strength of seal removal can be improved through the attention mechanism. Finally, the output image is fused with the image to be processed to obtain a result image in which redundant marks are removed. Therefore, the present invention is simple to operate and can ensure the image processing effect and the quality of the result image, thereby effectively solving the problems of complex operation and poor image processing effect existing in the related technology.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0025] FIG1 is a schematic diagram of an implementation environment according to the present invention;

[0026] FIG2 is a flow chart showing an image processing method according to an exemplary embodiment;

[0027] FIG3 is a flow chart of a training diffusion module in the embodiment corresponding to FIG2 ;

[0028] FIG4 is a schematic diagram of a color video obtained in the embodiment corresponding to FIG2 ;

[0029] FIG5 is a flow chart of an image processing method according to an application scenario;

[0030] FIG6 is a schematic diagram of an image processing method according to an application scenario;

[0031] FIG7 is a schematic diagram showing the results of an image processing method according to an application scenario;

[0032] FIG8 is a block diagram of an image processing apparatus according to an exemplary embodiment;

[0033] FIG9 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0034] Fig. 10 is a block diagram of an electronic device according to an exemplary embodiment.

[0035] The above-mentioned drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the present invention in any way, but rather to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0036] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0037] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0038] Deep learning technology: The concept of deep learning originates from the study of artificial neural networks. A multilayer perceptron with multiple hidden layers is an example of a deep learning architecture. Deep learning discovers distributed feature representations of data by combining lower-level features to form more abstract higher-level representations of attribute categories or features. Like machine learning methods, deep machine learning methods can be divided into supervised and unsupervised learning. The learning models developed under different learning frameworks vary significantly. For example, a convolutional neural network is a deep supervised machine learning model.

[0039] In the existing technology, the seal removal method based on generative adversarial network uses two independent classifiers to replace the original discriminant network, and combines the ResNet and Unet structures to construct a downsampling-refinement-upsampling generative network to generate images.

[0040] However, this method can only process local areas with seals, and cannot automatically detect the seal position for processing when the entire image is input. The effect of processing the entire image is very poor, the network training process is extremely unstable, the training and parameter adjustment steps are many, the operation is cumbersome, and the obtained seal removal effect is poor.

[0041] As can be seen from the above, the related technologies still have defects such as complex operations and poor image processing effects.

[0042] To this end, the present invention provides an image processing method that is simple to operate and provides excellent image processing results. This image processing method is suitable for use in document processing devices, which can be implemented as electronic devices integrating an image acquisition device and an image processing device. Such electronic devices include smartphones, desktop computers, laptop computers, servers, and the like. The image processing method in the embodiments of the present invention can be applied in a variety of scenarios, such as contract comparison and document processing.

[0043] Figure 1 is a schematic diagram of an implementation environment involved in an image processing method. It should be noted that this implementation environment is only an example adapted to the present invention and should not be considered as providing any limitation on the scope of application of the present invention.

[0044] The implementation environment includes a collection end 110 and a service end 130 .

[0045] Specifically, the acquisition end 110 can be considered as an image acquisition device, including but not limited to electronic devices with shooting functions such as cameras, cameras, and camcorders, and can also be considered as an integrated image acquisition and processing device, including but not limited to electronic devices with both shooting and image processing functions such as laptops equipped with cameras.

[0046] Server 130 can also be considered an image processing device, including but not limited to electronic devices with image processing capabilities, such as smartphones, desktop computers, laptop computers, and servers. It can also be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide background services, such as but not limited to image processing services.

[0047] A network communication connection is pre-established between the server 130 and the acquisition terminal 110 via a wired or wireless method, and data transmission between the server 130 and the acquisition terminal 110 is achieved via the network communication connection. The transmitted data includes but is not limited to: images to be processed, etc.

[0048] In one application scenario, the acquisition end 110 has both image acquisition and image processing functions. After the acquisition end 110 acquires the image to be processed, it calls the image processing service, performs position detection and image segmentation on the image to be processed, and obtains a segmented image of the image to be processed. The segmented image is then processed to obtain a mask image, and then a deep learning algorithm and an attention mechanism are used to extract features from the segmented image and the mask image. Finally, the output image is fused with the image to be processed to obtain a result image with redundant marks removed, which solves the problems of complex operation and poor image processing effect in related technologies.

[0049] Of course, in other application scenarios, image processing can also be achieved through the interaction between the acquisition end 110 and the service end 130.

[0050] In another application scenario, through the interaction between the acquisition terminal 110 and the server 130 , the acquisition terminal 110 acquires an image to be processed and uploads the image to be processed to the server 130 to request the server 130 to provide an image processing service.

[0051] At this time, for the server 130, after receiving the image to be processed uploaded by the acquisition end 110, it calls the image processing service, performs position detection and image segmentation on the image to be processed, obtains the segmented image of the image to be processed, and then performs image processing on the segmented image to obtain a mask image. Then, the deep learning algorithm and attention mechanism are used to extract features from the segmented image and the mask image. Finally, the output image is fused with the image to be processed to obtain the result image with redundant marks removed, so as to solve the problems of complex operation and poor image processing effect existing in related technologies.

[0052] Please refer to Figure 2. An embodiment of the present invention provides an image processing method, which is applicable to electronic devices. For example, the electronic device can be an electronic device with both shooting and image processing functions, such as a laptop computer equipped with a camera, or an electronic device with image processing functions, such as a smart phone, a desktop computer, a laptop computer, a server, etc.

[0053] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0054] As shown in FIG2 , the method may include the following steps:

[0055] Step 210 , obtaining an image to be processed, performing position detection and image segmentation on the image to be processed, and obtaining a segmented image of the image to be processed.

[0056] The segmented image is used to indicate the area to be processed in the image to be processed, and the area to be processed includes redundant marks to be removed. In this embodiment, the redundant marks can be seals. Of course, in other embodiments, the redundant marks can also be watermarks, etc., which is not a specific limitation here.

[0057] In one possible implementation, position detection and image segmentation of the image to be processed are performed using a detection model, which is a trained machine learning model capable of performing position detection and image segmentation of the area to be processed in the image to be processed. Specifically, the position of the area to be processed in the image to be processed is detected using the trained detection model to obtain position information of the area to be processed in the image to be processed; image segmentation is performed on the area to be processed in the image to be processed based on the position information to obtain a segmented image of the image to be processed. In this way, it is possible to automatically detect the seal of the entire image and automatically extract the seal portion for processing. Therefore, the present invention can accept images of any size and resolution, solving the problem that the prior art only allows manual input of the seal area.

[0058] In one possible implementation, the image to be processed is obtained by photographing and capturing the current environment using an image acquisition device, wherein the image acquisition device may be an electronic device with an image acquisition function, such as a smartphone equipped with a camera, a laptop computer, and the like.

[0059] It is understood that shooting can be a single shot or continuous shooting. In this case, for continuous shooting, a video can be obtained, and the image can be any frame in the video. For multiple shooting, multiple photos can be obtained, and the image can be any one of the multiple photos. In other words, the image in this embodiment can come from a dynamic image, such as multiple frames in a video or multiple photos, or from a static image, such as any frame in a video or any one of multiple photos. Accordingly, the image processing in this embodiment can be performed on a frame basis.

[0060] Step 230: Perform image processing on the segmented image to obtain a mask image.

[0061] Among them, the mask image and the segmented image correspond to each other. The mask image is used to distinguish the region of interest and the region of non-interest by opposite pixel values. After operating with the mask image, only the region of non-interest will be left in the obtained image. For example, the seal part to be removed is the region of interest, and the other parts are the region of non-interest. Therefore, the seal part can be removed by operating with the mask image.

[0062] In one possible implementation, step 230 includes the following steps:

[0063] Step S1 : screening the outline of the area to be processed in the segmented image to obtain the position of the outline of the area to be processed in the segmented image.

[0064] Step S2: performing image inversion processing on the image to be processed according to the position of the contour in the segmented image to obtain a mask image.

[0065] Among them, inversion refers to reversing the color phase of the image. The image is composed of pixels, and each pixel has an RGB color value. For example, if the RGB value of a pixel is 115, 220, and 120, then the inverted color RGB of the pixel is 110, 5, and 105. That is, the inverted color of a pixel or the entire image is 225 minus the RGB value of the original image.

[0066] In this embodiment, the inverse image of the redundant marks is obtained by acquiring the mask image, thereby preliminarily ensuring the accuracy of eliminating the redundant marks.

[0067] Step 250: Use a deep learning algorithm and an attention mechanism to perform feature extraction on the segmented image and the mask image to obtain an output image.

[0068] Specifically, as shown in FIG3 , step 250 may include the following steps:

[0069] Step 310: Perform multi-level feature extraction on the segmented image using a deep learning algorithm to obtain an intermediate feature map.

[0070] In one possible implementation, multi-level feature extraction of the segmented image is performed through a deep learning algorithm, which can be achieved through a feature extraction module in an image processing model.

[0071] In step 330 , the mask image and the intermediate feature map are subjected to multi-level feature fusion through the attention mechanism to obtain a fused feature map.

[0072] In one possible implementation, the mask image and the intermediate feature map are subjected to multi-level feature fusion through the attention mechanism to obtain a fused feature map, which can be implemented through the attention module in the image processing model.

[0073] In step 350 , a deep learning algorithm is used to perform multi-level feature extraction on the fused feature map to obtain an output feature map.

[0074] In one possible implementation, multi-level feature extraction is performed on the fused feature map through a deep learning algorithm to obtain an output feature map, which can also be implemented through a feature extraction module in an image processing model.

[0075] It is worth mentioning that the aforementioned method for performing position detection and image segmentation on the image to be processed to obtain the segmented image of the image to be processed can be implemented by a detection model trained in the image processing model.

[0076] The image processing model is a trained machine learning model capable of processing the image to be processed, wherein the machine learning module includes but is not limited to a convolutional neural network, a recurrent neural network, and a generative adversarial network. The training process of the image processing model may include the following steps:

[0077] Step a1: Obtain a training set.

[0078] The training set includes multiple training images, and the training images are images containing different redundant labels.

[0079] Step a2: Input the current training image in the training set into the machine learning model for training to obtain the loss value.

[0080] In step a3, if the loss value meets the convergence condition, the training is completed and the image processing model is obtained.

[0081] In step a4, if the loss value does not meet the convergence conditions, the model parameters of the machine learning model are updated, and other training images in the training set are obtained to input into the machine learning model to continue training.

[0082] Until the loss value meets the convergence condition, the model parameters of the machine learning model are stored, and the trained image processing model is obtained. At this time, the image processing model can perform image processing on the image to be processed to obtain a result image with redundant labels removed.

[0083] By training the machine learning model, an image processing model including a detection model, a feature extraction module and an attention module was obtained. This solved the problem of poor processing of the entire document image containing the seal, reduced the requirements for training and the required data set, and did not require position labeling. All pre-operation parts did not need to be labeled, and the program could be called to generate them. The training was simple and stable, without the need for tedious parameter adjustment. It had a good seal removal effect for input images of any size, and the model training process was simple and stable.

[0084] Step 370: perform region position prediction on the output feature map to obtain an output image.

[0085] Among them, the output image refers to the segmented image with the seal area removed. The output image is obtained by performing feature extraction on the segmented image and the mask image to predict the area position of the non-seal area in the segmented image, retaining the non-seal area in the segmented image and removing the seal area in the segmented image.

[0086] In this way, a high-quality output image is obtained by combining the mask image with a deep learning algorithm and an attention mechanism. This method is simple to operate while ensuring the image processing effect and the quality of the result image, thereby effectively solving the problems of complex operation and poor image processing effect in related technologies.

[0087] Step 270: perform image fusion on the output image and the image to be processed to obtain a result image with redundant marks removed.

[0088] After obtaining the output image with the seal area removed, it is fused with the image to be processed. The part with the seal removed can be fused with the part of the original image to be processed that does not contain the seal, thereby obtaining a result image with redundant marks removed.

[0089] Through the above process, the embodiment of the present invention obtains the image to be processed, then performs position detection and image segmentation on the image to be processed to obtain a segmented image of the image to be processed, and then performs image processing on the segmented image to obtain a mask image, and then uses a deep learning algorithm and an attention mechanism to extract features from the segmented image and the mask image. Then, a high-quality output image can be obtained according to the position of the seal through the mask image and deep learning. At the same time, the strength of seal removal can be improved through the attention mechanism. Finally, the output image is fused with the image to be processed to obtain a result image in which redundant marks are removed. Therefore, the present invention is simple to operate and can ensure the image processing effect and the quality of the result image, thereby effectively solving the problems of complex operation and poor image processing effect existing in the related technology.

[0090] Referring to FIG. 4 , in an exemplary embodiment, step 330 may include the following steps:

[0091] In step 410 , the current level one intermediate feature map is used as the key value K, and the mask image is used as the query value Q.

[0092] Step 430 : Perform attention calculation on the mask image and the intermediate feature map according to the key value K and the query value Q to obtain a first feature map.

[0093] In step 450 , the intermediate feature map is calculated to obtain a weight V, and the first feature map is weightedly calculated according to the weight V to obtain a second feature map.

[0094] In step 470 , the first feature map and the second feature map are fused to obtain a next-level intermediate feature map.

[0095] Specifically, as shown in FIG5 , F l-1 Represents the features of the previous layer output, takes the current level intermediate feature map as the key value K, and the mask image as the query value Q, performs dot product operations on the segmented image and the mask image according to the key value K and the query value Q respectively, and then performs a common operation on the results of the two to obtain the first feature map, obtains the weight V according to the current level intermediate feature map, performs dot product weighted calculation on the first feature map according to the weight V, and obtains the second feature map, and performs a fusion process of adding the first feature map and the second feature map to obtain the next level intermediate feature map.

[0096] Step 490 , continue to perform feature fusion on the next-level intermediate feature map and the mask image until the weight reaches a set threshold, thereby obtaining a fused feature map.

[0097] Specifically, first, the current level intermediate feature map is used as the key value K, and the mask image is used as the query value Q. Then, according to the key value K and the query value Q, attention calculation is performed on the mask image and the intermediate feature map to obtain the first feature map. The intermediate feature map is then calculated to obtain the weight V, and the first feature map is weighted according to the weight V to obtain the second feature map. The first feature map and the second feature map are fused to obtain the next level intermediate feature map. Finally, the next level intermediate feature map and the mask image are continued to be feature fused until the features are fused to the last level of the model, that is, the fused feature map is obtained.

[0098] Through the above process, the embodiment of the present invention performs multi-layer feature extraction on the segmented image and the mask image through a deep learning algorithm and an attention mechanism. Then, the image features of the non-seal area in the image to be processed can be retained as much as possible to obtain a high-quality output image. The strength of seal removal is further improved through the attention mechanism. Finally, the output image is fused with the image to be processed to obtain a result image with redundant marks removed, thereby effectively improving the image processing effect and solving the problem of poor image processing effect existing in related technologies.

[0099] FIG6 and FIG7 are schematic diagrams of an image processing method in an application scenario. In this application scenario, an electronic device such as a laptop computer may need to perform image processing.

[0100] In Figure 6, the image to be processed, Document with Seal, is first obtained. Then, the detection model Seal Detector is used to perform position detection and image segmentation on the image to be processed. The target image GT Seal Segmentation is used to supervise the detection model to obtain Loss, so that the detection model converges to obtain the output result Seal Segmentation. The output result Seal Segmentation of the detection model is cropped to obtain a mask image Mask, which is then detected to obtain a segmented image Seal Area. The feature extraction module Feature Extraction is used to extract features from the segmented image Seal Area. The mask image Mask is added to the attention module Seal2Removal-attention for attention calculation. The feature extraction module Feature Extraction is used to extract features from the result of the attention calculation to obtain the output image Seal Removal Results with redundant marks removed. The target result image GT Seal Removal Results is used to supervise the image processing model to obtain Loss, so as to obtain a more accurate output image Seal Removal Results. Finally, the output image Seal Removal Results is fused with the image to be processed, to obtain the result image after redundant marks are removed.

[0101] FIG7 shows the stamp removal effect of an embodiment of the present invention, which shows outstanding performance in terms of removing traces and retaining details.

[0102] In the above application scenario, image processing is achieved which is both simple to operate and can ensure the image processing effect and the quality of the result image, thereby effectively solving the problems of complex operation and poor image processing effect existing in related technologies.

[0103] The following are embodiments of the apparatus of the present invention, which can be used to perform the image processing method of the present invention. For details not disclosed in the apparatus embodiments of the present invention, please refer to the method embodiments of the image processing method of the present invention.

[0104] Please refer to FIG8 , an image processing apparatus 800 is provided in an embodiment of the present invention.

[0105] The apparatus 800 includes but is not limited to: an image segmentation module 810 , a mask acquisition module 830 , an image processing module 850 and a result acquisition module 870 .

[0106] The image segmentation module 810 is used to obtain an image to be processed, perform position detection and image segmentation on the image to be processed, and obtain a segmented image of the image to be processed; the segmented image is used to indicate the area to be processed in the image to be processed; the area to be processed contains redundant marks to be removed;

[0107] The mask acquisition module 830 is used to perform image processing on the segmented image to obtain a mask image; the area to be processed in the mask image corresponds to the area to be processed in the segmented image;

[0108] An image processing module 850 is used to extract features from the segmented image and the mask image using a deep learning algorithm and an attention mechanism to obtain an output image;

[0109] The result acquisition module 870 is used to perform image fusion on the output image and the image to be processed to obtain a result image after redundant marks are removed.

[0110] It should be noted that the image processing provided in the above embodiments only uses the division of the above-mentioned functional modules as an example when performing image processing. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the image processing device will be divided into different functional modules to complete all or part of the functions described above.

[0111] In addition, the image processing apparatus and the image processing method provided in the above embodiments belong to the same concept, wherein the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0112] Fig. 9 is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment. The electronic device is applicable to the acquisition terminal 110 in the implementation environment shown in Fig. 1 .

[0113] It should be noted that the electronic device is only an example adapted for the present invention and should not be considered to provide any limitation on the scope of use of the present invention. The electronic device should not be interpreted as needing to rely on or necessarily having one or more components in the exemplary electronic device 2000 shown in FIG9 .

[0114] The hardware structure of the electronic device 2000 may vary greatly due to different configurations or performances. As shown in FIG9 , the electronic device 2000 includes a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0115] Specifically, the power supply 210 is used to provide operating voltage for various hardware devices on the electronic device 2000 .

[0116] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices, for example, interacting between the acquisition terminal 110 and the service terminal 130 in the implementation environment shown in FIG1 .

[0117] Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, as shown in FIG9 , which is not specifically limited here.

[0118] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system 251, application 253 and data 255, etc. The storage method can be temporary storage or permanent storage.

[0119] Among them, the operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000 to enable the central processing unit 270 to calculate and process the massive data 255 in the memory 250. It can be Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0120] The application 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251. It may include at least one module (not shown in FIG9 ), each of which may include computer-readable instructions for the electronic device 2000. For example, an image processing device may be considered as an application 253 deployed on the electronic device 2000.

[0121] The data 255 may be photos, pictures, etc. stored in a disk, or may be image processing model parameters, etc. stored in the memory 250 .

[0122] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on the massive amount of data 255 in the memory 250. For example, the image processing method is performed by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0123] In addition, the present invention can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or combination thereof.

[0124] Referring to FIG. 10 , an embodiment of the present invention provides an electronic device 4000 , which may include an image acquisition device having an image acquisition function, etc.

[0125] In FIG. 10 , the electronic device 4000 includes at least one processor 4001 and at least one memory 4003 .

[0126] Data exchange between the processor 4001 and the memory 4003 can be achieved via at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. The communication bus 4002 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG10 shows only one thick line, but this does not mean that there is only one bus or one type of bus.

[0127] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0128] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0129] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to these.

[0130] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0131] The computer-readable instructions are executed by one or more processors 4001 to implement the image processing methods in the above-mentioned embodiments.

[0132] In addition, an embodiment of the present invention provides a storage medium having computer-readable instructions stored thereon. The computer-readable instructions are executed by one or more processors to implement the image processing method described above.

[0133] In an embodiment of the present invention, a computer program product is provided. The computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the image processing method described above.

[0134] Compared with the related art, the present invention has the following beneficial effects:

[0135] 1. The present invention first obtains an image to be processed, then performs position detection and image segmentation on the image to be processed to obtain a segmented image of the image to be processed, then performs image processing on the segmented image to obtain a mask image, and then uses a deep learning algorithm and an attention mechanism to extract features from the segmented image and the mask image. Then, a high-quality output image can be obtained according to the position of the seal through the mask image and deep learning. At the same time, the strength of seal removal can be improved through the attention mechanism. Finally, the output image is fused with the image to be processed to obtain a result image in which redundant marks are removed. Therefore, the present invention is simple to operate and can ensure the image processing effect and the quality of the result image, thereby effectively solving the problems of complex operation and poor image processing effect existing in the related art.

[0136] 2. The present invention provides an algorithm that includes seal position detection and seal removal. It does not require high-quality data sets, seal labeling, and other issues. It has a good seal removal effect for input images of any size, and the model training process is simple and stable.

[0137] 3. The present invention solves the problem of poor effect in processing the entire document image containing the seal, reduces the requirements for training and the required data set, does not require position marking, and all pre-operation parts do not need to be marked. It can be generated by calling the program. The training is simple and stable, and there is no need for a tedious parameter adjustment process; it improves the effect of seal removal, and shows outstanding performance in terms of removed traces and retained details.

[0138] 4. The present invention can remove seals from images of any size, automatically detect seals on the entire image, and automatically extract the seal part for processing. Therefore, it can accept images of any size and resolution, solving the problem in the prior art that only the seal area can be input.

[0139] 5. The present invention has a wide range of application scenarios. Digital document processing: Many documents (such as contracts, certification documents, etc.) usually need to be stamped. The present invention can be applied to scanned or photographed document images to remove the stamps and restore the original appearance of the document; Historical document research: When studying historical documents or ancient books, the present invention can help scholars restore the text so that they can better read and analyze the content; Digital image restoration: The present invention can be used to repair old photos or images, restore the original content, and improve image quality; Anti-piracy applications: In the field of digital media, such as e-books, digital music or movies, copyright protection is important. Some digital content may be attached with watermarks or logos to prevent piracy or unauthorized dissemination. The present invention can help legitimate users remove watermarks for personal use.

[0140] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0141] The above descriptions are only partial embodiments of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed, perform position detection and image segmentation on the image to be processed, and obtain a segmented image of the image to be processed; the segmented image is used to indicate a region to be processed in the image to be processed; the region to be processed contains redundant marks to be removed; Performing image processing on the segmented image to obtain a mask image; the area to be processed in the mask image corresponds to the area to be processed in the segmented image; Using a deep learning algorithm and an attention mechanism to extract features from the segmented image and the mask image to obtain an output image; The output image is fused with the image to be processed to obtain a result image in which the redundant marks are removed.

2. The method according to claim 1, characterized in that The performing position detection and image segmentation on the image to be processed to obtain a segmented image of the image to be processed includes: Using the trained detection model, detecting the position of the area to be processed in the image to be processed to obtain position information of the area to be processed in the image to be processed; Image segmentation is performed on the area to be processed in the image to be processed according to the position information to obtain a segmented image of the image to be processed.

3. The method according to claim 1, characterized in that The step of performing image processing on the segmented image to obtain a mask image comprises: Screening the contour of the area to be processed in the segmented image to obtain the position of the contour of the area to be processed in the segmented image; The image to be processed is subjected to image inversion processing according to the position of the contour in the segmented image to obtain the mask image.

4. The method according to claim 1, characterized in that The method of using a deep learning algorithm and an attention mechanism to extract features from the segmented image and the mask image to obtain an output image includes: Performing multi-level feature extraction on the segmented image by using the deep learning algorithm to obtain an intermediate feature map; Through the attention mechanism, the mask image and the intermediate feature map are subjected to multi-level feature fusion to obtain a fused feature map; Performing multi-level feature extraction on the fused feature map through the deep learning algorithm to obtain an output feature map; Performing region position prediction on the output feature map to obtain the output image.

5. The method according to claim 4, characterized in that The mask image and the intermediate feature map are subjected to multi-level feature fusion through the attention mechanism to obtain a fused feature map, including: The intermediate feature map of the current level is used as the key value K, and the mask image is used as the query value Q; Performing attention calculation on the mask image and the intermediate feature map according to the key value K and the query value Q to obtain a first feature map; Calculating the intermediate feature map to obtain a weight V, and performing weighted calculation on the first feature map according to the weight V to obtain a second feature map; The first feature map and the second feature map are fused to obtain a next-level intermediate feature map; Continue to perform feature fusion on the next-level intermediate feature map and the mask image until the weight reaches a set threshold, thereby obtaining the fused feature map.

6. The method according to any one of claims 1 to 5, characterized in that: The image processing process for the image to be processed is implemented by calling an image processing model, and the image processing model is a trained machine learning model with the ability to perform image processing on the image to be processed.

7. The method according to claim 6, characterized in that The image processing model includes a detection model, a feature extraction module and an attention module; The training process of the image processing model includes: Acquire a training set; the training set includes a plurality of training images, and the training images are images containing different redundant labels; Inputting the current training image in the training set into the machine learning model for training to obtain a loss value; If the loss value meets the convergence condition, the training is completed and the image processing model is obtained; otherwise, the model parameters of the machine learning model are updated, and other training images in the training set are obtained and input into the machine learning model to continue training until the loss value meets the convergence condition.

8. An image processing device, characterized in that: The device comprises: An image segmentation module is used to obtain an image to be processed, perform position detection and image segmentation on the image to be processed, and obtain a segmented image of the image to be processed; the segmented image is used to indicate a region to be processed in the image to be processed; the region to be processed contains redundant marks to be removed; A mask acquisition module, used for performing image processing on the segmented image to obtain a mask image; the area to be processed in the mask image corresponds to the area to be processed in the segmented image; An image processing module, used for extracting features from the segmented image and the mask image using a deep learning algorithm and an attention mechanism to obtain an output image; The result acquisition module is used to perform image fusion on the output image and the image to be processed to obtain a result image after the redundant marks are removed.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors so that the electronic device implements the image processing method according to any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the image processing method according to any one of claims 1 to 7.

Images