Method for removing shadow of document image and related device

Abstract

The invention provides a shadow removing method for a document image and a related device, and relates to the field of image processing. The method comprises the steps of obtaining a first document image comprising a shadow area, calling a shadow estimation model to process the first document image to obtain a corresponding estimated shadow image, and removing the shadow area in the first document image based on the estimated shadow image to obtain a second document image. The shadow estimation model is obtained through iterative training of a model loss function comprising a shadow loss function, and the shadow loss function comprises estimation of regression loss, total variation loss and color loss between the shadow image and the corresponding actual shadow area. In the training process, the color loss of the related shadow region is increased for supervised training, so that the estimated shadow image output by the shadow estimation model does not comprise dark color elements of the non-shadow region, the dark color elements of the non-shadow region are not removed as the shadow region, and the shadow region in the color document image can be effectively removed.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a document image shadow removal method and related devices. Background Art

[0002] With the rapid development of information technology and the growing demand for digitalization of office and study, digital document images are gradually replacing original physical documents, which can improve the efficiency of daily office and study. In addition, compared with physical documents, document images do not need to occupy physical space, are easy to manage, and are quick to process. There are many ways to obtain document images. For example, you can use the document scanning function of a mobile phone to digitize a physical document to obtain the corresponding document image. However, in the process of generating document images using the document scanning function of a mobile phone, due to the shooting environment (for example, single light source, obstruction, insufficient light, etc.), human factors, etc., part of the generated document image may be covered by shadows, which affects the reading of the document and the recognition of text information and various structured elements in the document.

[0003] In order to solve the problem of partial areas of document images being covered by shadows, current technologies are mainly divided into traditional algorithms and deep learning methods. Among them, traditional algorithms generally use water filling algorithms, which iteratively diffuse from the edge of the image to the center of the image and estimate the shadow by simulating the process of water filling, and use the estimated shadow to remove the shadow contained in the document image; deep learning methods generally use a single document image shadow removal deep learning network (Background Estimation Document Shadow Removal Network, BEDSR-Net) for processing. First, the background estimation module extracts the global background color of the document image and the spatial distribution information of the background and non-background pixels, and then realizes the shadow removal of the document image based on the global background color and spatial distribution information.

[0004] In current technology, traditional algorithms or deep learning methods for removing shadows from document images are suitable for non-color document images, but for color document images, current existing methods cannot effectively remove shadows from color document images. Summary of the invention

[0005] The present application provides a document image shadow removal method and related device, which can effectively remove the shadow area in the color document image by using the estimated shadow image output by the trained shadow estimation model, and can effectively retain the elements of the non-shadow area in the document image.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] The first aspect of the present application provides a method for removing shadows from a document image, the method comprising: obtaining a first document image including a shadow area, calling a shadow estimation model to process the first document image, obtaining a corresponding estimated shadow image, and removing the shadow area in the first document image based on the estimated shadow image to obtain a second document image. The shadow estimation model is obtained by iteratively training a model loss function including a shadow loss function, the shadow loss function including a regression loss, a total variation loss, and a color loss between the estimated shadow image and the corresponding actual shadow area. Since the color loss of the shadow area of ​​the document image is added to supervise the shadow estimation model during the training process of the shadow estimation model, the shadow estimation model will not identify the dark elements in the non-shadow area as the shadow area, and output an estimated shadow image that does not contain the dark elements in the non-shadow area, so that the dark elements in the non-shadow area in the first document image will not be removed as the shadow area, and the shadow area in the color document image can be effectively removed.

[0008] In one possible implementation, the color loss between the estimated shadow image output by the shadow estimation model and the corresponding actual shadow area image is obtained in the following manner: calculating a calculated color map of the estimated shadow image and a calculated color map of the shadow area image; calculating the regression loss between the calculated color map of the estimated shadow image and the calculated color map of the shadow area image to obtain the color loss between the estimated shadow image and the corresponding actual shadow area image.

[0009] In one possible implementation, the shadow estimation model includes a re-parameterized structure. The re-parameterized structure can ensure that the training resources are abundant and the structure constructed during the training process is large, so that the model can have better performance while ensuring that fewer resources are consumed during the reasoning process.

[0010] In a possible implementation, before acquiring the first document image, the method further includes: preprocessing the shadow estimation model, the preprocessing including one or more of the following: reparameterization processing, output removal processing, and model conversion processing. The structure of the shadow estimation model can be reduced, the parameters of the shadow estimation model are reduced, thereby reducing memory usage and improving the processing speed of the shadow estimation model.

[0011] In a possible implementation, after acquiring the first document image, the method further includes: performing normalization processing on the first document image to obtain a processed first document image; thereby calling the estimated shadow image to process the processed first document image to obtain an estimated shadow image corresponding to the first document image. This is useful for improving the processing performance of the shadow estimation model.

[0012] In a possible implementation, when the size of the first document image is inconsistent with the preset size, before calling the shadow estimation model to process the first document image and obtaining the estimated shadow image corresponding to the first document image, the size of the first document image is adjusted to the preset size to obtain the first image, wherein the preset size is the required size of the input image of the shadow estimation model, the shadow estimation model is called to process the first image to obtain the second image, and the size of the second image is adjusted to the size of the first document image, thereby obtaining the estimated shadow image corresponding to the first document image. Since the shadow area has less high-frequency information (i.e., less information / pixels of grayscale changes), for the shadow area image, enlarging or reducing the shadow area will hardly affect the content, so the training and reasoning of the resolution in the shadow estimation model can be reduced, effectively reducing the difficulty of training and processing, and will not affect the de-shadowing effect of the document image.

[0013] In a possible implementation, the model loss function also includes a shadow removal loss function, which includes: a regression loss between the target document image output by the shadow estimation model and the corresponding actual document image without shadows. Thus, the shadow estimation model can output the target document image with the shadow area removed, which can improve the effectiveness of removing the shadow area.

[0014] In a possible implementation, the method further includes: calling a shadow estimation model to process the first document image to obtain a target document image corresponding to the first document image, and obtaining a final second document image based on the second document image and the target document image. The target document image output by the shadow estimation model is added to optimize the effect of removing the shadow area of ​​the document image, thereby improving the effectiveness of removing the shadow area.

[0015] In a possible implementation, the model loss function also includes: a restoration loss function; the restoration loss function includes: a regression loss between a restored document image and a corresponding actual document image with shadows; wherein the restored document image is synthesized by an estimated shadow image output by the shadow estimation model and a target document image output by the shadow estimation model. Adding a restoration loss function can further supervise the training of the shadow estimation model and improve the accuracy of the shadow estimation model.

[0016] In a possible implementation, the structure of the shadow estimation model is a symmetric codec structure, which can ensure the amount of information output by the model and is more suitable for completing the underlying visual task of removing the shadow area of ​​the document image.

[0017] In one possible implementation, the regression loss is Huber loss, which combines the advantages of L1 loss and L2 loss and avoids the disadvantageous loss function as much as possible, reduces the impact of outliers on training, enhances the robustness of outliers, and reduces the sensitivity to outliers.

[0018] In a second aspect, the present application provides an electronic device, which includes a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the method of the first aspect above.

[0019] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed, the method of the first aspect described above is implemented.

[0020] In a fourth aspect, the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 An exemplary diagram of the composition of an electronic device provided in an embodiment of the present application;

[0022] Figure 2 An example diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of a shadow estimation model provided in an embodiment of the present application;

[0024] Figure 4A A structural diagram of a re-parameterization module during training provided in an embodiment of the present application;

[0025] Figure 4B A structural diagram of the reasoning of a re-parameterized module provided in an embodiment of the present application;

[0026] Figure 5A A flowchart of a shadow estimation model training method provided in an embodiment of the present application;

[0027] Figure 5B A flowchart of a method for obtaining a training data set provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of a process for removing shadows from a document image provided in an embodiment of the present application;

[0029] Figure 7 A simplified schematic diagram of the process of a document image shadow removal method provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of the process of another method for removing shadows from a document image provided in an embodiment of the present application;

[0031] Fig. 9 A schematic flow chart of another method for removing shadows from a document image provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of the process of another method for removing shadows from a document image provided in an embodiment of the present application;

[0033] Fig.11 A flowchart of another method for removing shadows from a document image provided in an embodiment of the present application

[0034] Fig. 12A A document image with a shadow area to be removed provided in an embodiment of the present application;

[0035] Fig. 12B A document image after removing the shadow area provided in an embodiment of the present application;

[0036] Fig.13A Another document image to be provided in an embodiment of the present application with a shadow area to be removed;

[0037] Fig. 13B Another document image after removing the shadow area is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0039] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0040] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given:

[0041] Grad-CAM (Gradient-weighted Class Activation Map) refers to the generation of a class activation heat map for an input image. It is a two-dimensional feature score network associated with a specific output category, and each position of the grid represents the importance of the category.

[0042] Convolutional Neural Networks (CNN) is a deep learning model or a multi-layer perceptron similar to an artificial neural network, which is often used to analyze visual images. It can effectively solve the shortcomings of fully connected neural networks: easy loss of spatial information, too many parameters and low efficiency, difficult training, and even easy to cause network overfitting.

[0043] Atrous Convolution is a method to increase the receptive field. Atrous convolution is used to solve the problem that in a fully convolutional neural network, the size of the output image must be consistent with the size of the input image and upsampling is required. However, since pooling operations are used in convolutional neural networks to increase the receptive field while reducing the resolution, upsampling cannot restore the loss of some detail information caused by pooling.

[0044] The advantages of a document image shadow removal method provided in an embodiment of the present application are compared and explained below in combination with the implementation method of removing the shadow area of ​​a document image in the current technology.

[0045] In one implementation, the shadow area of ​​a document image is removed based on a traditional algorithm, wherein a water filling algorithm is a traditional algorithm with a good effect in removing the shadow area of ​​a document image. The water filling algorithm is used as an example to introduce a method of removing the shadow area of ​​a document image using a traditional algorithm in the current technology. The water filling algorithm is to iteratively diffuse from the edge of the image to the center of the image and estimate the shadow by simulating the process of water filling, and use the estimated shadow to remove the shadow contained in the document image. The method of removing the shadow area of ​​a document image based on the water filling algorithm has a good effect on non-color document images. However, the removal of the shadow area of ​​a document image based on the water filling algorithm requires entering from the edge of the document image, and the dark elements at the edge of the color document image are easily regarded as the shadow area and removed, resulting in the removal of the dark elements at the edge of the color document image, so that the color document image after removing the shadow area cannot completely retain the elements of the non-shadow area in the document image, and the shadow area of ​​the color document image cannot be effectively removed.

[0046] In another implementation, the shadow area of ​​the document image is removed based on deep learning, wherein the most commonly used deep learning network for removing the shadow area of ​​the document image is the Background Estimation Document Shadow Removal Network (BEDSR-Net). Specifically, the removal of the shadow area of ​​the document image based on BEDSR-Net is mainly divided into two stages: in the first stage, the global background color of the document image is extracted through the background estimation module, and in the process of estimating the background color, the spatial distribution information of the background and shadow pixels in the document image is obtained, and the spatial distribution information is encoded into an attention map; in the second stage, the shadow removal module is used to remove the shadow area in the document image based on the estimated background color and the attention map, and a document image with the shadow area removed is obtained. The method for removing the shadow area of ​​the document image based on BEDSR-Net can achieve a better shadow removal effect for non-color document images or document images with a single background color. Furthermore, the attention in the first stage of BEDSR-Net Figure 1 It is generally obtained through Grad-CAM. Based on the estimated background color, the similarity between each pixel in the document image and the background is obtained, so as to obtain the spatial distribution information of the shadow and background pixels and present it in the form of a heat map. However, for color document images, the background color is rich and will be affected by other factors (for example: the light intensity when the document image is obtained). It is impossible to accurately obtain the attention map based on the estimated background color, and thus it is impossible to accurately obtain the shadow area, and thus it is impossible to effectively remove the shadow area of ​​the color document image.

[0047] In summary, the currently commonly used document image shadow area removal methods can effectively remove shadow areas for non-color document images, but cannot effectively remove shadow areas for color document images, and cannot even effectively retain the elements of the non-shadow areas in the document images.

[0048] In the present application, a shadow estimation model with a symmetric codec structure is used to realize the function of removing shadow areas of document images. Since the color loss of the shadow area of ​​the document image is added in the process of training the shadow estimation model (that is, the loss function between the calculated color map of the estimated shadow image output by the shadow estimation model and the calculated color map of the actual shadow area image), the training of the shadow estimation model is supervised so that the color distribution of the estimated shadow image output by the shadow estimation model is as close as possible to the color distribution of the actual shadow area image, thereby making the color distribution of the estimated shadow image output by the trained shadow estimation model closer to the actual situation, and will not identify the dark elements in the non-shadow area of ​​the document image to be removed as the shadow area, thereby will not remove the dark elements in the non-shadow area as the shadow area, and can effectively remove the shadow area in the color document image.

[0049] In some embodiments, the electronic device may be a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, etc. The present application does not impose any special restrictions on the specific form of the above electronic devices. In this embodiment, the structure of the electronic device may be as follows: Figure 1 As shown, Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0050] like Figure 1 As shown, the electronic device may include a processor 110, a display screen 120, a camera 130, a touch sensor 140, and an internal memory 150, etc.

[0051] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. For example: a first document image including a shadow area is processed using a shadow estimation model to obtain an estimated shadow image corresponding to the first document image, and based on the estimated shadow image, the shadow area of ​​the first document image is removed to obtain a second document image.

[0053] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0054] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0055] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0056] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 140, the charger, the flash, the camera 130, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 140 through the I2C interface, so that the processor 110 communicates with the touch sensor 140 through the I2C bus interface to realize the touch function of the electronic device.

[0057] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 120 and the camera 130. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 130 communicate via the CSI interface to implement the shooting function of the electronic device. The processor 110 and the display screen 120 communicate via the DSI interface to implement the display function of the electronic device.

[0058] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 130, the display 120, the wireless communication module, the audio module, the sensor module, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0059] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0060] The electronic device implements the display function through a GPU, a display screen 120, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 120 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0061] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 120, where N is a positive integer greater than 1.

[0062] A series of graphical user interfaces (GUIs) can be displayed on the display screen 120 of the electronic device, and these GUIs are the main screens of the electronic device. Generally speaking, the size of the display screen 120 of the electronic device is fixed, and only limited controls can be displayed on the display screen 120 of the electronic device. A control is a GUI element, which is a software component included in an application program, and controls all data processed by the application program and interactive operations on the data. Users can interact with the control through direct manipulation to read or edit relevant information of the application program. Generally speaking, a control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0063] The electronic device can realize the shooting function through the ISP, the camera 130, the video codec, the GPU, the display screen 120 and the application processor.

[0064] The ISP is used to process the data fed back by the camera 130. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 130.

[0065] The camera 130 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0066] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0067] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0068] NPU is a neural network (NN) computing processor. It can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize the intelligent functions of electronic devices, such as: shadow area removal of document images, image recognition, face recognition, voice recognition, text understanding, etc.

[0069] The touch sensor 140 is also called a "touch control device". The touch sensor 140 can be arranged on the display screen 120. The touch sensor 140 and the display screen 120 form a touch screen, also called a "touch control screen". The touch sensor 140 is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 120. In other embodiments, the touch sensor 140 can also be arranged on the surface of the electronic device, which is different from the position of the display screen 120.

[0070] The internal memory 150 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 150. For example, based on the estimated shadow image output by the shadow estimation model, the shadow area of ​​the first document image including the shadow area is removed. Among them, the program storage area can store an operating system, an application required for at least one function (such as removing the shadow area of ​​the document image, image scaling, etc.). The data storage area can store data created during the use of the electronic device (such as image data, audio data, etc.). In addition, the internal memory 150 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 150, and / or the instructions stored in the memory provided in the processor.

[0071] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.

[0072] The operating system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device.

[0073] Figure 2 It is a software structure block diagram of the electronic device according to an embodiment of the present application.

[0074] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer, and the kernel layer.

[0075] The application layer can include a series of application packages. Figure 2 As shown, the application package may include document scanning, gallery, memo, camera and other applications. Among them, document scanning specifically includes scanning a physical document into a corresponding document image, and can achieve a shadow removal effect on the document image; gallery, memo, camera can all be used as the entry for document scanning.

[0076] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0077] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0078] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0079] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0080] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0081] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0082] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0083] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0084] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0085] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0086] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0087] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0088] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0089] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0090] A 2D graphics engine is a drawing engine for 2D drawings.

[0091] The hardware abstraction layer is an abstract structure between the kernel layer and the upper layer. It is an encapsulation of the inner Linux driver and provides a unified interface for the upper layer. The upper layer application does not know the specific working implementation of the lower layer hardware and shields the implementation details of the lower layer. The hardware abstraction layer can include: camera module, image module, Bluetooth module, and wifi module, etc.

[0092] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0093] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

[0094] Combine the following Figure 3-4B , a deep learning model for removing shadow areas of document images provided in an embodiment of the present application is introduced in detail, that is, a shadow estimation model 300 provided in an embodiment of the present application.

[0095] Specifically, the shadow estimation model 300 adopts a symmetric codec structure based on a convolutional neural network (for example: U-Net structure, SegNet structure), and there is a jump connection between the codec. Among them, the codec structure actually refers to the entire deep learning model / network mainly composed of an encoder and a decoder. The encoder is mainly used to extract features from the input, while the decoder mainly performs further feature optimization and task processing on the features obtained from the input processed by the encoder. The symmetric codec structure based on the convolutional neural network can maintain the amount of information output by the model, and is more suitable for completing the low-level visual task of removing the shadow area of ​​the document image.

[0096] Combine the following Figure 3 The structure of a shadow estimation model 300 provided in an embodiment of the present application is introduced in detail.

[0097] Specifically, the shadow estimation model 300 includes a convolution part 310 , a down-sampling part 320 , an up-sampling part 330 , and an output part 340 .

[0098] The convolution part 310 is used to perform a convolution operation on the input first document image 30 , so as to extract features of the input first document image 30 .

[0099] The first document image 30 is a document image including a shadow area and is a document image that needs to have the shadow area removed.

[0100] Furthermore, the convolution part 310 has a uniform requirement for the size of the input first document image 30. Specifically, in the embodiment of the present application, the size of the input first document image 30 is specified to be 512*512 pixels.

[0101] Specifically, the convolution part 310 uses a convolution operation with a convolution step size of 2 and a convolution kernel size of 3*3 to extract features from the input first document image 30 .

[0102] Specifically, the downsampling part 320 is mainly composed of four stage modules (specifically, the four stage modules are the first stage module 321, the second stage module 322, the third stage module 323, and the fourth stage module 324). Downsampling processing is performed between each stage module, and downsampling processing is also performed between the convolution part 310 and the downsampling part 320.

[0103] Each stage module contains multiple basic modules, which are connected in series. The basic modules include residual modules, dense modules, anti-bottleneck modules, etc.

[0104] Specifically, the first-stage module 321 includes 2 basic modules, the second-stage module 322 includes 3 basic modules, the third-stage module 323 includes 3 basic modules, and the fourth-stage module includes 4 basic modules.

[0105] It should be noted that in the embodiment of the present application, the number of basic modules included in the four-stage modules in the upsampling part 320 of the shadow estimation model 300 is as shown above. In addition, the number of basic modules included in the four-stage modules can also be other numbers, which can be flexibly adjusted based on the number shown above according to actual conditions, and this application does not make specific limitations.

[0106] Furthermore, the basic module adopts a reparameterization module. The reparameterization module is used to achieve structural reparameterization. Structural reparameterization refers to first constructing a series of structures (generally used in the training process) and converting their parameters into another set of parameters (generally used for reasoning), thereby converting this series of structures into another series of structures. During use, it is necessary to ensure that the resources consumed during reasoning are small, and at the same time ensure that the performance during reasoning meets the requirements. Since the training resources are generally relatively abundant, the structure constructed during the training process is larger and can have better performance (for example: higher precision, accuracy, sparsity, etc.). During the reasoning process, the converted structure is smaller and retains the same performance as the structure during the training process.

[0107] Specifically, the shadow estimation model 300 adopts a re-parameterization module as a basic module. During the training process of the shadow estimation model 300, multiple branches are used in parallel to maintain a high learning ability. When the training of the shadow estimation model 300 is completed, the multiple branches are converted into one path, which may improve the speed of reasoning while ensuring the high accuracy of the trained shadow estimation model 300.

[0108] For ease of understanding, the following Figure 4A and Figure 4B The re-parameterization module provided in the embodiment of the present application is introduced in detail.

[0109] like Figure 4A The specific structure of the re-parameterization module during training shown in the figure includes k branches consisting of 3*3 convolutional layers and batch normalization layers, 1 branch consisting of 1*1 convolutional layers and batch normalization layers, and 1 branch consisting of only batch normalization layers. The outputs of all branches are added element by element, and then processed by the activation function to output the features of the layer. The activation function in the embodiment of the present application is a rectified linear unit (ReLU). Of course, other types of activation functions can also be used, and a suitable activation function can be selected according to the actual situation.

[0110] It should be noted that Figure 4A What is shown is only a structure of a re-parameterized module during training. In addition, other re-parameterized structures can be selected, and this application does not make specific limitations.

[0111] When the shadow estimation model 300 is trained, Figure 4A The multiple branches shown are converted into Figure 4B The path shown is used for the reasoning process of the shadow estimation model 300, that is, it is converted into a branch consisting of a 3*3 convolution layer and an activation function. Figure 4BThe branch shown in the figure consists of a 3*3 convolutional layer and an activation function. During the inference process, it can maintain the same Figure 4A The performance (e.g., accuracy) of the architecture shown is the same, but the speed at inference is greatly improved.

[0112] In the downsampling part 320 of the shadow estimation model 300, downsampling processing is performed between modules in each stage. Specifically, a mixed pooling method is used for downsampling processing.

[0113] Among them, the mixed pooling method replaces the conventional deterministic pooling operation with a random process, and randomly adopts the maximum pooling method and the average pooling method during model training. That is, the mixed pooling method changes the rules of pooling adjustment in the form of random selection, which solves the problems encountered by the maximum pooling method and the average pooling method to a certain extent. Mixed pooling can help prevent the phenomenon of network / model overfitting to a certain extent, improve precision (accuracy) by solving the overfitting problem, and the computational overhead required by the mixed pooling method is negligible, and its application range is relatively wide.

[0114] It should be noted that the embodiment of the present application adopts a hybrid pooling method for downsampling processing. In addition, other pooling methods can be selected for downsampling processing, such as random pooling, DPP pooling, etc. The pooling method can be selected according to actual usage, and this application does not make specific limitations.

[0115] Among them, the upsampling part 331 is mainly composed of a pooling module and three stage modules (specifically, the pooling module 331, the three stage modules are the fifth stage module 335, the sixth stage module 336, and the seventh stage module 337), the upsampling process is performed between the pooling module 331 and the fifth stage module 335, the upsampling process is performed between each stage module, and the upsampling process is also performed between the upsampling part 330 and the output part 340. And the features of the corresponding scale (level) in the downsampling part and the features extracted by the upsampling part are channel-connected in parallel, and then input into the next stage module for feature extraction.

[0116] Among them, the pooling module 331 adopts a spatial pooling method to improve the ability of the shadow estimation model 300 to extract multi-scale information. In order to solve the problem that the neural network needs to input an image of a fixed size, resulting in poor accuracy of the overall network, for example: the input of a certain neural network requires the image size to be 224*224, then no matter what the size of the input image is, it is necessary to adjust the size of the input image to 224*224 before inputting it into the neural network, resulting in deformation of the image, thereby causing the information contained in the image to be deformed, limiting the accuracy of the overall network. The entire process of spatial pooling has nothing to do with the size of the input, so it can arbitrarily process images of any size, and no matter what size the input image is, the output size is fixed, so that the network input original image does not need to be resized.

[0117] Furthermore, the spatial pooling method adopts the Atrous Spatial Pyramid Pooling (ASPP) method. ASPP uses multiple parallel atrous convolution layers with different sampling rates, processes the features extracted for each sampling rate in a separate branch, and fuses the features extracted by each branch to generate the final result. The ASPP method constructs convolution kernels with different receptive fields through different atrous rates to obtain multi-scale object information. It can capture information at different scales and has a significant improvement in estimating the shadow image of the document image and estimating the document image with the shadow area removed.

[0118] Each stage module contains multiple basic modules, which are connected in series. The basic modules include residual modules, dense modules, anti-bottleneck modules, etc.

[0119] Specifically, the fifth stage module 335 includes 2 basic modules, the sixth stage module 336 includes 2 basic modules, and the seventh stage module 337 includes 2 basic modules.

[0120] It should be noted that in the embodiment of the present application, the number of basic modules included in the three-stage modules in the downsampling part 330 of the shadow estimation model 300 is as shown above. In addition, the number of basic modules included in the three-stage modules can also be other numbers, which can be flexibly adjusted based on the number shown above according to actual conditions, and this application does not make specific limitations.

[0121] Furthermore, the base module among the three-stage modules in the upsampling part 330 of the shadow estimation model 300 is a re-parameterization module.

[0122] In the upsampling part 330 of the shadow estimation model 300, upsampling processing is performed between each stage module, and upsampling processing is performed between the seventh stage module 337 and the output part 340. Specifically, bilinear interpolation is used for upsampling processing.

[0123] Among them, bilinear interpolation, also known as bilinear interpolation, is based on the core idea of ​​performing linear interpolation in two directions, that is, using the four real pixel values ​​around the virtual point in the original image to jointly determine a pixel value in the target image, where the virtual point is obtained by inferring the pixel position of the target image in the original image. Based on the bilinear difference method, the upsampling process is smoother, the calculation is simple, and the consumption of computing resources is less.

[0124] It should be noted that in the embodiment of the present application, bilinear interpolation is used for upsampling. In addition, other interpolation methods, anti-pooling methods, transposed convolution methods, etc. can also be used for upsampling, which are not specifically limited in the present application. Among them, the output part 340 includes two output layer modules, namely a first output module 341 and a second output module 342. The output part 340 outputs the corresponding estimated shadow image 31 and the target document image 32 (that is, the document image after removing the shadow area) according to the features of the received extracted first document image 30.

[0125] The estimated shadow image 31 is a shadow area image output by the shadow estimation model 300 based on the first document image 30 and including information on the position distribution and color distribution of the shadow area.

[0126] The target document image 32 is a document image output by the shadow estimation model 300 based on the first document image 30 after removing the shadow area.

[0127] Specifically, the first output module 341 and the second output module 342 have the same structure, including a convolution with a convolution kernel size of 1*1 and a sigmoid activation function, and the number of output channels is 3.

[0128] In the neural convolution network / deep learning algorithm learning, it is usually necessary to set the input channel (in_channels) and the output channel (out_channels). The number of channels of the initially input image sample, that is, the number of input channels for the initial convolution process, depends on the type of image. For example, if the initially input image sample is a sample in RGB format, the number of input channels is 3. The number of output channels after the convolution operation depends on the number of convolution kernels, and the output channels after the convolution operation will also be used as the input channels of the convolution kernel of the next convolution. The number of input channels of the next convolution is the same as the number of output channels of the convolution operation.

[0129] Specifically, the number of channels for all operators in the shadow estimation model 300 is as follows: the number of input channels of the convolution part 310 is 3, because the format of the input first document image 30 is RGB format, and the number of output channels of the convolution part 310 is usually set to a multiple of 8. In the embodiment of the present application, the number of output channels of the convolution part 310 is set to 32, that is, the convolution part 310 includes 32 convolution kernels. In the downsampling part 310, the number of input channels and the number of output channels in the stage module immediately adjacent to each downsampling process are doubled. Since the downsampling process is performed in a mixed pooling manner, that is, the channels of the maximum pooling and the average pooling are connected in parallel, the number of output channels is doubled compared to the number of input channels. The pooling part 331 adopts a hollow space pooling pyramid method, so the number of input channels and the number of output channels of the pooling module 331 remain unchanged. In the upsampling part 340, the number of input channels in the stage module immediately adjacent to each upsampling process is half of that before the channels are connected in parallel. The number of output channels of the first output module 341 and the second output module 342 in the output part 340 is 3, that is, the pictures in RGB format are output.

[0130] Combine the following Figure 5A and Figure 5B A training method for a shadow estimation model 300 provided in an embodiment of the present application is introduced in detail.

[0131] S510 , loading the shadow estimation model 300 , and initializing parameters of the shadow estimation model 300 .

[0132] Parameter initialization refers to the process of initializing the weights and biases of each node before training the network model. Parameter initialization has a crucial impact on the convergence speed and performance of the model. Model training is to continuously iterate and update the weight parameters to achieve better performance. However, as the network depth (number of layers) increases, problems such as gradient vanishing or gradient exploding are very likely to occur during training. If any of these situations occur, the loss gradient is either too large or too small to flow backward favorably. If the network model can do this, it will take longer to converge. A good parameter initialization is very helpful in dealing with the problem of gradient vanishing or gradient exploding, which is beneficial to improving the convergence speed and performance of the network model.

[0133] Specifically, the He initialization method is used to initialize the parameters of the shadow estimation model 300.

[0134] Among them, He initialization, also known as Kaiming initialization, also known as MSRA initialization, corresponds to the nonlinear activation function (Relu and Prelu). The conditions for He initialization are: during forward propagation, the variance of the state value remains unchanged; during backward propagation, the variance of the gradient of the activation value remains unchanged. He initialization is a robust neural network parameter initialization method that can ensure that information can flow effectively during forward and backward propagation, so that the variance of input signals in different layers is roughly the same.

[0135] It should be noted that the parameter initialization method in the embodiment of the present application is He initialization. In addition to this, Xavier initialization can also be selected. The parameter initialization method can be selected in addition to all-0 initialization for parameter initialization, and this application does not make specific limitations.

[0136] S520: Obtain a training data set.

[0137] Specifically, firstly, a shadow area image and a shadow-free document image are obtained; the shadow area image and the shadow-free document image are enhanced and synthesized respectively to obtain a shadowed document image. Synthesized shadowed document image

[0138] Specific, combined Figure 5B The figure shows a flow chart of a method for obtaining a training data set provided in an embodiment of the present application.

[0139] S521. Obtain a shadow area image and a shadow-free document image.

[0140] The shadow-free document images include: shadow-free document images obtained by scanning documents with a scanner, electronic images of documents, real shadow-free document images taken with a camera, etc. The total number of shadow-free document images collected for training is generally more than 10,000, and the proportion of shadow-free document images of each category is required to be balanced as much as possible.

[0141] Among them, the shadow area image is obtained by collecting real shadows and synthetic shadows.

[0142] The real shadow collection scenes include: indoor soft light, indoor strong light, outdoor direct sunlight, outdoor no direct sunlight, etc. Specifically, in different scenes, the shadows generated by the occlusion of various shapes can be mapped on a pure white flat plate with only slow reflection and obtained by shooting with a camera.

[0143] Among them, the synthetic shadow can be generated by superimposing a randomly generated two-dimensional Gaussian superposition model, and on the basis of the synthetic shadow obtained by superimposing a randomly generated two-dimensional Gaussian superposition model, other arbitrary forms of shadows are superimposed to increase the complexity of the synthetic shadow, which is beneficial to the subsequent training of the shadow estimation model 300, so that the shadow estimation model 300 can have better performance.

[0144] S522: Enhance the shadow area image and the shadow-free document image respectively.

[0145] Specifically, for shadow-free document images, there are differences between the shadow-free document image (scanned document image) obtained by scanning the document with a scanner and the electronic version of the document (electronic document image) and the actual shadow-free document image, so a series of enhancement processing is performed on the scanned document image and the electronic document image to make the scanned document image and the electronic document image closer to the actual shadow-free document image, so that the shadow estimation model 300 subsequently trained based on the enhanced shadow-free document image has better performance and is more in line with the actual situation.

[0146] Among them, a series of enhancement processing performed on scanned document images and electronic document images includes but is not limited to: random cropping, random 90-degree rotation, resizing, adding noise, flipping, random scaling, random small-angle rotation, random translation, adding background, document distortion, adding occlusion, color jittering, contrast jittering, blurring, etc.

[0147] Specifically, for the shadow area image, there is a difference between the real shadow area image obtained by collecting the real shadow method and the synthetic shadow area image obtained by the synthetic shadow method and the shadow area of ​​the actual document image with a shadow area. Since the document image cannot be only a slow-reflecting pure white flat plate, it should be an image with text or pictures. Therefore, a series of enhancement processing is performed on the real shadow area image and the synthetic shadow area image to make the shadow area image closer to the actual shadow area image, so that the shadow estimation model 300 obtained by subsequent training has better performance and is more in line with the actual situation.

[0148] Among them, a series of enhancement processing performed on the shadow area image includes but is not limited to: random cropping, random 90-degree rotation, resizing, adding noise, flipping, random scaling, color jittering, etc.

[0149] S523: synthesize the enhanced shadow area image and the enhanced shadow-free document image to obtain a shadowed document image.

[0150] Specifically, the shadow area image is normalized, the values ​​of the pixels of the shadow area image are normalized to between 0 and 1, and the normalized shadow area image is multiplied pixel by pixel with the shadow-free document image to obtain the shadowed document image.

[0151] The synthesized document image with shadow, the enhanced shadow area image and the enhanced document image without shadow used for synthesis together constitute a set of training data.

[0152] S524: normalize the synthesized document image with shadow.

[0153] Among them, normalization: the goal of normalization is to find a certain mapping relationship to map the original data to the interval [a, b]. Generally, [a, b] will take the combination of [-1, 1], [0, 1]. Normalization of the synthesized shadowed document image is to adjust the eigenvalue size to a similar range. If normalization is not performed, if the eigenvalue of the image is large, the gradient value will also be large; if the eigenvalue of the image is small, the gradient value will also be small. When the model is back-propagated, the gradient value update is the same as the learning rate. When the learning rate is small, the small gradient value will lead to slow update. When the learning rate is large, the large gradient value will make the model difficult to converge. Therefore, in order to make the model training converge smoothly, the image is normalized and the eigenvalues ​​of different dimensions are adjusted to a similar range. A unified learning rate can be used to accelerate model training, which is beneficial to model training.

[0154] Specifically, the synthesized document image with shadows is normalized by using zero-mean normalization, which maps a set of data to a standard normal distribution with a mean of 0 and a standard deviation of 1, thereby eliminating the factors that lead to different results due to different distributions of various features.

[0155] Specifically, the values ​​of the pixels of the synthesized document image with shadows are first normalized to between 0 and 1, and then the value of each pixel is subtracted by 0.5 and multiplied by 4 to obtain a zero-mean normalization result.

[0156] It should be noted that in the embodiment of the present application, the synthesized shadowed document image is normalized by zero-mean normalization. In addition, the synthesized shadowed document image can also be normalized by maximum-minimum normalization, Z-Score normalization, standard deviation normalization, etc., and this application does not make specific limitations.

[0157] S530 , iteratively train the shadow estimation model 300 using the training data set.

[0158] Specifically, the shadow estimation model 300 is iteratively trained using the training data set based on the model loss function.

[0159] For ease of understanding, the following Figure 5A The specific steps of S530 shown in FIG. 5 are described below. S530 includes:

[0160] S531. Input the synthesized document image with shadow in the training data set into the shadow estimation model to obtain the estimated shadow image and the target document image output by the shadow estimation model.

[0161] S532. Determine a model loss function based on the estimated shadow image and the corresponding shadow area image, the target document image and the corresponding shadow-free document image, and the restored document image and the shadowed document image.

[0162] Among them, the model loss function is: L = L S +L I +L r .

[0163] Among them, L is the model loss function; L S is the shadow loss function; L I is the shadow removal loss function; L r is the recovery loss function.

[0164] For the convenience of description, in the embodiment of the present application, after the synthesized document image with shadow is input, the estimated shadow image output by the shadow estimation model 300 is assumed to be S', and the shadow area image corresponding to the input document image with shadow is assumed to be S (that is, the label of the estimated shadow image is the shadow area image corresponding to the input document image with shadow); the target document image output by the shadow estimation model 300 is assumed to be I', and the shadow-free document image corresponding to the input document image with shadow is assumed to be I (that is, the label of the target document image is the shadow-free document image corresponding to the input document image with shadow); the input synthesized document image with shadow is assumed to be I r .

[0165] The shadow loss function is the regression loss, total variation loss and color loss between the estimated shadow image and the corresponding shadow area image. The regression loss includes: L1 loss, L2 loss or Huber loss. For ease of understanding, the following regression loss is explained using Huber loss as an example.

[0166] Specifically, the shadow loss function is: L S =Huber(S,S')+TV(S,S')+Huber(C(S),C(S')).

[0167] Among them, Huber loss is Huber loss, also known as smoothed mean absolute error (smoothed L1 loss). It is a loss function that combines the advantages of mean absolute error (MAE, L1 loss) and mean squared error (MSE, L2 loss) and avoids the disadvantages as much as possible. Huber loss approaches L1 loss when the error is large; and approaches L2 loss when the error is small. Therefore, compared with L1 loss, Huber loss reduces the impact of outliers on training; compared with L2 loss, it enhances the robustness of outliers and reduces the sensitivity to outliers.

[0168] Specifically, Huber (S, S') is the Huber loss between the estimated shadow image output by the shadow estimation model 300 and the shadow area image of the shadowed document image used for synthesizing the input, and is used to supervise the training of the shadow estimation model 300 to output the estimated shadow image and the corresponding shadow area image as close as possible, so that the trained shadow estimation model 300 can accurately output the estimated shadow image.

[0169] Among them, TV loss is the total variation loss, and its purpose is to add some regularization terms to maintain the smoothness of the image during the training model. TV loss is a commonly used regularization term. By reducing TV loss, noise can be constrained to a certain extent, making the estimated shadow image output by the trained model smoother.

[0170] Specifically, TV(S,S') is the total variation loss between the estimated shadow image output by the shadow estimation model 300 and the shadow area image of the document image with shadow used for synthesizing the input, in order to make the shadow area image output by the trained shadow estimation model 300 smoother.

[0171] Wherein, C is the calculated color map, specifically the image itself divided by the mean of the image in each channel dimension (the average value of each pixel in each channel), thereby obtaining the calculated color map of the image. The Huber loss between the calculated color map C(S') of the estimated shadow image S' and the calculated color map C(S) of the corresponding shadow area image S can effectively enhance the color supervision between the estimated shadow image and the corresponding shadow area image, thereby reducing the color deviation of the estimated shadow image. As a result, the color distribution of the estimated shadow image output by the trained shadow estimation model 300 is closer to the actual situation, reducing the situation where the darker picture area in the color document image is identified as the shadow area, and is more suitable for color document images.

[0172] Specifically, Huber (C(S), C(S')) is the color loss between the estimated shadow image output by the shadow estimation model 300 and the shadow area image of the shadowed document image used for synthesizing the input, that is, the Huber loss between the calculated color map of the estimated shadow image S' and the calculated color map of the corresponding shadow area image S. It is used to supervise the training of the shadow estimation model 300 to ensure that the color distribution of the estimated shadow image output by the shadow estimation model 300 is as close as possible to the corresponding shadow area image, avoiding treating the dark elements at the edge of the document image as shadow areas, making the shadow estimation model 300 more suitable for color document images.

[0173] In summary, the shadow estimation model 300 is trained by supervising the shadow loss, so that the estimated shadow image output by the shadow estimation model 300 can be as close as possible to the real shadow area image, for example: color distribution, noise distribution, etc.

[0174] Among them, the de-shadowing loss function is the regression loss between the target document image and the corresponding shadow-free document image.

[0175] Specifically, the shadow removal loss function is: L I =Huber(I,I').

[0176] Specifically, Huber (I, I') is the Huber loss between the target document image output by the shadow estimation model 300 and the shadow-free document image used to synthesize the input shadowed document image, which is used to supervise the training of the shadow estimation model 300 to output the target document image and the corresponding shadow-free document image as close as possible, so that the trained shadow estimation model 300 can accurately output the target document image.

[0177] The restoration loss function is the regression loss between the restored document image and the shadowed document image.

[0178] The restored document image is a document image synthesized from the estimated shadow image output by the shadow estimation model 300 and the target document image.

[0179] Specifically, the restoration loss function is: L r = Huber(I r ,I r ').

[0180] Among them, I r I is the input synthetic shadow document image; r ' is a restored document image synthesized by the estimated shadow image S' output by the shadow estimation model 300 and the target document image I', that is, I r ' is the product of S' and I'.

[0181] Specifically, Huber (I r ,I r ') is the Huber loss between the estimated shadow image S' output by the shadow estimation model 300 and the target document image I', and the restored document image synthesized by the input shadowed document image, which is used to supervise the training that the restored document image synthesized by the estimated shadow image S' output by the shadow estimation model 300 and the target document image is as close as possible to the input shadowed document image.

[0182] Since the shadow loss function in the model loss function includes: the Huber loss (also called color loss) between the calculated color map of the estimated shadow image and the calculated color map of the actual shadow area image, the model loss function is used to supervise the training of the shadow estimation model. This can make the color distribution of the estimated shadow image output by the trained shadow estimation model closer to the actual situation, reduce the situation where darker image areas in color document images are identified as shadow areas, and is suitable for color document images.

[0183] Furthermore, the model loss function is used to simultaneously perform supervised training on the estimated shadow image, target document image, and restored document image synthesized from the estimated shadow image and the target document image output by the shadow estimation model. This can effectively utilize the semantic information of the shadow and non-shadow areas, thereby better improving the robustness of the shadow estimation model.

[0184] It should be noted that the Huber loss in the embodiment of the present application can be replaced by L1 loss, L2 loss, etc., which can be set according to actual conditions and is not specifically limited in the present application.

[0185] Furthermore, before training the shadow estimation model 300, relevant parameters of the optimizer are set.

[0186] Among them, the optimizer guides the various parameters of the loss function to update to the appropriate size in the correct direction during the back propagation process of deep learning / model training, so that the updated parameters make the value of the loss function continuously approach the global minimum.

[0187] Specifically, the optimizer uses the AdamW optimizer, and the initial learning rate is set to 0.0001.

[0188] It should be noted that in the embodiment of the present application, the optimizer adopts the AdamW optimizer. In addition, other optimizers can be selected according to actual conditions, such as: SGDM optimizer, Adam optimizer, etc., and this application does not make specific limitations.

[0189] Furthermore, since the shadow estimation model 300 has a uniform requirement for the size of the input first document image 30, before training the shadow estimation model 300, it is necessary to pre-adjust the size of the synthetic shadowed document image in the training data set to the uniform requirement of the shadow estimation model 300 for the input first document image 30.

[0190] S533: Based on the model loss function, iteratively train the shadow estimation model 300.

[0191] S540 , when a preset condition is met, stop training to obtain a trained shadow estimation model 300 .

[0192] Specifically, the shadow estimation model 300 is iteratively trained using the training data set until a preset condition is met, then the training is stopped, and the parameters of the shadow estimation model are saved to obtain a trained shadow estimation model 300 .

[0193] The preset conditions include: a preset number of iterations, and a preset condition related to a loss function. For example, when the preset condition is a preset number of iterations, the shadow estimation model 300 is iteratively trained, and when the number of iterations reaches the preset number of iterations, the training is stopped to obtain a trained shadow estimation model 300; when the preset condition is a preset loss function value, the shadow estimation model 300 is iteratively trained, and when the value of the model loss function is less than or equal to the preset loss function value, the training is stopped to obtain a trained shadow estimation model 300; when the preset condition is that the overall loss function approaches stability, the shadow estimation model 300 is iteratively trained, and when the model loss function approaches stability, the training is stopped to obtain a trained shadow estimation model 300.

[0194] Furthermore, in the embodiment of the present application, a pre-set number of iterations is selected as a preset condition. When the number of data samples in the training data set is 30,000, the maximum number of iteration cycles is set to 200 (the pre-set number of iterations is 200). The pre-set number of iterations can be adjusted within a certain range, but if it is set too large, it may easily cause the trained shadow estimation model 300 to overfit. If it is set too small, the training of the shadow estimation model 300 will be insufficient. It is necessary to set an appropriate number of iterations according to the actual situation.

[0195] Furthermore, in the training method of a shadow estimation model 300 provided by the embodiment of the present application described above, a training data set is obtained in advance, that is, before training the shadow estimation model 300, the shadow area image and the shadow-free document image are enhanced, and the corresponding shadow document image is synthesized to obtain training data for training the shadow estimation model 300. In this way, the enhancement and synthesis of the training data can be omitted during the training process, so that the training speed of the model is improved. In addition, in the process of training the shadow estimation model 300, the shadow area image and the shadow-free document image can be enhanced and synthesized to obtain a training data set. The samples in the training data set obtained in this way are more diverse than those in the training data set obtained in advance, which can reduce the domain offset of the model and improve the generalization ability of the model after training.

[0196] Specifically, when training the shadow estimation model 300, the training data (shadow area images and shadow-free document images) are randomly read and packaged into small batches through the data iterator, and then the training data (shadow area images and shadow-free document images) are enhanced and synthesized to obtain a training data set. Among them, the batch size is generally set to 16, and can also be appropriately adjusted according to the device video memory size, which is not specifically limited in this application.

[0197] Among them, the batch size is 16, and 16 groups of training data (shadow area images and shadow-free document images) are selected each time, that is, one set of training data packaged into a batch is selected each time for enhancement and synthesis processing, and is input into the shadow estimation model 300 respectively, and the parameter adjustment values ​​corresponding to the input training data are calculated, and the average value of all parameter adjustment values ​​is taken as the final parameter adjustment value of the batch, and the parameters of the shadow estimation model 300 are adjusted.

[0198] Embodiment 1:

[0199] After the shadow estimation model 300 is trained, it is necessary to use the shadow estimation model 300 to remove the shadow area of ​​the first document image. In this embodiment of the application, the size of the first document image meets the uniform requirement of the shadow estimation model 300 for the input image size, that is, the size of the first document image is 512*512 pixels.

[0200] Combine the following Figure 6 , Figure 7 , a document image shadow removal method provided in an embodiment of the present application is introduced in detail.

[0201] S610 , preprocessing the trained shadow estimation model 300 .

[0202] The preprocessing includes: re-parameterization processing, output removal processing, model conversion processing (eg, model quantization), etc. The trained shadow estimation model 300 is preprocessed according to the actual usage and the configuration of the platform where the shadow estimation model 300 is deployed.

[0203] Specifically, if a re-parameterized structure is used in the shadow estimation model 300, the trained shadow estimation model 300 needs to be re-parameterized. Specifically, multiple parallel branches are equivalently converted into one branch, that is, multiple parallel convolution kernels are combined into one convolution kernel (such as Figure 4A and Figure 4B As shown, Figure 4A The structure shown is equivalent to the following Figure 4B The structure shown in FIG. 1 is used to calculate the shadow estimation model 300, and the batch normalization parameter is fixed to the bias parameter of the convolution kernel in one branch. The shadow estimation model 300 after the re-parameterization process consumes less resources during use and ensures that the performance meets the requirements.

[0204] Furthermore, in an embodiment of the present application, the shadow estimation model 300 is directly used to process the first document image to obtain the corresponding target document image. Therefore, the shadow estimation model 300 is de-outputted, and the structure of the shadow estimation model 300 is adjusted to remove the first output module 341, that is, the output module (output layer) for outputting the estimated shadow image is removed. This reduces the structure of the model, reduces the parameters in the model, reduces memory usage, and improves processing speed.

[0205] Furthermore, according to the configuration performance of the platform where the shadow estimation model 300 is deployed, the shadow estimation model 300 is subjected to corresponding model conversion, among which the most common is model quantization. Model quantization can effectively compress model parameters and reduce the size of the model structure, thereby speeding up the processing / inference speed, so that the shadow estimation model 300 can be applied to the corresponding processor, such as: DSP processor, NPU processor.

[0206] S620: Acquire a first document image.

[0207] Specifically, the size of the first document image meets the uniform requirement of the shadow estimation model 300 for the input image size, and 512*512 pixels is taken as an example for explanation, that is, the size of the obtained first document image is 512*512 pixels.

[0208] S630: Perform normalization processing on the acquired first document image.

[0209] The normalization process for the first image is the same as the normalization process for the input synthesized document image with shadows when training the shadow estimation model 300. That is, in the embodiment of the present application, the normalization process is zero mean normalization.

[0210] It should be noted that the normalization processing methods include: zero mean normalization, maximum and minimum normalization, Z-Score normalization, standard deviation normalization, etc. As long as the normalization processing method performed on the input shadow document image when training the shadow estimation model 300 is the same, this application does not make any specific limitations.

[0211] S640 , based on the preprocessed shadow estimation model 300 , process the first document image and output a target document image.

[0212] The target document image is the document image after the shadow area is removed from the first document image, that is, the document image after the shadow area is removed, which may also be called the second document image.

[0213] like Figure 7A simplified schematic diagram of removing shadows from a document image is shown, where the size of the first document image meets the uniform requirements of the shadow estimation model 300 for the input image size, i.e., the size of the first document image does not need to be processed. For the convenience of explanation, the size of the first document image is described as 512*512 pixels. After the first document image is normalized, it is input into the preprocessed shadow estimation model 300. The shadow estimation model 300 directly outputs a target document image (i.e., a second document image) corresponding to the first document image based on the input first document image. The size of the target document image is 512*512 pixels.

[0214] The embodiment of the present application utilizes a trained shadow estimation model to perform shadow removal processing on the first document image. Since the shadow estimation model introduces color loss between the estimated shadow image and the actual shadow area image during the training process, the shadow estimation model training is supervised so that the color distribution of the estimated shadow image and the actual shadow area image are as close as possible, thereby avoiding the shadow estimation model removing dark elements in the document image as shadow areas, thereby failing to retain non-shadow area elements in the document image.

[0215] Furthermore, if the size of the first document image meets the uniform requirement of the shadow estimation model 300 for the input image size, there is no need to process the size of the first document image, so there will be no image content loss / image distortion, and the overall information of the first document image can be effectively utilized.

[0216] Embodiment 2:

[0217] When the size / resolution of the first document image is relatively large, take the first document image size as 1500*2000 pixels as an example (e.g. Figure 8 The shadow estimation model 300 is a simplified schematic diagram of the shadow removal of the first document image shown in FIG. 3 . The size meets the uniform requirements of the shadow estimation model 300 for the input image size. The pre-processed shadow estimation model 300 can perform shadow removal processing based on the input large-size / resolution first document image, and can make full use of the overall information of the input image to output the target document image of the same large size / resolution (i.e., the second document image). However, since the input first document image contains more image information / features, it will take a long time to process, which may produce a more obvious delay effect. Furthermore, if the first document image is downsampled and then input into the pre-processed shadow estimation model for processing, it is easy to cause image content loss / image distortion, resulting in poor accuracy of the output target document image.

[0218] Combine the following Fig. 9 , Fig.10 , another document image shadow removal method provided by an embodiment of the present application is introduced in detail.

[0219] S910 , preprocessing the trained shadow estimation model 300 .

[0220] Specifically, in the embodiment of the present application, the shadow removal task of the first document image is implemented based on the corresponding estimated shadow image output by the shadow estimation model 300. Therefore, the shadow estimation model 300 is subjected to output removal processing, the structure of the shadow estimation model 300 is adjusted, and the second output module 342 is removed, that is, the output module (output layer) for outputting the target document image is removed, thereby reducing the structure of the model, reducing the parameters in the model, reducing the memory usage, and improving the processing speed.

[0221] It should be noted that compared with S610 in Example 1, the pre-processing de-output processing steps in Example 2 of the present application are different, and the remaining pre-processing details can be found in S160.

[0222] S920: Acquire a first document image.

[0223] Specifically, there is no specific requirement for the size of the first document image, and it can be an image of large size / resolution, and this application does not make any specific limitation.

[0224] S930: Normalize the acquired first document image and adjust the size of the first document image to a preset size to obtain a first image.

[0225] The preset size is a unified requirement of the shadow estimation model 300 for the input image size.

[0226] Furthermore, in order to ensure the processing speed of the shadow estimation model while reducing the difficulty of training the shadow estimation model, the preset size can be set to 512*512 pixels.

[0227] Specifically, the size of the first document image is adjusted to a preset size by interpolation.

[0228] S940 . Process the first image based on the preprocessed shadow estimation model 300 , and output a corresponding second image.

[0229] The second image is a corresponding estimated shadow image of a preset size output by the shadow estimation model 300 .

[0230] Here, the estimated shadow image is an image of an estimated shadow area in the first document image (input image).

[0231] S950: Adjust the size of the corresponding second image to the size of the acquired first document image to obtain an estimated shadow image corresponding to the first document image.

[0232] Specifically, the size of the second image is adjusted from a preset size to the size of the acquired first document image by interpolation, that is, the size of the second image is consistent with the size of the acquired first document image.

[0233] S960: Based on the corresponding estimated shadow image, remove the shadow area of ​​the first document image to obtain a second document image.

[0234] Specifically, the shadow area in the first document image can be removed using a preset algorithm. Specifically, the preset algorithm is: second document image = first document image / corresponding estimated shadow image*255. That is, the first document image and the estimated shadow image are divided by pixels, thereby removing the shadow area in the first document image.

[0235] like Fig.10 Another schematic diagram of removing shadows of a first document image is shown, wherein the size of the first document image is 1500*2000 pixels as an example, and the preset size is 512*512 pixels as an example.

[0236] First, after planning and processing the first document image, the size of the first document image is adjusted from 1500*2000 pixels to 512*512 pixels to obtain a first image of 512*512 pixels. The first image is used as the input of the preprocessed shadow estimation model 300, and the shadow estimation model 300 outputs a second image corresponding to the first image based on the first image, and the second image is a corresponding estimated shadow image with a size of 512*512 pixels. The size of the second image is adjusted from 512*512 pixels to 1500*2000 pixels to obtain an estimated shadow image consistent with the size of the first document image. Finally, the first document image is processed according to a preset algorithm to obtain a document image with the shadow area removed, that is, a second document image.

[0237] The embodiment of the present application utilizes a trained shadow estimation model to perform shadow removal processing on the first document image. Since the shadow estimation model introduces color loss between the estimated shadow image and the actual shadow area image during the training process, the shadow estimation model training is supervised so that the color distribution of the estimated shadow image and the actual shadow area image are as close as possible, thereby avoiding the shadow estimation model removing dark elements in the document image as shadow areas, thereby failing to retain non-shadow area elements in the document image.

[0238] Furthermore, since the shadow area has less high-frequency information (i.e., less grayscale change information / pixels), for the shadow area image, enlarging or reducing it will have almost no effect on the content of the shadow area. Therefore, the training and reasoning of the resolution in the shadow estimation model can be reduced, effectively reducing the difficulty of training and processing, and will not affect the de-shadowing effect of the document image.

[0239] Embodiment three:

[0240] Combine the following Fig.11 , a document image shadow removal method provided in an embodiment of the present application is introduced in detail.

[0241] S1101. Acquire a first document image.

[0242] The first document image includes a shadow area.

[0243] S1102: Call a shadow estimation model to process the first document image to obtain an estimated shadow image corresponding to the first document image.

[0244] The estimated shadow image is an image of a shadow area in the first document image output by the shadow estimation model.

[0245] The shadow estimation model is obtained by iterative training using a model loss function including a shadow loss function.

[0246] Specifically, the shadow loss function includes: estimating the regression loss between the shadow image and the corresponding shadow area image, the total variation loss, and the color loss.

[0247] Furthermore, the model loss function also includes a shadow removal loss function. Specifically, the shadow removal loss function includes: a regression loss between the target document image and the corresponding shadow-free document image. This allows the shadow estimation model to output the corresponding target document image more accurately, so that the subsequent target document image is used as a sentence to update the second document image.

[0248] Furthermore, the model loss function also includes a restoration loss function. Specifically, the restoration loss function includes: a regression loss between the restored document image and the corresponding document image with shadows.

[0249] Specifically, the color loss between the estimated shadow image and the corresponding actual shadow area image is the regression loss between the calculated color map of the estimated shadow image and the calculated color map of the corresponding shadow area image.

[0250] Specifically, the regression loss may be Huber loss, or may be mean absolute error loss or mean square error loss, which is not specifically limited in this application.

[0251] S1103 . Based on the corresponding estimated shadow image, remove the shadow area in the first document image to obtain a second document image.

[0252] The document image removal method provided in the above embodiment removes the shadow area of ​​the first document image (the document image to be removed from the shadow) based on the estimated shadow image output by the shadow estimation model, wherein the shadow estimation model increases the color loss of the shadow area during the training process, so that the color distribution of the estimated shadow image output by the shadow estimation model is as close to the actual situation as possible, and the dark elements in the non-shadow area of ​​the first document image will not be identified as the shadow area and reflected in the estimated shadow image, so that the dark elements in the non-shadow area will not be removed as the shadow area, and the shadow area in the color document image can be effectively removed.

[0253] Furthermore, the method further includes: calling the shadow estimation model to process the first document image to obtain a target document image corresponding to the first document image; and obtaining a final second document image based on the second document image and the corresponding target document image. That is, combining the target document image and the second document image output by the shadow estimation model, a series of processing is performed on the second document image, so that the final second document image better meets actual needs.

[0254] The following is a comparison of the performance of removing the shadow area of ​​the document image between the shadow estimation model and the traditional algorithm (water-filling) in combination with Table 1. Among them, the performance includes: image performance and time performance, among which the image performance is divided into two categories, one for black and white document images (paper); the other for color document images (color). The evaluation indicators for image performance are: Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR); the evaluation indicator for time performance is delay (ms).

[0255] Among them, PSNR is the abbreviation of "Peak Signal to Noise Ratio", which is an objective standard for evaluating images. SSIM (structural similarity index) is an indicator for measuring the similarity between two images. The range of the structural similarity index is 0-1. When the two images are completely different, the structural similarity index is 0; when the two images are exactly the same, the structural similarity index is 1.

[0256] Specifically, the performance comparison results are shown in Table 1.

[0257] Table 1

[0258]

[0259] Among them, epoch means the number of times all data is sent into the network to complete a forward calculation and back propagation process. An epoch means that all training data is input into the model / network once.

[0260] As shown in the data in Table 1, the performance of the shadow area of ​​the document image of the shadow estimation model is better than that of the traditional algorithm (water-filling). That is, whether it is a black and white document image or a color document image, the image processing performance of the shadow estimation model is better than the water-filling algorithm, and the delay is also much lower than the water-filling algorithm.

[0261] Based on this, it can be seen that the color distribution of the output estimated shadow image of the shadow estimation model obtained by supervised training by introducing color loss is closer to the actual situation, and the dark elements in the non-shadow area will not be displayed as shadow areas on the estimated shadow image. Therefore, the effect of removing the shadow area of ​​the document image based on the estimated shadow image is better than the traditional algorithm in the prior art, and for color document images, the effect of removing the shadow area of ​​the color document image is better, and it is more suitable for color document images, and can effectively remove the shadow area in the color document image.

[0262] In order to more intuitively understand the effect of the document image deshadowing method provided by the present application, Fig. 12A , Fig. 12B , Fig.13A and Fig. 13B The document image shown can directly reflect the effect of the shadow removal task. Fig. 12A and Fig. 12B It reflects the effect of the traditional algorithm (water-filling) in performing the shadow removal task. Fig. 12A is a document image 1201 whose shadow area is to be removed, Fig. 12B for Fig. 12A The document image 1201 to be shadowed is shown, and the document image 1202 after the shadowed area is removed by using a conventional algorithm. Fig.13A and Fig. 13B It reflects the effect of implementing the shadow removal task of a document image shadow removal method provided by the present application. Fig.13A is a document image 1301 whose shadow area is to be removed, Fig. 13B for Fig.13A The document image 1301 with the shadow area to be removed is shown, and the document image 1302 after the shadow area is removed using a document image shadow removal method provided by the present application.

[0263] Specifically, according to Fig. 12A , Fig. 12B , Fig.13A and Fig. 13B As can be seen from the document image shown, the traditional algorithm for removing the shadow area in the document image will remove the dark elements in the non-shadow area as the shadow area, thereby causing the dark elements at the edge of the document image to be removed, making it impossible for the color document image with the shadow area removed to completely retain the elements of the non-shadow area in the document image, and unable to effectively remove the shadow area of ​​the color document image; however, the document image shadow removal method provided in the present application removes the shadow area in the document image, which can effectively remove the shadow area in the color document image, thereby completely retaining the elements of the non-shadow area in the document image, and effectively realizing the removal of the shadow area of ​​the color document image.

[0264] It should be noted that the above-mentioned shadow estimation model training method and shadow removal method can be applied to other images with shadow areas, such as natural environment images, group photos, etc. It is necessary to replace the corresponding training data set according to the type of application scene / image, and train the shadow estimation model. This application does not make specific limitations.

[0265] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed, the electronic device implements the above-mentioned functions or steps.

[0266] An embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, enables an electronic device to implement the above-mentioned functions or steps.

[0267] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for removing shadows from a document image, characterized in that: include: Acquire a first document image, wherein the first document image includes a shadow area; Calling a shadow estimation model to process the first document image to obtain an estimated shadow image corresponding to the first document image; Based on the estimated shadow image, removing the shadow area in the first document image to obtain a second document image; The shadow estimation model is obtained by iterative training using a model loss function including a shadow loss function; the shadow loss function includes: regression loss, total variation loss and color loss between the estimated shadow image output by the shadow estimation model and the corresponding actual shadow area image.

2. The method according to claim 1, characterized in that The color loss between the estimated shadow image output by the shadow estimation model and the corresponding actual shadow area image is obtained in the following manner: Calculating and obtaining a calculated color map of the estimated shadow image and a calculated color map of the corresponding actual shadow area image; A regression loss between a calculated color map of the estimated shadow image and a calculated color map of an actual shadow area image of the first document image is calculated to obtain a color loss between the estimated shadow image and the corresponding shadow area image.

3. The method according to claim 1, characterized in that: The shadow estimation model includes: a re-parameterized structure.

4. The method according to any one of claims 1 to 3, characterized in that Before acquiring the first document image, the method further includes: The shadow estimation model is preprocessed; wherein the preprocessing includes one or more of the following: reparameterization processing, de-output processing and model conversion processing.

5. The method according to any one of claims 1 to 4, characterized in that After acquiring the first document image, the method further includes: performing normalization processing on the first document image to obtain a processed first document image; The calling of the shadow estimation model to process the first document image to obtain an estimated shadow image corresponding to the first document image includes: calling the shadow estimation model to process the processed first document image to obtain an estimated shadow image corresponding to the first document image.

6. The method according to any one of claims 1 to 5, characterized in that: The size of the first document image is inconsistent with a preset size; wherein the preset size is a required size of an input image of the shadow estimation model; before calling the shadow estimation model to process the first document image to obtain an estimated shadow image corresponding to the first document image, the method further includes: Adjusting the size of the first document image to the preset size to obtain a first image; The calling of the shadow estimation model to process the first document image to obtain an estimated shadow image corresponding to the first document image includes: calling the shadow estimation model to process the first image to obtain a second image; wherein the second image is an estimated shadow image corresponding to the first image and having a preset size; The method further includes: adjusting the size of the second image to the size of the first document image to obtain an estimated shadow image corresponding to the first document image.

7. The method according to claim 1, characterized in that The model loss function also includes: a shadow removal loss function; the shadow removal loss function includes: a regression loss between a target document image output by the shadow estimation model and a corresponding actual shadow-free document image.

8. The method according to claim 7, characterized in that The method further comprises: Calling the shadow estimation model to process the first document image to obtain a target document image corresponding to the first document image; the target document image corresponding to the first document image is an image output by the shadow estimation model with the shadow area of ​​the first document image removed; Based on the second document image and the target document image, a final second document image is obtained.

9. The method according to claim 7, characterized in that: The model loss function also includes: a restoration loss function; the restoration loss function includes: a regression loss between a restored document image and a corresponding actual document image with shadow; wherein the restored document image is synthesized by an estimated shadow image output by the shadow estimation model and a target document image output by the shadow estimation model.

10. The method according to any one of claims 1 to 9, characterized in that: The structure of the shadow estimation model is a symmetric codec structure.

11. The method according to any one of claims 1 to 9, characterized in that: The regression loss includes: Huber loss.

12. An electronic device, characterized in that: including a processor and a memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed, the method according to any one of claims 1 to 11 is implemented.

Images