Fluorescence labeling document processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a processing method and device for a fluorescence labeling document, equipment and a storage medium. The processing method comprises the steps that a multispectral scanning image set of a target document at at least one scanning angle is collected; based on the multispectral scanning image set, performing fusion processing through a deep learning model to obtain a first fusion image; determining a fluorescence labeling area in the first fusion image based on optical characteristics of a fluorescent pigment in the target document; and performing image enhancement processing on the fluorescence labeling area to obtain a target image. By carrying out image enhancement processing on the fluorescent labeling area, the printing or scanning quality of fluorescent labeling can be improved.

Description

Technical Field

[0001] The present application relates to the field of image forming technology, and particularly relates to a method for processing a fluorescently marked document and an image forming device. Background Art

[0002] In scenarios such as office work and study, people often use highlighters to mark important content in documents for quick positioning and highlighting key information. However, existing copying technologies have many deficiencies when processing documents with fluorescent markings. Due to the characteristics of fluorescent pigments and the imaging principle of image forming devices, fluorescent markings often fade or have color deviations after copying / scanning, affecting the quality of the copied / scanned documents and the integrity of information, and bringing great inconvenience to users. Summary of the Invention

[0003] In view of this, the present application provides a method, device, equipment, and storage medium for processing a fluorescently marked document to facilitate solving the problem of low printing or scanning quality of fluorescent markings in the prior art.

[0004] In a first aspect, an embodiment of the present application provides a method for processing a fluorescently marked document, including: Collecting an original image in response to a triggering operation of a user on a marking mode; Determining a fluorescent marking area in the original image and fluorescent marking information related to the fluorescent marking area; Processing the fluorescent marking area to obtain a target image; Wherein, the marking mode includes an automatic marking mode. When the user triggers the automatic marking mode, the collecting the original image includes: collecting a multi-spectral scanning image set of a target document at at least one scanning angle; The determining the fluorescent marking area in the original image includes: performing fusion processing on the multi-spectral scanning image set through a deep learning model to obtain a first fusion image; determining the fluorescent marking area in the first fusion image based on the optical characteristics of the fluorescent pigment in the target document; The processing the fluorescent marking area to obtain a target image includes: performing image enhancement processing on the marking area to obtain a target image.

[0005] In an optional embodiment, the multi-spectral scanning image set at least includes: an ultraviolet band image set, an infrared band image set, or a visible light band image set; the performing fusion processing on the multi-spectral scanning image set through a deep learning model to obtain a first fusion image includes: Performing fusion processing on the images in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain a second fusion image set; Perform a fusion process on the second fusion image set to obtain the first fusion image.

[0006] In an alternative embodiment, the step of fusing the image sets in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain the second fusion image set includes: Fuse the ultraviolet band image set based on the maximum color value of the same pixel point in the images corresponding to each scanning angle to obtain an ultraviolet band fused image; Alternatively, fuse the infrared band image set based on region fusion of wavelet transform to obtain an infrared band fused image; Alternatively, fuse the visible light band image set based on the preset weights of each scanning angle to obtain a visible light band fused image.

[0007] In an alternative embodiment, after determining the fluorescent annotation area in the first fusion image based on the optical properties of the fluorescent pigment in the target document, the method further includes: Determine an enhancement coefficient based on the illumination intensity value of the current environment, where the enhancement coefficient is positively correlated with the illumination intensity value; Calibrate the first fusion image based on the enhancement coefficient and white balance; The step of determining the fluorescent annotation area in the first fusion image based on the optical properties of the fluorescent pigment in the target document includes: Determine the candidate areas of fluorescent annotation in the first fusion image based on the optical properties of the fluorescent pigment in the target document; Determine the fluorescent annotation areas in the candidate areas of fluorescent annotation based on the deep learning model.

[0008] In an alternative embodiment, the step of performing image enhancement processing on the annotation area includes: Adjust the contrast of the annotation area based on the histogram equalization algorithm; And / or, convert the color value of the annotation area to the corresponding gray value based on a pre-stored gray mapping table.

[0009] In an alternative embodiment, the annotation mode further includes a manual annotation mode. When the user triggers the manual annotation mode, the step of determining the fluorescent annotation area in the original image includes: Respond to the triggering operation of the user based on the display screen of the original image to determine the fluorescent annotation area marked by the user; The fluorescent annotation information related to the fluorescent annotation area includes the annotation color and / or transparency. The step of processing the fluorescent annotation area to obtain the target image includes: Generate a corresponding annotation layer based on the fluorescence-labeled region, label color, and / or transparency. Fuse the scanned image and the annotation layer to obtain the target image.

[0010] In an optional embodiment, before fusing the scanned image and the annotation layer to obtain the target image, the method further includes: Adjust the transparency of the annotation layer based on the background brightness value of the scanned image, where the transparency is negatively correlated with the background brightness value.

[0011] In a second aspect, an embodiment of the present application provides a processing device for a fluorescently labeled document, including: A response module, configured to collect a multi-spectral scanned image set of a target document at at least one scanning angle in response to a user's trigger operation for the automatic annotation mode; A fusion module, configured to perform fusion processing on the multi-spectral scanned image set through a deep learning model to obtain a first fused image; An annotation module, configured to determine a fluorescently labeled region in the first fused image based on the optical characteristics of the fluorescent pigment in the target document; An enhancement module, configured to perform image enhancement processing on the labeled region to obtain a target image.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including a memory for storing computer program instructions and a processor for executing the program instructions, where when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method according to any one of the first aspects above.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and when the program runs, it controls the device where the computer-readable storage medium is located to execute the method according to any one of the first aspects.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, where the computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer is caused to execute the method according to any one of the first aspects.

[0015] Adopting the solution provided by the embodiment of the present application, in response to a user's triggering operation on the annotation mode, an original image is collected; a fluorescence annotation area in the original image and fluorescence annotation information related to the fluorescence annotation area are determined; the fluorescence annotation area is processed to obtain a target image; wherein, the annotation mode includes an automatic annotation mode, and when the user triggers the automatic annotation mode, collecting the original image includes: collecting a multi-spectral scanning image set of a target document at at least one scanning angle; determining the fluorescence annotation area in the original image includes: based on the multi-spectral scanning image set, performing fusion processing through a deep learning model to obtain a first fused image; determining the fluorescence annotation area in the first fused image based on the optical characteristics of the fluorescent pigment in the target document; processing the fluorescence annotation area to obtain a target image includes: performing image enhancement processing on the annotation area to obtain a target image. By performing image enhancement processing on the fluorescence annotation area, the printing or scanning quality of the fluorescence annotation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 Flow schematic of a method for processing a fluorescence-annotated document provided by an embodiment of the present application Figure 1 ; Figure 2 Flow schematic of a method for processing a fluorescence-annotated document provided by an embodiment of the present application Figure 2 ; Figure 3 Flow schematic of a method for processing a fluorescence-annotated document provided by an embodiment of the present application Figure 3 ; Figure 4 Flow schematic of a method for processing a fluorescence-annotated document provided by an embodiment of the present application Figure 4 ; Figure 5 Structural schematic diagram of a device for processing a fluorescence-annotated document provided by an embodiment of the present application; Figure 6 Structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] In order to better understand the technical solutions of the present application, the embodiments of the present application will be described in detail below with reference to the drawings.

[0019] It should be clear that the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0020] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms of "a", "the", and "said" used in the embodiments of the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be understood that the term "and / or" used herein is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0022] When the image forming device performs operation tasks such as scanning and copying, if the original document has fluorescent markings, problems such as fading and color deviation often occur in the target document obtained by conventional scanning and copying. To address this problem, the embodiments of the present application provide a method for processing fluorescently marked documents to improve the scanning quality of fluorescent markings in the target document and the information integrity of the target document.

[0023] Figure 1 Schematic flow of a method for processing fluorescently marked documents provided by the embodiments of the present application Figure 1 , this method can be executed by an image forming device, such as Figure 1 shown, this method may include: Step 101, in response to a triggering operation by the user on the marking mode, collect the original image.

[0024] In addition to the normal mode, the image forming device in the embodiments of the present application is configured with a marking mode for processing fluorescent markings. If the original document has fluorescent markings, the user can select the corresponding marking mode when issuing an operation task.

[0025] Step 102, determine the fluorescent marking area in the original image and the fluorescent marking information related to the fluorescent marking area.

[0026] Step 103, process the fluorescent marking area to obtain a target image.

[0027] In the normal copying or scanning mode, after the image forming device acquires the original image, subsequent processes will be performed based on the original image. For example, in normal copying, the image forming device outputs a copied document after acquiring the original image. In the marking mode, after the image forming device collects the original image, it first needs to determine the fluorescent marking area in the original image and the fluorescent marking information related to the fluorescent marking area, such as the color of the fluorescent marking. Then, the image forming device processes the fluorescent marking area, such as increasing the contrast, replacing the color, etc., to highlight the fluorescent marking area and obtain the target image, and performs subsequent scanning or copying processes based on the target image.

[0028] In an alternative embodiment, the marking mode may specifically include: an automatic marking mode and a manual marking mode. When the user selects the automatic marking mode, the processing flow of the image forming device can refer to Figure 2 , mainly including: Step 201, in response to the user's trigger operation for the automatic marking mode, collect a multi-spectral scanning image set of the target document at at least one scanning angle.

[0029] When the image forming device scans the target document, there are certain differences in the data scanned multiple times at the same angle, and there are also differences in the data scanned at different angles. In order to improve the scanning quality as much as possible, the image forming device in the embodiment of the present application performs multi-spectral image acquisition on the target document from multiple scanning angles to obtain the corresponding multi-spectral scanning image set. Among them, the multi-spectral scanning image set may include: an ultraviolet band image set, an infrared band image set, and a visible light band image set.

[0030] Exemplarily, in actual operation, the user places the target document with fluorescent markings on a specially designed image acquisition platform. The camera of the image forming device automatically adjusts the position and focal length, and can perform image acquisition on the target document in ultraviolet light (300 - 400nm), visible light (400 - 700nm), and near-infrared light (700 - 1000nm) and other bands from three different angles (such as 0°, 45°, 90°). An image can be collected from each angle and each band. By performing multi-angle acquisition in the ultraviolet band, an ultraviolet band image set can be obtained. By performing multi-angle acquisition in the infrared band, an infrared band image set can be obtained. By performing multi-angle acquisition in the visible light band, a visible light band image set can be obtained. Optionally, the target document can also be imaged from more or fewer angles (such as 0°, 30°, 60°, 90°, etc.) and bands. The embodiment of the present application does not limit this.

[0031] In the embodiment of the present application, by performing multi-angle and multi-spectral image acquisition on the target document, image details can be obtained in multiple aspects, the quality of image acquisition can be improved, and problems such as information loss of the content of the fluorescent marking can be avoided.

[0032] Step 202: Based on the multi-spectral scanning image set, perform fusion processing through a deep learning model to obtain a first fused image.

[0033] Specifically, the image forming device can first perform preliminary fusion processing on the images in the multi-spectral scanning image set at the same angle or the same band through the deep learning model, and then perform further fusion processing on the images obtained after the preliminary fusion processing to obtain a first fused image. Alternatively, the image forming device can directly perform fusion processing on the multi-spectral scanning image set to obtain a first fused image.

[0034] In an alternative embodiment, the image forming device separately performs fusion processing on the image sets in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain a second fused image set corresponding to each spectral band. Then, the image forming device performs fusion processing on the second fused image set to obtain a first fused image.

[0035] Specifically, the second fused image set may include: an ultraviolet band fused image, an infrared band fused image, and a visible light band fused image. When performing fusion processing on the image set of the same band, fusion processing can be performed according to the preset weights of the images at each angle in the image set to obtain the ultraviolet band fused image, the infrared band fused image, and the visible light band fused image. When performing fusion processing on the second fused image set, due to the optical characteristics of the fluorescent pigment, there is a distinction between the images of the fluorescent labeled area and the non-fluorescent labeled area in the ultraviolet band fused image and the infrared band fused image. The images of the fluorescent labeled area in the ultraviolet band fused image and the infrared band fused image can be extracted and fused with the visible light band fused image.

[0036] In the embodiment of the present application, by first performing fusion processing on the image set of the same band to obtain a second fused image set, and then performing fusion processing on the second fused image set, a targeted fusion strategy can be adopted for different band image sets. At the same time, the first fused image combines the images of the fluorescent labeled area in the specific band image and the visible light band fused image, which is convenient for accurately determining the fluorescent labeled area subsequently and makes the first fused image closer to the target document.

[0037] In an alternative embodiment, the image fusion process may include: (1) The image forming device fuses the ultraviolet band image set based on the maximum color value of the same pixel point in the images corresponding to each scanning angle to obtain an ultraviolet band fused image. Specifically, the fluorescent label will produce a fluorescence effect under ultraviolet light irradiation. The maximum color value of the same pixel point in the images corresponding to each scanning angle is used as the color value of this pixel point in the ultraviolet band fused image to highlight the fluorescence characteristics and enhance the distinguishability between the fluorescent pigment and the ordinary pigment. (2) The image forming device fuses the infrared band image set based on region fusion of wavelet transform to obtain an infrared band fused image. Specifically, the image forming device decomposes the infrared band image set into a low-frequency part and a high-frequency part based on wavelet transform. The low-frequency part can use the 0-degree image, and the high-frequency part can use the 45-degree image for fusion to obtain an infrared band fused image, so as to achieve the effect of retaining the overall structure with the 0-degree image in the low-frequency part and enhancing the local texture with the 45-degree image in the high-frequency part. (3) The image forming device fuses the visible light band image set based on the preset weights of each scanning angle to obtain a visible light band fused image. Specifically, the preset weight of the 0-degree image can be greater than the preset weights of the images at the other angles to retain the color of the front view and supplement details from the side view. For example, the preset weight of the 0-degree image is set to 0.5, the preset weight of the 45-degree image is set to 0.3, and the preset weight of the 90-degree image is set to 0.2.

[0038] In the embodiment of the present application, by taking the maximum color value of the same pixel point at multiple angles in the ultraviolet band, the fluorescence characteristics can be highlighted, and the distinguishability between the fluorescent pigment and the ordinary pigment can be enhanced. Fusing the infrared band based on wavelet transform can enhance the local texture while maintaining the overall structure of the image. By weighted fusion of the visible light band, the color of the front view and the details of the side view can be retained.

[0039] Step 203: Determine the fluorescent label area in the first fused image based on the optical characteristics of the fluorescent pigment in the target document.

[0040] Specifically, fluorescent pigments have a special optical behavior, that is, they can absorb light of a specific wavelength (usually light of a shorter wavelength, such as ultraviolet light or blue light) and re-emit (reflect) light in a longer wavelength range. This property makes fluorescent pigments appear very bright and even "glow" under certain conditions. Ordinary pigments usually do not have this fluorescence effect. They only reflect light of a specific wavelength and do not absorb short-wavelength light and re-emit longer-wavelength light.

[0041] The image forming device can emit light of a specific wavelength (such as ultraviolet light) to irradiate the target document. After scanning the document, the ultraviolet fluorescent pigments in the fluorescent labeled area absorb light of short wavelength and re-emit light of longer wavelength to form images with prominent colors such as bright green and orange-yellow, while the ordinary pigment area forms dark images, etc. Therefore, there are significant differences between the fluorescent labeled area and the ordinary pigment area in the ultraviolet band fusion image, and the fluorescent labeled area in the ultraviolet band fusion image can be determined according to the color, and then the fluorescent labeled area in the first fusion image can be determined. Further, if the fluorescent labeled area includes near-infrared fluorescent pigments, after being irradiated with infrared light, the near-infrared fluorescent labeled area will emit light with a wavelength of 800-900 nanometers to form a fluorescent image with high contrast, while the ordinary pigments form dark or non-fluorescent images due to the lack of fluorescence effect, and the fluorescent labeled area in the infrared band fusion image can be determined according to the color, and then the fluorescent labeled area in the first fusion image can be determined.

[0042] In an alternative embodiment, the steps for the image forming device to determine the fluorescent labeled area may include: (1) Based on the optical properties of the fluorescent pigments in the target document, determine the candidate areas of the fluorescent labels in the first fusion image. Specifically, based on the color difference between the fluorescent labeled area and the ordinary pigment area caused by the optical properties of the fluorescent pigments, a fluorescent color threshold can be set to determine the areas in the ultraviolet band fusion image and / or the infrared band fusion image whose color values are greater than or equal to the fluorescent color threshold, and the areas corresponding to these areas in the first fusion image are used as the candidate areas of the fluorescent labels. (2) Based on the deep learning model, determine the fluorescent labeled areas in the candidate areas of the fluorescent labels. Specifically, the deep learning model can be an image segmentation model such as U-Net, Mask R-CNN, etc., and based on the deep learning model, further identify the fluorescent labeled areas in the candidate areas of the fluorescent labels in the first fusion image to exclude background interference.

[0043] In the embodiments of the present application, the image forming device sequentially identifies the fluorescent labeled area based on the optical properties of the fluorescent pigments and the deep learning model, which can improve the accuracy of identifying the fluorescent labeled area and provide guarantee for subsequent image processing.

[0044] Step 204, perform image enhancement processing on the labeled area to obtain the target image.

[0045] Specifically, the image enhancement processing can be to improve the contrast of the labeled area, replace the color of the labeled area with a more vivid color, etc., to achieve the effect of highlighting the labeled area.

[0046] In an alternative embodiment, the image forming apparatus may adjust the contrast of the marked area based on the histogram equalization algorithm, and may also convert the color values of the marked area into corresponding gray values based on a pre-stored gray mapping table. Different image enhancement processing methods may also be adopted for different job tasks. For example, in a color copying or scanning task, the image forming apparatus may perform image enhancement processing on the marked area by adjusting the contrast. For another example, in a black and white copying or scanning task, the image forming apparatus may perform image enhancement processing on the marked area by gray value conversion.

[0047] During the contrast adjustment process, the image forming apparatus first counts the number of pixels at each gray level in the image to form a histogram with the gray level on the horizontal axis and the number of pixels on the vertical axis, and then maps the original histogram to a histogram with a more uniform distribution, so that the number of pixels at all gray levels tends to be balanced. This method can stretch the pixel gray values originally concentrated in the darker or brighter areas to a wider gray range, thereby enhancing the overall contrast. For example, assuming that the pixel gray values of the original image are concentrated in 100 - 150 (darker), after equalization, they may be distributed to 0 - 255, and the details in the dark part become clearer.

[0048] At the same time, in order to avoid over-enhancement of histogram equalization such as noise amplification and local overexposure, or under-enhancement such as insignificant effects, the image forming apparatus may also adaptively adjust the equalization parameters according to the area and gray distribution of the fluorescent marked area. If the marked area is small, the enhancement intensity may need to be limited to avoid noise interference; if the area is large, the enhancement intensity can be increased to cover more details. If the gray levels are concentrated, such as in a high-brightness fluorescent area, a non-linear transformation can be performed on the histogram of the image during equalization to compress the highlight area and avoid overexposure; if the gray levels are dispersed, a linear expansion can be performed on the histogram of the image to uniformly stretch and enhance the overall contrast.

[0049] In the embodiment of the present application, based on the histogram equalization algorithm and adaptive adjustment, the contrast of the fluorescent marked area can be reasonably adjusted, which can not only distinguish the fluorescent marked area from the ordinary pigment area, but also avoid the problems of background noise interference or over-brightness of the fluorescent area.

[0050] Fluorescent markings such as yellow, pink, green, etc. have high distinctiveness in color mode, but may become blurred during black-and-white copying or scanning due to the influence of the gray-scale conversion algorithm. For example, a yellow fluorescent marking may be close to white in black-and-white mode, resulting in the marking almost disappearing. Therefore, the image forming device can pre-define a gray-scale mapping table for mapping fluorescent markings of different colors to preset gray-scale values, without relying on the default gray-scale conversion algorithm. For example, a yellow fluorescent marking can be converted to a darker gray, such as a gray-scale value of 100, to ensure distinctiveness from the white background, and a pink fluorescent marking can be converted to an even darker gray, such as a gray-scale value of 80, to increase the contrast. Other background colors such as white can be converted to pure white with a gray-scale value of 255.

[0051] In the embodiments of the present application, after specific gray-scale mapping, the fluorescent markings will appear as darker gray in black-and-white copies or scans, while the background remains white or light gray, thus ensuring high visibility of the fluorescent markings.

[0052] In an alternative embodiment, after identifying the fluorescent marking area in the first fused image, the image forming device can also calibrate the first fused image. First, the image forming device determines an enhancement coefficient based on the light intensity value of the current environment, where the enhancement coefficient is positively correlated with the light intensity value. After that, the image forming device calibrates the first fused image based on the enhancement coefficient and white balance.

[0053] Specifically, the enhancement coefficient can be calculated according to the light intensity using the formula: Enhancement coefficient = K × exp(Ambient light intensity value / 100), where exp is the exponential function, and the ambient light intensity value refers to the real-time detected light intensity, with the unit of lux. The enhancement coefficient is used to adjust image processing parameters, such as adjusting RGB values, to adapt to images under different lighting conditions. K is a scaling factor used to adjust the scale of the enhancement coefficient to meet actual needs. The value of the reference value K can be selected from between 0.1 and 0.5 according to the expected change range of the enhancement coefficient. When the value of K is larger, the change of the enhancement coefficient is more obvious under high light; when the value of K is smaller, the change is relatively gentle.

[0054] The process of calibrating the first fused image based on the enhancement coefficient and white balance may include: (1) Extract RGB values: Extract the RGB values of the blank area from the first fused image; (2) Calculate the average value: Calculate the average values of the R, G, and B color channels of all pixel points in the blank area; (3) Determine the gain value: Divide the average value of each color channel by the sum of the average values of the three channels, and the obtained normalized ratio is the gain value; (4) Apply the gain value: Multiply the initial value of each color channel by the enhancement coefficient and the gain value to obtain the calibrated value of each color channel. For example, for the R channel: Output R value = Initial R value Enhancement coefficient R-channel gain value.

[0055] In the embodiments of the present application, the image forming device calibrates the first fused image based on the enhancement coefficient and white balance, making each color in the image close to the standard color under different light intensities, ensuring that the content is clearly visible and the color is accurate, and making the first fused image closer to the target document.

[0056] In an alternative embodiment, the above deep learning model can be an improved U-Net model, and the specific improvements are as follows: (1) Improve the architecture of the model. First, expand the traditional RGB input channels to multi-spectral input channels such as infrared band channels, visible light band channels, and ultraviolet band channels. Each band channel can also include multiple channels at different angles to improve the distinguishability of fluorescence annotations. Exemplarily, when collecting images of the target document in the infrared band, visible light band, and ultraviolet band from three angles (0°, 45°, 90°), the multi-spectral input channels include: three channels of 0°, 45°, and 90° in the infrared band, three channels of 0°, 45°, and 90° in the visible light band, and three channels of 0°, 45°, and 90° in the ultraviolet band, for a total of nine channels. Second, introduce an attention mechanism. Embed a Convolutional Block Attention Module (CBAM) in the skip connections of the U-Net model to enhance the ability to capture small regions of fluorescence annotations.

[0057] (2) Optimize the model training strategy. First, increase the model training tasks. The main task can be to segment the fluorescence annotation area, and the auxiliary task can be to classify different fluorescent pigments such as different brands of highlighters and different colors of fluorescent pigments, so as to facilitate subsequent targeted processing of fluorescence annotations of different fluorescent pigments and improve the generalization of the model. First, increase the scenarios covered by the training data, simulate different lighting conditions such as low light and strong light, as well as different paper backgrounds such as white, yellow, and gray, and generate training data for training to cover more scenarios.

[0058] (3) Lightweight deployment. Quantize the model from 32-bit floating point (FP32) to 8-bit integer (INT8), reduce the number of parameters, and effectively improve the running speed to meet the computing power limitations of the image forming device.

[0059] In the embodiments of the present application, the U-Net model obtained through the above improvements identifies the fluorescence annotation area, can automatically extract features from the data without manual design, so as to achieve high-precision segmentation of the fluorescence annotation area, has strong adaptability to different lighting, backgrounds, and colors, and can also process multiple tasks such as segmentation and classification at the same time, accurately identify the fluorescence annotation area, and perform targeted processing on the fluorescence annotations of different fluorescent pigments.

[0060] The solution provided by the embodiments of this application, in response to a triggering operation by the user for the automatic annotation mode, collects a multi-spectral scan image set of the target document at at least one scan angle; based on the multi-spectral scan image set, performs fusion processing through a deep learning model to obtain a first fused image; determines the fluorescent annotation area in the first fused image based on the optical characteristics of the fluorescent pigment in the target document; performs image enhancement processing on the annotation area to obtain a target image. By collecting images of the target document through multi-angle and multi-spectral means, the quality of image collection is improved. Based on the optical characteristics of the fluorescent pigment and the deep learning model, the fluorescent annotation area is accurately identified. By performing image enhancement processing on the annotation area to achieve the effect of highlighting the annotation area, the obtained target image is close to the target document while ensuring high visibility of the fluorescent annotation. Taking the target image as the scanned image of the target document, or printing the target image as a copy of the target document can improve the printing or scanning quality of the fluorescent annotation.

[0061] In an alternative embodiment, the annotation mode further includes a manual annotation mode. When the user triggers the manual annotation mode, the processing flow of the image forming device can refer to Figure 3 , mainly including: Step 301, in response to a triggering operation by the user based on the display screen of the original image, determines the fluorescent annotation area marked by the user and the fluorescent annotation information related to the fluorescent annotation area.

[0062] Among them, the fluorescent annotation information related to the fluorescent annotation area includes annotation color and / or transparency. After the user selects the manual annotation mode, the touch screen of the image forming device can display the original image. The user can select the annotation area by framing it on the touch screen, or can also perform annotation through a terminal device such as a mobile phone connected to the image forming device. During the annotation process, the user can select fluorescent annotation information such as annotation color and annotation transparency.

[0063] Optionally, text, table recognition algorithms such as optical character recognition algorithms can be used to recognize text and table areas to assist the user in quickly annotating, and multiple annotation methods such as framing, lasso, magic wand, etc. can be provided.

[0064] Step 302, generates a corresponding annotation layer based on the fluorescent annotation area, annotation color and / or transparency.

[0065] Optionally, when the user does not select the annotation color or transparency, a corresponding annotation layer can be generated according to a preset default color such as yellow or a default transparency such as 50%.

[0066] Step 303, fuses the scanned image and the annotation layer to obtain a target image.

[0067] Specifically, each pixel of the scanned image and the annotation layer can be fused to obtain the target image. For the R, G, and B values of the same pixel in the scanned image, the annotation layer, and the target image, the value of the target image = the value of the scanned image × (1 - preset weight) + the value of the annotation layer × preset weight. Among them, the preset weight can be set by the user according to their needs. The greater the preset weight, the greater the influence on the annotation color and transparency; conversely, the smaller the preset weight, the greater the influence of the scanned image. Optionally, the image forming device can store the annotation color, transparency, and preset weight selected by the user, and automatically annotate each page of the image according to the fluorescent annotation area selected by the user when performing multi-page copying or scanning.

[0068] In an alternative embodiment, before fusing the scanned image and the annotation layer to obtain the target image, the image forming device can adjust the transparency of the annotation layer based on the background brightness value of the scanned image, where the transparency is negatively correlated with the background brightness value. Specifically, the following formula can be used for adjustment: adjusted transparency = original transparency × (1 - background brightness value / 255).

[0069] In the embodiments of the present application, by configuring an annotation tool in the image forming device or the terminal device, the user can manually annotate the fluorescent annotation area and preview the annotation effect in real time. By fusing the image pixel by pixel and automatically adjusting the annotation transparency according to the background brightness, a target image with complete information can be obtained, avoiding the difficulty of distinguishing due to color distortion of the fluorescent annotation. Using the target image as the scanned image of the target document, or printing the target image as a copy of the target document, improves the quality of copying or scanning fluorescent annotation documents.

[0070] Figure 4 Schematic flow of a method for processing a fluorescent annotation document provided by an embodiment of the present application Figure 4 , as Figure 4 shown, the method may include: Step 401, the user selects an annotation mode.

[0071] The annotation mode includes: an automatic annotation mode and a manual annotation mode. When the user selects the manual annotation mode, step 402 is entered; when the user selects the automatic annotation mode, step 405 is entered.

[0072] Step 402, obtain the original image, and the user selects the fluorescent annotation area and fluorescent annotation information in the original image.

[0073] The fluorescent annotation information includes the annotation color and transparency.

[0074] Step 403, generate an annotation layer.

[0075] Generate a corresponding annotation layer based on the fluorescent annotation area, annotation color, and transparency.

[0076] Step 404: Merge the original image and the annotation layer to obtain the target image.

[0077] Step 405: Obtain the original images with multi - spectral and multi - angle.

[0078] By collecting multi - angle images in the ultraviolet band, infrared band, and visible light band, the integrity of the original image information can be ensured.

[0079] Step 406: Adjust the original image according to the light intensity value and white balance.

[0080] Calibrating the original image through the light intensity value and white balance can ensure that the content is clearly visible and the color is accurate.

[0081] Step 407: Identify the fluorescent annotation area.

[0082] Based on the optical properties of fluorescent pigments, identifying the fluorescent annotation area through a deep - learning model can improve the accuracy of identifying the fluorescent annotation area and provide guarantee for subsequent image processing.

[0083] Step 408: Perform image enhancement processing on the fluorescent annotation area to obtain the target image.

[0084] In the color mode, image enhancement processing can be achieved by adjusting the contrast. It can not only distinguish the fluorescent annotation area from the ordinary area but also avoid the problems of background noise interference or over - bright fluorescent areas. In the black - and - white mode, image enhancement processing can be achieved through a specific gray - scale mapping. After the specific gray - scale mapping, the fluorescent annotation will appear as a darker gray in the black - and - white copy, while the background remains white or light gray, thus ensuring the high visibility of the fluorescent annotation.

[0085] To implement the above - mentioned method, the image forming device in the embodiments of the present application is configured with corresponding hardware modules, which may include: a multi - spectral scanning head that supports multi - angle scanning in the visible light band, infrared band, and ultraviolet band; an ambient light sensor that detects the intensity of the scanning ambient light in real time; and a touch - interactive screen that supports multi - point touch and pressure sensing.

[0086] Figure 5 It is a schematic structural diagram of a processing device for a fluorescent - annotated document provided by an embodiment of the present application. As Figure 5 shown, the device may include: A response module 510, configured to collect a multi - spectral scanning image set of a target document at at least one scanning angle in response to a user's triggering operation on the automatic annotation mode.

[0087] A fusion module 520, configured to perform fusion processing through a deep - learning model based on the multi - spectral scanning image set to obtain a first fusion image.

[0088] A labeling module 530 for determining a fluorescent labeling region in a first fused image based on the optical characteristics of fluorescent pigments in a target document.

[0089] An enhancement module 540 for performing image enhancement processing on the labeled region to obtain a target image.

[0090] Corresponding to the above embodiments, the present application also provides an electronic device. Figure 6 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 600 may include: a processor 601, a memory 602, and a communication unit 603. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiments of the present application. It can be a bus structure, a star structure, include more or fewer components than shown in the figure, combine certain components, or have different component arrangements.

[0091] Among them, the communication unit 603 is used to establish a communication channel so that the electronic device can communicate with other devices. Receive user data sent by other devices or send user data to other devices.

[0092] The processor 601 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines. By running or executing software programs, instructions, and / or modules stored in the memory 602, and by calling data stored in the memory, it performs various functions of the electronic device and / or processes data. The processor can be composed of an integrated circuit (IC). For example, it can be composed of a single packaged IC, or composed of multiple packaged ICs with the same or different functions connected together. For example, the processor 601 may only include a central processing unit (CPU). In the embodiment of the present application, the CPU can be a single operation core or include multiple operation cores.

[0093] The memory 602 is used to store the execution instructions of the processor 601. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0094] When the execution instructions in the memory 602 are executed by the processor 601, the electronic device 600 can execute some or all of the steps in the above embodiments.

[0095] In a specific implementation, the present application further provides a computer storage medium. The computer storage medium can store a program, and when the program is executed, it can include some or all of the steps in the embodiments of the method for processing a fluorescently marked document provided by the present application. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), or the like.

[0096] In a specific implementation, the present application further provides a computer program product. The computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer is caused to execute some or all of the steps in the embodiments of the method for processing a fluorescently marked document provided by the present application.

[0097] The embodiments of the present application further provide a non-temporary computer-readable storage medium. The non-temporary computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the method for processing a fluorescently marked document provided by the embodiments of the present application.

[0098] The above non-temporary computer-readable storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (hereinafter referred to as ROM), an erasable programmable read-only memory (hereinafter referred to as EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0099] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0100] The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber, etc., or any suitable combination of the foregoing.

[0101] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments of the present application.

[0102] For the same or similar parts among the various embodiments in this specification, reference can be made to each other. In particular, for the device embodiments and the terminal embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method for processing a fluorescently labeled document, characterized in that, Including: Collecting an original image in response to a user's triggering operation on a marking mode; Determining a fluorescent marking area in the original image and fluorescent marking information related to the fluorescent marking area; Processing the fluorescent marking area to obtain a target image; Wherein, the marking mode includes an automatic marking mode. When the user triggers the automatic marking mode, the collecting of the original image includes: collecting a multi-spectral scanning image set of a target document at at least one scanning angle; The determining of the fluorescent marking area in the original image includes: performing fusion processing on the multi-spectral scanning image set through a deep learning model to obtain a first fusion image; determining the fluorescent marking area in the first fusion image based on the optical characteristics of the fluorescent pigment in the target document; The processing of the fluorescent marking area to obtain a target image includes: performing image enhancement processing on the marking area to obtain a target image.

2. The method according to claim 1, wherein The multi-spectral scanning image set at least includes: an ultraviolet band image set, an infrared band image set or a visible light band image set; the performing of fusion processing on the multi-spectral scanning image set through a deep learning model to obtain a first fusion image includes: Performing fusion processing on the image sets in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain a second fusion image set; Performing fusion processing on the second fusion image set to obtain the first fusion image.

3. The method according to claim 2, characterized in that, The performing of fusion on the image sets in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain a second fusion image set includes: Fusing the ultraviolet band image set based on the maximum color value of the same pixel point in the images corresponding to each scanning angle to obtain an ultraviolet band fusion image; Or, fusing the infrared band image set based on region fusion of wavelet transform to obtain an infrared band fusion image; Or, fusing the visible light band image set based on the preset weights corresponding to each scanning angle to obtain a visible light band fusion image.

4. The method according to claim 1, wherein After determining the fluorescent marking area in the first fusion image based on the optical characteristics of the fluorescent pigment in the target document, the method further includes: Determining an enhancement coefficient based on the light intensity value of the current environment, wherein the enhancement coefficient is positively correlated with the light intensity value; Calibrating the first fusion image based on the enhancement coefficient and white balance; The determining of the fluorescent marking area in the first fusion image based on the optical characteristics of the fluorescent pigment in the target document includes: Determining a candidate area for fluorescent marking in the first fusion image based on the optical characteristics of the fluorescent pigment in the target document; Determining the fluorescent marking area in the candidate area for fluorescent marking based on the deep learning model.

5. The method according to claim 1, wherein The performing of image enhancement processing on the marking area includes: Adjusting the contrast of the marking area based on the histogram equalization algorithm; And / or converting the color value of the marking area to a corresponding gray value based on a pre-stored gray mapping table.

6. The method according to claim 1, characterized in that, The annotation mode further includes a manual annotation mode. When the user triggers the manual annotation mode, determining the fluorescence annotation area in the original image includes: Responding to a triggering operation of the user based on the display screen of the original image, determining the fluorescence annotation area annotated by the user; The fluorescence annotation information related to the fluorescence annotation area includes annotation color and / or transparency. Processing the fluorescence annotation area to obtain a target image includes: Generating a corresponding annotation layer based on the fluorescence annotation area, annotation color and / or transparency; Fusing the scanned image and the annotation layer to obtain the target image.

7. The method according to claim 6, wherein Before fusing the scanned image and the annotation layer to obtain the target image, the method further includes: Adjusting the transparency of the annotation layer based on the background brightness value of the scanned image, where the transparency is negatively correlated with the background brightness value.

8. A processing device for a fluorescently labeled document, characterized in that, Including: A response module, configured to collect a multi-spectral scanned image set of a target document at at least one scanning angle in response to a triggering operation of the user on the automatic annotation mode; A fusion module, configured to perform fusion processing on the multi-spectral scanned image set through a deep learning model to obtain a first fused image; An annotation module, configured to determine a fluorescence annotation area in the first fused image based on the optical characteristics of the fluorescent pigment in the target document; An enhancement module, configured to perform image enhancement processing on the annotation area to obtain a target image.

9. An electronic device, characterized in that, Including a memory for storing computer program instructions and a processor for executing the program instructions. When the computer program instructions are executed by the processor, the electronic device is caused to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program. When the program runs, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images