A method, device, equipment and storage medium for processing fluorescent annotated documents

Abstract

Embodiments of the present application provide a method, apparatus, device, and storage medium for processing fluorescently annotated documents. The method collects a set of multispectral scanned images of a target document at at least one scanning angle; fuses the multispectral scanned images using a deep learning model to obtain a first fused image; determines the fluorescently annotated region in the first fused image based on the optical properties of the fluorescent pigment in the target document; and performs image enhancement processing on the fluorescently annotated region to obtain a target image. Image enhancement of the fluorescently annotated region can improve the printing or scanning quality of the fluorescent annotation.

Description

Technical Field

[0001] The present application relates to the field of image forming technology, and in particular to a method for processing fluorescent annotated documents and an image forming device. Background Art

[0002] In office and school settings, people often use highlighters to mark important content in documents, allowing them to quickly locate and highlight key information. However, existing copying technology has many shortcomings when processing documents with fluorescent annotations. Due to the characteristics of fluorescent pigments and the imaging principles of image-forming devices, fluorescent annotations often fade and exhibit color deviations after copying or scanning, affecting the quality of copied or scanned documents and the integrity of the information, causing significant inconvenience to users. Summary of the Invention

[0003] In view of this, the present application provides a method, apparatus, device and storage medium for processing fluorescent annotated documents, so as to solve the problem of low printing or scanning quality of fluorescent annotations in the prior art.

[0004] In a first aspect, an embodiment of the present application provides a method for processing a fluorescently annotated document, comprising:

[0005] In response to a user triggering operation on the annotation mode, acquiring an original image;

[0006] Determining a fluorescent annotated area in the original image and fluorescent annotated information related to the fluorescent annotated area;

[0007] Processing the fluorescent marked area to obtain a target image;

[0008] Wherein, the annotation mode includes an automatic annotation mode, and when the user triggers the automatic annotation mode, the acquiring of the original image includes: acquiring a multispectral scanning image set of the target document at at least one scanning angle;

[0009] Determining the fluorescent annotated area in the original image includes: performing fusion processing on the multispectral scan image set through a deep learning model to obtain a first fused image; and determining the fluorescent annotated area in the first fused image based on the optical properties of the fluorescent pigment in the target document;

[0010] The processing of the fluorescent marked area to obtain the target image includes: performing image enhancement processing on the marked area to obtain the target image.

[0011] In an optional embodiment, the multispectral scanning image set includes at least: an ultraviolet band image set, an infrared band image set, or a visible light band image set; and the fusion processing based on the multispectral scanning image set through a deep learning model to obtain the first fused image includes:

[0012] fusing the image sets in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain a second fused image set;

[0013] The second fused image set is fused to obtain the first fused image.

[0014] In an optional embodiment, fusing the image sets in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain the second fused image set includes:

[0015] Based on the maximum color value of the same pixel point in the images corresponding to each of the scanning angles, the ultraviolet band image set is fused to obtain an ultraviolet band fused image;

[0016] Alternatively, the infrared band image set is fused based on regional fusion of wavelet transform to obtain an infrared band fused image;

[0017] Alternatively, the visible light band image set is fused based on the preset weights of the respective scanning angles to obtain a visible light band fused image.

[0018] In an optional embodiment, after determining the fluorescent annotated area in the first fused image based on the optical properties of the fluorescent pigment in the target document, the method further includes:

[0019] Determining an enhancement coefficient based on the light intensity value of the current environment, wherein the enhancement coefficient is positively correlated with the light intensity value;

[0020] calibrating the first fused image based on the enhancement coefficient and white balance;

[0021] The determining of the fluorescent marked area in the first fused image based on the optical properties of the fluorescent pigment in the target document includes:

[0022] determining, based on optical properties of the fluorescent pigment in the target document, a candidate region for fluorescent annotation in the first fused image;

[0023] Based on the deep learning model, a fluorescent annotated region in the fluorescent annotated candidate region is determined.

[0024] In an optional embodiment, performing image enhancement processing on the marked area includes:

[0025] Adjusting the contrast of the marked area based on a histogram equalization algorithm;

[0026] And / or, the color value of the marked area is converted into a corresponding grayscale value based on a pre-stored grayscale mapping table.

[0027] In an optional embodiment, the annotation mode further includes a manual annotation mode. When the user triggers the manual annotation mode, determining the fluorescent annotation area in the original image includes:

[0028] In response to a triggering operation by the user based on the display screen of the original image, determining a fluorescent marking area marked by the user;

[0029] The fluorescent annotation information related to the fluorescent annotation area includes annotation color and / or transparency, and the processing of the fluorescent annotation area to obtain a target image includes:

[0030] Generate a corresponding annotation layer based on the fluorescent annotation area, annotation color and / or transparency;

[0031] The scanned image and the annotated layer are fused to obtain the target image.

[0032] In an optional embodiment, before fusing the scanned image and the annotated layer to obtain the target image, the method further includes:

[0033] The transparency of the annotation layer is adjusted based on a background brightness value of the scanned image, wherein the transparency is negatively correlated with the background brightness value.

[0034] In a second aspect, an embodiment of the present application provides a device for processing fluorescently annotated documents, comprising:

[0035] a response module, configured to acquire a multispectral scanning image set of a target document at at least one scanning angle in response to a user triggering operation on the automatic annotation mode;

[0036] A fusion module, configured to perform fusion processing based on the multispectral scanning image set through a deep learning model to obtain a first fused image;

[0037] a marking module, configured to determine a fluorescent marking area in the first fused image based on the optical properties of the fluorescent pigment in the target document;

[0038] The enhancement module is used to perform image enhancement processing on the marked area to obtain a target image.

[0039] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the electronic device is triggered to execute any of the methods described in the first aspect above.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any method described in the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any one of the methods described in the first aspect.

[0042] Using the solution provided in an embodiment of the present application, in response to a user triggering an annotation mode, an original image is captured; a fluorescent annotation area in the original image and fluorescent annotation information related to the fluorescent annotation area are determined; and the fluorescent annotation area is processed to obtain a target image. The annotation mode includes an automatic annotation mode. When the user triggers the automatic annotation mode, capturing the original image includes: capturing a multispectral scanning image set of a target document at at least one scanning angle; determining the fluorescent annotation area in the original image includes: performing fusion processing based on the multispectral scanning image set using a deep learning model to obtain a first fused image; determining the fluorescent annotation area in the first fused image based on the optical properties of the fluorescent pigment in the target document; and processing the fluorescent annotation area to obtain a target image includes: performing image enhancement processing on the annotation area to obtain the target image. By performing image enhancement processing on the fluorescent annotation area, the printing or scanning quality of the fluorescent annotation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A schematic diagram of a method for processing fluorescent annotated documents provided in an embodiment of the present application Figure 1 ;

[0045] Figure 2 A schematic diagram of a method for processing fluorescent annotated documents provided in an embodiment of the present application Figure 2 ;

[0046] Figure 3 A schematic diagram of a method for processing fluorescent annotated documents provided in an embodiment of the present application Figure 3 ;

[0047] Figure 4 A schematic diagram of a method for processing fluorescent annotated documents provided in an embodiment of the present application Figure 4 ;

[0048] Figure 5 A schematic diagram of the structure of a device for processing fluorescent annotated documents provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0051] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0052] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0053] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0054] When an image forming device performs tasks such as scanning and copying, if the original document contains fluorescent annotations, conventional scanning and copying often results in fading, color deviation, and other issues in the target document. To address this issue, embodiments of the present application provide a method for processing fluorescently annotated documents, which is used to improve the scanning quality of fluorescent annotations in the target document and the information integrity of the target document.

[0055] Figure 1 A schematic diagram of a method for processing fluorescent annotated documents provided in an embodiment of the present application Figure 1 , the method may be performed by an image forming device, such as Figure 1 As shown, the method may include:

[0056] Step 101: In response to a user triggering an annotation mode, an original image is captured.

[0057] In addition to the normal mode, the image forming device of the embodiment of the present application is configured with an annotation mode for processing fluorescent annotations. If the original document contains fluorescent annotations, the user can select the corresponding annotation mode when issuing the job task.

[0058] Step 102: Determine the fluorescent annotated area in the original image and the fluorescent annotated information related to the fluorescent annotated area.

[0059] Step 103: Process the fluorescent marked area to obtain a target image.

[0060] In conventional copying or scanning mode, after the image forming device acquires the original image, it executes subsequent processes based on the original image. For example, in conventional copying, the image forming device immediately outputs a copy document after acquiring the original image. In annotation mode, however, after acquiring the original image, the image forming device first identifies the fluorescently annotated areas within the original image, along with relevant fluorescent annotation information, such as the color of the fluorescent annotation. The image forming device then processes the fluorescently annotated areas, performing processes such as contrast enhancement and color replacement, to highlight the fluorescently annotated areas and ultimately produce the target image. The target image is then used as the basis for subsequent scanning or copying processes.

[0061] In an optional embodiment, the marking mode may include: automatic marking mode and manual marking mode. When the user selects the automatic marking mode, the processing flow of the image forming device may refer to Figure 2 , mainly including:

[0062] Step 201 : In response to a user triggering an automatic annotation mode, a multispectral scanning image set of a target document at at least one scanning angle is collected.

[0063] When an image forming device scans a target document, data from multiple scans at the same angle will differ, as will data from scans at different angles. To maximize scanning quality, the image forming device of an embodiment of the present application captures multispectral images of the target document from multiple scanning angles, generating a corresponding multispectral scan image set. The multispectral scan image set may include an ultraviolet band image set, an infrared band image set, and a visible light band image set.

[0064] For example, in actual operation, a user places a target document with fluorescent annotations on a specially designed image acquisition platform. The camera of the imaging device automatically adjusts its position and focal length to capture images of the target document from three different angles (e.g., 0°, 45°, and 90°) in wavelength bands such as ultraviolet light (300-400nm), visible light (400-700nm), and near-infrared light (700-1000nm). An image is captured from each angle and wavelength band. Multi-angle capture in the ultraviolet band produces a set of ultraviolet band images, multi-angle capture in the infrared band produces a set of infrared band images, and multi-angle capture in the visible light band produces a set of visible light band images. Optionally, images of the target document can be captured from more or fewer angles (e.g., 0°, 30°, 60°, 90°, etc.) and wavelength bands, although this is not a limitation in this embodiment of the present application.

[0065] In the embodiment of the present application, by performing multi-angle and multi-spectral image acquisition on the target document, image details can be obtained from multiple aspects, the quality of image acquisition can be improved, and problems such as missing information in the fluorescently annotated content can be avoided.

[0066] Step 202: Based on the multispectral scanning image set, a fusion process is performed through a deep learning model to obtain a first fused image.

[0067] Specifically, the image forming device can first perform preliminary fusion processing on images of the same angle or the same band in the multispectral scanning image set through a deep learning model, and then further fuse the images obtained after the preliminary fusion processing to obtain a first fused image. It can also directly perform fusion processing on the multispectral scanning image set through a deep learning model to obtain a first fused image.

[0068] In an optional embodiment, the image forming device fuses the image sets of the ultraviolet band, the infrared band, and the visible light band, respectively, to obtain second fused image sets corresponding to the respective spectral bands. The image forming device then fuses the second fused image sets to obtain the first fused image.

[0069] Specifically, the second fused image set may include: ultraviolet band fused images, infrared band fused images and visible light band fused images. When the same band image set is fused, the fusion processing can be performed according to the preset weights of the images at each angle in the image set to obtain ultraviolet band fused images, infrared band fused images and visible light band fused images; when the second fused image set is fused, due to the optical properties of the fluorescent pigment, the images of the fluorescent marked areas and the non-fluorescent marked areas in the ultraviolet band fused images and the infrared band fused images are distinguishable. The images of the fluorescent marked areas in the ultraviolet band fused images and the infrared band fused images can be extracted and fused with the visible light band fused image.

[0070] In an embodiment of the present application, by first fusing the same band image set to obtain a second fused image set, and then fusing the second fused image set, targeted fusion strategies can be adopted for different band image sets. At the same time, the first fused image combines the image of the fluorescent annotated area in the specific band image and the visible light band fused image, which facilitates the subsequent accurate determination of the fluorescent annotated area and makes the first fused image closer to the target document.

[0071] In an optional embodiment, the image fusion process may include: (1) the image forming device fuses the ultraviolet band image set based on the maximum color value of the same pixel point in the image corresponding to each scanning angle to obtain the ultraviolet band fusion image. Specifically, the fluorescent label will produce a fluorescent effect under ultraviolet light irradiation, and the maximum color value of the same pixel point in the image corresponding to each scanning angle is used as the color value of the pixel point in the ultraviolet band fusion image to highlight the fluorescent characteristics and enhance the distinction between fluorescent pigments and ordinary pigments. (2) the image forming device fuses the infrared band image set based on regional fusion of wavelet transform to obtain the infrared band fusion image. Specifically, the image forming device decomposes the infrared band image set into a low-frequency part and a high-frequency part based on wavelet transform. The low-frequency part can use the 0-degree image and the high-frequency part can use the 45-degree image. The infrared band fusion image is obtained by fusion, so as to achieve the effect of using the 0-degree image to retain the overall structure of the low-frequency part and using the 45-degree image to enhance the local texture of the high-frequency part. (3) The image forming device fuses the visible light band image set based on the preset weights of each scanning angle to obtain a visible light band fused image. Specifically, the preset weight of the 0-degree image can be greater than the preset weights of the images at other angles to preserve the color at the front view and supplement the details at the side view. For example, the preset weight of the 0-degree image is set to 0.5, the preset weight of the 45-degree image is set to 0.3, and the preset weight of the 90-degree image is set to 0.2.

[0072] In the embodiments of this application, by taking the maximum color value of the same pixel at multiple angles in the ultraviolet band, fluorescent properties can be highlighted, enhancing the distinction between fluorescent pigments and ordinary pigments. Fusion of the infrared bands based on wavelet transforms can simultaneously maintain the overall image structure while enhancing local texture. Weighted fusion of the visible light bands can preserve color from the front view and details from the side view.

[0073] Step 203 : determining a fluorescent annotated area in the first fused image based on the optical properties of the fluorescent pigment in the target document.

[0074] Specifically, fluorescent pigments exhibit a unique optical behavior: they absorb specific wavelengths of light (usually shorter wavelengths, such as ultraviolet or blue) and re-emit (reflect) light at longer wavelengths. This property makes fluorescent pigments appear very bright, even "glowing" under certain conditions. Ordinary pigments, on the other hand, typically do not exhibit this fluorescent effect. They simply reflect specific wavelengths of light, rather than absorbing short wavelengths and re-emitting longer wavelengths.

[0075] The image forming device can emit light of a specific wavelength (e.g., ultraviolet light) to illuminate the target document. After scanning the document, the ultraviolet fluorescent pigment in the fluorescently labeled area absorbs the short-wavelength light and re-emit light of a longer wavelength, forming an image with prominent colors such as bright green and orange. The ordinary pigment area forms an image with dark colors, etc. Therefore, there is a significant difference between the fluorescently labeled area and the ordinary pigment area in the ultraviolet band fused image. The fluorescently labeled area in the ultraviolet band fused image can be determined based on color, and the fluorescently labeled area in the first fused image can be determined. Furthermore, if the fluorescently labeled area includes a near-infrared fluorescent pigment, after infrared light irradiation, the near-infrared fluorescent labeled area will emit a wavelength of 800-900 nanometers to form a high-contrast fluorescent image. Ordinary pigments, due to their lack of fluorescence effect, form a dark or non-fluorescent image. The fluorescently labeled area in the infrared band fused image can be determined based on color, and the fluorescently labeled area in the first fused image can be determined.

[0076] In an optional embodiment, the step of determining the fluorescent annotation area by the image forming device may include: (1) determining the candidate area for fluorescent annotation in the first fused image based on the optical properties of the fluorescent pigment in the target document. Specifically, based on the color difference between the fluorescent annotation area and the ordinary pigment area caused by the optical properties of the fluorescent pigment, a fluorescent color threshold can be set to determine the area in the ultraviolet band fused image and / or the infrared band fused image whose color value is greater than or equal to the fluorescent color threshold, and the area corresponding to the area in the first fused image is used as the candidate area for fluorescent annotation. (2) determining the fluorescent annotation area in the candidate area for fluorescent annotation based on a deep learning model. Specifically, the deep learning model can be an image segmentation model such as U-Net, Mask R-CNN, etc., and further identifying the fluorescent annotation area in the candidate area for fluorescent annotation in the first fused image based on the deep learning model to eliminate background interference.

[0077] In an embodiment of the present application, the image forming device sequentially identifies the fluorescent labeled area based on the optical properties of the fluorescent pigment and the deep learning model, which can improve the accuracy of the fluorescent labeled area identification and provide a guarantee for subsequent image processing.

[0078] Step 204: perform image enhancement processing on the marked area to obtain a target image.

[0079] Specifically, the image enhancement processing may be to increase the contrast of the marked area, replace the color of the marked area with a brighter color, etc., so as to achieve the effect of highlighting the marked area.

[0080] In an optional embodiment, the image forming device may adjust the contrast of the annotated area based on a histogram equalization algorithm, or may convert the color values ​​of the annotated area to corresponding grayscale values ​​based on a pre-stored grayscale mapping table. Different image enhancement methods may also be employed for different tasks. For example, in a color copying or scanning task, the image forming device may enhance the image of the annotated area by adjusting the contrast. In another example, in a black-and-white copying or scanning task, the image forming device may enhance the image of the annotated area by converting grayscale values.

[0081] During contrast adjustment, the image forming device first counts the number of pixels at each grayscale level in the image, forming a histogram with grayscale levels on the horizontal axis and pixel count on the vertical axis. The original histogram is then mapped to a more evenly distributed histogram, balancing the number of pixels across all grayscale levels. This method stretches the grayscale values ​​of pixels that were originally concentrated in darker or brighter areas across a wider grayscale range, thereby enhancing overall contrast. For example, if the original image's pixel grayscale values ​​were concentrated between 100 and 150 (darker), after equalization, they might be distributed between 0 and 255, making dark details more distinct.

[0082] To avoid excessive histogram equalization enhancement (e.g., noise amplification, localized overexposure), or insufficient enhancement (e.g., lack of effect), the image forming device can adaptively adjust the equalization parameters based on the area and grayscale distribution of the fluorescently marked region. If the marked region is small, the enhancement intensity may need to be limited to avoid noise interference; if the region is large, the enhancement intensity can be increased to cover more detail. If grayscale is concentrated, such as in bright fluorescent areas, the image histogram can be nonlinearly transformed during equalization to compress the highlights and avoid overexposure. If grayscale is dispersed, the image histogram can be linearly expanded to evenly stretch it and improve overall contrast.

[0083] In the embodiment of the present application, based on the histogram equalization algorithm and adaptive adjustment, the contrast of the fluorescent marked area can be reasonably adjusted, which can not only distinguish the fluorescent marked area from the ordinary pigment area, but also avoid background noise interference or the problem of excessive brightness of the fluorescent area.

[0084] Fluorescent labels such as yellow, pink, and green have high distinction in color mode, but when copied or scanned in black and white, they may become blurred due to the influence of the grayscale conversion algorithm. For example, a yellow fluorescent label may be close to white in black and white mode, causing the label to almost disappear. Therefore, the image forming device can pre-define a grayscale mapping table to map fluorescent labels of different colors to preset grayscale values, rather than relying on the default grayscale conversion algorithm. For example, a yellow fluorescent label can be converted to a darker gray, such as a grayscale value of 100, to ensure distinction from the white background. A pink fluorescent label can be converted to an even darker gray, such as a grayscale value of 80, to improve contrast. Other background colors such as white can be converted to pure white with a grayscale value of 255.

[0085] In the embodiment of the present application, after a specific grayscale mapping, the fluorescent marking will appear darker gray in a black and white copy or scan, while the background remains white or light gray, thereby ensuring high visibility of the fluorescent marking.

[0086] In an optional embodiment, after the image forming device identifies the fluorescently annotated area in the first fused image, it may further calibrate the first fused image. First, the image forming device determines an enhancement factor based on the current ambient light intensity, where the enhancement factor and the light intensity are positively correlated. The image forming device then calibrates the first fused image based on the enhancement factor and white balance.

[0087] Specifically, the enhancement factor can be calculated based on light intensity using the formula: Enhancement factor = K × exp(ambient light intensity value / 100), where exp is an exponential function and ambient light intensity value refers to the real-time detected light intensity, expressed in lux. The enhancement factor is used to adjust image processing parameters, such as RGB values, to accommodate images under varying lighting conditions. K is a scaling factor used to adjust the enhancement factor scale to suit actual needs. The base value K can be selected between 0.1 and 0.5, depending on the desired range of the enhancement factor. Larger K values ​​result in more pronounced changes in the enhancement factor under bright lighting conditions; smaller K values ​​result in more gradual changes.

[0088] Based on the enhancement coefficient and white balance, the process of calibrating the first fused image may include: (1) extracting RGB values: extracting the RGB values ​​of the blank area from the first fused image; (2) calculating the average value: calculating the average value of the three color channels R, G, and B of all pixels in the blank area; (3) determining the gain value: dividing the average value of each color channel by the sum of the average values ​​of the three channels to obtain a normalized ratio, which is the gain value; (4) applying the gain value: multiplying the initial value of each color channel by the enhancement coefficient and the gain value to obtain the value of each color channel after calibration. For example, for the R channel: output R value = initial R value Enhancement coefficient R channel gain value.

[0089] In an embodiment of the present application, the image forming device calibrates the first fused image based on the enhancement coefficient and white balance, so that each color in the image is close to the standard color under different light intensities, ensuring that the content is clearly visible and the color is accurate, and making the first fused image closer to the target document.

[0090] In an optional embodiment, the above-mentioned deep learning model may be an improved U-Net model, with the following specific improvements:

[0091] (1) Improve the model architecture. First, the traditional RGB input channels are expanded to multispectral input channels such as infrared band channels, visible light band channels, and ultraviolet band channels. Each band channel can also include multiple channels at different angles to improve the discrimination of fluorescent annotations. For example, when collecting images of target documents in the infrared band, visible light band, and ultraviolet band from three angles (0°, 45°, and 90°), the multispectral input channels include: three channels at 0°, 45°, and 90° for the infrared band, three channels at 0°, 45°, and 90° for the visible light band, and three channels at 0°, 45°, and 90° for the ultraviolet band, for a total of nine channels. Second, introduce the attention mechanism. Embed the Convolutional Block Attention Module (CBAM) in the jump connection of the U-Net model to enhance the ability to capture small areas of fluorescent annotations.

[0092] (2) Optimize the model training strategy. First, increase the model training tasks. The main task can be to segment the fluorescent labeling area, and the auxiliary task can be to classify different fluorescent pigments, such as different brands of fluorescent pens and different colors of fluorescent pigments, so that the fluorescent labels of different fluorescent pigments can be processed in a targeted manner in the future, thereby improving the generalization of the model. First, increase the scenes covered by the training data, simulate different lighting conditions such as weak light, strong light, etc., and different paper background colors such as white, yellow, gray, etc., to generate training data for training to cover more scenes.

[0093] (3) Lightweight deployment. Quantizing the model from 32-bit floating point (FP32) to 8-bit integer (INT8) reduces the number of parameters, effectively improving the running speed and meeting the computing power limitations of image forming equipment.

[0094] In the embodiment of the present application, the U-Net model obtained through the above-mentioned improvement recognizes the fluorescent labeled area and can automatically extract features from the data without manual design to achieve high-precision segmentation of the fluorescent labeled area. It has strong adaptability to different lighting, backgrounds, and colors. It can also simultaneously process multiple tasks such as segmentation and classification, accurately identify the fluorescent labeled area, and perform targeted processing on the fluorescent labels of different fluorescent pigments.

[0095] The solution provided in the embodiment of the present application is to collect a set of multispectral scanned images of a target document at at least one scanning angle in response to a user triggering an automatic annotation mode; perform fusion processing based on the multispectral scanned image set through a deep learning model to obtain a first fused image; determine the fluorescent annotation area in the first fused image based on the optical properties of the fluorescent pigment in the target document; and perform image enhancement processing on the annotation area to obtain a target image. By performing multi-angle and multi-spectral image acquisition on the target document, the quality of image acquisition is improved, the fluorescent annotation area is accurately identified based on the optical properties of the fluorescent pigment and a deep learning model, and the image enhancement processing is performed on the annotation area to achieve the effect of highlighting the annotation area, so that the obtained target image is close to the target document while ensuring high visibility of the fluorescent annotation. The target image is used as a scanned image of the target document, or the target image is printed as a copy of the target document, which can improve the printing or scanning quality of the fluorescent annotation.

[0096] In an optional embodiment, the annotation mode also includes a manual annotation mode. When the user triggers the manual annotation mode, the processing flow of the image forming device can refer to Figure 3 , mainly including:

[0097] Step 301 : In response to a triggering operation by a user based on a display screen of an original image, a fluorescent annotation area annotated by the user and fluorescent annotation information related to the fluorescent annotation area are determined.

[0098] The fluorescent annotation information associated with the fluorescently annotated area includes the annotation color and / or transparency. When the user selects manual annotation mode, the touchscreen of the imaging device displays the original image. The user can then select the annotation area using the touchscreen or perform annotation using a mobile phone or other terminal device connected to the imaging device. During the annotation process, the user can select fluorescent annotation information such as the annotation color and transparency.

[0099] Optionally, text and table recognition algorithms such as optical character recognition algorithms can be used to identify text and table areas to assist users in quick annotation. Multiple annotation methods can be provided, such as box selection, lasso, magic wand, etc.

[0100] Step 302: Generate a corresponding annotation layer based on the fluorescent annotation area, annotation color and / or transparency.

[0101] Optionally, when the user does not select a label color or transparency, a corresponding label layer may be generated based on a preset default color such as yellow or a default transparency such as 50%.

[0102] Step 303: Fuse the scanned image and the annotation layer to obtain a target image.

[0103] Specifically, each pixel of the scanned image and the annotation layer can be fused to obtain the target image. For the R, G, and B values ​​of the same pixel in the scanned image, the annotation layer, and the target image, the value of the target image = the value of the scanned image × (1-preset weight) + the value of the annotation layer × the preset weight. Among them, the preset weight can be set according to user needs. The larger the preset weight, the greater the impact of the annotation color and transparency; conversely, the smaller the preset weight, the greater the impact on the scanned image. Optionally, the image forming device can store the annotation color and transparency and preset weight selected by the user. When copying or scanning multiple pages, each page of the image is automatically annotated according to the fluorescent annotation area selected by the user.

[0104] In an optional embodiment, before fusing the scanned image and the annotation layer to generate the target image, the image forming device may adjust the transparency of the annotation layer based on the background brightness of the scanned image. The transparency is negatively correlated with the background brightness. Specifically, the adjustment can be made according to the following formula: Adjusted transparency = Original transparency × (1 - Background brightness / 255).

[0105] In an embodiment of the present application, by configuring a marking tool in an image forming device or a terminal device, a user can manually mark a fluorescent marking area and preview the marking effect in real time. By performing image fusion pixel by pixel and automatically adjusting the marking transparency according to the background brightness, a target image with complete information can be obtained, avoiding color distortion of the fluorescent marking that makes it difficult to distinguish. The target image can be used as a scanned image of the target document, or the target image can be printed as a copy of the target document, thereby improving the quality of copying or scanning fluorescent marked documents.

[0106] Figure 4 A schematic diagram of a method for processing fluorescent annotated documents provided in an embodiment of the present application Figure 4 ,like Figure 4 As shown, the method may include:

[0107] Step 401: The user selects a marking mode.

[0108] The marking mode includes: automatic marking mode and manual marking mode. When the user selects the manual marking mode, the process proceeds to step 402 . When the user selects the automatic marking mode, the process proceeds to step 405 .

[0109] Step 402: Acquire the original image, and the user selects the fluorescent annotation area and fluorescent annotation information in the original image.

[0110] Fluorescent annotation information includes annotation color and transparency.

[0111] Step 403: Generate a labeling layer.

[0112] Generate corresponding annotation layers based on the fluorescent annotation area, annotation color, and transparency.

[0113] Step 404: fuse the original image and the annotation layer to obtain a target image.

[0114] Step 405: Acquire multi-spectral and multi-angle original images.

[0115] By collecting multi-angle images in the ultraviolet, infrared and visible light bands, the integrity of the original image information can be ensured.

[0116] Step 406: Adjust the original image according to the light intensity value and white balance.

[0117] Calibrating raw images using light intensity values ​​and white balance ensures content is visible and colors are accurate.

[0118] Step 407: Identify the fluorescent marked area.

[0119] Based on the optical properties of fluorescent pigments, the deep learning model is used to identify fluorescent labeled areas, which can improve the accuracy of fluorescent labeled area identification and provide guarantee for subsequent image processing.

[0120] Step 408: Perform image enhancement processing on the fluorescent marked area to obtain a target image.

[0121] In color mode, image enhancement can be achieved by adjusting contrast, distinguishing fluorescent annotation areas from ordinary areas while avoiding background noise or overly bright fluorescent areas. In black and white mode, image enhancement can be achieved through a specific grayscale mapping process. After this grayscale mapping, fluorescent annotations will appear darker gray in black and white copies, while the background remains white or light gray, ensuring high visibility of fluorescent annotations.

[0122] In order to implement the above method, the image forming device of the embodiment of the present application is configured with corresponding hardware modules, which may include: a multi-spectral scanning head that supports multi-angle scanning in the visible light band, infrared band and ultraviolet band; an ambient light sensor that detects the intensity of the scanning ambient light in real time; and a touch interactive screen that supports multi-touch and pressure sensing.

[0123] Figure 5 This is a schematic diagram of the structure of a device for processing fluorescently annotated documents provided in an embodiment of the present application. Figure 5 As shown, the device may include:

[0124] The response module 510 is configured to collect a set of multispectral scanned images of the target document at at least one scanning angle in response to a user triggering operation on the automatic annotation mode.

[0125] The fusion module 520 is configured to perform fusion processing based on the multispectral scanning image set through a deep learning model to obtain a first fused image.

[0126] The annotation module 530 is configured to determine a fluorescent annotation area in the first fused image based on the optical properties of the fluorescent pigment in the target document.

[0127] The enhancement module 540 is used to perform image enhancement processing on the marked area to obtain a target image.

[0128] Corresponding to the above embodiments, the present application also provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 600 may include: a processor 601, a memory 602, and a communication unit 603. These components communicate via one or more buses. Those skilled in the art will understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiments of the present application. It can be a bus structure or a star structure, and can also include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0129] The communication unit 603 is configured to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices, or send user data to other devices.

[0130] The processor 601 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs, instructions, and / or modules stored in the memory 602, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 601 can include only a central processing unit (CPU). In the embodiment of the present application, the CPU can be a single computing core or multiple computing cores.

[0131] The memory 602 is used to store execution instructions of the processor 601. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0132] When the execution instructions in the memory 602 are executed by the processor 601 , the electronic device 600 is enabled to execute part or all of the steps in the above embodiments.

[0133] In a specific implementation, this application also provides a computer storage medium, wherein the computer storage medium may store a program that, when executed, may include some or all of the steps of each embodiment of the method for processing fluorescently annotated documents provided in this application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0134] In a specific implementation, the present application also provides a computer program product, wherein the computer program product includes executable instructions, which, when executed on a computer, enable the computer to execute some or all of the steps in each embodiment of the method for processing fluorescent annotated documents provided in the present application.

[0135] An embodiment of the present application further provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable the computer to execute the method for processing fluorescent annotated documents provided in an embodiment of the present application.

[0136] The aforementioned non-transitory computer-readable storage medium may take the form of any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0137] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0138] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, etc., or any suitable combination of the foregoing.

[0139] Those skilled in the art will clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present application, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0140] In this specification, reference can be made to the same or similar parts between the various embodiments. In particular, for the device embodiment and the terminal embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

Claims

1. A method for processing fluorescent annotated documents, characterized in that: include: In response to a user triggering operation on the annotation mode, acquiring an original image; Determining a fluorescent annotated area in the original image and fluorescent annotated information related to the fluorescent annotated area; Processing the fluorescent marked area to obtain a target image; Wherein, the annotation mode includes an automatic annotation mode, and when the user triggers the automatic annotation mode, the acquiring of the original image includes: acquiring a multispectral scanning image set of the target document at at least one scanning angle; Determining the fluorescent annotated area in the original image includes: performing fusion processing on the multispectral scan image set through a deep learning model to obtain a first fused image; and determining the fluorescent annotated area in the first fused image based on the optical properties of the fluorescent pigment in the target document; The processing of the fluorescent marked area to obtain the target image includes: performing image enhancement processing on the marked area to obtain the target image; The multispectral scanning image set includes at least: an ultraviolet band image set, an infrared band image set, or a visible light band image set; the fusion processing based on the multispectral scanning image set through the deep learning model to obtain the first fused image includes: fusing the image sets in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain a second fused image set; performing a fusion process on the second fused image set to obtain the first fused image; After determining the fluorescent annotated area in the first fused image based on the optical properties of the fluorescent pigment in the target document, the method further includes: Determining an enhancement coefficient based on the light intensity value of the current environment, wherein the enhancement coefficient is positively correlated with the light intensity value; calibrating the first fused image based on the enhancement coefficient and white balance; The determining of the fluorescent marked area in the first fused image based on the optical properties of the fluorescent pigment in the target document includes: determining, based on optical properties of the fluorescent pigment in the target document, a candidate region for fluorescent annotation in the first fused image; Based on the deep learning model, a fluorescent annotated region in the fluorescent annotated candidate region is determined.

2. The method according to claim 1, characterized in that The fusing the image sets in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain a second fused image set includes: Based on the maximum color value of the same pixel point in the images corresponding to each of the scanning angles, the ultraviolet band image set is fused to obtain an ultraviolet band fused image; Alternatively, the infrared band image set is fused based on regional fusion of wavelet transform to obtain an infrared band fused image; Alternatively, the visible light band image set is fused based on the preset weights of the respective scanning angles to obtain a visible light band fused image.

3. The method according to claim 1, characterized in that The performing image enhancement processing on the marked area includes: Adjusting the contrast of the marked area based on a histogram equalization algorithm; And / or, the color value of the marked area is converted into a corresponding grayscale value based on a pre-stored grayscale mapping table.

4. The method according to claim 1, wherein The annotation mode also includes a manual annotation mode. When the user triggers the manual annotation mode, determining the fluorescent annotation area in the original image includes: In response to a triggering operation by the user based on the display screen of the original image, determining a fluorescent marking area marked by the user; The fluorescent annotation information related to the fluorescent annotation area includes annotation color and / or transparency, and the processing of the fluorescent annotation area to obtain a target image includes: Generate a corresponding annotation layer based on the fluorescent annotation area, annotation color and / or transparency; The scanned image and the annotated layer are fused to obtain the target image.

5. The method according to claim 4, characterized in that Before fusing the scanned image and the annotated layer to obtain the target image, the method further includes: The transparency of the annotation layer is adjusted based on a background brightness value of the scanned image, wherein the transparency is negatively correlated with the background brightness value.

6. A device for processing fluorescently annotated documents, characterized in that: include: a response module, configured to acquire a multispectral scanning image set of a target document at at least one scanning angle in response to a user triggering operation on the automatic annotation mode; A fusion module, configured to perform fusion processing based on the multispectral scanning image set through a deep learning model to obtain a first fused image; a marking module, configured to determine a fluorescent marking area in the first fused image based on the optical properties of the fluorescent pigment in the target document; An enhancement module, configured to perform image enhancement processing on the marked area to obtain a target image; The multispectral scanning image set includes at least: an ultraviolet band image set, an infrared band image set, or a visible light band image set; the fusion processing based on the multispectral scanning image set through the deep learning model to obtain the first fused image includes: fusing the image sets in the ultraviolet band image set, the infrared band image set, and the visible light band image set to obtain a second fused image set; performing a fusion process on the second fused image set to obtain the first fused image; The apparatus further includes: a calibration module configured to determine an enhancement coefficient based on a light intensity value of a current environment, wherein the enhancement coefficient is positively correlated with the light intensity value; and calibrate the first fused image based on the enhancement coefficient and white balance; The determining of the fluorescent marked area in the first fused image based on the optical properties of the fluorescent pigment in the target document includes: determining, based on optical properties of the fluorescent pigment in the target document, a candidate region for fluorescent annotation in the first fused image; Based on the deep learning model, a fluorescent annotated region in the fluorescent annotated candidate region is determined.

7. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 5.

Images