Document processing method and computer program product

Abstract

According to the document processing method and the computer program product provided by the invention, the document image is segmented into different document elements according to the semantic type of the document image, so that the integrity of the internal logic structure of the document image is ensured. Meanwhile, the importance score of each document element is determined, and the most suitable coding strategy is matched for each document element based on the importance score and the semantic type of the document element. Therefore, the compression degree can be determined according to the importance degree of the document elements, and the subjective visual quality can be ensured while the compression rate is greatly improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer data compression technology, and in particular to a document processing method and a computer program product. Background Technology

[0002] With the advancement of digitalization, massive amounts of documents are stored, transmitted, and processed in the form of digital images. Document compression technology is crucial for effectively conserving storage space and network bandwidth. However, current compression processes often "flatten" documents into a series of image layers, resulting in the loss of semantic information. Furthermore, fixed compression algorithms are typically used for elements of the same type. For example, areas identified as "images," regardless of whether their content is a smooth-toned photograph or a line drawing or technical drawing containing a large amount of high-frequency detail, may all be uniformly compressed using JPEG.

[0003] Therefore, there is an urgent need for a method that can overcome the above-mentioned defects, achieve better visual quality, and preserve the original semantic structure of the document. Summary of the Invention

[0004] This disclosure provides a document processing method and a computer program product. It determines the importance score of different document elements and uses these scores to determine different compression strategies for each element. Thus, the compression level is determined by the importance of the document elements, significantly improving the compression ratio while maintaining subjective visual quality.

[0005] In one aspect, this embodiment provides a document processing method, the method comprising: segmenting a document image into multiple document elements based on the semantic category of the document image in a file to be compressed; determining importance scores for the multiple document elements, the importance scores being used to reflect the degree of attention paid to the document elements; determining an encoding strategy for each document element based on the semantic category and importance score of each document element; and compressing each document element in the document image based on the encoding strategy of each document element.

[0006] In embodiments of this disclosure, segmenting a document image into multiple document elements based on the semantic category of the document image in the file to be compressed includes: when a user triggers a document upload control on a terminal device, obtaining a file to be compressed, the file to be compressed including at least one page of document images; when a user triggers a document compression control on a terminal device, segmenting the document image into multiple document elements based on the semantic category of the document images in the file to be compressed; or, after an image forming device scans and obtains a file to be compressed, when a user triggers a document compression control on an image forming device, segmenting the document image into multiple document elements based on the semantic category of the document images in the file to be compressed.

[0007] In embodiments of this disclosure, after segmenting a document image into multiple document elements based on the semantic category of the document image in the file to be compressed, the method further includes: displaying the semantic category of each document element in the document image in a semantic interface; receiving a user's instruction to change the semantic category of any document element in the semantic interface, changing the semantic category of the document element, and using the changed semantic category as the semantic category of the document element.

[0008] In embodiments of this disclosure, determining the importance scores of multiple document elements includes: determining the attention coefficient of each document element based on a visual saliency model; and determining the importance score of each document element based on the attention coefficient and the bounding box of each document element.

[0009] In embodiments of this disclosure, after determining the attention coefficient of each document element, the method further includes: displaying the attention coefficient of each document element in an attention interface; receiving a user's instruction to change the attention coefficient of any document element in the attention interface, changing the attention coefficient of the document element, and dynamically adjusting the attention coefficients of other document elements; and using the changed attention coefficient of each document element as its respective attention coefficient.

[0010] In embodiments of this disclosure, an encoding strategy for each document element is determined based on its semantic category and importance score. This includes: if the document element is a vectorizable document element, it is vectorized to compress the document element; vectorizable document elements include at least text-based document elements and graphic-based document elements; or if the document element is an image-based document element, its color complexity is determined to establish an encoding strategy for the image-based document element.

[0011] In embodiments of this disclosure, if a document element is a vectorizable document element, the document element is vectorized and encoded, including: if the vectorizable document element is a text-type document element, extracting the text information of the text-type document element, wherein the text information includes at least one or more of the following: text content, font, font size, color, and position; and saving the text information as vector data.

[0012] In embodiments of this disclosure, if the document element is a vectorizable document element, the document element is vectorized and encoded, including: when the vectorizable document element is a graphic document element, extracting the wireframe information and / or text information of the graphic document element, wherein the wireframe information includes at least the wireframe row and column relationship, and the text information includes at least one or more of the following: text content, font, font size, color, and position; and storing structured data, wherein the structured data is used to describe the wireframe information and / or text information.

[0013] In embodiments of this disclosure, determining the color complexity of image-type document elements to determine the encoding strategy for image-type document elements includes: when the color complexity is low, performing implicit vectorization compression on image-type document elements based on importance scores; when the color complexity is high, determining whether the image-type document element belongs to a photographic image or a line image, and determining the encoding strategy for image-type document elements based on the determination result and importance scores.

[0014] In embodiments of this disclosure, after determining the encoding strategy for each document element based on its semantic category and importance score, the method further includes: displaying the encoding strategy corresponding to each document element in a strategy interface; receiving a user's strategy change instruction for any document element in the strategy interface, changing the encoding strategy of the document element, and using the changed encoding strategy as the encoding strategy for the document element.

[0015] In embodiments of this disclosure, the method further includes: when the file to be compressed includes multiple pages of document images, comparing multiple document elements included in the current document image with multiple document elements included in the previous document image for any two adjacent pages of document images; if the current document image and the previous document image have completely identical document elements, then when compressing the current document image, storing reference pointer information corresponding to the completely identical document elements, the reference pointer information being used to point to the completely identical document elements in the previous document image; if the current document image and the previous document image have partially identical document elements, then when compressing the current document image, storing difference data corresponding to the partially identical document elements; wherein, the difference data is used to represent the difference between the partially identical document elements in the current document image and in the previous document image.

[0016] In the embodiments of this disclosure, after compressing each document element in the document image based on the encoding strategy of each document element, the method further includes: generating a structure directory file based on the association relationship of multiple document elements in the file to be compressed; the structure directory file includes at least one or more of the following: the size of each page of document image, the identifiers, semantic categories, bounding boxes, levels, reference pointer information, and logical relationships between the multiple document elements in each page of document image; and generating a decompressed file package based on the structure directory file and the compressed file of the file to be compressed, wherein the compressed file contains the compressed data of all document elements of all document images in the file to be compressed.

[0017] In embodiments of this disclosure, the method further includes: opening the decompressed file package using a specified reader, and displaying each document in the file to be compressed based on the structure directory file and the compressed file; receiving user editing instructions for the documents, and editing and modifying the documents; and saving the edited documents.

[0018] In another aspect, this embodiment provides a document processing apparatus, comprising: a semantic segmentation module for segmenting a document image into multiple document elements based on the semantic category of the document image in a file to be compressed; a first determination module for determining importance scores of the multiple document elements, the importance scores reflecting the degree of attention paid to the document elements; a second determination module for determining an encoding strategy for each document element based on the semantic category and importance score of each document element; and a document compression module for compressing each document element in the document image based on the encoding strategy of each document element.

[0019] In another aspect, this embodiment provides a computer program product, including a computer program that, when executed by a processor, implements the document processing method described above.

[0020] In another aspect, this embodiment provides a computer-readable storage medium for storing computer-readable instructions that, when executed by a processor, cause the processor to perform the document processing method described above. Attached Figure Description

[0021] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 A schematic diagram illustrating a scanning scenario according to an embodiment of the present disclosure is provided.

[0023] Figure 2 A flowchart illustrating a document processing method according to an embodiment of the present disclosure is shown.

[0024] Figure 3 The flowchart illustrating an embodiment of the present disclosure for determining the importance scores of a plurality of document elements is shown.

[0025] Figure 4 A flowchart illustrating another document processing method according to an embodiment of the present disclosure is shown.

[0026] Figure 5 The diagram illustrates another document processing method according to an embodiment of the present disclosure.

[0027] Figure 6 The diagram illustrates a document processing method according to an embodiment of the present disclosure.

[0028] Figure 7 A schematic diagram illustrating a semantic interface according to an embodiment of the present disclosure is provided.

[0029] Figure 8 A schematic diagram of an attention interface according to an embodiment of the present disclosure is shown.

[0030] Figure 9 A schematic diagram illustrating a strategy interface according to an embodiment of the present disclosure is provided.

[0031] Figure 10 A schematic diagram illustrating a strategy modification interface according to an embodiment of the present disclosure is provided.

[0032] Figure 11 A schematic diagram illustrating the interface for decompressing a file package according to an embodiment of the present disclosure is provided.

[0033] Figure 12 A block diagram illustrating a document processing apparatus according to an embodiment of the present disclosure is shown.

[0034] Figure 13 A block diagram illustrating a computer program product according to an embodiment of the present disclosure is shown schematically.

[0035] Figure 14 A block diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0037] Existing mainstream document compression technologies have the following significant drawbacks when processing modern complex documents: Insufficient segmentation accuracy: Traditional mixed raster content (MRC) models rely primarily on low-level physical features such as color, brightness, and edges for layering. When processing documents with tightly packed text and images, complex background textures, old documents, or those with low contrast or uneven lighting, the segmentation results are poor. This often leads to jagged or blurred text edges, and important graphic elements are incorrectly classified into the background layer, resulting in lost details and severely impacting the visual quality and information integrity of the compressed document.

[0038] Compression strategies are often rigid and inflexible: Existing technologies typically employ fixed compression algorithms for the same type of elements. For example, all areas identified as "images," regardless of whether their content is a smooth-toned photograph or a line drawing or technical drawing containing a large amount of high-frequency detail, may be uniformly compressed using the Joint Photographic Experts Group (JPEG) format. This "one-size-fits-all" strategy is suboptimal: compressing line drawings with JPEG produces artifacts such as ringing and blockiness; while compressing photographs using lossless methods such as portable network graphics (PNG) results in excessively large file sizes, defeating the purpose of compression.

[0039] Loss of structural and semantic information: The compression process often "flattens" a document into a superposition of several image layers. The inherent logical structural information of the document, such as the row and column relationships of tables, the item order of lists, and the hierarchy of paragraphs, is completely destroyed. This not only makes the compressed document difficult for machines to further parse and utilize (such as data extraction and content search), but also poses a significant obstacle to accessible access for visually impaired individuals.

[0040] Based on this, embodiments of this disclosure provide a document processing method that can segment document images based on their semantics, which helps ensure the integrity of the document image's logical structure. Simultaneously, an importance score is determined for each segmented document element, reflecting its level of attention within the document image. Furthermore, an optimal encoding strategy is determined based on the semantic type and importance score of the document elements. This allows for the selection of different compression methods for document elements with varying levels of importance and semantic types, saving compression space while maintaining the logical integrity and visual quality of the document image. Specifically, the higher the importance score of a document element, the more attention it receives from the user, resulting in higher compression quality and visual quality from the compression method or encoding strategy.

[0041] This disclosure provides a document processing system that can be deployed on a terminal device or an image forming device, wherein the document processing system is used to implement a document processing method.

[0042] A document processing system includes at least a file acquisition module, a semantic segmentation module, a first determination module, a second determination module, a document compression module, and a document decompression module.

[0043] The file acquisition module is used to acquire the file to be compressed.

[0044] The semantic segmentation module is used to perform semantic recognition on each page of the document image in the file to be compressed. Based on the semantic category of each page of the document image in the file to be compressed, each page of the document image is segmented into multiple document elements.

[0045] The first determination module is used to determine the importance scores of multiple document elements in the document image on each page.

[0046] The second determination module is used to determine the encoding strategy for each document element based on its semantic category and importance score.

[0047] The document compression module is used to compress each document element in each page of document images based on the encoding strategy of each document element.

[0048] The document decompression module is used to decompress compressed files by invoking a specified reader, restoring and displaying each document in the file to be compressed, and responding to user editing instructions to edit and modify the document content. The edited document can then be saved again. Here, "compressed file" refers to the file after the file to be compressed has been compressed.

[0049] The following section will combine... Figures 1-14 A document processing method according to an embodiment of the present disclosure will be described.

[0050] Figure 1 A schematic diagram illustrating a scanning scenario according to an embodiment of the present disclosure is provided.

[0051] like Figure 1 As shown, the system includes an image forming apparatus 101 and a terminal device 102. A user can scan a file to be compressed using the image forming apparatus 101 and transmit the file to the terminal device 102. The terminal device 102 can receive the file and compress it. The image forming apparatus 101 can be a device with functions such as printing, scanning, copying, and faxing, while the terminal device 102 can include network devices such as smartphones, tablets, laptops, and desktop computers. Of course, the image forming apparatus 101 itself can compress the scanned file, eliminating the need for the terminal device 102 to compress it.

[0052] Specifically, the terminal device 102 or the image forming device 101 can perform semantic segmentation on the document image in the file to be compressed, dividing it into multiple independent document elements. Furthermore, the importance score of each document element can be determined, and different encoding strategies can be determined for different document elements based on their importance scores and semantic categories, thereby achieving multi-level, precise compression of the document image.

[0053] After compressing the file, the terminal device 102 or the image forming device 101 can display the compression result. When decompression is needed, a designated reader can be used to decompress the file. Designating a reader also allows the user to modify specific content within the document. This allows users to independently modify the scanned document content without needing to modify the original document and scan it again, reducing user operations and providing convenience.

[0054] In scanning scenarios, there are two types: push scan and pull scan. For pull-and-scan: When a user scans an original document using a drag-and-drop scanner, the scanning driver installed on the terminal device sets the scanning parameters and starts the scan. These parameters include scan resolution, number of copies, scan area, and whether scan compression is required. After the terminal device sends a scan command to the image forming device, the image forming device executes the scan according to the parameters, transmitting the scanned file back to the terminal device. Upon receiving the scanned file, if the scanning parameters indicate that compression is not required, the user can upload the scanned file to a document processing system for compression and decompression. If the scanning parameters indicate that compression is required, the terminal device automatically invokes the document processing system upon receiving the scanned file, which then compresses and decompresses the file.

[0055] For push-broom: When a user scans an original document using a push-broom, they set the scanning parameters on the image forming device and start the scan. These parameters include: scan resolution, number of copies, scan area, and whether scan compression is required. The image forming device executes the scan according to these parameters, producing a scanned file. If the scanning parameters indicate that scan compression is not required, the scanned file is directly transmitted to the terminal device or other destination. If the scanning parameters indicate that scan compression is required, the image forming device uses the document processing system to compress the scanned file. The compressed file is then transmitted to the terminal device or other destination (such as an email address).

[0056] Figure 2 A flowchart illustrating a document processing method according to an embodiment of the present disclosure is shown.

[0057] like Figure 2 As shown, the document processing method of this disclosure embodiment can be applied to a terminal device or an image forming device, and the method includes S201, S202, S203 and S204: S201. Based on the semantic category of the document image in the file to be compressed, the document image is segmented into multiple document elements.

[0058] In this embodiment of the disclosure, a terminal device or an image forming apparatus acquires a file to be compressed. The file to be compressed includes at least one page of document image. The file to be compressed can refer to a file that needs to be compressed, and can be obtained by scanning the original document using an image forming apparatus or a terminal device (such as a mobile phone), or can be transmitted to the terminal device or image forming apparatus via other communication methods (such as social media software).

[0059] When acquiring a file to be compressed, users can either open the document processing system on their terminal device and upload the file, or the image forming device or the terminal device can automatically call the document processing system and upload the file.

[0060] In one embodiment, a user can open a document processing system and display a file acquisition interface provided by the file acquisition module of the system on a terminal device. This file acquisition interface includes a document upload control. The user can upload a file to be compressed by triggering the document upload control. The document upload control can be displayed as a link, button, or similar format. The file to be compressed includes at least one page of document image. The content of the document image can include tables, text, images, graphics, patterns, and other formats.

[0061] In one embodiment, when the terminal device detects a compression requirement for a file to be compressed, it automatically invokes the document processing system and uploads the acquired file to be compressed to the system. The terminal device can determine whether there is a compression requirement for the file based on scanning parameters.

[0062] In one embodiment, after the image forming device scans and obtains the file to be compressed, the image forming device identifies whether the file to be compressed has a compression requirement based on the scanning parameters. When a compression requirement for the file to be compressed is identified, the image forming device automatically calls the document processing system and uploads the scanned file to be compressed to the document processing system.

[0063] After uploading the file to be compressed in the document processing system, the terminal device or image forming device displays the document processing interface, which includes at least document compression controls and document decompression controls.

[0064] When a user triggers the document compression control in the document processing interface, the document compression process begins. This involves semantic recognition of the document images in the file to be compressed, and segmenting the document images into multiple document elements based on their semantic categories.

[0065] The document compression process is as follows: The semantic segmentation model deployed in the semantic segmentation module of the document processing system can be used to semantically identify each page of the document image in the file to be compressed, obtaining the semantic category of each page. Based on the semantic category of each page, semantic segmentation is performed on the document image to obtain multiple document elements. Semantic categories can include, but are not limited to: titles, graphics, images, tables, paragraphs, etc. Document elements represent instances of objects in the document image. It is understandable that different texts or paragraphs in a document image have different meanings. By distinguishing the semantic categories of the document image content, related content belonging to the same logic can be effectively grouped together, thus facilitating the classification and compression of the document image.

[0066] Using a semantic segmentation model, semantic recognition is performed on each page of the document image in the compressed file to obtain the semantic category of each page. Based on the semantic category of each page, semantic segmentation is performed on the document image to obtain multiple document elements. This includes: determining multiple semantic masks for the document image based on the semantic segmentation model, where each semantic mask is used to determine at least the semantic category; and segmenting the document image into multiple document elements based on the multiple semantic masks.

[0067] In this embodiment of the disclosure, the semantic segmentation model can be used to accurately identify the page content of a document image. The semantic segmentation model can employ a panoramic semantic segmentation model from deep learning. In a preferred embodiment, the semantic segmentation model can use a panoptic feature pyramid network (PFPN) architecture to distinguish individual instance objects in the document image.

[0068] In this embodiment, the semantic segmentation model can not only perform semantic classification on document images, but also perform instance segmentation on image regions belonging to the same semantic category, that is, distinguish different instance objects as document elements. For example, during semantic classification, the first, second, and third paragraphs in the document image all belong to the semantic category of "text". Then, for the semantic category of "text", the first, second, and third paragraphs can be further distinguished, that is, the first paragraph can be distinguished as "text 1", the second paragraph as "text 2", and the third paragraph as "text 3", to distinguish different instance objects belonging to the same semantic category.

[0069] In this embodiment of the disclosure, semantic recognition of the document image is essentially pixel-level classification, that is, assigning a semantic category label to each pixel in the document image. Furthermore, after semantic recognition, the semantic segmentation model can also separate each pixel according to the semantic category. In some embodiments, the segment anything model (SAM) included in the semantic segmentation model can be used to separate each pixel according to the semantic category.

[0070] In some embodiments, SAM can precisely segment each pixel, accurately locating and dividing the complete region of any independent object based on target features. By separating pixels of different semantic categories, it ensures that categories do not overlap, and by separating different instance objects of the same semantic category, it ensures that instances are not confused.

[0071] In this embodiment of the disclosure, the semantic segmentation model can also repair specific objects after separation based on pixel position and semantics. In some embodiments, diffusion models and state-of-the-art (SOTA) included in the semantic segmentation model can be used to repair specific objects after separation. For example, for large title text on a cover background, the original font position will be left blank after background separation. After repair using a diffusion model, if the storage space occupied is smaller (continuous patterns have better storage efficiency than complex and varied patterns), selective repair can be performed.

[0072] In this embodiment, a large and diverse set of annotated document datasets (such as public layout networks and document layout networks) can be used to train and fine-tune the model to obtain the final semantic segmentation model. Through training, the model can identify and accurately delineate various predefined semantic regions and perform instance segmentation based on the semantic recognition results, such as semantic categories {text, title, list, table, background, image, graphic}. Furthermore, specific regions can be further subdivided, such as the sky in image A being A-1 and the person in image A being A-2.

[0073] In this embodiment of the disclosure, a trained semantic segmentation model can be used to accurately identify document images and output multiple semantic masks for the document images. In some embodiments, the semantic mask can be a binary semantic mask. Each semantic mask corresponds to a semantic category, and a pixel with a mask value of 1 for that semantic category indicates that the pixel belongs to that semantic category. For example, in a semantic mask specifically identifying the semantic category "paragraph," all pixels belonging to "paragraph" will be marked as 1, while pixels in other places (such as headings or blank spaces) will be marked as 0. As another example, in a semantic mask specifically identifying the semantic category "heading," only the semantic mask corresponding to pixels belonging to "heading" will be 1.

[0074] In this embodiment of the disclosure, each pixel has a corresponding semantic mask based on different semantic categories. By analyzing multiple semantic masks, a document image can be segmented into multiple document elements. In some embodiments, multiple pixels belonging to the same semantic category and whose pixel positions are adjacent can be identified as a single document element. For example, for the semantic category of "text," pixels corresponding to multiple consecutive semantic masks with a value of 1 can be grouped into a single document element.

[0075] In this embodiment, a unique instance ID can be assigned to each independent document element. Simultaneously, metadata such as the bounding box, precise outline, and hierarchical relationship on the page for each independent document element is determined. The bounding box refers to enclosing the paragraph within a rectangle, recording the box's position (e.g., "top left corner at page (10, 20), bottom right corner at (500, 300)"). The bounding box allows for convenient and quick location of the corresponding document element. The precise outline refers to a more detailed record of the paragraph's edge shape (e.g., if the last line of text in a paragraph is short, the precise outline would be "wider at the top and narrower at the bottom"). The hierarchical relationship refers to the positional relationship between document elements; for example, a table might be below the second paragraph and above the third paragraph. The hierarchical relationship determines the contextual order of document elements.

[0076] In the embodiments of this disclosure, the deep learning-based semantic segmentation model can accurately identify and separate complex document elements, such as images with embedded text and tables spanning multiple lines, thereby ensuring the integrity of content elements and maintaining accurate semantic content.

[0077] In this embodiment of the disclosure, after segmenting the document image into multiple document elements based on the semantic category of the document image in the file to be compressed, the method further includes: displaying a semantic interface in the document processing system of the terminal device or image forming device, wherein the semantic interface displays the semantic category of each document element in the document image. In the semantic interface, a user's instruction to change the semantic category of any document element is received, the semantic category of the document element is changed, and the changed semantic category is used as the semantic category of the document element.

[0078] When a semantic category needs to be changed, the user triggers any document element in the semantic interface, and various semantic categories are displayed next to that document element. When the user selects a semantic category from the displayed categories, a semantic category change command is triggered. Based on the semantic category change command, the semantic category of the document element is changed, and the semantic category carried in the semantic category change command becomes the new semantic category. The trigger can be a single click, double click, long press, etc.

[0079] For example, if a paragraph labeled "text" is displayed in the semantic interface, the user can reselect "image" from the various semantic categories for that paragraph, thus changing the semantic category of the paragraph from "text" to "image".

[0080] S202. Determine the importance scores of multiple document elements. The importance scores are used to reflect at least the degree to which document elements are given attention.

[0081] In embodiments of this disclosure, importance scores can be used to reflect the degree of attention paid to document elements in a document image. The more easily a document element is noticed, the higher its importance score. Here, attention can refer to visual attention.

[0082] For example, headings are usually at the top of document images and have large fonts. From a visual attention perspective, document elements with headings will be the focus of attention, so document elements with the semantic category of headings have a higher importance score.

[0083] Figure 3 A flowchart illustrating an embodiment of the present disclosure for determining importance scores of multiple document elements is shown. Figure 3 As shown, S202 above includes S301 and S302: S301. Based on the visual saliency model, determine the attention coefficient of each document element.

[0084] In this embodiment of the disclosure, the attention coefficient of each document element can be obtained through a visual saliency model. The attention coefficient reflects the degree to which a document element attracts attention. The sum of the attention coefficients in the document image is 100%, and document elements with higher visual attention have larger attention coefficients. For example, the visual saliency model can be used to simulate the content that a person instinctively notices first when viewing a page of document images; for example, they will prioritize noticeable content such as titles, bolded key data, and faces in images. The visual saliency model assigns higher attention coefficients to these prominent elements.

[0085] In some embodiments, a large number of document image samples can be input into the model to train the model (e.g., a stereo attention mechanism (SAM)) to obtain a visual saliency model.

[0086] In this embodiment of the disclosure, after determining the attention coefficient of each document element, a saliency heatmap corresponding to the document image can be generated based on the attention coefficient of each document element. This heatmap can represent the degree of visual attention a user pays to each document element in the document image. Typically, areas such as titles, faces, and key data have high saliency, while areas such as backgrounds and non-key data have low saliency.

[0087] In this embodiment of the disclosure, after determining the attention coefficient of image-type document elements, the method further includes: displaying an attention interface on a terminal device or an image forming device, whereby the attention coefficient of each document element is displayed. The attention interface receives a user's instruction to change the attention coefficient of any document element, changes the attention coefficient of that document element, and dynamically adjusts the attention coefficients of other document elements. The changed attention coefficient of each document element is then used as its respective attention coefficient.

[0088] In some embodiments, a user can choose to change the attention coefficient of a specific document element. Based on the modified attention coefficient, the attention coefficients of other document elements near or related to that element, or all other document elements, can be automatically adjusted. The modified attention coefficient is then used as the attention coefficient for other document elements, providing a basis for subsequent compression.

[0089] S302. Determine the importance score of a document element based on its attention coefficient and bounding box.

[0090] In this embodiment, the terminal device or image forming device can superimpose the attention coefficient of a document element with the bounding box involved in that document element to determine the importance score of the document element. This importance score can serve as an important basis for subsequent selection of encoding strategies. A document element involves at least one bounding box. For example, the document element "hair" can be distinguished from other document elements by its bounding box, and the document element "hair" may consist of multiple bounding boxes.

[0091] In some embodiments, for any document element, when the document element involves multiple bounding boxes and the document elements corresponding to the multiple bounding boxes are all the same document element, the attention coefficient of the document element can be used as the importance score of the document element.

[0092] In some embodiments, for any document element, when the document element involves multiple bounding boxes and the document elements corresponding to the multiple bounding boxes are not all the same document element, the importance score of the document element can be determined based on the attention coefficients of the document elements corresponding to the multiple bounding boxes and the area proportions of different document elements within the bounding boxes of the document element. Specifically, the importance score of the document element can be obtained by weighted summing the area proportions of different document elements within the bounding boxes of the document element and the attention coefficients of the different document elements. For example, the bounding box of the document element "eye" also includes a portion of the document element "face". The "eye" document element accounts for 90% of the bounding box, and the "face" document element accounts for 10%. The attention coefficient of the "eye" document element is 50, and the attention coefficient of the "face" document element is 30. Therefore, the importance score for the document element "eye" can be: 50 × 90% + 30 × 10% = 48.

[0093] According to embodiments of this disclosure, a visual saliency model is used to assign a higher attention coefficient to the regions in a document image that are of interest to the human visual system, so as to comprehensively determine the importance score of each document element, and select a compression method according to the importance score, thereby reducing the amount of file data while ensuring the subjective visual quality of the user's attention to the region.

[0094] S203. Determine the encoding strategy for each document element based on its semantic category and importance score.

[0095] S204. Based on the encoding strategy of each document element, compress each document element in the document image.

[0096] In this embodiment of the disclosure, the terminal device or image forming device can select the optimal compression method, i.e., encoding strategy, for each document element based on its semantic category and importance score. After determining the encoding strategy for each document element, the determined encoding strategy is used to compress each document element. Alternatively, after the user redetermines the encoding strategy for the document element, the corresponding document element is compressed according to the redetermined encoding strategy.

[0097] According to embodiments of this disclosure, a document image is segmented into different document elements based on the semantic relationships of the document image, thereby helping to ensure the integrity of the document image's inherent logical structure. Simultaneously, an importance score is determined for each document element, and based on the importance score and the semantic type of the document element, the most suitable encoding strategy is matched for each document element. Therefore, the compression level can be determined by the semantic type and importance of the document elements, improving the overall compression rate while maintaining subjective visual quality.

[0098] The following provides a detailed explanation of the encoding strategy for each document element.

[0099] In this embodiment of the disclosure, the encoding strategy for each document element is determined based on its semantic category and importance score. This includes: determining whether a document element is a vectorizable document element based on its semantic category; if the document element is vectorizable, performing vectorized encoding on it to compress it; vectorizable document elements include at least text-based and image-based document elements; if the document element is not vectorizable, i.e., if it is an image-based document element, determining its color complexity to determine the encoding strategy for that image-based document element.

[0100] In some embodiments, text-based document elements, such as letters and Chinese characters, have fixed shapes and do not require recording every pixel; it is sufficient to describe: this is the letter 'A', font, font size, and color. Graphical document elements, such as rectangles, circles, lines, tables, and other geometric shapes, are regular and easily described using parameters (position, size, color). Document elements that can be described using structured statements can be determined to be vectorizable. Conversely, photos and patterns (such as landscape photos, face photos, and animated patterns) have varying pixel colors and lack a fixed pattern, making them difficult to vectorize.

[0101] In this embodiment of the disclosure, after vectorizable document elements are vectorized and encoded, their structured statements can be directly stored, thereby saving storage space. For example, position (30, 20) is a 50×50px 8-bit RGB rectangle, which occupies 50×50×3=7500 bytes before vectorization. After vectorization, the structured statement is: {type=shape, shape=rectangle, bounds=[30,20,50,50],fill_color=#FF0000,stroke=null}, which only requires <100 bytes, with a compression ratio of 100 / 7500.

[0102] According to the embodiments of this disclosure, after semantic segmentation of the document image, text, vector graphics, and other content are stored in their native, high-compression form (such as character encoding or vector instructions) instead of pixel arrays. This ensures that important information is not lost while maximizing space savings. For example, a "red title box" in the document image (important, a rectangular graphic, and vectorizable) is preferentially converted into a structured descriptive statement, while less important complex images (such as background decorative images) may not be vectorized or stored in a low-precision manner.

[0103] In this embodiment of the disclosure, if the document element is a vectorizable document element, the document element is vectorized and encoded, including: when the vectorizable document element is a text-type document element, extracting the text information of the text-type document element, wherein the text information includes at least one or more of the following: text content, font, font size, color, and position; and saving the text information as vector data.

[0104] In some embodiments, an optical character recognition (OCR) engine (such as Tesseract or a more advanced deep learning OCR model) can be invoked to extract information such as the text content, font, font size, color, and position of an element. For example, recognizing the words "Annual Report" in a document image and simultaneously extracting information such as the font, font size, color, and position of these words, i.e., the text content is "Annual Report," the font is Microsoft YaHei, the font size is 24, the color is #FF0000, and the position is (50, 30).

[0105] According to the embodiments of this disclosure, text information recognized by OCR can be directly stored, which is a near-lossless method with an extremely high compression ratio. Since it stores the text itself rather than pixels, the text remains clear and undamaged regardless of magnification, making it a near-lossless method. Furthermore, storing text content plus style descriptions significantly saves compression space, resulting in an extremely high compression ratio. In addition, when modifications are needed to the stored text-based document elements, their structured semantics can be directly modified.

[0106] In this embodiment of the disclosure, if the document element is a vectorizable document element, the document element is vectorized and encoded, including: when the vectorizable document element is a graphic document element, extracting the wireframe information and / or text information of the graphic document element, wherein the wireframe information includes at least the wireframe row and column relationship, and the text information includes at least one or more of the following: text content, font, font size, color, and position; and storing structured data, wherein the structured data is used to describe the wireframe information and / or text information.

[0107] In this embodiment of the disclosure, when the vectorizable document element is a graphic document element, the wireframe information and / or text information of the graphic document element can be identified. The wireframe information can be used to represent the shape, color, and row / column relationships of the graphic, such as a square, rectangle, or curve, and the row / column relationships can be used to represent the positional relationships between multiple wireframes. The text information is similar to that of the text document element described above and can include one or more of the following: text content, font, font size, color, and position. Furthermore, the graphic document element can be described using structured statements, thereby directly storing the structured statements.

[0108] For example, when the graphic document element is a table document element, the wireframe structure of the table can be identified, the row and column relationships of the cells can be parsed, and the text content within each cell can be analyzed. Thus, the table topology (i.e., the skeleton information of the table, such as the position and size of each cell) and cell content (i.e., the specific text information in each cell, such as the text content of the cell in the 2nd row and 1st column being "Zhang San," with the font being SimSun, size 12, etc.) can be described using structured data (such as JSON or XML).

[0109] According to the embodiments of this disclosure, directly storing structured data avoids the problems of pixel images taking up too much space, being difficult to edit, and becoming blurry when magnified, while accurately preserving the logic and details of the content. Furthermore, if one wants to modify the table content, only the relevant fields in the structured data need to be modified, without needing to change the entire image.

[0110] In this embodiment of the disclosure, determining the color complexity of image-type document elements to determine the encoding strategy of image-type document elements based on the color complexity includes: when the color complexity is low, performing implicit vectorization compression on image-type document elements based on importance scores; when the color complexity is high, determining whether the image-type document elements belong to photo-type images or line-type images, and determining the encoding strategy of image-type document elements based on the determination result and the importance scores.

[0111] In this embodiment, low color complexity can refer to a limited number of colors, such as a document element that is a solid color image, or an image with few colors. High color complexity, on the other hand, refers to a large number of colors that are difficult to separate, such as landscape photos or portraits with many colors and transitions. When the color complexity of an image-type document element is low, it can be implicitly vectorized.

[0112] In this embodiment of the disclosure, when the document element is an image-type document element, it can be first determined whether implicit vectorization is possible based on the color complexity of the image-type document element. In some embodiments, implicit vectorization can refer to saving space by reducing unnecessary colors / details for images with simple colors, while preserving the original image shape. For example, for image-type document elements with two colors, implicit vectorization can be used, such as using 0 and 1 to represent the colors of the image-type document element, thereby reducing the memory footprint of the image.

[0113] When image-based document elements can be implicitly vectorized, compression quality parameters can be determined based on their importance scores to compress them. For image-based document elements with higher importance scores (such as core illustrations), higher compression quality parameters are appropriate. That is, a higher importance score indicates that the image-based document element is more important to the user, and setting higher compression quality parameters allows for less loss of detail and higher image clarity during compression. Conversely, for image-based document elements with lower importance scores (such as decorative background images), lower compression quality parameters are appropriate.

[0114] According to embodiments of this disclosure, image-type document elements with low color complexity are implicitly vectorized, and compression quality parameters are determined based on importance scores. This ensures that important image-type document elements suffer minimal detail loss during compression, maintaining image clarity. Less important image-type document elements can be compressed more, thus saving space. This balances clarity and storage space while meeting the user's priorities.

[0115] In this embodiment, when color complexity is low, latent vectorization is performed on image-type document elements based on importance scores. This includes: inputting image-type document elements with low color complexity into a latent vector generative model, so that the latent vector generative model outputs latent vectors for the image-type document elements. The latent vectors are stored and used to reconstruct the image-type document elements. For example, a 1MB ID photo might generate a latent vector of only tens of bytes, thus saving space. In this embodiment, the latent vector generative model can be a generative model specifically optimized for document elements, such as a variational autoencoder (VAE) or a generative adversarial network (GAN). The latent vector generative model is trained to understand and reconstruct images of specific types (e.g., low color complexity), such as "ID photos" or "company logos." That is, a large number of images of specific types can be used to train the generative model to generate the latent vector generative model, enabling the latent vector generative model to output latent vectors for specific types of images.

[0116] According to the embodiments of this disclosure, the latent vectors of document elements are output and stored using a latent vector generative model. Compared with traditional compressed data, the latent vectors occupy less memory space, thus saving storage space. Moreover, the color and shape of the document elements directly restored using the latent vectors are almost indistinguishable from the original document elements.

[0117] In this embodiment of the disclosure, when the color complexity is high, it is determined whether the image-type document element belongs to the photo-type image or the line-type image, and the encoding strategy of the image-type document element is determined based on the judgment result and the importance score.

[0118] In this embodiment of the disclosure, for image-type document elements with high color complexity, it is possible to determine whether they are photo-type images or line-type images by calculating features such as texture complexity (e.g., Laplacian operator variance), color histogram, and edge density of the image-type document elements.

[0119] In some embodiments, if the image-type document element is a line drawing, a lossless compression algorithm, such as PNG or WebP lossless compression format, is used to perfectly preserve sharp edges. If the image-type document element is a photograph, a more advanced lossy compression algorithm, such as AV1 Image File Format or JPEG eXtra Large image format, is used. Furthermore, the compression quality parameters can be determined based on an importance score, using higher quality parameters for image-type document elements with high importance scores and lower quality parameters for image-type document elements with low importance scores.

[0120] In this embodiment of the disclosure, after determining the encoding strategy for each document element based on the semantic category and importance score of each document element, the method further includes: displaying a strategy interface on a terminal device or an image forming device, displaying the encoding strategy corresponding to each document element on the strategy interface; receiving a user's strategy change instruction for any document element in the strategy interface, changing the encoding strategy of the document element, and using the changed encoding strategy as the encoding strategy of the document element.

[0121] Figure 4 A flowchart illustrating another document processing method according to an embodiment of the present disclosure is shown.

[0122] like Figure 4 As shown, the document processing method also includes S401, S402a, and S402b: S401. When the file to be compressed includes multiple pages of document images, for any two adjacent document images, compare the multiple document elements included in the current document image with the multiple document elements included in the previous document image.

[0123] In this embodiment of the disclosure, when the file to be compressed includes multiple pages of document images, when processing the Nth page (N>1), the document elements included in the current page document image can be compared with the document elements included in the previous page document image. For example, the hash value or feature vector of each document element can be calculated, and the two page document images can be compared by comparing the similarity of the hash values ​​or feature vectors between the document elements of the two pages. In some cases, the document elements included in the current page can also be compared with the document elements included in the previous M pages (1<M<N) document images.

[0124] S402a. If the current document image and the previous document image have the same document element, then when compressing the current document image, store the reference pointer information corresponding to the same document element. The reference pointer information is used to point to the same document element in the previous document image.

[0125] In this embodiment of the disclosure, when the current document image and the previous document image contain identical document elements (i.e., for a certain document element, the content of the document element in the current document image is exactly the same as the content of the document element in the previous document image), for the identical document elements, when compressing the current page document image, the reference pointer information corresponding to the document element is stored. Specifically, the reference pointer information corresponding to the document element can be stored in a compressed file containing the compressed data of the current page document image, and the reference pointer information corresponding to the document element is used to indicate the positional information of the document element. That is, in the current document image and the previous document image, the document element in the previous document image can be retrieved through the reference pointer information corresponding to the document element in the current document image.

[0126] For example, if the content of the header on page 2 is exactly the same as the content of the header on page 1, then when compressing the image of page 2, the reference pointer information of the header is saved, and this reference pointer information can point to the header on page 1.

[0127] S402b: If the current document image and the previous document image have some identical document elements, then when compressing the current document image, store the difference data corresponding to the partially identical document elements; wherein, the difference data is used to represent the difference between the partially identical document elements in the current document image and in the previous document image.

[0128] In this embodiment of the disclosure, when there are partially identical document elements in the current document image and the previous document image (for a certain document element, the content of the document element in the current document image is partially the same as the content of the document element in the previous document image), for a certain document element, the content difference of the document element in the current document image compared with the document element in the previous document image can be calculated as differential data and stored.

[0129] For example, in the chart on page 2, based on the chart on page 1, the "2024 sales" is changed to "2025 sales," and the data is changed from "100" to "120." When compressing the image on page 2, the content difference between the chart on page 2 and the chart on page 1 is calculated and stored as differential data. In some embodiments, after calculating the differential data, it can be compressed to obtain differential compressed data, which is then saved, further saving storage space.

[0130] According to embodiments of this disclosure, by storing reference pointer information and storing differential data, the problem of repeatedly compressing the same content when compressing images of multi-page documents can be solved, reducing compression computation power, reducing the amount of data after final compression, and also saving storage space. Figure 5 A flowchart illustrating another document processing method according to an embodiment of the present disclosure is shown.

[0131] like Figure 5 As shown, after compressing each document element in the document image using an encoding strategy based on each document element, steps S501 and S502 are also included: S501. Based on the relationship between multiple document elements in the file to be compressed, generate a structure directory file. The structure directory file shall include at least one or more of the following: the size of each page document image, the identifiers of the multiple document elements included in each page document image, semantic categories, bounding boxes, levels, reference pointer information, and logical relationships between the multiple document elements included in each page document image.

[0132] For each page of the document, use images such as Figures 2-4 After compression, a structure directory file is generated. This file can include the size information of each page's document image (size, margins, etc.), the identifier of each document element within the multiple document elements included on each page, the semantic category of each document element, the bounding box of each document element, the hierarchical relationship between multiple document elements, reference pointer information for document elements, and the logical relationships between multiple document elements. The structure directory file can be used to describe the relationships between all document elements in each page's document image, as well as the sequential relationships between document images on each page.

[0133] In this embodiment of the disclosure, the manifest.json file can be a JSON or XML file. The manifest.json file serves as both a "table of contents" and a "manual" for the multi-page document images. The manifest.json file describes the logical hierarchy of the entire document in a tree structure, clearly illustrating the relationships between all document elements on each page and the sequential relationships between pages.

[0134] S502. Based on the structure directory file and the compressed file of the file to be compressed, generate a decompressed file package, wherein the compressed file contains the compressed data of all document elements of all document images in the file to be compressed.

[0135] After compressing all document images, a compressed file is generated to store the data resulting from the compression of all document images within the file to be compressed. Based on the structure directory file and the compressed file of the file to be compressed, an uncompressed archive is generated. The file format of the uncompressed archive can be, for example, .sdoc or SemanticDocument. The uncompressed archive can contain a structure directory file (manifest.json) and the compressed file.

[0136] In some embodiments, a corresponding compressed file is generated for each document image. One compressed file corresponds to one document image. After all document images are compressed, multiple compressed files are saved to a compressed folder.

[0137] The compressed file contains compressed data for each document element in the document image. Examples include text_01.txt, table_02.json, figure_03.avif, and logo_04.vec (an implicit vector binary file). When generating the decompressed file package, the package contains a structured directory file and at least one compressed file.

[0138] In this embodiment of the disclosure, when browsing compressed files, a designated reader can be used to open the decompressed file package and display each document in the file to be compressed based on the structure directory file and compressed files within the decompressed file package. Each displayed document can be edited and modified by the user. The designated reader is part of the document processing system and is used in conjunction with the decompressed file package.

[0139] When a user needs to modify the content of a compressed document, the system receives the user's editing instructions for each displayed document and allows the user to edit and modify the document.

[0140] In this embodiment of the disclosure, the terminal device or image forming device can decode and reconstruct each document image in the compressed file based on the structure directory file and the compressed file in the decompressed file package. The reconstructed document content can be edited and modified.

[0141] For example, a specified reader can be used to parse the manifest.json file and the compressed file, generate multiple blank canvases based on the size information (size, margins, etc.) of the document images on each page, and then render the compressed data of the document elements included in the document images on each page of the compressed file on the corresponding canvas to generate document elements, thereby displaying each document in the file to be compressed.

[0142] In this embodiment, after opening the decompressed file package with a specified reader, the text of each document is rendered as optional, searchable, and editable text content. Tables and charts are dynamically drawn based on their structured statements, making them interactive and copyable. Images are reconstructed using corresponding decoders (AVIF decoder, PNG decoder, or generative model decoder). The final product presented to the user is a visually identical document to the original, but internally fully structured and interactive—a "living" document. Therefore, the user can edit the reconstructed document and, furthermore, receive editing instructions to modify and retain corresponding document elements.

[0143] According to embodiments of this disclosure, the document after decoding and reconstructing the decompressed file package is "alive." The reconstructed document is not only a visual reproduction of the original document but also supports modification and editing functions. Tables can be directly copied to spreadsheet software, and text can be searched and edited, providing unprecedented convenience for downstream data mining, information retrieval, and accessibility applications. For example, users can sort table data or interact with charts in a specified reader.

[0144] Figure 6 The diagram illustrates a document processing flow according to an embodiment of the present disclosure.

[0145] like Figure 6 As shown, S601: Obtain the document image from the file to be compressed. The document image format can be a raster image (such as PNG, JPEG, TIFF, BMP) or a page description language file (such as PDF). Figure 6 As shown, the document image to be compressed may include text A, text B, image A, image B, and a table.

[0146] S602: Segmenting Document Images. Using a semantic segmentation model, perform semantic recognition and segmentation on document images, outputting the bounding boxes and semantic categories of each document element.

[0147] Among them, such as Figure 6 As shown, semantic segmentation can yield multiple independent document elements, such as text A, tables, graphics, images of people, etc. Furthermore, individual pixels can be separated to obtain... Figure 6The pixel areas shown include text A, table, graphic, and image A (person). ID-01 can be a unique ID for text A, and similarly, table, graphic, and image A can each have a corresponding ID. Further, the separated pixels are repaired to obtain multiple independent document elements, such as text A, table, graphic, graphic A-1 (person), and image A-2 (grass).

[0148] S603: Determine Importance Scores. Based on the visual saliency model and the bounding boxes of document elements, determine the importance scores of document elements. For example... Figure 6 As shown, an importance score can be assigned to each document element, such as text A having an importance score of coefficient 1, text B having an importance score of coefficient 2, and so on.

[0149] S604: Determine the encoding strategy. Based on the importance score and semantic category of the document element, determine a different encoding strategy for each document element and implement the different encoding strategies.

[0150] For example, vectorizable document elements are vectorized. If they are structured elements such as text or graphics, OCR or structural parsing is performed to encode them into text streams or structured data. Alternatively, implicitly vectorizable document elements are implicitly vectorized. If they are image elements, it is first determined whether they can be implicitly vectorized. If so, a generative AI encoder can be selectively used to compress them into implicit vectors. If not, their content features are analyzed, and an appropriate lossy or lossless encoder and parameters are selected for compression based on their importance scores. Alternatively, other objects are rasterized, and the optimal encoding strategy is dynamically selected from a pre-defined encoding strategy library based on the importance score for compression.

[0151] S605: Compression. The file to be compressed is compressed using a defined encoding strategy. If the file contains multi-page document images, differential and incremental compression are performed to eliminate redundant content across pages. Finally, all compressed data (i.e., compressed files) and the structured directory file containing metadata are packaged into a structured .sdoc file, which serves as the decompression package.

[0152] S606: Output. Outputs the final uncompressed file package.

[0153] Figures 7-11 The diagram illustrates a document processing interface interaction according to an embodiment of the present disclosure.

[0154] Figure 7A schematic diagram illustrating a semantic interface according to an embodiment of the present disclosure is provided. After a user uploads a file to be compressed through a file acquisition interface on a terminal device, the terminal device performs semantic segmentation on the file to be compressed and displays the segmentation results to the user through the semantic interface 700. The semantic interface 700 may include corresponding buttons or links for changing semantic category 701, next step 702, skip 703, and next page 704. A user can select a document element and click to display the semantic category of that document element, or they can click to select and change the semantic category. For example, if a user selects "Notification about XXXX" and right-clicks to select to display the semantic category of the document element, the semantic category of that document element can be displayed in the semantic interface 700. Alternatively, a user can click to change the semantic category 701, thereby displaying changeable semantic categories in the semantic interface 700, such as... Figure 7 As shown, "Marked as: Text, Graphic, Image, More" can be displayed next to "Notification about XXXX". Furthermore, users can select "Graphics" to change the semantic category of "Notification about XXXX" from text to graphic.

[0155] If the file to be compressed includes multiple pages of document images, you can edit the multiple pages using "Next Page 704". After making changes, you can click Next 702, or click Skip 703 if you do not want to change the document element type.

[0156] Figure 8 A schematic diagram illustrating an attention interface according to an embodiment of the present disclosure is provided. After a user clicks the Next 702 button or the Skip 703 button in the semantic interface 700, they are redirected to the attention interface 800. For example... Figure 8 As shown, the attention interface 800 may include skip 801, attention change 802, next step 803, and next page 804. The attention interface 800 can display a saliency heatmap of each document element. The saliency heatmap can be used to reflect the attention coefficients of different regions in the document image, for example... Figure 8 As shown, the attention interface 800 displays a saliency heatmap of an animal. This animal image is segmented into multiple document elements during semantic category segmentation, and the saliency heatmap of the image can be further displayed in the attention interface 800.

[0157] The saliency heatmap uses color to reflect the level of attention coefficients; for example, areas with higher attention coefficients are redder, and areas with lower attention coefficients are bluer. This allows users to intuitively determine the attention coefficient of each part of the image. If a user wants to change the attention coefficient of a specific area, they can select that area and modify it, receiving instructions to do so. For example, a user can select an area and click "Change Attention" (802), then enter the changed attention coefficient. Alternatively, the attention coefficient can be modified using a scroll bar; sliding the bar changes the color of the area, providing a clear visual indication of the change. If the user doesn't want to change the attention coefficient, they can click "Skip" (801). Users can also click "Next Page" (804) to modify the next document image. After changing the attention coefficient, the attention interface (800) can re-display the saliency heatmap. Furthermore, the user can click the "Next" button (803) to proceed to the "Confirm Strategy" interface.

[0158] After modifying the attention coefficients of different parts of the document image, the modified attention coefficients are used as the attention coefficients of the corresponding parts. Further, the bounding box of the document element is superimposed with multiple attention coefficients included within the bounding box to determine the importance score of the document element. Different attention coefficients can correspond to different colors. In some embodiments, document elements with high attention coefficients (high attention level) can be displayed in red, and document elements with low attention coefficients (low attention level) can be displayed in blue. It should be understood that red and blue are merely examples.

[0159] Figure 9 A schematic diagram illustrating a strategy interface according to an embodiment of the present disclosure is provided. After a user clicks "Skip" (801) or "Next" (803) in the attention interface (800), they are redirected to the strategy interface (900), as shown below. Figure 9 As shown, the strategy interface 900 includes buttons for Next 901, Advanced Settings 902, and Next Page 903. In the strategy interface 900, the terminal device can display the encoding strategy for each document element to the user. For example, if the current document image includes multiple document elements, one of which is a paragraph with a vectorizable compression strategy, and another document element is a photograph with an AV1 image file format compression strategy. The strategy interface 900 may also include memory resources allocated to document elements, such as... Figure 9As shown, the semantic category of the photo is "image," the compression method is "JPEG2000," and the allocated memory resources are "30%." The allocated memory resources refer to the percentage of total available resources allocated to implement the current compression strategy. Specifically, based on the importance score of the document element, device (terminal device or image forming device) memory resources are allocated to that document element; the higher the importance score, the more memory resources are allocated. Based on the semantic category of the document element and the allocated memory resources, the encoding strategy (i.e., compression method) for that document element is determined. The more memory resources allocated, the higher the compression quality of the determined encoding strategy, and the clearer the compressed document element.

[0160] In addition, users can select a document element and modify its encoding strategy; for example, users can change the encoding strategy of a document element by right-clicking. Alternatively, users can click Advanced Settings 902 to modify the encoding strategy of each document element. In some embodiments, users can click Advanced Settings 902 to navigate to the strategy modification interface 1000, such as... Figure 10 As shown.

[0161] Figure 10 A schematic diagram illustrating a strategy modification interface according to an embodiment of this disclosure is provided. Figure 10 As shown, the strategy modification interface 1000 includes options for semantic category change 1001, compression method change 1002, memory resource change 1003, and an OK button 1004. Semantic category change option 1001 can include changing to image, graphics, or custom. Compression method change option 1002 can include JPEG200 or CCITT TIFF. Memory resource change option 1003 can include high, medium, or low memory resources. After making the changes, the user can click the OK button 1004 to compress each document element in the file to be compressed, resulting in a compressed file.

[0162] For example, if the current document element is an image, semantic category option 1001 can include image, graphic, or custom. If the current document element is text, and... Figure 7 Similarly, in a semantic interface, semantic category options can include tables, images, graphics, and custom categories. In this case, a document element with a semantic category of text can be converted to a document element with a semantic category of image or graphics. For example... Figure 10 As shown, users can reselect the compression method for the photo, for example, change it to "CCITT TIFF", and change the allocated memory resources to "40%".

[0163] It should be noted that after a user changes the compression method, the document processing system can allocate appropriate memory resources to that document element based on the newly selected compression method. Memory resources can, to some extent, reflect the compression quality of the document element; higher compression quality means more compression time, and vice versa. For example, a document element with 60% memory resources will have higher compression quality than one with 20% memory resources. In some cases, if a user needs faster or higher quality compression of a document element, they can change the allocated memory resources, for example, adjusting them from 20% to 40% or from 60% to 40%.

[0164] After compressing the compressed file into a compressed file, the compressed file and the structure directory file are packaged into a structured .sdoc file, which is then used as an unzipped file package and displayed in the unzipped file package interface.

[0165] Figure 11 A schematic diagram illustrating the interface for decompressing a file package according to an embodiment of this disclosure is provided. Figure 11 As shown, when a user clicks the "Confirm" button (1004) in the policy modification interface (1000), they can be redirected to the file decompression interface (1100). The file decompression interface (1100) includes "Edit" (1101) and "Save" (1102) buttons. When the user clicks the "Save" (1102) button, the decompressed file package is saved. When the user clicks "Edit" (1101), they can use a specified reader to parse the compressed multi-page document. Within the parsed multi-page document, the user can edit the content. For example, the above method can compress a single-page PDF document image into an editable document. The text, tables, etc., included in the document image are transformed into editable text and tables through the compression method. Therefore, the user can edit the PDF document image.

[0166] Figure 12 A block diagram illustrating a document processing apparatus according to an embodiment of the present disclosure is shown.

[0167] like Figure 12 As shown, the document processing device 1200 includes a semantic segmentation module 1201, a first determination module 1202, a second determination module 1203, and a document compression module 1204.

[0168] The semantic segmentation module 1201 is used to segment the document image into multiple document elements based on the semantic category of the document image in the file to be compressed; the first determination module 1202 is used to determine the importance score of the multiple document elements, the importance score being used to reflect the degree of attention the document elements are; the second determination module 1203 is used to determine the encoding strategy of each document element based on the semantic category and importance score of each document element; and the document compression module 1204 is used to compress each document element in the document image based on the encoding strategy of each document element.

[0169] Figure 13 A block diagram illustrating a computer program product according to an embodiment of the present disclosure is shown schematically. like Figure 13 As shown, a computer program product 1300 according to an embodiment of this disclosure includes a computer program 1301, which, when executed by a processor, implements the document processing method described above.

[0170] Figure 14 A block diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure is shown schematically. like Figure 14 As shown, an embodiment of the present disclosure provides a computer-readable storage medium 1400 for storing computer-readable instructions 1401, which, when executed by a processor, cause the processor to perform the document processing method described above.

[0171] The above description, with reference to the accompanying drawings, illustrates a document processing method and computer program product according to embodiments of the present disclosure. By segmenting a document image into different document elements based on the semantic relationships of the document image, it helps to ensure the integrity of the document image's inherent logical structure. Simultaneously, it determines the importance score of each document element and matches the most suitable encoding strategy to each document element based on the importance score and the semantic type of the document element. Therefore, the degree of compression can be determined by the importance of the document elements, significantly improving the compression ratio while maintaining subjective visual quality. By implementing different encoding strategies for document elements with different semantic types and importance scores, compression can be performed more effectively and in a more targeted manner.

[0172] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0173] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0174] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0175] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0176] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described above can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0177] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0178] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A document processing method, characterized in that, The method includes: Based on the semantic category of the document image in the file to be compressed, the document image is segmented into multiple document elements; Determine the importance scores of multiple document elements, the importance scores being used to reflect the degree of attention paid to the document elements; Based on the semantic category and importance score of each document element, determine the encoding strategy for each document element; and Each document element in the document image is compressed based on the encoding strategy of each document element.

2. The document processing method according to claim 1, characterized in that, The process of segmenting the document image into multiple document elements based on the semantic category of the document image in the file to be compressed includes: When a user triggers the document upload control on a terminal device, a file to be compressed is obtained, the file to be compressed including at least one page of document images; When the user triggers the document compression control on the terminal device, the document image is segmented into multiple document elements based on the semantic category of the document image in the file to be compressed; or, After the image forming device scans and obtains the file to be compressed, when the user triggers the document compression control on the image forming device, the document image is segmented into multiple document elements based on the semantic category of the document image in the file to be compressed.

3. The document processing method according to claim 1, characterized in that, After segmenting the document image into multiple document elements based on the semantic category of the document image in the file to be compressed, the method further includes: The semantic interface displays the semantic category of each document element in the document image; In the semantic interface, the system receives a user's instruction to change the semantic category of any document element, changes the semantic category of the document element, and uses the changed semantic category as the semantic category of the document element.

4. The document processing method according to claim 1, characterized in that, The determination of importance scores for the multiple document elements includes: Based on the visual saliency model, the attention coefficient of each document element is determined; An importance score for each document element is determined based on the attention coefficient and the bounding box of each document element.

5. The document processing method according to claim 4, characterized in that, After determining the attention coefficient for each document element, the method further includes: The attention coefficient for each document element is displayed in the attention interface; In the attention interface, the user's instruction to change the attention coefficient of any document element is received, the attention coefficient of the document element is changed, and the attention coefficients of other document elements are dynamically adjusted. The attention coefficient of each document element after the change is used as its respective attention coefficient.

6. The document processing method according to claim 1, characterized in that, The step of determining the encoding strategy for each document element based on its semantic category and importance score includes: If the document element is a vectorizable document element, the document element is vectorized and encoded to compress it. The vectorizable document element includes at least text-based document elements and graphic-based document elements; or If the document element is an image-type document element, determine the color complexity of the image-type document element to determine the encoding strategy of the image-type document element.

7. The document processing method according to claim 6, characterized in that, If the document element is a vectorizable document element, the process of vectorizing the document element includes: When the vectorizable document element is a text-type document element, extract the text information of the text-type document element, wherein the text information includes at least one or more of the following: text content, font, font size, color, and position; and The text information is saved as vector data.

8. The document processing method according to claim 6, characterized in that, If the document element is a vectorizable document element, the process of vectorizing the document element includes: When the vectorizable document element is a graphic document element, extract the wireframe information and / or text information of the graphic document element. The wireframe information includes at least the wireframe row and column relationships, and the text information includes at least one or more of the following: text content, font, font size, color, and position. The system stores structured data, which is used to describe the wireframe information and / or the text information.

9. The document processing method according to claim 6, characterized in that, The step of determining the color complexity of the image-type document element to determine the encoding strategy of the image-type document element includes: When the color complexity is low, the image-type document elements are implicitly vectorized and compressed based on the importance score; In cases of high color complexity, the image-type document element is determined to be either a photographic image or a line image, and the encoding strategy for the image-type document element is determined based on the determination result and the importance score.

10. The document processing method according to claim 1, characterized in that, After determining the encoding strategy for each document element based on its semantic category and importance score, the method further includes: The strategy interface displays the encoding strategy corresponding to each document element; In the policy interface, the system receives policy change instructions from the user for any document element, changes the encoding policy of the document element, and uses the changed encoding policy as the encoding policy for the document element.

11. The document processing method according to claim 1, characterized in that, The method further includes: When the file to be compressed includes multiple pages of document images, for any two adjacent document images, the multiple document elements included in the current document image are compared with the multiple document elements included in the previous document image; If the current document image and the previous document image have the same document element, then when compressing the current document image, the reference pointer information corresponding to the same document element is stored, and the reference pointer information is used to point to the same document element in the previous document image; If the current document image and the previous document image have some identical document elements, then when compressing the current document image, the difference data corresponding to the partially identical document elements is stored; wherein, the difference data is used to represent the difference between the partially identical document elements in the current document image and in the previous document image.

12. The document processing method according to any one of claims 1 to 11, characterized in that, After compressing each document element in the document image based on the encoding strategy for each document element, the method further includes: Based on the association relationship of multiple document elements in the file to be compressed, a structure directory file is generated. The structure directory file includes at least one or more of the following: the size of each page of the document image, the identifiers of the multiple document elements included in each page of the document image, the semantic category, the bounding box, the level, the reference pointer information, and the logical relationship between the multiple document elements included in each page of the document image. Based on the structure directory file and the compressed file of the file to be compressed, a decompressed file package is generated, wherein the compressed file contains the compressed data of all document elements of all document images in the file to be compressed.

13. The document processing method according to claim 12, characterized in that, The method further includes: Open the decompressed file package with the specified reader, and display each document in the file to be compressed based on the structure directory file and the compressed file; Receive user editing instructions for a document and edit or modify the document accordingly.

14. A document processing apparatus, characterized in that, The device includes: The semantic segmentation module is used to segment the document image into multiple document elements based on the semantic category of the document image in the file to be compressed; The first determining module is used to determine the importance scores of a plurality of document elements, the importance scores being used to reflect the degree of attention paid to the document elements; The second determining module is used to determine the encoding strategy of each document element based on the semantic category and the importance score of each document element; A document compression module is used to compress each document element in the document image based on the encoding strategy of each document element.

15. A computer program product, comprising a computer program, characterized in that, When a computer program is executed by a processor, it implements the document processing method as described in any one of claims 1 to 13.

16. A computer-readable storage medium for storing computer-readable instructions, characterized in that, When computer-readable instructions are executed by a processor, the processor performs the document processing method as described in any one of claims 1 to 13.

Images