Format conversion method and system based on multi-modal fusion and generative adversarial network

Abstract

The invention relates to the technical field of document and image processing, and provides a format conversion method and system based on multi-modal fusion and generative adversarial networks, and the method comprises the steps: obtaining document layout information of a document format original file through a deep learning model according to the extracted document multi-modal information of the original file, obtaining document conversion style information by utilizing a document generative adversarial network model according to the document layout information and the first target style requirement, and converting the original file into a PDF format file according to the document conversion style information; for the image format original file, obtaining image understanding information by using an image analysis model according to the extracted image multi-modal information of the original file, and obtaining image conversion style information by using an image generative adversarial network model according to the image understanding information, a target image type requirement and a second target format requirement, and converting the original file into a PDF format file according to the image conversion style information. And PDF files meeting specific requirements are efficiently and accurately formed.

Description

Technical Field

[0001] This application relates to the field of document and image processing technology, specifically to a format conversion method and system based on multimodal fusion and generative adversarial networks. Background Technology

[0002] With the increasing prevalence of paperless office practices, a large amount of information is presented in document or image format. However, some professional fields require converting these documents or images into PDF files that meet specific style requirements. The most common format conversion method currently available is to convert documents or images to PDF files while preserving their original style. This method, however, cannot guarantee that the converted PDF file meets specific style requirements.

[0003] For document format conversion, some technologies employ template matching and rule-based conversion to ensure the converted PDF file meets specific style requirements. However, for converting complex documents, these methods only perform preliminary text structuring without deeply integrating images, tables, and the overall hierarchical information of the document. This makes it difficult for the converted PDF file to fully retain the complex content of the original document, leading to problems such as content loss and table misalignment.

[0004] For image format conversion, in order to make the converted PDF file conform to specific style requirements, some technologies combine text tags with image recognition. However, they cannot perform targeted format conversion according to the professional characteristics of the image type, and the converted PDF file is difficult to meet the requirements of professional fields.

[0005] Therefore, there is an urgent need to propose a method that can accurately and efficiently convert documents into PDF files that meet specific style requirements, and accurately and efficiently convert images into PDF files that meet image type requirements and specific style requirements. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a format conversion method and system based on multimodal fusion and generative adversarial networks. It integrates multimodal information such as text, graphics, and hierarchy of a document to generate a PDF file that meets specific style requirements, and integrates multimodal information such as basic information and metadata of an image to generate a PDF file that meets the target image type requirements and specific style requirements. The conversion quality is improved by introducing a generative adversarial network model.

[0007] The technical solution adopted in this application is as follows:

[0008] Firstly, this application provides a format conversion method based on multimodal fusion and generative adversarial networks, the method comprising:

[0009] Obtain the original file and determine its format type; the format type includes document format and image format.

[0010] If the original file is in document format, extract the document multimodal information from the original file, use a deep learning model to obtain document layout information based on the document multimodal information, use a document generative adversarial network model to obtain document conversion style information based on the document layout information and the first target style requirements, and convert the original file into a PDF file based on the document conversion style information.

[0011] If the original file is in image format, extract the image multimodal information from the original file, obtain image understanding information using an image analysis model based on the image multimodal information, obtain image conversion style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, and convert the original file into a PDF file based on the image conversion style information.

[0012] Secondly, this application provides a format conversion system based on multimodal fusion and generative adversarial networks, the system comprising:

[0013] The judgment unit is used to obtain the original file and determine the format type of the original file; the format type includes document format and image format;

[0014] The document format conversion unit is used to extract the document multimodal information of the original file if the original file is in document format, obtain the document layout information using a deep learning model based on the document multimodal information, obtain the document conversion style information using a document generative adversarial network model based on the document layout information and the first target style requirement, and convert the original file into a PDF format file based on the document conversion style information.

[0015] The image format conversion unit is used to extract the multimodal information of the original file if the original file is in image format, obtain image understanding information using an image analysis model based on the multimodal information, obtain image conversion style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, and convert the original file into a PDF format file based on the image conversion style information.

[0016] Thirdly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described format conversion method based on multimodal fusion and generative adversarial networks.

[0017] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described format conversion method based on multimodal fusion and generative adversarial networks.

[0018] The above-mentioned technical solution adopted in this application can achieve the following beneficial effects:

[0019] The format conversion method based on multimodal fusion and generative adversarial networks provided in this application first obtains the original file and determines whether the original file is in document format or image format. For the original file in document format, document layout information is obtained using a deep learning model based on the extracted document multimodal information of the original file. Document conversion style information is obtained using a document generative adversarial network model based on the document layout information and the first target style requirement. The original file is then converted into a PDF file based on the document conversion style information. For the original file in image format, image understanding information is obtained using an image analysis model based on the extracted image multimodal information of the original file. Image conversion style information is obtained using an image generative adversarial network model based on the image understanding information, the target image type requirement, and the second target style requirement. The original file is then converted into a PDF file based on the image conversion style information.

[0020] The method provided in this application has the following advantages:

[0021] The conversion method provided in this application can completely preserve complex content such as paragraphs, tables, images, worksheets, and illustrations in the original document format, and maintain multi-level nested structures. The content loss rate of the converted PDF file is close to 0, and the problem of style misalignment is effectively solved.

[0022] The conversion method provided in this application can perform targeted conversion of original image files, especially professional images such as medical images and engineering drawings, based on their professional characteristics. In medical image conversion tests, the proportion of key indicators such as image clarity and annotation completeness in the converted PDF files that meet the presentation standards of medical professionals reaches over 95%, satisfying the usage requirements of the professional field.

[0023] With the optimization of the generative adversarial network model, the conversion results are significantly improved in terms of accuracy and professionalism, regardless of the original document or image format. The generated PDF files have a similarity of over 98% to the real files in terms of style compliance and content presentation. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 A flowchart illustrating a format conversion method according to an embodiment of this application is shown;

[0026] Figure 2 A flowchart illustrating a format conversion method according to another embodiment of this application is shown;

[0027] Figure 3 This diagram illustrates the structure of a format conversion system according to an embodiment of the present application.

[0028] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of this application is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The concept behind this application is that a large number of documents or images need to be converted into PDF files. For some professional fields, there are specific style requirements for the converted PDF files. For example, the banking industry has special requirements for the font, font size, margins, and line spacing of the converted PDF files; or, for medical images, there are special requirements for the presentation standards of the converted PDF files. Existing common format conversion methods usually preserve the original style of the document or image, failing to meet the requirement that the converted PDF file conforms to specific style requirements. For document format conversion, some technologies use template matching and rule-based conversion methods, but for complex documents, they only perform preliminary text structuring processing without deeply integrating images, tables, and the overall hierarchical information of the document. This makes it difficult for the converted PDF file to fully retain the complex content of the original document, leading to problems such as content loss and table misalignment. For image format conversion, some technologies combine text tags with image recognition, but they cannot perform targeted format conversion based on the professional characteristics of the image type, making it difficult for the converted PDF file to meet the requirements of professional fields.

[0031] To address the aforementioned drawbacks, this application proposes a format conversion method and system based on multimodal fusion and generative adversarial networks. It integrates multimodal information such as text, graphics, and hierarchy of a document to generate a PDF file that meets specific style requirements, and integrates multimodal information such as basic information and metadata of an image to generate a PDF file that meets the target image type requirements and specific style requirements. The conversion quality is improved by introducing a generative adversarial network model.

[0032] Figure 1A flowchart illustrating a format conversion method based on multimodal fusion and generative adversarial networks proposed in one embodiment of this application is shown. Figure 1 As shown, the method in this embodiment includes steps S110 to S130.

[0033] Step S110: Obtain the original file and determine the format type of the original file; wherein, the format type includes document format and image format.

[0034] The format conversion method proposed in this embodiment can be applied to the server side. The server side can be implemented using a standalone server or a service platform composed of multiple servers.

[0035] In this embodiment, the target format conversion is PDF. Therefore, the server first obtains the original file, which is not in PDF format. In practice, the server can actively or passively retrieve the original file to be processed from the database.

[0036] For example, in some situations, personnel need to instruct the server to process a specific raw file. In this case, the personnel can cache the raw file in a specific storage area of ​​the database and instruct the server to retrieve the raw file from that specific storage area for processing. Conversely, in some cases, a large number of raw files need to be processed. In this case, the large number of raw files can be cached in a non-specific storage area of ​​the database, and the server can proactively retrieve the raw files from the non-specific storage area for processing sequentially.

[0037] After obtaining the original file, the server can determine its format type based on its file extension. If the file extension is ".doc", ".docx", ".xls", ".xlsx", etc., the original file is a document format; if the file extension is ".jpg", ".png", etc., the original file is an image format. The server can also determine the original file format type using other existing methods.

[0038] The server employs different processing methods for the original file depending on whether it is identified as a document or an image.

[0039] Step S120: If the original file is in document format, extract the document multimodal information of the original file, obtain the document layout information using a deep learning model based on the document multimodal information, obtain the document conversion style information using a document generative adversarial network model based on the document layout information and the first target style requirements, and convert the original file into a PDF format file based on the document conversion style information.

[0040] If the server determines that the original file is in document format, it identifies elements such as paragraphs, tables, images, worksheets, and illustrations in the document, and parses the document's multimodal information such as text, graphics, and hierarchy. It then performs feature processing on the document's multimodal information so that it can be effectively utilized by the deep learning model.

[0041] The multimodal information of the document after feature processing is input into a deep learning model. The deep learning model then performs a deep understanding of the multimodal information of the document after feature processing to obtain the document layout information.

[0042] A document generation adversarial network (GAN) model participates in the conversion process. The GAN model can include a first generator and a first discriminator. The first generator generates preliminary document conversion style information based on document layout information and learns the first target style requirement. The first discriminator evaluates the realism of the preliminary document conversion style information and feeds back an error signal to the first generator based on the difference between the preliminary document conversion style information and a real document that conforms to the first target style requirement. The first generator adjusts its generation strategy based on the error signal, optimizing the preliminary document conversion style information to more closely resemble a real document that conforms to the first target style requirement. Through continuous adversarial training and optimization, the first generator ultimately optimizes and generates document conversion style information.

[0043] The primary target style requirements are used to ensure that paragraphs, images, tables, worksheets, and illustrations in the converted PDF file conform to specific styles. For example, primary target style requirements may include details such as margins, line spacing, font, font size, table size, table layout, image size, and image layout.

[0044] The server can create an initial PDF file and then add paragraphs, tables, images, worksheets, illustrations, and other elements from the original document to this initial PDF file according to the document conversion style information, thus forming a PDF file. The converted PDF file can completely retain all the complex content of the original document and conforms to the primary target style requirements.

[0045] Step S130: If the original file is in image format, extract the image multimodal information of the original file, obtain image understanding information using an image analysis model based on the image multimodal information, obtain image conversion style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, and convert the original file into a PDF format file based on the image conversion style information.

[0046] If the server determines that the original file is in image format, it identifies basic image information and metadata elements in the image, parses the global and attribute multimodal information of the image, performs feature processing on the multimodal information of the image, and enables it to be effectively utilized by the image analysis model.

[0047] The multimodal information of the image after feature processing is input into the image analysis model. The image analysis model performs in-depth understanding of the multimodal information of the image after feature processing to obtain image understanding information.

[0048] Image generative adversarial networks (GANs) participate in the conversion process. A GAN model can include a second generator and a second discriminator. The second generator, based on image understanding information and learning the target image type requirements and second target style requirements, generates preliminary image conversion style information. The second discriminator evaluates the quality of the preliminary image conversion style information and feeds back an error signal to the first generator based on the difference between the preliminary target conversion style information and a real image that meets the target image type requirements and second target style requirements. The second generator adjusts its generation strategy based on the error signal, optimizing the preliminary image conversion style information to more closely approximate the quality of a real image that meets the target image type requirements and second target style requirements. Through continuous adversarial training and optimization, the second generator ultimately optimizes and generates image conversion style information.

[0049] The first target image type requirement indicates that the converted PDF file's image conforms to specific presentation requirements. For example, for medical images, the target image type requirement may include: contrast, brightness, sharpness, and region of interest (ROI) annotations; for engineering drawings, the target image type requirement may include: sharpness, compression ratio, resolution, and dimension annotations. The second target style requirement indicates that the converted PDF file's image conforms to a specific style. For example, the second target style requirement may include details such as the space occupied on the PDF page and image size.

[0050] The server can create an initial PDF file or obtain a PDF file to be embedded, and add the image to the initial PDF file or the PDF file to be embedded according to the image conversion style information, thereby forming a PDF file. The converted PDF file can ensure that the image meets the presentation requirements of the professional field and also meets the second target style requirements.

[0051] from Figure 1As shown in the method, the format conversion method based on multimodal fusion and generative adversarial networks provided in this application first obtains the original file and determines its format type; the format type includes document format and image format; for the original file in document format, the document multimodal information of the original file is extracted, and the document layout information is obtained using a deep learning model based on the document multimodal information; the document conversion style information is obtained using a document generative adversarial network model based on the document layout information and the first target style requirement; and the original file is converted into a PDF format file based on the document conversion style information. For the original file in image format, the image multimodal information of the original file is extracted, and the image understanding information is obtained using an image analysis model based on the image multimodal information; the image conversion style information is obtained using an image generative adversarial network model based on the image understanding information, the target image type requirement, and the second target style requirement; and the original file is converted into a PDF format file based on the image conversion style information.

[0052] The method provided in this application has the following advantages:

[0053] The conversion method provided in this application can completely preserve complex content such as paragraphs, tables, images, worksheets, and illustrations in the original document format, and maintain multi-level nested structures. The content loss rate of the converted PDF file is close to 0, and the problem of style misalignment is effectively solved.

[0054] The conversion method provided in this application can perform targeted conversion of original image files, especially professional images such as medical images and engineering drawings, based on their professional characteristics. In medical image conversion tests, the proportion of key indicators such as image clarity and annotation completeness in the converted PDF files that meet the presentation standards of medical professionals reaches over 95%, satisfying the usage requirements of the professional field.

[0055] With the optimization of the generative adversarial network model, the conversion results are significantly improved in terms of accuracy and professionalism, regardless of the original document or image format. The generated PDF files have a similarity of over 98% to the real files in terms of style compliance and content presentation.

[0056] The following sections describe in detail the conversion methods for original files in document format and original files in image format.

[0057] In some optional implementations, in the above method, step S120, extracting the document multimodal information of the original file, includes: extracting paragraphs, tables, and images from the original file in Word format, parsing the paragraph text and paragraph structure of the paragraphs, parsing the table text and table structure of the tables, and parsing the image information and image structure of the images; or, extracting worksheets and illustrations from the original file in Excel format, parsing the worksheet text and worksheet structure of the worksheets, and parsing the illustration information and illustration structure of the illustrations.

[0058] When the server extracts multimodal information from the original file, different extraction methods can be used depending on whether the original file is in Word or Excel format. Examples are given below for both Java and Python environments.

[0059] In a Java environment, the fully functional Apache POI library can be used to extract multimodal information from raw documents. For Word format raw documents, the XWPFDocument class can be used to extract elements such as paragraphs, tables, and images. It iterates through the paragraphs in the raw document, parsing them to extract paragraph text (including but not limited to the text content) and paragraph structure (including but not limited to paragraph formatting attributes). For tables inserted in Word format raw documents, the tables are parsed to extract table text (including but not limited to the text and data within the table) and table structure (including but not limited to table formatting attributes). For images embedded in Word format raw documents, the images are parsed to extract image information (including but not limited to the image's binary data) and image format (including but not limited to image formatting attributes). For Excel format raw documents, the XSSFWorkbook class can be used to extract worksheets, illustrations, and other elements from the raw document. Parse worksheets to extract worksheet text (including but not limited to text, data, formulas, etc.) and worksheet structure (including but not limited to cell styles, worksheet name, data types, etc.); for embedded inserts in Excel format original files, parse and extract illustration information (including but not limited to binary data of illustrations, etc.) and illustration format (including but not limited to layout attributes of illustrations, etc.).

[0060] In a Python environment, for Word documents, the `Document` class and similar classes from the `python-docx` library can be used to extract multimodal information. This involves extracting elements such as paragraphs, tables, and images, and then iterating through these elements to extract paragraph text and structure, table text and structure, and image information and structure. For Excel documents, the pandas library can be used to extract multimodal information. This involves extracting worksheets, illustrations, and other elements, and then extracting worksheet text and structure, illustration information and formatting. Furthermore, the pandas library can convert tabular data into a DataFrame structure, facilitating subsequent data processing and analysis.

[0061] In some optional implementations, in the above method, step S120, obtaining document layout information using a deep learning model based on document multimodal information, and obtaining document conversion style information using a document generative adversarial network model based on the document layout information and the first target style requirement, includes: preprocessing text to form text features, extracting feature vectors of information to form graphic features, constructing a hierarchical structure tree based on the structure to form hierarchical features, and fusing text features, graphic features, and hierarchical features into multimodal document features; inputting the multimodal document features into a deep learning model based on the Transformer architecture and self-attention mechanism to obtain document layout information; the document generative adversarial network model includes a first generator and a first discriminator, the first generator generates document conversion style information based on the document layout information and the first target style requirement, and the first discriminator performs adversarial training on the results of the first generator to optimize the document conversion style information.

[0062] After the server extracts the multimodal information of the document, it needs to perform feature processing on the multimodal information so that it can be effectively utilized by subsequent deep learning models.

[0063] For text (including paragraph and table text for Word format and worksheet text for Excel format), preprocessing operations such as word segmentation, part-of-speech tagging, and named entity recognition are performed to transform the text into structured text features.

[0064] For information (including image information for Word format original files, and illustration information for Excel format original files), extract its feature vectors (such as color, texture, etc.) to form graphic features.

[0065] For the structure (including paragraph structure, table structure, and image structure for Word format original files, and worksheet structure and illustration structure for Excel format original files), a hierarchical structure tree is constructed to clarify the nesting relationships and order between elements such as paragraphs, tables, and images, or between elements such as worksheets and illustrations, thus forming hierarchical features.

[0066] After fusing text features, graphic features, and hierarchical features into multimodal document features, these multimodal document features are then input into a deep learning model based on the Transformer architecture and self-attention mechanism.

[0067] The self-attention mechanism in the Transformer architecture enables deep learning models to simultaneously focus on elements at different locations within a document's original file, allowing for a comprehensive analysis of textual semantics, graphical structure, and hierarchical information. For example, when processing complex Word documents, deep learning models can understand the relationships between text and data in tables and paragraph descriptions, as well as the position and function of images and tables within the overall hierarchical structure of the Word document. Through this deep understanding, deep learning models can generate document layout information. This layout information may include, but is not limited to, paragraph text formatting (font size, font, line spacing, etc.), table drawing methods (borders, headers, cell attributes, etc.), and image embedding positions and sizes.

[0068] A document generation adversarial network (GAN) model is involved in the format conversion process. The GAN model consists of a first generator and a first discriminator. The first generator generates preliminary document conversion style information based on document layout information and by learning the first target style requirements.

[0069] After the first generator generates preliminary document conversion style information, the first discriminator compares this information with a real document that meets the first target style requirements to determine the authenticity of the preliminary document conversion style information. The first discriminator feeds back the difference between the preliminary document conversion style information and the real document as an error signal to the first generator.

[0070] The first generator further learns the features of real documents based on the error signals, attempting to optimize the initial document transformation style information. Through continuous adversarial training and optimization, the first generator finally generates optimized document transformation style information.

[0071] In some optional implementations, in the above method, step S120, converting the original file into a PDF file according to the document conversion style information, includes: creating a first initial PDF file; adding paragraphs, tables, and pictures to the first initial PDF file to form a PDF file according to the document conversion style information; or, adding worksheets and illustrations to the first initial PDF file to form a PDF file according to the document conversion style information.

[0072] The following examples will still use Java and Python environments respectively.

[0073] In a Java environment, a first initial PDF file is created. Then, using the iText library, elements such as paragraphs, tables, pictures, worksheets, and illustrations are added to the first initial PDF file according to specific style requirements based on the document transformation style information optimized by the first generator.

[0074] In a Python environment, an initial PDF file is created. Then, using either the `reporatlab` or `fpdf` library, elements such as paragraphs, tables, images, worksheets, and illustrations are added to the initial PDF file according to specific style requirements, based on document conversion style information optimized by the first generator. The `reporatlab` library is flexible and comprehensive, while the `fpdf` library is simpler and easier to use.

[0075] At this point, the server has converted the original document file into a PDF file that conforms to the first target style requirement. The converted PDF file not only retains all the content of the original file but also meets the specific style requirements of the PDF format.

[0076] In some optional implementations, in the above method, step S130, extracting the image multimodal information of the original file, includes: reading the original file in image format; parsing the image basic information and metadata of the original file.

[0077] The server extracts multimodal information from the original image files. The following example uses a Python environment.

[0078] The Pillow library can be used to read raw files in formats such as JPG and PNG, and parse the basic image information and metadata of the raw files. The Pillow library can extract basic image information such as image pixels, image size, and image color mode, as well as metadata such as image creation time, modification time, file size, camera model, and lens parameters. This multimodal image information provides rich context for subsequent image analysis, helping to understand the image content more accurately.

[0079] In some optional implementations, in the above method, step S130, obtaining image understanding information using an image analysis model based on image multimodal information, and obtaining image transformation style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, includes: forming global features based on basic image information, forming attribute features based on metadata, and fusing global features and attribute features into multimodal image features; inputting the multimodal image features into an image analysis model based on Transformer architecture and self-attention mechanism to obtain image understanding information; the image generative adversarial network model includes a second generator and a second discriminator, the second generator generates image transformation style information based on image understanding information, target image type requirements, and second target style requirements, and the second discriminator performs adversarial training optimization on the results of the second generator to optimize the image transformation style information.

[0080] After the server extracts multimodal information from the image, it performs feature processing on this information to enable it to be effectively utilized by subsequent image analysis models. Global features are formed based on the basic image information, and attribute features are formed based on metadata. Finally, the global features and attribute features are fused into multimodal image features.

[0081] Multimodal image features are input into an image analysis model based on the Transformer architecture and self-attention mechanism.

[0082] The self-attention mechanism in the Transformer architecture analyzes multimodal image features, enabling it to identify image types, spatial relationships within images, and semantic information. For example, the identified image types can include, but are not limited to, medical images, engineering drawings, and ordinary photographs; for medical images, it can identify regions of interest (such as key areas of organs or key areas of lesions); for engineering drawings, it can identify lines, symbols, and dimension annotations. This comprehensive understanding of multimodal image features provides a basis for accurate subsequent conversion.

[0083] Image generative adversarial networks (GANs) participate in the format conversion process. A GAN model consists of a second generator and a second discriminator. The second generator, based on image understanding information and learning the target image type requirements and second target style requirements, generates preliminary image conversion style information. For example, for engineering drawings, it generates image clarity, compression rate, resolution, and dimension annotations in PDF format that meet engineering professional image presentation standards; for medical images, it generates contrast, brightness, clarity, and region of interest annotations in PDF format that meet medical professional image presentation standards.

[0084] After the second generator generates preliminary image conversion style information, the second discriminator compares this information with a real image that meets both the target image type requirements and the second target style requirements to determine the quality of the preliminary image conversion style information. The second discriminator then feeds back the quality difference between the preliminary image conversion style information and the real image as an error signal to the second generator.

[0085] The second generator further learns the features of the real image based on the error signal, attempting to optimize the initial image transformation style information. Through continuous adversarial training and optimization, the second generator finally generates optimized image transformation style information.

[0086] In some optional implementations, in the above method, step S130, converting the original file into a PDF file according to the image conversion style information, includes: creating a second initial PDF file, adding an image to the second initial PDF file according to the image conversion style information to form a PDF file; or, obtaining a PDF file to be embedded, adding an image to the PDF file to be embedded according to the image conversion style information to form a PDF file.

[0087] The following example will still use the Python environment.

[0088] Converting raw image files to PDF format can be done through direct conversion or embedding. For direct conversion, a second initial PDF file can be created, and then the image can be added to this second initial PDF file using the `img2pdf` library based on the image conversion style information. For embedding, the PDF file to be embedded can be obtained, and then the image can be added to this PDF file using the `reportlab` library based on the image conversion style information.

[0089] At this point, the server has converted the original image file into a PDF file that meets the requirements of the target image type and the second target style. The converted PDF file ensures that the image meets the presentation requirements of the professional field (including clarity, completeness, professionalism, etc.) and also conforms to the specific style requirements of PDF files.

[0090] Existing technologies typically employ rule-based multimodal fusion and conversion methods. This involves pre-defining a series of conversion rules to connect original files of different formats with PDF files. Multimodal information is then processed to convert the original files of different formats into PDF files based on these rules. However, this method lacks flexibility and struggles to handle complex and ever-changing original file structures and content, as well as the evolving style requirements of PDF files. Furthermore, the development and maintenance of conversion rules are costly; when encountering new original file formats and content or new special style requirements for PDF files, extensive manual adjustments to the conversion rules are necessary, making adaptive conversion impossible.

[0091] The format conversion method proposed in this application, based on multimodal fusion and generative adversarial networks, integrates multimodal information such as text, graphics, and hierarchy of documents to generate PDF files that meet specific style requirements, and integrates multimodal information such as basic information and metadata of images to generate PDF files that meet the requirements of target image type and specific style requirements. By introducing a generative adversarial network model, the conversion quality is improved, enabling rapid adaptation and high-quality conversion.

[0092] For example, for a newly emerging spreadsheet source file with a special format, rule-based methods may take weeks to develop and debug new conversion rules, while the format conversion method proposed in this application can achieve accurate conversion efficiently.

[0093] Figure 2 A flowchart illustrating a format conversion method based on multimodal fusion and generative adversarial networks, proposed in another embodiment of this application, is shown. According to... Figure 2 As shown, the method in this embodiment includes steps S201 to S217.

[0094] Step S201: Obtain the original file and determine its format type. If it is a document format, proceed to step S202; if it is an image format, proceed to step S212.

[0095] Step S202: Determine the format type of the original document. If it is a Word document, proceed to step S203 or S204; if it is an Excel document, proceed to step S205 or S206.

[0096] Step S203: In the Java environment, use the XWPFDocument class to extract paragraphs, tables, and images from the original file, extracting paragraph text and structure, table text and structure, and image information and format. Proceed to step S207.

[0097] Step S204: In the Python environment, use the python-docx library to extract paragraphs, tables, and images from the original file, extracting paragraph text and structure, table text and structure, and image information and format. Proceed to step S207.

[0098] Step S205: In the Java environment, use the XSSFWorkbook class to extract the worksheets and illustrations from the original file, extracting the worksheet text, worksheet structure, illustration information, and illustration format. Proceed to step S207.

[0099] Step S206: In the Python environment, use the Pandas library to extract the worksheets and illustrations from the original file, extracting the worksheet text, worksheet structure, illustration information, and illustration format. Proceed to step S207.

[0100] Step S207: Preprocess the text to form text features, extract feature vectors from the information to form graphical features, construct a hierarchical structure tree based on the structure to form hierarchical features, and fuse the text features, graphical features, and hierarchical features into multimodal document features. Proceed to step S208.

[0101] Step S208: Input the multimodal document features into a deep learning model based on the Transformer architecture and self-attention mechanism to obtain document layout information. Proceed to step S209.

[0102] Step S209: The first generator of the document generation adversarial network model generates document transformation style information based on the document layout information and the first target style requirements. The first discriminator of the document generation adversarial network model performs adversarial training on the result of the first generator to optimize the document transformation style information. Proceed to step S210 or step S211.

[0103] Step S210: For the Java environment, create a first initial PDF file and use the iText library to add paragraphs, tables, pictures or worksheets, and illustrations to the first initial PDF file according to the document conversion style information.

[0104] Step S211: In the Python environment, create a first initial PDF file and use the reporatlab library or fpdf library to add paragraphs, tables, pictures or worksheets and illustrations to the first initial PDF file according to the document conversion style information.

[0105] Step S212: In the Python environment, extract the basic image information and metadata of the original file using the Pillow library. Proceed to step S213.

[0106] Step S213: Generate global features based on basic image information and attribute features based on metadata. Fuse the global features and attribute features into multimodal image features. Proceed to step S214.

[0107] Step S214: Input the multimodal image features into the image analysis model based on the Transformer architecture and self-attention mechanism to obtain image understanding information. Proceed to step S215.

[0108] In step S215, the second generator of the image generative adversarial network model generates image transformation style information based on image understanding information, target image type requirements, and second target style requirements. The second discriminator of the image generative adversarial network model performs adversarial training on the result of the second generator to optimize the image transformation style information. Proceed to step S216 or step S217.

[0109] Step S216: In the Python environment, create a second initial PDF file and use the img2pdf library to add images to the second initial PDF file based on image conversion style information.

[0110] Step S217: In the Python environment, obtain the PDF file to be embedded, and use the reportlab library to add the image to the PDF file based on the image conversion style information.

[0111] Figure 3 A schematic diagram of the structure of a format conversion system based on multimodal fusion and generative adversarial networks, according to an embodiment of this application, is shown. Figure 3 As shown, the system 300 includes:

[0112] The judgment unit 310 is used to obtain the original file and determine the format type of the original file; wherein, the format type includes document format and image format;

[0113] The document format conversion unit 320 is used to extract the document multimodal information of the original file if the original file is in document format, obtain the document layout information using a deep learning model based on the document multimodal information, obtain the document conversion style information using a document generative adversarial network model based on the document layout information and the first target style requirement, and convert the original file into a PDF format file based on the document conversion style information.

[0114] The image format conversion unit 330 is used to extract the image multimodal information of the original file if the original file is in image format, obtain image understanding information using an image analysis model based on the image multimodal information, obtain image conversion style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, and convert the original file into a PDF format file based on the image conversion style information.

[0115] In some embodiments of this application, in the above-described system 300, the document format conversion unit 320 includes: a document multimodal information extraction module, used to extract paragraphs, tables and images from the original Word format file, parse the paragraph text and paragraph structure of the paragraphs, parse the table text and table structure of the tables, and parse the image information and image structure of the images; or, to extract worksheets and illustrations from the original Excel format file, parse the worksheet text and worksheet structure of the worksheets, and parse the illustration information and illustration structure of the illustrations.

[0116] In some embodiments of this application, in the above-described system 300, the document format conversion unit 320 includes: a document conversion style information generation module, used to preprocess text to form text features, extract feature vectors of information to form graphic features, construct a hierarchical structure tree based on the structure to form hierarchical features, and fuse text features, graphic features and hierarchical features into multimodal document features; input the multimodal document features into a deep learning model based on the Transformer architecture and self-attention mechanism to obtain document layout information; the document generation adversarial network model includes a first generator and a first discriminator, the first generator generates document conversion style information according to the document layout information and the first target style requirement, and the first discriminator performs adversarial training optimization on the result of the first generator to optimize the document conversion style information.

[0117] In some embodiments of this application, in the system 300 described above, the document format conversion unit 320 includes: a document style conversion module, used to create a first initial PDF file; add paragraphs, tables and pictures to the first initial PDF file according to document conversion style information to form a PDF format file; or add worksheets and illustrations to the first initial PDF document according to document conversion style information to form a PDF format file.

[0118] In some embodiments of this application, in the system 300 described above, the image format conversion unit 330 includes: an image multimodal information extraction module, used to read the original file of the image format; and to parse the image basic information and metadata of the original file.

[0119] In some embodiments of this application, in the above-described system 300, the image format conversion unit 330 includes: an image conversion style information generation module, used to form global features based on image basic information, form attribute features based on metadata, and fuse the global features and attribute features into multimodal image features; input the multimodal image features into an image analysis model based on Transformer architecture and self-attention mechanism to obtain image understanding information; the image generative adversarial network model includes a second generator and a second discriminator, the second generator generates image conversion style information based on image understanding information, target image type requirements, and second target style requirements, and the second discriminator performs adversarial training optimization on the results of the second generator to optimize the image conversion style information.

[0120] In some embodiments of this application, in the system 300 described above, the image format conversion unit 330 includes: an image style conversion module, used to create a second initial PDF file, and add an image to the second initial PDF file according to image conversion style information to form a PDF format file; or, to obtain a PDF file to be embedded, and add an image to the PDF file to be embedded according to image conversion style information to form a PDF format file.

[0121] It should be noted that the format conversion system 300 based on multimodal fusion and generative adversarial networks can implement the aforementioned format conversion methods based on multimodal fusion and generative adversarial networks, which will not be elaborated further.

[0122] Figure 4 This application shows a schematic diagram of the structure of a computer device according to one embodiment of the present application. Figure 4 As shown, the internal structure of this computer device may include a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external devices via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a format conversion method based on multimodal fusion and generative adversarial networks.

[0123] In one embodiment, the computer device provided in this application includes a memory and a processor. The memory stores a database and a computer program that can run on the processor. When the processor executes the computer program, it performs the following steps:

[0124] Obtain the original file and determine its format type; the format type includes document format and image format.

[0125] If the original file is in document format, extract the document multimodal information from the original file, use a deep learning model to obtain document layout information based on the document multimodal information, use a document generative adversarial network model to obtain document conversion style information based on the document layout information and the first target style requirements, and convert the original file into a PDF file based on the document conversion style information.

[0126] If the original file is in image format, extract the image multimodal information from the original file, obtain image understanding information using an image analysis model based on the image multimodal information, obtain image conversion style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, and convert the original file into a PDF file based on the image conversion style information.

[0127] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, the computer program performing the following steps when executed by a processor:

[0128] Obtain the original file and determine its format type; the format type includes document format and image format.

[0129] If the original file is in document format, extract the document multimodal information from the original file, use a deep learning model to obtain document layout information based on the document multimodal information, use a document generative adversarial network model to obtain document conversion style information based on the document layout information and the first target style requirements, and convert the original file into a PDF file based on the document conversion style information.

[0130] If the original file is in image format, extract the image multimodal information from the original file, obtain image understanding information using an image analysis model based on the image multimodal information, obtain image conversion style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, and convert the original file into a PDF file based on the image conversion style information.

[0131] It should be noted that the functions or steps that can be implemented by the computer device or computer-readable storage medium described above can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0134] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A format conversion method based on multimodal fusion and generative adversarial networks, characterized in that, The method includes: Obtain the original file and determine the format type of the original file; wherein, the format type includes document format and image format; If the original file is in the document format, extract the document multimodal information of the original file, obtain the document layout information using a deep learning model based on the document multimodal information, obtain the document conversion style information using a document generative adversarial network model based on the document layout information and the first target style requirement, and convert the original file into a PDF format file based on the document conversion style information; If the original file is in the image format, extract the image multimodal information of the original file, obtain image understanding information using an image analysis model based on the image multimodal information, obtain image conversion style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, and convert the original file into the PDF format file based on the image conversion style information.

2. The format conversion method based on multimodal fusion and generative adversarial networks according to claim 1, characterized in that, The extraction of document multimodal information from the original file includes: Extract paragraphs, tables, and images from the original Word document; parse the paragraph text and structure of the paragraphs; parse the table text and structure of the tables; and parse the image information and structure of the images. or, Extract the worksheets and illustrations from the original file in Excel format, parse the worksheet text and worksheet structure, and parse the illustration information and illustration structure.

3. The format conversion method based on multimodal fusion and generative adversarial networks according to claim 2, characterized in that, The step of obtaining document layout information using a deep learning model based on the document's multimodal information, and obtaining document transformation style information using a document generative adversarial network model based on the document layout information and the first target style requirement, includes: The text is preprocessed to form text features, the feature vectors of the information are extracted to form graphic features, a hierarchical structure tree is constructed based on the structure to form hierarchical features, and the text features, graphic features and hierarchical features are fused into multimodal document features; The multimodal document features are input into the deep learning model based on the Transformer architecture and self-attention mechanism to obtain the document layout information; The document generation adversarial network model includes a first generator and a first discriminator. The first generator generates the document transformation style information based on the document layout information and the first target style requirement. The first discriminator performs adversarial training on the results of the first generator to optimize the document transformation style information.

4. The format conversion method based on multimodal fusion and generative adversarial networks according to claim 2, characterized in that, The step of converting the original file into a PDF file based on the document conversion style information includes: Create the first initial PDF file; Based on the document conversion style information, the paragraphs, tables, and images are added to the first initial PDF file to form the PDF format file; or, The worksheet and illustrations are added to the first initial PDF document according to the document conversion style information to form the PDF format file.

5. The format conversion method based on multimodal fusion and generative adversarial networks according to claim 1, characterized in that, The extraction of image multimodal information from the original file includes: Read the original file in the image format; The image basic information and metadata of the original file are parsed.

6. The format conversion method based on multimodal fusion and generative adversarial networks according to claim 5, characterized in that, The step of obtaining image understanding information using an image analysis model based on the image multimodal information, and obtaining image transformation style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, includes: Global features are formed based on the image basic information, attribute features are formed based on the metadata, and the global features and attribute features are fused into multimodal image features; The multimodal image features are input into the image analysis model based on the Transformer architecture and self-attention mechanism to obtain the image understanding information; The image generative adversarial network model includes a second generator and a second discriminator. The second generator generates the image transformation style information based on the image understanding information, the target image type requirements, and the second target style requirements. The second discriminator performs adversarial training on the results of the second generator to optimize the image transformation style information.

7. The format conversion method based on multimodal fusion and generative adversarial networks according to claim 5, characterized in that, The step of converting the original file into the PDF format file based on the image conversion style information includes: Create a second initial PDF file, and add the image to the second initial PDF file according to the image conversion style information to form the PDF format file; or, Obtain the PDF file to be embedded, and add the image to the PDF file to form the PDF format file according to the image conversion style information.

8. A format conversion system based on multimodal fusion and generative adversarial networks, characterized in that, The system includes: A judgment unit is used to acquire the original file and determine the format type of the original file; wherein, the format type includes document format and image format; The document format conversion unit is configured to, if the original file is the document format, extract the document multimodal information of the original file, obtain document layout information using a deep learning model based on the document multimodal information, obtain document conversion style information using a document generative adversarial network model based on the document layout information and a first target style requirement, and convert the original file into a PDF format file based on the document conversion style information; An image format conversion unit is configured to, if the original file is in the image format, extract the image multimodal information of the original file, obtain image understanding information using an image analysis model based on the image multimodal information, obtain image conversion style information using an image generative adversarial network model based on the image understanding information, target image type requirements, and second target format requirements, and convert the original file into the PDF format file based on the image conversion style information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the format conversion method based on multimodal fusion and generative adversarial networks as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is instructed by the processor, it implements the steps of the format conversion method based on multimodal fusion and generative adversarial networks as described in any one of claims 1 to 7.

Images