PPT file content extraction method and system based on large language model

Abstract

The invention relates to a PPT file content extraction method and system based on a large language model, electronic equipment and a storage medium. The PPT file content extraction method comprises the steps that a PPT file is obtained through a crawler, and a video processing tool extracts PPT key frames and audio-to-text in a video; pPT image processing relates to enhancement, segmentation, OCR and object detection; the large language model application covers a multi-modal input layer, a vision-language fusion network and model fine tuning; and converting the PPT and video PPT data into HTML (Hypertext Markup Language) files. Compared with a PPT file processing method in the related technology, the method can automatically generate data required by large model training in batches. A large number of high-quality structured data sets are provided, the large model can better learn features and rules of PPT content, detailed text description can be generated according to visual information of PPT, PPT description is improved to be more accurate, and the quality of the data sets is higher.

Description

Technical Field

[0001] The present application relates to the field of PPT file processing, and in particular to a PPT file processing method, system, computer device and computer-readable storage medium based on a large language model. Background Art

[0002] At present, there are many ways to produce and analyze PPT (PowerPoint presentation) content. Traditional PPT production mainly relies on manual editing, template selection or automatic layout based on preset content. However, such methods have limitations in content understanding and expression. For complex content or dynamically changing scenes, the generated effects appear to be relatively rigid. Many existing automated PPT generation methods are often based on simple template matching and keyword extraction, lacking in-depth analysis of PPT visual information (such as images, charts, etc.) and natural language generation capabilities.

[0003] In addition, the current methods for extracting PPT content are relatively simple, mostly using OCR (optical character recognition) technology or image processing technology to extract text and simple visual information. However, PPT not only covers text content, but also contains multi-dimensional information such as images and animation effects, which are of great significance to the completeness and accuracy of content understanding. Existing technical means are usually difficult to effectively integrate and reproduce the multi-dimensional information in PPT.

[0004] In summary, the existing technology lacks efficient methods for PPT content generation and conversion, and there is also a missing problem in the PPT large model data set. Summary of the invention

[0005] The embodiments of the present application provide a method, system, computer device and computer-readable storage medium for extracting PPT file content based on a large language model, so as to at least solve the problem of lack of efficient methods for generating and converting PPT content in related technologies.

[0006] In a first aspect, an embodiment of the present application provides a method for extracting content from a PPT file based on a large language model, the method comprising: Obtaining a PPT file, and preprocessing the PPT file to obtain preprocessed data; Extracting the content of each page in the PPT file based on the preprocessed data by using a customized large language model to obtain page description information, wherein the customized large language model is obtained by fine-tuning the large language model; Each PPT page is converted into an HTML file, and the HTML file and the page description information are combined into a structured data set for large model training.

[0007] In some embodiments, before obtaining the PPT file, the method further includes: A training set of annotated PPT images and their corresponding description information are obtained, and the network structure of the customized large language model is fine-tuned to obtain the customized large language model, wherein the customized large language model includes a multimodal input layer, a visual language fusion network and a basic model structure.

[0008] In some embodiments, the method further comprises: Through the multimodal input layer, a plurality of different types of feature extraction modules are used to respectively extract information of the preprocessed data of each modality, and the information extracted from the data of each different modality is fused to obtain fused information; The fusion information is received through a visual language fusion network, and the visual features and text features extracted from the fusion information are fused to obtain a fusion feature representation for extracting page description information, wherein the visual language fusion network is set in the Transformer structure of the large language model.

[0009] In some embodiments, the method further comprises: Configure corresponding description templates for different types of PPT pages, and use the description templates to constrain the customized large language model to extract content from each PPT page to obtain the page description information; After obtaining the page description information, verifying and correcting the page description information through a post-processing algorithm, and outputting the page description information if the verification passes; Also, context information associated with the PPT page currently being processed in the batch is collected, and the currently obtained page description information is verified using the context information, and when the verification passes, the page description information is output.

[0010] In some embodiments, obtaining the PPT file includes: using a crawler program to crawl public data from the Internet to obtain the PPT file; And, extracting key frames containing PPT files from the video data set, extracting PPT images and audio information associated with the PPT images from the key frames through a video processing tool library, The audio information is converted into text information, and the PPT image and the text information are combined to obtain the PPT file.

[0011] In some embodiments, preprocessing the PPT file includes: By using image processing technology, the recognizability of the PPT page is optimized; By using an image segmentation algorithm, the optimized PPT page is segmented into different types of page areas, wherein the page elements include: a text box area, a chart area, and a picture area; The text information of the text box area is identified and obtained by using the OCR technology, and the content information of the chart area and the picture area is obtained by using the target detection algorithm.

[0012] In a second aspect, an embodiment of the present application provides a PPT file content extraction system based on a large language model, the system comprising: an acquisition module, a description information extraction module and a data set generation module, wherein: The acquisition module is used to acquire the PPT file and preprocess the PPT file to obtain preprocessed data; The description information extraction module is used to extract the content of each page in the PPT file based on the preprocessed data through a customized large language model to obtain page description information, wherein the customized large language model is obtained by fine-tuning the large language model; The data set generation module is used to convert each PPT page into an HTML file, and to form a structured data set with the HTML file and the page description information for large model training.

[0013] In some embodiments, the system further comprises: a model fine-tuning training module, wherein: The model fine-tuning training module is used to obtain a training set of annotated PPT images and their corresponding description information before obtaining the PPT file, and to fine-tune the network structure of the customized large language model to obtain the customized large language model, wherein the customized large language model includes a multimodal input layer, a visual language fusion network and a basic model structure.

[0014] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0016] Compared with the related art, the PPT file content extraction method based on the large language model provided in the embodiment of the present application obtains the PPT file through a crawler, and the video processing tool extracts the PPT key frames and audio in the video to text; PPT image processing involves enhancement, segmentation, OCR and object detection; the large language model application covers multimodal input layer, visual-language fusion network and model fine-tuning; and converts PPT and video PPT data into HTML files. Compared with the PPT file processing method in the related art, the data required for large model training can be automatically generated in batches. With a large number of high-quality structured data sets, the large model can better learn the characteristics and rules of the PPT content, and can generate detailed text descriptions based on the visual information of the PPT, improve the description of the PPT to be more accurate, and the data set quality is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of a method for extracting content from a PPT file based on a large language model according to an embodiment of the present application; Figure 2 is a flow chart of another method for extracting content from a PPT file based on a large language model according to an embodiment of the present application; Figure 3 It is a structural block diagram of a PPT file content extraction system based on a large language model according to an embodiment of the present application; Figure 4 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0019] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0020] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0021] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] At present, there are many ways to produce and analyze PPT (PowerPoint presentation) content. Traditional PPT production relies on manual editing, template selection, or automatic layout based on preset content, but these methods have certain limitations in content understanding and expression capabilities, especially in complex content or dynamically changing scenarios, the generation effect is relatively rigid. Many existing automated PPT generation methods also rely on some simple template matching and keyword extraction, lacking in-depth understanding of PPT visual information (such as images, charts, etc.) and natural language generation capabilities.

[0023] In addition, the current methods for extracting PPT content are usually relatively simple, mainly through OCR (optical character recognition) technology or image processing technology to extract text and simple visual information. However, PPT contains not only text content, but also multi-dimensional information such as images and animation effects, which are crucial to the completeness and accuracy of content understanding. Existing technologies are usually unable to effectively integrate and reproduce the multi-dimensional information in PPT.

[0024] Therefore, the existing technology still has certain limitations in PPT content generation, extraction and understanding, especially the lack of efficient methods for PPT content generation and conversion, and the lack of large PPT model data sets.

[0025] In view of this, the embodiment of the present application provides a PPT file content extraction method based on a large language model, aiming to solve the deficiencies in PPT content generation and extraction in the prior art, and thus the problem of the lack of a large PPT model data set. The technical solution of the present application can efficiently extract multidimensional information from PPT files or videos by combining the text generation capability of a large language model with image processing technology, and generate accurate and comprehensive PPT descriptions through natural language generation technology, thereby obtaining a high-quality PPT description data set.

[0026] Figure 1 is a flow chart of a method for extracting content from a PPT file based on a large language model according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps: S101, obtaining a PPT file, and preprocessing the PPT file to obtain preprocessed data; This step can be divided into the following sub-steps: Step 1: Use a crawler to crawl public data from the Internet and obtain PPT files; Among them, this type of PPT files is widely stored in various document libraries. For example, some professional academic document libraries may contain a large number of PPTs related to academic research; there are also rich PPTs in the form of teaching courseware on educational platforms; there are also open resource libraries, which bring together PPT resources shared by different fields and creators.

[0027] Step 2, extract key frames containing PPT files from the video data set, extract PPT images and audio information associated with the PPT images from the key frames through the video processing tool library, and convert the audio information into text information; Among them, there are usually clips of PPT presentations in the video. Since the video is composed of a series of continuous frames, in order to obtain the key information related to the PPT, it is necessary to extract the key frames containing the PPT content. This is based on the presentation characteristics of the PPT presentation in the video. Usually, the frames when the PPT page switches or the key content is displayed have key information.

[0028] Specifically, it can be achieved with the help of video processing tools such as FFmpeg and OpenCV. It should also be noted that when extracting key frames, it is also necessary to extract the audio information of the relevant frames. After the audio information is extracted, it is converted into text information. This step is usually implemented using speech recognition technology, and the converted text information will be used as a description of the PPT image. The purpose of this is to present the content in the audio in text form, so as to better combine it with the PPT image information to obtain the PPT file, and provide more comprehensive information for subsequent processing and analysis.

[0029] In addition, before using the large language model to generate PPT text descriptions based on PPT files, the PPT images need to be preprocessed in a series of steps to provide high-quality, easy-to-understand input for the large language model. Specifically, the steps include: 1) Image enhancement and preprocessing, the purpose is to provide high-quality, easy-to-understand input for large language models and improve the recognizability of images. Image processing technologies such as Pillow and OpenCV can be used. Specific operations include: Denoising: Remove possible noise in the PPT image, such as noise generated during image acquisition, to make the image clearer.

[0030] Sharpening: Enhance the edges and details of the image to make the text, graphics and other elements in the image more prominent, making it easier to identify and process later.

[0031] Contrast adjustment: Adjust the contrast of the image to make the bright and dark parts of the image more distinct, further improving the clarity and recognizability of the image.

[0032] 2) Region segmentation: The purpose is to identify the various regions in the PPT, including text boxes, charts, pictures and other elements, so that the content of each region can be processed separately; this can be achieved through image segmentation algorithms.

[0033] It should be noted that the content of different regions has different characteristics and processing methods. For example, the text in the text box needs to be accurately recognized and extracted, the chart needs to analyze its structure and data, and the picture needs to be classified or described. In this embodiment, through region segmentation, appropriate processing methods can be used for different regions to improve the accuracy and efficiency of processing.

[0034] 3) OCR and object detection Text content recognition (OCR) aims to identify the text content in PPT images. You can use OCR technology such as Tesseract or a custom deep learning model. It is understood that this preprocessing step can convert the text information in the image into an editable text format for subsequent processing and analysis, such as combining it with a large language model to generate a more accurate description of the PPT content.

[0035] In addition, for chart and image information extraction (object detection and image classification), the layout and content information of elements such as charts and images are extracted.

[0036] It can be understood that in this step, for charts, their structure and data information can be obtained, such as the number of columns and the data represented by the height of the bar chart; for images, their categories and some key features can be understood. This information is crucial for a comprehensive understanding of the PPT content and can provide richer input for the large language model, enabling it to generate more accurate and detailed PPT text descriptions.

[0037] Through the above step S101, PPT files are obtained through different means to provide the original data basis for subsequent processing, covering a variety of topics and styles, and increasing the representativeness of the data set. The preprocessing step provides high-quality, easy-to-understand input for the large language model, improves the accuracy and efficiency of PPT content processing, and then generates more accurate and detailed text descriptions.

[0038] S102, processing each page content in the PPT file based on the preprocessed data through a customized large language model to obtain page description information, wherein the customized large language model is obtained by fine-tuning the large language model, including: a multimodal input layer and a visual-language fusion network layer; In this embodiment, a customized large language model obtained by improving a large language model (such as chatGPT) is used to process each page of the input PPT file to obtain page description information for representing the page information, specifically: Considering that traditional large language models use text data as input, and the application scenarios of the application scheme involve multimodal information such as images, texts, and audios, in order to make the model adapt to such complex input situations, a multimodal input layer is introduced, where: First, the image-related information of the PPT page includes text content, chart structure, and image content recognized by OCR. For example, OCR technology can convert the text in the PPT image into editable text, chart structure information can be obtained through image processing and analysis algorithms, and image content information is a feature description of the image itself.

[0039] In addition, image and audio-to-text information are extracted from the video keyframes. The PPT presentation keyframes in the video contain important image information, while the audio-to-text information can supplement the understanding of the PPT content. These information are pre-processed by the image processing module and the audio recognition module respectively to ensure the quality and validity of the data.

[0040] Furthermore, information such as the layout and chart types of the PPT are very important for understanding the overall structure and content organization of the PPT. Before being passed into the model, it will also be processed accordingly to better integrate with other modal information.

[0041] In this embodiment, after the information of the above different modes is preprocessed by each dedicated feature extraction module, the information of all modes will be fused together and passed to the core part of the model (i.e., the structure for extracting page description information). It can be understood that such a fusion method ensures that the model can take into account multiple information sources at the same time, so as to understand the PPT content more comprehensively.

[0042] Furthermore, the customized large language model in this embodiment also includes a visual language fusion network, which fuses visual features with text information to form a unified representation by adding an image feature extraction network to the traditional Transformer structure.

[0043] For image input, convolutional neural network (CNN) or visual transformer (ViT) can be used to extract image features. Among them, CNN has good local feature extraction capabilities and can capture detailed information in the image, such as texture, edges and other features in the image. ViT is better at processing global image information and can grasp the characteristics of the image as a whole. By selecting a suitable image feature extraction method, more effective image features can be obtained according to the characteristics of the PPT image.

[0044] In terms of text input, the use of large language model infrastructure (such as the GPT series) can fully utilize the advantages of large language models in text processing to accurately analyze and process input text information.

[0045] It can be understood that by fusing image features and text features into the same vector space, the model can generate text descriptions that match the image content. This fusion method breaks the limitation of the separation of visual and language information in traditional models, allowing the model to better understand the relationship between PPT images and text, thereby generating text descriptions that are more in line with the actual situation.

[0046] In addition, it should be noted that the multimodal data fusion in the multimodal input layer ensures that various modal information can enter the core part of the model in a preliminarily processed form, providing a comprehensive data basis for further fusion operations of the vision-language fusion network.

[0047] Furthermore, the visual language fusion network more accurately fuses image and text features in the internal structure of the model, which further improves and optimizes the fusion effect of the first step. It focuses on fusing visual features (mainly image features) and text features at a deeper and more targeted level. The ultimate goal is to enable the model to generate accurate text descriptions that are more in line with the actual content of the PPT.

[0048] Under the above model architecture, the large language model is fine-tuned based on a large number of annotated PPT images and their descriptions. These annotated data contain various PPT samples, covering different styles, layouts and contents, such as technical reports, academic speeches, business reports, etc. By using rich and diverse annotated data, the model can better learn the characteristics and rules of different types of PPTs.

[0049] In the specific fine-tuning process, the model not only learns how to generate accurate text descriptions, but also learns how to combine the information extracted from the image to generate output in a specific format. For example, for a PPT page containing a chart, the model needs to learn to generate an analysis description that meets the requirements based on the chart structure and related text information, and present it in a specific format. This learning method can make the model more adaptable to the application scenarios of the present invention and improve the accuracy and practicality of the content generated by the model.

[0050] In addition, in order to ensure that the generated description meets expectations and avoid irrelevant or invalid information, the following means can be used to constrain and correct the output of the large language model during the model's specific generation of page content descriptions: a. Template constraints guide the large language model to generate structured text, so that the generated description meets expectations and avoids irrelevant or invalid information; An example of a template constraint is as follows: For chart-type PPT images, the template is "Chart title, Chart content:". When the model processes this type of image, it will first determine the chart title and then describe the chart content according to the requirements of the template, so that the generated text has a clear structure and focus; For PPT images of pictures, the template can be "Picture description: ______". The model will describe the picture under the guidance of this template, making the description more targeted and standardized.

[0051] b. Post-processing and correction: After the large language model generates a preliminary text description, rules and post-processing algorithms are applied to check and correct the description to ensure that it conforms to the actual content.

[0052] For example, a text correction model can be set up to detect whether a description that does not match the image content is generated and make corrections.

[0053] After the large language model generates a preliminary text description, it is first checked using the set rules. For example, it checks whether there are grammatical errors and whether it conforms to basic logical relationships. Then, it uses the text correction model for further testing. The model can compare the generated description with the actual content of the image to determine whether there is any discrepancy. If there is a discrepancy, such as describing an element that is not in the image, it will be corrected so that the final description matches the image content.

[0054] c. Contextual verification: Use the contextual information of the PPT to verify and ensure that the description generated by the large language model is consistent with the content of the previous and next pages. For example, if the previous page is a chart and the next page is a chart analysis, when the model generates the description of the next page, it will generate accurate chart analysis text based on the content and structure of the previous page chart and the logical relationship between the previous and next pages. This ensures that the description of the entire PPT is coherent in content and conforms to the actual presentation logic.

[0055] Through the above constraints, template constraints provide description templates for PPT images, guide the model to generate description information, and avoid invalid information; post-processing and correction use rules and model checks to correct the initial description to ensure that it conforms to the actual content; context verification uses context information to ensure that the description is consistent with the content of the previous and next pages, ensuring coherence and logic. Overall, the accuracy of the customized large language model in generating page description content has been improved.

[0056] S103, converting each PPT page into an HTML file, and combining the HTML file and page description information into a structured data set for large model training.

[0057] In this embodiment, each page of the PPT is converted into a structured HTML format for further display and editing. It can be understood that the display effect of the HTML data needs to be exactly the same as that of the PPT.

[0058] Specifically, the PPT format data is parsed and the layout information of the text, images and other elements on each page is converted into HTML data. This conversion allows the PPT content to maintain its original structure and display effect in the new format, facilitating subsequent data operations and utilization.

[0059] In addition, for the PPT data in the video, its structural information is obtained and converted into HTML data. Computer vision technology can be used to structure the image. First, the key frames containing PPT content are extracted from the video, and then these key frame images are processed to obtain the PPT structural information, and finally converted into HTML data.

[0060] Figure 2 is a flowchart of another method for extracting content from a PPT file based on a large language model according to an embodiment of the present application, such as Figure 2As shown in the figure, PPT files and video files are collected respectively. After a series of data processing steps, the data pairs of PPT text description and HTML files are finally obtained. Users can use these results to view, edit or reorganize the PPT content. For example, users can quickly understand the main content of the PPT based on the text description, perform visual editing operations through HTML files, and change the layout and element order of the PPT. The data pairs composed of the page description information of the PPT and the structure file finally constitute the data set, which enhances the editability and reusability of the content.

[0061] Through the above steps S101 to S103, the PPT file is obtained through the crawler, the video processing tool extracts the PPT key frames and audio in the video to text; PPT image processing involves enhancement, segmentation, OCR and object detection; the large language model application covers multimodal input layer, visual-language fusion network and model fine-tuning; and PPT and video PPT data are converted into HTML files. Compared with the PPT file processing method in the related art, the data required for large model training can be automatically generated in batches. With a large number of high-quality structured data sets, the large model can better learn the characteristics and rules of PPT content, thereby improving the ability to generate and extract PPT content.

[0062] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0063] This embodiment also provides a PPT file content extraction system based on a large language model. Figure 3 is a structural block diagram of a PPT file content extraction system based on a large language model according to an embodiment of the present application, such as Figure 3 As shown, the system includes: an acquisition module 10, a description information extraction module 20 and a data set generation module 30, wherein: The acquisition module 10 is used to acquire the PPT file and preprocess the PPT file to obtain preprocessed data; This module uses different means to obtain PPT files to provide the raw data basis for subsequent processing, covering a variety of topics and styles, and increasing the representativeness of the data set. The preprocessing step provides high-quality, easy-to-understand input for the large language model, improves the accuracy and efficiency of PPT content processing, and generates more accurate and detailed text descriptions.

[0064] The description information extraction module 20 is used to extract the content of each page in the PPT file based on the preprocessed data through a customized large language model to obtain page description information, wherein the customized large language model is obtained by fine-tuning the large language model; By improving the large language model and further setting constraints and verification mechanisms, the model is guided to generate description information to avoid invalid information; post-processing and correction use rules and model checks to correct the initial description to ensure that it conforms to the actual content; context verification uses context information to ensure that the description is consistent with the previous and next page content, ensuring coherence and logic. Overall, the accuracy of the customized large language model in the process of generating page description content has been improved.

[0065] The data set generation module 30 is used to convert each PPT page into an HTML file, and to form a structured data set with the HTML file and page description information for large model training.

[0066] Through the above system, after obtaining the PPT file, the preprocessing steps involve enhancement, segmentation, OCR and object detection; the application of the large language model covers multimodal input layer, visual-language fusion network and model fine-tuning; and the conversion of PPT and video PPT data into HTML files. Compared with the PPT file processing method in the related art, the data required for large model training can be automatically generated in batches. With a large number of high-quality structured data sets, the large model can better learn the characteristics and rules of the PPT content, and can generate detailed text descriptions based on the visual information of the PPT, improve the description of the PPT to be more accurate, and the quality of the data set is higher. In addition, since PPT files can be collected from different types of data sources, it is not only suitable for the generation and extraction of static PPT files, but also can process PPT content in videos, and has strong versatility and applicability.

[0067] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 4 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 4 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a content extraction method for a PPT file based on a large language model, and the database is used to store data.

[0068] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0069] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0070] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for extracting PPT file content based on a large language model, characterized in that: The method comprises: Obtaining a PPT file, and preprocessing the PPT file to obtain preprocessed data; Extracting the content of each page in the PPT file based on the preprocessed data by using a customized large language model to obtain page description information, wherein the customized large language model is obtained by fine-tuning the large language model; Each PPT page is converted into an HTML file, and the HTML file and the page description information are combined into a structured data set for large model training.

2. The method according to claim 1, characterized in that Before obtaining the PPT file, the method further includes: A training set of annotated PPT images and their corresponding description information are obtained, and the network structure of the customized large language model is fine-tuned to obtain the customized large language model, wherein the customized large language model includes a multimodal input layer, a visual language fusion network and a basic model structure.

3. The method according to claim 2, characterized in that The method further comprises: Through the multimodal input layer, a plurality of different types of feature extraction modules are used to respectively extract information of the preprocessed data of each modality, and the information extracted from the data of each different modality is fused to obtain fused information; The fusion information is received through a visual language fusion network, and the visual features and text features extracted from the fusion information are fused to obtain a fusion feature representation for extracting page description information, wherein the visual language fusion network is set in the Transformer structure of the large language model.

4. The method according to claim 3, characterized in that The method further comprises: Configure corresponding description templates for different types of PPT pages, and use the description templates to constrain the customized large language model to extract content from each PPT page to obtain the page description information; After obtaining the page description information, verifying and correcting the page description information through a post-processing algorithm, and outputting the page description information if the verification passes; Also, context information associated with the PPT page currently being processed in the batch is collected, and the currently obtained page description information is verified using the context information, and when the verification passes, the page description information is output.

5. The method according to claim 1, characterized in that Obtaining the PPT file includes: using a crawler program to crawl public data from the Internet to obtain the PPT file; And, extracting key frames containing PPT files from the video data set, extracting PPT images and audio information associated with the PPT images from the key frames through a video processing tool library, The audio information is converted into text information, and the PPT image and the text information are combined to obtain the PPT file.

6. The method according to claim 5, characterized in that Preprocessing the PPT file includes: By using image processing technology, the recognizability of the PPT page is optimized; By using an image segmentation algorithm, the optimized PPT page is segmented into different types of page areas, wherein the page elements include: a text box area, a chart area, and a picture area; The text information of the text box area is identified and obtained by using the OCR technology, and the content information of the chart area and the picture area is obtained by using the target detection algorithm.

7. A PPT file content extraction system based on a large language model, characterized in that: The system comprises: an acquisition module, a description information extraction module and a data set generation module, wherein: The acquisition module is used to acquire the PPT file and preprocess the PPT file to obtain preprocessed data; The description information extraction module is used to extract the content of each page in the PPT file based on the preprocessed data through a customized large language model to obtain page description information, wherein the customized large language model is obtained by fine-tuning the large language model; The data set generation module is used to convert each PPT page into an HTML file, and to form a structured data set with the HTML file and the page description information for large model training.

8. The system according to claim 7, characterized in that The system also includes: a model fine-tuning training module, wherein: The model fine-tuning training module is used to obtain a training set of annotated PPT images and their corresponding description information before obtaining the PPT file, and to fine-tune the network structure of the customized large language model to obtain the customized large language model, wherein the customized large language model includes a multimodal input layer, a visual language fusion network and a basic model structure.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images