Method and system for displaying document through voice conversation based on large model

Abstract

The invention discloses a method for displaying documents based on voice dialogue of a large model, which comprises the following steps of: S100, extracting document data of an uploaded file by a cloud large model, and storing the document data in a knowledge base through identification and judgment; marking the positions of the small pictures in the characters in a manner of placing special characters and ID information, and storing the marked small pictures in partition blocks, corresponding to the ID information, of a knowledge base, so as to establish the knowledge base; s200, the application end transmits the voice audio information to a cloud big model, and the cloud big model accesses a knowledge base and transmits document data associated with the voice dialogue to the application end; s300, the application end reads document data, and when a special character is recognized, ID information in the special character is transmitted to the cloud large model; and step S400, the cloud big model extracts small pictures corresponding to the ID information in the knowledge base, and returns the small pictures to the application end, and the application end synchronously displays the document data. The document content can be completely displayed through voice dialogue.

Description

Technical Field

[0001] The present invention relates to the technical field of voice-controlled document display in connected vehicles, and in particular to a method and system for displaying documents through voice dialogue based on a large model. Background Art

[0002] In the era of rapid development of big models, more and more products are applying big model capabilities, such as: in-vehicle big models, which are mainly used in voice and intelligent driving. In terms of voice, they can cover all small talk, enhance voice capabilities, and make voice recognition more natural. In the field of in-vehicle, big models also understand natural speech better than traditional speech. Among the big model capabilities, the knowledge base is an especially important reserve. The knowledge base requires documents to be uploaded and scanned and stored. However, in the process of outputting documents, small pictures between texts cannot be displayed, resulting in the following problems in the in-vehicle system:

[0003] First: small pictures / icons cannot be exported;

[0004] Second: the lack of pictures / icons in the process of users answering knowledge questions increases the difficulty of understanding;

[0005] Third: User experience declines. Summary of the invention

[0006] The main purpose of the present invention is to provide a method and system for displaying documents in voice dialogue based on a large model to address the shortcomings of the prior art that documents cannot be fully displayed during system voice dialogue, especially small pictures between texts.

[0007] A method for presenting documents based on a voice dialogue of a large model comprises the following steps:

[0008] Step S100: The cloud-based large model extracts the document data of the uploaded file, and stores the text data information, large image data information, and table data information in the document data in the corresponding partition blocks of the knowledge base through recognition and judgment; the position of the small image in the text is marked by placing special characters plus ID information, and then the small image data information in the document data is stored in the partition block corresponding to the ID information of the knowledge base according to the marked ID information, thereby establishing a knowledge base;

[0009] Step S200: the application end collects the voice audio information of the voice conversation and transmits the voice audio information to the cloud big model. The cloud big model accesses the knowledge base and transmits the document data associated with the voice conversation to the application end;

[0010] Step S300: the application reads the document data, and when a special character is recognized, transmits the ID information in the special character to the cloud-based large model;

[0011] Step S400: The cloud-based large model extracts the small image corresponding to the ID information in the knowledge base and transmits it back to the application end, which synchronously displays the document data.

[0012] Preferably, the step S100 includes: the cloud-based large model scans and identifies the text, pictures, and tables in the uploaded file through the Pdfplumber library and OCR technology, and extracts them by category; then the extracted text is identified as a title or text according to the text size and position coordinates, and the extracted picture is identified as a large picture or a small picture by comparing the picture size with a set value. When a small picture is identified, the position of the small picture in the text is marked by placing special characters plus an ID; finally, the extracted text data information is stored in the text partition in the order of title first and text later, the extracted large picture data information is stored in the large picture partition in the scanning order, the extracted table data information is stored in the table partition in the scanning order, and the extracted small picture data information is stored in the IP address partition corresponding to the ID according to the marked ID information; thereby establishing a knowledge base in the cloud.

[0013] Preferably, step S200 includes: the application end collects the voice audio information of the current conversation sound through the device microphone, and then transmits it to the cloud voice platform through the wireless network, the cloud voice platform performs semantic conversion processing on the voice audio information, and transmits the semantic information to the cloud big model, the cloud big model parses the semantic information to obtain multiple related keywords, the cloud big model accesses the knowledge base of the field corresponding to the related keywords according to the multiple related keywords, retrieves the document data in the knowledge base, obtains the related document data, and transmits the text data information, large image data information, and table data information in the document data to the application end.

[0014] Preferably, step S300 includes: the application end reads the text data information, large image data information, and table data information in the order of reception. When a special character is recognized during the reading process, the reading is paused, the ID information in the special character is obtained, and the ID information is transmitted to the cloud large model.

[0015] Preferably, the step S400 includes: the cloud-based large model searches the knowledge base according to the ID information, extracts the small image stored at the IP address location corresponding to the ID, and transmits it back to the application end; the application end temporarily stores the small image in the display space of the corresponding category according to the data information category, and the display screen synchronously displays the data information according to the storage order.

[0016] The present invention also provides a system for presenting documents based on voice dialogue of a large model, comprising: a cloud and an application end, wherein the cloud includes a first extraction module, a first recognition module, a judgment module, a first transmission module, a marking module, and an access module, wherein the first extraction module is used to classify and extract document data of an uploaded file, the first recognition module is used to recognize and classify text data information, large image data information, small image data information, and table data information in the extracted document data, the judgment module is used to determine whether the data category in the document data is consistent with the partition block category in the knowledge base, the marking module is used to mark the position of the small image in the text by placing special characters plus ID, the first transmission module is used to store the text data information, large image data information, and table data information in the document data in the corresponding partition block of the knowledge base, the first transmission module is used to transmit the document data associated with the voice dialogue to the application end, the first transmission module is used to transmit the small image corresponding to the ID in the knowledge base to the application end, the first transmission module is used to store the small image data information in the document data in the partition block corresponding to the ID information in the knowledge base according to the marked ID information, and the access module is used to access the knowledge base;

[0017] The application end includes a collection module, a second transmission module, an access module, a reading module, a second recognition module, a second extraction module, and a display module; the collection module is used to collect voice and audio information of voice conversations, the second transmission module is used to transmit the voice and audio information to a large cloud model, the second transmission module is used to transmit the ID in the special characters to the large cloud model, the access module is used to access a knowledge base, the reading module is used to read document data, the second recognition module is used to recognize special characters in text data information, the second extraction module is used to extract small pictures corresponding to the ID in the knowledge base, and the display module is used to synchronously display document data.

[0018] Preferably, the first recognition module is also used to scan and recognize text, pictures, and tables in the uploaded file through the Pdfplumber library in collaboration with OCR technology. The first recognition module is also used to recognize the title or text of the extracted text according to the text size and position coordinates. The first recognition module is also used to recognize the large picture or small picture of the extracted picture by comparing the picture size with a set value. The first transmission module is also used to store the extracted text data information in the order of title first and text later in the text partition, store the extracted large picture data information in the large picture partition in the scanning order, store the extracted table data information in the table partition in the scanning order, and store the extracted small picture data information in the IP address partition corresponding to the ID according to the marked ID information.

[0019] Preferably, the cloud also includes a conversion module, a parsing module, and a retrieval module. The conversion module is used to perform semantic conversion processing on voice audio information, the parsing module is used to parse semantic information, and the retrieval module is used to retrieve document data in the knowledge base. The first transmission module is also used to transmit semantic information to a large cloud model, the first extraction module is also used to obtain associated document data, and the access module is also used to access a knowledge base in a field corresponding to an associated keyword based on multiple associated keywords.

[0020] Preferably, the reading module is also used to read text data information, large image data information, and table data information in the order in which they are received.

[0021] Preferably, the retrieval module is also used to retrieve the knowledge base according to the ID information, the first extraction module is also used to extract the small picture stored at the IP address location corresponding to the ID, the second transmission module is also used to temporarily store the data information in the display space of the corresponding category according to the category of the data information, and the display module is also used to synchronously display the data information according to the storage order.

[0022] The method for displaying documents through voice dialogue based on a large model provided by the present invention has the following beneficial effects: the cloud-based large model extracts the document data of the uploaded file, and stores the text data information, large image data information, and table data information in the document data in the corresponding partition blocks of the knowledge base through recognition and judgment, and marks the position of the small image in the text by placing special characters plus ID information, and then stores the small image data information in the document data in the partition blocks corresponding to the ID information of the knowledge base according to the marked ID information, thereby establishing a knowledge base; by identifying the small image in the processing of the document, marking and storing the position of the small image in the document, a data source and a display position data source are provided for subsequent reading and retrieval of the small image for display on the screen; the application end collects the voice dialogue information, and the small image is displayed in the document. The cloud-based big model accesses the knowledge base and transmits the document data associated with the voice conversation to the application end; the voice audio information is parsed into recognizable semantic text information, which provides the original data basis for the big model to access the corresponding database by domain and extract document data; the application end reads the document data, and when special characters are recognized, the ID information in the special characters is transmitted to the cloud-based big model; during the upper screen reading process, the read ID is provided to the big model, and the address data is provided for the big model to find the corresponding small picture; the cloud-based big model extracts the small picture corresponding to the ID information in the knowledge base and transmits it back to the application end, which synchronously displays the document data; after receiving the small picture, the display of the small picture is realized, and the entire document data is fully displayed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative work:

[0024] Figure 1 The figure is a flow chart of a method for presenting documents based on voice dialogue in a large model provided by an embodiment of the present invention.

[0025] Figure 2 The figure shows a schematic diagram of the system module structure of a system for presenting documents through voice dialogue based on a large model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Typical embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0028] The general idea of ​​the present invention is: in view of the shortcomings of the prior art that documents cannot be fully displayed during the system voice dialogue process, especially the small pictures between texts, the present invention provides a method for displaying documents through voice dialogue based on a large model, extracting the document data of the uploaded file through the cloud large model, storing the text data information, large picture data information, and table data information in the document data in the corresponding partition blocks of the knowledge base through recognition and judgment, marking the position of the small picture in the text by placing special characters plus ID information, and then storing the small picture data information in the document data in the partition block corresponding to the ID information of the knowledge base according to the marked ID information, thereby establishing a knowledge base; by identifying the small picture in the processing of the document, marking and storing the position of the small picture in the document, data is provided for subsequent reading and retrieval of the small picture for display on the screen. Source and display location data source; the application side collects the voice and audio information of the voice conversation, and transmits the voice and audio information to the cloud-based big model. The cloud-based big model accesses the knowledge base and transmits the document data associated with the voice conversation to the application side; the voice and audio information is parsed into recognizable semantic text information, which provides the original data basis for the big model to access the corresponding database by domain and extract document data; the application side reads the document data, and when special characters are recognized, the ID information in the special characters is transmitted to the cloud-based big model; during the upper screen reading process, the read ID is provided to the big model, and address data is provided for the big model to find the corresponding small picture; the cloud-based big model extracts the small picture corresponding to the ID information in the knowledge base, and transmits it back to the application side, which synchronously displays the document data; after receiving the small picture, the display of the small picture is realized, and the entire document data is fully displayed.

[0029] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other. Figure 1 , Figure 1 The figure is a flow chart of a method for displaying a document based on a voice dialogue of a large model provided by an embodiment of the present invention. In this embodiment, a method for displaying a document based on a voice dialogue of a large model includes:

[0030] Step S100: The cloud-based large model extracts the document data of the uploaded file, and stores the text data information, large image data information, and table data information in the document data in the corresponding partition blocks of the knowledge base through recognition and judgment; the position of the small image in the text is marked by placing special characters plus ID information, and then the small image data information in the document data is stored in the partition block of the knowledge base corresponding to the ID information according to the marked ID information, thereby establishing a knowledge base.

[0031] By identifying small images during document processing and marking and storing the positions of the small images in the document, a data source is provided for subsequent document reading and small image retrieval for display on the screen, as well as the source of the display position data.

[0032] The cloud-based big model has the function of processing and extracting document files, and also has the function of communicating with the application end, voice platform, and knowledge base. To display documents on the application end, it is necessary to retrieve and transmit relevant document data to the application end, and the application end reads it in a streaming manner and displays it on the screen. As a storage object for document-related data, the knowledge base needs to be established and improved in advance to support the big model and the application end through data transmission, so that the application end can communicate through voice to realize the demand for displaying documents.

[0033] How to establish and improve the knowledge base so as to provide comprehensive data for subsequent application terminals and realize comprehensive document display, specifically, the step S100 includes: the cloud-based large model scans and identifies the text, pictures, and tables in the uploaded file through the Pdfplumber library and the OCR technology, and extracts them by category; then the extracted text is identified as the title or the text according to the text size and position coordinates, and the extracted picture is identified as a large picture or a small picture by comparing the size of the picture with the set value. When the small picture is identified, the position of the small picture in the text is marked by placing special characters plus ID; finally, the extracted text data information is stored in the text partition in the order of title first and text later, the extracted large picture data information is stored in the large picture partition in the scanning order, the extracted table data information is stored in the table partition in the scanning order, and the extracted small picture data information is stored in the IP address partition corresponding to the ID according to the marked ID information; thereby establishing a knowledge base in the cloud.

[0034] Take the example of uploading PDF files to the knowledge base. Before the knowledge base is stored, the PDF files are scanned and recognized by the cloud-based big model using the Pdfplumber library in conjunction with OCR technology. As a PDF parsing tool, the Pdfplumber library uses OCR technology to clean and preprocess PDF files, so that the text, pictures, and table data in the PDF files can be extracted by category. The extracted text, pictures, and table data are temporarily stored in the big model, which needs to further process these data. In a PDF file, text is divided into title and text. Usually, title text is bolder and larger, while text is lighter and smaller. Images are divided into separate images and some small images interspersed between text. Separate images are usually larger in size and capacity, while small images between text are usually significantly smaller. The text layout of the document usually puts the title first and the text second. By combining the coordinates of the text in the document, the text size, the thickness of the text, etc., the text can be distinguished as the title or the text. After distinguishing the title and the text, they are transferred to the text block corresponding to the knowledge base in the order of title first and text later. For the classification of images, the image size and image capacity are compared with the set value. Images larger than the set value are separate images, and images smaller than the set value are small images between text. For separate images, they are stored in the large image block corresponding to the knowledge base in the order of scanning. For small pictures between words, after they are identified, special symbols are placed at the position of the words in the document as a unique mark for the small pictures. Different small pictures are represented by unique IDs, and the IDs are placed in the middle of special symbols, such as: text 1 + ￥￥ + ID + ￥￥ + text 2. After the marking is completed, the small pictures are stored in the small picture block corresponding to the knowledge base, and the unique IP stored in each small picture is bound to the unique ID, providing an association basis for the subsequent large model to read the ID and find the small picture. At the same time, special symbols are marked in the text and stored in the text. In this way, after each file is processed, it is stored in the knowledge base to establish a knowledge base.

[0035] Step S200: The application end collects the voice audio information of the voice conversation and transmits the voice audio information to the cloud-based big model. The cloud-based big model accesses the knowledge base and transmits the document data associated with the voice conversation to the application end.

[0036] Parse speech audio information into recognizable semantic text information, providing the original data basis for large models to access corresponding databases by domain and extract document data.

[0037] In an embodiment of the present invention, the vehicle system is voice controlled and has recording function, transmission function, processing function, storage function, etc. It can record, it can perform various data interactions with cloud servers and smart devices, and it also has computing and processing capabilities. It can also store various data information obtained from the outside or the cloud, such as data tables, etc.

[0038] As for how to realize document display through voice conversation, specifically, the step S200 includes: the application end collects the voice audio information of the current conversation sound through the device microphone, and then transmits it to the cloud voice platform through the wireless network. The cloud voice platform performs semantic conversion processing on the voice audio information and transmits the semantic information to the cloud big model. The cloud big model parses the semantic information to obtain multiple related keywords. The cloud big model accesses the knowledge base of the field corresponding to the related keywords according to the multiple related keywords, searches the document data in the knowledge base, obtains the related document data, and transmits the text data information, large image data information, and table data information in the document data to the application end.

[0039] The voice platform can be a voice provider server in the cloud, or a voice processing system installed locally in the vehicle system, etc. However, the local voice processing system is relatively simple in processing voice, and is not as comprehensive and accurate as the cloud. This embodiment uses a cloud voice platform, which can parse voice audio information and output semantic information. After the voice platform completes the analysis, it is transmitted to the cloud model via wireless communication. The cloud model parses the voice semantic information to obtain multiple related keywords. For example, the semantic information "vehicle system function description" can be parsed into the vehicle system manual, vehicle interface manual, vehicle function manual, etc. The cloud inputs the related keywords into the knowledge base of the corresponding field, and retrieves the document data with the same or similar name and the same or similar data content as the keyword from the data stored in the knowledge base. In the block storage space, the text, large picture, and table data information of the partition block are transmitted to the application end.

[0040] Step S300: The application reads the document data, and when a special character is recognized, the ID information in the special character is transmitted to the cloud-based large model.

[0041] During the reading process of the upper screen, the read ID is provided to the large model, providing address data for the large model to find the corresponding small picture.

[0042] The application receives the document data transmitted by the knowledge base, reads the document data in the order of reception, and displays it on the screen. When special characters are recognized in the text data, it communicates with the large model and transmits the ID data in the special characters read to the large model, which then proceeds to the next step of processing.

[0043] As for how the application side obtains the small picture, specifically, the step S300 includes: the application side reads the text data information, the large picture data information, and the table data information in the order of reception. When a special character is recognized during the reading process, the reading is paused, the ID information in the special character is obtained, and the ID information is transmitted to the cloud large model.

[0044] The application receives the text data and reads it. When a special symbol is identified during reading, the text reading is paused, and the application extracts the ID data of the special symbol and transmits it to the big model. The big model performs the next step of processing on the ID data.

[0045] Step S400: The cloud-based large model extracts the small image corresponding to the ID information in the knowledge base and transmits it back to the application end, which synchronously displays the document data.

[0046] The cloud-based large model retrieves the small image from the knowledge base through the ID data and transmits it to the application end. The application end receives the small image and displays it, fully displaying the entire document data.

[0047] As for how the cloud-based large model processes the ID data transmitted by the application end after receiving it to realize the transmission of small pictures, specifically, the step S400 includes: the cloud-based large model searches the knowledge base according to the ID information, extracts the small picture stored at the IP address location corresponding to the ID, and transmits it back to the application end. The application end temporarily stores the small picture in the display space of the corresponding category according to the data information category, and the display screen synchronously displays the data information according to the storage order.

[0048] When the large model receives the ID data information transmitted by the application end, it finds the IP address corresponding to the ID data based on the corresponding relationship, enters the knowledge base to find the small picture stored in the IP address, extracts the small picture, and transmits it to the application end. The application end receives the small picture and immediately displays it on the screen, thereby realizing the display of the small picture after the text. After the text is displayed, the application end further reads other received text data, image data, and table data, and displays them on the screen in the order of reception, thereby realizing the display of the entire document.

[0049] Based on the above method, the present invention provides a method for displaying documents based on voice dialogue of a large model. The document data of the uploaded file is extracted through the cloud large model. The text data information, large image data information, and table data information in the document data are stored in the corresponding partition blocks of the knowledge base through recognition and judgment. The position of the small image in the text is marked by placing special characters plus ID information, and then the small image data information in the document data is stored in the partition block corresponding to the ID information of the knowledge base according to the marked ID information, thereby establishing a knowledge base; by identifying the small image in the processing of the document, the position of the small image in the document is marked and stored, so as to provide a data source and a display position data source for subsequent on-screen reading and retrieval of the small image for display; the application end collects the voice dialogue information and the small image in the document. The cloud-based big model accesses the knowledge base and transmits the document data associated with the voice conversation to the application end; the voice audio information is parsed into recognizable semantic text information, which provides the original data basis for the big model to access the corresponding database by domain and extract document data; the application end reads the document data, and when special characters are recognized, the ID information in the special characters is transmitted to the cloud-based big model; during the upper screen reading process, the read ID is provided to the big model, and the address data is provided for the big model to find the corresponding small picture; the cloud-based big model extracts the small picture corresponding to the ID information in the knowledge base and transmits it back to the application end, which synchronously displays the document data; after receiving the small picture, the display of the small picture is realized, and the entire document data is fully displayed.

[0050] Accordingly, a system for displaying documents based on voice dialogues of large models, referring to Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a system module structure of a system for displaying speech dialogue documents based on a large model provided by an embodiment of the present invention. The system displays speech dialogue documents using the method described above, and includes:

[0051] The cloud side and the application side, the cloud side includes a first extraction module, a first recognition module, a judgment module, a first transmission module, a marking module, and an access module. The first extraction module is used to classify and extract document data of uploaded files. The first recognition module is used to identify and classify text data information, large image data information, small image data information, and table data information in the extracted document data. The judgment module is used to determine whether the data category in the document data is consistent with the partition block category in the knowledge base. The marking module is used to mark the position of the small image in the text by placing special characters plus IDs. The first transmission module is used to store the text data information, large image data information, and table data information in the document data in the corresponding partition blocks of the knowledge base. The first transmission module is used to transmit the document data associated with the voice dialogue to the application side. The first transmission module is used to The small picture corresponding to the ID is transmitted to the application end, the first transmission module is used to store the small picture data information in the document data in the partition block corresponding to the ID information of the knowledge base according to the marked ID information, and the access module is used to access the knowledge base; the application end includes a collection module, a second transmission module, an access module, a reading module, a second recognition module, a second extraction module, and a display module; the collection module is used to collect voice and audio information of voice conversations, the second transmission module is used to transmit the voice and audio information to the cloud-based large model, the second transmission module is used to transmit the ID in the special characters to the cloud-based large model, the access module is used to access the knowledge base, the reading module is used to read the document data, the second recognition module is used to recognize special characters in the text data information, the second extraction module is used to extract the small picture corresponding to the ID in the knowledge base, and the display module is used to synchronously display the document data.

[0052] Preferably, the first recognition module is also used to scan and recognize text, pictures, and tables in the uploaded file through the Pdfplumber library in collaboration with OCR technology. The first recognition module is also used to recognize the title or text of the extracted text according to the text size and position coordinates. The first recognition module is also used to recognize the large picture or small picture of the extracted picture by comparing the picture size with a set value. The first transmission module is also used to store the extracted text data information in the order of title first and text later in the text partition, store the extracted large picture data information in the large picture partition in the scanning order, store the extracted table data information in the table partition in the scanning order, and store the extracted small picture data information in the IP address partition corresponding to the ID according to the marked ID information.

[0053] Preferably, the cloud also includes a conversion module, a parsing module, and a retrieval module. The conversion module is used to perform semantic conversion processing on voice audio information, the parsing module is used to parse semantic information, and the retrieval module is used to retrieve document data in the knowledge base. The first transmission module is also used to transmit semantic information to a large cloud model, the first extraction module is also used to obtain associated document data, and the access module is also used to access a knowledge base in a field corresponding to an associated keyword based on multiple associated keywords.

[0054] Preferably, the reading module is also used to read text data information, large image data information, and table data information in the order in which they are received.

[0055] Preferably, the retrieval module is also used to retrieve the knowledge base according to the ID information, the first extraction module is also used to extract the small picture stored at the IP address location corresponding to the ID, the second transmission module is also used to temporarily store the data information in the display space of the corresponding category according to the category of the data information, and the display module is also used to synchronously display the data information according to the storage order.

[0056] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0057] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0058] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0059] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0060] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0061] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

Claims

1. A method for presenting documents based on voice dialogue of a large model, characterized in that: The following steps are involved: Step S100: The cloud-based large model extracts document data of the uploaded file, and stores text data information, large image data information, and table data information in the document data in corresponding partition blocks of the knowledge base through recognition and judgment; The position of the small image in the text is marked by placing special characters plus ID information, and then the small image data information in the document data is stored in a partition block corresponding to the ID information in the knowledge base according to the marked ID information, thereby establishing a knowledge base; Step S200: the application end collects the voice audio information of the voice conversation and transmits the voice audio information to the cloud big model. The cloud big model accesses the knowledge base and transmits the document data associated with the voice conversation to the application end; Step S300: the application reads the document data, and when a special character is recognized, transmits the ID information in the special character to the cloud-based large model; Step S400: The cloud-based large model extracts the small image corresponding to the ID information in the knowledge base and transmits it back to the application end, which synchronously displays the document data.

2. A method for presenting a document based on a voice dialogue of a large model as claimed in claim 1, characterized in that: The step S100 includes: the cloud-based large model scans and identifies the text, pictures, and tables in the uploaded file through the Pdfplumber library and the OCR technology, and extracts them by category; then the extracted text is identified as a title or text according to the text size and position coordinates, and the extracted picture is identified as a large picture or a small picture by comparing the picture size with the set value. When the small picture is identified, the position of the small picture in the text is marked by placing special characters plus ID; finally, the extracted text data information is stored in the text partition in the order of title first and text later, the extracted large picture data information is stored in the large picture partition in the scanning order, the extracted table data information is stored in the table partition in the scanning order, and the extracted small picture data information is stored in the IP address partition corresponding to the ID according to the marked ID information; thereby establishing a knowledge base in the cloud.

3. A method for presenting documents based on voice dialogues in a large model as claimed in claim 2, characterized in that: The step S200 includes: the application end collects the voice audio information of the current conversation sound through the device microphone, and then transmits it to the cloud voice platform through the wireless network. The cloud voice platform performs semantic conversion processing on the voice audio information and transmits the semantic information to the cloud big model. The cloud big model parses the semantic information to obtain multiple related keywords. The cloud big model accesses the knowledge base of the field corresponding to the related keywords according to the multiple related keywords, retrieves the document data in the knowledge base, obtains the related document data, and transmits the text data information, large image data information, and table data information in the document data to the application end.

4. A method for presenting documents based on voice dialogues in a large model as claimed in claim 3, characterized in that: The step S300 includes: the application end reads the text data information, the large image data information, and the table data information in the order of reception. When a special character is recognized during the reading process, the reading is paused, the ID information in the special character is obtained, and the ID information is transmitted to the cloud large model.

5. A method for presenting documents based on voice dialogues in a large model as claimed in claim 4, characterized in that: The step S400 includes: the cloud-based large model searches the knowledge base according to the ID information, extracts the small image stored at the IP address location corresponding to the ID, and transmits it back to the application end. The application end temporarily stores the small image in the display space of the corresponding category according to the data information category, and the display screen synchronously displays the data information according to the storage order.

6. A system for displaying documents based on voice dialogues in a large model, characterized in that: include: The cloud end and the application end, the cloud end includes a first extraction module, a first recognition module, a judgment module, a first transmission module, a marking module, and an access module. The first extraction module is used to classify and extract document data of uploaded files. The first recognition module is used to identify and classify text data information, large image data information, small image data information, and table data information in the extracted document data. The judgment module is used to determine whether the data category in the document data is consistent with the partition block category in the knowledge base. The marking module is used to mark the position of the small image in the text by placing special characters plus IDs. The first transmission module is used to store the text data information, large image data information, and table data information in the document data in the corresponding partition blocks of the knowledge base. The first transmission module is used to transmit document data associated with voice dialogues to the application end. The first transmission module is used to transmit the small image corresponding to the ID in the knowledge base to the application end. The first transmission module is used to store the small image data information in the document data in the partition blocks corresponding to the ID information in the knowledge base according to the marked ID information. The access module is used to access the knowledge base; The application end includes a collection module, a second transmission module, an access module, a reading module, a second recognition module, a second extraction module, and a display module; the collection module is used to collect voice and audio information of voice conversations, the second transmission module is used to transmit the voice and audio information to a large cloud model, the second transmission module is used to transmit the ID in the special characters to the large cloud model, the access module is used to access a knowledge base, the reading module is used to read document data, the second recognition module is used to recognize special characters in text data information, the second extraction module is used to extract small pictures corresponding to the ID in the knowledge base, and the display module is used to synchronously display document data.

7. A system for displaying documents of speech dialogues based on a large model as claimed in claim 6, characterized in that: The first recognition module is also used to scan and recognize text, pictures, and tables in the uploaded file through the Pdfplumber library and OCR technology. The first recognition module is also used to recognize the title or text of the extracted text according to the text size and position coordinates. The first recognition module is also used to recognize the large picture or small picture by comparing the size of the extracted picture with the set value. The first transmission module is also used to store the extracted text data information in the order of title first and text later in the text partition, store the extracted large picture data information in the large picture partition in the scanning order, store the extracted table data information in the table partition in the scanning order, and store the extracted small picture data information in the IP address partition corresponding to the ID according to the marked ID information.

8. A system for presenting documents of speech dialogue based on a large model as claimed in claim 7, characterized in that: The cloud also includes a conversion module, a parsing module, and a retrieval module. The conversion module is used to perform semantic conversion processing on voice audio information, the parsing module is used to parse semantic information, and the retrieval module is used to retrieve document data in the knowledge base. The first transmission module is also used to transmit semantic information to a large cloud model, the first extraction module is also used to obtain associated document data, and the access module is also used to access a knowledge base in a field corresponding to an associated keyword based on multiple associated keywords.

9. A system for presenting documents of speech dialogue based on a large model as claimed in claim 8, characterized in that: The reading module is also used to read text data information, large image data information, and table data information in the order of reception.

10. A system for presenting documents of speech dialogue based on a large model as claimed in claim 9, characterized in that: The retrieval module is also used to retrieve the knowledge base according to the ID information, the first extraction module is also used to extract the small picture stored at the IP address location corresponding to the ID, the second transmission module is also used to temporarily store the data information in the display space of the corresponding category according to the category of the data information, and the display module is also used to synchronously display the data information according to the storage order.

Images