Information carrier processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an information carrier processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an information carrier, and extracting to-be-processed text content in the information carrier; performing format conversion processing on the to-be-processed text content to obtain target text content; obtaining an appeal dialogue text of the user; and in response to the appeal dialogue text, performing content understanding processing on the target text content to obtain a result corresponding to the appeal. The method comprises the following steps: extracting to-be-processed text content in an information carrier, performing format conversion processing on the to-be-processed text content to obtain target text content, responding to an appeal dialogue text of a user, performing content understanding processing on the target text content, and obtaining a result corresponding to the appeal, thereby helping the user to assist in reading the information carrier. The time spent by the user is reduced, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an information carrier processing method, device, electronic device and storage medium. Background Art

[0002] Currently, most information carriers, such as electronic documents, are electronic documents mainly in PDF format converted from typesetting tools and typesetting documents. The page numbers are dozens or even hundreds of pages, resulting in a lot of content, which requires users to spend a lot of time reading them by themselves. Summary of the invention

[0003] In order to solve the above technical problem that it takes a long time for users to read the information carrier by themselves, the embodiments of the present application provide an information carrier processing method, device, electronic device and storage medium. The specific technical solution is as follows:

[0004] In a first aspect of an embodiment of the present application, a method for processing an information carrier is first provided, the method comprising:

[0005] Acquiring an information carrier and extracting text content to be processed in the information carrier;

[0006] Performing format conversion processing on the text content to be processed to obtain target text content;

[0007] Get the user's demand dialogue text;

[0008] In response to the appeal dialogue text, the target text content is processed for content understanding to obtain a result corresponding to the appeal.

[0009] In an optional implementation, extracting the text content to be processed in the information carrier includes:

[0010] According to the carrier type of the information carrier, the text content to be processed in the information carrier is obtained.

[0011] In an optional implementation, extracting the text content to be processed in the information carrier includes:

[0012] In response to an opening operation on the information carrier, displaying the information carrier and determining a carrier type of the information carrier;

[0013] According to the carrier type of the information carrier, the text content to be processed in the information carrier is obtained.

[0014] In an optional implementation, the obtaining of the user's demand dialogue text includes:

[0015] In response to an auxiliary reading mode wake-up operation on the information carrier, displaying an auxiliary reading dialog box;

[0016] The user's demand dialogue text is obtained through the auxiliary reading dialog box.

[0017] In an optional implementation, in response to the appeal dialogue text, performing content understanding processing on the target text content to obtain a result corresponding to the appeal includes:

[0018] In response to the appeal dialogue text, identifying a user intention corresponding to the appeal dialogue text;

[0019] In the case where the user intention represents a user's generalized demand for the information carrier, extracting typesetting information from the target text content;

[0020] Calling a large language model to perform summary processing on the typesetting information, generating a summary corresponding to the information carrier, and displaying the summary in the auxiliary reading dialog box;

[0021] and / or,

[0022] The large language model is called to perform candidate prompt question summarization processing on the typesetting information, generate candidate prompt questions corresponding to the information carrier, and display the candidate prompt questions in the auxiliary reading dialog box.

[0023] In an optional implementation, in response to the appeal dialogue text, performing content understanding processing on the target text content to obtain a result corresponding to the appeal includes:

[0024] In response to the appeal dialogue text, identifying a user intention corresponding to the appeal dialogue text;

[0025] In the case where the user intention represents a user's question demand for the information carrier, determining the text content related to the demand dialogue text in the target text content;

[0026] Calling a large language model to perform question-answering processing on the relevant text content and the appeal dialogue text, and generating a question-answering result of the appeal dialogue text;

[0027] The question and answer result is displayed in the auxiliary reading dialog box.

[0028] In an optional implementation, after determining the relevant text content of the appeal dialogue text in the target text content, the method further includes:

[0029] The position of the relevant content text in the information carrier is determined, the page number corresponding to the position is searched, and the page number and / or the text content corresponding to the page number is displayed in the auxiliary reading dialog box.

[0030] In an optional embodiment, the method further comprises:

[0031] In response to a translation operation on the information carrier, determining a target language corresponding to the translation operation;

[0032] The translation engine is called to translate the target text content, generate the target text content corresponding to the target language, and display it in the auxiliary reading dialog box.

[0033] In an optional implementation, the acquiring information carrier includes:

[0034] Detect whether there is text content to be processed within the image acquisition range of the image acquisition module of the mobile terminal;

[0035] When there is text content to be processed within the image acquisition range, calling the image acquisition module of the mobile terminal to acquire an image of the text content to be processed to obtain an image;

[0036] Performing correction processing on the image to obtain a corrected image, and using the corrected image as an information carrier of the text content to be processed;

[0037] The extracting the text content to be processed in the information carrier includes:

[0038] A text recognition method for the image type of the information carrier is determined, and the text content to be processed in the information carrier is obtained by using the text recognition method.

[0039] In an optional implementation, after performing format conversion processing on the text content to be processed to obtain target text content, the method further includes:

[0040] Extracting the typesetting information in the target text content, and calling the large language model to perform summary processing on the typesetting information to generate a summary corresponding to the information carrier;

[0041] Calling a large language model to perform candidate prompt question summarization processing on the typesetting information to generate candidate prompt questions corresponding to the information carrier;

[0042] Displaying an auxiliary reading dialog box, displaying the summary on a summary interface of the auxiliary reading dialog box, and displaying the candidate prompt questions on a question-and-answer interface of the auxiliary reading dialog box;

[0043] The obtaining of the user's demand dialogue text includes:

[0044] In response to an interface switching operation on the auxiliary reading dialog box, switching from the summary interface to the question-and-answer interface;

[0045] In the question-and-answer interface, the user's demand dialogue text is obtained.

[0046] In an optional implementation, the extracting the layout information in the target text content includes:

[0047] Detecting whether the language corresponding to the target text content is a preset language;

[0048] When the language is a preset language, extracting typesetting information from the target text content;

[0049] In the case where the language is not a preset language, calling a translation engine to translate the target text content to generate target text content corresponding to the preset language;

[0050] Extract the layout information from the target text content in the preset language.

[0051] In an optional implementation, in response to the appeal dialogue text, performing content processing on the target text content to obtain a result corresponding to the appeal includes:

[0052] In response to the appeal dialogue text, identifying a user intention corresponding to the appeal dialogue text;

[0053] In the case where the user intention represents a user's question demand for the information carrier, determining the text content related to the demand dialogue text in the target text content;

[0054] The large language model is called to perform question-answering processing on the relevant text content and the appeal dialogue text, generate a question-answering result of the appeal dialogue text, and display the question-answering result in the question-answering interface.

[0055] In an optional implementation, after determining the relevant text content of the appeal dialogue text in the target text content, the method further includes:

[0056] Determine the position of the relevant content text in the information carrier, find the page number corresponding to the position, and display the page number and / or the text content corresponding to the page number in the question-and-answer interface.

[0057] In an optional implementation, the determining of the relevant text content of the appeal dialogue text in the target text content includes:

[0058] Vectorizing the appeal dialogue text to obtain an appeal vector;

[0059] Performing vectorization processing on the target text content to obtain a carrier vector;

[0060] According to the matching result between the appeal vector and the carrier vector, the relevant text content of the appeal dialogue text in the target text content is determined.

[0061] In an optional implementation, the vectorizing the target text content to obtain a carrier vector includes:

[0062] Segmenting the target text content according to a preset first granularity to obtain a plurality of first sub-text contents;

[0063] Performing vectorization processing on each of the first sub-text contents to obtain a first granularity carrier vector;

[0064] Segmenting the target text content according to a preset second granularity to obtain a plurality of second sub-text contents;

[0065] Vectorization processing is performed on each of the second text sub-contents to obtain a second granularity carrier vector.

[0066] In an optional implementation, determining the relevant text content of the appeal dialogue text in the target text content according to the matching result of the appeal vector and the carrier vector includes:

[0067] Searching for a first granularity carrier vector matching the demand vector from a plurality of the first granularity carrier vectors;

[0068] Determine the first subtext content corresponding to the first granularity carrier vector matched by the appeal vector as the relevant text content of the appeal dialogue text in the target text content;

[0069] or,

[0070] Searching for a second granularity carrier vector matching the demand vector from a plurality of the second granularity carrier vectors;

[0071] The second sub-text content corresponding to the second granularity carrier vector matched by the appeal vector is determined as the relevant text content of the appeal dialogue text in the target text content.

[0072] In a second aspect of the embodiments of the present application, an information carrier processing device is further provided, the device comprising:

[0073] A carrier acquisition module, used for acquiring information carriers;

[0074] A content extraction module, used to extract the text content to be processed in the information carrier;

[0075] A content conversion module, used for performing format conversion processing on the text content to be processed to obtain target text content;

[0076] The text acquisition module is used to obtain the user's demand dialogue text;

[0077] The content processing module is used to respond to the appeal dialogue text, perform content understanding processing on the target text content, and obtain a result corresponding to the appeal.

[0078] In an optional implementation, the content extraction module is specifically used to:

[0079] According to the carrier type of the information carrier, the text content to be processed in the information carrier is obtained.

[0080] In an optional implementation, the content extraction module is specifically used to:

[0081] In response to an opening operation on the information carrier, displaying the information carrier and determining a carrier type of the information carrier;

[0082] According to the carrier type of the information carrier, the text content to be processed in the information carrier is obtained.

[0083] In an optional implementation, the text acquisition module is specifically used to:

[0084] In response to an auxiliary reading mode wake-up operation on the information carrier, displaying an auxiliary reading dialog box;

[0085] The user's demand dialogue text is obtained through the auxiliary reading dialog box.

[0086] In an optional implementation, the content processing module is specifically used to:

[0087] In response to the appeal dialogue text, identifying a user intention corresponding to the appeal dialogue text;

[0088] In the case where the user intention represents a user's generalized demand for the information carrier, extracting typesetting information from the target text content;

[0089] Calling a large language model to perform summary processing on the typesetting information, generating a summary corresponding to the information carrier, and displaying the summary in the auxiliary reading dialog box;

[0090] and / or,

[0091] The large language model is called to perform candidate prompt question summarization processing on the typesetting information, generate candidate prompt questions corresponding to the information carrier, and display the candidate prompt questions in the auxiliary reading dialog box.

[0092] In an optional implementation, the content processing module specifically includes:

[0093] An intention recognition submodule, for identifying the user intention corresponding to the demand dialogue text in response to the demand dialogue text;

[0094] A content determination submodule, for determining the text content related to the appeal dialogue text in the target text content, when the user intention represents the user's question appeal for the information carrier;

[0095] A result generation submodule, for calling a large language model to perform question-answering processing on the relevant text content and the appeal dialogue text, and generating a question-answering result for the appeal dialogue text;

[0096] The result display submodule is used to display the question and answer result in the auxiliary reading dialog box.

[0097] In an optional embodiment, the device further comprises:

[0098] The page number display submodule is used to determine the position of the relevant content text in the information carrier, find the page number corresponding to the position, and display the page number and / or the text content corresponding to the page number in the auxiliary reading dialog box.

[0099] In an optional embodiment, the device further comprises:

[0100] A translation module, configured to determine a target language corresponding to a translation operation performed on the information carrier in response to the translation operation performed on the information carrier;

[0101] The translation engine is called to translate the target text content, generate the target text content corresponding to the target language, and display it in the auxiliary reading dialog box.

[0102] In an optional implementation, the carrier acquisition module is specifically used to:

[0103] Detect whether there is text content to be processed within the image acquisition range of the image acquisition module of the mobile terminal;

[0104] When there is text content to be processed within the image acquisition range, calling the image acquisition module of the mobile terminal to acquire an image of the text content to be processed to obtain an image;

[0105] Performing correction processing on the image to obtain a corrected image, and using the corrected image as an information carrier of the text content to be processed;

[0106] The content extraction module is specifically used for:

[0107] A text recognition method for the image type of the information carrier is determined, and the text content to be processed in the information carrier is obtained by using the text recognition method.

[0108] In an optional embodiment, the device further comprises:

[0109] An information extraction module, used to extract typesetting information from the target text content;

[0110] A summary generation module, used for calling a large language model to perform summary processing on the typesetting information, and generating a summary corresponding to the information carrier;

[0111] A question generation module, used for calling a large language model to perform candidate prompt question summarization processing on the typesetting information, and generating candidate prompt questions corresponding to the information carrier;

[0112] A summary and question display module, used to display an auxiliary reading dialog box, display the summary on the summary interface of the auxiliary reading dialog box, and display the candidate prompt questions on the question-answer interface of the auxiliary reading dialog box;

[0113] The text acquisition module is specifically used for:

[0114] In response to an interface switching operation on the auxiliary reading dialog box, switching from the summary interface to the question-and-answer interface;

[0115] In the question-and-answer interface, the user's demand dialogue text is obtained.

[0116] In an optional implementation, the information extraction module is specifically used to:

[0117] Detecting whether the language corresponding to the target text content is a preset language;

[0118] When the language is a preset language, extracting typesetting information from the target text content;

[0119] In the case where the language is not a preset language, calling a translation engine to translate the target text content to generate target text content corresponding to the preset language;

[0120] Extract the layout information from the target text content in the preset language.

[0121] In an optional implementation, the content processing module specifically includes:

[0122] An intention recognition submodule, for identifying the user intention corresponding to the demand dialogue text in response to the demand dialogue text;

[0123] A content determination submodule, for determining the text content related to the appeal dialogue text in the target text content, when the user intention represents the user's question appeal for the information carrier;

[0124] A result generation submodule, for calling a large language model to perform question-answering processing on the relevant text content and the appeal dialogue text, and generating a question-answering result for the appeal dialogue text;

[0125] The result display submodule is used to display the question and answer results in the question and answer interface.

[0126] In an optional embodiment, the device further comprises:

[0127] The page number display submodule is used to determine the position of the relevant content text in the information carrier, find the page number corresponding to the position, and display the page number and / or the text content corresponding to the page number in the question and answer interface.

[0128] In an optional implementation, the content determination submodule specifically includes:

[0129] A text processing unit, used for performing vectorization processing on the appeal dialogue text to obtain an appeal vector;

[0130] A content processing unit, used for performing vectorization processing on the target text content to obtain a carrier vector;

[0131] The content determination unit is used to determine the relevant text content of the appeal dialogue text in the target text content according to the matching result of the appeal vector and the carrier vector.

[0132] In an optional implementation, the content processing unit is specifically configured to:

[0133] Segmenting the target text content according to a preset first granularity to obtain a plurality of first sub-text contents;

[0134] Performing vectorization processing on each of the first sub-text contents to obtain a first granularity carrier vector;

[0135] Segmenting the target text content according to a preset second granularity to obtain a plurality of second sub-text contents;

[0136] Vectorization processing is performed on each of the second text sub-contents to obtain a second granularity carrier vector.

[0137] In an optional implementation manner, the content determination unit is specifically configured to:

[0138] Searching for a first granularity carrier vector matching the demand vector from a plurality of the first granularity carrier vectors;

[0139] Determine the first subtext content corresponding to the first granularity carrier vector matched by the appeal vector as the relevant text content of the appeal dialogue text in the target text content;

[0140] or,

[0141] Searching for a second granularity carrier vector matching the demand vector from a plurality of the second granularity carrier vectors;

[0142] The second sub-text content corresponding to the second granularity carrier vector matched by the appeal vector is determined as the relevant text content of the appeal dialogue text in the target text content.

[0143] In a third aspect of the embodiments of the present application, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0144] Memory, used to store computer programs;

[0145] The processor is used to implement any information carrier processing method described in the first aspect when executing the program stored in the memory.

[0146] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes the information carrier processing method described in any one of the first aspects above.

[0147] In a fifth aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned information carrier processing methods.

[0148] The technical solution provided by the embodiment of the present application obtains the information carrier, extracts the text content to be processed in the information carrier, performs format conversion processing on the text content to be processed to obtain the target text content, obtains the user's demand dialogue text, responds to the demand dialogue text, performs content understanding processing on the target text content, and obtains the result corresponding to the demand. By extracting the text content to be processed in the information carrier, performing format conversion processing on the text content to be processed to obtain the target text content, responding to the user's demand dialogue text, performing content understanding processing on the target text content, and obtaining the result corresponding to the demand, it helps the user to assist in reading the information carrier, reduces the time spent by the user, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0149] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0150] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0151] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0152] Figure 1 It is a schematic diagram of an implementation process of an information carrier processing method shown in an embodiment of the present application;

[0153] Figure 2 It is a schematic diagram of an implementation flow of another information carrier processing method shown in an embodiment of the present application;

[0154] Figure 3 A schematic diagram of displaying an auxiliary reading dialog box shown in an embodiment of the present application;

[0155] Figure 4 A schematic diagram of an implementation flow of a method for determining relevant text content shown in an embodiment of the present application;

[0156] Figure 5 It is a schematic diagram of an implementation flow of another information carrier processing method shown in an embodiment of the present application;

[0157] Figure 6 A schematic diagram of a mobile terminal collecting images shown in an embodiment of the present application;

[0158] Figure 7 A schematic diagram of displaying a summary of an insight interface in an auxiliary reading dialog box shown in an embodiment of the present application;

[0159] Figure 8 A schematic diagram of a display problem of an exploration interface in an auxiliary reading dialog box shown in an embodiment of the present application;

[0160] Fig. 9 A schematic diagram of a user voice input request dialogue text in a search interface shown in an embodiment of the present application;

[0161] Fig.10 A schematic diagram of displaying question and answer results and page numbers in a search interface shown in an embodiment of the present application;

[0162] Fig.11 It is a structural schematic diagram of an information carrier processing device shown in an embodiment of the present application;

[0163] Fig.12 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0164] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0165] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0166] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an information carrier processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0167] S101, obtaining an information carrier, and extracting text content to be processed in the information carrier.

[0168] In an embodiment of the present application, an information carrier can be obtained, and the text content to be processed in the information carrier can be extracted. The text content to be processed can be obtained through screening. For example, the text content to be processed extracted from an information carrier (such as a document) containing written notes can be the content after the written notes are hidden, or it can be the content without hiding the written notes.

[0169] It should be noted that for documents (such as PDF documents, Word documents, PPT documents), if the document is photographed / scanned, then the information carrier can be an image. Among them, for a document, if it is scanned and the format is converted, a scanned PDF can be obtained, and the information carrier can be a scanned PDF (essentially still a document).

[0170] In this way, the information carrier can be a variety of documents, such as Word documents, TXT documents or PDF documents, or scanned PDFs or images, which means that the information carrier can be of various types and carry certain text content and other information. The embodiments of the present application do not limit this.

[0171] Thus, the carrier type of the information carrier can be determined. Here, the carrier type of the information carrier can be multiple different carrier types. The corresponding carrier type can be, for example, a Word document, a TXT document, a PDF document, a scanned PDF or an image, etc., which means that the carrier type here is no longer limited to a PDF document.

[0172] For information carriers, the text content to be processed in the information carriers can be obtained according to the carrier type. Among them, the information carrier may be a streaming document or a non-streaming document, such as a layout document, a scanned PDF or a picture. The streaming document can directly obtain its text content to be processed, while the non-streaming document needs special processing.

[0173] For example, when the carrier type is a Word document or a TXT document, the text content to be processed in the information carrier can be directly obtained. When the carrier type is a PDF document, the PDF parsing engine corresponding to the PDF document can be called to parse the information carrier to obtain the text content to be processed. When the carrier type is a scanned PDF or image, OCR (optical character recognition) can be used for recognition to obtain the text content to be processed in the information carrier.

[0174] S102, performing format conversion processing on the text content to be processed to obtain target text content.

[0175] In the embodiment of the present application, for the text content to be processed, in order to facilitate the subsequent processing flow, it is necessary to perform a layout conversion on it, thereby calling a layout conversion engine to perform a layout conversion process on the text content to be processed to obtain a target text content, where the target text content can be regarded as a structured json (JavaScript Object Notation) format text content, which is equivalent to a structured streaming document. Among them, for the specific layout conversion, reference can be made to the existing solutions, and the embodiment of the present application will not be repeated here one by one.

[0176] It should be noted that the target text content is obtained by performing a format conversion process on the text content to be processed in the information carrier. Compared with the text content to be processed, the target text content has, for example, typesetting information (for example, it can be paragraphs, titles, multiple columns, etc.), and text attribute information (for example, it can be text color, text bold, text italic, etc.). Compared with directly performing content understanding processing on the text content to be processed in the information carrier, the above-mentioned typesetting information and text attribute information in the target text content can help to better and more conveniently perform content understanding processing on the text content to be processed in the information carrier, so as to more accurately summarize and answer questions about the text content to be processed in the information carrier. For example, in the subsequent process of summarizing the text content to be processed in the information carrier to generate summaries and candidate prompt questions, the above-mentioned typesetting information can allow the large language model to better and more conveniently understand the text content to be processed in the information carrier, so as to generate accurate summaries and candidate prompt questions.

[0177] S103, obtaining the user's demand dialogue text.

[0178] In an embodiment of the present application, the user can be assisted in reading the above-mentioned information carrier, so that the user can input a demand dialogue text, thereby obtaining the user's demand dialogue text. Here, the user's demand dialogue text can represent some reading demands of the user for the information carrier, such as summarizing the text content to be processed in the information carrier, or asking questions based on the text content to be processed in the information carrier. The embodiment of the present application does not limit this.

[0179] S104, in response to the appeal dialogue text, performing content understanding processing on the target text content to obtain a result corresponding to the appeal.

[0180] In an embodiment of the present application, for the user's demand dialogue text, the target text content can be understood and processed in response to the user's demand dialogue text to obtain a result corresponding to the demand, thereby realizing the processing of the information carrier, which can help the user to assist in reading the text content to be processed in the information carrier.

[0181] It should be noted that the results corresponding to the above-mentioned demands may be, for example, a summary obtained by summarizing the text content to be processed in the information carrier, or a question-and-answer result generated by questions raised based on the text content to be processed in the information carrier, and the embodiments of the present application are not limited to this.

[0182] Through the above description of the technical solution provided by the embodiment of the present application, by obtaining the text content to be processed in information carriers of different carrier types, performing format conversion processing on the text content to be processed to obtain the target text content, responding to the user's demand dialogue text, performing content understanding processing on the target text content, and obtaining the result corresponding to the demand, it helps users to assist in reading the information carrier, reduces the time spent by users, and improves the user experience. And it can support information carriers of different carrier types, further improving the user experience.

[0183] In the present application, the information carrier processing method provided in the present application can be applied to, for example, a PC terminal, or a mobile terminal (such as a mobile phone), etc. For the PC terminal and the mobile terminal, etc., the information carrier processing method provided in the present application is slightly different in form. Subsequently, the information carrier provided in the present application will be explained respectively by taking the PC terminal and the mobile terminal as examples.

[0184] Among them, on the PC side, the information carrier processing method can be as follows Figure 2 The method described in the figure can be presented in the form of a document reader such as a PDF document or a Word document to assist in reading comprehension and answer questions. Specifically, Figure 2 FIG. 1 is a schematic diagram of an implementation flow of another information carrier processing method provided in an embodiment of the present application. The method is applied to a PC, for example, and may specifically include the following steps:

[0185] S201, acquiring an information carrier, displaying the information carrier in response to an opening operation on the information carrier, and determining a carrier type of the information carrier.

[0186] S202: Acquire the text content to be processed in the information carrier according to the carrier type of the information carrier.

[0187] S203, performing format conversion processing on the text content to be processed to obtain target text content.

[0188] In the embodiment of the present application, the user often uploads the information carrier to the cloud, and can use the download link to download the information carrier, thereby obtaining the information carrier. The user can perform an opening operation on the information carrier, so that the information carrier can be displayed in response to the opening operation on the information carrier.

[0189] It should be noted that the information carrier is generally opened with the aid of a reader. For example, for a Word document, the user performs an opening operation, and the information carrier can be opened through a document reader in response to the opening operation on the information carrier, and the information carrier can be displayed in the document reader, which is not limited in the embodiments of the present application.

[0190] In addition, when the user performs the operation of opening the information carrier, it means that the user may have a need for assisted reading and question-and-answering on the information carrier. At this time, the carrier type of the information carrier can be determined, and the text content to be processed in the information carrier can be obtained according to the carrier type of the information carrier. The format of the text content to be processed is converted to obtain the target text content, in preparation for subsequent assisted reading and question-and-answering.

[0191] S204, in response to the auxiliary reading mode wake-up operation on the information carrier, displaying an auxiliary reading dialog box.

[0192] S205, obtaining the user's demand dialogue text through the auxiliary reading dialog box.

[0193] In an embodiment of the present application, for a reader (such as a document reader, an image reader), an auxiliary reading system is embedded therein, so that after a user opens an information carrier through the reader, the user can perform an auxiliary reading mode wake-up operation on the information carrier in the reader, and in response to the auxiliary reading mode wake-up operation performed on the information carrier in the reader, the auxiliary reading system is awakened and an auxiliary reading dialog box is displayed, that is, in response to the auxiliary reading mode wake-up operation on the information carrier, the auxiliary reading system is awakened and an auxiliary reading dialog box is displayed.

[0194] It should be noted that for readers (such as document readers, image readers, etc.), corresponding buttons are set therein, such as an "assisted reading" button. The user can click this button to execute the wake-up operation of the assisted reading mode for the information carrier, thereby waking up the assisted reading system in response to the wake-up operation of the assisted reading mode for the information carrier, and displaying the assisted reading dialog box on one side of the reader (such as a document reader, image reader, etc.).

[0195] In addition, when a reader (eg, a document reader) is referred to as a side, it is understood that it generally refers to the right side of the reader (eg, a document reader), such as Figure 3 As shown, the right side conforms to the user's habit of using a reader (such as a document reader). Of course, it can also be the left side of the reader (such as a document reader), or the upper side of the reader (such as a document reader), or the lower side of the reader (such as a document reader). The embodiment of the present application is not limited to this.

[0196] For the assisted reading dialog box, the user can interact with the assisted reading system in the assisted reading dialog box, so that the user can enter his or her own demand dialogue text in the assisted reading dialog box, and the assisted reading system can obtain the user's demand dialogue text, so as to perform operations related to assisted reading according to the user's demand dialogue text.

[0197] S206, in response to the demand dialogue text, identifying the user intention corresponding to the demand dialogue text.

[0198] S207, when the user intention represents the user's question demand for the information carrier, determine the relevant text content of the demand dialogue text in the target text content.

[0199] S208, calling the large language model to perform question-answering processing on the relevant text content and the appeal dialogue text, and generating a question-answering result of the appeal dialogue text.

[0200] S209, displaying the question and answer result in the auxiliary reading dialog box.

[0201] In an embodiment of the present application, the auxiliary reading system can respond to the user's demand dialogue text and identify the user intention corresponding to the demand dialogue text. When the user intention represents the user's question demand regarding the information carrier, it means that the user has asked a question regarding the information carrier. Therefore, the relevant text content of the demand dialogue text in the target text content can be determined, and the large language model can be called to perform question and answer processing on the relevant text content and the demand dialogue text, generate the question and answer results of the demand dialogue text, and display the question and answer results in the auxiliary reading dialog box.

[0202] The Large Language Model (LLM) is a model based on machine learning and natural language processing technology. It learns to serve human language understanding and generation by training a large amount of text data. The core idea of ​​LLM is to learn the patterns and language structures of natural speech through large-scale unsupervised training, which can simulate the human language cognition and generation process to a certain extent.

[0203] For example, Figure 3As shown, the user enters the demand dialogue text in the auxiliary reading dialog box: "What are the preparation methods of electrospun nanofibers?" The auxiliary reading system can respond to the above user's demand dialogue text and identify the user intention corresponding to the demand dialogue text. Here, the user intention represents the user's question demand for the information carrier, which means that the user has asked a question about the information carrier. Therefore, the relevant text content of the demand dialogue text ("What are the preparation methods of electrospun nanofibers") in the target text content can be determined, and the large language model is called to perform question and answer processing on the relevant text content and the demand dialogue text, and generate the question and answer results of the demand dialogue text ("There are many preparation methods for electrospun nanofibers. Generally speaking, the preparation process of electrospun nanofibers includes the following steps: first of all..."), and the question and answer results are displayed in the auxiliary reading dialog box.

[0204] In addition, in order to facilitate the tracing of the questions asked by the user, the page number of the text content related to the question asked by the user in the information carrier and / or the text content corresponding to the page number can be displayed while displaying the question and answer results in the auxiliary reading dialog box. Specifically, after determining the relevant text content of the appeal dialogue text in the target text content, determine the position of the relevant content text in the information carrier, find the page number corresponding to the position, and display the page number and / or the text content corresponding to the page number in the auxiliary reading dialog box. For example, if the relevant pages are 3, 5, and 9, they can be displayed below the question and answer results, such as Figure 3 In this way, in the auxiliary reading dialog box, not only the question and answer results are displayed, but also the page number of the text content related to the question asked by the user in the information carrier and / or the text content corresponding to the page number are displayed, so that the user can easily trace the question asked.

[0205] Among them, for determining the relevant text content of the appeal dialogue text in the target text content, you can refer to the following: Figure 4 As shown in the method. Figure 4 FIG. 1 is a schematic diagram of an implementation flow of a method for determining relevant text content provided in an embodiment of the present application. The method may specifically include the following steps:

[0206] S401, vectorizing the appeal dialogue text to obtain an appeal vector.

[0207] In the embodiment of the present application, the user's demand dialogue text can be vectorized to obtain a demand vector.

[0208] The appeal vector may be an embedding vector, and thus the appeal dialogue text may be vectorized through a preset embedding layer to obtain the embedding vector corresponding to the appeal dialogue text.

[0209] S402, vectorizing the target text content to obtain a carrier vector.

[0210] S403, determining the relevant text content of the appeal dialogue text in the target text content according to the matching result of the appeal vector and the carrier vector.

[0211] In the embodiment of the present application, the target text content can also be vectorized to obtain a carrier vector.

[0212] The carrier vector may also be an embedding vector, so that the target text content may be vectorized through a preset embedding layer to obtain the embedding vector corresponding to the target text content.

[0213] Therefore, the appeal vector and the carrier vector can be matched, and the relevant text content of the appeal dialogue text in the target text content can be determined based on the matching results of the two. Here, the appeal vector and the carrier vector are matched, which is actually the matching of the embedding vector corresponding to the appeal dialogue text and the embedding vector corresponding to the target text content.

[0214] Among them, the user's demand dialogue text may be a relatively detailed question or a relatively rough question. In order to adapt to questions of different levels, the determination of the relevant text content of the demand dialogue text in the target text content can also be processed at different granularities.

[0215] Based on this, for the target text content, the target text content is segmented according to a preset first granularity to obtain multiple first sub-text contents, each first sub-text content is vectorized to obtain a first granularity carrier vector, and the target text content is segmented according to a preset second granularity to obtain multiple second sub-text contents, each second text sub-content is vectorized to obtain a second granularity carrier vector.

[0216] Thus, a first granularity carrier vector matching the appeal vector can be searched from among multiple first granularity carrier vectors, and the first subtext content corresponding to the first granularity carrier vector matching the appeal vector can be determined as the relevant text content of the appeal dialogue text in the target text content. Alternatively, a second granularity carrier vector matching the appeal vector can be searched from among multiple second granularity carrier vectors, and the second subtext content corresponding to the second granularity carrier vector matching the appeal vector can be determined as the relevant text content of the appeal dialogue text in the target text content.

[0217] It should be noted that, for the demand vector, if the similarity between the demand vector and the first granularity carrier vector is higher than a threshold, the demand vector is considered to match the first granularity carrier vector. Similarly, if the similarity between the demand vector and the second granularity carrier vector is higher than a threshold, the demand vector is considered to match the second granularity carrier vector. For the calculation of the similarity between vectors, reference can be made to the existing algorithms, and the embodiments of the present application will not be described one by one here.

[0218] In addition, the above-mentioned preset first granularity can be a paragraph, and the preset second granularity can be a sentence. If the user's demand dialogue text can represent that the question asked by the user is relatively rough, then from multiple first-granularity carrier vectors, a first-granularity carrier vector matching the demand vector is searched, and the first sub-text content corresponding to the first-granularity carrier vector matching the demand vector is determined as the relevant text content of the demand dialogue text in the target text content. If the user's demand dialogue text can represent that the question asked by the user is relatively fine, then from multiple second-granularity carrier vectors, a second-granularity carrier vector matching the demand vector is searched, and the second sub-text content corresponding to the second-granularity carrier vector matching the demand vector is determined as the relevant text content of the demand dialogue text in the target text content.

[0219] For example, the target text content is segmented into paragraphs to obtain multiple first sub-text contents (i.e., the text content of each paragraph), each first sub-text content is vectorized to obtain a first granularity carrier vector (i.e., embedding vector), the target text content is segmented into sentences to obtain multiple second sub-text contents (i.e., the text content of each sentence), each second text sub-content is vectorized to obtain a second granularity carrier vector (i.e., embedding vector).

[0220] If the user's demand dialogue text represents the question asked by the user in a relatively rough way, then the relevant text content of the demand dialogue text in the target text content can be determined from the paragraph granularity, that is, from multiple first-granularity carrier vectors, a first-granularity carrier vector matching the demand vector is searched, and the first sub-text content corresponding to the first-granularity carrier vector matching the demand vector is determined as the relevant text content of the demand dialogue text in the target text content; if the user's demand dialogue text represents the question asked by the user in a relatively fine way, then the relevant text content of the demand dialogue text in the target text content can be determined from the sentence granularity, that is, from multiple second-granularity carrier vectors, a second-granularity carrier vector matching the demand vector is searched, and the second sub-text content corresponding to the second-granularity carrier vector matching the demand vector is determined as the relevant text content of the demand dialogue text in the target text content.

[0221] In addition, for information carriers, embodiments of the present application can summarize the text content to be processed in the information carrier according to the user's demands, thereby generating a summary for the user to view. Therefore, in response to the demand dialogue text, the user intent corresponding to the demand dialogue text is identified. When the user intent represents the user's summary demand for the information carrier, the typesetting information in the target text content is extracted, and the large language model is called to summarize the typesetting information, generate a summary corresponding to the information carrier, and display the summary in the auxiliary reading dialog box, or call the large language model to summarize the typesetting information for candidate prompt questions, generate candidate prompt questions corresponding to the information carrier, and display the candidate prompt questions in the auxiliary reading dialog box, such as Figure 3 shown.

[0222] It should be noted that the typesetting information may be, for example, paragraphs, titles (such as first-level titles, second-level titles, etc.), multiple columns, etc. In addition, in the process of generating summaries, text attribute information (such as text color, text bold, text italic, etc.) may also be used, that is, the typesetting information and text attribute information in the target text content are extracted, and the large language model is called to perform summary processing on the typesetting information and text attribute information to generate a summary corresponding to the information carrier, or the large language model is called to perform candidate prompt question summary processing on the typesetting information and text attribute information to generate candidate prompt questions corresponding to the information carrier.

[0223] In the embodiment of the present application, for the information carrier, the recorded language of the text content to be processed is various, such as English, so a translation function can be provided to facilitate language conversion and assist the user in reading the text content to be processed in the information carrier. Based on this, the user can perform a translation operation on the information carrier in a reader (such as a document reader), so that in response to the translation operation on the information carrier, the target language corresponding to the translation operation is determined, the translation engine is called to translate the target text content, and the target text content corresponding to the target language is generated and displayed in the auxiliary reading dialog box.

[0224] Through the above description of the technical solution provided by the embodiment of the present application, by acquiring information carriers of different carrier types, in response to the opening operation on the information carrier, the information carrier is displayed, and according to the carrier type of the information carrier, the text content to be processed in the information carrier is acquired, and the format conversion processing is performed on the text content to be processed to obtain the target text content, and in response to the auxiliary reading mode wake-up operation on the information carrier, the auxiliary reading dialog box is displayed, and the user's demand dialogue text is acquired through the auxiliary reading dialog box. In response to the demand dialogue text, the user intention corresponding to the demand dialogue text is identified, and when the user intention represents the user's question demand for the information carrier, the relevant text content of the demand dialogue text in the target text content is determined, and the large language model is called to perform question and answer processing on the relevant text content and the demand dialogue text, and the question and answer results of the demand dialogue text are generated, and the question and answer results are displayed in the auxiliary reading dialog box, so as to help users assist in reading information carriers, support information carriers of different carrier types, reduce the time spent by users, improve user experience, and inquire about the content of the text to be processed in the information carrier through multiple rounds of dialogue. Of course, it can also help users summarize the content of the text to be processed in the information carrier, thereby generating a summary to help users gain insight into important information therein.

[0225] On the mobile side, the information carrier processing method can be as follows Figure 5 The method described in the figure can be applied to photo-taking scenarios to assist in reading comprehension and answer questions. Figure 5 FIG. 1 is a schematic diagram of an implementation flow of another information carrier processing method provided in an embodiment of the present application. The method is applied to, for example, a mobile terminal and may specifically include the following steps:

[0226] S501, detecting whether there is text content to be processed within the image acquisition range of the image acquisition module of the mobile terminal.

[0227] S502: When there is text content to be processed within the image acquisition range, call the image acquisition module of the mobile terminal to perform image acquisition on the text content to be processed to obtain an image.

[0228] S503, correcting the image to obtain a corrected image, and using the corrected image as an information carrier of the text content to be processed.

[0229] In an embodiment of the present application, a mobile terminal (or mobile terminal) usually carries an image acquisition module, such as a camera. A user can use the image acquisition module of the mobile terminal to take a photo of a text content to be processed, thereby generating an image. The image will serve as an information carrier of the text content to be processed, and the text content to be processed in the image will be subsequently processed to assist the user in reading the text content to be processed.

[0230] Based on this, it is detected whether there is text content to be processed within the image acquisition range of the image acquisition module of the mobile terminal. If there is text content to be processed within the image acquisition range, the image acquisition module of the mobile terminal is called to perform image acquisition on the text content to be processed, obtain an image, perform correction processing on the image, obtain the corrected image, and use the corrected image as the information carrier of the text content to be processed, such as Figure 6 shown.

[0231] It should be noted that the correction of the image may specifically be quadrilateral correction and curvature correction, etc., which is not limited in the embodiments of the present application. The specific correction principle may refer to the existing correction principle.

[0232] S504, determining a text recognition method for the image type of the information carrier, and using the text recognition method to obtain the text content to be processed in the information carrier.

[0233] In an embodiment of the present application, for the information carrier, that is, the above-mentioned corrected image, the text recognition method of its image type (that is, the aforementioned OCR recognition) can be determined, and this text recognition method can be used to obtain the information carrier, that is, the text content to be processed in the corrected image. For example, the text content to be processed can be an English contract.

[0234] S505, performing format conversion processing on the text content to be processed to obtain target text content, and extracting typesetting information in the target text content.

[0235] S506, calling the large language model to perform summary processing on the typesetting information to generate a summary corresponding to the information carrier, calling the large language model to perform candidate prompt question summary processing on the typesetting information to generate candidate prompt questions corresponding to the information carrier.

[0236] S507, displaying an auxiliary reading dialog box, displaying a summary on a summary interface of the auxiliary reading dialog box, and displaying candidate prompt questions on a question-and-answer interface of the auxiliary reading dialog box.

[0237] In an embodiment of the present application, for the text content to be processed, it can be subjected to layout conversion processing to obtain the target text content, and the typesetting information in the target text content can be extracted. Then, the large language model can be called to summarize the typesetting information to generate a summary corresponding to the information carrier. At the same time, the large language model can also be called to summarize the typesetting information to generate candidate prompt questions corresponding to the information carrier. The questions here are candidate prompt questions that may be of interest to the user and are generated based on the target text content.

[0238] After generating the summary and candidate prompt questions, the auxiliary reading dialog box can be displayed. There are two interfaces in the auxiliary reading dialog box, namely the summary interface and the inquiry interface. The interfaces can be switched. For the summary generated above, the summary can be displayed in the summary interface of the auxiliary reading dialog box. For the candidate prompt questions generated above, the candidate prompt questions can be displayed in the question-and-answer interface of the auxiliary reading dialog box. For example, taking a contract as an example, the summary interface is the "Insight" interface, and the summary of this contract can be displayed in the "Insight" interface, such as Figure 7 As shown, the inquiry interface is an "exploration" interface, and candidate prompt questions can be displayed in the "exploration interface", such as Figure 8 As shown, the questions here are candidate prompt questions that the user may be interested in, generated based on this contract.

[0239] S508, in response to the interface switching operation for the auxiliary reading dialog box, the summary interface is switched to the question-and-answer interface, and the user's demand dialogue text is obtained in the question-and-answer interface.

[0240] S509, in response to the demand dialogue text, identifying the user intention corresponding to the demand dialogue text.

[0241] S510, when the user intention represents the user's question demand for the information carrier, determine the relevant text content of the demand dialogue text in the target text content.

[0242] S511, calling the large language model to perform question-answering processing on the relevant text content and the appeal dialogue text, generating a question-answering result of the appeal dialogue text, and displaying the question-answering result in the question-answering interface.

[0243] In an embodiment of the present application, for the auxiliary reading dialog box, the user can perform an interface switching operation, switching from the summary interface to the question and answer interface, thereby responding to the interface switching operation for the auxiliary reading dialog box, switching from the summary interface to the question and answer interface. In the question and answer interface, the user can enter the demand dialogue text, thereby obtaining the user's demand dialogue text.

[0244] In the embodiment of the present application, the user's demand dialogue text can be input by voice, and then voice recognition is performed to obtain the user's demand dialogue text, such as Fig. 9 shown.

[0245] For the user's demand dialogue text, the user intention corresponding to the demand dialogue text can be identified in response to the demand dialogue text. When the user intention represents the user's question demand for the information carrier, the relevant text content of the demand dialogue text in the target text content is determined, and the large language model is called to perform question and answer processing on the relevant text content and the demand dialogue text to generate a question and answer result of the demand dialogue text, and the question and answer result is displayed in the question and answer interface. For example, the answer result can be "The name of Company A is...", such as Fig.10 The question and answer results can be output in the form of voice broadcast.

[0246] In addition, in order to facilitate the tracing of the questions asked by the user, the page number of the text content related to the question asked by the user in the information carrier and / or the text content corresponding to the page number can be displayed while displaying the question and answer results in the question and answer interface. Specifically, after determining the relevant text content of the appeal dialogue text in the target text content, determine the position of the relevant content text in the information carrier, find the page number corresponding to the position, and display the page number and / or the text content corresponding to the page number in the question and answer interface. If the relevant page is 1, it can be displayed below the question and answer results, such as Fig.10 In this way, in the question-and-answer interface, not only the question-and-answer results are displayed, but also the page number of the text content related to the question asked by the user in the information carrier and / or the text content corresponding to the page number are displayed, so that the user can easily trace the question asked.

[0247] In addition, for the text content to be processed, for example, it may be in English instead of Chinese. Therefore, the language of the target text content obtained by performing format conversion on the text content to be processed remains unchanged. If the typesetting information in the target text content is directly extracted and then processed by a large language model, the summary and questions generated may be in English instead of Chinese, which affects the user's reading.

[0248] Based on this, it is possible to detect whether the language corresponding to the target text content is a preset language (such as Chinese); if the language is the preset language, extract the typesetting information in the target text content; if the language is not the preset language, call the translation engine to translate the target text content to generate the target text content corresponding to the preset language; extract the typesetting information in the target text content of the preset language.

[0249] In addition, the appeal dialogue text is vectorized to obtain an appeal vector, and the target text content is vectorized to obtain a carrier vector. According to the matching result of the appeal vector and the carrier vector, the relevant text content of the appeal dialogue text in the target text content is determined. Specifically, the relevant text content of the appeal dialogue text in the target text content can be determined by referring to the following. Figure 4 The method shown in the embodiment of the present application will not be described in detail here.

[0250] Through the above description of the technical solution provided by the embodiment of the present application, an image is collected as an information carrier of the text content to be processed through an image acquisition module on a mobile terminal, a text recognition method of an image type of the information carrier is used to obtain the text content to be processed in the information carrier, a layout conversion process is performed on the text content to be processed to obtain a target text content, typesetting information in the target text content is extracted, a large language model is called to perform summary processing on the typesetting information, a summary corresponding to the information carrier is generated, a large language model is called to perform candidate prompt question summary processing on the typesetting information, candidate prompt questions corresponding to the information carrier are generated, a summary is displayed in a summary interface of an auxiliary reading dialog box, and candidate prompt questions are displayed in a question-and-answer interface of an auxiliary reading dialog box, and in response to an interface switching operation for the auxiliary reading dialog box, Switch from the summary interface to the question-and-answer interface, and in the question-and-answer interface, obtain the user's demand dialogue text, respond to the demand dialogue text, identify the user's intention corresponding to the demand dialogue text, and when the user intention represents the user's question demand for the information carrier, determine the relevant text content of the demand dialogue text in the target text content, call the large language model to perform question-and-answer processing on the relevant text content and the demand dialogue text, generate the question-and-answer results of the demand dialogue text, and display the question-and-answer results in the question-and-answer interface. In this way, taking photos can help users assist in reading, improve the user experience, and through multiple rounds of dialogue, inquire about the text content to be processed in the information carrier. Of course, it can also help users summarize the text content to be processed in the information carrier, thereby generating a summary and helping users gain insight into important information.

[0251] Corresponding to the above method embodiment, the present application embodiment also provides an information carrier processing device, such as Fig.11 As shown, the device may include: a carrier acquisition module 1110 , a content extraction module 1120 , a content conversion module 1130 , a text acquisition module 1140 , and a content processing module 1150 .

[0252] The carrier acquisition module 1110 is used to acquire the information carrier;

[0253] A content extraction module 1120, used to extract the text content to be processed in the information carrier;

[0254] The content conversion module 1130 is used to perform format conversion processing on the text content to be processed to obtain target text content;

[0255] The text acquisition module 1140 is used to acquire the user's demand dialogue text;

[0256] The content processing module 1150 is used to respond to the appeal dialogue text, perform content understanding processing on the target text content, and obtain a result corresponding to the appeal.

[0257] In an optional implementation, the content extraction module is specifically used to:

[0258] According to the carrier type of the information carrier, the text content to be processed in the information carrier is obtained.

[0259] In an optional implementation, the content extraction module is specifically used to:

[0260] In response to an opening operation on the information carrier, displaying the information carrier and determining a carrier type of the information carrier;

[0261] According to the carrier type of the information carrier, the text content to be processed in the information carrier is obtained.

[0262] In an optional implementation, the text acquisition module is specifically used to:

[0263] In response to an auxiliary reading mode wake-up operation on the information carrier, displaying an auxiliary reading dialog box;

[0264] The user's demand dialogue text is obtained through the auxiliary reading dialog box.

[0265] In an optional implementation, the content processing module is specifically used to:

[0266] In response to the appeal dialogue text, identifying a user intention corresponding to the appeal dialogue text;

[0267] In the case where the user intention represents a user's generalized demand for the information carrier, extracting typesetting information from the target text content;

[0268] Calling a large language model to perform summary processing on the typesetting information, generating a summary corresponding to the information carrier, and displaying the summary in the auxiliary reading dialog box;

[0269] and / or,

[0270] The large language model is called to perform candidate prompt question summarization processing on the typesetting information, generate candidate prompt questions corresponding to the information carrier, and display the candidate prompt questions in the auxiliary reading dialog box.

[0271] In an optional implementation, the content processing module specifically includes:

[0272] An intention recognition submodule, for identifying the user intention corresponding to the demand dialogue text in response to the demand dialogue text;

[0273] A content determination submodule, for determining the text content related to the appeal dialogue text in the target text content, when the user intention represents the user's question appeal for the information carrier;

[0274] A result generation submodule, for calling a large language model to perform question-answering processing on the relevant text content and the appeal dialogue text, and generating a question-answering result for the appeal dialogue text;

[0275] The result display submodule is used to display the question and answer result in the auxiliary reading dialog box.

[0276] In an optional embodiment, the device further comprises:

[0277] The page number display submodule is used to determine the position of the relevant content text in the information carrier, find the page number corresponding to the position, and display the page number and / or the text content corresponding to the page number in the auxiliary reading dialog box.

[0278] In an optional embodiment, the device further comprises:

[0279] A translation module, configured to determine a target language corresponding to a translation operation performed on the information carrier in response to the translation operation performed on the information carrier;

[0280] The translation engine is called to translate the target text content, generate the target text content corresponding to the target language, and display it in the auxiliary reading dialog box.

[0281] In an optional implementation, the carrier acquisition module is specifically used to:

[0282] Detect whether there is text content to be processed within the image acquisition range of the image acquisition module of the mobile terminal;

[0283] When there is text content to be processed within the image acquisition range, calling the image acquisition module of the mobile terminal to acquire an image of the text content to be processed to obtain an image;

[0284] Performing correction processing on the image to obtain a corrected image, and using the corrected image as an information carrier of the text content to be processed;

[0285] The content extraction module is specifically used for:

[0286] A text recognition method for the image type of the information carrier is determined, and the text content to be processed in the information carrier is obtained by using the text recognition method.

[0287] In an optional embodiment, the device further comprises:

[0288] An information extraction module, used to extract typesetting information from the target text content;

[0289] A summary generation module, used for calling a large language model to perform summary processing on the typesetting information, and generating a summary corresponding to the information carrier;

[0290] A question generation module, used for calling a large language model to perform candidate prompt question summarization processing on the typesetting information, and generating candidate prompt questions corresponding to the information carrier;

[0291] A summary and question display module, used to display an auxiliary reading dialog box, display the summary on the summary interface of the auxiliary reading dialog box, and display the candidate prompt questions on the question-answer interface of the auxiliary reading dialog box;

[0292] The text acquisition module is specifically used for:

[0293] In response to an interface switching operation on the auxiliary reading dialog box, switching from the summary interface to the question-and-answer interface;

[0294] In the question-and-answer interface, the user's demand dialogue text is obtained.

[0295] In an optional implementation, the information extraction module is specifically used to:

[0296] Detecting whether the language corresponding to the target text content is a preset language;

[0297] When the language is a preset language, extracting typesetting information from the target text content;

[0298] In the case where the language is not a preset language, calling a translation engine to translate the target text content to generate target text content corresponding to the preset language;

[0299] Extract the layout information from the target text content in the preset language.

[0300] In an optional implementation, the content processing module specifically includes:

[0301] An intention recognition submodule, for identifying the user intention corresponding to the demand dialogue text in response to the demand dialogue text;

[0302] A content determination submodule, for determining the text content related to the appeal dialogue text in the target text content, when the user intention represents the user's question appeal for the information carrier;

[0303] A result generation submodule, for calling a large language model to perform question-answering processing on the relevant text content and the appeal dialogue text, and generating a question-answering result for the appeal dialogue text;

[0304] The result display submodule is used to display the question and answer results in the question and answer interface.

[0305] In an optional embodiment, the device further comprises:

[0306] The page number display submodule is used to determine the position of the relevant content text in the information carrier, find the page number corresponding to the position, and display the page number and / or the text content corresponding to the page number in the question and answer interface.

[0307] In an optional implementation, the content determination submodule specifically includes:

[0308] A text processing unit, used for performing vectorization processing on the appeal dialogue text to obtain an appeal vector;

[0309] A content processing unit, used for performing vectorization processing on the target text content to obtain a carrier vector;

[0310] The content determination unit is used to determine the relevant text content of the appeal dialogue text in the target text content according to the matching result of the appeal vector and the carrier vector.

[0311] In an optional implementation, the content processing unit is specifically configured to:

[0312] Segmenting the target text content according to a preset first granularity to obtain a plurality of first sub-text contents;

[0313] Performing vectorization processing on each of the first sub-text contents to obtain a first granularity carrier vector;

[0314] Segmenting the target text content according to a preset second granularity to obtain a plurality of second sub-text contents;

[0315] Vectorization processing is performed on each of the second text sub-contents to obtain a second granularity carrier vector.

[0316] In an optional implementation manner, the content determination unit is specifically configured to:

[0317] Searching for a first granularity carrier vector matching the demand vector from a plurality of the first granularity carrier vectors;

[0318] Determine the first subtext content corresponding to the first granularity carrier vector matched by the appeal vector as the relevant text content of the appeal dialogue text in the target text content;

[0319] or,

[0320] Searching for a second granularity carrier vector matching the demand vector from a plurality of the second granularity carrier vectors;

[0321] The second sub-text content corresponding to the second granularity carrier vector matched by the appeal vector is determined as the relevant text content of the appeal dialogue text in the target text content.

[0322] The present application also provides an electronic device, such as Fig.12 As shown, it includes a processor 121, a communication interface 122, a memory 123 and a communication bus 124, wherein the processor 121, the communication interface 122, and the memory 123 communicate with each other through the communication bus 124.

[0323] Memory 123, used for storing computer programs;

[0324] The processor 121 is used to execute the program stored in the memory 123 to implement the following steps:

[0325] Acquire an information carrier, extract the text content to be processed in the information carrier; perform format conversion processing on the text content to be processed to obtain target text content; acquire the user's demand dialogue text; respond to the demand dialogue text, perform content understanding processing on the target text content, and obtain a result corresponding to the demand.

[0326] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0327] The communication interface is used for communication between the above electronic device and other devices.

[0328] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0329] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0330] In another embodiment provided in the present application, a storage medium is provided, in which instructions are stored. When the storage medium is run on a computer, the computer executes the information carrier processing method described in any one of the above embodiments.

[0331] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the information carrier processing method described in any one of the above embodiments.

[0332] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0333] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0334] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0335] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for processing an information carrier, characterized in that: The method comprises: Acquiring an information carrier and extracting text content to be processed in the information carrier; Performing format conversion processing on the text content to be processed to obtain target text content; Get the user's demand dialogue text; In response to the appeal dialogue text, the target text content is processed for content understanding to obtain a result corresponding to the appeal.

2. The method according to claim 1, characterized in that The extracting the text content to be processed in the information carrier includes: According to the carrier type of the information carrier, the text content to be processed in the information carrier is obtained.

3. The method according to claim 1, characterized in that The extracting the text content to be processed in the information carrier includes: In response to an opening operation on the information carrier, displaying the information carrier and determining a carrier type of the information carrier; According to the carrier type of the information carrier, the text content to be processed in the information carrier is obtained.

4. The method according to claim 3, characterized in that The obtaining of the user's demand dialogue text includes: In response to an auxiliary reading mode wake-up operation on the information carrier, displaying an auxiliary reading dialog box; The user's demand dialogue text is obtained through the auxiliary reading dialog box.

5. The method according to claim 4, characterized in that The step of performing content understanding processing on the target text content in response to the appeal dialogue text to obtain a result corresponding to the appeal includes: In response to the appeal dialogue text, identifying a user intention corresponding to the appeal dialogue text; In the case where the user intention represents a user's generalized demand for the information carrier, extracting typesetting information from the target text content; Calling a large language model to perform summary processing on the typesetting information, generating a summary corresponding to the information carrier, and displaying the summary in the auxiliary reading dialog box; and / or, The large language model is called to perform candidate prompt question summarization processing on the typesetting information, generate candidate prompt questions corresponding to the information carrier, and display the candidate prompt questions in the auxiliary reading dialog box.

6. The method according to claim 4, characterized in that The step of performing content understanding processing on the target text content in response to the appeal dialogue text to obtain a result corresponding to the appeal includes: In response to the appeal dialogue text, identifying a user intention corresponding to the appeal dialogue text; In the case where the user intention represents a user's question demand for the information carrier, determining the text content related to the demand dialogue text in the target text content; Calling a large language model to perform question-answering processing on the relevant text content and the appeal dialogue text, and generating a question-answering result of the appeal dialogue text; The question and answer result is displayed in the auxiliary reading dialog box.

7. The method according to claim 6, characterized in that After determining the relevant text content of the appeal dialogue text in the target text content, the method further includes: The position of the relevant content text in the information carrier is determined, the page number corresponding to the position is found, and the page number and / or the text content corresponding to the page number is displayed in the auxiliary reading dialog box.

8. The method according to claim 4, characterized in that The method further comprises: In response to a translation operation on the information carrier, determining a target language corresponding to the translation operation; The translation engine is called to translate the target text content, generate the target text content corresponding to the target language, and display it in the auxiliary reading dialog box.

9. The method according to claim 1, characterized in that: The obtaining of information carrier comprises: Detect whether there is text content to be processed within the image acquisition range of the image acquisition module of the mobile terminal; When there is text content to be processed within the image acquisition range, calling the image acquisition module of the mobile terminal to acquire an image of the text content to be processed to obtain an image; Performing correction processing on the image to obtain a corrected image, and using the corrected image as an information carrier of the text content to be processed; The extracting the text content to be processed in the information carrier includes: A text recognition method for the image type of the information carrier is determined, and the text content to be processed in the information carrier is obtained by using the text recognition method.

10. The method according to claim 9, characterized in that After performing format conversion processing on the text content to be processed to obtain target text content, the method further includes: Extracting the typesetting information in the target text content, and calling the large language model to perform summary processing on the typesetting information to generate a summary corresponding to the information carrier; Calling a large language model to perform candidate prompt question summarization processing on the typesetting information to generate candidate prompt questions corresponding to the information carrier; Displaying an auxiliary reading dialog box, displaying the summary on a summary interface of the auxiliary reading dialog box, and displaying the candidate prompt questions on a question-and-answer interface of the auxiliary reading dialog box; The obtaining of the user's demand dialogue text includes: In response to an interface switching operation on the auxiliary reading dialog box, switching from the summary interface to the question-and-answer interface; In the question-and-answer interface, the user's demand dialogue text is obtained.

11. The method according to claim 10, characterized in that The extracting the typesetting information in the target text content includes: Detecting whether the language corresponding to the target text content is a preset language; When the language is a preset language, extracting typesetting information from the target text content; In the case where the language is not a preset language, calling a translation engine to translate the target text content to generate target text content corresponding to the preset language; Extract the layout information from the target text content in the preset language.

12. The method according to claim 10, characterized in that The step of performing content understanding processing on the target text content in response to the appeal dialogue text to obtain a result corresponding to the appeal includes: In response to the appeal dialogue text, identifying a user intention corresponding to the appeal dialogue text; In the case where the user intention represents a user's question demand for the information carrier, determining the text content related to the demand dialogue text in the target text content; The large language model is called to perform question-answering processing on the relevant text content and the appeal dialogue text, generate a question-answering result of the appeal dialogue text, and display the question-answering result in the question-answering interface.

13. The method according to claim 12, characterized in that After determining the relevant text content of the appeal dialogue text in the target text content, the method further includes: Determine the position of the relevant content text in the information carrier, find the page number corresponding to the position, and display the page number and / or the text content corresponding to the page number in the question-and-answer interface.

14. The method according to claim 6 or 12, characterized in that: The determining of the relevant text content of the appeal dialogue text in the target text content includes: Vectorizing the appeal dialogue text to obtain an appeal vector; Performing vectorization processing on the target text content to obtain a carrier vector; According to the matching result between the appeal vector and the carrier vector, the relevant text content of the appeal dialogue text in the target text content is determined.

15. The method according to claim 14, characterized in that The vectorization process of the target text content to obtain a carrier vector includes: Segmenting the target text content according to a preset first granularity to obtain a plurality of first sub-text contents; Performing vectorization processing on each of the first sub-text contents to obtain a first granularity carrier vector; Segmenting the target text content according to a preset second granularity to obtain a plurality of second sub-text contents; Vectorization processing is performed on each of the second text sub-contents to obtain a second granularity carrier vector.

16. The method according to claim 15, characterized in that The step of determining the text content related to the appeal dialogue text in the target text content according to the matching result between the appeal vector and the carrier vector includes: Searching for a first granularity carrier vector matching the demand vector from a plurality of the first granularity carrier vectors; Determine the first subtext content corresponding to the first granularity carrier vector matched by the appeal vector as the relevant text content of the appeal dialogue text in the target text content; or, Searching for a second granularity carrier vector matching the demand vector from a plurality of the second granularity carrier vectors; The second sub-text content corresponding to the second granularity carrier vector matched by the appeal vector is determined as the relevant text content of the appeal dialogue text in the target text content.

17. An information carrier processing device, characterized in that: The device comprises: A carrier acquisition module, used for acquiring information carriers; A content extraction module, used to extract the text content to be processed in the information carrier; A content conversion module, used for performing format conversion processing on the text content to be processed to obtain target text content; The text acquisition module is used to obtain the user's demand dialogue text; The content processing module is used to respond to the appeal dialogue text, perform content understanding processing on the target text content, and obtain a result corresponding to the appeal.

18. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-16 when executing a program stored in a memory.

19. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Images