Intelligent question and answer interaction method and device

Abstract

The invention relates to the technical field related to artificial intelligence, in particular to an intelligent question and answer interaction method and device. The method comprises: acquiring an original image; extracting text information in the original image; obtaining an answer text input by the user; and based on the text information and a preset large language model module, replying the answer text. Through the setting, according to the scheme provided by the invention, the original image can be identified, and then the text content can be understood and analyzed by the system through the large language model module instead of identifying the text on the image. Therefore, the system can provide more intelligent and humanized interaction experience.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence related technologies, and specifically to an intelligent question-answering interaction method and device. Background Art

[0002] With the development of artificial intelligence technology, text recognition (OCR) has become a very common and important technology for enterprise digitalization. However, the existing image text recognition (OCR) system can generally only directly return the text content recognized on the image, or use rules or information extraction models to further process the recognized content, structure the content, and return the key information on the image in the form of key-value pairs. It does not have the function of question-answering interaction. Summary of the invention

[0003] In view of this, the embodiments of the present application are directed to providing an intelligent question-answering interaction method and device.

[0004] This application provides an intelligent question-answering interactive method, including:

[0005] Get the original image;

[0006] Extracting text information from the original image;

[0007] Get the answer text entered by the user;

[0008] Based on the text information and the preset large language model module, the response text is replied.

[0009] In some embodiments, before extracting the text information in the original image, the method further includes:

[0010] Performing a preprocessing operation on the original image;

[0011] The preprocessing operation includes at least one of an angle correction operation, a noise removal operation, a watermark removal operation and a stamp removal operation.

[0012] In some embodiments, extracting text information from the original image includes:

[0013] Performing text detection on the original image to obtain coordinate information of each text line;

[0014] Cropping the original image based on the coordinate information to obtain a text line image;

[0015] Use a preset neural network model to recognize multiple text line images and obtain the target text information in each text line image.

[0016] In some embodiments, replying to the answer text based on the text information and a preset large language model module includes:

[0017] Generate a prompt word based on the target text information;

[0018] The large language model is used to respond to the response text based on the prompt word and the response text.

[0019] In some embodiments, it also includes:

[0020] Get the prompt word information entered by the user;

[0021] Based on the prompt word information, a prompt word is generated.

[0022] In some embodiments, the large language model also includes a historical conversation storage function for responding to the conversation text based on historical conversations and learned knowledge and prompt words.

[0023] The present application also provides an intelligent question-answering interactive device, comprising:

[0024] An acquisition module, used for acquiring the original image;

[0025] An extraction module, used to extract text information in the original image;

[0026] The acquisition module is also used to obtain the answer text input by the user;

[0027] The reply module is used to reply to the reply text based on the text information and a preset large language model module.

[0028] The present application also provides an electronic device, including:

[0029] A processor, and a memory for storing a program executable by the processor;

[0030] The processor is used to implement the above-mentioned intelligent question-answering interaction method by running the program in the memory.

[0031] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the intelligent question-answering interaction method as described above.

[0032] In the solution provided by the present application, the original image is first obtained; the text information in the original image is extracted; the reply text input by the user is obtained; and the reply text is replied based on the text information and the preset large language model module. With such a setting, the solution provided by the present application can recognize the original image, and then through the large language model module, the system can understand and analyze the text content, rather than just recognize the text on the image. This enables the system to provide a more intelligent and humanized interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0034] Figure 1 It is a flowchart of an intelligent question-answering interaction method provided by an embodiment of the present application.

[0035] Figure 2 It is a partial flow chart of a method provided by an embodiment of the present application.

[0036] Figure 3 It is a structural diagram of an intelligent question-answering interactive device provided by an embodiment of the present application.

[0037] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Application Overview

[0040] With the development of artificial intelligence technology, text recognition (OCR) has become a very common and important technology for enterprise digitalization. However, the existing image text recognition (OCR) system can generally only directly return the text content recognized on the image, or use rules or information extraction models to further process the recognized content, structure the content, and return the key information on the image in the form of key-value pairs. It does not have the function of question-answering interaction.

[0041] To realize the question-and-answer interaction function, another solution is to use a large multimodal model, directly encode the image and question text, and input them into the large model. The large model outputs the answer to realize the end-to-end question-and-answer interaction function.

[0042] The main idea of ​​the existing image text recognition system is to first perform OCR text recognition, and then design a set of rules to extract key information based on business needs and return fixed field information. This approach has obvious disadvantages:

[0043] 1. Poor generalization ability: if the text layout in the image is slightly adjusted or the style changes, the extraction rules may need to be modified.

[0044] Second, the expansion capability is poor. If new key information fields need to be added, rules need to be added. The number of key information fields is proportional to the number of rules. The more key information fields are required, the more rules need to be designed. Moreover, adding new rules may affect existing rules, and the modification cost is high.

[0045] 3. The system cannot truly understand the text content, cannot realize question-answer interaction and intelligent analysis, can only recognize and extract the text content on the image, and cannot answer different questions raised by users.

[0046] Another existing solution is to implement end-to-end question-answering interaction based on a multimodal large model. Although this can make the entire interactive system process simpler, the current multimodal large model technology also has certain shortcomings:

[0047] 1. Large multimodal models require huge computing resources, which general enterprises do not have. Although this patent also requires a large computing power to support the use of a large language model, the computing power required is smaller than that required for a large multimodal model that supports image, text and other modal information.

[0048] Second, the accuracy of multimodal large models in dense and fine-grained perception tasks is not high, and they cannot accurately perceive all the text information on the image, so they cannot give accurate answers, and often give irrelevant answers, which cannot achieve a good interactive experience. However, this patent uses the current more mature OCR algorithm to accurately identify the text information on the image, and then gives the text information to the large language model, which can greatly improve the accuracy of the answer, while reducing the consumption of computing resources, and providing users with a better interactive experience.

[0049] In order to solve the above problems, this patent adds a large language model module to the general image text recognition process, and uses a large language model (LLM) to implement intelligent interactive question and answer. Not only does it no longer need to design a complete set of rules to extract key information, but it also allows flexible setting of prompt words according to different needs. The large language model can extract information based on the fields required by the user, improving the system's generalization and expansion capabilities. In addition, the system is not limited to text information on the image. It can understand and analyze the text content based on the text content on the image and the knowledge learned by the large language model itself, analyze and answer questions raised by users, meet different needs of users, and help users better analyze.

[0050] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0051] Exemplary Methods

[0052] Figure 1 FIG. 1 is a flow chart of an intelligent question-answering interactive method provided by an embodiment of the present application. Figure 1 As shown, the method includes the following contents.

[0053] Step S110, obtaining an original image;

[0054] Step S120, extracting text information in the original image;

[0055] Step S130, obtaining the answer text input by the user;

[0056] Step S140: reply to the answer text based on the text information and the preset large language model module.

[0057] The solution provided in this application can improve the intelligence of interaction: through the large language model module, the system can understand and analyze the text content, not just recognize the text on the image. This enables the system to provide a more intelligent and humanized interactive experience. Enhance the generalization ability of the system: the large language model can handle a variety of different text information. Even if the text layout or style in the image changes, the system can adapt and accurately extract information, reducing the need for rule modification. Improve the scalability of the system: when it is necessary to add key information fields or adjust the interaction logic, the system can achieve this by adjusting the prompt word template or the parameters of the large language model without redesigning complex rule sets, reducing the cost of expanding new functions. Realize multi-round dialogue interaction: the system can conduct context-related multi-round dialogues based on the information stored in historical dialogues, making the interaction more coherent and natural, and improving the user experience. Reduce computing power resource consumption: compared with the multimodal large model, this application uses the OCR algorithm to first accurately identify the text information on the image, and then interacts with the large language model, which can reduce the demand for computing power resources and make the system more efficient. Improve the accuracy of answers: By combining mature OCR technology with large language models, the system can more accurately understand users' questions and give relevant answers, reducing irrelevant answers and improving the accuracy of interactions. Optimize user experience: Users can choose different prompt word templates according to their needs, or flexibly set their own prompt words, so that the system can better meet the personalized needs of users. Support diverse data sources: The system supports multiple data sources such as photos, scanned documents, PDFs, etc., allowing users to easily obtain original images from different sources, improving the applicability and convenience of the system. Through the above process, this patented technology provides an efficient, intelligent, and user-friendly question-and-answer interaction system that can be widely used in situations requiring image text recognition and natural language processing.

[0058] Further, see Figure 2 This patent provides an intelligent question-answering interaction system based on image text recognition and large language model. It combines image text recognition and large language model technology to realize intelligent question-answering interaction function, has multi-round question-answering interaction capability, and provides users with key information extraction, intelligent analysis and other functions. It has the characteristics of strong generalization ability, strong expansion ability and high accuracy.

[0059] The specific steps of this patent are as follows: Figure 2 As shown, it mainly includes three modules: preprocessing module, text recognition module and large language model module.

[0060] In some embodiments, before extracting the text information in the original image, the method further includes: performing a preprocessing operation on the original image; the preprocessing operation includes: at least one of an angle correction operation, a denoising operation, a watermark removal operation, and a stamp removal operation. Specifically, the steps include:

[0061] Step 1: Image import, supporting data sources such as photos, scanned documents, PDF, etc., and converting them into image data uniformly

[0062] The second step is image preprocessing. Due to the diversity and complexity of the imported data, the imported data may be tilted or rotated. To solve this problem, this step first uses a neural network model based on deep learning to identify the angle of the imported image, and corrects the angle of the bank statement image according to the identified angle, thereby obtaining positive image data. The supported angles include 0 degrees, 90 degrees, 180 degrees, and 270 degrees. After obtaining the positive image, the image is preprocessed by denoising, watermark removal, and seal removal to provide a clear and positive image for subsequent recognition, thereby improving the accuracy of text recognition in subsequent processes.

[0063] Furthermore, extracting the text information in the original image includes: performing text detection on the original image to obtain the coordinate information of each text line; cropping the original image based on the coordinate information to obtain a text line image; using a preset neural network model to recognize multiple text line images to obtain the target text information in each text line image. Specifically, it includes the following steps: (refer to Figure 2 Chinese text recognition module: This module implements OCR technology based on deep learning algorithm, using a two-stage algorithm of text detection and text recognition)

[0064] The third step is text detection. This step implements text detection based on the neural network model, detects all text lines in the image, obtains the coordinate information of each text line, and crops the image according to the coordinate information to obtain multiple text line images.

[0065] Step 4: Text recognition: Use a neural network model to implement text recognition, identify multiple text lines in the previous step, and obtain the text information in each text line image.

[0066] Furthermore, based on the text information and the preset large language model module, the reply to the answer text includes: generating prompt words based on the target text information; replying to the answer text based on the prompt words and the answer text through the large language model. The intelligent question-answering interaction method also includes: obtaining prompt word information input by the user; generating prompt words based on the prompt word information. The large language model includes a historical conversation storage function, which is used to reply to the answer text based on historical conversations and the knowledge and prompt words that have been learned. The specific implementation steps are as follows: (refer to Figure 2 Zhongda Language Model Module: This template includes prompt word engineering, historical dialogue repository, and large language model)

[0067] Step 5: Prompt Engineering: This step presets relevant prompt word templates based on the characteristics of the large language model and related techniques of prompt word engineering, such as few-shot learning and chain of thought technology (CoT). Users can choose different prompt word templates according to their own questions or needs, or flexibly set their own prompt words based on actual needs.

[0068] After setting the initial prompt, the system will automatically add the text information recognized by the text recognition module to the prompt.

[0069] Step 6: Based on the information stored in the historical conversation repository, determine whether this interaction is a new conversation, that is, whether it is the first interaction with the large model. The user can also decide whether to start a new conversation. If it is the first interaction with the large language model, the prompt obtained in the previous step is directly sent to the next step. If it is not the first interaction with the large language model, the system will automatically add the historical conversation information to the prompt before entering the next step, so that the large language model can answer according to the historical context, so as to realize the function of multiple rounds of interaction with the large language model.

[0070] Step 7: Send the set prompt to the large language model, and the large language model will give a corresponding answer based on the text content on the image and the knowledge learned by the large language model itself.

[0071] Step 8: Feedback the answer of the large language model to the user, and the user can decide whether to have the next round of conversation according to the needs. If you want to continue the conversation, you can re-enter step 5 to repeat the conversation. At the same time, the system will store the conversation information in the database, save the historical conversation information so that the large language model has the ability to answer according to the historical context, and realize the function of multiple rounds of interaction with the large language model; if there is no need to continue the conversation, the system will end the conversation.

[0072] In summary, in the solution provided by this application, by designing a large language model module, the entire recognition system has the advantages of strong generalization ability and strong scalability, and can truly understand and analyze text content, not limited to text information on images, and use the knowledge learned by the large language model from a large amount of training data to provide users with multiple rounds of intelligent interactive answers. The solution provided by this application uses an OCR algorithm with low computing power consumption and high accuracy, so that the entire interactive system has lower computing power consumption, higher accuracy, and better interactive experience compared to the solution that only uses a multimodal large model.

[0073] Exemplary Devices

[0074] The device embodiments of the present application can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0075] Figure 3 FIG. 1 is a block diagram of an intelligent question-answering interactive device provided by an embodiment of the present application. Figure 3 As shown, the device comprises:

[0076] An acquisition module 31 is used to acquire an original image;

[0077] An extraction module 32, used to extract text information in the original image;

[0078] The acquisition module 31 is also used to acquire the answer text input by the user;

[0079] The reply module 33 is used to reply to the reply text based on the text information and a preset large language model module.

[0080] In some embodiments, before extracting the text information in the original image, the method further includes:

[0081] Performing a preprocessing operation on the original image;

[0082] The preprocessing operation includes at least one of an angle correction operation, a noise removal operation, a watermark removal operation and a stamp removal operation.

[0083] In some embodiments, extracting text information from the original image includes:

[0084] Performing text detection on the original image to obtain coordinate information of each text line;

[0085] Cropping the original image based on the coordinate information to obtain a text line image;

[0086] Use a preset neural network model to recognize multiple text line images and obtain the target text information in each text line image.

[0087] In some embodiments, replying to the answer text based on the text information and a preset large language model module includes:

[0088] Generate a prompt word based on the target text information;

[0089] The large language model is used to respond to the response text based on the prompt word and the response text.

[0090] In some embodiments, it is also used to:

[0091] Get the prompt word information entered by the user;

[0092] Based on the prompt word information, a prompt word is generated.

[0093] In some embodiments, the large language model includes a historical conversation storage function, which is used to respond to the conversation text based on historical conversations and learned knowledge and prompt words.

[0094] Exemplary Electronic Devices

[0095] Below, reference Figure 4 To describe an electronic device according to an embodiment of the present application. Figure 4 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0096] like Figure 4 As shown, electronic device 400 includes one or more processors 410 and memory 420 .

[0097] The processor 410 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0098] The memory 420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 410 may run the program instructions to implement the intelligent question-answering interaction method of each embodiment of the present application described above and / or other desired functions. Various contents such as category correspondences may also be stored in the computer-readable storage medium.

[0099] In one example, the electronic device 400 may further include: an input device 430 and an output device 440 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0100] In addition, the input device 430 may also include, for example, a keyboard, a mouse, an interface, etc. The output device 440 may output various information to the outside, including analysis results, etc. The output device 440 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0101] Of course, to simplify, Figure 4Only some of the components in the electronic device related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0102] Exemplary computer program products and computer-readable storage media

[0103] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the intelligent question-and-answer interaction method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0104] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0105] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the intelligent question-and-answer interaction method according to various embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0106] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0107] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An intelligent question-answering interactive method, characterized in that: include: Get the original image; Extracting text information from the original image; Get the answer text entered by the user; Based on the text information and the preset large language model module, the response text is replied.

2. The intelligent question-answering interactive method according to claim 1, characterized in that: Before extracting the text information in the original image, the method further includes: Performing a preprocessing operation on the original image; The preprocessing operation includes at least one of an angle correction operation, a noise removal operation, a watermark removal operation and a stamp removal operation.

3. The intelligent question-answering interactive method according to claim 1, characterized in that: Extracting text information from the original image, including: Performing text detection on the original image to obtain coordinate information of each text line; Cropping the original image based on the coordinate information to obtain a text line image; Use a preset neural network model to recognize multiple text line images and obtain the target text information in each text line image.

4. The intelligent question-answering interactive method according to claim 3, characterized in that: Replying to the answer text based on the text information and the preset large language model module includes: Generate a prompt word based on the target text information; The large language model is used to respond to the response text based on the prompt word and the response text.

5. The intelligent question-answering interactive method according to claim 4, characterized in that: Also includes: Get the prompt word information entered by the user; Based on the prompt word information, a prompt word is generated.

6. The intelligent question-answering interactive method according to claim 4, characterized in that: Also includes: The large language model includes a historical dialogue storage function, which is used to respond to the dialogue text based on the historical dialogue and the learned knowledge and prompt words.

7. An intelligent question-answering interactive device, characterized in that: include: An acquisition module, used for acquiring the original image; An extraction module, used to extract text information in the original image; The acquisition module is also used to obtain the answer text input by the user; The reply module is used to reply to the reply text based on the text information and a preset large language model module.

8. An electronic device, characterized in that: include: A processor, and a memory for storing a program executable by the processor; The processor is used to implement the intelligent question-answering interaction method according to any one of claims 1 to 6 by running the program in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the intelligent question-answering interaction method according to any one of claims 1 to 6.

Images