Knowledge-based question-and-answer methods, devices, equipment, and storage media

Abstract

This application discloses a knowledge-based question-answering method, apparatus, device, and storage medium. The method includes: acquiring a user-inputted image and related question text; identifying the text information contained in the image; providing the text information as question-answering knowledge to a large language model, enabling the large language model to understand the text information contained in the image and use it as contextual information for the knowledge-based question-answering dialogue; and then inputting the user-inputted question text as a prompt instruction to the large language model, which can understand and respond to the question text based on the aforementioned understood text information contained in the image. This application supports multimodal question input, allowing users to conduct knowledge-based question-answering based on images, thus expanding the application scenarios of knowledge-based question-answering. Furthermore, the solution of this application enables the large language model to understand the text information contained in the image, thereby providing more accurate responses matching the question text.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and more specifically, to a knowledge question answering method, apparatus, device and storage medium. Background Technology

[0002] Knowledge-based question answering refers to the process where a user gives a command to a machine, and the machine provides a corresponding response to that command.

[0003] Traditional question answering typically relies on knowledge bases or knowledge graphs to retrieve user commands, find matching answers within those bases, and output them to the user. In recent years, with the development of artificial intelligence technology, especially the emergence of large language models (LLMs), combining LLMs with question answering can significantly improve accuracy. LLMs are deep neural network models based on the Transformer architecture, possessing superior semantic understanding and multi-turn dialogue capabilities. Common LLM models include GPT-4 and other LLMs developed by various companies.

[0004] However, current knowledge question answering solutions based on large language models are limited to text input to text output or text input to image output, which is too simplistic in terms of the modality of input information and limits the application scenarios of knowledge question answering. Summary of the Invention

[0005] In view of the above problems, this application is proposed to provide a knowledge question-answering method, apparatus, device, and storage medium to realize multimodal question-answering input information, supporting users to use images and text as input for knowledge question-answering, and providing matching response content. The specific solution is as follows:

[0006] Firstly, it provides a knowledge-based question-and-answer method, including:

[0007] Obtain the image input by the user and the question text related to the image, wherein the question text is used to describe the question raised by the user in the current round;

[0008] Identify the text information contained in the image;

[0009] The text information is input as question-answering knowledge into a pre-configured large language model;

[0010] The question text is used as a prompt instruction and input into the large language model to obtain the response content of the large language model to the question text.

[0011] Preferably, before inputting the text information as question-answering knowledge into the large language model, the method further includes:

[0012] Identify the language type of the text information as the source language;

[0013] When the conditions for translating the text information are met, the text information in the source language is translated to obtain the translated text information in the target language, wherein the target language is the language that matches the user.

[0014] The process of inputting the text information as question-answering knowledge into the large language model includes:

[0015] The translated text information is then input into the large language model as question-answering knowledge.

[0016] Preferably, the process of detecting whether the conditions for translating the text information are met includes:

[0017] The large language model is used to identify whether the question text contains a text translation intent. If so, the conditions for translating the text information are met.

[0018] or,

[0019] When the source language is different from the target language matched by the user, the conditions for translating the text information are determined to be met.

[0020] Preferably, the image and the related question text are entered in the same round of question-and-answer interaction, or the image and the related question text are entered in different rounds of question-and-answer interaction, and the image is entered earlier than the question text.

[0021] Preferably, when the intent of the question text related to the image is to translate the content in the image, the response output by the large language model to the question text is the translated text information.

[0022] Preferably, when the intent of the question text related to the image is to restore the translation to the image, the response content output by the large language model to the question text is an edited image of the image, wherein the edited image is an image in which the translated text information is replaced with the original text information according to the corresponding position in the image.

[0023] Preferably, the text information is input as question-answering knowledge into a pre-configured large language model, including:

[0024] Obtain a pre-configured knowledge prompt instruction template, wherein the knowledge prompt instruction template includes an image information slot;

[0025] The text information is filled into the image information slot to obtain the edited knowledge prompt instruction, which is then input into the pre-configured large language model.

[0026] Preferably, obtaining the image input by the user and the question text related to the image includes:

[0027] The question-and-answer interface is displayed, which includes image input controls and text dialog boxes;

[0028] In response to a user's action of uploading an image via the image input control, the system obtains the image input by the user.

[0029] In response to the user's input of question text in the text dialog box, the system obtains the question text related to the image entered by the user.

[0030] Preferably, obtaining the image input by the user includes:

[0031] Obtain the video input by the user, and extract video frame images from the video;

[0032] The identification of text information contained in the image includes:

[0033] Identify the text information contained in each video frame image;

[0034] The step of inputting the text information as question-answering knowledge into a pre-configured large language model includes:

[0035] The timestamp of each video frame and its contained text information are used as input to the pre-configured large language model as question-answering knowledge.

[0036] Secondly, a knowledge question-answering device is provided, comprising:

[0037] The input information acquisition unit is used to acquire the image input by the user and the question text related to the image, wherein the question text is used to describe the question raised by the user in the current round;

[0038] The image recognition unit is used to recognize the text information contained in the image;

[0039] A knowledge input unit is used to input the text information as question-answering knowledge into a pre-configured large language model.

[0040] The question-and-answer interaction unit is used to input the question text as a prompt into the large language model and obtain the response content of the large language model in response to the question text.

[0041] Thirdly, a knowledge question-answering device is provided, including: a memory and a processor;

[0042] The memory is used to store programs;

[0043] The processor is used to execute the program to implement the various steps of the knowledge question-answering method as described above.

[0044] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the various steps of the knowledge question-answering method as described above.

[0045] By employing the above technical solution, the knowledge-based question-answering process of this application supports users inputting questions through text and images. Based on this, the image input by the user is recognized to obtain the text information contained within the image. This text information is then provided as question-answering knowledge to a large language model, enabling the large language model to understand the text information contained in the image and use it as contextual information for the knowledge-based question-answering dialogue. Furthermore, the question text input by the user is input as a prompt to the large language model. Based on the aforementioned understanding of the text information contained in the image, the large language model can understand and respond to the question text, obtaining the answer content. As can be seen, this embodiment supports multimodal question input formats, allowing users to conduct knowledge-based question-answering based on images, thus expanding the application scenarios of knowledge-based question-answering. Moreover, according to the solution of this application, the large language model can understand the text information contained in the image, thereby providing more accurate answers matching the question text. Attached Figure Description

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0047] Figure 1 This application provides a schematic flowchart of a knowledge question answering method.

[0048] Figures 2-7 Several different interactive interfaces for knowledge-based Q&A are illustrated in the diagrams.

[0049] Figure 8 This is a schematic diagram of another knowledge question answering method provided in an embodiment of this application;

[0050] Figure 9 A schematic diagram of a knowledge question-answering device provided in an embodiment of this application;

[0051] Figure 10 This is a schematic diagram of the structure of a knowledge question-answering device provided in an embodiment of this application. Detailed Implementation

[0052] Before introducing the proposed solution, let's first explain the English terms used in this document:

[0053] Prompt: Instructions. When conversing with AI (such as a large language model), you need to send instructions to the AI. These can be a text description, such as "Please recommend a popular song for me" when you talk to the AI, or a parameter description in a certain format, such as describing the relevant drawing parameters to ask the AI ​​to draw a picture in a certain format.

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] This application provides a knowledge question answering solution that can be applied to various knowledge question answering scenarios, such as human-computer dialogue scenarios provided by hardware devices such as intelligent robots, and knowledge question answering scenarios provided by various types of applications.

[0056] The proposed solution can be implemented based on a terminal with data processing capabilities, such as a mobile phone, computer, learning machine, or intelligent robot.

[0057] Next, combined Figure 1 The knowledge-based question-answering method described in this application may include the following steps:

[0058] Step S100: Obtain the image input by the user and the question text related to the image.

[0059] Specifically, in a knowledge-based question-and-answer scenario, users can input questions in the form of images or text. The question text describes the question the user is asking in the current round, and this question can be related to the image.

[0060] In a knowledge-based Q&A session, the user and the machine can interact in multiple rounds. Therefore, the image and question text entered by the user in this step can be entered within the same round of Q&A interaction. Alternatively, the image and question text can be entered in different rounds of Q&A interaction, with the image entered earlier than the question text. That is, the user can enter an image in one round of Q&A interaction and then enter the related question text in the next round.

[0061] Understandably, users can enter different question texts in multiple rounds of question-and-answer interaction for the same input image.

[0062] Reference Figure 2-7 In the first round of question-and-answer interaction, users can simultaneously input a menu image containing English content and the question text: "Please translate this menu content into Chinese." Alternatively, users can input the menu image in the first round and then input the question text: "Please translate this menu content into Chinese" in the second round. This application does not restrict the specific interaction format.

[0063] Further reference Figure 7 It can be seen that, for the same input image, users can engage in multiple rounds of question-and-answer interactions with the machine, such as... Figure 7 In the second round, the user can input a further question: "Help me restore the translated content to the image," and the machine will output the restored image. In the third round, the user can input a further question: "What are the ingredients for smoked beef?" and the machine will output the corresponding response.

[0064] Step S110: Identify the text information contained in the image.

[0065] Specifically, this application can call the corresponding image recognition engine to recognize the text information in the image. For example, it can call the OCR engine to perform OCR recognition on the image and obtain the recognized text information.

[0066] Step S120: Input the text information as question-answering knowledge into the pre-configured large language model.

[0067] Specifically, in order for the large language model to understand the questions posed by users regarding the images, this step involves inputting the text information identified from the images as question-answering knowledge into the large language model. This allows the large language model to understand the text information contained in the images and use it as contextual information for the knowledge-based question-answering dialogue. Consequently, it can better understand the image-related question text input by the user in the current round and subsequent rounds, and provide the correct response.

[0068] In one optional implementation, text information can be used as the Prompt knowledge input for the large language model. Specifically, this application can pre-configure an instruction template for inputting knowledge information into the large language model in the form of a Prompt. The knowledge prompt instruction template may include image information slots for filling in the text information recognized from the image. Based on this, in this step, the text information recognized from the input image can be filled into the image information slots in the knowledge prompt instruction template to obtain the edited knowledge prompt instruction, which is then input into the large language model.

[0069] For example, assuming the text information is: "Napoleon crossed the Alps at the St. Bernard Pass", after filling the image information slot with the text information, the resulting edited knowledge prompt instruction Prompt is: "Please remember the following: This is an image with the following information: Napoleon crossed the Alps at the St. Bernard Pass".

[0070] Step S130: Input the question text as a prompt into the large language model to obtain the response content of the large language model to the question text.

[0071] Specifically, after completing step S120 above, the large language model has understood the text information contained in the image. Based on this, the user-inputted question text can be used as a prompt and input into the large language model to obtain the response content of the large language model to the question text.

[0072] The knowledge-based question-answering method provided in this application supports users inputting questions via text and images. Based on this, the method recognizes the images input by the user, obtains the text information contained within the images, and provides this text information as question-answering knowledge to a large language model. This allows the large language model to understand the text information contained in the images and use it as contextual information for the knowledge-based question-answering dialogue. Furthermore, the user-inputted question text is input as a prompt to the large language model, which can then understand and respond to the question text based on the aforementioned understood text information contained in the images, thus obtaining the answer content. As can be seen, this embodiment supports multimodal question input formats, allowing users to conduct knowledge-based question-answering based on images, expanding the application scenarios of knowledge-based question-answering. Moreover, the solution according to this application enables the large language model to understand the text information contained in images, thereby providing more accurate responses matching the question text.

[0073] Reference Figure 2 As shown, this embodiment provides an optional implementation method for obtaining user-input images and image-related question text.

[0074] This embodiment provides a question-and-answer interface for display to the user. This interface includes an image input control M1 and a text dialog box M2.

[0075] In this embodiment, the system can respond to a user's action of uploading an image via the image input control M1, and obtain the image entered by the user. It can also respond to a user's action of entering question text in the text dialog box M2, and obtain the question text related to the image entered by the user.

[0076] For the image input control M1, a pop-up window appears after the user triggers the control. This pop-up allows the user to select a local image file for upload, or drag and drop the target image into the designated area to complete the upload. After uploading, the image enters the image cache area, as shown below. Figure 3 As shown. At this point, the user can click the "Send" control to upload the image to the machine, completing the image input process. In addition, the user can also enter question text in the text dialog box M2, such as... Figure 4 As shown, the input question text is "Please translate this menu content into Chinese." After entering the question text, you can click the "Send" button to complete the input process for both the image and the question text.

[0077] It should be noted that the knowledge question answering method provided in this embodiment supports multimodal input methods for question input. In addition to the image and text input modes described in the above embodiments, it can also support video input.

[0078] After obtaining the video input by the user, video frame images can be extracted from the video, and then the text information contained in each video frame image can be identified, such as subtitles displayed in the video frame images. The video frame images extracted from the video can be each frame of the video or keyframe images.

[0079] Building upon this foundation, to enable the large language model to learn the text information contained in the video, the text information identified in each video frame can be input as question-answering knowledge into the large language model. Of course, considering that different video frames have different timestamps, and that this timestamp information and text information are related and also constitute a form of knowledge information, the timestamp of each video frame and the text information contained in the video frame can be combined, and the combined information can be input as question-answering knowledge into the large language model.

[0080] In the video input mode, users can ask questions about the input video, such as "Please translate the subtitles in the video" or "Please describe the background of the oil painting in the video".

[0081] In some embodiments of this application, considering that users may prefer different languages ​​in different application regions and scenarios, and that different large language models may use different languages ​​in their training data during the training phase, resulting in varying abilities to understand knowledge in different languages, the knowledge question answering method provided in this embodiment can include a language identification and processing step. Specific steps can be found in [reference needed]. Figure 8 As shown:

[0082] Step S200: Obtain the image input by the user and the question text related to the image.

[0083] Step S210: Identify the text information contained in the image.

[0084] Steps S200-S210 correspond one-to-one with steps S100-S110 in the aforementioned embodiments, and are described in detail above, so they will not be repeated here.

[0085] Step S220: Identify the language type of the text information as the source language.

[0086] Step S230: Check whether the conditions for translating the text information are met. If yes, proceed to steps S240-S250; otherwise, proceed to step S260.

[0087] This application provides several different implementation methods for detecting whether the conditions for translating text information are met, as follows:

[0088] Large language models can be used to identify whether the problem text contains the intention to translate the text. If so, the conditions for translating the text information are met.

[0089] Specifically, the large language model can understand and identify the intent of the user's input question text. If it is determined that the intent of the question text includes the intent to translate text (for example, the question text "Help me translate this menu content into Chinese" mentioned earlier), then it can be determined that the conditions for translating the text information are met, that is, the text information in the source language can be translated into the target language.

[0090] Furthermore, it can determine whether the conditions for translating text information are met when the identified source language is different from the target language matched by the user.

[0091] The target language is the language that matches the user. In this embodiment, the target language that matches the user can be determined in various ways, including but not limited to the following:

[0092] 1. Use the language of the user's input question text as the target language.

[0093] 2. Set the default language as the target language.

[0094] 3. Identify the current user's identity through sensors and query the stored target language corresponding to the current user's identity.

[0095] Step S240: Translate the source language text information to obtain the target language translated text information.

[0096] Specifically, if the large language model has translation capabilities, it can be invoked to translate text information in the source language. In addition, other translation engines can also be invoked to translate text information in the source language.

[0097] Step S250: Input the translated text information as question-answering knowledge into the large language model.

[0098] Step S260: Input the text information as question-answering knowledge into the large language model.

[0099] Step S270: Input the question text as a prompt into the large language model to obtain the response content of the large language model to the question text.

[0100] The knowledge-based question-answering method provided in this embodiment adds a process for identifying the language of the text information. Furthermore, if the conditions for translating the text information are met, the source language text information can be translated to obtain the translated text information in the target language that matches the user. Simultaneously, the translated text information in the target language can be used as question-answering knowledge and input into a large language model, enabling the large language model to better understand the text information contained in the image.

[0101] In another optional implementation, after step S240 above, step S250 can also simultaneously input the translated text information and the untranslated text information into the large language model, so that the large language model can understand the image content based on the text information of the two languages.

[0102] In some embodiments of this application, optional example scenarios of the knowledge question-answering method of this application are described. For example... Figures 4-5 As shown:

[0103] When the user's input of a question text related to an image indicates an intent to translate the content within the image, the corresponding large language model outputs the translated text as the response to the question text. For example... Figure 4 As shown, the user entered the question text "Please translate this menu content into Chinese". Figure 5 The example translates the content of an English menu and outputs the translated menu. (It should be noted that...) Figure 5 This is only a reference example and not a real interactive interface; it only demonstrates the translation result of a portion of the text information in the image. Of course, the original text information before translation can also be output simultaneously.

[0104] like Figure 6 As shown:

[0105] When the user's input of a question text related to an image is intended to restore the translated text to the image, the large language model outputs a response to the question text that is an edited image. This edited image is an image in which the translated text information is replaced with the original text information according to the corresponding position in the image.

[0106] Figure 6 For example, if the user inputs the question text "Help me restore the translated content to the image", the corresponding machine output will be an edited image in which the original English content is replaced by the translated Chinese content.

[0107] It's important to note that when the large language model possesses image editing capabilities, image editing can be performed directly through it. When the large language model lacks image editing capabilities, it can output instructions to the system, which then responds to these instructions and invokes a third-party image editing engine to perform the image editing operation. The large language model can understand the user's intent in the question text, parse it into image editing instructions that the image editing engine can recognize, and then send these instructions to the image editing engine through the system. The engine can then perform image editing according to these instructions, producing the edited image, which is ultimately displayed to the user. Image editing is not limited to... Figure 6 In addition to the example of editing text within an image, users can also issue editing instructions for other elements in the image via question text, including but not limited to: deleting, adding, moving, and resizing elements in the image.

[0108] Of course, the above Figure 4-5 , Figure 6 Only a few optional scenarios for image-based knowledge answering are illustrated. In addition, users can also input other types of questions, which are not exhaustive in this embodiment.

[0109] The knowledge question answering device provided in the embodiments of this application is described below. The knowledge question answering device described below can be referred to in correspondence with the knowledge question answering method described above.

[0110] See Figure 9 , Figure 9 This is a schematic diagram of the structure of a knowledge question-answering device disclosed in an embodiment of this application.

[0111] like Figure 9 As shown, the device may include:

[0112] The input information acquisition unit 11 is used to acquire the image input by the user and the question text related to the image, wherein the question text is used to describe the question raised by the user in the current round;

[0113] Image recognition unit 12 is used to recognize text information contained in the image;

[0114] Knowledge input unit 13 is used to input the text information as question-answering knowledge into a pre-configured large language model;

[0115] The question-and-answer interaction unit 14 is used to input the question text as a prompt into the large language model and obtain the response content of the large language model to the question text.

[0116] Optionally, the apparatus of this application may further include:

[0117] A source language identification unit is used to identify the language type of the text information as the source language.

[0118] A translation unit is used to translate the text information in the source language when the conditions for translating the text information are met, to obtain the translated text information in the target language, wherein the target language is a language that matches the user.

[0119] The corresponding knowledge input unit is specifically used to input the translated text information as question-answering knowledge into the large language model.

[0120] Optionally, the process by which the translation unit detects whether the conditions for translating the text information are met may include:

[0121] The large language model is used to identify whether the question text contains a text translation intent. If so, the conditions for translating the text information are met.

[0122] or,

[0123] When the source language is different from the target language matched by the user, the conditions for translating the text information are determined to be met.

[0124] Optionally, when the intent of the image-related question text acquired by the input information acquisition unit is to translate the content in the image, the response content output by the large language model to the question text is the translated text information.

[0125] Optionally, when the intent of the image-related question text acquired by the input information acquisition unit is to restore the translation to the image, the response content output by the large language model to the question text is an edited image of the image. The edited image is an image in which the translated text information is replaced with the original text information according to the corresponding position in the image.

[0126] Optionally, the process by which the knowledge input unit inputs the text information as question-answering knowledge into the pre-configured large language model may include:

[0127] Obtain a pre-configured knowledge prompt instruction template, wherein the knowledge prompt instruction template includes an image information slot;

[0128] The text information is filled into the image information slot to obtain the edited knowledge prompt instruction, which is then input into the pre-configured large language model.

[0129] Optionally, the process by which the input information acquisition unit acquires the user-inputted image and related question text may include:

[0130] The question-and-answer interface is displayed, which includes image input controls and text dialog boxes;

[0131] In response to a user's action of uploading an image via the image input control, the system obtains the image input by the user.

[0132] In response to the user's input of question text in the text dialog box, the system obtains the question text related to the image entered by the user.

[0133] Alternatively, the process by which the input information acquisition unit acquires the image input by the user may include: acquiring the video input by the user and extracting video frame images from the video; then, the process by which the image recognition unit identifies the text information contained in the image may include: identifying the text information contained in each video frame image; then, the process by which the knowledge input unit inputs the text information as question-answering knowledge into the pre-configured large language model may include: inputting the timestamp of each video frame image and its contained text information as question-answering knowledge into the pre-configured large language model.

[0134] The knowledge-based question-answering device provided in this application embodiment can be applied to knowledge-based question-answering devices, such as mobile phones, computers, learning machines, and intelligent robots. Optionally, Figure 10 The hardware structure block diagram of the knowledge question answering device is shown below. Figure 10 The hardware structure of a knowledge-based question-and-answer device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0135] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0136] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0137] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0138] The memory stores a program, which the processor can call. The program is used for:

[0139] Obtain the image input by the user and the question text related to the image, wherein the question text is used to describe the question raised by the user in the current round;

[0140] Identify the text information contained in the image;

[0141] The text information is input as question-answering knowledge into a pre-configured large language model;

[0142] The question text is used as a prompt instruction and input into the large language model to obtain the response content of the large language model to the question text.

[0143] Optionally, the refined and extended functions of the program can be found in the description above.

[0144] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0145] Obtain the image input by the user and the question text related to the image, wherein the question text is used to describe the question raised by the user in the current round;

[0146] Identify the text information contained in the image;

[0147] The text information is input as question-answering knowledge into a pre-configured large language model;

[0148] The question text is used as a prompt instruction and input into the large language model to obtain the response content of the large language model to the question text.

[0149] Optionally, the refined and extended functions of the program can be found in the description above.

[0150] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0151] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0152] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A knowledge-based question-and-answer method, characterized in that, include: Obtain the image input by the user and the question text related to the image, wherein the question text is used to describe the question raised by the user in the current round; Identify the text information contained in the image; Identify the language type of the text information as the source language; The large language model is used to identify whether the question text contains a text translation intent. If so, it is determined that the conditions for translating the text information are met; or, when the source language is different from the target language matched by the user, it is determined that the conditions for translating the text information are met. When the conditions for translating the text information are met, the text information in the source language is translated to obtain the translated text information in the target language, wherein the target language is the language that matches the user. The translated text information is used as question-and-answer knowledge and is input into a pre-configured large language model in the form of a prompt instruction. This enables the large language model to understand the text information contained in the image, use it as context information for the knowledge-based question-and-answer dialogue, assist in understanding the image-related question text input by the user in the current round and subsequent rounds, and provide the correct response. The question text is used as a prompt instruction and input into the large language model to obtain the response content of the large language model to the question text.

2. The method according to claim 1, characterized in that, The image and the related question text are entered in the same round of question-and-answer interaction, or the image and the related question text are entered in different rounds of question-and-answer interaction, and the image is entered earlier than the question text.

3. The method according to claim 1, characterized in that, When the intent of the question text related to the image is to translate the content in the image, the response output by the large language model to the question text is the translated text information.

4. The method according to claim 1, characterized in that, When the intent of the question text related to the image is to restore the translation to the image, the response output by the large language model to the question text is an edited image of the image, which is an image in which the translated text information is replaced with the original text information according to the corresponding position in the image.

5. The method according to claim 1, characterized in that, The text information is input as question-answering knowledge into a pre-configured large language model, including: Obtain a pre-configured knowledge prompt instruction template, wherein the knowledge prompt instruction template includes an image information slot; The text information is filled into the image information slot to obtain the edited knowledge prompt instruction, which is then input into the pre-configured large language model.

6. The method according to any one of claims 1-5, characterized in that, The process of obtaining the user-input image and related question text includes: The question-and-answer interface is displayed, which includes image input controls and text dialog boxes; In response to a user's action of uploading an image via the image input control, the system obtains the image input by the user. In response to the user's input of question text in the text dialog box, the system obtains the question text related to the image entered by the user.

7. The method according to any one of claims 1-5, characterized in that, The process of obtaining the image input by the user includes: Obtain the video input by the user, and extract video frame images from the video; The identification of text information contained in the image includes: Identify the text information contained in each video frame image; The step of inputting the text information as question-answering knowledge into a pre-configured large language model includes: The timestamp of each video frame and its contained text information are used as input to the pre-configured large language model as question-answering knowledge.

8. A knowledge-based question-and-answer device, characterized in that, include: The input information acquisition unit is used to acquire the image input by the user and the question text related to the image, wherein the question text is used to describe the question raised by the user in the current round; The image recognition unit is used to recognize the text information contained in the image; The knowledge input unit is used to identify the language type of the text information as the source language; it uses a large language model to identify whether the question text contains a text translation intention, and if so, it determines that the conditions for translating the text information are met; or, when the source language is different from the target language matched by the user, it determines that the conditions for translating the text information are met. When the conditions for translating the text information are met, the text information in the source language is translated to obtain the translated text information in the target language, wherein the target language is the language that matches the user. The translated text information is used as question-and-answer knowledge and is input into a pre-configured large language model in the form of a prompt instruction. This enables the large language model to understand the text information contained in the image, use it as context information for the knowledge-based question-and-answer dialogue, assist in understanding the image-related question text input by the user in the current round and subsequent rounds, and provide the correct response. The question-and-answer interaction unit is used to input the question text as a prompt into the large language model and obtain the response content of the large language model in response to the question text.

9. A knowledge-based question-and-answer device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the knowledge question-answering method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the knowledge question-answering method as described in any one of claims 1 to 7.

Images