Intelligent question answering method, system, electronic device and readable storage medium

Abstract

The present application discloses an intelligent question-answering method, system, electronic device and readable storage medium, the method comprising: obtaining a question to be answered and a multimodal file associated with the question to be answered; performing intent recognition on the question to be answered, and determining a target processing task corresponding to the question to be answered; calling a target multimodal large model or target code tool that matches the target processing task, processing the multimodal file based on the target processing task, and generating an answer to the question to be answered.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an intelligent question-answering method, system, electronic device, and readable storage medium. Background Art

[0002] The intelligent question-answering system integrates natural language processing technology and information retrieval technology. Users usually ask questions in the form of voice or text, and submit them to the question-answering robot for recognition and response. For professional fields, such as government affairs, by pre-setting corresponding questions and answers in the knowledge base of the intelligent question-answering system, extracting keywords from user questions, matching questions and answers based on rules, and responding to users' questions about local policies and government affairs in a human-computer interactive way.

[0003] The current intelligent question-answering system has achieved remarkable results in answering general questions, but it is difficult to fully mine and effectively apply information from multimodal data, and cannot provide effective answers to questions in certain professional fields. For example, it can extract text from images based on optical character recognition (OCR) technology but cannot analyze the information of the image, and cannot perform statistical analysis on the data. Summary of the invention

[0004] The embodiments of the present application provide an intelligent question-answering method, system, electronic device, and readable storage medium, which can solve the problem of difficulty in fully mining and effectively applying information in multimodal data, and the inability to provide effective answers to questions in certain professional fields.

[0005] In a first aspect, an embodiment of the present application provides an intelligent question-answering method, which includes: obtaining a question to be answered and a multimodal file associated with the question to be answered; performing intent recognition on the question to be answered, and determining a target processing task corresponding to the question to be answered; calling a target multimodal large model or target code tool that matches the target processing task, processing the multimodal file based on the target processing task, and generating an answer to the question to be answered.

[0006] In the second aspect, an embodiment of the present application provides an intelligent question-answering system, which includes: an acquisition module for acquiring questions to be answered and multimodal files associated with the questions to be answered; a multimodal agent processing module for performing intent recognition on the questions to be answered and determining a target processing task corresponding to the questions to be answered; and calling a target multimodal large model or target code tool that matches the target processing task, processing the multimodal file based on the target processing task, and generating an answer to the question to be answered.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises a program or instructions, and when the program or instructions are executed, the steps of the method described in the first aspect are implemented.

[0011] In an embodiment of the present application, by obtaining a question to be answered and a multimodal file associated with the question to be answered, the intent of the question to be answered is identified, the target processing task corresponding to the question to be answered is determined, the target multimodal big model or target code tool matching the target processing task is called, the multimodal file is processed based on the target processing task, and the answer to the question to be answered is generated. The intention of the question to be answered is realized, and the calling of the target multimodal big model or target code tool to process the multimodal file is supported. The target multimodal big model or target code tool matches the target processing task corresponding to the question to be answered, so that the multimodal file can be more fully understood and analyzed, and a more accurate answer can be generated to answer the question to be answered. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of an intelligent question-answering method provided in an embodiment of the present application;

[0013] Figure 2 It is a flowchart of another intelligent question-answering method provided in an embodiment of the present application;

[0014] Figure 3a It is a schematic diagram of the answer to an intelligent question-answering method provided in an embodiment of the present application;

[0015] Figure 3b is a schematic diagram of an answer to another intelligent question-answering method provided in an embodiment of the present application;

[0016] Figure 4 is a structural diagram of an intelligent question-answering system provided in an embodiment of the present application;

[0017] Figure 5 It is a flowchart of another intelligent question-answering method provided in an embodiment of the present application;

[0018] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0021] The following, in conjunction with the accompanying drawings, describes in detail the intelligent question-answering method, system, electronic device, and readable storage medium provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0022] Figure 1 The flowchart of an intelligent question-answering method provided by an embodiment of the present application is shown, and the method can be executed by an electronic device. Figure 1 , the method may include the following steps.

[0023] Step 102: Obtain questions to be answered and multimodal files associated with the questions to be answered.

[0024] The questions to be answered may be in various forms, such as text, voice, etc. The multimodal files associated with the questions to be answered may include files of various modes, such as text, image, video, audio, etc., and may also include documents of various formats, such as portable document format (PDF), docx format, presentation format (PowerPoint, PPT), table format (excel), Hypertext Markup Language (HTML), json format, etc. The obtained multimodal files may be any combination of the above-mentioned files of various modes and documents of various formats, such as text+image, text+video, text+PDF format document+image, text+docx format document+video, text+excel format document+audio, audio+image, audio+video, etc.

[0025] Step 104: perform intent recognition on the question to be answered and determine the target processing task corresponding to the question to be answered.

[0026] Among them, the intention recognition of the questions to be answered is to identify what kind of processing and analysis to perform on the multimodal files, rather than processing and analyzing the multimodal files. The questions to be answered can be understood and analyzed through a basic large language model to obtain the target processing task corresponding to the questions to be answered.

[0027] Step 106, calling a target multimodal macro model or a target code tool that matches the target processing task, processing the multimodal file based on the target processing task, and generating an answer to the question to be answered.

[0028] For example, if the target processing task is a video file parsing question-answering task, the video-to-text (video2txt) model can be called to process the video file to generate a text description, and answer questions based on the text description. If the target processing task is an audio file parsing question-answering task, the speech recognition model can be called to process the audio file to generate text subtitles, and answer questions based on the text subtitles. If the target processing task is to identify a specific target object in an image, the image recognition model can be called to identify each object in the image and determine the target object from it. If the target processing task is document content analysis, the document parsing retrieval-augmented generation (rag) tool can be called to process and analyze the document. For another example, if the target processing task is to generate a corresponding image or video based on a scene described in text, the text-image generation model or the text-video generation model can be called to generate a corresponding image or video based on the text description. For another example, for a PDF document with mixed text and images, multiple models can be called to combine to complete the target processing task. The target multimodal large model is a model that can process the corresponding target processing task, and the target code tool is a tool that can generate corresponding code for the target processing task to process the target processing task. The above models of different modes can be stored in a model library to facilitate unified management and scheduling of various models.

[0029] In an embodiment of the present application, by obtaining a question to be answered and a multimodal file associated with the question to be answered, the intent of the question to be answered is identified, the target processing task corresponding to the question to be answered is determined, the target multimodal big model or target code tool matching the target processing task is called, the multimodal file is processed based on the target processing task, and the answer to the question to be answered is generated. The intention of the question to be answered is realized, and the calling of the target multimodal big model or target code tool to process the multimodal file is supported. The target multimodal big model or target code tool matches the target processing task corresponding to the question to be answered, so that the multimodal file can be more fully understood and analyzed, and a more accurate answer can be generated to answer the question to be answered.

[0030] In one implementation, the above-mentioned step 106 calls a target multimodal large model or a target code tool that matches the target processing task, processes the multimodal file based on the target processing task, and generates an answer to the question to be answered, which may include the following steps.

[0031] Step 1061, based on the target processing task, determine at least one processing flow corresponding to the target processing task and a target multimodal large model or target code tool corresponding to each of the processing flows.

[0032] Among them, the target processing task may include complex business processes. To achieve the target processing task, it is necessary to implement each processing flow included therein. For each processing flow, it is necessary to call the target multimodal big model or target code tool corresponding to each processing flow. The multimodal big model or target code tool required to process each processing flow may include a combination of multiple multimodal big models or target code tools.

[0033] Step 1062: for each of the processing flows, call the target multimodal macro model or the target code tool to process the multimodal file to generate an answer to the question to be answered.

[0034] In an embodiment of the present application, at least one processing flow required to implement the target processing task is determined according to the target processing task, and the target multimodal big model or target code tool corresponding to each processing flow is determined, so that for each processing flow, the determined target multimodal big model or target code tool is called to process the multimodal file and generate the answer to the question to be answered.

[0035] In an achievable embodiment, the target processing task is to search for images with specific scenes in a PDF document that is a mixture of images and texts. The specific scene is someone having an open-air barbecue on the lawn by the river at dusk. First, the text recognition model is called to recognize the text content in the PDF document, and a description without relevant keywords in the text content is obtained, but there is a description of time and place. Secondly, the code tool is called to generate business logic code to detect the images in the PDF document, and a corresponding text description is generated for each image. Furthermore, the text description corresponding to each image is matched with the description of the specific scene in the target processing task for similarity, and then matched in combination with the contextual text information adjacent to each image, and each image is matched in turn until the matching judgment of all images is completed, and the answer to the question to be answered corresponding to the target processing task is given according to the result of the matching judgment. The above-mentioned processing flows together constitute the target processing task, and the multimodal files are processed in turn to realize each processing flow, so that the target processing task can be realized and the answer to the question to be answered can be generated.

[0036] In one implementation, before calling the target multimodal large model or target code tool that matches the target processing task in the above-mentioned step 106, processing the multimodal file based on the target processing task, and generating the answer to the question to be answered, the method also includes the following steps.

[0037] Step 1051, determining the target professional field of the question to be answered based on the question to be answered and the multimodal documents associated with the question to be answered.

[0038] The target professional field of the question to be answered may be determined directly through the question to be answered, or may be determined jointly through the question to be answered and a multimodal file associated with the question to be answered.

[0039] For example, if the question to be answered directly describes "in a certain field", the target professional field of the question to be answered can be determined. For another example, if the multimodal file associated with the question to be answered is weather-related data, and the question to be answered is to predict the weather on a certain day in the future, the target professional field of the question to be answered is determined to be the field of meteorology. Or if the question to be answered is the impact of weather on the incidence of a certain disease, the target professional field of the question to be answered is determined to be the field of epidemiology.

[0040] Step 1052, determining the target multimodal large model or the target code tool from the model library of the target professional field.

[0041] In an embodiment of the present application, the target professional field of the question to be answered is determined based on the question to be answered and the multimodal file associated with the question to be answered, and the target multimodal big model or the target code tool is determined from the model library of the target professional field. Therefore, when processing the target processing task, a professional target multimodal big model or the target code tool that is more suitable for processing the target processing task is called, which can not only realize the processing of complex target processing tasks, but also improve the processing accuracy of the target processing tasks.

[0042] In another possible embodiment, see Figure 2 , the question to be answered is to predict the growth trend of the gross domestic product in region A. The target processing task corresponding to the question to be answered is to analyze the growth trend of the gross domestic product (GDP) in region A based on the economic data of region A. The target processing task can be decomposed into: reading data, building an ARIMA model, and using the ARIMA model for prediction, and generating an economic development trend analysis based on the prediction results. Among them, the target processing task belongs to the field of data statistical analysis, so the professional data processing big model and code tools are called from the pre-configured data statistical analysis model library, and the data is subjected to professional data statistical analysis that cannot be achieved by the basic big model through the data processing big model and code tools corresponding to each processing flow called.

[0043] In one implementation, before performing intent recognition on the question to be answered and determining the target processing task corresponding to the question to be answered in the above step 104, the method further includes the following steps.

[0044] Step 1031, obtaining multimodal professional data in the target professional field.

[0045] For example, in government-related fields, multimodal professional data includes officially released data in various forms such as text, images, video, voice, and mixed text and graphics. It may also include multimodal professional data in the target professional field obtained from other trusted channels.

[0046] Step 1032: extracting each entity object corresponding to the multimodal professional data and the relationship between the entity objects through a multimodal feature extraction model.

[0047] Among them, the multimodal feature extraction model is used to extract features from multimodal professional data to obtain various entity objects in the multimodal professional data, such as time, place, people, etc., as well as the relationships between entity objects, such as the relationship between time and place, the relationship between time and people, etc.

[0048] Step 1033: construct a professional knowledge graph based on the entity objects and the relationships between the entity objects.

[0049] Among them, the professional knowledge graph presents various entity objects and the relationships between entity objects, so that for a certain entity object, entity objects related to the entity object and the relationships between them can be found from the professional knowledge graph to facilitate understanding of the entity object.

[0050] Step 1034, storing the professional knowledge graph in a professional knowledge base.

[0051] Optionally, the professional knowledge graph in the professional knowledge base is stored in the form of vectors, and vectorization tools can be used to uniformly search the professional knowledge graph in the professional knowledge base.

[0052] In an embodiment of the present application, multimodal professional data of a target professional field is acquired, and entity objects corresponding to the multimodal professional data and the relationships between the entity objects are extracted through a multimodal feature extraction model. A professional knowledge graph is constructed based on the entity objects and the relationships between the entity objects, and the professional knowledge graph is then stored in a professional knowledge base, thereby realizing the construction of a professional knowledge base of the target professional field for retrieving and acquiring data in the target professional field.

[0053] In one implementation, the above step 104 performs intent recognition on the question to be answered and determines the target processing task corresponding to the question to be answered, including the following steps.

[0054] Step 1041 , identifying the question to be answered, obtaining the target entity object corresponding to the question to be answered and the relationship between the target entity objects.

[0055] The target entity objects corresponding to the questions to be answered are objects with actual meanings, such as time, place, person, etc. The relationship between the target entity objects is obtained to facilitate understanding of the meaning of the questions to be answered.

[0056] Step 1042, searching for a target professional knowledge graph from the professional knowledge base; wherein the target professional knowledge graph is related to the target entity object or the relationship between the target entity objects.

[0057] Among them, according to the identification of the question to be answered, a target professional knowledge graph related to the question to be answered is retrieved from the professional knowledge base, and the target professional knowledge graph is related to the target entity object or the relationship between the target entity objects, which may include: the similarity between the entity object in the target professional knowledge graph and the target entity object is greater than a first preset threshold, and may also include that the similarity between the relationship between the entity objects and the relationship between the target entity objects is greater than a second preset threshold. With the help of the target professional knowledge graph, the target entity objects in the question to be answered and the relationship between the target entity objects are more deeply understood.

[0058] Step 1043: confirm or modify the target entity object or the relationship between the target entity objects according to the target professional knowledge graph.

[0059] The questions to be answered may be texts with correct or incorrect expressions. Since the target professional knowledge graph has high information accuracy, the target entity objects or the relationship between target entity objects can be confirmed or corrected based on the pre-built professional knowledge graph to improve the correctness of understanding the questions to be answered.

[0060] Step 1044, determining the target processing task corresponding to the question to be answered based on the confirmed or revised question to be answered.

[0061] In an embodiment of the present application, by identifying the question to be answered, the target entity object corresponding to the question to be answered and the relationship between the target entity objects are obtained, and then the target professional knowledge graph related to the target entity object or the relationship between the target entity objects is searched from the professional knowledge base, so that the target entity object or the relationship between the target entity objects is confirmed or corrected according to the target professional knowledge graph, and then according to the confirmed or corrected question to be answered, the target processing task corresponding to the question to be answered is determined, thereby improving the accuracy of intention recognition of the question to be answered.

[0062] In one implementation, after performing intent recognition on the question to be answered and determining the target processing task corresponding to the question to be answered in the above step 104, the method further includes: searching for the answer to the question to be answered from the professional knowledge base based on the target processing task.

[0063] In an embodiment of the present application, after determining the target processing task corresponding to the question to be answered, based on the target processing task, the answer to the question to be answered is searched from the professional knowledge base. When the answer to the question to be answered can be found from the professional knowledge base, that is, according to the entity objects in each professional knowledge graph in the professional knowledge base and the relationship between the entity objects, the question to be answered can be answered and the answer to the question to be answered is generated. When the answer to the question to be answered cannot be found from the professional knowledge base, that is, the question to be answered cannot be answered through the entity objects in each professional knowledge graph and the relationship between the entity objects, and the target processing task corresponding to the question to be answered includes a complex processing flow, the target processing task is decomposed into at least one processing flow and the target multimodal large model or target code tool corresponding to each processing flow is determined, so as to call the target multimodal large model or target code tool to process the multimodal file and generate the answer to the question to be answered. It is realized that when the professional knowledge graph based on the professional knowledge base can answer the question to be answered, the answer to the question to be answered is searched from the professional knowledge base; when the professional knowledge graph based on the professional knowledge base cannot answer the question to be answered, the answer to the question to be answered is generated by processing the multimodal file by calling the target multimodal large model or target code tool matching the target processing task. It can not only select the appropriate way to answer the question to be answered according to the complex situation of the target processing task corresponding to the question to be answered, but also improve the efficiency and accuracy of answering the question to be answered.

[0064] Optionally, when the above-mentioned professional knowledge base, target multimodal big model or target code tool are unable to answer the question to be answered, you can call a search engine to search the Internet for information related to the question to be answered, and call a document parsing big model or code tool to organize and analyze the relevant information for user reference.

[0065] In one implementation, the output form of the generated answer to the question to be answered can also be multimodal, such as a combination of text, image, video, audio, etc. Figure 3a and Figure 3bBased on the question to be answered, "Generate a statistical chart and analysis report based on the annual regional GDP data of area A from 2001 to 2022", the intention of the question to be answered is recognized, and based on the target processing task corresponding to the question to be answered, the target multimodal large model or target code tool matching the target processing task is called to process the provided data to generate the following: Figure 3a and Figure 3b The statistical charts and text-based analysis reports shown in the figure can be used to generate statistical charts and analysis reports (such as Figure 3a ), or generate a bar graph and analysis report (as shown Figure 3b as shown).

[0066] Figure 4 A schematic diagram of the structure of an intelligent question-answering system provided in an embodiment of the present application is shown. Figure 4 The system includes an acquisition module 41 and a multimodal agent processing module 42, and the multimodal agent processing module 42 integrates at least one multimodal large model or code tool. The multimodal large model or code tool may include a speech recognition large model, a picture and text generation large model, a video analysis large model, a search engine, etc.

[0067] Among them, the acquisition module 41 is used to obtain the questions to be answered and the multimodal files associated with the questions to be answered; the multimodal agent processing module 42 is used to identify the intention of the questions to be answered, determine the target processing tasks corresponding to the questions to be answered; and call the target multimodal large model or target code tool that matches the target processing task, process the multimodal files based on the target processing task, and generate answers to the questions to be answered.

[0068] The target multimodal large model is one or more of the at least one multimodal large model mentioned above. The target code tool is one or more of the at least one code tool mentioned above.

[0069] In one implementation, the multimodal agent processing module 42 calls the target multimodal big model or target code tool that matches the target processing task, processes the multimodal file based on the target processing task, and generates the answer to the question to be answered, which may include: according to the target processing task, determining at least one processing flow corresponding to the target processing task and the target multimodal big model or target code tool corresponding to each processing flow; for each processing flow, calling the target multimodal big model or the target code tool to process the multimodal file to generate the answer to the question to be answered.

[0070] In one implementation, the multimodal agent processing module 42 can also be used to determine the target professional field of the question to be answered based on the question to be answered and the multimodal files associated with the question to be answered; and determine the target multimodal large model or the target code tool from the model library of the target professional field.

[0071] In one implementation, the multimodal intelligent agent processing module 42 can also be used to obtain multimodal professional data in the target professional field; extract the entity objects corresponding to the multimodal professional data and the relationships between the entity objects through a multimodal feature extraction model; construct a professional knowledge map based on the entity objects and the relationships between the entity objects; and store the professional knowledge map in a professional knowledge base.

[0072] In one implementation, the multimodal intelligent agent processing module 42 performs intent recognition on the question to be answered and determines the target processing task corresponding to the question to be answered, which may include: identifying the question to be answered, obtaining the target entity object corresponding to the question to be answered and the relationship between the target entity objects; searching for a target professional knowledge graph from the professional knowledge base; wherein the target professional knowledge graph is related to the target entity object or the relationship between the target entity objects; confirming or correcting the target entity object or the relationship between the target entity objects according to the target professional knowledge graph; and determining the target processing task corresponding to the question to be answered based on the confirmed or corrected question to be answered.

[0073] In one implementation, the multimodal agent processing module 42 may also be used to search for answers to the questions to be answered from the professional knowledge base based on the target processing task.

[0074] Figure 5 FIG. 1 shows a flow chart of another intelligent question-answering method provided in another achievable embodiment of the present application, see Figure 5 , the method may include the following steps.

[0075] Step 501, obtaining questions to be answered and table documents associated with the questions to be answered.

[0076] Among them, the table document associated with the question to be answered is a table document of the GDP of area A over the years. The question to be answered is: Which year has the highest GDP growth rate in area A over the years and give the value.

[0077] Step 502: Use a large language model to identify the intent of the question to be answered and determine the target processing task corresponding to the question to be answered.

[0078] Among them, a large language model is used to segment the questions to be answered, identify entity objects and the relationship between entity objects, obtain the intention of the questions to be answered, and determine the target processing task corresponding to the questions to be answered.

[0079] Step 503, calling the code tool that matches the target processing task to process the table document and generate a standardized answer text.

[0080] Among them, the standardized answer text is: The year with the highest GDP growth rate in Area A was 2011, with a growth rate of 19.58%.

[0081] In an embodiment of the present application, a large language model is used to identify the intent of the question to be answered, determine the target processing task, and call a code tool that matches the target processing task to process the table document, such as reading the table document, identifying row attributes and column attributes, calculating the gross domestic product growth rate, comparing values, etc., to determine the year with the highest GDP growth rate in area A over the years and advance the corresponding GDP growth rate, thereby determining the answer to the question to be answered, and realizing the calling of the code tool to perform professional calculations and analysis on the problems in this professional field.

[0082] The embodiment of the present application also provides an electronic device, which is used to execute the above-mentioned intelligent question-answering method. Figure 6 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. Electronic devices may have relatively large differences due to different configurations or performances, and may include a processor (processor) 601, a communication interface (CommunicationsInterface) 602, a memory (memory) 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 can call a computer program stored in the memory 603 and can be run on the processor 601 to execute the various steps of the above-mentioned intelligent question-answering method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0083] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0084] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0085] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0086] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned intelligent question and answer method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0087] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0088] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned intelligent question and answer method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0089] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0090] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes a program or instructions. When the program or instructions are executed, the various processes of the above-mentioned intelligent question and answer method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0091] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0093] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An intelligent question-answering method, characterized in that: include: Obtaining questions to be answered and multimodal files associated with the questions to be answered; Performing intent recognition on the question to be answered, and determining a target processing task corresponding to the question to be answered; Calling a target multimodal macro model or target code tool that matches the target processing task, processing the multimodal file based on the target processing task, and generating an answer to the question to be answered; The calling of the target multimodal big model or target code tool that matches the target processing task, processing the multimodal file based on the target processing task, and generating the answer to the question to be answered includes: determining, according to the target processing task, at least one processing flow corresponding to the target processing task and the target multimodal big model or target code tool corresponding to each processing flow; for each processing flow, calling the target multimodal big model or the target code tool to process the multimodal file and generating the answer to the question to be answered; Wherein, before calling the target multimodal big model or target code tool matching the target processing task, processing the multimodal file based on the target processing task, and generating the answer to the question to be answered, the method further includes: determining the target professional field of the question to be answered according to the question to be answered and the multimodal file associated with the question to be answered; determining the target multimodal big model or the target code tool from the model library of the target professional field; Wherein, before performing intent recognition on the question to be answered and determining the target processing task corresponding to the question to be answered, the method further includes: obtaining multimodal professional data of the target professional field; extracting each entity object corresponding to the multimodal professional data and the relationship between the entity objects through a multimodal feature extraction model; constructing a professional knowledge graph according to the each entity object and the relationship between the entity objects; and storing the professional knowledge graph in a professional knowledge base; Among them, the intention recognition of the question to be answered and the determination of the target processing task corresponding to the question to be answered include: identifying the question to be answered, obtaining the target entity object corresponding to the question to be answered and the relationship between the target entity objects; searching for a target professional knowledge graph from the professional knowledge base; wherein the target professional knowledge graph is related to the target entity object or the relationship between the target entity objects; confirming or correcting the target entity object or the relationship between the target entity objects according to the target professional knowledge graph; and determining the target processing task corresponding to the question to be answered according to the confirmed or corrected question to be answered.

2. The method according to claim 1, characterized in that After performing intent recognition on the question to be answered and determining the target processing task corresponding to the question to be answered, the method further includes: Based on the target processing task, the answer to the question to be answered is searched from the professional knowledge base.

3. An intelligent question-answering system, characterized in that: include: An acquisition module, used for acquiring questions to be answered and multimodal files associated with the questions to be answered; A multimodal agent processing module, used to identify the intention of the question to be answered and determine the target processing task corresponding to the question to be answered; and calling a target multimodal big model or target code tool that matches the target processing task, processing the multimodal file based on the target processing task, and generating an answer to the question to be answered; wherein the multimodal agent processing module integrates at least one multimodal big model or code tool, the target multimodal big model is one or more of the at least one multimodal big model, and the target code tool is one or more of the at least one code tool; The multimodal agent processing module is used to call the target multimodal large model or target code tool that matches the target processing task, and process the multimodal file based on the target processing task to generate the answer to the question to be answered, including: determining at least one processing flow corresponding to the target processing task and the target multimodal large model or target code tool corresponding to each processing flow according to the target processing task; for each processing flow, calling the target multimodal large model or the target code tool to process the multimodal file to generate the answer to the question to be answered; The multimodal agent processing module 42 is further used to determine the target professional field of the question to be answered according to the question to be answered and the multimodal file associated with the question to be answered; determine the target multimodal large model or the target code tool from the model library of the target professional field; The multimodal agent processing module is further used to obtain multimodal professional data of the target professional field; extract the entity objects corresponding to the multimodal professional data and the relationship between the entity objects through the multimodal feature extraction model; construct a professional knowledge map according to the entity objects and the relationship between the entity objects; and store the professional knowledge map in the professional knowledge base; Among them, the multimodal intelligent agent processing module is used to identify the intention of the question to be answered and determine the target processing task corresponding to the question to be answered, including: identifying the question to be answered, obtaining the target entity object corresponding to the question to be answered and the relationship between the target entity objects; searching for a target professional knowledge graph from the professional knowledge base; wherein the target professional knowledge graph is related to the target entity object or the relationship between the target entity objects; confirming or correcting the target entity object or the relationship between the target entity objects according to the target professional knowledge graph; and determining the target processing task corresponding to the question to be answered based on the confirmed or corrected question to be answered.

4. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the intelligent question-answering method as described in any one of claims 1 to 2.

5. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the intelligent question-answering method as described in any one of claims 1 to 2 are implemented.

6. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes a program or instructions, which, when executed, implement the steps of the intelligent question-answering method as described in any one of claims 1 to 2.

Images