Question and answer method and device, electronic equipment and storage medium

Abstract

The invention provides a question and answer method and device, electronic equipment and a storage medium, and relates to the field of data processing, in particular to the field of artificial intelligence. According to the specific implementation scheme, initial questioning data sent by a client side are received, the initial questioning data comprise a first image, a second image and a user questioning text, and the second image is generated after a user executes track circling operation in a picture area of the first image; constructing a multi-modal cue word according to the initial question data; according to the multi-modal cue word, in combination with a preset large language model and an agent library, target reply content corresponding to the initial question data is generated, and the agent library comprises a plurality of candidate agents; and sending the target reply content to the client.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and more particularly to the field of artificial intelligence, specifically to a question-answering method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of educational informatization, more and more learning devices and applications are beginning to support "screenshot Q&A" or "photo Q&A" functions. These functions typically process the entire image uploaded by the user and combine it with a question-and-answer model to generate a response. However, these systems struggle to deeply understand the contextual information within the image. For example, in English learning, the same word may have different meanings in different contexts. Traditional screenshot Q&A systems often fail to obtain sufficient contextual information when faced with partial screenshots, thus potentially failing to accurately understand the user's question or provide a precise answer. Summary of the Invention

[0003] This disclosure provides a question-and-answer method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a question-answering method is provided, comprising: receiving initial question data sent by a client, wherein the initial question data includes a first image, a second image, and user question text, the second image being generated by the user performing a trajectory drawing operation within the screen area of ​​the first image; constructing multimodal prompt words based on the initial question data; generating target response content corresponding to the initial question data based on the multimodal prompt words, combined with a preset large language model and an intelligent agent library, wherein the intelligent agent library contains multiple candidate intelligent agents; and sending the target response content to the client.

[0005] According to another aspect of this disclosure, a question-and-answer method is provided, comprising: in response to detecting a user's selection operation on a question-and-answer control, capturing a complete image of the current interface as a first image; triggering the current interface to enter a drawing mode, the drawing mode supporting the user to draw a trajectory on the screen area of ​​the first image; obtaining a second image obtained after the user performs the trajectory drawing operation within the screen area of ​​the first image; and obtaining the user's question text; sending the first image, the second image, and the user's question text as initial question data to the cloud, the cloud being used to perform question parsing based on the initial question data; receiving the target reply content sent by the cloud, and presenting the target reply content in the user session.

[0006] According to another aspect of this disclosure, a question-answering device is provided, comprising: a receiving module for receiving initial question data sent by a client, wherein the initial question data includes a first image, a second image, and user question text, and the second image is generated by the user performing a trajectory drawing operation within the screen area of ​​the first image; a construction module for constructing multimodal prompt words based on the initial question data; a generation module for generating target response content corresponding to the initial question data based on the multimodal prompt words, combined with a preset large language model and an intelligent agent library, wherein the intelligent agent library contains multiple candidate intelligent agents; and a sending module for sending the target response content to the client.

[0007] According to another aspect of this disclosure, a question-and-answer device is provided, comprising: a monitoring module, configured to capture a complete image of the current interface as a first image in response to detecting a user's selection operation on a question-and-answer control; a triggering module, configured to trigger the current interface to enter a drawing mode, wherein the drawing mode supports the user to draw a trajectory on the screen area of ​​the first image; an acquisition module, configured to acquire a second image obtained after the user performs the trajectory drawing operation within the screen area of ​​the first image, and to acquire the user's question text; a sending module, configured to send the first image, the second image, and the user's question text as initial question data to the cloud, wherein the cloud is used to perform question parsing based on the initial question data; and a receiving module, configured to receive the target reply content sent by the cloud and present the target reply content in the user session.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described question-and-answer method.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the above-described question-and-answer method.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described question-and-answer method.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure.

[0013] Figure 1 This is a schematic diagram illustrating an exemplary implementation of a question-and-answer method according to an exemplary embodiment of the present disclosure.

[0014] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a question-and-answer method according to an exemplary embodiment of the present disclosure.

[0015] Figure 3 This is a schematic diagram illustrating an exemplary implementation of a question-and-answer method according to an exemplary embodiment of the present disclosure.

[0016] Figure 4 This is a schematic diagram illustrating an exemplary implementation of a question-and-answer method according to an exemplary embodiment of the present disclosure.

[0017] Figure 5 This is a schematic diagram illustrating an exemplary implementation of a question-and-answer method according to an exemplary embodiment of the present disclosure.

[0018] Figure 6 This is a schematic diagram illustrating the interaction between a client and the cloud in accordance with an exemplary embodiment of the present disclosure of a question-and-answer method.

[0019] Figure 7 This is a schematic diagram of a question-and-answer device according to an exemplary embodiment of the present disclosure.

[0020] Figure 8 This is a schematic diagram of a question-and-answer device according to an exemplary embodiment of the present disclosure.

[0021] Figure 9 This is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] Image processing is the technique of using computers to analyze images to achieve desired results. It is also known as image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by capturing images using equipment such as industrial cameras, video cameras, and scanners. The elements of this array are called pixels, and their values ​​are called grayscale values. Image processing techniques generally include three parts: image compression, enhancement and restoration, and matching, description, and recognition.

[0024] Deep learning (DL) is a new research direction in the field of machine learning (ML), bringing it closer to its original goal—artificial intelligence. Deep learning learns the inherent laws and hierarchical representations of sample data; the information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, capable of recognizing data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies.

[0025] Computer vision is the science of enabling machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting 'information' from images or multidimensional data. Here, 'information' refers to what Shannon defines as information that can be used to help make a 'decision.' Because perception can be seen as extracting information from sensory signals, computer vision can also be viewed as the science of enabling artificial systems to 'perceive' from images or multidimensional data.

[0026] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. AI hardware technologies generally include computer vision, speech recognition, natural language processing, as well as learning / deep learning, big data processing, and knowledge graph technologies.

[0027] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] Figure 1 This is a schematic diagram illustrating an exemplary implementation of a question-answering method shown in this application, which can be applied to the cloud, such as... Figure 1 As shown, this question-and-answer method can be introduced through the following steps S101~S104.

[0029] S101, Receive initial question data sent by the client, wherein the initial question data includes a first image, a second image and user question text, and the second image is generated by the user performing a trajectory drawing operation within the screen area of ​​the first image.

[0030] The first image refers to the original, complete image provided by the user.

[0031] The second image refers to the image generated after the user performs a trajectory drawing operation on the first image.

[0032] The user question text is the text in which the user asks a question about the area circled in the second image.

[0033] In simple terms, the first image can be understood as the background image of the second image. Compared with the first image, the second image has an additional circled trajectory, while the rest is the same. This second image can clearly show the positional relationship between the circled area and the overall first image, which facilitates accurate recognition later.

[0034] For example, suppose the first image is an image containing an English article, and the second image is an image generated after the user circles an English word in the first image. The user's question text might be: "What does this word mean in the article?"

[0035] For example, suppose the first image is an image containing multiple math word problems, and the second image is an image generated after the user circles one of the math word problems in the first image. The user's question text might be: "How do I solve this word problem?"

[0036] The initial query data can be encapsulated by the client and sent to the cloud in the form of a data packet.

[0037] S102, Construct multimodal prompt words based on the initial question data.

[0038] The multimodal prompts can be constructed based on the first image, the second image, and the user's question text. The multimodal prompts will guide the following large language model to understand the user's question.

[0039] S103. Based on the multimodal prompt words, combined with the preset large language model and intelligent agent library, generate the target response content corresponding to the initial question data. The intelligent agent library contains multiple candidate intelligent agents.

[0040] In artificial intelligence, an agent is a system or program capable of independently performing tasks, making decisions, and interacting with its environment or users. In artificial intelligence, an agent typically refers to an automated system with specific functions and tasks, capable of autonomously judging, reasoning, and operating based on input information or environmental changes. Depending on its design and objectives, an agent can focus on different domains or tasks, such as explaining mathematical problems, translating languages, or analyzing data.

[0041] In this application, the agent library contains multiple candidate agents, each corresponding to different functional characteristics. For example, the agent library may contain a large number of candidate agents, such as those for explaining math problems, those for categorizing and organizing incorrect answers, and those for interpreting English text.

[0042] Each candidate agent can be a trained agent that can be directly invoked.

[0043] In some feasible approaches, the user's intent can be understood through multimodal prompts based on a pre-defined large language model, thereby determining the target agent to be invoked from the agent library, and invoking the target agent to generate the target response content corresponding to the initial question data based on the multimodal prompts.

[0044] In some feasible approaches, the user's intent can be understood through multimodal cue words based on a pre-set large language model, and the target response content corresponding to the initial question data can be generated directly based on the user's intent using the pre-set large language model.

[0045] S104, Send the target reply content to the client.

[0046] This application proposes a question-answering method, comprising: receiving initial question data sent by a client, wherein the initial question data includes a first image, a second image, and user question text, and the second image is generated by the user performing a trajectory drawing operation within the image area of ​​the first image; constructing multimodal prompt words based on the initial question data; generating target response content corresponding to the initial question data based on the multimodal prompt words, combined with a preset large language model and an intelligent agent library, wherein the intelligent agent library contains multiple candidate intelligent agents; and sending the target response content to the client. In this application, the user can directly circle the area of ​​interest on the first image to form the second image. When performing question analysis, the first image provides a global perspective, helping to understand the context of the area circled by the user; by using the multimodal prompt word construction method, not only the text of the user's question is analyzed, but also the first and second images are considered, which greatly improves the relevance and accuracy of the answer and avoids misunderstandings or information loss that may be caused by simple text input; since each intelligent agent is independent and trainable, as the intelligent agent library is continuously enriched, the entire question-answering system becomes more flexible and can adapt to diverse user needs.

[0047] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a question-and-answer method shown in this application, such as... Figure 2 As shown, this question-and-answer method can be introduced through the following steps S201~S206.

[0048] S201, Receive initial question data sent by the client, wherein the initial question data includes a first image, a second image and user question text, and the second image is generated by the user performing a trajectory drawing operation within the screen area of ​​the first image.

[0049] For details on the specific implementation of step S201, please refer to the relevant parts of the above embodiments, which will not be repeated here.

[0050] S202, determine the global-local relationship between the first image and the second image.

[0051] The global-local relationship between the first image and the second image is as follows: the first image provides the global context, and the second image locates the core local region to be analyzed.

[0052] S203, based on global-local relationships, generates intermediate prompt words by combining the user's question text.

[0053] Among them, intermediate prompts can be used to describe global and local relationships as well as to describe the user's question text.

[0054] For example, suppose the first image is an image containing an English article, and the second image is an image generated after the user circles an English word in the first image. The user's question text might be: "What does this word mean in the article?" Then the intermediate prompt could be: "The first image provides the global context, the second image locates the core local area to be analyzed, and the user wants to know what the word they circled in the second image means in the article."

[0055] S204, construct multimodal prompts based on the first image, the second image, the user's question text, and intermediate prompts.

[0056] In some implementations, the first image, the second image, the user's question text, and the intermediate prompt words generated above are combined to form multimodal prompt words.

[0057] S205. Based on the multimodal prompt words, combined with the preset large language model and intelligent agent library, generate the target response content corresponding to the initial question data. The intelligent agent library contains multiple candidate intelligent agents.

[0058] S206, Send the target response content to the client.

[0059] For details on the specific implementation of steps S205 to S206, please refer to the relevant parts of the above embodiments, which will not be repeated here.

[0060] In this embodiment, the user can directly circle the area of ​​interest on the first image to form the second image. When analyzing the question, the first image provides a global perspective, which helps to understand the context of the area circled by the user. By using a multimodal prompt word construction method, not only the text of the user's question is analyzed, but also the first and second images are taken into account, which greatly improves the relevance and accuracy of the answer.

[0061] Figure 3 This is a schematic diagram illustrating an exemplary implementation of a question-and-answer method shown in this application, such as... Figure 3 As shown, this question-and-answer method can be introduced through the following steps S301 to S309.

[0062] S301, Receive initial question data sent by the client, wherein the initial question data includes a first image, a second image and user question text, and the second image is generated by the user performing a trajectory drawing operation within the screen area of ​​the first image.

[0063] S302, determine the global-local relationship between the first image and the second image.

[0064] S303 generates intermediate prompt words based on global-local relationships and user question text.

[0065] S304, construct multimodal prompts based on the first image, the second image, the user's question text, and intermediate prompts.

[0066] For details on the specific implementation of steps S301 to S304, please refer to the relevant parts of the above embodiments, which will not be repeated here.

[0067] S305 uses a large language model to identify the intent of multimodal prompts and obtain the question intent.

[0068] In this application, the large language model has an intent recognition function, which can be used to identify the intent of multimodal prompt words and obtain the question intent.

[0069] S306. Based on the question's intent and the functional characteristics of each candidate agent in the agent library, match the target agent from the candidate agents.

[0070] As described above, the agent library contains multiple candidate agents, each corresponding to different functional characteristics. The large language model can match the target agent to be invoked from among the candidate agents based on the question intent and the functional characteristics corresponding to each candidate agent in the agent library.

[0071] There can be one or more target intelligent agents.

[0072] For example, if the user asks the question "What does this word mean in the article?", the final target agent matched will be the candidate agent used for English interpretation.

[0073] For example, if the user's question text is "What does this word mean in the article, and can you give me more examples of sentences containing this word?", the final matched target agents will be the candidate agents for interpreting English and the candidate agents for providing examples of English sentences.

[0074] If there are multiple target agents, multiple target agents can be invoked in parallel according to the questioning intent, or multiple target agents can be invoked sequentially according to the questioning intent.

[0075] S307, If a target agent is matched, the target agent is invoked based on the multimodal prompt words to obtain the target response content.

[0076] It's easy to understand that, generally, different candidate agents correspond to different input parameters. To achieve accurate invocation of the agent, after matching the target agent, the input parameters corresponding to the target agent can be extracted from the multimodal prompts. The input parameters are then input into the target agent to obtain the initial response content output by the target agent. Finally, the target response content is generated based on the initial response content.

[0077] As one feasible approach, the initial response content can be directly used as the target response content.

[0078] As a feasible approach, a large language model can be used to organize and refine the initial response content according to the question's intent, resulting in the target response content. (This is especially important when there are multiple target agents; in such cases, the large language model needs to organize and refine the initial responses output by each agent according to the question's intent to obtain the target response content.) By organizing and optimizing the initial response content using a large language model, the response becomes more fluent and understandable, ensuring that the tone and style of the response meet user needs and providing a more natural and user-friendly interactive experience.

[0079] S308, in response to the failure to match the target agent, outputs the target response content based on multimodal cue words through a large language model.

[0080] If no target agent is matched as described above, the multimodal prompt words are directly input into the large language model to obtain the target response content output by the large language model. In other words, this ensures that even if no target agent is matched as described above, a fallback response content can still be generated and returned to the user, avoiding the situation where the response content is empty.

[0081] S309, Send the target response content to the client.

[0082] In this embodiment, the user can directly circle the area of ​​interest on the first image to form a second image. During question analysis, the first image provides a global perspective, helping to understand the context of the user-circled area. A large language model is used to identify the intent of the multimodal prompts, obtaining the question's intent. Based on the question intent and the functional characteristics of each candidate agent in the agent library, a target agent is matched from the candidate agents. If a target agent is matched, it is invoked based on the multimodal prompts to obtain the target response content. If no target agent is matched, the multimodal prompts are directly input into the large language model to obtain the target response content output by the large language model. That is, it ensures that even if no target agent is matched, a fallback response is generated and returned to the user, avoiding an empty response.

[0083] Figure 4 This is a schematic diagram illustrating an exemplary implementation of a question-and-answer method shown in this application, such as... Figure 4 As shown, this question-and-answer method can be introduced through the following steps S401~S407.

[0084] S401, Receive initial question data sent by the client, wherein the initial question data includes a first image, a second image and user question text, and the second image is generated by the user performing a trajectory drawing operation within the screen area of ​​the first image.

[0085] S402, Construct multimodal prompts based on the initial question data.

[0086] S403: Based on the multimodal prompt words, combined with the preset large language model and intelligent agent library, generate the target response content corresponding to the initial question data. The intelligent agent library contains multiple candidate intelligent agents.

[0087] S404, send the target response content to the client.

[0088] For details on the specific implementation of steps S401 to S404, please refer to the relevant parts of the above embodiments, which will not be repeated here.

[0089] S405, receive the follow-up text and session identifier sent by the client, wherein the session identifier is a unique identifier used to represent the user session.

[0090] Furthermore, it is not difficult to understand that after receiving the target response, users often have the need to ask questions about the current target response. For example, after receiving the meaning of the circled English word in the current article, the user may ask, "What other meanings does this word have in other contexts?" In other words, the question-and-answer method proposed in this application supports users to ask questions in multiple rounds. When the user asks a follow-up question in each round, the method receives the follow-up question text and session identifier sent by the client. The session identifier is a unique identifier used to represent the user's session.

[0091] S406, determine the session context information corresponding to the follow-up text based on the session identifier.

[0092] S407 constructs a new round of multimodal prompt words based on follow-up text and conversation context information, and combines a large language model and intelligent agent library to generate a new round of target response content, and sends the new round of target response content to the client.

[0093] This application's embodiments introduce a multi-round questioning function, allowing users to continue asking questions based on the previous answer, forming a more natural and coherent dialogue process. This makes the question-and-answer process no longer a simple "question and answer," but rather better meets the user's in-depth needs and improves the overall interactive experience.

[0094] Figure 5 This is an exemplary schematic diagram illustrating an implementation of a question-and-answer method shown in this application, such as... Figure 5 As shown, this question-and-answer method is applied to the client side of the terminal device and can be introduced through the following steps S501~S505.

[0095] S501, in response to detecting the user's selection of the question-and-answer control, captures the complete image of the current interface as the first image.

[0096] The terminal devices to which this method can be applied include, but are not limited to, mobile phones, tablets, laptops, wearable devices, and other terminal devices.

[0097] In some implementations, the question-and-answer control proposed in this application can be embedded at the operating system level of the terminal device. That is, this solution is no longer limited to specific question-and-answer applications, but can be applied to third-party applications in all scenarios. Users can use the question-and-answer method proposed in this application to achieve instant Q&A functionality on mainstream third-party learning platforms (such as online question banks, foreign language learning apps, video course software, etc.), making full use of the high-quality educational resources of third-party platforms, breaking the "resource closure" of existing learning devices, and expanding the functional boundaries and usage value of the product.

[0098] In some implementations, the question-and-answer control proposed in this application resides on the edge of the terminal device screen as a "global floating button." Users can trigger the question-and-answer method proposed in this application to provide instant answers by clicking this button on any interface (including third-party applications, video playback interfaces, document reading interfaces, etc.), without needing to exit the current application.

[0099] In some implementations, users can select the question-and-answer control through single-click, double-click, or voice activation.

[0100] Once the user selects the question-and-answer control, the system captures the complete image of the current screen as the first image and generates a corresponding image cache to ensure the real-time and complete extraction of screen content and avoid content loss due to screen refresh.

[0101] The first image can also be understood as the background image of the second image below, which contains all the visual elements of the current interface.

[0102] S502 triggers the current interface to enter the drawing mode, which allows users to draw trajectories on the screen area of ​​the first image.

[0103] After the user selects the question-and-answer control, or after capturing the complete image of the current interface as the first image, the current interface is triggered to enter the drawing mode.

[0104] S503, obtain the second image obtained after the user performs a trajectory drawing operation within the screen area of ​​the first image, and obtain the user's question text.

[0105] When a user is detected drawing a trajectory in the drawing mode, the system records the user's drawing trajectory in real time and obtains a second image after the drawing is completed. This second image, compared to the first image, can be simply understood as having the drawn trajectory added; the rest is identical. The second image clearly shows the positional relationship between the selected area and the overall first image, facilitating accurate subsequent recognition.

[0106] In some implementations, after acquiring the second image, the user can automatically enter a user session interface where the user can directly input their question in text form or in voice form.

[0107] If the user inputs a question in speech, the speech can be converted into a text version of the question using a speech recognition algorithm.

[0108] For example, if a user is reading an English article, after the user selects the question-and-answer control, the entire image of the current screen is captured as the first image, and the current screen enters the drawing mode. The second image is then obtained after the user performs a drawing operation within the area of ​​the first image. For instance, if a specific English word in the first image is drawn, the second image will differ from the first image in that the drawn word is surrounded by additional drawing lines. The user's input question could be, "What does this word mean here?"

[0109] S504, the first image, the second image, and the user's question text are sent to the cloud as initial question data, which the cloud uses to parse the question based on the initial question data.

[0110] In some implementations, the first image, the second image, and the user's question text are used as initial question data and encapsulated into a data packet, which is then sent to the cloud. Encapsulation into a data packet ensures that all data related to the initial question is transmitted together, guaranteeing data consistency. The cloud can obtain all necessary initial question data at once, avoiding misunderstandings or inaccurate answers caused by data loss or disordered transmission.

[0111] The S505 receives the target response content sent from the cloud and displays the target response content in the user's session.

[0112] In this embodiment, by introducing a "circle drawing mode", users can directly circle the area of ​​interest on the first image to form a second image. The first image, the second image, and the user's question text are sent to the cloud as initial question data. When the question is analyzed in the cloud, the first image provides a global perspective, which helps to understand the context of the area circled by the user, ensuring that the question is understood and answered more accurately.

[0113] Furthermore, it's easy to understand that after receiving the target response, users often have a need to ask follow-up questions based on the current response. For example, after receiving the meaning of the circled English word in the current article, a user might ask, "What other meanings might this word have in other contexts?" In other words, the question-and-answer method proposed in this application supports multiple rounds of questioning. Each time a user asks a follow-up question, the method retrieves the text entered by the user in the user's conversation; retrieves the conversation identifier corresponding to the user's conversation (which is unique); sends the follow-up text and the conversation identifier to the cloud for a new round of question parsing. The cloud determines the conversation context information corresponding to the follow-up text based on the conversation identifier, and then generates a new round of target response content based on the follow-up text and conversation context information, sending the new round of target response content to the client; the client receives the new round of target response content sent from the cloud and presents it in the user's conversation. By introducing a multi-round questioning function, users can continue to ask questions based on the previous answer, forming a more natural and coherent dialogue process. This makes the question and answer process no longer a simple "question and answer", but can better meet the user's in-depth needs and improve the overall interactive experience.

[0114] Figure 6 This application illustrates an interaction diagram between a question-and-answer method and the cloud, such as... Figure 6 As shown, if the client on the terminal device detects the user's selection operation on the question and answer control, it captures the complete image of the current interface as the first image, triggers the current interface to enter the drawing mode, obtains the second image obtained after the user performs a trajectory drawing operation within the screen area of ​​the first image, and obtains the user's question text. Then, the first image, the second image and the user's question text are used as initial question data and encapsulated into a data packet, and then sent to the cloud in the form of a data packet.

[0115] The cloud-based system can be divided into a cloud client and a cloud server. The cloud server has a pre-set intelligent agent library. The cloud client receives data packets (containing a first image, a second image, and the user's question text) sent by the client. It then determines the global and local relationships between the first and second images and generates intermediate prompt words based on these relationships and the user's question text. Finally, it constructs multimodal prompt words based on the first image, the second image, the user's question text, and the intermediate prompt words. The cloud client has a pre-set large language model. This model performs intent recognition on the multimodal prompt words to obtain the question intent. Based on the question intent and the functional characteristics of each candidate intelligent agent in the intelligent agent library, it matches a target intelligent agent from the candidate agents. If a target intelligent agent is matched, the target intelligent agent in the cloud server is invoked based on the multimodal prompt words to generate initial response content. This initial response content is then linguistically organized to obtain the target response content. If no target intelligent agent is matched, the multimodal prompt words are directly input into the large language model to obtain the target response content output by the large language model. Finally, the cloud client sends the target response content to the client on the terminal device.

[0116] Figure 7 This is a schematic diagram of a question-and-answer device shown in this application, which is applied in the cloud. The question-and-answer device 700 includes a receiving module 701, a building module 702, a generating module 703, and a sending module 704.

[0117] The receiving module 701 is used to receive initial question data sent by the client. The initial question data includes a first image, a second image, and user question text. The second image is generated by the user performing a trajectory drawing operation within the screen area of ​​the first image.

[0118] Among them, the construction module 702 is used to construct multimodal prompt words based on the initial question data.

[0119] The generation module 703 is used to generate the target response content corresponding to the initial question data based on the multimodal prompt words, combined with the preset large language model and intelligent agent library. The intelligent agent library contains multiple candidate intelligent agents.

[0120] The sending module 704 is used to send the target reply content to the client.

[0121] In this device, users can directly circle the area of ​​interest on the first image to form a second image. When analyzing questions, the first image provides a global perspective, helping to understand the context of the area circled by the user. By using a multimodal prompt word construction method, not only the text of the user's question is analyzed, but also the first and second images are considered, which greatly improves the relevance and accuracy of the answer and avoids misunderstandings or information loss that may be caused by simple text input. Since each agent is independent and trainable, as the agent library is continuously enriched, the entire question-answering system becomes more flexible and can adapt to diverse user needs.

[0122] Furthermore, the construction module 702 is also used to: determine the global-local relationship between the first image and the second image; generate intermediate prompt words based on the global-local relationship and the user's question text; and construct multimodal prompt words based on the first image, the second image, the user's question text, and the intermediate prompt words.

[0123] Furthermore, the generation module 703 is also used to: perform intent recognition on the multimodal prompt words through a large language model to obtain the question intent; based on the question intent and combined with the functional characteristics corresponding to each candidate agent, match the target agent from the candidate agents; and call the target agent according to the multimodal prompt words to obtain the target response content.

[0124] Furthermore, the generation module 703 is also used to: extract the input parameters corresponding to the target agent from the multimodal prompt words; input the input parameters into the target agent to obtain the initial response content output by the target agent; and generate the target response content based on the initial response content.

[0125] Furthermore, the generation module 703 is also used to: use the initial response content as the target response content; or, organize and compose the initial response content according to the question intent using a large language model to obtain the target response content.

[0126] Furthermore, the generation module 703 is also used to: in response to the failure to match the target agent, output the target response content based on multimodal prompt words through a large language model.

[0127] Furthermore, the target intelligent agent can be one or more.

[0128] Furthermore, the receiving module 701 is also used to receive the follow-up question text and session identifier sent by the client, wherein the session identifier is a unique identifier used to represent the user session; the building module 702 is also used to: determine the session context information corresponding to the follow-up question text based on the session identifier; and build a new round of multimodal prompt words based on the follow-up question text and the session context information; the generating module 703 is also used to: generate a new round of target response content based on the new round of multimodal prompt words combined with a large language model and an intelligent agent library; and the sending module 704 is also used to: send the new round of target response content to the client.

[0129] Figure 8 This is a schematic diagram of a question-and-answer device shown in this application, applied to a client on the terminal device side. The question-and-answer device 800 includes a monitoring module 801, a triggering module 802, an acquisition module 803, a sending module 804, and a receiving module 805.

[0130] The monitoring module 801 is used to capture the complete image of the current interface as the first image in response to the detection of the user's selection operation on the question and answer control.

[0131] Among them, the trigger module 802 is used to trigger the current interface to enter the drawing mode. The drawing mode allows users to draw trajectories on the screen area of ​​the first image.

[0132] The acquisition module 803 is used to acquire the second image obtained after the user performs a trajectory drawing operation within the screen area of ​​the first image, and to acquire the user's question text.

[0133] The sending module 804 is used to send the first image, the second image, and the user's question text as initial question data to the cloud, where the cloud uses the initial question data to perform question parsing.

[0134] The receiving module 805 is used to receive the target reply content sent from the cloud and present the target reply content in the user session.

[0135] In this device, by introducing a "circle drawing mode," users can directly circle the area of ​​interest on the first image to form a second image. The first image, the second image, and the user's question text are sent to the cloud as initial question data. When the question is analyzed in the cloud, the first image provides a global perspective, which helps to understand the context of the area circled by the user, ensuring that the question is understood and answered more accurately.

[0136] Furthermore, the sending module 804 is also used to: encapsulate the first image, the second image, and the user's question text as initial question data into a data packet and then send it to the cloud.

[0137] Furthermore, the acquisition module 803 is also used to: acquire the follow-up question text entered by the user in the user session; acquire the session identifier corresponding to the user session, wherein the session identifier is a unique identifier; the sending module 804 is also used to: send the follow-up question text and the session identifier to the cloud for a new round of question parsing; the receiving module 805 is also used to: receive the new round of target response content sent from the cloud, and present the new round of target response content in the user session.

[0138] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0139] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0140] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0141] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0142] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as question-and-answer methods. For example, in some embodiments, the question-and-answer method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the question-and-answer method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform question-and-answer methods by any other suitable means (e.g., by means of firmware).

[0143] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0147] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0148] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0149] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0150] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A question-and-answer method, comprising: Receive initial question data sent by the client, wherein the initial question data includes a first image, a second image and user question text, and the second image is generated by the user performing a trajectory drawing operation within the screen area of ​​the first image; Construct multimodal prompt words based on the initial question data; Based on the multimodal prompt words, combined with a preset large language model and an intelligent agent library, the target response content corresponding to the initial question data is generated, wherein the intelligent agent library contains multiple candidate intelligent agents; The target response content is sent to the client.

2. The method according to claim 1, wherein, The step of constructing multimodal prompt words based on the initial question data includes: Determine the global and local relationships between the first image and the second image; Based on the global-local relationship, intermediate prompt words are generated in conjunction with the user's question text; A multimodal prompt word is constructed based on the first image, the second image, the user's question text, and the intermediate prompt word.

3. The method according to claim 1 or 2, wherein, The step of generating the target response content corresponding to the initial question data based on the multimodal prompt words, combined with a preset large language model and intelligent agent library, includes: The intent of the question is obtained by performing intent recognition on the multimodal prompt words using the large language model. Based on the question intent and the functional characteristics of each candidate agent, the target agent is matched from the candidate agents. The target agent is invoked based on the multimodal prompt words to obtain the target response content.

4. The method according to claim 3, wherein, The step of invoking the target agent based on the multimodal cue words to obtain the target response content includes: Extract the input parameters corresponding to the target agent from the multimodal prompts; The input parameters are input to the target agent to obtain the initial response content output by the target agent; The target response content is generated based on the initial response content.

5. The method according to claim 4, wherein, The process of generating the target response content based on the initial response content includes: Use the initial response content as the target response content; or... The target response content is obtained by organizing and arranging the initial response content according to the question intent using the large language model.

6. The method according to claim 3, wherein, The method further includes: In response to the failure to match the target agent, the target response content is output based on the multimodal cue words through the large language model.

7. The method according to claim 3, wherein, The target intelligent agent can be one or more.

8. The method according to claim 1, wherein, After sending the target response content to the client, the process further includes: Receive follow-up text and session identifier sent by the client, wherein the session identifier is a unique identifier used to represent a user session; The session context information corresponding to the follow-up question text is determined based on the session identifier; Based on the follow-up question text and the conversation context information, a new round of multimodal prompt words is constructed, and combined with the large language model and the intelligent agent library, a new round of target response content is generated, and the new round of target response content is sent to the client.

9. A question-and-answer method, comprising: In response to detecting a user's selection of a question-and-answer control, capture the complete image of the current interface as the first image; The current interface is triggered to enter the drawing mode, which allows the user to draw a trajectory on the screen area of ​​the first image; Obtain the second image obtained after the user performs a trajectory drawing operation within the screen area of ​​the first image, and obtain the user's question text; The first image, the second image, and the user's question text are sent to the cloud as initial question data, and the cloud is used to parse the question based on the initial question data. Receive the target reply content sent from the cloud and present the target reply content in the user session.

10. The method according to claim 9, wherein, The step of sending the first image, the second image, and the user's question text as initial question data to the cloud includes: The first image, the second image, and the user's question text are encapsulated into a data packet as initial question data and then sent to the cloud.

11. The method according to claim 9, wherein, After presenting the target response content to the user, the process further includes: Obtain the follow-up question text entered by the user in the user session; Obtain the session identifier corresponding to the user session, wherein the session identifier is a unique identifier; The follow-up question text and the session identifier are sent to the cloud for a new round of question analysis; Receive the new round of target response content sent from the cloud, and present the new round of target response content in the user session.

12. A question-and-answer device, comprising: The receiving module is used to receive initial question data sent by the client, wherein the initial question data includes a first image, a second image and user question text, and the second image is generated by the user performing a trajectory drawing operation within the screen area of ​​the first image; The construction module is used to construct multimodal prompt words based on the initial question data; The generation module is used to generate the target response content corresponding to the initial question data based on the multimodal prompt words, combined with a preset large language model and an intelligent agent library, wherein the intelligent agent library contains multiple candidate intelligent agents; The sending module is used to send the target reply content to the client.

13. A question-and-answer device, comprising: The monitoring module is used to detect when a user selects a question-and-answer control and capture the complete image of the current interface as the first image. The trigger module is used to trigger the current interface to enter the drawing mode, which allows the user to draw a trajectory on the screen area of ​​the first image. The acquisition module is used to acquire a second image obtained after the user performs a trajectory drawing operation within the screen area of ​​the first image, and to acquire the user's question text; The sending module is used to send the first image, the second image, and the user's question text as initial question data to the cloud, whereby the cloud is used to parse the question based on the initial question data. The receiving module is used to receive the target reply content sent by the cloud and present the target reply content in the user session.

14. An electronic device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8 or 9-11.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8 or 9-11.

16. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8 or 9-11.

Images