Human-computer interaction method, computing device, server, storage medium and program product

Abstract

The application provides a human-computer interaction method, a computing device, a server, a storage medium and a program product. The method relates to the field of artificial intelligence, and comprises the following steps: searching according to input query information to obtain a search result including information of a webpage related to the query information; according to the search result, an area image matched with the query information is intercepted from the webpage, and text information of the webpage is obtained, so that deep understanding and intelligent retrieval of the content of the webpage are realized; not only text information related to the query information can be searched, but also an area image matched with the query information can be intercepted from the webpage; further, reply information of the query information is generated according to the query information, the area image and the text information of the webpage; the intuitive visual information provided by the area image matched with the query information and the text information of the webpage are combined to generate the reply information, so that the quality of the generated reply information can be improved, the user query can be responded to with higher quality, and the user question can be better answered.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a human-computer interaction method, computing device, server, storage medium, and program product. Background Technology

[0002] With the rapid development of internet technology, the way people obtain information has undergone tremendous changes. However, facing a massive amount of information, finding the desired content quickly and accurately has become a challenge. Currently, the development of Large Language Models (LLMs) has brought new possibilities to this end. For a user's question, an artificial intelligence model generates an answer based on the search results by searching for relevant information.

[0003] However, for questions that require a combination of real-time data and visual recognition capabilities, such as "What is the profile picture that OpenAI uses on Twitter?", traditional search engines often struggle to provide search results that can directly and effectively answer the question. This results in low-quality responses generated by AI models that fail to adequately address user questions. Summary of the Invention

[0004] This application provides a human-computer interaction method, computing device, server, storage medium, and program product to solve the problem that the quality of responses generated by artificial intelligence models is low and they cannot answer user questions well.

[0005] Firstly, this application provides a human-computer interaction method, including:

[0006] The system performs a search based on the input query information to obtain search results, which include information about web pages related to the query information.

[0007] Based on the search results, extract the region image that matches the query information from the webpage, and obtain the text information of the webpage;

[0008] Based on the query information, the regional image, and the text information of the webpage, a response to the query information is generated.

[0009] Secondly, this application provides a computing device, characterized in that it comprises:

[0010] Memory and processor;

[0011] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the methods provided by any of the foregoing aspects.

[0012] Thirdly, this application provides a server, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the server to perform the methods provided in any of the foregoing aspects.

[0013] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method provided in any of the foregoing aspects.

[0014] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in any of the foregoing aspects.

[0015] This application provides a human-computer interaction method, computing device, server, storage medium, and program product. The method searches based on input query information to obtain search results, including information from web pages related to the query information. Based on the search results, it extracts an image of a region from the web page that matches the query information and obtains the text information of the web page, achieving deep understanding and intelligent retrieval of the web page content. This not only retrieves text information related to the query information but also extracts an image of a region from the web page that matches the query information. Furthermore, based on the query information, the image of the region, and the text information of the web page, it generates a response to the query. By combining the intuitive visual information provided by the image of the region matching the query information with the text information of the web page to generate the response, the quality of the generated response can be improved, providing a higher-quality response to user queries and thus better answering user questions. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] Figure 1 A schematic diagram of an example system architecture to which this application applies;

[0018] Figure 2 A flowchart illustrating a human-computer interaction method provided in an exemplary embodiment of this application;

[0019] Figure 3 A flowchart illustrating a human-computer interaction method provided in another exemplary embodiment of this application;

[0020] Figure 4 Example diagram of region image extraction provided for an exemplary embodiment of this application;

[0021] Figure 5A flowchart for obtaining a webpage screenshot provided as an exemplary embodiment of this application;

[0022] Figure 6 A flowchart illustrating a human-computer interaction method provided in an exemplary embodiment of this application;

[0023] Figure 7 This is a structural block diagram of a computing device according to an embodiment of this application;

[0024] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of this application.

[0025] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0028] First, let me explain the terms used in this application:

[0029] Intelligent agent: In computer science, an agent is an entity that possesses autonomy, responsiveness, sociality, and initiative. It can be a software program or a hardware device used to perform specific tasks.

[0030] Multimodal: refers to technologies or systems that can process and integrate multiple types / modal data or information (e.g., text, audio, video, etc.).

[0031] Screenshot: An image representing a webpage rendered using HTML.

[0032] Web actions: Performing actions on a webpage such as clicking links, scrolling through pages, and filling in information.

[0033] Large models refer to deep learning models with a massive number of parameters, typically containing hundreds of millions, tens of billions, or even trillions of parameters. Large models are also known as foundation models (FM), which are pre-trained on large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. These models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and Multi-modal Pre-training Models.

[0034] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0035] This application provides a human-computer interaction method that searches based on input query information to obtain search results including web pages related to the query information. Based on the search results, it extracts an image of a region from the web page that matches the query information and obtains the text information of the web page, achieving deep understanding and intelligent retrieval of the web page content. This method not only retrieves text information related to the query information but also extracts an image of a region from the web page that matches the query information. Furthermore, based on the query information, the image of the region, and the text information of the web page, it generates a response to the query. Combining the intuitive visual information provided by the image of the region matching the query information with the text information of the web page to generate the response improves the quality of the generated response, providing a higher-quality response to user queries and thus better answering user questions.

[0036] Figure 1 This is a schematic diagram of an example system architecture to which this application applies. Figure 1As shown, the system architecture includes a server and endpoint devices. The server and endpoint devices have a communication link, enabling communication between them.

[0037] In this context, a server is a computing device deployed in the cloud or locally, such as a cloud cluster. The server is responsible for generating corresponding responses based on the user's input query.

[0038] Edge devices can be electronic devices that run downstream human-computer interaction tasks. Specifically, they can be hardware devices with network communication, computing, and information display functions, including but not limited to smartphones, tablets, desktop computers, local servers, and cloud servers. For example, edge devices can be client devices that run human-computer interaction systems such as intelligent customer service, intelligent robots, and knowledge-based question-and-answer systems.

[0039] Users input query information through their endpoint devices, and the endpoint devices send the user-input query information to the server.

[0040] The server receives query information from the client device, performs a search based on the query information, and obtains search results, including information about web pages related to the query information. Based on the search results, it extracts an image of a region from the web page that matches the query information and obtains the text information of the web page. Based on the query information, the region image, and the text information of the web page, it generates a response to the query. Furthermore, the server returns the response to the query to the client device. The client device then outputs the response to the query to the user.

[0041] In one example scenario, the client device runs the client of the intelligent question-answering system, and the server is the cloud server of the intelligent question-answering system. Users ask questions through the client device, such as "What is the profile picture that organization A uses on platform B?" The client device then sends the user's question to the server.

[0042] Based on the user's question, if the server determines that a search is required to answer the current user's question, it searches for relevant web pages. Further, based on the search results, the server extracts an image of the region matching the question from the web page and obtains the web page's text information, achieving deep understanding and intelligent retrieval of the web page content. This allows it to not only search for text information related to the question but also extract an image of the region matching the question. Finally, based on the question, the region image, and the web page's text information, the server generates an answer to the question. The server returns the answer to the question to the client device. The client device then outputs the answer to the question to the user.

[0043] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0044] Figure 2 This is a flowchart illustrating a human-computer interaction method provided in an exemplary embodiment of this application. The execution entity in this embodiment is the server in the aforementioned system architecture. Figure 2 As shown, the specific steps of this method are as follows:

[0045] Step S201: Search according to the input query information to obtain search results, which include information about web pages related to the query information.

[0046] The input query information refers to the user's input information (i.e., Query), which is usually natural language text.

[0047] In this embodiment, before generating a response to the query, a search is performed based on the query to obtain search results for generating the response. These search results include information about at least one webpage related to the query. The search results include, but are not limited to, the following information about the webpage: Uniform Resource Locator (URL), title, and summary. The search results may also include information such as the webpage's date and publisher; these can be set according to actual application requirements and are not specifically limited here.

[0048] For example, the server can invoke a search engine based on the query information, causing the search engine to perform a search based on the query information and return the search results to the server.

[0049] Optionally, before performing a search based on the query information, the server can rewrite the query information to obtain the rewritten query information (called the rewritten query). Based on the rewritten query, the search engine is invoked to perform a search and obtain the search results. By rewriting the query information, the search engine can better understand the user's search intent and return more accurate and relevant results, thereby improving the search engine's recall and accuracy.

[0050] Optionally, by rewriting the query information, one or more rewritten queries related to the input query information but with different expressions can be generated. These rewritten queries can supplement the input query information. Searching based on both the input query information and the rewritten queries yields richer and more accurate search results.

[0051] It should be noted that in practical applications, query rewriting schemes can combine various methods and techniques to meet the search requirements of different application scenarios and needs. Methods and techniques for query rewriting include, but are not limited to: synonym replacement, hyponym / hypernym replacement, query expansion, query error correction, query generalization or refinement, and query reorganization. In this embodiment, any query rewriting scheme can be used to rewrite the input query information; no specific limitation is made here.

[0052] Furthermore, the search engine used in this embodiment can be any third-party search engine or a self-developed search engine; no specific limitation is made here. Exemplarily, the server sends a search request to the search engine, which includes the input query information and / or a modified query. In response to receiving the search request, the search engine performs a search based on the input query information and / or the modified query, and returns the search results to the server. The server receives the search results returned by the search engine.

[0053] Step S202: Based on the search results, extract the region image from the webpage that matches the query information and obtain the text information of the webpage.

[0054] In this step, the server extracts the text information of the web pages related to the query information based on the information in the search results. The search results include the web page's URL, title, and summary.

[0055] Specifically, the server sends an HTTP (Hypertext Transfer Protocol) request to retrieve webpage data based on the webpage's URL. Webpage data refers to the data of the webpage itself, that is, the data object representing a webpage. The webpage data is parsed to obtain the text contained within the webpage, resulting in the webpage's content text. The webpage's text information includes the webpage's content text. Optionally, the webpage's text information may include the webpage's content text, as well as the webpage's title and summary.

[0056] Furthermore, rendering the webpage data into a visual page yields the rendered webpage. Taking a screenshot of the rendered page yields a webpage screenshot. Performing semantic understanding and analysis on the screenshot extracts the target region (object box) associated with the query information, and cropping an image of the target region from the webpage yields the image of the region matching the query information.

[0057] Step S203: Generate response information for the query based on the query information, the region image, and the text information of the webpage.

[0058] In this step, the query information, the image of the area on the webpage that matches the query information, and the text information of the webpage are input into the intelligent interaction model. The intelligent interaction model then generates a response to the query based on the query information, the image of the area on the webpage that matches the query information, and the text information of the webpage.

[0059] Among them, the intelligent interaction model can be the human-computer interaction model of the human-computer interaction system. Specifically, it can be various large language models (LLMs) with multimodal input capabilities, which can receive text and image input. No specific limitation is made here.

[0060] For example, the intelligent interaction model can be deployed and run on a server. The server inputs the query information, the image of the area in the webpage that matches the query information, and the text information of the webpage into the intelligent interaction model. The intelligent interaction model then generates a response to the query based on the query information, the image of the area in the webpage that matches the query information, and the text information of the webpage.

[0061] For example, the intelligent interaction model can be deployed and run on other cloud servers. The server of the human-computer interaction system sends a request to the cloud server to invoke the intelligent interaction model. This request includes query information, an image of the region on the webpage that matches the query information, and the text information of the webpage. In response to the request, the cloud server inputs the query information, the image of the region on the webpage that matches the query information, and the text information of the webpage into the intelligent interaction model. Based on the query information, the intelligent interaction model generates a response to the query and returns the response to the server of the human-computer interaction system. The server of the human-computer interaction system receives the response returned by the cloud server.

[0062] In this embodiment, a search is performed based on the input query information to obtain search results including information from web pages related to the query information. Based on the search results, an image of a region matching the query information is extracted from the web page, and the text information of the web page is obtained, achieving a deep understanding and intelligent retrieval of the web page content. This not only allows for the search of text information related to the query information but also the extraction of an image of a region on the web page that matches the query information. Furthermore, based on the query information, the image of the region, and the text information of the web page, a response to the query information is generated. By combining the intuitive visual information provided by the image of the region matching the query information with the text information of the web page to generate the response information, the quality of the generated response information can be improved, thereby providing a higher quality response to user queries and better answering user questions.

[0063] Figure 3This is a flowchart illustrating a human-computer interaction method provided in another exemplary embodiment of this application. Based on the foregoing embodiments, in an optional embodiment, after receiving input query information, the server can determine whether a search is required to respond to the current query information. If a search is required to respond to the current query information, a search is performed based on the query information to obtain search results. A region image matching the query information is extracted from the web pages included in the search results, and the text information of the web pages is obtained. Based on the query information, the region image, and the text information of the web pages, a response to the query information is generated. If a search is not required to respond to the query information, a response to the query information is directly generated through an intelligent interaction model.

[0064] Specifically, such as Figure 3 As shown, the specific steps of this method are as follows:

[0065] Step S301: Obtain the input query information.

[0066] The input query information refers to the user's input information (i.e., the query), which is usually natural language text. For example, a typical user query would be "What is the profile picture OpenAI uses on Twitter?".

[0067] Step S302: Determine whether the query information needs to be searched.

[0068] In this embodiment, after receiving the input query information, the server first determines whether the current query information needs to be searched.

[0069] Specifically, the server inputs the query information into the search decision model, which then determines whether the query information needs to be searched.

[0070] Optionally, in multi-turn dialogue scenarios, the query information input in the current turn may contain historical dialogue information. Historical dialogue information refers to the interaction information between the user and the human-computer interaction system in previous turns of the current session, including the historical queries input by the user in each historical turn and the historical responses received. In this step, the query information input in the current turn and the historical dialogue information are input together into the search decision model. The search decision model determines whether the response to the query information input in the current turn needs to be searched based on the query information input in the current turn and the historical dialogue information, that is, whether the response information generated in the current turn needs to be searched.

[0071] In this step, if it is determined that a search is required to respond to the query information, steps S303-S305 are executed. A search is performed based on the query information. Based on the search results, an image of the region that matches the query information is extracted from the web pages related to the query information, and the text information of the web page is obtained. Then, based on the query information, the region image, and the text information of the web page, the response information to the query information is generated.

[0072] If it is determined that no search is needed to answer the query information, then step S306 is executed to directly generate the answer information for the query information through the intelligent interaction model.

[0073] In this embodiment, the search decision model used is a trained classification model or a model with similar functionality; any classification model can be used. The search decision model can perform classification prediction based on the input query information. There are two possible classification prediction results: one is that a search is required, and the other is that a search is not required.

[0074] The training data used to train the search decision model can include user history dialogues and annotation information. The annotation information indicates the search decision results corresponding to the user's history dialogues, specifically whether a search is required for the response to the last input query in the user's history dialogues. By training the classification model using the training data, the search decision model can be obtained.

[0075] For example, user history dialogues from the training data are input into a classification model. The model predicts whether a search is needed for the response to the last input query in the user's history dialogue, thus obtaining a classification prediction result. Based on the classification prediction result and the annotation information of the user's history dialogues, a cross-entropy loss function is calculated. The parameters of the classification model are then updated through backpropagation based on the cross-entropy loss function value. Once training is complete, a search decision model is obtained.

[0076] The training strategy used to train the search and judgment model, including but not limited to the selected optimization algorithm and learning rate, can be configured and adjusted according to the actual application scenario, and no specific limitations are made here.

[0077] Step S303: If it is determined that the query information needs to be searched, then the search is performed based on the query information to obtain the search results.

[0078] If it is determined that a search is required to respond to the query, a search is performed based on the query to obtain search results for generating the response, which include information from at least one webpage related to the query.

[0079] The search results include, but are not limited to, the following information about the webpage: URL, title, and summary. Search results may also include information such as the webpage's date and publisher; these can be set according to actual application needs and are not specifically limited here.

[0080] In this step, the search results are obtained based on the query information. For the specific implementation principle, please refer to the relevant content of step S201 above, which will not be repeated here.

[0081] Step S304: Based on the search results, extract the region image that matches the query information from the webpage and obtain the text information of the webpage.

[0082] After obtaining search results related to the query information, the server extracts the text information of the relevant web pages based on the information in the search results. The search results include the web page's URL, title, and summary.

[0083] In this embodiment, when extracting text information from a webpage, the server can send an HTTP (Hypertext Transfer Protocol) request based on the webpage's URL to obtain the webpage data. The server parses the webpage data to obtain the text content contained in the webpage, thus obtaining the webpage's content text. Optionally, the server can cache the content text corresponding to each URL. When the content text of a webpage with the same URL needs to be extracted subsequently, the content text corresponding to the URL can be directly read from the cache, which can improve the efficiency of extracting webpage text information, thereby improving the efficiency and responsiveness of human-computer interaction.

[0084] Optionally, when extracting the text information of a webpage, the server can also send a webpage parsing request to the webpage parsing service. This request contains the URLs of the various webpages included in the search results. In response to the request, the webpage parsing service parses the corresponding webpage based on its URL, obtains the webpage's content text, and returns the content text to the server. The server receives the webpage content text returned by the webpage parsing service.

[0085] The server uses the content text of the webpage as the text information of the webpage. Optionally, the server uses the content text of the webpage, as well as the title and summary of the webpage, as the text information of the webpage; that is, the text information of the webpage may include the content text of the webpage, as well as the title and summary of the webpage.

[0086] In this embodiment, when extracting an image of a region matching the query information from a webpage, the server can obtain a screenshot of the webpage based on the search results. The query information and the webpage screenshot are then input into a Vision Language Model (VLM), which determines the target region in the webpage screenshot that matches the query information. The server then extracts an image of the target region from the webpage screenshot, thus obtaining the image of the region matching the query information.

[0087] Specifically, the server can send an HTTP request to retrieve webpage data based on the URL of the webpage included in the search results. Rendering this webpage data into a visual page yields the rendered webpage. Taking a screenshot of this rendered page yields a webpage screenshot.

[0088] Furthermore, the query information and webpage screenshot are input into the Visual Language Model (VLM) for semantic analysis and understanding, extracting the target region in the webpage screenshot that is associated with the query information. The image within the target region in the webpage screenshot that is associated with the query information contains the content and information needed to generate the response to the query.

[0089] The Visual Language Model (VLM) can be any pre-trained VLM. The query information is inserted into the corresponding position of a pre-defined region-locating prompt to obtain text prompts. These text prompts guide the VLM in extracting the target region associated with the query information from the input webpage screenshot. The text prompts and webpage screenshots are then combined... Figure 1 The input is a visual language model (VLM) for inference and prediction, and the output is the location information of the target area in the webpage screenshot that is associated with the query information.

[0090] For example, the region location prompt could be "Given this [query], help me extract the region related to the given image", where [query] is the corresponding location of the query information to be inserted.

[0091] For example, if the input query is "What is OpenAI's Twitter profile picture?", the search results could include the URL of OpenAI's Twitter page (such as its homepage or official website) that contains OpenAI's profile picture. Based on this URL, the server can obtain a screenshot of the page (e.g., ...). Figure 4 (As shown in the image). Further, the query information and a screenshot of the webpage will be... Figure 1 The input VLM is used for inference and prediction, and the output is the location of the target area in the webpage screenshot that is associated with the query "What is the profile picture of OpenAI Twitter?", such as... Figure 4The target area shown is the area where the OpenAI avatar is located.

[0092] Furthermore, based on the location information of the target area in the webpage screenshot that is associated with the query information, the server can extract the area image of the target area from the webpage screenshot to obtain the area image that matches the query information.

[0093] It should be noted that the Visual Language Model (VLM) can be deployed and run on the server of the human-computer interaction system. Optionally, the VLM can be deployed and run on other cloud servers. The server of the human-computer interaction system inserts the query information into the corresponding position of the preset area location prompt to obtain text prompt information. Based on the text prompt information and a screenshot of the webpage, a request to invoke the VLM is sent to the cloud server. This request includes the text prompt information and the screenshot of the webpage. The cloud server responds to the request by retrieving the text prompt information and the screenshot of the webpage. Figure 1 The system takes a visual language model (VLM) as input for inference and prediction, outputs the location information of the target area in the webpage screenshot that is associated with the query information, and returns the location information of the target area in the webpage screenshot that is associated with the query information to the server of the human-computer interaction system.

[0094] Step S305: Input the query information, regional image, and webpage text information into the intelligent interaction model. The intelligent interaction model generates a response to the query information based on the query information, regional image, and webpage text information.

[0095] In this step, the server generates a response / prompt message based on the query information, the region image, and the text information of the webpage. This response / prompt message is used to prompt the intelligent interaction model to generate a response to the query based on the region image and the text information of the webpage. The prompt message is then input into the intelligent interaction model, which generates the response to the query based on the prompt message.

[0096] For example, the query information and the text information of the webpage are inserted into the corresponding positions in a preset response generation prompt to obtain response generation prompt information. The response generation prompt information is used to instruct the intelligent interaction model to generate response information for the query information based on given reference data. The given reference data includes the text information of the webpage in the response generation prompt information and the image input to the model.

[0097] For example, the response generation prompt can be:

[0098] "You are an AI assistant. Please generate an answer based on the given question, referring to the provided knowledge base and images."

[0099] #Given question: [Query information]

[0100] #knowledge base

[0101] Please keep the following materials in mind; they may be helpful in answering the question.

[0102] "[Text information of the webpage]" refers to the location where the query information is to be inserted, and the text information of the webpage is to be inserted. The given image is a screenshot of the area input into the intelligent interaction model along with the response generation prompt.

[0103] The response will generate a prompt message and area screenshot. Figure 1 The system inputs an intelligent interactive model to perform reasoning, generating and outputting responses to queries.

[0104] It should be noted that the intelligent interaction model can be deployed and run on the server of the human-computer interaction system. Optionally, the intelligent interaction model can also be deployed and run on other cloud servers. The server of the human-computer interaction system inserts the query information and the text information of the webpage into the corresponding positions in the preset response generation prompt, obtains the response generation prompt information, and sends a request to the cloud server to invoke the intelligent interaction model based on the response generation prompt information and the area screenshot. This request includes the response generation prompt information and the area screenshot. The cloud server responds to this request by sending the response generation prompt information and the area screenshot. Figure 1 The system inputs an intelligent interaction model to perform reasoning, generates a response to the query, and returns the response to the server of the human-computer interaction system. The server of the human-computer interaction system then receives the response from the cloud server.

[0105] Step S306: If it is determined that the query information does not need to be searched, the query information is input into the intelligent interaction model, and the intelligent interaction model generates the response information for the query information.

[0106] If it is determined that no search is needed to answer the query, the answer can be generated directly through an intelligent interaction model.

[0107] In this embodiment, upon receiving the input query information, a search determination model adaptively determines whether to perform a search based on the current query information. Only when a search is deemed necessary to respond to the current query information is a search performed to obtain search results. This involves extracting an image of the region matching the query information from the webpage and obtaining the webpage's text information, achieving deep understanding and intelligent retrieval of the webpage content. This not only retrieves text information related to the query information but also extracts an image of the region matching the query information from the webpage. Based on the query information, the region image, and the webpage's text information, a response to the query is generated, improving the quality of the generated response and thus providing a higher-quality response to user queries and better answers to user questions. Conversely, if a search is deemed unnecessary to respond to the query information, no search is required. The intelligent interaction model directly generates the response, reducing unnecessary searches and improving the efficiency and timeliness of human-computer interaction.

[0108] The solution in this embodiment can provide more intelligent and personalized information search and question-and-answer services for different user queries, which can improve the efficiency and accuracy of information search, thereby improving the quality of human-computer interaction.

[0109] Figure 5 This is a flowchart illustrating the process of obtaining a webpage screenshot, provided as an exemplary embodiment of this application. In an optional embodiment, as shown... Figure 5 As shown, the specific process for obtaining screenshots of web pages related to the search results is as follows:

[0110] Step S501: Obtain search results, which include the URLs of web pages related to the query information.

[0111] Step S502: Obtain the rendered page of the webpage based on the webpage's URL.

[0112] In this step, the server retrieves webpage data by sending an HTTP request to the server corresponding to the URL of the webpage contained in the search results. Further, the server renders the webpage data into a visual page, thus obtaining the rendered webpage. The principle behind rendering webpage data into a visual page is the same as the principle of webpage rendering in a browser, and can be implemented using any webpage rendering method, which will not be elaborated here.

[0113] Step S503: Take a screenshot of the rendered page of the webpage to obtain a webpage screenshot.

[0114] After obtaining the rendered page of the webpage, the server captures an image of the rendered page to obtain a screenshot of the webpage.

[0115] Specifically, the server can capture an image of the rendered page that currently falls within the visible area (viewport) based on the set visible area size, thus obtaining a screenshot of the webpage. The visible area (viewport) size can be set and adjusted according to actual application needs, and is not specifically limited here.

[0116] Furthermore, the server performs a scrolling operation on the rendered page to refresh the images that fall into the visible area of ​​the rendered page, and captures the images that fall into the visible area of ​​the currently rendered page to obtain other webpage screenshots.

[0117] Optionally, before performing a scrolling operation on the rendered page, the server can determine whether a scrolling operation is needed based on the size of the rendered page and the size of the viewport (i.e., the viewport). If the size of the rendered page is larger than the viewport size, then a scrolling operation is needed. If the size of the rendered page is smaller than or equal to the viewport size, then a scrolling operation is not needed.

[0118] Optionally, the server can also input the rendered page and the size of the visible area (i.e., viewport) of the webpage into a pre-trained scrolling operation determination model. The scrolling operation determination model then determines whether a scrolling operation needs to be performed on the rendered page. Specifically, the scrolling operation determination model can be implemented using any classification model. The model performs classification prediction based on the input rendered page and the size of the visible area (i.e., viewport). There are two possible classification predictions: one is that a scrolling operation needs to be performed on the rendered page, and the other is that a scrolling operation does not need to be performed.

[0119] When it is determined that scrolling is required on the rendered page, the server can determine the number of scrolling operations to be performed based on the ratio of the rendered page size to the viewport size. The server then performs the corresponding number of scrolling operations to refresh the image of the rendered page that falls within the viewport, and captures the image of the rendered page falling within the viewport after each scrolling operation to obtain a screenshot of the webpage.

[0120] If it is determined that scrolling is not necessary for the rendered page, then scrolling will no longer be performed on the rendered page.

[0121] Step S504: Parse the rendered page of the webpage and extract the links contained in the rendered page of the webpage.

[0122] After obtaining the rendered page of the webpage, the server parses the rendered page to extract the links contained within it. This step can utilize any webpage parsing algorithm or tool to parse the webpage and extract its links; no specific limitations are specified here.

[0123] Step S505: Perform a link opening operation on the links contained in the rendered page of the webpage to obtain the rendered page corresponding to the link.

[0124] For any link contained in the rendered page of a webpage, the server performs a link-opening operation to open the rendered page corresponding to the link. In this embodiment, the rendered page corresponding to the link is referred to as the link page.

[0125] Step S506: Take a screenshot of the rendered page corresponding to the link to obtain a screenshot of the link page. The webpage screenshot includes the screenshot of the link page.

[0126] For any link to a corresponding page, the server takes a screenshot of the rendered page corresponding to the link to obtain a screenshot of the link page.

[0127] The specific implementation principle of taking a screenshot of the linked page in this step is the same as that of taking a screenshot of the rendered page of the webpage in the aforementioned step S503. For details, please refer to the relevant content, which will not be repeated here.

[0128] It should be noted that for any linked page, it is also possible to parse and extract the second-level links contained within the linked page (links contained in the original webpage are called first-level links), perform a link-open operation on the second-level links, obtain the rendered page corresponding to the second-level links (called the second-level link page), and take a screenshot of the second-level link page. This process is repeated until a screenshot of the link page at the nth level is taken. The screenshots of each level of link page are also used as screenshots of the webpage. Here, n is a positive integer, which can be set according to the actual application scenario and requirements; no specific limitation is made here.

[0129] The solution in this embodiment can simulate human web browsing behavior (such as scrolling pages and clicking links) by performing webpage scrolling and link opening operations, thereby enabling access to and collection of deeper webpage content, which is difficult for traditional search engines to achieve. This capability is particularly important for obtaining specific or hidden information.

[0130] In this embodiment, for any link in a webpage related to the query information, the linked page is opened and a screenshot of the linked page is taken. The screenshot of the linked page and the screenshot of the rendered webpage are then compared. Figure 1 As a screenshot of a webpage, it can fully capture relevant content for searching information, improving the efficiency and accuracy of information retrieval, thereby enhancing the quality of human-computer interaction.

[0131] Figure 6 A flowchart of the human-computer interaction method provided in the embodiments of this application is shown below. Figure 6As shown, the overall process framework of the human-computer interaction method in this embodiment is as follows:

[0132] The system receives query information input from the user. The diagram shows an example where the input query is "What is the profile picture of OpenAITwitter?"

[0133] The system automatically determines whether a search is necessary based on the current query information. If a search is deemed required, the query is rewritten. For example, the expression "OpenAITwitter" is incomplete; the rewritten query could be "What is the profile picture that OpenAI uses on Twitter?".

[0134] The search is performed based on the rewritten query to obtain search results. The search results include the URLs of web pages related to the query information.

[0135] Based on the URL of the webpage related to the query information, the system simulates human webpage browsing behavior and obtains images of the regions on the webpage that match the query information. Specifically, this includes: taking a screenshot of the visible area of ​​the rendered webpage; taking a new screenshot after scrolling the rendered page; and opening n layers of linked pages on the webpage and obtaining screenshots of those linked pages. For detailed implementation, please refer to the relevant content in the foregoing embodiments, which will not be repeated here. Furthermore, the system uses a Visual Language Model (VLM) to extract images of the regions on the webpage that match the query information.

[0136] Based on the query information, the text information of the web pages related to the query information, and the area image, generate a response and a prompt.

[0137] The response prompt is input into the LLM for inference, generating and outputting a response to the query. For example, the generated response shown in the figure is "OpenAI's Twitter profile picture is its company logo".

[0138] The solution in this embodiment can be implemented using an intelligent agent. This agent can receive query information, determine search results, take screenshots of web pages, perform web page operations (such as opening links and scrolling), and generate accurate response information using an intelligent interaction model. By integrating web page operations and screenshot capabilities into the intelligent agent, it can perform more complex operations, such as opening links and scrolling pages, to obtain deeper information. The intelligent agent also has multimodal information processing capabilities, capable of processing not only text information but also image information and performing interactive operations on web pages. This multimodal information processing capability allows the intelligent agent to acquire and analyze data from different perspectives, thereby generating more comprehensive and accurate answers.

[0139] Intelligent agents can integrate various resources, including using search engines to search for relevant web pages, and can also combine various technical means such as screenshots and web page content extraction to obtain richer and more comprehensive information, thereby improving the efficiency and accuracy of information retrieval and thus improving the quality of human-computer interaction.

[0140] Furthermore, by using intelligent agents to achieve end-to-end automated processes, from receiving input query information to generating the final response, the entire process is highly automated and requires no human intervention. This end-to-end automated process greatly improves the efficiency of human-computer interaction while also reducing the possibility of errors.

[0141] Figure 7 This is a structural block diagram of a computing device according to an embodiment of this application. Figure 7 As shown, the computing device may include one or more (only one is shown in the figure) processors 701 and memory 702. The memory 702 stores computer programs / instructions, and the processor 701 executes the computer programs / instructions. When the computer programs / instructions are executed by the processor 701, they implement the technical solutions provided in any of the aforementioned method embodiments. Their specific functions and the technical effects they can achieve are similar and will not be repeated here.

[0142] The aforementioned computing device can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one model machines, mobile phones, tablet computers, or other portable smart terminals, and the computing device may have the model in the above embodiments of this application pre-installed.

[0143] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0144] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0145] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Figure 8 As shown, the server includes a memory 801 and a processor 802. The memory 801 stores computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 802 is communicatively connected to the memory 801 and executes the computer-executable instructions stored in the memory 801 to implement the technical solutions provided in any of the above method embodiments. Their specific functions and the technical effects they achieve are similar and will not be repeated here.

[0146] Optional, such as Figure 8 As shown, the server also includes other components such as a firewall 803, a load balancer 804, a communication component 805, and a power supply component 806. Figure 8 The diagram only shows some components and does not mean that the server only includes... Figure 8 The components shown. Figure 8 This example uses a cloud server deployed in the cloud as an example, but the server can also be deployed locally. This embodiment does not make any specific limitations here.

[0147] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method of any of the foregoing embodiments. The specific functions and technical effects to be achieved are not described here.

[0148] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. The execution of the computer program by the at least one processor causes the server to perform the technical solution provided in any of the above method embodiments. The specific functions and the technical effects that can be achieved are not described here.

[0149] This application provides a chip, including a processing module and a communication interface. The processing module is capable of executing the technical solution of the server in the aforementioned method embodiments. Optionally, the chip further includes a storage module (e.g., a memory), which stores instructions. The processing module executes the instructions stored in the storage module, and the execution of the instructions stored in the storage module causes the processing module to execute the technical solution provided in any of the aforementioned method embodiments.

[0150] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0151] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules in at least one processor.

[0152] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0153] The aforementioned storage device can be object storage service (OSS).

[0154] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0155] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as mobile hotspots (WiFi), second-generation (2G), third-generation (3G), fourth-generation (4G) / Long Term Evolution (LTE), fifth-generation (5G), or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), infrared, Ultra Wide Band (UWB), Bluetooth, and other technologies.

[0156] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0157] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0158] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components within an electronic device or host device.

[0159] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0160] The order of the embodiments described above is merely for illustrative purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0162] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0163] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A human-computer interaction method, characterized in that, include: The system performs a search based on the input query information to obtain search results, which include information about web pages related to the query information. Based on the search results, extract the region image that matches the query information from the webpage, and obtain the text information of the webpage; Based on the query information, the regional image, and the text information of the webpage, a response to the query information is generated.

2. The method according to claim 1, characterized in that, The step of searching based on the query information to obtain search results includes: The query information is input into the search determination model, and the search determination model determines whether the query information needs to be searched. If it is determined that the query information needs to be searched, then a search is performed based on the query information to obtain search results.

3. The method according to claim 1, characterized in that, The step of searching based on the query information to obtain search results includes: The query information is rewritten to obtain a rewritten query; Based on the rewritten query, the search engine is invoked to perform a search and obtain the search results.

4. The method according to claim 1, characterized in that, The search results include the Uniform Resource Locator (URL), title, and summary of the webpage; obtaining the text information of the webpage includes: Based on the URL of the webpage, obtain the content text of the webpage, which includes the content text, title, and summary of the webpage.

5. The method according to claim 1, characterized in that, The step of extracting a region image from the webpage that matches the query information based on the search results includes: Based on the search results, obtain a screenshot of the webpage; The query information and the webpage screenshot are input into a visual language model, and the visual language model is used to determine the target area in the webpage screenshot that matches the query information. Extract the region image of the target area from the screenshot of the webpage.

6. The method according to claim 5, characterized in that, The search results include the URLs of web pages related to the query information. Obtaining a screenshot of the web page based on the search results includes: Based on the URL of the webpage, obtain the rendered page of the webpage; Take a screenshot of the rendered page of the webpage to obtain a webpage screenshot.

7. The method according to claim 6, characterized in that, Also includes: Parse the rendered page of the webpage and extract the links contained in the rendered page of the webpage; Perform a link opening operation on the links contained in the rendered page of the webpage to obtain the rendered page corresponding to the link; A screenshot of the rendered page corresponding to the link is taken to obtain a screenshot of the link page. The webpage screenshot includes the screenshot of the link page.

8. The method according to claim 6 or 7, characterized in that, Taking a screenshot of any of the rendered pages, including: Based on the set visible area size, extract the image from the rendered page that falls within the visible area; Perform a scrolling operation on the rendered page, refresh the image falling into the visible area of ​​the rendered page, and capture the image falling into the visible area of ​​the current rendered page.

9. The method according to any one of claims 1-7, characterized in that, The step of generating response information for the query based on the query information, the regional image, and the text information of the webpage includes: Based on the query information, the region image, and the text information of the webpage, a prompt message is generated. The prompt message is used to prompt the intelligent interaction model to generate a response message for the query information based on the region image and the text information of the webpage. The prompt information is input into the intelligent interaction model, which then generates a response to the query based on the prompt information.

10. The method according to claim 2, characterized in that, Also includes: If it is determined that no search is required to answer the query, the query is input into the intelligent interaction model, and the intelligent interaction model generates the answer to the query.

11. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein when the computer programs / instructions are executed by the processor, they implement the method according to any one of claims 1-10.

12. A server, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the server to perform the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images