Page interaction methods, electronic devices and computer program products

Abstract

This disclosure provides a page interaction method, an electronic device, and a computer program product. The method first responds to receiving an interaction request by extracting the text content to be responded to and the interaction context of the page that initiated the interaction request from the interaction request. The text content to be responded to represents the request intent of the interaction request. Then, the text content is modified through the interaction context. The modified text content includes the representation information of the target object to which the interaction request is directed and which is intended to be processed. The representation information can uniquely identify the target object. Finally, an execution instruction is issued to the intelligent agent based on the modified text content, and feedback is given to the requester of the interaction request after the execution result of the execution instruction is obtained.

Description

Technical Field

[0001] This disclosure specifically relates to page interaction methods, electronic devices, readable storage media, and computer program products. Background Technology

[0002] Intelligent agents are important tools that provide users with a variety of functional services. When users browse information using applications, they can make queries and ask questions through conversations with intelligent agents to obtain query results or answers to questions.

[0003] When users want to consult content on a webpage by interacting with an AI agent, the common practice is for developers to provide a corresponding entry point within the application's page. Accessing this entry point allows users to directly ask questions to the AI ​​agent without manually typing. However, in some scenarios, because the question posed to the AI ​​agent cannot specify the target of the consultation, the application cannot provide this functionality.

[0004] For example, if a user wants to consult the AI ​​agent provided by an application that offers real estate services about the details and comparison results of multiple houses, even if the program provides a corresponding entry point, and the user enters the entry point and automatically asks a question, the AI ​​agent will not receive the question content that includes the house being asked. The AI ​​agent cannot know which houses the user wants to compare, and therefore cannot fulfill the user's needs. Summary of the Invention

[0005] This disclosure provides page interaction methods, electronic devices, readable storage media, and computer program products.

[0006] The first aspect of this disclosure proposes a page interaction method, comprising: in response to receiving an interaction request, extracting text content to be responded to and an interaction context of the page initiating the interaction request from the interaction request, wherein the text content to be responded to represents the request intent of the interaction request; modifying the text content through the interaction context, wherein the modified text content includes representation information of a target object to which the interaction request is directed and intended to be processed, wherein the representation information can uniquely identify the target object; and issuing an execution instruction to an agent based on the modified text content, and providing feedback to the requester of the interaction request after obtaining the execution result of the execution instruction.

[0007] According to some embodiments of this disclosure, the interaction context includes interaction information formed by interacting with page elements on the page that initiates the interaction request; modifying the text content through the interaction context includes: modifying the part of the text content to be modified as determined by the interaction information, wherein the part to be modified corresponds to the target object to which the interaction request is directed and intended to be processed.

[0008] According to some embodiments of this disclosure, the interaction information includes interaction behavior information, which includes the identification information of the page element that has been interacted with and the interaction method of interacting with the page element.

[0009] According to some embodiments of this disclosure, the text content to be responded to is obtained through the configuration information of a target control provided by the application that initiated the interaction request, and the target control is used to trigger a session with the agent.

[0010] According to some embodiments of this disclosure, different target controls correspond to different processing intentions; modifying the part to be modified in the text content determined by the interaction information includes: calling a rewrite service corresponding to the processing intention to modify the part to be modified in the text content determined by the interaction information.

[0011] According to some embodiments of this disclosure, at least a portion of the target controls are global target controls, which are used to provide a session entry point with the agent in multiple different pages. The interaction context also includes a page identifier of the page that initiates the interaction request. Calling a rewrite service corresponding to the processing intent to modify the part to be modified in the text content determined by the interaction information includes: calling a rewrite service corresponding to the processing intent to modify the part to be modified in the text content determined by the interaction information and the page identifier.

[0012] According to some embodiments of this disclosure, the execution result of the execution instruction is provided to the requester of the interaction request through an interactive page. The interactive page is used to provide an interactive environment for interacting with the intelligent agent, and the page that initiates the interaction request is the previous page of the interactive page.

[0013] According to some embodiments of this disclosure, in response to receiving an interaction request, image content is extracted from the interaction request; before issuing an execution instruction to the agent based on the modified text content, the semantic information represented by the extracted image content is first determined; issuing an execution instruction to the agent based on the modified text content includes: adding the semantic information to the modified text content to form an execution instruction, and sending the execution instruction to the agent.

[0014] A second aspect of this disclosure provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform the page interaction method described in any of the above embodiments.

[0015] A third aspect of this disclosure provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the page interaction method described in any of the above embodiments.

[0016] This disclosure provides a fourth aspect of a computer program product, the computer program product comprising a computer program, which, when executed by a processor, is used to implement the page interaction method described in any of the above embodiments. Attached Figure Description

[0017] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0018] Figure 1 The illustration shows an application scenario diagram of some embodiments of the page interaction method of this disclosure.

[0019] Figures 2-6 A schematic diagram of the overall flow of a page interaction method M100 according to some embodiments of the present disclosure is shown.

[0020] Figure 7 A flowchart of a multimodal page interaction method M100 according to some embodiments of the present disclosure is shown.

[0021] Figure 8 This is a schematic block diagram of the structure of a page interaction device according to one embodiment of the present disclosure.

[0022] Figure 9 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure. Detailed Implementation

[0023] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0024] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments and is not restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0027] When customers use an app to browse information, they may want to inquire about the content displayed in the app, or they may want to use the app to process and modify data related to business matters. In this case, they can ask questions or give instructions to the intelligent agent through the intelligent agent dialogue page provided by the app, thereby obtaining answers to their questions or completing data processing.

[0028] Taking the real estate sector as an example, when a customer browses property listings on a real estate app and wants to compare and analyze the basic information of multiple properties in a new development, they can click on the corresponding control provided by the app to automatically submit this analysis request directly in the chat window. However, clicking this control only provides the AI ​​agent with the user's intention to perform the analysis, such as issuing an instruction to "compare and analyze the basic information of the properties," but it cannot specify which properties should be compared.

[0029] Therefore, this disclosure proposes a page interaction method.

[0030] Figure 1The diagram illustrates an application scenario of a page interaction method according to some embodiments of this disclosure. In this application scenario, a user terminal 10 and a server terminal 20 may be included. The user terminal 10 can communicate with the server terminal 20 to send and receive data or commands. In this disclosure, both the user terminal 10 and the server terminal 20 include at least one processor and at least one memory.

[0031] For example, client 10 can be a terminal device held by the user, and server 20 can be a device that provides data browsing, querying, and other business services to the user. When a user browses content provided by the app on client 10 and wants to query or modify some of the content, they initiate an interaction request by triggering controls provided by the app. The interaction request includes text indicating the intent of the request and an interaction context formed on the app page before the user initiates the interaction request. The text is the user query, and the interaction context represents information about the user's actions on the app page. At this time, the object to be processed in the user query is vague and ambiguous. After receiving the interaction request, server 20 rewrites the user query according to the interaction context, changing the intent to remain the same but clarifying the object to be processed, and rewriting the vague referents into clear and specific referents. Then, an intelligent agent answers the rewritten user query and sends the answer back to client 10 as a response to the user's question.

[0032] exist Figure 1 The shape and structure of the user terminal 10 and the server terminal 20 shown should not be construed as limiting the scope of protection of this disclosure. In this disclosure, "terminal device" can be different types of electronic devices, such as mobile phones, tablets, laptops, or desktop computers. In addition, the server terminal 20 can include a server, which can be a physical server or a cloud server. This disclosure does not limit the type of server.

[0033] Figure 2 A schematic diagram illustrating the overall flow of a page interaction method M100 according to some embodiments of this disclosure is shown. For example... Figure 2 The method shown includes steps S110, S120, and S130. This method can be executed by electronic devices such as mobile phones and computers.

[0034] S110, in response to receiving an interaction request, extract the text content to be responded to and the interaction context of the page that initiated the interaction request from the interaction request. The text content to be responded to represents the request intent of the interaction request.

[0035] S120, modify the text content through the interaction context. The modified text content includes the representation information of the target object to which the interaction request is directed and which is intended to be processed. The representation information can uniquely identify the target object.

[0036] S130: Based on the modified text content, an execution instruction is issued to the agent, and after receiving the execution result of the instruction, feedback is sent to the requester of the interaction request.

[0037] When using the app, if a user wants to query or modify the content displayed on the current page, they can initiate an interaction request through the methods provided by the app, such as clicking a button provided by the app to initiate an interaction request with the intelligent agent.

[0038] The text content to be responded to represents the user's intent, and this text content can be determined by how the interaction request is initiated. For example, if a user wants to compare and analyze multiple property listings on a page, and the page provides a button B1 with the words "Property Information Comparison" to provide this comparison function, then the initiation method could be clicking button B1 to initiate the interaction request. There is a mapping relationship between button B1 and the intent of "Property Information Comparison." When button B1 is clicked, pre-configured text content is automatically used as the representation of the user's intent. For example, the text content could be "Compare and analyze the basic information of multiple properties." In this case, the target object of the user's intent is vague and needs to be clarified.

[0039] Interaction context represents the user's interactions with the page before initiating an interaction request, and can be used to determine the target object the user intends to process. At the moment the user triggers an interaction request, the application determines the method of initiation to obtain the text content to be responded to, and simultaneously collects user behavior information prior to triggering the interaction request as the interaction context, such as browsing the page, selecting controls, etc. Then, it generates an interaction request containing the aforementioned text content and interaction context, and sends it to the server.

[0040] After receiving an interaction request, the server extracts text content representing the user's intent and interaction context that can be used to determine the target object for processing the user's intent. Based on the user's interaction with the page information provided by the interaction context, the server rewrites the text content.

[0041] The target object is the business object that the user intends to process, such as a customer, property listing, dataset, or a specific business transaction. The target object can be represented by an ID, order number, or other information that uniquely identifies and determines it. If the original text contains vague references to the target object, then an identifier that specifically represents the target object is used to replace these vague references. For example, changing "conduct a comparative analysis of basic information for multiple properties" to "conduct a comparative analysis of basic information for properties with IDs [ID1, ID2, ...]" is used. If the original text does not contain vague references to the target object, then an identifier that specifically represents the target object is added to the original text to clearly indicate the target object. For example, changing "analyze customer churn risk" to "analyze the customer churn risk for customer ID XXX" results in the rewritten text. In the new text, the target object that the user intends to process is clear and explicit.

[0042] The new text content is sent to the agent as an execution instruction. The agent executes the instructions in the new text content, such as asking questions or making queries, and obtains the execution result. The server sends the execution result to the client, and the client displays the execution result, thus completing the response to the user.

[0043] Understandably, the user intent in an interaction request could be to generate an analysis report, recommend content, or modify configuration settings, such as changing appointment time parameters. In this case, the object whose value needs to be modified can be clearly defined by modifying the text content, and the intelligent agent can then execute the modification. Afterwards, the user is notified that the object's value has been modified. The processing method for the target object corresponds to the user intent and can include one or more of the following: querying, modifying the task.

[0044] According to the page interaction method proposed in the embodiments of this disclosure, when a user triggers a session, not only is text corresponding to the triggering method and conforming to the user's interaction intent automatically transmitted, but also an interaction context that can be used to semantically enhance the text is transmitted. In the case where the text uses vague references or omits the object of intent processing, the multi-dimensional context information generated by the user during the interaction process is used to semantically enhance the text representing the user's interaction intent, complete the missing sentence structure in the text, and automatically rewrite the text into a clear, complete, and targeted instruction. This significantly improves the accuracy of the agent's understanding of the user's intent, avoids invalid interactions caused by unclear references, and provides a more intelligent and smoother interaction experience.

[0045] For example, the interaction context may include interaction information generated by interacting with page elements on the page that initiated the interaction request.

[0046] The page initiating the interaction request can be a business page, such as a property listing page, which displays summary information of all available properties in a particular neighborhood in a list format. Page elements can be interactive controls on the page, such as checkboxes, clickable options, input boxes, etc.

[0047] Interaction information is a record generated when an interaction is performed on one or more page elements within the current page (such as clicking, selecting, or modifying) before an interaction request is initiated.

[0048] Interaction information can include interaction behavior information. Interaction behavior information can include the identification information of page elements that have been interacted with, as well as the interaction methods used to interact with page elements.

[0049] Interaction behavior information can be a structured record of the user's specific actions. The identification information is the unique ID of the business entity corresponding to the manipulated page element, such as customer ID or property ID. The interaction method is the type of action the user performs on that page element, such as selecting a checkbox, clicking a property card, or entering a value.

[0050] Different types of interaction methods represent different user processing needs, and the interaction method is adapted to the way the interaction request is initiated, that is, compatible with the text content to be responded to. For example, if a user clicks button B1 to trigger "Compare Housing Information," before this trigger, the user would usually select housings in the housing list or browse the basic information of multiple housings by clicking on housing cards. Therefore, the text content "Compare and analyze the basic information of multiple houses" in the interaction request matches the selected or browsed housing IDs (identification information) in the interaction information, and can be combined as the basis for rewriting the text content.

[0051] Users can select properties from multiple listings using checkboxes and then initiate an interaction request to compare multiple properties. The application then redirects to an interaction page with the agent, and the IDs of the selected properties become interactive information within the interaction context. Similarly, users can click on multiple property cards sequentially on a business page to access property details and view specific parameters, and then initiate an interaction request to compare multiple properties. In this case, the IDs of the properties viewed on that business page, or the card IDs of the property cards, become interactive information within the interaction context.

[0052] Figure 3 A schematic diagram illustrating the overall flow of a page interaction method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 3In step S120, modifying the text content through the interaction context can specifically involve modifying the portion of the text content to be modified, as determined by the interaction information. The portion to be modified corresponds to the target object pointed to by the interaction request and intended for processing.

[0053] The text content obtained through the method of initiating the interaction request is vague and cannot be uniquely determined regarding the target object that the interaction request points to and intends to process. However, the interaction information can uniquely determine which objects the user intends to process. Combined with the specific processing methods already provided in the original text content, the specific objects to be processed and the processing methods can be rewritten.

[0054] Suppose an app has multiple business pages, including a new home sales page, a used home sales page, etc. These business pages display a list of properties for the corresponding business type, and each page can have a button for providing comparative analysis functions. After a user performs actions on two different pages and clicks the corresponding button, the text content of the interaction requests initiated by these two triggers can be the same (both intending to compare multiple properties). However, the interaction information will differ depending on the business type (the target objects are different properties), resulting in different execution instructions after rewriting.

[0055] This implementation can accurately identify the target object of the user's intent processing and understand the user's intent to process the target object, thereby generating more targeted and executable instructions and supporting a wider range of business scenarios.

[0056] For example, the text content to be responded to can be obtained through the configuration information of the target control provided by the application that initiated the interaction request. The target control is used to trigger a session with the agent. Different target controls can correspond to different processing intentions.

[0057] The target control provided by the application can be a control provided on the page that initiates the interaction request, such as a button. The configuration information of the target control is metadata pre-defined during the application development or configuration phase. This can include preset text information used to represent the processing intent, i.e., the text content to be responded to (the original text content). After the target control is triggered (e.g., clicked), the preset text information corresponding to that control is used as the text content to be responded to, combined with the interaction context to form an interaction request. In multiple business pages provided by the app, the same business page can provide multiple different target controls, each corresponding to a different function, to meet various user intents.

[0058] This implementation method, by reading the configuration information of the target control that triggered the request, can directly and reliably obtain the user's processing intent, thereby enabling adaptive rewriting of the original text content. Furthermore, by using multiple target controls to provide the user with various executable functions, it enhances the system's flexibility and scalability.

[0059] Figure 4 A schematic diagram illustrating the overall flow of a page interaction method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 4 In step S120, the method of modifying the part to be modified in the text content determined by the interaction information may specifically include: calling the rewrite service corresponding to the processing intent to modify the part to be modified in the text content determined by the interaction information.

[0060] A rewriting service is used to modify the text content of the input agent. It can be a functional module that encapsulates rewriting logic. Each rewriting service can be used to handle only one type of text rewriting task under one processing intent. The rewriting service takes the original text and interaction information as input and outputs the rewritten instructions.

[0061] The rewriting service can be provided by decorators, which can dynamically add new behaviors or functionalities to objects without modifying the original class code. Each decorator can extract key business data from specific contextual information and integrate it into the complete logic of the original text content, thereby automatically rewriting the user's short query into a complete instruction that the intelligent agent can accurately parse.

[0062] Decorators are pluggable. When a new business type is added, meaning a new intent and processing function is added, only a new class needs to be created, implementing the decorator's interface, and registered with the manager. If the intent and processing function need to be modified, only the corresponding decorator needs to be modified, without affecting other modules.

[0063] Multiple decorators can correspond to a single decorator manager, which can contain a registry to store the names of all available decorators. During initialization, the server can load each decorator, and the manager receives intent information and other content through an exposed entry point, then automatically selects the appropriate decorator based on the intent information.

[0064] Each decorator can receive the original parameter and return the rewritten string. For example, if the original parameter is "Compare and analyze the basic information of multiple houses", the returned string will be "Compare and analyze the basic information of houses with house IDs [ID1, ID2, ...]".

[0065] This implementation provides an independent rewriting service for each type or category of processing intent. When an interaction request is received, it can be dynamically routed to the corresponding rewriting service based on the processing intent. This facilitates the expansion and maintenance of the rewriting service, improving the system's flexibility and scalability. When a new business function (such as generating XX solution) needs to be added, only a new rewriting service needs to be developed and registered in the scheduler, without modifying any existing code, greatly improving development efficiency and system stability.

[0066] For example, at least some of the target controls can be global target controls. Global target controls are used to provide a session entry point with the agent across multiple different pages.

[0067] The target control used to trigger a conversation with the agent can be a built-in function button on the page or a globally floating bubble button. For example, business page A may have a fixed button; clicking this button will navigate to interactive page D and trigger a conversation. Interactive page D may also have a fixed button; clicking this button will not navigate to another page but will only trigger a conversation. If there has already been one or more conversations with the agent, clicking this button will continue a new round of conversation based on the previous conversation content, and the conversation context will not be lost.

[0068] Applications can be configured with bubble buttons as global buttons. These bubble buttons will always appear on the upper layer of the application's interface when a user navigates between different business pages. The bubble buttons are functionally associated with multiple different business pages, all supporting the same processing intent. For example, both the new home sales page and the used home sales page support property comparison analysis. Therefore, developers can set the button providing this comparison analysis function as a bubble button. Clicking this button will redirect to an interactive page, and the interaction request will contain the property selected by the user on the previous business page.

[0069] The interaction context can also include the page identifier of the page that initiated the interaction request. The page identifier indicates the previous page before the interaction, i.e., the page before the jump. The page identifier can include a page ID. When an interaction request is initiated through a global target control, the page identifier, combined with the page element identifier information in the interaction behavior information, ultimately determines which specific target objects the user interacted with. The page identifier can also include information indicating the page's business category. The same list of property IDs may have different business semantics on different business pages. Combining the business category with the page element identifier information also allows the rewrite service to better understand the source and meaning of the interaction information, ensuring the correctness of the rewrite logic.

[0070] Figure 5 A schematic diagram illustrating the overall flow of a page interaction method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 5In step S120, the method of calling the rewrite service corresponding to the processing intent to modify the part to be modified in the text content determined by the interaction information can be as follows: calling the rewrite service corresponding to the processing intent to modify the part to be modified in the text content determined by the interaction information and the page identifier.

[0071] If a user initiates an interaction request on a business page by triggering a global target control, that business page is the page before the redirect, and the page identifier can be the page ID of that business page. The rewrite service uses the preset text of the global target control as the text content to be responded to, and rewrites the text content by combining the page ID, the page element where the interaction occurred, and the interaction method.

[0072] If a user has already entered the interactive page and initiates an interaction request through a target control inherent to the interactive page or a global target control provided by the program, since there is no page redirection, the page identifier is known, i.e., the ID of the interactive page. Therefore, the interaction context can be directly obtained from the interactive page to rewrite the text content.

[0073] This implementation method, by introducing global target controls and page identifiers, embeds intelligent interaction capabilities into any business scenario of the entire application, providing diverse interactive service entry points while ensuring the integrity and accuracy of contextual information in cross-page scenarios. Furthermore, compared to using URLs (Uniform Resource Locator) for cross-page parameter passing, this implementation method can transmit complex and longer parameters.

[0074] For example, the execution result of the instruction can be provided to the requester of the interaction request through an interactive page. The interactive page can be used to provide an interactive environment for interacting with the agent. The interactive page is a dedicated page for hosting dialogue between the user and the agent. It provides a standardized interactive environment for displaying historical messages, input fields, and the agent's responses. The agent's query results in response to user questions or the results of data modifications to business matters indicated by the user can be displayed on the interactive page, for example, arranged in a conversational format. Users can continuously ask questions to the agent on the interactive page and view the agent's responses.

[0075] The page initiating the interaction request can be the previous page from the interaction page. In other words, this implementation can either allow for a continuous session within the interaction page (intra-page session) or a session initiated by navigating to the interaction page from a business page (cross-page session). Multiple different pages initiating interaction requests can redirect to the same interaction page, meaning the business side uses a single interaction page to connect with interactions across multiple business pages. Regardless of whether the user initiates the request from the customer details page, property listing page, or any other page, the request will ultimately converge on the same interaction page for subsequent communication.

[0076] Using a unified interactive page can improve the consistency of user experience, reduce the complexity of business pages, and facilitate function iteration and data analysis through centralized dialogue management.

[0077] Figure 6 A schematic diagram illustrating the overall flow of a page interaction method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 6 In step S110, in response to receiving an interaction request, the text content to be responded to and the interaction context of the page that initiated the interaction request are extracted from the interaction request. Specifically, in response to receiving an interaction request, the text content to be responded to, the image content, and the interaction context of the page that initiated the interaction request are extracted from the interaction request.

[0078] The page interaction method M100 further includes step S129, which is performed before step S130. Step S129 is: determining the semantic information represented by the extracted image content.

[0079] In step S130, the specific method of issuing an execution instruction to the agent based on the modified text content can be as follows: adding semantic information to the modified text content to form an execution instruction, and sending the execution instruction to the agent.

[0080] Image content can originate from page elements within a business page. In this case, the image content is part of the interaction context, and the content selected by the user on the business page can include one or more images. If it's an in-page conversation within an interactive page, the image content can also be an image clicked or uploaded by the user on the interactive page. In this case, the image content originates from the user and may not belong to the interaction context but rather be an independent item like the text content. Image content can collaboratively rewrite the original text content with the interaction context. The interaction context is used to clarify the target object that the user intends to process, and the semantics represented by the image content can be used to describe the content of the target object.

[0081] Semantic information is structured information that can be described in text after analyzing image content (such as OCR text recognition, object detection, and scene understanding). For example, identifying the floor plan from a house photo and describing it in natural language. Image content can provide supplementary information from another modality by offering semantic information, thus combining with modified text content to fully express the user's intent, enabling agents to handle more complex, vision-based queries.

[0082] Figure 7 A flowchart of a multimodal page interaction method M100 according to some embodiments of this disclosure is shown. (See also...) Figure 7 A user clicked on the floor plan of property F on its details page to zoom in. The user then clicked a global bubble button on the details page to initiate an interaction request. The intent of this bubble button was to provide a layout description of the property. Parsing the interaction request yielded the original query text "Provide a layout description of this property," as well as interaction behavior information from the interaction context. This included the floor plan of property F and information indicating the click interaction method. The page identifier in the interaction context was the ID of the details page for property F.

[0083] The corresponding rewrite service is invoked through a global target control. Based on the interaction behavior information and page identifiers in the interaction context, the original query text is rewritten to obtain the modified query text, which reads "Layout introduction of house with house ID XXXXX". The house ID is obtained from the details page ID. Simultaneously, when an image is detected in the interaction request, natural language representations of the house structure and interior layout are obtained through image analysis, which reads "Three-bedroom apartment, north-south facing, large living room". The content obtained from image analysis is added to the modified query text, i.e., the two are concatenated to obtain the final query text. The final query text is then input into the agent to obtain the query results.

[0084] This implementation extends the input of interactive requests from plain text to multimodal input that combines text and images, realizing intelligent interaction that integrates text and images. By supporting multimodal input, the program can handle complex tasks involving visual information, providing users with more powerful and intuitive service capabilities.

[0085] Based on any of the above embodiments, this disclosure also provides a page interaction device. Figure 8 This is a schematic block diagram of the structure of a page interaction device according to one embodiment of this disclosure. Figure 8 As shown, the page interaction device includes: a context extraction module 110, a text modification module 120, and an instruction execution module 130.

[0086] The context extraction module 110 is used to extract the text content to be responded to and the interaction context of the page that initiated the interaction request in response to the received interaction request. The text content to be responded to represents the request intent of the interaction request.

[0087] The text modification module 120 is used to modify text content through the interaction context. The modified text content includes the representation information of the target object to which the interaction request is directed and which is intended to be processed. The representation information can uniquely identify the target object.

[0088] The instruction execution module 130 is used to issue execution instructions to the agent based on the modified text content, and to provide feedback to the requester of the interaction request after receiving the execution result of the execution instructions.

[0089] The aforementioned page interaction device can be in the form of computer software, and each module of the aforementioned page interaction device can be implemented through computer software modules. The specific implementation process of the functions and roles of each module in the aforementioned device is detailed in the corresponding steps of the aforementioned method, and will not be repeated here.

[0090] The execution entity of the page interaction method in the specific embodiments of this disclosure can be an electronic device such as a server.

[0091] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the page interaction method of any of the embodiments described above.

[0092] Figure 9 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure.

[0093] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0094] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0095] The processor 1200 can be a central processing unit (CPU). The processor 1200 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0096] The memory 1300 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions of the computer program in the embodiments of this disclosure. The processor 1200 implements the page interaction method by running the non-transitory software programs, instructions, and modules stored in the memory 1300.

[0097] The memory 1300 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created by the processor 1200. Furthermore, the memory 1300 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1300 may optionally include memory remotely located relative to the processor 1200, and these remote memories may be connected to the processor 1200 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0098] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0099] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of this disclosure are performed wholly or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM, or other programmable device.

[0100] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0101] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0106] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0107] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

[0108] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0109] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0110] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0111] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0112] At the same time, it is understood that the data involved in this disclosed technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

Claims

1. A page interaction method, characterized in that, include: In response to receiving an interaction request, extract the text content to be responded to and the interaction context of the page that initiated the interaction request from the interaction request, wherein the text content to be responded to represents the request intent of the interaction request; The text content is modified through the interaction context. The modified text content includes representation information of the target object to which the interaction request is directed and which is intended to be processed. The representation information can uniquely identify the target object. as well as Based on the modified text content, an execution instruction is issued to the agent, and after receiving the execution result of the execution instruction, feedback is sent to the requester of the interaction request.

2. The page interaction method according to claim 1, characterized in that, The interaction context includes the interaction information formed by interacting with page elements on the page that initiated the interaction request; Modifying the text content through the interaction context includes: The text content to be modified, as determined by the interaction information, is modified. The part to be modified corresponds to the target object that the interaction request points to and intends to process.

3. The page interaction method according to claim 2, characterized in that, The interaction information includes interaction behavior information, which includes the identification information of the page element that has been interacted with and the interaction method of interacting with the page element.

4. The page interaction method according to claim 2 or 3, characterized in that, The text content to be responded to is obtained through the configuration information of the target control provided by the application that initiated the interaction request. The target control is used to trigger a session with the agent.

5. The page interaction method according to claim 4, characterized in that, Different target controls correspond to different processing intentions; Modifying the portion of the text content to be modified as determined by the interaction information includes: The rewrite service corresponding to the processing intent is invoked to modify the part of the text content to be modified as determined by the interaction information.

6. The page interaction method according to claim 5, characterized in that, At least some of the target controls are global target controls, which are used to provide a session entry point with the agent in multiple different pages, and the interaction context also includes a page identifier of the page that initiates the interaction request; Invoking a rewrite service corresponding to the processing intent to modify the portion of the text content to be modified as determined by the interaction information, including: The rewrite service corresponding to the processing intent is invoked to modify the part of the text content to be modified, as determined by the interaction information and the page identifier.

7. The page interaction method according to claim 1 or 6, characterized in that, The execution result of the execution instruction is provided to the requester of the interaction request through the interactive page. The interactive page is used to provide an interactive environment for interacting with the intelligent agent. The page that initiates the interaction request is the previous page of the interactive page.

8. The page interaction method according to claim 1, characterized in that, In response to receiving an interaction request, image content is also extracted from the interaction request; before issuing an execution instruction to the agent based on the modified text content, the semantic information represented by the extracted image content is first determined; Based on the modified text content, execution instructions are issued to the agent, including: The semantic information is added to the modified text content to form an execution instruction, which is then sent to the intelligent agent.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the page interaction method according to any one of claims 1 to 8.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, is used to implement the page interaction method according to any one of claims 1 to 8.

Images