Method, device, equipment and program product for interface interaction

Abstract

Provided are an interface interaction method, device, equipment and program product. The method proposed herein includes: receiving a first request, the first request being used to instruct a smart system to execute a first task, the first request representing identification of at least one item of media content; presenting at least one element at an associated position of the at least one item of media content, the at least one element representing a first processing procedure of the smart system on the at least one item of media content, the first processing procedure corresponding to a first stage of the first task; and outputting first information, the first information being from the smart system, the first information including an execution result of the first task. In this way, the correspondence between the processing procedure of the smart system and the execution result of the task can be intuitively reviewed, thereby improving the explainability and credibility of the processing result.

Description

Technical Field

[0001] The examples in this article generally relate to the field of computer science, and in particular to methods, apparatuses, devices, and program products for user interface interaction. Background Technology

[0002] In multimodal question-answering scenarios, users often need to ask questions to intelligent systems based on image content, hoping to obtain answers related to specific locations, local details, or cross-regional relationships within the image. However, when the intelligent system presents the processing results only in text form, it is difficult for users to intuitively determine the correspondence between the processing results and specific regions in the image. Summary of the Invention

[0003] In a first aspect, a method for interface interaction is provided. The method includes: receiving a first request, the first request being used to instruct an intelligent system to perform a first task, the first request representing the identification of at least one piece of media content; presenting at an associated location of the at least one piece of media content, the at least one element representing a first processing procedure of the intelligent system on the at least one piece of media content, the first processing procedure corresponding to a first stage of the first task; and outputting first information, the first information being from the intelligent system, the first information including the execution result of the first task.

[0004] In a second aspect, an apparatus for interface interaction is provided. The apparatus includes: a receiving module configured to receive a first request, the first request being used to instruct an intelligent system to perform a first task, the first request representing an identification of at least one piece of media content; a first presentation module configured to present at least one element at an associated location of the at least one piece of media content, the at least one element representing a first processing procedure of the intelligent system on the at least one piece of media content, the first processing procedure corresponding to a first stage of the first task; and an output module configured to output first information, the first information being from the intelligent system, the first information including the execution result of the first task.

[0005] In a third aspect, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0007] In this way, by presenting elements representing the intelligent system's processing at relevant locations in the media content, users can intuitively perceive the intelligent system's process of locating, analyzing, and reasoning about various objects in the media content, and thereby verify the correspondence between the intelligent system's output and the media content, thus improving the interpretability and credibility of the processing results.

[0008] It should be understood that the content described in this section is not intended to limit the key or important features of the examples in this article, nor is it intended to restrict the scope of the solution. Other features will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the various examples herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of the example environment is shown; Figures 2A to 2F Example interfaces for some scenarios are shown; Figures 3A to 3D Example interfaces for some scenarios are shown; Figures 4A to 4D Example interfaces for some scenarios are shown; Figure 5 The flowcharts show example processes of interface interactions in some scenarios; Figure 6 Schematic block diagrams of example devices for interface interaction in some scenarios are shown; and Figure 7 A block diagram of an electronic device capable of implementing multiple illustrative scenarios is shown. Detailed Implementation

[0010] The examples in this document will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples presented herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0011] It should be noted that the headings of any section / subsection provided herein are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0012] In the description of the examples in this document, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0013] The examples in this article may involve users' historical interaction data, the acquisition of such data, and / or its use. All of these aspects comply with relevant laws, regulations, and rules. In the examples presented here, all collection, acquisition, processing, manipulation, forwarding, and use of historical interaction data are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any potentially involved historical interaction data or information should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0014] As used in this article, the term "intelligent system" can refer to a system capable of autonomous control based on machine learning models. An intelligent system is, for example, a virtual object or physical entity that can make decisions and autonomously execute actions based on machine learning models to achieve preset goals or complete preset tasks. An intelligent system can be an automated program that understands user intent and can utilize models or invoke tools to complete various types of tasks. In some cases, examples of intelligent systems may include, but are not limited to: intelligent agents, robot programs, chatbots, digital avatars, intelligent customer service, digital assistants, etc. Alternatively, an intelligent system can also be an intelligent role implemented based on a machine learning model. An "intelligent system" can process user requests based on generative models (e.g., language models, multimodal models) to perform specified types of tasks. In some cases, an intelligent system may also be associated with a virtual account, which may have a corresponding avatar or nickname.

[0015] As used in this article, the term "media content" refers to visual or multimedia content that can be processed by intelligent systems, including but not limited to user-uploaded images, images returned by search tools, images generated by generative models, video frames, document scans, or any combination of the above.

[0016] As used in this article, the term "element" refers to a visual marker or component presented at a relevant location within media content, used to characterize the intelligent system's processing of the media content. Elements may include, but are not limited to, numbered markers, counting indicators, connecting lines, trajectory paths, magnified views, coordinate labels, selected areas, or animation effects.

[0017] As used herein, the term "task" refers to a set of operations performed by an intelligent system in response to a request, having one or more processing objectives. A task may include performing at least one of the following operations on one or more media contents: identification, location, counting, comparison, relation identification, trajectory identification, size measurement, information extraction, or result generation. A task may be completed by a single operation or may include multiple interconnected or coordinated processing procedures.

[0018] As used in this document, the term "request" refers to information or instructions that enable an intelligent system to start, configure, continue, or adjust a task. Requests can include natural language content, media content, task parameters, object identifiers, locations, conditions, user actions, session context, and any combination thereof. Requests can be formed through a single user input, assembled from content provided by the user in multiple interaction steps, or based on the current interface state or existing session content.

[0019] The term "processing" as used in this article refers to the process by which an intelligent system analyzes, understands, and reasons about content during the execution of a task, including but not limited to object recognition, object localization, object counting, relationship identification, trajectory identification, region comparison, or size measurement.

[0020] As used herein, the term "stage" refers to a part of a task that can be distinguished from other parts based on at least one of the following: processing objective, processing object, input data, intermediate results, processing time period, or interface presentation state. A stage may include one or more processing steps and may output intermediate results for use by other stages. Different stages may be executed sequentially, partially overlappingly, or upon fulfillment of certain conditions; a sequential relationship between two stages is indicated only when one stage is explicitly stated to be a subsequent stage of another.

[0021] The term “structured data” as used in this article refers to data that describes the processing of an intelligent system in a predefined format, including but not limited to structured representations describing processing steps, coordinate information, animation patterns, and final results.

[0022] The term "rendering information" as used in this article refers to information generated from structured data to guide the visual presentation of elements, including but not limited to annotation layer styles, animation effects, overlay methods, and interaction logic.

[0023] The term "component" as used in this article refers to a visual container in an interface used to display media content and carry elements. Components may include, but are not limited to, image display areas, visualization canvases, or media display areas with annotation overlays.

[0024] As used in this article, the term "object" refers to a unit of content within media content that can be identified, located, marked, compared, counted, or relationed. An object can correspond to an entity represented by the media content, such as a person, animal, object, building, celestial body, or device control, or it can correspond to a region, outline, surface, text, field, icon, node, path location, or other manipulable visual unit within the media content. An object can be a complete entity or a part of an entity; the same unit of content can also be categorized into different objects based on different task requirements.

[0025] As used in this document, the term "type" refers to a category or classification dimension formed based on one or more attributes of an object, task, relationship, element, or processing step. Types can be derived based on name, semantics, shape, color, texture, text, location, function, processing target, interaction method, or a combination of these attributes. For example, the type of an object could include socks, containers, controls, stars, planets, or satellites, and the type of a task could include object identification, object counting, relationship identification, trajectory identification, or size measurement. Types can be explicitly obtained from a request or derived by the intelligent system based on the request, media content, or session context.

[0026] As used in this document, the term "relationship" refers to the association between two or more objects, areas, media content, processing steps, or processing results. Relationships can include matching, pairing, adjacency, relative, containment, subordination, belonging, surrounding, sequential, directional, spatial, correspondence, succession, or other relationships that describe the connection between related content. Relationships can be directional or non-directional, and can be presented through at least one of the following methods: connecting elements, identical identifiers, different styles, spatial proximity, or textual information.

[0027] As used in this document, "first information" refers to information related to the first task, provided directly or indirectly by the intelligent system. First information includes at least the execution result of the first task, and may also include at least one of the following: intermediate conclusions, number of objects, object relationships, trajectory determination, dimensional data, processing steps, processing basis, confidence information, explanatory text, and association identifiers with at least one of these elements. First information may be output in the form of text, numerical values, images, audio, video, structured messages, or a combination of the above. First information may be output all at once or in parts as the first task is executed.

[0028] As used in this document, the term "associated location" refers to a location that has a spatial or display correspondence with an object, region, processing step, or processing result. An associated location can be located inside, on the boundary, at the center, or near an object or region, or it can be located at a corresponding display location within a component that carries media content, at the endpoint of a connecting element, or at another location that maintains an identifiable correspondence with the related content. Therefore, presenting an element at an associated location of media content does not require the element to perfectly coincide with the original pixel location of the object. As media content is scaled, cropped, moved, or displayed on different components, the associated location can be transformed or updated according to the display state of the media content.

[0029] The term "first part" as used herein refers to a portion of the at least one media content that corresponds to the processing object, intermediate processing result, or processing evidence of the first stage. The first part can be a continuous or non-continuous region within the media content, and may include local image regions in an image, at least a frame or a portion of a time segment in a video, pages, paragraphs, fields, or character regions in a document, and local models or spatial regions in three-dimensional media content. The first part can be obtained by the intelligent system based on the object location, boundaries, features, relationships, or intermediate processing results acquired in the first stage, or it can be obtained based on the user's selection of the media content. The first part can establish a correspondence with the at least one media content through information such as coordinates, region boundaries, object identifiers, time ranges, page identifiers, or content identifiers. The term "first" is used only to distinguish the corresponding part and does not limit the part's position, order of appearance, importance, or data storage order within the media content.

[0030] As used in this document, the term "dynamic process" refers to a process in which at least one element undergoes a presentation change over time, during processing steps, or in the state of the interface. Presentation changes can include at least one of the following: appearance, disappearance, movement, extension, scaling, blinking, color change, transparency change, content update, style switching, stopping presentation, or replacing presentation. The terms "second part," "third part," etc., within a dynamic process can correspond to different time intervals, processing steps, sets of elements, or states of the interface. Stopping the presentation of a part can include hiding, removing, reducing visibility, stopping updates, or replacing the part with another part.

[0031] As used in this document, the term "action" refers to interactive input that can be received by an electronic device and trigger changes in the state of the interface, task, or element presentation. Actions can include clicks, touches, long presses, swipes, drags, selections, inputs, voice commands, eye inputs, gesture inputs, or other interaction methods. Actions can directly affect elements, media content, parts of information, or interface components, or indirectly affect content by referencing relevant content.

[0032] As mentioned above, in multimodal question-answering scenarios, intelligent systems can typically read images uploaded by users and answer questions in text form regarding objects, relationships, quantities, regional differences, or inference conclusions within the images. However, when questions rely on specific locations, local details, cross-regional comparisons, or relationships between multiple images, simply describing the processing results in text makes it difficult for users to determine whether the intelligent system correctly identified the target region or to verify the system's reasoning basis.

[0033] A user interface interaction scheme is proposed. According to the scheme, an electronic device receives a request to instruct an intelligent system to perform a task utilizing media content, presents elements representing the intelligent system's processing at a relevant location within the media content, and outputs information including the task execution result.

[0034] Through the above approach, users can directly see visual markers corresponding to the positioning, analysis, and reasoning processes of the intelligent system within the media content. This allows them to intuitively judge whether the intelligent system has correctly identified the target area, established the correct object relationships, or tracked the correct path. This enables them to directly verify the correspondence between the intelligent system's processing procedures and task execution results, thereby improving the interpretability and credibility of the processing results. Because the elements are presented in relevant locations within the media content, users do not need to repeatedly switch between text descriptions and images to verify the processing results, reducing the number of interface jumps. Consequently, the client does not need to load a separate result details page, reducing client-side page rendering overhead and cache usage, and consequently reducing the number of detail page requests sent to the server.

[0035] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0036] Example Environment Figure 1 A schematic diagram of example environment 100 is shown. (e.g.) Figure 1 As shown, example environment 100 may include electronic device 110.

[0037] In this example environment 100, electronic device 110 can run an application 120 that supports user interface interaction. Application 120 can be any suitable type of application for user interface interaction, including but not limited to: multimodal question-and-answer applications, image understanding applications, smart assistant applications, or other suitable applications. User 140 can interact with application 120 via electronic device 110 and / or its attached devices.

[0038] exist Figure 1 In environment 100, if application 120 is active, electronic device 110 can use application 120 to present interface 150 for supporting interface interaction.

[0039] In some cases, electronic device 110 communicates with server 130 to provide services to application 120. Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, virtual reality (VR) / augmented reality (AR) devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some cases, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0040] Server 130 can be a single physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and basic cloud computing services. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for applications 120 that support user interface interaction in electronic devices 110.

[0041] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection can include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections. In some cases, server 130 and electronic device 110 can exchange signaling information through their communication connection.

[0042] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme.

[0043] The following description of the example will continue with reference to the accompanying drawings.

[0044] Example Interaction The examples described herein can be implemented on electronic device 110. It should be noted that the operations described relative to electronic device 110 can be performed by a related application 120 running on electronic device 110. The interface described relative to electronic device 110 can be provided by application 120; for example, electronic device 110 presents the corresponding interface by running application 120. Furthermore, some operations described relative to electronic device 110 may require the assistance of server 130 to complete.

[0045] Figures 2A to 2F Example interfaces 200A to 200F are shown for counting and identifying object relationships in at least one type of media content, according to certain scenarios. Interfaces 200A to 200F can, for example, be provided by... Figure 1 The electronic device 110 shown is provided. See below for reference. Figures 2A to 2F This describes the process of counting objects and identifying object relationships.

[0046] In some scenarios, electronic device 110 receives a first request. The first request instructs the intelligent system to perform a first task. The first request represents the identification of at least one piece of media content. As an example, electronic device 110 may present interface 200A. Electronic device 110 can receive the first request through interface 200A. The first request may include content of any type, such as text content, image content, audio content, video content, etc. For example, electronic device 110 may receive input content through an input component in interface 200A to trigger the intelligent system to perform the first task as the input content. Here, the first task matches the first request; that is, the first task may be described by input content from user 140, such as "Help me see how many unmatched socks there are." Upon receiving such input content, the intelligent system can determine that the first task is an object identification and matching task based on the input content, and perform the aforementioned task based on at least one piece of media content.

[0047] In some scenarios, the first request represents the identification of at least one piece of media content, meaning that the first request includes information that enables the electronic device 110 or the intelligent system to obtain or access at least one piece of media content. Specifically, the first request may directly include at least one piece of media content, such as image data, video data, audio data, or document data; it may also include the access address of at least one piece of media content, such as a Uniform Resource Locator (URL), a Uniform Resource Identifier (URLId), a storage path, or an object storage address; and it may also include the identifier of the at least one piece of media content, such as a media content identifier, a file identifier, an object key, a content hash value, a message location identifier, etc. The aforementioned media content, access address, and identifier may be included individually or in combination in the first request. The electronic device 110 or the intelligent system can parse the corresponding information in the first request, obtain the at least one piece of media content from local storage, cache, database, server, or other storage locations, and establish a correspondence between the media content and the first task.

[0048] In some cases, the electronic device 110 may present an input component for interacting with the intelligent system. Furthermore, the electronic device 110 may receive a first request through the input component. The first request includes at least one type of media content and natural language content. The natural language content describes the requirements of the first task.

[0049] As an example, such as Figure 2A As shown, on interface 200A, electronic device 110 presents input component 202 and at least one piece of media content 206. Input component 202 is used to receive at least one piece of media content and natural language content. In this example, user 140 can submit at least one piece of media content and natural language content to the intelligent system as a first request through input component 202. The natural language content can describe the requirements of the first task, for example, the natural language content could be "How many unpaired socks are there in the picture?". At least one piece of media content includes multiple objects to be processed, such as characters, items, animals, buildings, etc. in the image. After receiving the natural language content and at least one piece of media content, electronic device 110 can present message 204 and at least one piece of media content 206 on interface 200A.

[0050] In this way, the intelligent system can perform targeted processing tasks on media content, improving the efficiency and accuracy of task processing. Furthermore, the electronic device 110 can complete the request assembly without navigating between different pages, reducing the number of user steps. Correspondingly, the electronic device 110 only needs to send one request containing media content and a task description to the server 130, reducing the number of edge-cloud interaction rounds and lowering edge-cloud communication bandwidth usage.

[0051] In some cases, in response to receiving a first request, the intelligent system can perform a first processing procedure on at least one piece of media content 206. The first processing procedure corresponds to a first stage of a first task. During the first processing procedure, the intelligent system can identify objects in at least one piece of media content 206, and obtain the location, type, number, or relationships between the identified objects, and generate structured data describing the first processing procedure. Based on the structured data, the intelligent system can generate at least one element. Further, after at least one element is generated, the electronic device 110 can present at least one element at an associated location in at least one piece of media content 206. Thus, the objects, locations, numbers, or relationships identified by the intelligent system in the first stage can be presented to the user through at least one element.

[0052] In this way, users can verify the objects, locations, numbers, or relationships identified by the intelligent system within at least one piece of media content 206 without relying solely on the textual conclusions in the initial information. The electronic device 110 can reduce the number of times a separate results page needs to be loaded to view the processing basis and decrease the likelihood of errors in the correspondence between media content and processing results.

[0053] In some cases, presenting at least one element includes: acquiring structured data describing a first processing procedure; and presenting at least one element at an associated location of at least one piece of media content based on the structured data. Specifically, in response to receiving a first request, the intelligent system can perform a first processing procedure on at least one piece of media content and generate structured data describing the first processing procedure. The structured data may include at least one processing step, each processing step may include at least one of a step identifier, step text, object identifier, media content identifier, coordinate set, processing result, and animation mode. After acquiring the structured data, the electronic device 110 can obtain the media content corresponding to the corresponding processing step based on the media content identifier, obtain the associated location of at least one element based on the object identifier and coordinate set, and present at least one element according to the processing order indicated by the step identifier. For example, in Figure 2B In the object identification process shown, the structured data may include the object identifier and position coordinates of each sock. The electronic device 110 can present elements 210-1, 210-2, 212-1 and 212-2 at the associated positions of the corresponding socks based on the above position coordinates.

[0054] In some cases, at least one element includes a first element. The first element is used to mark the first object. At least one piece of media content includes a first object. As an example, such as Figure 2BAs shown, the intelligent system can obtain a first object (e.g., object 211-1 or 211-2) from a plurality of objects to be processed comprising at least one media content 206, and generate elements 210-1, 210-2, 212-1, and / or 212-2 corresponding to the first object. The electronic device 110 can present at least one media content 206 within a first component 208 of the interface 200B, and present corresponding elements at associated positions of the first object. For example, elements 210-1 and 210-2 are presented at associated positions of object 211-1, and elements 212-1 and 212-2 are presented at associated positions of object 211-2, so that the user can perceive the first object marked by each first element.

[0055] In this way, users can identify objects identified by the intelligent system while viewing media content, without having to search for objects again from a list of objects or text descriptions. Consequently, the electronic device 110 can reduce the number of interface switching times and the number of interface nodes used to present a separate list of objects.

[0056] In some cases, presenting at least one element at an associated location of at least one piece of media content based on structured data may include: generating rendering information based on the structured data, wherein the rendering information characterizes the rendering style of at least one element; and presenting at least one element based on the rendering information. As an example, after acquiring structured data, electronic device 110 can select a target rendering template matching the current processing step from multiple rendering templates based on the processing step type and the type of the first task in the structured data, and input the object identifier, object position, object relationship, and element rendering order from the structured data into the target rendering template to generate rendering information. The rendering information may include at least one of element type, element shape, element color, line width, transparency, text style, element level, animation mode, and rendering time. As an example, such as... Figure 2B As shown, when the processing step is an object identification or object counting step, the intelligent system can generate rendering information for sequentially presenting elements 210-1, 210-2, 212-1, and 212-2. The electronic device 110 can present the aforementioned elements at the corresponding positions on the socks according to the element styles and presentation order indicated by the rendering information.

[0057] In some cases, the style of the first element is derived based on the type of the first task. For example, the intelligent system may determine the type of the first task based on the natural language content and / or at least one type of media content in the first request. The type of the first task may include, for example, at least one of object identification type, object counting type, object relationship identification type, trajectory tracking type, and object size measurement type. The electronic device 110 may select a target style corresponding to the type of the first task from a variety of candidate styles, and present the first element in the target style. Different candidate styles may differ in at least one of shape, color, border, numbering, text, transparency, and / or animation mode. For example, such as... Figure 2B As shown, when the first request instructs the identification and counting of socks in at least one piece of media content 206, the intelligent system can determine that the first task includes an object identification type and an object counting type. The electronic device 110 can present elements 212-1 and 212-2 in an object marking style to identify the socks in at least one piece of media content 206, and elements 210-1 and 210-2 in a style including a count number to represent the counting order of the socks. In the case of a trajectory tracking type, the first element can adopt a trajectory style including nodes and connecting lines; in the case of an object size measurement type, the first element can adopt a measurement style including endpoint markers, size lines, and / or size information.

[0058] In some cases, presenting at least one element may include presenting the first element at a first position, which is derived from a second position, representing the location of the first object within at least one piece of media content. The intelligent system may, after identifying the first object, obtain the second position of the first object within the at least one piece of media content. The second position may, for example, be represented by the center point coordinates, bounding box coordinates, outline coordinates, or the image region occupied by the first object.

[0059] In some cases, the second position may be a position represented in the original coordinate system of at least one media content 206. For example, the second position may be represented using original pixel coordinates. Alternatively, the second position may also be represented using relative coordinates, normalized coordinates, grid coordinates, or other coordinate forms that can characterize the position of the first object in at least one media content 206. The electronic device 110 can obtain the display position and display size of at least one media content 206 in the first component 208, and based on the display position, display size, and second position, convert the second position into a first position in the display coordinate system of the interface 200B.

[0060] As an example, such as Figure 2BAs shown, the electronic device 110 can obtain the first positions of elements 210-1, 210-2, 212-1, and 212-2 respectively based on the position of the corresponding sock in at least one piece of media content 206, and present the aforementioned elements at the associated position of the corresponding sock. When at least one piece of media content 206 is presented in the first component 208 at a predetermined scaling ratio, the electronic device 110 can convert the second position according to the scaling ratio, and adjust the converted position based on the margin, cropping area, or offset of at least one piece of media content 206 in the first component 208 to obtain the first position.

[0061] In some situations, when the size of the first component 208 changes, at least one piece of media content 206 is scaled, or the display position of at least one piece of media content 206 changes in the interface, the electronic device 110 can re-acquire the first position based on the second position and move the first element to the re-acquired first position. Thus, the first element can continue to be displayed at the associated position of the first object after the interface state changes, reducing the possibility of positional shift between the first element and the first object, and facilitating user verification of the first object identified by the intelligent system.

[0062] In this way, the first position of the first element can change with the scaling, cropping, or display position of at least one piece of media content 206, and the second position of the first object is reacquired, so that the first element is continuously presented at the associated position of the first object. The electronic device 110 can complete the position update of the first element without triggering the intelligent system to re-identify the first object or requesting the server 130 to return the object position, thereby reducing position calibration requests and reducing the possibility of misalignment between the first element and the first object in different display states.

[0063] In some cases, presenting at least one element may include: obtaining size information of at least one media content; obtaining at least one display parameter of the at least one element based on the size information; and presenting the at least one element based on the at least one display parameter. The size information may include at least one of the following: the original width, original height, display width, display height, scaling ratio, cropping area, display offset, and screen pixel density of the at least one media content 206. After obtaining the second position of the first object, the electronic device 110 may obtain the size information of the at least one media content 206 in the first component 208, and convert the second position to a first position in the first component 208 based on the ratio between the original size and the display size. Further, the electronic device 110 may obtain at least one of the following display parameters of the first element based on the size information: marker radius, border width, font size, label offset, transparency, and animation movement distance. As an example, such as Figure 2BAs shown, the electronic device 110 can present the first position and display parameters of elements 210-1, 210-2, 212-1 and 212-2 according to the display size of at least one media content 206 in the first component 208, and present the above elements according to the acquired first position and display parameters.

[0064] In some cases, the first task is at least used to identify objects of a first type, where the first object corresponds to the first type. As an example, the first task may include an object identification task, which can be used to identify objects in at least one media content 206 that correspond to the first type. For example, the first type could be a sock type, and the first task could be a sock identification task. The intelligent system can identify each object in at least one media content 206 based on the first type to obtain multiple socks in at least one media content 206. Further, after the objects of the first type are identified, the intelligent system can generate at least one element to identify the socks identified in at least one media content 206. As an example, such as... Figure 2B As shown, after object recognition, the electronic device 110 can present elements 212-1 and 212-2 in the interface 200B to identify objects of a first type in at least one piece of media content 206. Elements 212-1 and 212-2 can be one implementation of the first element. Elements 212-1 and 212-2 are used to mark a first object corresponding to the first type (e.g., object 211-2). For example, elements 212-1 and 212-2 are used to mark socks in at least one piece of media content 206.

[0065] In this way, the intelligent system can narrow down the scope of object identifiers based on the first type defined by the first request, and generate the first element only for the first object corresponding to the first type. The user can directly verify the object identified as a sock based on elements 212-1 and 212-2, reducing the manual exclusion of irrelevant objects. Correspondingly, the intelligent system does not need to perform subsequent relation identification or counting processing on objects that do not correspond to the first type, thereby reducing the number of candidate objects to be processed and the corresponding computational load.

[0066] In some cases, the first element includes a first count identifier, which corresponds to a first object, and the first stage includes counting a first group of objects. The first group of objects corresponds to a first type and includes the first object. As an example, the intelligent system can traverse multiple identified socks according to the positional order of the objects in at least one piece of media content 206 or the object identifier order. For each identified object corresponding to the first type, the intelligent system can assign a count number to the corresponding object and update the current count of the first group of objects. Figure 2BAs shown, the electronic device 110 can present elements 210-1 and 210-2 sequentially according to the above counting order. The text related to the number in elements 210-1 and 210-2 can serve as first counting identifiers to mark the corresponding first objects. In this way, the electronic device 110 can display the number of first objects through elements in at least one piece of media content 206, allowing the user to verify the counting results of the intelligent system one by one according to each first counting identifier.

[0067] In some cases, at least one element also includes a second element. The second element represents a first relationship between the first object and the second object. As an example, the first relationship can be a pairing relationship. After identifying multiple socks, the intelligent system can extract the color, texture, shape, pattern, text, and / or other visual features of each sock, and determine whether the first object and the second object satisfy a pairing relationship based on the extracted visual features. As an example, such as... Figure 2C As shown in interface 200C, electronic device 110 can present elements 214-1 and 214-2 between the associated positions of a first object and a second object that satisfy a pairing relationship. Elements 214-1 and 214-2 can serve as implementations of the second element to represent that the first object and the second object satisfy the first relationship (e.g., socks at both ends of element 214-1 are the same pair of socks).

[0068] In this way, the second element can directly present the first relationship between the first and second objects, allowing users to confirm the object relationship without repeatedly comparing multiple objects with textual descriptions. Correspondingly, the electronic device 110 can locally present the second element based on the object identifier and object location, eliminating the need to load separate relationship details for each object combination, thereby reducing client-side rendering overhead and the number of end-to-cloud queries.

[0069] In some cases, the first stage includes identifying objects that satisfy a first relationship, which is derived from a first request. For example, the intelligent system can derive the first relationship as a pairing relationship between socks from the natural language content of the first request, "How many socks in the image are not paired?". Based on this pairing relationship, the intelligent system can perform pairwise comparisons or group matching on multiple socks identified during the object identification process to obtain combinations of objects that satisfy the pairing relationship. In response to a combination of objects satisfying the pairing relationship, the electronic device 110 can present a second element between the first and second objects in that combination to identify objects that satisfy the first relationship (i.e., the pairing relationship). In response to a corresponding object not being included in any combination of objects satisfying the pairing relationship, the electronic device 110 can highlight the object in a subsequent interface to indicate that the object does not satisfy the first relationship.

[0070] In this way, the first relationship can be directly obtained from the task requirements in the first request. The intelligent system can filter objects that meet the first relationship according to the relationship type specified by the user, without requiring the user to input matching rules separately. The electronic device 110 can present the combination of objects that meet the first relationship and the objects that do not meet the first relationship respectively, reducing multiple rounds of relationship queries and enabling the user to continuously review the object matching and non-matching results.

[0071] In some cases, the second element is used to connect the first and second objects. For example, such as... Figure 2C As shown, the intelligent system can obtain the display positions of the first object and the second object in the first component 208, and define the two display positions or their vicinity as connection endpoints. The intelligent system can progressively draw element 214-1 or element 214-2 from one connection endpoint to the other to connect the first object and the second object. When the first relationship is directional, the second element may also include a directed connection element (e.g., a line segment including an arrow). The direction of the directed connection element can characterize the first relationship. For example, in navigational, causal, or operational sequence relationships, the electronic device 110 may present an arrow on one side of the second object to indicate the direction of the relationship from the first object to the second object. Figure 2C The pairing relationship shown can be undirected, so elements 214-1 and 214-2 may not include direction markers.

[0072] In this way, the connection endpoints of the second element can correspond to the display positions of the first and second objects, ensuring that the connection relationship remains aligned with the corresponding objects. For directional first relationships, directed connection elements can also simultaneously convey both the object relationship and the relationship direction, reducing the need for additional directional text. The electronic device 110 can generate connection elements based on object coordinates and direction identifiers, thereby reusing the same relationship data to support the interface presentation of both undirected and directed relationships.

[0073] In some cases, at least one piece of media content includes a first piece of media content and a second piece of media content. The first piece of media content includes a first object, and the second piece of media content includes a second object. For example, such as... Figure 2EAs shown, electronic device 110 can present first media content 213-1 and second media content 213-2 in interface 200E. The intelligent system can identify objects in the first and second media content respectively, and based on the visual characteristics of the objects, determine a first object 211-2 and a second object 215 that satisfy a first relationship (e.g., a pairing relationship between socks) across the media content. Electronic device 110 can present element 214-3 according to the position of the first object 211-2 in the first media content, and present element 214-4 according to the position of the second object (e.g., object 215) in the second media content. Elements 214-3 and 214-4 can use the same numbering, color, or other associated style to jointly implement the second element, enabling the user to perceive that the first object in the first media content and the second object in the second media content satisfy the first relationship.

[0074] In this way, the primary relationship across media content can be uniformly presented through related elements distributed across different media content, eliminating the need for users to open the first and second media content separately and manually compare object features. Electronic device 110 can utilize the media content index to map object locations to corresponding media content and synchronously update elements 214-3 and 214-4 within the same interface, thereby reducing the number of media content switching attempts and the memory usage caused by repeated loading.

[0075] In some cases, at least one element displays a magnified view of the first part, which represents the intermediate processing result of the first stage. At least one piece of media content includes the first part. As an example, during the process of identifying a first object or a first relationship, the intelligent system may acquire visual evidence used for processing the first task, such as the texture, text, boundaries, or other local details of the first object, and match the first part including such local details from at least one piece of media content. Further, the intelligent system may acquire reference information describing the position and extent of the first part, and extract the first part from at least one piece of media content based on this reference information. The electronic device 110 may magnify the extracted local content, generate an element displaying a magnified view of the first part, and present the element at the associated position corresponding to the corresponding object. This magnified view can represent the intermediate processing result of the first stage, and may display the object identified by the intelligent system in the first stage, local features further obtained from the object, or parts of at least one piece of media content that need further verification in subsequent processing.

[0076] like Figure 2FAs shown, the first task can be to match socks in at least one piece of media content 206. In the first stage, the intelligent system identifies multiple socks in at least one piece of media content 206 and extracts a region containing local patterns, textures, color transitions, or outline features as a first part for each sock to be compared. The intelligent system can extract corresponding local content from at least one piece of media content 206 based on the position of the first part and present elements 212-3 and 212-4 at the associated position of the corresponding sock. Elements 212-3 and 212-4 can respectively display magnified views of the corresponding first part. Thus, the user can view the first object identified by the intelligent system or the local details upon which the first relationship is based without leaving the current interface.

[0077] In this way, users can review details such as textures, text, or borders while keeping the overall media content visible. The electronic device 110 can locally capture and zoom in on a first area based on at least one piece of already loaded media content 206, without requesting a separate local image or jumping to a details page, thereby reducing the amount of data transmitted between the device and the cloud and the page loading overhead.

[0078] In some cases, electronic device 110 outputs first information. This first information comes from the intelligent system and includes the result of executing a first task. For example, such as... Figure 2D As shown, the intelligent system can trigger the presentation of socks marked by elements 212-1 and 212-2 that do not meet the pairing relationship based on the object identification result and the pairing relationship identification result, and generate first information including the number of unpaired socks. The electronic device 110 can output the first information, for example, displaying "There are two unpaired socks" in the session interface. The first information can also be associated with the object identifier, location, or processing step identifier of elements 216-1 and 216-2.

[0079] In some cases, at least one element includes a fourth element. Electronic device 110 can receive a first operation that triggers a fourth portion of the first information. The fourth portion describes the processing associated with the fourth element; further, in response to receiving the first operation, electronic device 110 presents the fourth element in a first style. As an example, such as... Figure 2DAs shown, after the electronic device 110 outputs the first information including "There are two unpaired socks," the user can click, touch, or reference the aforementioned text content (corresponding to the fourth part) in the first information as a first operation. The electronic device 110 can, based on the object identifier associated with the aforementioned text content, designate elements 216-1 and 216-2 as the fourth elements, and present elements 216-1 and 216-2 with styles such as highlighting, flashing, bolding borders, or changing color, while reducing the display intensity of other elements. In other scenarios, the first information may also include first content describing the intermediate processing of the first task, as the fourth part. When the first content is triggered, the electronic device 110 can query the fourth elements (e.g., elements 210-1 and 210-2) that match the first content, and present elements 210-1 and 210-2 with styles such as highlighting, flashing, bolding borders, or changing color.

[0080] In this way, the partial conclusions in the first piece of information can trigger corresponding visual evidence in at least one piece of media content, allowing users to directly locate the object, area, or trajectory used by the intelligent system from the text results. The electronic device 110 can complete the style switching of the fourth element based on the locally stored step identifier, object identifier, or trajectory identifier, without having to re-trigger the intelligent system to execute the first task or initiate additional correlation queries to the server 130, thereby reducing the number of steps required for users to search for visual evidence and the amount of data involved in end-to-end cloud communication.

[0081] Figures 3A to 3D Example interfaces 300A to 300D for identifying trajectories are shown, depending on certain scenarios. Interfaces 300A to 300D can, for example, be provided by... Figure 1 The electronic device 110 shown is provided. See below for reference. Figures 3A to 3D This describes the process of trajectory identification.

[0082] In some cases, the first request is used to identify at least one trajectory that meets the conditions. For example, such as... Figure 3A As shown, the electronic device 110 can present an input component 302 and receive at least one piece of media content 306 and natural language content 304 as a first request through the input component 302. The natural language content 304 could be, for example, "Help me find a way out of this maze." In response to receiving the first request, the intelligent system can determine that the first task is a trajectory identification task and obtain the conditions indicated by the first request, including reaching the maze exit from the maze entrance and avoiding impassable areas. The intelligent system can process at least one piece of media content 306 based on the above conditions to identify at least one candidate trajectory.

[0083] In this way, the intelligent system can limit the trajectory search range based on the target location and access conditions indicated by the first request, avoiding the generation of candidate trajectories irrelevant to the user's needs. The electronic device 110 provides natural language content 304 and at least one piece of media content 306 as the same first request to the intelligent system, which can reduce the number of interaction rounds used to supplement trajectory conditions and reduce the computational load generated by performing trajectory searches in irrelevant areas.

[0084] In some cases, the first element is used to mark the first object, and at least one piece of media content includes the first object. For example, such as... Figure 3B As shown, the intelligent system can perform area recognition on at least one media content 306 to obtain wall areas and multiple passable areas. Any passable area or the entrance location of a passable area can be used as a first object. The intelligent system can obtain the location and adjacency relationship of each first object and generate elements 310-1, 310-2, and 310-3 to mark each first object. Further, the electronic device 110 can present at least one media content 306 and elements 310-1, 310-2, and 310-3 in the first component 308, enabling the user to view the candidate entrances or passable areas obtained by the intelligent system.

[0085] In this way, elements 310-1 to 310-3 can be marked as the first objects to be used as trajectory nodes before trajectory generation, allowing users to confirm the candidate entrances and passable areas adopted by the intelligent system. Accordingly, subsequent trajectories can directly reuse the acquired first object positions and adjacency relationships, eliminating the need to repeatedly perform area recognition with each trajectory extension, thereby reducing redundant calculations and the possibility of misalignment between trajectory nodes and media content.

[0086] In some cases, the first element represents the first trajectory. The first trajectory is obtained based on the distribution of the first objects. As an example, an intelligent system can use multiple first objects as trajectory nodes, and based on the position and adjacency of each first object, sequentially obtain the next first object starting from the initial first object, until the target position is reached or it cannot continue. The intelligent system can use the sequentially obtained multiple first objects and their connection relationships as the first trajectory for identification. Figure 3C As shown, the electronic device 110 can sequentially present the nodes and connecting lines in element 312-1 according to the node order in the first trajectory. As the nodes and connecting lines appear sequentially, element 312-1 can gradually extend from the entrance along multiple passable areas to the exit to represent the first trajectory obtained based on the distribution of the first object.

[0087] In this way, the first trajectory is superimposed on the associated positions of multiple first objects in the form of nodes and connecting lines. The user can directly check whether the trajectory passes through the correct passable area along element 312-1. The electronic device 110 can generate the first trajectory at once according to the node sequence without loading the result pages of each node separately, thereby reducing interface jumps and the interface resources used to present the scattered trajectory steps.

[0088] In some cases, presenting at least one element includes presenting a dynamic process associated with at least one element. As an example, when presenting element 312-1, electronic device 110 may first present the starting node of the first trajectory, then present subsequent nodes in the order of nodes in the first trajectory, and progressively extend connecting lines between adjacent nodes. For each node presented, electronic device 110 may also simultaneously highlight the passable area corresponding to that node, or present text explaining the current processing step. When the first trajectory reaches the exit, electronic device 110 may stop extending connecting lines and update element 312-1 to a style indicating that the first trajectory meets the conditions.

[0089] In this way, users can observe the process of the intelligent system gradually perceiving the first trajectory from the starting position, based on the distribution of multiple first objects, rather than just viewing the final route. Electronic device 110 can update element 312-1 locally based on the node order in the structured data, without having to request rendering results from server 130 separately for each trajectory node, thereby reducing the number of edge-cloud interaction rounds and the amount of communication data.

[0090] In some cases, at least one element also includes a third element. The third element indicates whether the first trajectory satisfies the conditions. As an example, after obtaining the first trajectory, the intelligent system can determine whether the first trajectory reaches the target exit, whether it passes through a permitted area, and whether it intersects with an obstacle. In response to the first trajectory not satisfying the conditions indicated by the first request, the intelligent system can generate a negative judgment result associated with the first trajectory. Figure 3D As shown, electronic device 110 can display element 312-2 in interface 300D. The trajectory portion in element 312-2 can serve as an implementation of the first element, used to identify the first trajectory; element 314, associated with element 312-2, can serve as an implementation of the third element. Element 314 includes the text "Path Not Found," which can be used to indicate that the first trajectory does not meet the conditions. Electronic device 110 can also stop displaying subsequent nodes at the position where the trajectory cannot continue to extend, and display the first trajectory with a different color, line type, or transparency than the feasible trajectory.

[0091] In this way, the third element can be presented in association with the processing status of the first trajectory, allowing the user to simultaneously view the area traversed by the candidate trajectory, the location where the trajectory stops, and the judgment result of the trajectory not meeting the conditions, thereby verifying the reason why the intelligent system rejected the candidate trajectory. The electronic device 110 can update the style of the first trajectory and present the third element according to the node order and judgment result included in the same structured data, reducing additional verification requests for trajectory conditions.

[0092] Figures 4A to 4D Example interfaces 400A to 400D are shown for measuring the size and quantity of objects according to some scenarios. Interfaces 400A to 400D can, for example, be derived from... Figure 1 The electronic device 110 shown is provided. See below for reference. Figures 4A to 4D This describes the process of calculating the size and quantity of objects.

[0093] In some cases, electronic device 110 receives a first request, which instructs the intelligent system to perform tasks such as measuring object dimensions and calculating the number of objects to be placed. As an example, such as... Figure 4A As shown, the electronic device 110 can present an input component 402 and receive at least one piece of media content 406 and natural language content 404 as a first request via the input component 402. The natural language content 404 could be, for example, "How many of the same bowls can fit on this table?" In response to receiving the first request, the intelligent system can identify a first task including identifying the bowl as the first object, measuring the size of the bowl, calculating the available area of ​​the tabletop, and calculating the number of bowls that can be placed based on the bowl's size and the available area.

[0094] In this way, the intelligent system can determine the processing order between object identification, size measurement, and quantity calculation based on the same first request, so that the result of the previous processing step can be directly used as the input for the next processing step. The electronic device 110 does not need to receive multiple requests for object identification, size measurement, and quantity calculation separately, thereby reducing user input operations and end-to-end cloud interaction rounds, and keeping multiple processing stages within the same task context.

[0095] In some cases, presenting at least one element may include presenting the first element at a first position, which is obtained based on a second position of the first object within at least one piece of media content. As an example, the intelligent system may identify a bowl as the first object and obtain a first boundary position and a second boundary position of the bowl. The first boundary position and the second boundary position may serve as coordinates representing the second position of the first object within at least one piece of media content 406. Figure 4BAs shown, the electronic device 110 can convert the first boundary position and the second boundary position into a corresponding first position based on the display size of at least one media content 406 in the first component 408, and present elements 410-1 to 410-4 at the converted first position. Elements 410-1 to 410-4 can serve as implementations of the first element to mark the boundary position of the first object and present coordinate or size information associated with the boundary position.

[0096] In some cases, at least one element is a first set of elements, and the electronic device 110 may also present a second set of elements at an associated location of at least one piece of media content, the second set of elements representing a second phase of the first task.

[0097] As an example, such as Figure 4C As shown, after measuring the length of the bowl (the first object), the intelligent system can proceed to the second stage of the first task and identify the first and second boundary positions of the table. Elements 412-1 to 412-5 can be implemented as a second set of elements to characterize the second stage of measuring the table length and calculating the number of bowls that can be placed on it based on the table length. The electronic device 110 can present endpoint markers at the positions corresponding to elements 412-2 and 412-3, and progressively extend elements 412-4 from the first boundary position of the table to the second boundary position to connect the two boundary positions of the table. The intelligent system can calculate the length of the table based on the distance between the two boundary positions, the scale information of at least one media content 406, and / or reference object information, and use the length of the bowl obtained in the first stage as a scale reference to calculate the number of bowls that can be placed on the table based on the length of the table and the length of the bowl. The electronic device 110 can present the boundary coordinates of the table through element 412-1, and present the length of the table, the length of the bowl, and the number of objects obtained based on the above lengths through element 412-5, and output the number of objects as at least a part of the first information.

[0098] In some scenarios, such as Figure 4B As shown, elements 410-1 to 410-4 can be considered as an implementation of the first group of elements, used to characterize the first stage of measuring the length of the bowl; as Figure 4CAs shown, elements 412-1 to 412-5 can be implemented as a second set of elements, representing the second stage of measuring the length of the table and counting the objects. In the second stage, the intelligent system uses the length of the bowl obtained in the first stage and calculates the number of objects based on the ratio between the table length and the bowl length obtained in the second stage. Further, the first stage may include processing a first set of media content, and the second stage may include processing a second set of media content. At least one set of media content may include both the first and second sets of media content, and the first set of media content is different from the second set of media content. As an example, such as... Figure 4D As shown, the electronic device 110 can present a first set of media content and a first set of elements in a first component 408, and a second set of media content and a second set of elements in another component 414, so as to display the processing basis for bowl length measurement and table length measurement respectively.

[0099] In this way, the first and second sets of elements correspond to the bowl length measurement and table length measurement, respectively, allowing the user to distinguish between the two processing stages and confirm how the bowl length obtained in the first stage is used for object count calculation in the second stage. The electronic device 110 can present the two sets of elements sequentially in the same first component 408 according to the stage identifiers, or it can present the two sets of elements side-by-side through the first component 408 and another component 414, without needing to create separate task pages for the two measurement stages. The server 130 can also return the boundary coordinates, length data, and presentation order of the two stages in a single response, thereby reducing repeated measurement requests and edge-cloud interaction rounds.

[0100] In some cases, presenting the second set of elements includes: presenting the first component, which displays at least one piece of media content; stopping the presentation of the first set of elements in the first component and presenting the second set of elements in the first component. As an example, in Figure 4B Towards Figure 4CDuring the switching process, electronic device 110 can first present elements 410-1 to 410-4 in the first component 408 to demonstrate the first stage of measuring the length of the bowl. In response to the intelligent system completing the bowl length measurement and starting to measure the table length, electronic device 110 can maintain the presentation of at least one piece of media content 406 in the first component 408, stop presenting elements 410-1 to 410-4 in the first component 408, and present elements 412-1 to 412-5 in the same first component 408. Electronic device 110 can first present elements 412-2 and 412-3 to mark the two boundary positions of the table, then gradually extend the presentation of element 412-4, and present the boundary coordinates of the table, the length of the table, and the number of objects through elements 412-1 and 412-5. Thus, the first component 408 can switch from the bowl length measurement state to the table length measurement and object counting state without reloading at least one piece of media content 406.

[0101] In this way, the first component 408 can switch from the first group of elements to the second group of elements while keeping at least one piece of media content 406 and the user's visual context unchanged. The user can compare the bowl length obtained in the first stage with the table length and number of objects obtained in the second stage without repositioning the bowl and table. Accordingly, the electronic device 110 does not need to reload at least one piece of media content 406 for the stage switch and can reuse the bowl length saved in the first stage, thereby reducing page rebuilding, media resource reading, repeated measurement, and cache usage.

[0102] In some cases, presenting at least one element may include displaying at least one element through a dynamic process, where the dynamic process demonstrates the dynamic changes of at least one element. For example, such as... Figure 4B As shown, when measuring the length of a bowl, the electronic device 110 can display a dynamic process. Such a dynamic process could be, for example, presenting multiple elements from at least one piece of media content 406 in sequence. For instance, element 410-2 is presented first, followed by element 410-3, and then the connecting element between elements 410-2 and 410-3. After the above elements have been presented, the electronic device 110 can present element 410-1 to represent the length of the bowl.

[0103] In some cases, at least one element may include a second part and a third part, and the rendering dynamic process may include stopping the rendering of the second part and rendering the third part. As an example, in Figure 4B Towards Figure 4CDuring the switching process, the process of measuring the length of the bowl based on elements 410-1 to 410-4 can be considered the second part of the dynamic process, and the process of measuring the length of the table based on elements 412-1 to 412-5 and counting the objects can be considered the third part of the dynamic process. The electronic device 110 can first sequentially present elements 410-2 and 410-3 at the two boundary positions of the bowl, and present the boundary coordinates and bowl length included by elements 410-1 and 410-4. In response to the completion of the second part, the electronic device 110 stops presenting elements 410-1 to 410-4 and begins the third part: sequentially presenting elements 412-2 and 412-3 at the two boundary positions of the table, causing element 412-4 to gradually extend from the first boundary position of the table to the second boundary position, and presenting the boundary coordinates of the table, the length of the table, and the number of objects obtained based on the table length and the bowl length through elements 412-1 and 412-5.

[0104] In this way, the electronic device 110 can present a dynamic process in chronological order, first measuring the length of the bowl, then measuring the length of the table, and counting the number of objects, allowing the user to observe how the measurement results of the second part become the basis for the calculation of the third part. By stopping the presentation of the second part and presenting the third part in the same first component 408, the electronic device 110 can reuse at least one piece of media content 406 and the obtained bowl length, only updating the elements corresponding to the current stage, reducing occlusion caused by displaying too many markers at the same time, and reducing repeated loading of media content, updating of interface nodes, and separate requests for different stages.

[0105] In addition to the examples shown in the accompanying drawings, the first task may also include object matching, file page content positioning, step-by-step transitions between multiple media contents, quantity counting, or spatial graphic relationship identification. At least one element may correspond to a selection box element, outline marker, number marker, connector, direction marker, zoom-in element, measurement element, text label, or information element, depending on the specific task. The electronic device 110 may present the aforementioned elements at the corresponding positions of at least one media content according to the processing objective described in the first request, and update the style, position, or display status of the elements as the processing stage of the first task progresses.

[0106] As an example, the first task could be to identify target stars, planets, and satellites from a night sky image and to identify the galactic affiliation or orbital relationship between the identified celestial bodies. At least one element could include dot markers respectively presented at the associated locations of each celestial body, circular markers surrounding the celestial bodies, text labels for distinguishing celestial body types, and relationship lines connecting the related celestial bodies. The electronic device 110 could, upon identifying a star, present a dot marker of a first color at the center of the star and display a "Star" label near the marker; upon identifying a planet or satellite, it could present circular markers of a second or third color respectively and display a "Planet" or "Satellite" label.

[0107] Furthermore, the intelligent system can identify the relationships between celestial bodies based on their image features, relative positions, and the relationship type indicated by the first request. In response to multiple celestial bodies belonging to the same galaxy, the electronic device 110 can mark the corresponding celestial bodies with the same color or number and connect the corresponding marks sequentially with solid lines to indicate that these celestial bodies belong to the same galaxy. In response to one celestial body being a planet and another celestial body being a satellite of that planet, the electronic device 110 can present a directed connecting line from the planet's circular marker to the satellite's circular marker, and display an "orbiting" label next to the line to indicate that the satellite orbits the planet. In response to the user selecting a description of a galaxy or orbital relationship from the first information, the electronic device 110 can highlight the corresponding celestial body's marker and connecting line, and reduce the display intensity of other celestial body markers, allowing the user to verify the specific celestial body corresponding to the relationship.

[0108] As an example, the first task could be to count the number of items of the same type on a shelf. At least one element could include a selection box surrounding each item, a counting indicator displayed near the selection box, and an information element displaying the number of items on each shelf level and the total number. Electronic device 110 could start by selecting each identified item from the first shelf level and display the counting indicator from left to right. After completing the traversal of the first shelf level, it could display the number of items on that shelf level near the first shelf level, and then continue selecting and counting items on the next shelf level. In response to the completion of counting on all shelf levels, electronic device 110 could display the number of items on each level and the total number, allowing the user to verify the counting results one by one based on the selection box and the counting indicator.

[0109] As an example, the first task could be to plan the operation sequence required to complete device initialization based on the device panel image. At least one element could include node markers presented at the associated locations of the power control, mode control, and confirmation control, respectively; sequence number markers indicating the operation sequence; directed trajectories connecting adjacent node markers; and a judgment element indicating whether the operation sequence meets preset conditions. The electronic device 110 could first present a first node marker including the sequence number "1" at the power control, then present a second node marker including the sequence number "2" at the mode control, and gradually extend a directed trajectories from the first node marker to the second node marker. Subsequently, the electronic device 110 could present a third node marker including the sequence number "3" at the confirmation control, and continue to extend the directed trajectories to form an operation path that passes through the three controls in sequence.

[0110] In response to the aforementioned operation path satisfying the operational constraints of device initialization, the electronic device 110 can display a complete directed trajectory using a first color, and present a judgment element at the end of the trajectory to indicate "correct sequence". In response to a candidate operation path passing the confirmation control before the mode control, the intelligent system can determine that the candidate operation path does not meet the operational constraints. The electronic device 110 can stop extending the trajectory at the confirmation control and present a judgment element to indicate "operation order does not meet requirements". Thus, the user can not only view the specific controls that need to be operated, but also understand the order of operations for each control based on node markers, sequence numbers, and the directed trajectory.

[0111] As an example, the first task could be to identify components with the same shape as the sample from an image of a storage area containing multiple components. At least one element could include a bounding box element surrounding the sample object, multiple contour points marking the protrusions and openings of the sample object, a bounding box element surrounding the candidate components, and an association identifier indicating the matching result. The electronic device 110 could first present the bounding box element surrounding the sample object at its outer boundary, and then sequentially highlight the protrusions and openings; subsequently, it could present comparison points at corresponding positions of each candidate component. When the protrusions and openings of a candidate component correspond to the sample object, the electronic device 110 could present the candidate component with an association identifier that has the same number as the sample object; when the corresponding positions do not correspond, it could stop highlighting the candidate component and continue comparing the next candidate component.

[0112] As an example, the first task could be to locate the device name, the target field in the parameter table, and the explanatory paragraph associated with the target field in a document image. At least one element could include a selection element surrounding the device name in the document, a selection element surrounding the target field, a guide line connecting the target field and the explanatory paragraph, a numbered marker indicating the search order, and a zoom-in element displaying the field content. The electronic device 110 could first display the selection element surrounding the device name in the page title area, then scroll to the parameter table area and display the selection element surrounding the target field; after obtaining the explanatory paragraph associated with the target field, a guide line could be extended from the selection element surrounding the target field to the explanatory paragraph. In response to the user selecting the target field name from the first information, the electronic device 110 could scroll the field to the visible area and display a zoom-in element so that the user can verify the field content.

[0113] As another example, the first task can be to identify two adjacent surfaces after folding, based on the unfolded diagram of the spatial figure. At least one element may include a numbering mark presented at the center of each surface, a first connecting element indicating adjacent edges in the unfolded diagram, a direction mark indicating the folding direction, and a result element for highlighting the target surface. The electronic device 110 may first present the numbering marks for each surface sequentially, and then, starting from the reference surface, present the first connecting element and direction mark along the shared edge to indicate the folding order of the surfaces. When two surfaces are adjacent after folding, the electronic device 110 may highlight the numbering marks of these two surfaces with the same color and present the result connecting element between them, allowing the user to verify the adjacency relationship based on the folding path.

[0114] In the example presented in this paper, the intelligent system can employ a two-stage processing architecture to visualize the media content processing process. In the first stage, the intelligent system generates structured data describing the processing based on the media content and user requests. In the second stage, the electronic device 110 generates a visualization with annotation layers and animation effects based on this structured data and the original media content. This two-stage architecture allows the intelligent system to focus on understanding and reasoning about the media content, while the generation of the visualization is handled by dedicated rendering logic, thereby reducing the probability of coordinate offsets or rendering anomalies in the visualization output.

[0115] In some cases, structured data can take the form of a list of steps, a final result, and a list of media content. Each step can include a step index, a text description, a coordinate set, and an animation mode. The text description can be organized according to the structure of "observation basis, intermediate conclusions, and next step," and the coordinate set can describe the positional information of points, boxes, paths, or matched objects within the media content.

[0116] In some cases, the coordinates output by the intelligent system can adopt a unified relative coordinate system, such as relative coordinates within a predetermined integer range. Each coordinate can carry a media content index to support single-image and multi-image scenarios. When displaying elements, the electronic device 110 can convert the relative coordinates into display coordinates based on the actual size of the media content, thereby ensuring that the elements can still be accurately superimposed on the associated positions of the media content under different screen sizes and scaling ratios.

[0117] In some cases, when generating elements, the intelligent system can automatically select the corresponding animation template based on the type of processing step. For example, it can include at least one of the following animation templates: a path growth template for path tracing, a pop-up template for counting and listing, a matching connection template for pairing and connecting lines, a box selection zoom template for local box selection and zooming, a node walk template for node hop traversal, and a default template for pure conclusion steps that do not require coordinates.

[0118] In some situations, when presenting elements, the electronic device 110 can adaptively set the display parameters of the elements based on the size information of the media content, including but not limited to the radius of the annotation points, the size of the text font, the visibility of the labels, the line width of the connecting lines, and the transparency of the annotation layer. When the annotations are dense, the electronic device 110 can automatically enlarge key annotation points or hide unnecessary labels to maintain the clarity of the visual presentation.

[0119] In some cases, the processing of the aforementioned intelligent system can be performed entirely by the electronic device 110 alone. For example, the electronic device 110 can run a lightweight visual language model locally to perform understanding and reasoning on media content, generate structured data, and complete the rendering and presentation of elements locally.

[0120] In some cases, the processing of the aforementioned intelligent system can be entirely performed by the server 130. For example, the electronic device 110 sends media content and user requests to the server 130, which performs understanding and reasoning, generates structured data and rendering information, and returns the visualization results to the electronic device 110 for display.

[0121] In some cases, the processing of the aforementioned intelligent system can be jointly executed by electronic device 110 and server 130. For example, electronic device 110 can be responsible for local preprocessing and coordinate normalization of media content, while server 130 can be responsible for deep understanding and reasoning based on media content, generating structured data and returning it to electronic device 110, which then renders and presents the elements locally based on the structured data.

[0122] In some cases, the above solutions can also be applied to various scenarios, including but not limited to: visual reasoning scenarios (such as spatial reasoning, shadow matching, or stereoscopic unfolding), navigation and path planning scenarios, document analysis scenarios (such as annotating key clauses or paragraphs in document images), device operation guidance scenarios (such as annotating operation buttons and operation sequences on device interface screenshots), and data visualization scenarios (such as annotating trend changes or abnormal areas on data charts). It should be understood that the above application scenarios are merely illustrative and not restrictive.

[0123] Example process Figure 5 A flowchart of an example process 500 for interface interaction is shown, based on some scenarios. Process 500 can be implemented at electronic device 110. See below for reference. Figure 5 To describe process 500.

[0124] like Figure 5 As shown, in block 510, electronic device 110 receives a first request, which is used to instruct the intelligent system to perform a first task, and the first request represents the identification of at least one piece of media content.

[0125] In box 520, electronic device 110 presents at least one element at the associated location of at least one media content, the at least one element representing a first processing procedure of the intelligent system on at least one media content, the first processing procedure corresponding to a first stage of a first task.

[0126] In box 530, electronic device 110 outputs first information, which comes from the intelligent system and includes the execution result of the first task.

[0127] In some cases, at least one element may include a first element, which is used to mark a first object, and at least one piece of media content includes the first object.

[0128] In this way, the first element marks the first object at its associated location within the media content, allowing users to intuitively confirm the position of the object identified by the intelligent system within the media content. Consequently, the electronic device 110 does not need to provide users with a separate object list page to explain the location of each object individually, thus reducing client-side page rendering overhead and cache usage.

[0129] In some cases, the style of the first element is derived from the type of the first task.

[0130] In this way, different task types correspond to different styles of first elements, allowing users to quickly identify the processing mode of the current task through the visual style of the elements. Accordingly, the electronic device 110 selects the corresponding rendering template based on the task type, eliminating the need to load a general rendering resource package, thus reducing the client's rendering resource loading and memory usage.

[0131] In some cases, presenting at least one element may include: presenting the first element at a first position, the first position being derived from a second position, which is the position of the first object within at least one piece of media content.

[0132] In this way, the presentation position of the first element corresponds to the actual position of the first object in the media content, allowing users to verify the positional correspondence between the marker and the object. Accordingly, the coordinate mapping based on the object's position is completed by the local rendering logic, eliminating the need for the electronic device 110 to request position calibration data from the server 130, thus reducing the amount of data involved in edge-cloud communication.

[0133] In some cases, the first task can at least be used to identify an object of the first type, and the first object corresponds to the first type.

[0134] In this way, the intelligent system can identify specific types of objects based on a first request, and users can view the identification results for a specific type of object. Correspondingly, the intelligent system performs in-depth analysis only on the target type of object, reducing the computational overhead of the server 130.

[0135] In some cases, the first element may include a first counting identifier, which corresponds to a first object, and the first phase includes counting a first group of objects, which corresponds to a first type and includes the first object.

[0136] In this way, the first count identifier is directly displayed at the associated location of each object, allowing the user to check the number of each object one by one and confirm the total. Consequently, the electronic device 110 displays all count identifiers and the total number of items simultaneously in a single media content view, eliminating the need to load a separate statistics page for the counting results, thus reducing the number of page renderings and document object model refreshes on the client side.

[0137] In some cases, at least one element may also include a second element, which may characterize the first relationship between the first object and the second object.

[0138] In this way, the second element directly indicates the relationship between objects on the media content in the form of connecting lines, allowing users to identify associated objects without reading a text list. Consequently, the relationship visualization is rendered directly overlaid on the media content, eliminating the need for the electronic device 110 to load separate relationship details for each pair of associated objects, thus reducing rendering overhead on the client and query load on the server 130.

[0139] In some cases, the first phase may include identifying objects that satisfy a first relationship, which is obtained based on a first request.

[0140] In this way, the intelligent system automatically identifies objects that satisfy the relationship based on the relationship type obtained from the first request, without requiring the user to manually specify pairing rules. Accordingly, the relationship identification is completed by the server 130 in one go and returned through structured data, eliminating the need for the electronic device 110 to perform multiple rounds of interactive queries, thus reducing the number of interaction rounds between the end and the cloud and the consumption of communication bandwidth.

[0141] In some cases, the second element is used to connect the first object and the second object.

[0142] In this way, the second element explicitly indicates the relationship between the two objects in the form of a connection, allowing users to intuitively identify the associated objects along the connection line. Correspondingly, the rendering of the connection element is completed locally based on the coordinate information of the two objects, eliminating the need for the electronic device 110 to request additional relationship display data from the server 130, thus reducing the amount of data involved in edge-cloud communication.

[0143] In some cases, the second element includes a directed connection element, the direction of which represents the first relation.

[0144] In this way, directed connection elements represent asymmetric relationships between objects through directional information, allowing users to visually distinguish the directionality of the relationship. Accordingly, the directional information is contained in the coordinate set of structured data, and the electronic device 110 only needs to add directional markers during local rendering without additional data requests, reducing the amount of data in edge-cloud communication.

[0145] In some cases, at least one type of media content may include a first type of media content and a second type of media content, wherein the first type of media content includes a first object and the second type of media content includes a second object.

[0146] In this way, the intelligent system can establish object relationships across multiple media content sets. Users can confirm the object correspondences across media content sets in a single interface, without having to manually compare multiple media content sets. Consequently, the electronic device 110 simultaneously presents multiple media content sets and their relationship markers on the same interface, reducing page load times and memory usage.

[0147] In some cases, the first element can characterize the first trajectory, which is obtained based on the distribution of the first object.

[0148] In this way, the first element presents path information based on object distribution in the form of a trajectory, allowing users to track the reasoning process along the visual path. Correspondingly, trajectory rendering is completed locally based on the coordinate sequence in the structured data, eliminating the need for the electronic device 110 to request rendering data separately for each path node, thus reducing the number of edge-cloud interaction rounds.

[0149] In some cases, the first request is used to identify at least one trajectory that satisfies the conditions, and at least one element may also include a third element that can characterize whether the first trajectory satisfies the conditions.

[0150] In this way, the third element clearly visually indicates whether the trajectory meets the conditions, allowing users to easily distinguish between feasible and infeasible paths. Consequently, the condition determination result is directly included in the structured data, and the electronic device 110 can display the determination mark during local rendering without initiating additional condition verification requests to the server 130, thus reducing the amount of data in edge-cloud communication and the processing load on the server 130.

[0151] In some cases, at least one element may display a magnified view of the first part, which represents the intermediate processing result of the first stage, and at least one piece of media content includes the first part.

[0152] In this way, the first element displays a magnified view directly at the relevant location within the media content, allowing users to view details without using zoom gestures or navigating to a details page. Correspondingly, the magnified view is cropped and scaled from the media content by local rendering logic, eliminating the need to request high-resolution local images from the server, thus reducing data transmission volume in edge-cloud communication.

[0153] In some cases, at least one element is a first set of elements, and the method may also include: presenting a second set of elements at an associated location of at least one piece of media content, the second set of elements representing a second phase of the first task.

[0154] In some cases, presenting the second set of elements includes: presenting the first component, which displays at least one piece of media content; and stopping the presentation of the first set of elements in the first component and presenting the second set of elements in the first component.

[0155] In this way, the first component sequentially replaces and presents different groups of elements at different stages, allowing the user to observe the stage evolution of the processing within the same visual container. Accordingly, the replacement of element groups is completed by the local rendering logic based on the stage index, eliminating the need for the electronic device 110 to reload media content or re-request rendering data from the server 130 for stage switching, thus reducing the client's page rendering overhead and the amount of data exchanged between the client and the cloud.

[0156] In some cases, the first phase may include processing a first set of media content, and the second phase may include processing a second set of media content, wherein at least one set of media content includes both the first and second sets of media content, and the first set of media content is different from the second set of media content.

[0157] In this way, different media content is processed at different stages, and the processing results of each stage are presented separately on the interface, allowing users to clearly compare the individual processing procedures of each group of media content. Accordingly, the electronic device 110 presents the results of each stage side by side in different components of the same interface, eliminating the need to switch between multiple separate pages, thus reducing the number of page loads and rendering overhead for the client.

[0158] In some cases, presenting at least one element may include: displaying at least one element through a dynamic process, wherein the dynamic process displays the dynamic changes of at least one element.

[0159] In this way, the dynamic process is presented as an animation of element changes, allowing users to observe the temporal sequence and evolutionary logic of the intelligent system's processing. Accordingly, the animation rendering is completed by a local animation template based on the step sequence in the structured data, eliminating the need for the electronic device 110 to request rendering data from the server 130 for each animation frame, thus reducing the number of edge-cloud interaction rounds and the amount of communication data.

[0160] In some cases, at least one element may include a second part and a third part, and the rendering dynamic process may include: stopping the rendering of the second part and rendering the third part.

[0161] In this way, by stopping the presentation of the second part and presenting the third part, the user can observe the phased changes of elements within the same visual area. Accordingly, the switching of element parts is handled by local rendering logic, and the electronic device 110 does not need to reload media content for partial switching, reducing the number of document object model refreshes and rendering overhead on the client side.

[0162] In some cases, at least one element includes a fourth element, the electronic device 110 can receive a first operation, the first operation being used to trigger a fourth part of the first information, the fourth part describing the processing associated with the fourth element; and the fourth element is presented in a first style.

[0163] In this way, users can start from any part of the execution result and trigger the corresponding element to be presented in the first style through operation, establishing a two-way linkage between the result text and the visual markup of the media content. Correspondingly, the style switching is completed by the local rendering logic based on the step index in the structured data, without the need to initiate additional association query requests to the server 130, reducing the amount of data in the end-to-cloud communication and the processing load of the server 130.

[0164] In some cases, receiving a first request may include: presenting an input component for interacting with an intelligent system; and receiving a first request via the input component, the first request including at least one piece of media content and natural language content, the natural language content describing the requirements of a first task.

[0165] In this way, users submit both media content and a natural language description of their task requirements simultaneously through the input component, enabling the intelligent system to perform targeted processing of the media content based on the user's intent. Consequently, the electronic device 110 only needs to send one request containing the media content and task description to the server 130, reducing the number of edge-cloud interaction rounds and lowering the bandwidth consumption of edge-cloud communication.

[0166] In some cases, presenting at least one element may include: acquiring structured data that describes a first processing procedure; and presenting at least one element at a relevant location in at least one piece of media content based on the structured data.

[0167] In this way, the presentation of elements is based on structured data describing the processing, ensuring that the visualization is consistent with the reasoning process of the intelligent system. Accordingly, the structured data, as an intermediate representation, decouples reasoning and rendering, allowing the rendering logic of the electronic device 110 to run independently of the reasoning logic of the intelligent system. This reduces the probability of rendering anomalies and the number of times a re-request is needed due to rendering errors.

[0168] In some cases, presenting at least one element at an associated location of at least one piece of media content based on structured data includes: generating rendering information based on structured data, wherein the rendering information characterizes the rendering style of at least one element; and presenting at least one element based on the rendering information.

[0169] In this way, rendering information is generated based on structured data before elements are presented, decoupling the generation of rendering styles from the inference output of the intelligent system. Accordingly, the generation of rendering information can be completed by a local rendering template, and the electronic device 110 does not need to delegate the rendering logic to the server 130, reducing the amount of data exchanged between the end and the cloud and the computational load on the server 130.

[0170] In some cases, presenting at least one element may include: obtaining size information of at least one media content; obtaining at least one display parameter of the at least one element based on the size information; and presenting the at least one element based on the at least one display parameter.

[0171] In this way, annotations can be accurately superimposed on the relevant positions of the media content even under different screen sizes and scaling ratios. Accordingly, the determination of display parameters is completed by the local rendering logic based on the media content size, and the electronic device 110 does not need to request rendering parameters for different screen sizes from the server 130, thus reducing the amount of data in edge-cloud communication.

[0172] Example devices and equipment A corresponding apparatus for implementing the above methods or processes is also provided. Figure 6 Block diagrams of a device 600 for user interface interaction are shown in several scenarios. Device 600 can be implemented as or included in electronic device 110. The various modules / components in device 600 can be implemented by hardware, software, firmware, or any combination thereof.

[0173] like Figure 6 As shown, the device 600 includes a receiving module 610 configured to receive a first request, the first request being used to instruct the intelligent system to perform a first task, the first request representing the identification of at least one piece of media content; a first presentation module 620 configured to present at least one element at an associated location of at least one piece of media content, the at least one element representing a first processing procedure of the intelligent system on at least one piece of media content, the first processing procedure corresponding to a first stage of the first task; and an output module 630 configured to output first information, the first information being from the intelligent system, the first information including the execution result of the first task.

[0174] In some cases, the first presentation module 620 may also be configured such that at least one element includes a first element, the first element being used to mark a first object, and at least one piece of media content includes the first object.

[0175] In some cases, the first presentation module 620 can also be configured such that the style of the first element is derived from the type of the first task.

[0176] In some cases, the first presentation module 620 may also be configured to present at least one element including: presenting the first element at a first position, the first position being derived from a second position, the second position being the position of the first object in at least one piece of media content.

[0177] In some cases, the first task is used to identify at least an object of the first type, and the first object corresponds to the first type.

[0178] In some cases, the first element includes a first counting identifier, which corresponds to a first object, and the first phase includes counting a first group of objects, which corresponds to a first type and includes the first object.

[0179] In some cases, at least one element also includes a second element, which represents the first relationship between the first object and the second object.

[0180] In some cases, the first phase includes identifying objects that satisfy a first relationship, which is obtained based on a first request.

[0181] In some cases, the second element is used to connect the first object and the second object.

[0182] In some cases, the second element includes a directed connection element, the direction of which represents the first relation.

[0183] In some cases, at least one media content includes a first media content and a second media content, wherein the first media content includes a first object and the second media content includes a second object.

[0184] In some cases, the first element represents the first trajectory, which is obtained based on the distribution of the first object.

[0185] In some cases, the first request is used to identify at least one trajectory that satisfies the conditions, and at least one element also includes a third element that characterizes whether the first trajectory satisfies the conditions.

[0186] In some cases, at least one element displays a magnified view of the first part, the magnified view representing the intermediate processing result of the first stage, and at least one piece of media content includes the first part.

[0187] In some cases, at least one element is a first group of elements, and the first presentation module 620 can also be configured to present a second group of elements at the associated location of at least one media content, the second group of elements representing the second stage of the first task.

[0188] In some cases, the first presentation module 620 may also be configured to: present a first component that displays at least one piece of media content; and stop presenting a first set of elements in the first component and present a second set of elements in the first component.

[0189] In some cases, the first phase includes processing a first set of media content, the second phase includes processing a second set of media content, and at least one set of media content includes both the first and second sets of media content, with the first set of media content being different from the second set of media content.

[0190] In some cases, the first presentation module 620 can also be configured to: display at least one element through a dynamic process, wherein the dynamic process displays the dynamic changes of at least one element.

[0191] In some cases, at least one element includes a second part and a third part, and the first presentation module 620 can also be configured to: stop presenting the second part and present the third part.

[0192] In some cases, at least one element includes a fourth element, and the receiving module 610 may also be configured to receive a first operation, the first operation being used to trigger a fourth part of the first information, the fourth part describing the processing associated with the fourth element; and to present the fourth element in a first style.

[0193] In some cases, the receiving module 610 may also be configured to: present an input component for interacting with the intelligent system; and receive a first request through the input component, the first request including at least one piece of media content and natural language content, the natural language content describing the requirements of the first task.

[0194] In some cases, the first presentation module 620 may also be configured to: acquire structured data describing the first processing procedure; and, based on the structured data, present at least one element at the associated location of at least one piece of media content.

[0195] In some cases, the first presentation module 620 may also be configured to: generate rendering information based on structured data, the rendering information representing the rendering style of at least one element; and present at least one element based on the rendering information.

[0196] In some cases, the first presentation module 620 may also be configured to: acquire size information of at least one piece of media content; acquire at least one display parameter of at least one element based on the size information; and present at least one element based on the at least one display parameter.

[0197] The modules included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some cases, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 600 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0198] Figure 7 A block diagram of an electronic device 700 in which one or more examples may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein. Figure 7 The illustrated electronic device 700 can be used to implement the electronic device 110 discussed above.

[0199] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processing units or processors 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processor 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0200] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0201] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various examples.

[0202] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0203] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0204] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0205] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0206] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0207] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0208] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0209] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for interface interaction, comprising: Receive a first request, the first request being used to instruct the intelligent system to perform a first task, the first request representing the identification of at least one piece of media content; At the associated location of the at least one media content, at least one element is presented, the at least one element representing a first processing procedure of the intelligent system on the at least one media content, the first processing procedure corresponding to a first stage of the first task; and Output first information, which comes from the intelligent system, including the execution result of the first task.

2. The method of claim 1, wherein the at least one element includes a first element for marking a first object, and the at least one media content includes the first object.

3. The method of claim 2, wherein presenting the at least one element comprises: The first element is presented at a first position, which is based on a second position, which is the position of the first object within the at least one piece of media content.

4. The method of claim 2, wherein the first task is at least used to identify an object of a first type, the first object corresponding to the first type.

5. The method of claim 4, wherein the first element includes a first counting identifier corresponding to the first object, and the first stage includes counting a first group of objects corresponding to the first type, the first group of objects including the first object.

6. The method of claim 2, wherein the at least one element further comprises a second element, the second element representing a first relationship between the first object and the second object.

7. The method of claim 6, wherein the first stage includes identifying objects that satisfy the first relationship, the first relationship being obtained based on the first request.

8. The method of claim 6, wherein the at least one media content includes first media content and second media content, the first media content including the first object, and the second media content including the second object.

9. The method of claim 2, wherein the first element represents a first trajectory, the first trajectory being obtained based on the distribution of the first object.

10. The method of claim 9, wherein the first request is used to identify at least one trajectory that satisfies the condition, and the at least one element further includes a third element, the third element characterizing whether the first trajectory satisfies the condition.

11. The method of claim 1, wherein the at least one element displays a magnified view of the first portion, the magnified view representing an intermediate processing result of the first stage, and the at least one media content includes the first portion.

12. The method of claim 1, wherein the at least one element is a first group of elements, the method further comprising: At the associated location of the at least one piece of media content, a second set of elements is presented, the second set of elements representing the second phase of the first task.

13. The method of claim 12, wherein presenting the second set of elements comprises: A first component is presented, which displays the at least one piece of media content; as well as Stop displaying the first group of elements in the first component, and display the second group of elements in the first component.

14. The method of claim 12, wherein the first stage includes processing a first set of media content, the second stage includes processing a second set of media content, and the at least one set of media content includes the first set of media content and the second set of media content, wherein the first set of media content is different from the second set of media content.

15. The method of claim 1, wherein the presentation of at least one element comprises: The at least one element is displayed through a dynamic process, which demonstrates the dynamic changes of the at least one element.

16. The method of claim 15, wherein the at least one element comprises a second portion and a third portion, and the dynamic presentation process comprises: Stop displaying the second part and display the third part.

17. The method of claim 1, wherein the at least one element comprises a fourth element, and the method further comprises: Receive a first operation, the first operation being used to trigger a fourth part of the first information, the fourth part being used to describe the processing procedure associated with the fourth element; as well as The fourth element is presented in the first style.

18. A device for interface interaction, comprising: The receiving module is configured to receive a first request, the first request being used to instruct the intelligent system to perform a first task, the first request representing an identifier for at least one piece of media content; A first presentation module is configured to present at least one element at an associated location of the at least one media content, the at least one element representing a first processing procedure of the intelligent system on the at least one media content, the first processing procedure corresponding to a first stage of the first task; as well as The output module is configured to output first information, which comes from the intelligent system and includes the execution result of the first task.

19. An electronic device comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 17 when executed by the at least one processor.

20. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 17.

Images