Information interaction method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses an information interaction method and device, electronic equipment, a storage medium and a program product, and belongs to the technical field of augmented reality. The method comprises the following steps: obtaining interaction behavior information generated in an interaction interface of the augmented reality equipment, wherein the interaction behavior information comprises watching duration of watching at least one display object in a display window of the interaction interface; under the condition that the interactive behavior information represents that the gaze duration reaches a preset gaze duration, obtaining space attribute information of the display window in the interactive interface, semantic information of the display window and use preference information of the user for the augmented reality device, the semantic information being determined according to type information and state information of the display window; the space attribute information is determined according to size information and position information of the display window; target interaction information is determined according to the interaction behavior information, the spatial attribute information, the semantic information and the use preference information, and the target interaction information is associated with the operation intention of the user; and displaying the target interaction information on the interaction interface.

Description

Technical Field

[0001] The present application belongs to the field of extended reality technology, and specifically relates to an information interaction method, device, electronic device, storage medium and program product. Background Art

[0002] With the development of science and technology, more and more users are using extended reality (XR) devices for online interaction. The interactive interface of existing XR devices mainly follows the design ideas of mobile platforms such as mobile phones, and tends to fold the operation instructions into the interface elements to achieve an overall simple design style. Although this design improves the visual experience, it increases the complexity of the operation to a certain extent.

[0003] Currently, users can operate XR devices through handles, gestures, and eyeballs. Since the operation instructions are usually folded into the interface elements, the user's operation is cumbersome, which reduces the interaction efficiency and accelerates the user's hand and eye fatigue. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an information interaction method, device, electronic device, storage medium and program product, which can solve the problem of cumbersome user operations, thereby reducing interaction efficiency and accelerating user hand-eye fatigue.

[0005] In a first aspect, an embodiment of the present application provides an information interaction method, the method comprising:

[0006] Acquire interactive behavior information generated in an interactive interface of an extended reality device, the interactive interface comprising at least one display window, the interactive behavior information being interactive process information of the at least one display window, the interactive behavior information comprising a gaze duration of each of at least one display object in the display window;

[0007] When the interaction behavior information indicates that the gaze duration reaches a preset gaze duration, obtaining spatial attribute information of the display window on the interaction interface, semantic information of the display window, and the user's usage preference information for the extended reality device, wherein the semantic information is determined according to type information and state information of the display window, and the spatial attribute information is determined according to size information and position information of the display window;

[0008] Determining target interaction information according to the interaction behavior information, the spatial attribute information, the semantic information and the usage preference information, wherein the target interaction information is associated with the operation intention of the user;

[0009] The target interaction information is displayed on the interaction interface.

[0010] In a second aspect, an embodiment of the present application provides an information interaction device, the device comprising:

[0011] A first acquisition module is used to acquire interaction behavior information generated in an interaction interface of an extended reality device, wherein the interaction interface includes at least one display window, and the interaction behavior information is interaction process information of the at least one display window, wherein the interaction behavior information includes a gaze duration of each of at least one display object in the display window;

[0012] A second acquisition module is used to acquire, when the interaction behavior information indicates that the gaze duration reaches a preset gaze duration, spatial attribute information of the display window on the interaction interface, semantic information of the display window, and usage preference information of the user for the extended reality device, wherein the semantic information is determined according to type information and state information of the display window, and the spatial attribute information is determined according to size information and position information of the display window;

[0013] a determination module, configured to determine target interaction information according to the interaction behavior information, the spatial attribute information, the semantic information and the usage preference information, wherein the target interaction information is associated with the operation intention of the user;

[0014] A display module is used to display the target interaction information on the interaction interface.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0019] In an embodiment of the present application, since the target interaction information is associated with the user's operation intention, by acquiring the interaction behavior information generated in the interaction interface of the extended reality device, and when the interaction behavior information represents that the user's gaze time on a display object in the display window reaches a preset gaze time, the target interaction information is determined according to the interaction behavior information, the spatial attribute information of the display window in the interaction interface, semantic information and the user's usage preference information for the extended reality device, and the target interaction information is displayed in the interaction interface. That is, by predicting the user's next operation intention during the process of the user interacting with the extended reality device, and displaying the target interaction information associated with the operation intention, the interaction result can be directly displayed, or the user can interact with the extended reality device based on the target interaction information, thereby simplifying the user's interaction operation, improving the interaction efficiency, and alleviating the user's hand-eye fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of an information interaction method provided in an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of determining activation weights provided in an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of determining the association relationship between two modal information based on a cross-attention mechanism provided by an embodiment of the present application;

[0023] Figure 4 is another schematic diagram of determining the association relationship between two modal information based on the cross attention mechanism provided by an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of determining target interaction information using a multimodal large model provided by an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of an interactive interface provided in an embodiment of the present application;

[0027] Figure 8 is a structural diagram of an information interaction device provided in an embodiment of the present application;

[0028] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0029] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0032] For existing XR interaction designs, most control methods can be summarized into two operation primitives: "navigation" and "selection". Common "navigation" operation implementation forms include: ray navigation based on handle tracking, motion navigation based on gesture tracking, motion navigation based on eye tracking, etc.; common "selection" operation implementation forms include: button selection based on handle, motion selection based on gesture recognition, etc.

[0033] Whether it is navigation operations that rely on controller tracking, gesture tracking, and eye tracking, or selection operations that rely on controller buttons and gesture recognition, the underlying logic relies on frequent movements of "hands" and "eyes". Based on this, operating XR devices through eye-hand interaction for a long time will greatly cause eye and arm fatigue to users.

[0034] In addition, as described in the background technology section, the interactive interface of existing XR devices mainly follows the design ideas of mobile platforms such as mobile phones, and tends to fold the operation instructions into the interface elements to achieve an overall simple design style. Although this design tendency improves the visual experience, it increases the complexity of the operation to a certain extent. For example, "copy text from the reader to the browser for search", the process goes through: a) the handle operates the cursor to select the text; b) long press the key to trigger the menu and click the "copy" command; c) select the browser address bar; d) long press the key again to trigger the menu and click the "paste" command; e) click the "search" button and a series of actions. Even if the eye tracking capability is integrated, it mainly simplifies the "handle movement" type of operation, and the overall link has not been optimized much in terms of interaction efficiency, which will accelerate user eye and hand fatigue.

[0035] Based on this, in order to solve the problems of the prior art, the embodiments of the present application provide an information interaction method, device, electronic device, readable storage medium and computer program product. Among them, the information interaction method can be applied to extended reality scenarios, and can also be applied to digital intelligent promotion (such as advertising, services, etc.) and natural intuitive interaction in the field of robotics, etc., without limitation here.

[0036] The information interaction provided by the embodiments of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0037] Figure 1 FIG. 1 is a flow chart of an information interaction method provided by an embodiment of the present application. Figure 1 As shown, the information interaction method provided in the embodiment of the present application includes the following steps S110-S140, and each step is explained in detail below.

[0038] S110, obtaining interactive behavior information of the user in the interactive interface of the extended reality device, where the interactive interface includes at least one display window, and the interactive behavior information is interactive process information of the at least one display window, and the interactive behavior information includes the gaze duration of each of at least one display object in the display window.

[0039] Here, multiple applications can be installed in the Extended Reality (XR) device. A display window corresponding to the application can be displayed in the interactive interface. Display objects such as text labels, text boxes, images, and controls can be displayed in the display window. Among them, the text box can be described using text near the user's viewpoint. The image can be described using the region of interest (POI) near the user's viewpoint. The control can correspond to functions such as "confirm", "cancel", and "share".

[0040] From the perspective of time, interactive behavior information may include the user's interactive behavior from turning on the XR device to obtaining the interactive behavior information. From the perspective of interaction type, interactive behavior information may include viewpoint information and target operation behavior. Among them, viewpoint information may include the user's gaze order and gaze duration on the objects displayed in the display window. Viewpoint information may belong to the "navigation operation" information mentioned above. Target operation behavior may include the user's selection operation on the control. Target operation behavior may belong to the "selected operation" mentioned above. From the perspective of information type, interactive behavior information may include display object sequence, dwell time information, and operation fragments.

[0041] Based on this, in order to improve the accuracy of subsequent interactive behavior prediction based on the interactive behavior information, in some embodiments, the above S110 may specifically include:

[0042] receiving a first input from a user on a display object in at least one display window;

[0043] In response to a first input, acquiring a display object sequence corresponding to a plurality of display objects, a gaze duration for each display object, and a target operation behavior for operating at least one display window;

[0044] Determine the user's stay time information for each display object according to the gaze time corresponding to each display object within the first preset time;

[0045] According to the target operation behavior, the display object sequence is divided into multiple operation segments;

[0046] The interactive behavior information is determined based on the display object sequence, dwell time information, and operation fragments.

[0047] Here, the first input includes but is not limited to: gaze input by the user through the eyeball, touch input of the user to the display object through a touch device such as a finger or a stylus, voice commands input by the user, specific gestures input by the user, other feasible inputs, etc. The first input can be determined according to actual usage needs, and the embodiments of the present application are not limited. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a dragging gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. Based on this, the above-mentioned first input can, for example, be: a user's browsing input of a display object in at least one display window.

[0048] The display object sequence may represent an interaction sequence generated in the order of input when the user performs a first input on multiple display objects. In addition, the target operation behavior of the user operating the display window may specifically be an operation of the user selecting a control in the display window. The selection operation may include, for example, operations such as the user clicking a "share" control, clicking an "edit" control, and clicking a "confirm" control.

[0049] As an example, by monitoring the user's interactive behavior, a display object sequence can be generated. The display object sequence can be recorded as: M1-viewpoint moves out of the text label | M2-viewpoint moves into the image display area | M3-viewpoint hovers near the image (0.2, 0.4) | M4-button triggers the menu | M5-presses the "share" button | M6-viewpoint moves out of the image area | M7-viewpoint moves into the browser search box | M8-button input text | M9-button triggers search. Among them, M1-M4, M6-M8 can belong to the "move" and "select" interface level operations that do not involve background functions. That is, M1-M4, M6-M8 can belong to the "navigation operation" mentioned above. In addition, M5 can be an operation for "triggering the sharing function", and M9 can be an operation for "triggering the search function". M5 and M9 can belong to the "selection operation" involving background functions. That is, the target operation behavior can include M5 and M9 in the above example.

[0050] As an example, if the display object sequence is long, in order to better predict the user's next interactive behavior based on the display object sequence, that is, predict the user's operation intention, the display object sequence can be split into multiple segments. Specifically, since "navigation operation" is a prerequisite for "selection operation" and "selection operation" is the ultimate goal of "navigation operation", the display object sequence can be split using the target operation behavior as an operation separator to obtain multiple operation segments. The operation segment can be the minimum operation sequence of moving and navigating to "selection operation". The length of the operation segment can reflect the complexity of the operation. The longer the operation segment, the higher the complexity of the operation segment can be, and the higher the importance of the operation segment to the subsequent interactive behavior prediction can be. On the contrary, the shorter the operation segment, the lower the complexity of the operation segment can be, and the lower the importance of the operation segment to the subsequent interactive behavior prediction can be. Based on the above example, the operation segment can include M1-M5 and M6-M9.

[0051] In addition, the first preset duration may be a preset time window, such as 10 minutes. The dwell duration information may indicate the proportion of the gaze duration corresponding to each display object within the first preset duration, or the proportional relationship between the gaze durations corresponding to multiple display objects. The dwell duration information may be represented by a sliding histogram. The dwell duration information may reflect the user's attention to the display window.

[0052] As an example, by monitoring the user's interactive behavior, the user's gaze duration on each displayed object can be obtained. By statistically analyzing multiple gaze durations within a first preset duration, the user's stay duration information on each displayed object can be obtained.

[0053] As a more specific example, the gaze point is sampled at a preset frequency (such as 10 Hz) within a first preset time length (such as 10 minutes), and the proportion of each display window that the gaze point has focused on within the first preset time length is counted to obtain a sliding histogram.

[0054] In this way, by determining the interactive behavior information based on a longer display object sequence, a shorter operation segment and the dwell time information obtained by statistical analysis of each display object, the interactive behavior information can be made richer, thereby improving the accuracy of subsequent interactive behavior prediction based on the interactive behavior information.

[0055] S120, when the interaction behavior information represents that the gaze duration reaches a preset gaze duration, obtain spatial attribute information of the display window in the interaction interface, semantic information of the display window, and user preference information for the extended reality device, wherein the semantic information is determined based on the type information and status information of the display window, and the spatial attribute information is determined based on the size information and position information of the display window.

[0056] Here, if the user's gaze duration on a displayed object reaches a preset gaze duration, the user's next interaction behavior can be predicted based on the interaction behavior information generated by the user's interaction with the extended reality device.

[0057] In order to improve the accuracy of predicting the user's next interactive behavior, before predicting the interactive behavior, the spatial attribute information of the display window in the interactive interface, the semantic information of the display window and the user's usage preference information for the XR device can also be obtained, and the interactive behavior can be predicted based on the above-mentioned interactive behavior information, spatial attribute information, semantic information and usage preference information and other multimodal information. Among them, from the perspective of time dimension, spatial attribute information and semantic information can belong to instant information, interactive behavior information can belong to short-term information, and usage preference information can belong to long-term information.

[0058] The spatial attribute information of the display window on the interactive interface may include size information and position information of the display window on the interactive interface.

[0059] Based on this, in order to improve the prediction efficiency of subsequent prediction of interactive behaviors, in some embodiments, the above-mentioned acquisition of spatial attribute information of the display window in the interactive interface may specifically include:

[0060] Get the first size information and anchor point information of the display window in the interactive interface;

[0061] Mapping the first size information into the target coordinate system to obtain second size information;

[0062] Project the anchor point corresponding to the anchor point information onto the unit sphere to obtain the anchor point pose information;

[0063] The spatial attribute information is determined according to the second size information and the anchor point pose information.

[0064] Here, the first size information may be the actual size information of the display window in the interactive interface. The second size information may be the size information obtained after mapping the first size information to the target coordinate system. The second size information may be the same as or different from the first size information. Among them, the target coordinate system may be, for example, a normalized device coordinate system (Normalized Device Coordinates, NDC). The first size information of different display windows in the interactive interface may be in different data dimensions. Therefore, in order to improve the standardization of subsequent data processing processes, the first size information may be mapped to the target coordinate system, that is, the first size information is standardized to obtain the second size information. In addition, the larger the size of the display window, the higher the user's attention to the display window. The smaller the size of the display window, the lower the user's attention to the display window.

[0065] In addition, the anchor point information may be the position information of the anchor point of the display window. The unit sphere may be a sphere with a radius of 1. The anchor point pose information may be expressed using a quaternion. By uniformly projecting the anchor points onto the unit sphere, the anchor point pose information is obtained, and the position information of the anchor points may be standardized to improve the standardization of subsequent data processing processes. If the anchor point pose information indicates that the anchor point is at the main viewing angle in front of the unit sphere, the importance of the display content in the display window corresponding to the anchor point may be higher. If the anchor point pose information indicates that the anchor point is at the edge of the unit sphere, the importance of the display content in the display window corresponding to the anchor point may be lower.

[0066] In this way, by determining the spatial attribute information based on the second size information and the anchor point posture information after the standardization process, the standardization of the spatial attribute information can be improved, thereby improving the prediction efficiency of the subsequent prediction of the interactive behavior.

[0067] In addition, the semantic information of the display window can be determined based on the type information and state information of the display window. Among them, the type information can be determined based on the type of content displayed in the display window. The type information can be expressed in phrases. The type information can include dynamic types such as games and videos and static types such as pages and texts. When predicting interactive behaviors, the importance of dynamic types can be higher than that of static types. In addition, the state information can include that the display window is in an activated state or that the display window is in an inactivated state. That is, the display window can include a first display window whose state information is in an activated state and at least one second display window whose state information is in an inactivated state.

[0068] Based on this, in order to improve the accuracy of subsequent interactive behavior prediction, in some embodiments, the semantic information of the display window may specifically include:

[0069] Get the type information and status information of the display window, where the status information is used to indicate whether the display window is activated;

[0070] When the type information is text information, the type information is digitized to obtain a digitized result corresponding to the type information;

[0071] Get the activation weight corresponding to the state information;

[0072] The semantic information is determined based on the numerical results and activation weights.

[0073] Here, the numerical processing of the type information may be to map the text information to numerical information to obtain a numerical result. For example, "game" may be mapped to "10" and "page" may be mapped to "5". The numerical value may indicate the importance of the type information. The larger the numerical value, the higher the importance of the type information in the subsequent interactive behavior prediction.

[0074] In addition, if the display window is in an activated state, the activation weight of the display window may be 1; if the display window is in an inactivated state, the activation weight of the display window may be 0. However, in actual situations, even if the display window is in an inactivated state, its data still contributes to the prediction of the interactive behavior.

[0075] Therefore, in order to improve the accuracy of subsequent interactive behavior prediction, in some embodiments, the above-mentioned acquisition of the activation weight corresponding to the state information may specifically include:

[0076] Acquire a window distance between the first display window and each second display window to obtain at least one window distance;

[0077] An activation weight corresponding to each display window is determined according to at least one window distance.

[0078] Usually, the first display window is the display window selected by the user. The second display window may be the display window not selected by the user. The display window not selected by the user may be a "foreground" window or a "background" window. The "foreground" window may be closer to the first display window. The "background" window may be farther from the first display window.

[0079] As an example, if the interactive interface includes a first display window and two second display windows A and B, and the window distance between the first display window and the second display window A is 3, and the window distance between the first display window and the second display window B is 5, then the activation weight corresponding to the second display window A may be 3 / 8, and the activation weight corresponding to the second display window B may be 5 / 8. In addition, the activation weight corresponding to the first display window may be 1. The higher the activation weight, the higher the user's attention to the display window.

[0080] The embodiment of the present application determines the activation weight according to the window distance, so as to retain the contribution of the second display window to the interactive behavior prediction, thereby improving the accuracy of subsequent interactive behavior prediction.

[0081] Based on this, in order to improve the accuracy of the window distance and further improve the accuracy of subsequent interactive behavior prediction, in some embodiments, the above-mentioned obtaining the window distance between the first display window and each second display window may specifically include:

[0082] Acquire a first projection of an anchor point of the first display window on the unit sphere and a second projection of the second display window on the unit sphere;

[0083] The spherical distance between the first projection and each second projection is determined as the window distance.

[0084] like Figure 2 As shown, taking the interactive interface including four display windows as an example, 200 may represent the spherical surface of the unit sphere, 201 may represent the first projection, 202 may represent the second projection, and 203 may represent the window distance d i Wherein, the value of i can range from 0 to N, and N can represent the number of second display windows. d0 can be 0, representing the spherical distance between the first projection and the first projection, and d1, d2, and d3 can represent the spherical distance between the first projection and the three second projections, respectively.

[0085] By determining the window distance according to the spherical distance between the projections of the anchor points of the display window on the unit sphere, the accuracy of the window distance can be improved, and the accuracy of subsequent interactive behavior prediction can be further improved.

[0086] In addition, after determining at least one window distance, the activation weight can be calculated as described in the above example, or the activation weight can be calculated by the following formula:

[0087] in,

[0088] In the above formula, w iIt can be an activation weight, and λ can be a configurable hyperparameter, which is reserved for developers to make customized adjustments to control the overall weight. Determining the activation weight through the above formula can improve the scientific nature of the activation weight and further improve the accuracy of subsequent interactive behavior prediction.

[0089] In summary, by determining semantic information according to numerical results and activation weights, and then predicting interactive behaviors based on semantic information, the accuracy of interactive behavior prediction can be improved.

[0090] In addition, the user's usage preference information for the XR device may include the user's usage preference information for multiple applications in the XR device. The usage preference information may indicate that the user prefers to use a certain application in the XR device, and the time period when the user prefers to use a certain application.

[0091] Based on this, in order to further improve the accuracy of interactive behavior prediction, in some embodiments, the above-mentioned obtaining of the user's usage preference information for the extended reality device may specifically include:

[0092] Obtaining usage ratio information of multiple applications in the extended reality device by the user within a second preset time period and usage frequency information of the target application by the user within multiple time periods, where the second preset time period includes multiple time periods, and the target application is any one of the multiple applications;

[0093] The usage preference information is determined based on the usage ratio information and the usage frequency information.

[0094] Here, the usage ratio information can indicate the proportion of the user's usage time of the application to the second preset time. The usage frequency information can indicate the frequency of the user's use of the target application within the target time period. The usage ratio information and the usage frequency information can be regarded as prior knowledge under long-term statistics (i.e., the second preset time). The second preset time can be, for example, two weeks to one month. The operating system of the extended reality device can record events such as the opening, switching, and stopping of the application, and then track the usage time information of the application within the second preset time. By performing statistical analysis on the usage time information of multiple applications within the second preset time, the usage ratio information of multiple applications can be determined. The greater the user's usage ratio of the application within the second preset time, it can be indicated that the user has a greater demand for the application. The smaller the user's usage ratio of the application within the second preset time, it can be indicated that the user has a smaller demand for the application.

[0095] In addition, considering that most users' daily activities have certain regularity, the second preset duration can be divided according to time periods. By counting the frequency of users' use of a certain application in each time period, users' use habits of the application can be counted. The higher the frequency of use of a certain application in a certain time period, the more important the application is in that time period.

[0096] In this way, by determining the usage preference information based on the usage ratio information and the usage frequency information, it is possible to fully consider the user's usage needs and usage habits for the XR device when predicting interactive behaviors, thereby further improving the accuracy of interactive behavior predictions.

[0097] S130, determining target interaction information according to the interaction behavior information, the spatial attribute information, the semantic information and the usage preference information, where the target interaction information is associated with the user's operation intention.

[0098] The target interaction information may characterize the user's next interaction behavior. The target interaction information may include a quick interaction control and / or interaction result information for performing the next operation. There may be at least one quick interaction control and interaction result information. For example, the quick interaction control may be a "share" control, a "screenshot" control, etc. The interaction result information may be, for example, explanation information, a price comparison list of goods, etc.

[0099] Based on this, in order to improve the accuracy of interaction behavior prediction, in some embodiments, determining the target interaction information based on the interaction behavior information, spatial attribute information, semantic information and usage preference information may specifically include:

[0100] The interaction behavior information, spatial attribute information, semantic information and usage preference information are input into the multimodal big model, and the multimodal big model is used to determine the target interaction information.

[0101] The multimodal big model may be an artificial intelligence big model capable of processing multimodal information. The multimodal information may include interactive behavior information, spatial attribute information, semantic information, and usage preference information. The multimodal big model may be, for example, a multimodal Transformer model.

[0102] In some embodiments, the above-mentioned inputting the interaction behavior information, spatial attribute information, semantic information and usage preference information into the multimodal macro model and using the multimodal macro model to determine the target interaction information may specifically include:

[0103] Input the interactive behavior information, spatial attribute information, semantic information and usage preference information into the multimodal large model, and determine the association relationship between the first modal information and each second modal information based on the cross attention mechanism for each first modal information, where the first modal information is any one of the interactive behavior information, spatial attribute information, semantic information and usage preference information, and the second modal information is any one of the interactive behavior information, spatial attribute information, semantic information and usage preference information except the first modal information;

[0104] Based on the self-attention mechanism, for each first modal information, the internal dependency relationship of the first modal information is determined based on the association relationship;

[0105] The target interaction information is determined according to the association relationship and internal dependency relationship corresponding to each first modal information.

[0106] For example, when the first modal information is spatial attribute information and the second modal information is semantic information, a schematic diagram of determining the association relationship between the first modal information and the second modal information based on the cross attention mechanism using a multimodal large model can be as follows: Figure 3 shown.

[0107] For example, when the first modal information is spatial attribute information and the second modal information is interactive behavior information, a schematic diagram of determining the association relationship between the first modal information and the second modal information based on the cross attention mechanism using a multimodal large model can be as follows: Figure 4 shown.

[0108] For example, if the spatial attribute information is recorded as C1, the semantic information is recorded as C2, the interactive behavior information is recorded as C3, and the usage preference information is recorded as C4, then the schematic diagram of determining the target interactive information by using the multimodal large model provided in the embodiment of the present application can be as follows: Figure 5 shown.

[0109] The embodiment of the present application can improve the accuracy of interactive behavior prediction by using a multimodal large model to determine target interaction information.

[0110] S140, displaying target interaction information on the interaction interface.

[0111] The target interaction information can be displayed in a floating window on the interactive interface. As the user continues to interact with the XR device, the target interaction information displayed in the floating window can be dynamically adjusted.

[0112] As an example, when a user is reading a text, the explanation corresponding to the phrase the user is interested in can be displayed; when a user is browsing an e-commerce catalog, a product price comparison list can be displayed; when the user's viewpoint first shifts from the text display area to the video playback area and then stares at an animal in the video content for a while, a "screenshot" control and a text introduction to the animal can be displayed at the same time.

[0113] If the target interaction information is a quick interaction control, the user can continue to interact by directly clicking the quick interaction control in the interaction interface, without having to find the interaction control through layer-by-layer operations, thus simplifying the user operation. If the target interaction information is the interaction result information, the user can directly obtain the interaction result, further simplifying the user operation.

[0114] In an embodiment of the present application, since the target interaction information is associated with the user's operation intention, by acquiring the interaction behavior information generated in the interaction interface of the extended reality device, and when the interaction behavior information represents that the user's gaze time on a display object in the display window reaches a preset gaze time, the target interaction information is determined according to the interaction behavior information, the spatial attribute information of the display window in the interaction interface, semantic information and the user's usage preference information for the extended reality device, and the target interaction information is displayed in the interaction interface. That is, by predicting the user's next operation intention during the process of the user interacting with the extended reality device, and displaying the target interaction information associated with the operation intention, the interaction result can be directly displayed, or the user can interact with the extended reality device based on the target interaction information, thereby simplifying the user's interaction operation, improving the interaction efficiency, and alleviating the user's hand-eye fatigue.

[0115] In order to better describe the entire solution, some specific examples are given based on the above embodiments.

[0116] First, the multimodal large model can be deployed in the XR device or in the cloud, which is not limited here. Usually, since the multimodal large model itself is large in scale, in order to ensure the normal operation of the XR device, the multimodal large model can be deployed in the cloud, and the "end + cloud" system architecture can be used to implement the solution in the embodiment of this application. When the multimodal large model is deployed in the cloud, the schematic diagram of the system architecture in the embodiment of this application can be as follows: Figure 6 shown.

[0117] In addition, if Figure 7As shown, 700 can represent the user, and 701 can represent the interactive interface of the XR device. 702 can represent the movement trajectory of the user's viewpoint (in this example, the viewpoint first moves from the text display area to the video playback area, then stares at the pet in the video content for a while, and finally moves to the browser search box). 703 can represent the prediction of the multimodal large model, which guesses that the user's first intention may be to know the pet's breed, habits and other information, so some introductory content is displayed in the floating window. 704 can represent that the model infers that the user's second intention is to take a screenshot, so an additional "screenshot" control is displayed. 705 can represent that the user's current viewpoint stays in the search box, so the search keyword is predicted, that is, the recommended phrase is displayed. 706 and 607 can represent the quick interaction controls ranked 2nd and 3rd, corresponding to the "Share" control and "Search" control respectively.

[0118] It should be noted that when making predictions in this example, the interactive behavior information may include the first sub-interactive behavior information from the time the XR device is turned on to before the viewpoint moves into the text display area, and the second sub-interactive behavior information from the time the viewpoint moves from the text display area to the video playback area, then stares at the pet in the video content for a period of time, and finally moves to the browser search box.

[0119] Therefore, by perceiving the interactive behavior information, semantic information, spatial attribute information, usage preference information and the like through the XR device, the interactive behavior information, semantic information, spatial attribute information and usage preference information are input into the multimodal big model, and the multimodal big model is used to determine the target interaction information and dynamically display the target interaction information on the interactive interface of the XR device, which helps to enrich the information expression of the XR interface by combining the user's potential demands in real time, improve the interaction efficiency and relieve the user's eye and hand fatigue.

[0120] The information interaction method provided in the embodiment of the present application can be executed by an information interaction device. In the embodiment of the present application, the information interaction device provided in the embodiment of the present application is described by taking the method for executing information interaction by the information interaction device as an example.

[0121] Figure 8 FIG. 1 is a schematic diagram showing the structure of an information interaction device provided in an embodiment of the present application. Figure 8 As shown, the information interaction device 800 provided in the embodiment of the present application may include:

[0122] A first acquisition module 801 is used to acquire interaction behavior information generated in an interaction interface of an extended reality device, where the interaction interface includes at least one display window, and the interaction behavior information is interaction process information of at least one display window, and the interaction behavior information includes a gaze duration of each of at least one display object in the display window;

[0123] The second acquisition module 802 is used to acquire the spatial attribute information of the display window in the interactive interface, the semantic information of the display window, and the user's preference information for the extended reality device when the gaze duration represented by the interactive behavior information reaches a preset gaze duration, wherein the semantic information is determined according to the type information and state information of the display window, and the spatial attribute information is determined according to the size information and position information of the display window;

[0124] A determination module 803 is used to determine target interaction information according to the interaction behavior information, the spatial attribute information, the semantic information and the usage preference information, where the target interaction information is associated with the user's operation intention;

[0125] The display module 804 is used to display the target interaction information on the interaction interface.

[0126] The information interaction device 800 is described in detail below, as shown below:

[0127] In some embodiments, the first acquisition module 801 may specifically include:

[0128] A receiving unit, configured to receive a first input from a user on a display object in at least one display window;

[0129] A first acquisition unit, configured to acquire, in response to a first input, a display object sequence corresponding to a plurality of display objects, a gaze duration for gazing at each display object, and a target operation behavior for operating at least one display window;

[0130] A first determining unit, configured to determine the user's stay time information for each display object according to the gaze time corresponding to each display object within a first preset time;

[0131] A splitting unit, used to split the display object sequence into multiple operation segments according to the target operation behavior;

[0132] The second determining unit is used to determine the interactive behavior information according to the display object sequence, the stay duration information and the operation fragment.

[0133] In some embodiments, the second acquisition module 802 may further include:

[0134] A second acquisition unit, used to acquire type information and state information of the display window, where the state information is used to indicate whether the display window is activated;

[0135] A processing unit, configured to perform numerical processing on the type information when the type information is text information, to obtain a numerical result corresponding to the type information;

[0136] A third acquisition unit, used to acquire an activation weight corresponding to the state information;

[0137] The third determining unit is used to determine the semantic information according to the numerical result and the activation weight.

[0138] In some embodiments, the display window includes a first display window whose status information is in an activated state and at least one second display window whose status information is in an inactivated state. Based on this, the second acquisition module 802 may further include:

[0139] a fourth acquiring unit, configured to acquire a window distance between the first display window and each second display window to obtain at least one window distance;

[0140] The fourth determining unit is used to determine the activation weight corresponding to each display window according to at least one window distance.

[0141] In some embodiments, the fourth obtaining unit may specifically include:

[0142] An acquisition subunit, configured to acquire a first projection of an anchor point of a first display window on a unit sphere and a second projection of a second display window on a unit sphere;

[0143] The first determining subunit is used to determine the spherical distance between the first projection and each second projection as the window distance.

[0144] In some embodiments, the second acquisition module 802 may further include:

[0145] a fifth acquisition unit, configured to acquire usage ratio information of multiple applications in the extended reality device by the user within a second preset time period and usage frequency information of the target application by the user within multiple time periods, wherein the second preset time period includes multiple time periods, and the target application is any one of the multiple applications;

[0146] The fifth determining unit is used to determine the usage preference information according to the usage ratio information and the usage frequency information.

[0147] In an embodiment of the present application, since the target interaction information is associated with the user's operation intention, by acquiring the interaction behavior information generated in the interaction interface of the extended reality device, and when the interaction behavior information represents that the user's gaze time on a display object in the display window reaches a preset gaze time, the target interaction information is determined according to the interaction behavior information, the spatial attribute information of the display window in the interaction interface, semantic information and the user's usage preference information for the extended reality device, and the target interaction information is displayed in the interaction interface. That is, by predicting the user's next operation intention during the process of the user interacting with the extended reality device, and displaying the target interaction information associated with the operation intention, the interaction result can be directly displayed, or the user can interact with the extended reality device based on the target interaction information, thereby simplifying the user's interaction operation, improving the interaction efficiency, and alleviating the user's hand-eye fatigue.

[0148] The information interaction device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0149] The information interaction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0150] The information interaction device provided in the embodiment of the present application can achieve Figures 1 to 7 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0151] Alternatively, if Fig. 9As shown, an embodiment of the present application also provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores programs or instructions that can be executed on the processor 901, and when the program or instructions are executed by the processor 901, the various steps of the above-mentioned information interaction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0152] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0153] Fig.10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0154] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0155] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0156] The processor 1010 is used to obtain interactive behavior information generated in an interactive interface of an extended reality device, where the interactive interface includes at least one display window, and the interactive behavior information is interactive process information of the at least one display window, and the interactive behavior information includes a gaze duration of at least one display object in the display window; when the interactive behavior information indicates that the gaze duration reaches a preset gaze duration, obtain spatial attribute information of the display window in the interactive interface, semantic information of the display window, and user preference information for the extended reality device, where the semantic information is determined according to type information and state information of the display window, and the spatial attribute information is determined according to size information and position information of the display window; determine target interactive information according to the interactive behavior information, the spatial attribute information, the semantic information, and the usage preference information, and the target interactive information is associated with the user's operation intention;

[0157] The display unit 1006 is used to display the target interaction information on the interaction interface.

[0158] The embodiments of the present application predict the user's next operation intention during the process of interaction between the user and the extended reality device, and display the target interaction information associated with the operation intention, so as to directly display the interaction result or enable the user to interact with the extended reality device based on the target interaction information, thereby simplifying the user's interactive operation, improving the interaction efficiency, and alleviating the user's hand-eye fatigue.

[0159] Optionally, the user input unit 1007 is used to receive a first input from a user on a display object in at least one display window;

[0160] The processor 1010 is further configured to:

[0161] In response to a first input, acquiring a display object sequence corresponding to a plurality of display objects, a gaze duration for each display object, and a target operation behavior for operating at least one display window;

[0162] According to the gaze duration corresponding to each display object within the first preset duration, the user's stay duration information for each display object is determined; according to the target operation behavior, the display object sequence is divided into multiple operation segments; and the interactive behavior information is determined according to the display object sequence, the stay duration information and the operation segments.

[0163] Optionally, the processor 1010 is further configured to:

[0164] Get the type information and status information of the display window, where the status information is used to indicate whether the display window is activated;

[0165] When the type information is text information, the type information is digitized to obtain a digitized result corresponding to the type information;

[0166] The activation weight corresponding to the state information is obtained; and the semantic information is determined according to the numerical result and the activation weight.

[0167] Optionally, the display window includes a first display window whose status information is in an activated state and at least one second display window whose status information is in an inactivated state. Based on this, the processor 1010 is further configured to:

[0168] Acquire a window distance between the first display window and each second display window to obtain at least one window distance;

[0169] An activation weight corresponding to each display window is determined according to at least one window distance.

[0170] Optionally, the processor 1010 is further configured to:

[0171] Acquire a first projection of an anchor point of the first display window on the unit sphere and a second projection of the second display window on the unit sphere;

[0172] The spherical distance between the first projection and each second projection is determined as the window distance.

[0173] Optionally, the processor 1010 is further configured to:

[0174] Obtaining usage ratio information of multiple applications in the extended reality device by the user within a second preset time period and usage frequency information of the target application by the user within multiple time periods, where the second preset time period includes multiple time periods, and the target application is any one of the multiple applications;

[0175] The usage preference information is determined based on the usage ratio information and the usage frequency information.

[0176] The embodiments of the present application help to enrich the information expression of the XR interface by combining the user's potential demands in real time, improve the interaction efficiency and relieve the user's eye and hand fatigue.

[0177] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0178] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0179] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0180] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned information interaction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0181] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0182] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned information interaction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0183] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0184] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned information interaction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0185] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0187] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An information interaction method, characterized in that: include: Acquire interactive behavior information generated in an interactive interface of an extended reality device, the interactive interface comprising at least one display window, the interactive behavior information being interactive process information of the at least one display window, the interactive behavior information comprising a gaze duration of each of at least one display object in the display window; When the interaction behavior information indicates that the gaze duration reaches a preset gaze duration, obtaining spatial attribute information of the display window on the interaction interface, semantic information of the display window, and user preference information for the extended reality device, wherein the semantic information is determined according to type information and state information of the display window, and the spatial attribute information is determined according to size information and position information of the display window; Determining target interaction information according to the interaction behavior information, the spatial attribute information, the semantic information and the usage preference information, wherein the target interaction information is associated with the operation intention of the user; The target interaction information is displayed on the interaction interface.

2. The method according to claim 1, characterized in that: The obtaining of interactive behavior information generated in the interactive interface of the extended reality device includes: receiving a first input from the user on the display object in the at least one display window; In response to the first input, acquiring a display object sequence corresponding to the plurality of display objects, a gaze duration for gazing at each of the display objects, and a target operation behavior for operating the at least one display window; Determining the duration information of the user's stay on each of the displayed objects according to the gaze durations respectively corresponding to each of the displayed objects within the first preset duration; According to the target operation behavior, splitting the display object sequence into multiple operation segments; The interactive behavior information is determined according to the display object sequence, the stay duration information and the operation fragment.

3. The method according to claim 1, characterized in that: Acquiring semantic information of the display window, including: Acquire type information and state information of the display window, wherein the state information is used to indicate whether the display window is activated; In the case where the type information is text information, performing numerical processing on the type information to obtain a numerical result corresponding to the type information; Obtaining an activation weight corresponding to the state information; The semantic information is determined according to the quantified result and the activation weight.

4. The method according to claim 3, characterized in that The display window includes a first display window whose state information is in an activated state and at least one second display window whose state information is in an inactivated state, and acquiring the activation weight corresponding to the state information includes: Acquire a window distance between the first display window and each of the second display windows to obtain at least one window distance; The activation weight corresponding to each of the display windows is determined according to the at least one window distance.

5. The method according to claim 4, characterized in that The obtaining the window distance between the first display window and each of the second display windows includes: Acquire a first projection of an anchor point of the first display window on a unit sphere and a second projection of the second display window on the unit sphere; A spherical distance between the first projection and each of the second projections is determined as the window distance.

6. The method according to claim 1, characterized in that Acquiring the user's usage preference information for the extended reality device, including: Acquire usage ratio information of multiple applications in the extended reality device by the user within a second preset time period and usage frequency information of a target application by the user within multiple time periods, wherein the second preset time period includes the multiple time periods, and the target application is any one of the multiple applications; The usage preference information is determined according to the usage ratio information and the usage frequency information.

7. An information interaction device, characterized in that: include: A first acquisition module is used to acquire interaction behavior information generated in an interaction interface of an extended reality device, wherein the interaction interface includes at least one display window, and the interaction behavior information is interaction process information of the at least one display window, wherein the interaction behavior information includes a gaze duration of each of at least one display object in the display window; A second acquisition module is used to acquire, when the interaction behavior information indicates that the gaze duration reaches a preset gaze duration, spatial attribute information of the display window on the interaction interface, semantic information of the display window, and user preference information for the extended reality device, wherein the semantic information is determined according to type information and state information of the display window, and the spatial attribute information is determined according to size information and position information of the display window; a determination module, configured to determine target interaction information according to the interaction behavior information, the spatial attribute information, the semantic information and the usage preference information, wherein the target interaction information is associated with the operation intention of the user; A display module is used to display the target interaction information on the interaction interface.

8. The device according to claim 7, characterized in that The first acquisition module includes: A receiving unit, configured to receive a first input from the user on the display object in the at least one display window; A first acquisition unit, configured to acquire, in response to the first input, a display object sequence corresponding to the plurality of display objects, a gaze duration for gazing at each of the display objects, and a target operation behavior for operating the at least one display window; A first determining unit, configured to determine the duration information of the user's stay on each of the displayed objects according to the gaze durations respectively corresponding to each of the displayed objects within a first preset duration; A splitting unit, used for splitting the display object sequence into multiple operation segments according to the target operation behavior; The second determining unit is used to determine the interactive behavior information according to the display object sequence, the stay duration information and the operation fragment.

9. An electronic device, characterized in that: It includes a processor and a memory, the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the information interaction method according to any one of claims 1 to 6 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the information interaction method according to any one of claims 1 to 6 are implemented.

11. A computer program product, characterized in that The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps of the information interaction method according to any one of claims 1 to 6.

Images