Interaction method, device, computer equipment, computer readable storage medium and computer program product

Abstract

The application provides an interaction method and device, computer equipment, computer readable storage medium and computer program product; the method comprises: obtaining user input information, generating initial output text based on the user input information, and controlling the display of the initial output text on a display interface; obtaining user eye image data and user hand point cloud data; determining a first region in the display interface based on the user eye image data, determining target text information in the initial output text corresponding to the first region, and the first region representing a potential interaction region; determining whether the first region is a target region based on the user hand point cloud data, and the target region representing a region to be interacted; in the case where it is determined that the first region is the target region, generating interaction information related to the target text information.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to an interaction method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] Currently, when interacting with mobile AI tools, these tools typically only provide feedback and output answers based on user-input questions and / or commands, and may also offer related question guidance. When users are dissatisfied with the output, they need to actively send adjustment commands one or more times. During this process, users gradually lose patience due to unsatisfactory responses, resulting in a poor user experience. Furthermore, in responding to adjustment commands, AI tools need to re-output new answers, including at least some historical responses, each time. This is not only inefficient but also repeatedly consumes excessive screen space and system memory, leading to resource waste. Summary of the Invention

[0003] This application provides an interaction method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve interaction efficiency.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an interaction method, the method including: Obtain user input information, generate initial output text based on the user input information, and control the display of the initial output text on the display interface; Acquire user eye image data and user hand point cloud data; Based on the user's eye image data, a first region in the display interface is determined, and target text information corresponding to the first region in the initial output text is determined. The first region represents a potential interactive area. Based on the user's hand point cloud data, it is determined whether the first region is a target region, where the target region represents the region to be interacted with; If the first region is determined to be the target region, interactive information related to the target text information is generated.

[0005] In the above scheme, the user's eye image data includes multiple eye images sorted in chronological order. Determining the first region in the display interface based on the user's eye image data includes: extracting features from the multiple eye images to obtain eye movement feature information corresponding to each eye image. The eye movement feature information includes first position information and second position information related to the user's eyeball. The first position information represents the pupil position information, and the second position information represents the Pulchin spot position information. Based on the first position information and second position information corresponding to each eye image in the multiple eye images, the first region in the display interface is determined.

[0006] In the above scheme, determining the first region in the display interface based on the first and second position information corresponding to each eye image in the plurality of eye images includes: determining the first gaze direction vector of the user at a first moment based on the first and second position information corresponding to the first eye image; the first eye image is any one of the plurality of eye images, and the first moment is the time corresponding to the first eye image; determining one or more feedback regions in the display interface based on the first position information corresponding to each of the plurality of eye images; and determining a first region satisfying a first constraint condition from the one or more feedback regions based on the gaze direction vector at each of the plurality of moments, wherein the first constraint condition includes: the angle between the gaze direction vector and the center direction vector of the first feedback region is less than a first threshold, and the duration of the gaze direction vector in the first feedback region is greater than or equal to a second threshold, and the first feedback region is any one of the one or more feedback regions.

[0007] In the above scheme, the user's hand point cloud data includes multiple hand point cloud data ordered in chronological order. The step of determining whether the first region is a target region based on the user's hand point cloud data includes: starting from a second moment associated with the first region, calculating the first cosine value of the angle between the hand movement direction vector and the target direction vector at each moment, obtaining one or more first cosine values; wherein, the second moment is the moment when the duration of the user's gaze direction vector's stay in the first region is started, the hand movement direction vector at each moment is the direction vector from the center point of each moment to the center point of the second moment, and the target direction vector is the direction vector from the center point of the second moment to the center point of the first region; based on the first cosine values ​​of each of the one or more moments, determining a first trend value for each moment, where the first trend value represents the degree of approach of the fourth moment relative to the third moment towards the first region, and the fourth moment is the next moment in the chronological order of the third moment; and determining whether the first region is a target region based on the first trend values ​​of each moment.

[0008] In the above scheme, determining the first trend value at each moment based on the first cosine value of each moment in the one or more moments includes: determining the first two-dimensional position information corresponding to the center point of the user's hand point cloud data at multiple moments; determining the first distance information from the center point of the first region to the center point of the first region at each moment based on the target position information of the center point of the first region and the first two-dimensional position information of the center point of each moment in the one or more moments; calculating the angle between the user's hand movement direction vector and the target direction vector at each moment based on the first cosine value of each moment in the one or more moments to obtain the first angle corresponding to each moment; when the first angle corresponding to the third moment and the first angle corresponding to the fourth moment satisfy the first preset condition, calculating the first trend value of the fourth moment based on the first distance information and the first cosine value associated with the third moment and the first distance information and the first cosine value associated with the fourth moment, wherein the first preset condition is that the first angle corresponding to each moment in two adjacent moments is within the preset angle range; when the first angle corresponding to the third moment and the first angle corresponding to the fourth moment do not satisfy the first preset condition, setting the first trend value of the fourth moment to 0.

[0009] In the above scheme, determining whether the first region is the target region based on the first trend value at each time step includes: determining a first number of times when the first trend value is greater than 0 based on the first trend value at each of the one or more times step steps; and determining the first region as the target region when the first number is greater than or equal to a third threshold.

[0010] In the above scheme, determining the first two-dimensional position information corresponding to the center point of the user's hand point cloud data at multiple times includes: performing two-dimensional transformation processing on the user's hand point cloud data at multiple times based on preset camera parameters to obtain a set of two-dimensional position information corresponding to the user's hand point cloud data at each time; and calculating the position information corresponding to the center point of the user's hand point cloud data at each time based on the set of two-dimensional position information corresponding to the user's hand point cloud data at each time.

[0011] This application provides a computer device, the computer device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the interactive method provided in the embodiments of this application.

[0012] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the interactive method provided in this application when executed by a processor.

[0013] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the interaction method provided in this application.

[0014] This application provides an interactive method, apparatus, computer device, computer-readable storage medium, and computer program product. On one hand, for initial output text generated in response to user input, it acquires user eye image data and user hand point cloud data when the user browses the initial output text. Based on the user eye image data, it determines a potential first region within a first text region corresponding to the initial output text. Based on the user hand point cloud data, it determines whether the first region is a target region for interaction. By collecting user eye image data and user hand point cloud data in real time, it accurately identifies the text region the user expects to interact with and accurately predicts the user's intention regarding the initial output text. On the other hand, if the first region is determined to be a target region, it generates interactive information related to the target text information within the target region. It actively interacts with the user through this interactive information, allowing the user to make immediate adjustments and presentations based on the initial output text. This greatly reduces the user's operation steps, improves user experience, enhances interaction efficiency and smoothness, and increases the space utilization of the display interface. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the interaction method according to an embodiment of this application; Figure 2This is a schematic diagram of the text region corresponding to the initial output text in an embodiment of this application; Figure 3 This is a schematic diagram of interactive information in an embodiment of this application; Figure 4 This is a schematic diagram of the process for determining the first region according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the process of determining the target region according to an embodiment of this application; Figure 6 This is a schematic diagram of the process for determining the first tendency value in an embodiment of this application; Figure 7 This is a schematic diagram of the composition structure of the interactive device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0018] The terms “first,” “second,” etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Before providing a detailed description of the technical solution of this application, a brief introduction to the relevant technologies will be given first.

[0020] Currently, with the development of the internet and the advancement of artificial intelligence technology, human-computer interaction has entered an intelligent stage. AI tools are gradually becoming more widespread and are rapidly iterating from single-function to multimodal and intelligent agent forms. AI tools are software or platforms developed based on artificial intelligence technology. Mobile AI tools refer to AI applications adapted to mobile devices such as smartphones and tablets. Relying on edge or cloud-based AI capabilities, they can simulate human intelligence and assist users in efficiently completing various intelligent tasks in portable scenarios. Depending on different business needs, AI tools can generally be divided into general dialogue, visual generation, audio processing, and industry-specific types. For example, the most widely used general dialogue AI tools can perform intelligent retrieval, intelligent question answering, and intelligent writing through text-based dialogue interaction.

[0021] However, current dialogue-based AI tools require users to actively input adjustment commands to the AI ​​tool once or multiple times when they are not satisfied with the output response. If the AI ​​tool's adjusted output response still fails to satisfy the user, the user experience will gradually deteriorate. Furthermore, in the process of passively responding to adjustment commands, the AI ​​tool reduces interaction efficiency by re-outputting new responses that include at least some of the previous responses each time. It also wastes resources by repeatedly occupying too much screen space and system memory.

[0022] To address this issue and improve the reduced interaction efficiency caused by passive interaction with AI tools, we are currently exploring the use of eye-tracking to enable AI tools to perceive the user's reaction to the received responses and proactively adjust their responses accordingly, thereby improving the efficiency of interaction with users.

[0023] Eye-tracking technology is a non-invasive intelligent technology that captures and analyzes eye movement information, transforming the physiological characteristics of the eye into computable data to achieve "eye-tracking interaction." The core principle of this technology is to emit near-infrared light invisible to the human eye through a device, forming a reflective spot on the corneal surface. Combined with a high-speed camera, it collects eye images in real time. Through algorithm analysis of the relative displacement between the pupil center and the reflection point, the specific coordinates of the gaze are finally mapped to obtain the specific coordinate position, thereby providing accurate data support for subsequent business operations.

[0024] Currently, the specific implementation methods of human-computer interaction based on eye-tracking technology vary depending on the business scenario or business needs. For example, in the medical scenario, considering the characteristics of different types of users such as brain activity states and eye diseases, user feature data is collected and combined with eye movement feature information to extract features, so as to obtain eye movement feature data that accurately reflects the user's true eye movement intentions. This allows the obtained eye control commands to be adapted to different users, thereby improving the accuracy of eye control interaction and bringing users a smoother and more natural interactive experience.

[0025] To address the business requirement of eye use status detection, the intersection points of the human eye's line of sight and the screen, along with their corresponding pixel coordinates, are continuously acquired during the process of the human eye viewing the screen. When the number of continuously acquired intersection points reaches a preset threshold, the discrete value corresponding to the intersection point set is used to determine whether the human eye is in a fixation state, and the corresponding fixation point and its corresponding pixel coordinates are determined.

[0026] For remote assistance applications, remote assistance can be provided by observing the assistant's gaze and verbal behavior. This allows for the estimation of the assistant's gaze position on the captured image based on facial information, semantic analysis of the assistant's voice information, and image understanding of the captured image to establish a match, thereby determining the target location indicated by the assistant in the captured image.

[0027] However, the eye-tracking technology in the above methods is relatively complex and relies on precise instruments and equipment. High-precision eye tracking requires dedicated infrared cameras and high-performance processing chips, resulting in high hardware costs. At the same time, functions such as dynamic gaze point rendering have high computing power requirements, which puts too much pressure on the battery life of portable devices. In other words, the current human-computer interaction methods based on eye-tracking technology require high hardware and computing power costs.

[0028] Based on this, this application proposes an interaction method, which is applied to (mobile) terminals or mobile devices with human-computer interaction functions (such as those with deployed AI tools), specifically mobile phones, tablets, and other portable electronic devices that can be carried around and support network interaction. Figure 1 This is a flowchart illustrating the interaction method of an embodiment of this application; as shown Figure 1 As shown, the method includes: Step 101: Obtain user input information, generate initial output text based on the user input information, and control the display of the initial output text on the display interface; Step 102: Acquire user's eye image data and user's hand point cloud data; Step 103: Determine a first region in the display interface based on the user's eye image data, and determine the target text information in the initial output text that corresponds to the first region, where the first region represents a potential interactive area; Step 104: Based on the user's hand point cloud data, determine whether the first region is the target region, where the target region represents the region to be interacted with; Step 105: If the first region is determined to be the target region, generate interactive information related to the target text information.

[0029] In this embodiment, the terminal obtains user input information and generates initial output text in response to the user input information.

[0030] In this embodiment, the user input information is text information that the user inputs to the terminal according to the target intent. For example, it may be questions, requests, instructions, etc., raised by the user. This embodiment does not impose specific restrictions on the input method of the input information. For example, the user can upload real-time or stored video data, voice data, image data, and text data through the interface. The terminal performs preprocessing such as feature extraction and modality conversion on the original multimodal data uploaded by the user to obtain the input information in text form.

[0031] The initial output text is the text information initially generated by the terminal based on the user input information, that is, the unadjusted initial text information. This application embodiment does not impose specific restrictions on the method of generating the initial output text. For example, it can understand the user's intent in the input information through natural language processing (NLP), rely on the massive knowledge reserves of pre-trained models such as underlying neural networks, and use the attention mechanism of the Transformer architecture as the core to generate the initial output text associated with the input information.

[0032] In this embodiment, after generating initial output text in response to user input information, the terminal controls the display device to output the initial output text to the user on the display interface, and begins to collect user eye image data and user hand point cloud data when the user browses the initial output text.

[0033] In this embodiment, the display device is used to convert the generated initial output text into visual information and present it to the user through the display interface. This application embodiment does not impose specific limitations on the display device, and it can be electrically connected to the terminal through the graphics card interface.

[0034] User eye image data refers to the feature information of the eye area when a user browses the initial output text. This application embodiment does not impose specific restrictions on the method of collecting user eye image data. For example, a (high-precision) camera can be built into the terminal and combined with active infrared illumination technology. When the user starts browsing, the terminal immediately activates the camera and uses the corneal reflection principle to capture the movement features of the user's eye area when browsing the initial output text to obtain user eye image data.

[0035] User hand point cloud data refers to the feature information of the finger parts when the user browses the initial output text. This application embodiment does not impose specific restrictions on the collection method of user hand point cloud data. For example, a depth camera can be built into the terminal. When the user starts browsing, the terminal immediately activates the depth camera and uses depth perception technology to obtain the motion features of the user's finger parts when browsing the initial output text, thereby obtaining user hand point cloud data.

[0036] In this embodiment, after obtaining the user's eye image data when the user browses the initial output text, based on the user's eye image data, a potential first region is determined in the text area corresponding to the initial output text in the display interface, and the target text information in the initial output text corresponding to the first region is determined.

[0037] In this embodiment, the text area corresponding to the initial output text is the complete area of ​​the initial output text displayed to the user through the display interface, that is, the user can browse the area of ​​the initial output text. This embodiment does not impose specific restrictions on the size information of the text area corresponding to the initial output text. The size information of the text area is related to the size information of the display interface and the amount of text in the initial output text.

[0038] For example, let's take human-computer interaction between a user and a mobile phone as an example. Figure 2 This is a schematic diagram of the text region corresponding to the initial output text in an embodiment of this application; as shown below. Figure 2 As shown, in response to the user's input message "Please help me write an activity plan", the terminal generates and outputs the initial output text "Of course! Here is an activity plan: XX, XX..." on the screen via the display device. The area occupied by the initial output text on the screen is the text area corresponding to the initial output text.

[0039] The first region, also known as the pre-feedback region or pre-interaction region, refers to the potential interaction region, that is, the text region where the user may expect to interact further with the terminal, or the text region where the user may expect the terminal to adjust. The first region is determined based on the movement characteristics of the user's eyeballs when browsing the initial output text. The first region includes text information that the user may expect to interact further or adjust, that is, the first region includes pre-feedback or pre-interaction text information.

[0040] The target text information is the text information in the first area. When the first area is the target area, the target text information is the text information in which the user determines that they expect to make further interaction or to adjust their expectations. That is, the target text information is quasi-feedback or quasi-interaction text information.

[0041] In this embodiment, after determining the first region, the system will determine whether the first region is the target region based on the user's hand point cloud data obtained when the user browses the initial output text.

[0042] In this embodiment, the target area can also be called the quasi-feedback area or quasi-interaction area, which refers to the determined area to be interacted with, that is, the text area that the user determines to want to interact further with the terminal, or the text area that the user determines to want the terminal to adjust. The target area is obtained by filtering one or more determined first areas based on the movement characteristics of the user's finger when browsing the initial output text.

[0043] In this embodiment, after determining the target area, interactive information related to the target text information within the target area is generated, and the interactive information is actively output to the user through the display interface so that the user can provide real-time feedback based on the initial output text.

[0044] In this embodiment, the interactive information is related to the target text information within the target area. For example, it may be prompts or guidance information given based on the specific content and / or context of the target text information, which is used to actively prompt or guide the user to further refine, supplement or modify the target text information within the target area.

[0045] It should be noted that the embodiments of this application do not impose specific limitations on the operation of interactive information prompts. For example, targeted prompts can be made based on the specific characteristics of the target text information in the target area. For example, based on... Figure 2 The text region corresponding to the initial output text shown. Figure 3 This is a schematic diagram of interactive information in an embodiment of this application; as shown Figure 3 As shown in a, after determining the first region as the target region, when the target text information within the target region does not contain punctuation marks (such as complete or partial words or phrases), the interactive information can be "polish" and / or "modify". "Polish" is used to prompt the user whether the tone and style of the target text information need to be changed without changing the semantics of the target text information; "modify" is used to prompt the user whether the target text information needs to be replaced with a new word to change the semantics.

[0046] like Figure 3 As shown in b, after determining the first region as the target region, when the target text information within the target region contains punctuation marks (such as two complete or partial words, phrases, or sentences connected by punctuation marks), the interactive information can be "insert," which prompts the user whether a statement needs to be added before the punctuation mark, or whether a new statement needs to be added after the punctuation mark.

[0047] In this embodiment, on the one hand, for the initial output text generated from the acquired user input information, the system acquires user eye image data and user hand point cloud data when the user browses the initial output text. Based on the user eye image data, it determines a potential first region in the display interface. Based on the user hand point cloud data, it determines whether the first region is the target region for interaction. By collecting the user's eye image data and user hand point cloud data in real time, the system accurately identifies the text region that the user expects to interact with and accurately predicts the user's mental intention regarding the initial output text.

[0048] On the other hand, when the first region is determined as the target region, interactive information related to the target text information within the first region is generated. The interactive information is used to actively interact with the user, allowing the user to make real-time adjustments and presentations based on the initial output text. This greatly reduces the user's operation steps, improves the user experience, enhances the efficiency and smoothness of interaction, and increases the space utilization of the display interface.

[0049] In some optional implementations, the user eye image data includes multiple eye images ordered chronologically. Determining the first region in the display interface based on the user eye image data includes: extracting features from each of the multiple eye images to obtain eye movement feature information corresponding to each eye image. The eye movement feature information includes first position information and second position information related to the user's eyeball, where the first position information represents pupil position information and the second position information represents Pulchin spot position information. Based on the first position information and second position information corresponding to each of the multiple eye images, the first region in the display interface is determined.

[0050] In this embodiment, the user's eye image data includes an eye image sequence. The eye image sequence includes multiple eye images at different times, ordered chronologically from the moment when the user's eye image data is first collected. The time interval between adjacent moments is the same. This embodiment does not impose specific restrictions on the moment when the user's eye image data is collected. For example, multiple frames of eye images can be collected, with each frame of eye image corresponding to an eye image at each moment.

[0051] It should be noted that the embodiments of this application do not impose specific limitations on eye images. For example, when using a high-precision infrared camera, the eye images acquired at each moment can be grayscale digital images in the near-infrared band under active infrared illumination conditions.

[0052] In this embodiment, after acquiring eye images at multiple moments when the user browses the initial output text, feature extraction is performed on the eye images at each moment to obtain eye movement feature information corresponding to each eye image. Based on the eye movement feature information corresponding to each eye image at multiple moments, a first region that the user may want to adjust is determined in the first text region.

[0053] In this embodiment, the eye movement feature information includes first position information and second position information related to the user's eyeball. The first position information represents the pupil position information, which specifically includes the pixel coordinates of the pupil center in each eye image. The pupil center moves with eyeball rotation. This embodiment does not impose specific limitations on the extraction method of the first position information. For example, the least squares method can be used to fit the pupil contour to extract the pupil pixel coordinates.

[0054] The second location information represents the location information of the Pulcyn spot. The Pulcyn spot is a bright reflective spot produced when an infrared light source illuminates the cornea. With the light source and head positions fixed, the position of the Pulcyn spot relative to the eyeball remains almost constant, thus it can be used as a static reference. The Pulcyn spot location information specifically includes the pixel coordinates of the Pulcyn spot in each eye image. This application does not impose specific limitations on the extraction method of the second location information. For example, using the obtained pupil center as a reference, a binary large object (Blob) algorithm can be used to identify bright connected regions, and the bright spot features can be enhanced using a Gaussian-Laplacian operator to extract the pixel coordinates of the Pulcyn spot.

[0055] It should be noted that, in the embodiments of this application, after obtaining the first position information and the second position information of the eye image at each time moment through feature extraction, a Kalman filter can be used to perform noise reduction and smoothing on the eye movement feature information in order to eliminate noise interference caused by eye tremors and slight head movements.

[0056] Figure 4 This is a schematic diagram illustrating the process of determining the first region according to an embodiment of this application; as shown Figure 4 The step of determining the first region in the display interface based on the first and second position information corresponding to each eye image in the plurality of eye images includes: Step 201: Based on the first position information and the second position information corresponding to the first eye image, determine the first gaze direction vector of the user at the first moment, and determine the angular velocity and displacement information of the first gaze direction vector; the first eye image is any one of the plurality of eye images, and the first moment is the time corresponding to the first eye image.

[0057] In this embodiment, the specific process of determining the first region based on the eye movement feature information of the eye images at each time is as follows: First, based on the first position information and the second position information corresponding to each eye image, the user's gaze direction vector at each time is determined.

[0058] In this embodiment, the gaze direction vector, also known as the gaze direction vector, refers to a geometric vector with magnitude and direction that starts from the pupil of the human eye and points to the gaze point of the current display interface. It is used to quantitatively describe the spatial or planar direction information of "which direction the user is looking in". The gaze direction vector at the current moment is calculated based on the first position information (i.e., pupil position information) and the second position information (i.e., Pulchin spot position information) at the current moment.

[0059] For example, based on the pupil center pixel coordinates of frame t. and the pixel coordinates of the t-th frame's Pulchin spot. Calculate the gaze direction vector in frame t. The process can be designed as follows: (1) In this context, the relative displacement vector of the Pulcim spot pixel coordinates and the pupil center pixel coordinates on the screen plane in the t-th frame is used as an approximation or direct proxy for the gaze direction vector. The calculated gaze direction vector in the t-th frame is the gaze direction vector on the screen plane (two-dimensional), and its magnitude and direction change with eye movement.

[0060] It should be noted that, in the embodiments of this application, after determining the line-of-sight direction vector at each moment, the angular velocity and displacement information of the line-of-sight direction vector at each moment are determined based on the line-of-sight direction vectors of two adjacent moments. The angular velocity and displacement information are used to constrain the relatively stationary region.

[0061] In this embodiment, the angular velocity of the line-of-sight vector is used to describe how fast the line-of-sight direction rotates over time. Under the approximation of the screen plane, it can be calculated by the rate of change of the angle between the line-of-sight vectors at adjacent moments.

[0062] For example, the angular velocity of the gaze direction vector from frame t to frame t+1. The calculation process can be designed as follows: (2) in, This represents the angle between the view direction vectors of frame t and frame t+1; This represents the time interval between frame t+1 and frame t.

[0063] It can be based on the gaze direction vector of frame t. With the gaze direction vector of frame t+1 The inverse cosine of the included angle is calculated, for example, The calculation process can be designed as follows: (3) In this embodiment, the displacement information of the gaze direction vector may include the displacement vector of the gaze direction vector, which is used to describe the change in position and direction of the user's gaze direction between adjacent moments or time periods. That is, the change in position of the gaze direction vector (or gaze point) on the screen plane between two moments, which is used to measure whether the user's gaze has moved significantly.

[0064] For example, from the gaze direction vector of frame t The gaze direction vector up to frame t+1 displacement vector The calculation process can be designed as follows: (4) Furthermore, based on displacement vectors The corresponding displacement can be calculated. : (5) Step 202: Based on the first position information corresponding to each of the plurality of eye images, determine one or more feedback areas in the display interface.

[0065] In this embodiment, one or more feedback areas are determined in the display interface based on the first position information corresponding to each eye image.

[0066] In this embodiment, the feedback region refers to the area of ​​the user's gaze formed on the screen, calculated based on the user's first position information at each moment (i.e., the pixel coordinates of the pupil center). The feedback region can be represented by a two-dimensional rectangle, specifically by the two-dimensional coordinates of the four vertices of the rectangle. For example, a feedback region can be described as: The top left, top right, bottom right, and bottom left vertices of the feedback region are represented from left to right.

[0067] It should be noted that the embodiments of this application do not impose specific restrictions on the method of determining each feedback area using the pixel coordinates of the pupil center. For example, the pupil center of the user at the current moment can be taken as the center of the user's gaze area, and a rectangular area can be extended in all directions according to the preset field of view (FOV) or screen size ratio, which is the feedback area at the current moment.

[0068] Step 203: Based on the gaze direction vector at each of the multiple time points, determine a first region satisfying a first constraint condition from the one or more feedback regions, wherein the first constraint condition includes: the angle between the gaze direction vector and the center direction vector of the first feedback region is less than a first threshold, and the dwell time of the gaze direction vector in the first feedback region is greater than or equal to a second threshold, and the first feedback region is any one of the one or more feedback regions.

[0069] In this embodiment, after obtaining the user's gaze direction vector at each moment, the feedback region that satisfies the first constraint condition is selected from one or more determined feedback regions as the first region.

[0070] The first region, as a potential user interaction area, refers to the relatively static region that satisfies the first constraint condition during the user's browsing of the initial output text. The relatively static region is determined based on the angular velocity and displacement information of the line-of-sight vector at each moment. For example, "relatively static" can be quantitatively expressed as: the angular velocity of the line-of-sight vector between two moments (i.e., within a certain time period) is less than a preset angular velocity threshold. ), and the displacement vector of the line-of-sight direction vector is less than the preset displacement threshold ( In this case, the feedback region can be determined to be a relatively static feedback region.

[0071] The first constraint is used to filter out a first region by further constraining the degree of change in the user's gaze in one or more relatively static feedback regions. The first constraint specifically includes: the angle between the gaze direction vector and the center direction vector of any of the one or more feedback regions is less than a first threshold, and the duration of the gaze direction vector in the first feedback region is greater than or equal to a second threshold.

[0072] The center direction vector of the first feedback region refers to the direction vector from the user's eyeball (or pupil) to the center of the first feedback region, which can be calculated based on the first position information (i.e. the pixel coordinates of the pupil center) and the pixel coordinates of the center of the first feedback region.

[0073] The angle between the line of sight vector and the center direction vector of the first feedback region refers to the projection deviation angle of the line of sight vector in the first feedback region. This projection deviation angle can be calculated based on the inverse cosine of the cosine of the angle between the line of sight vector at the current moment and the center direction vector of the first feedback region. The specific calculation process is similar to the calculation process shown in formula (3), and will not be elaborated here.

[0074] The first threshold is a preset viewing angle tolerance value, used to constrain the projection deviation angle of the current viewing direction vector in the first feedback area. For example, it can be preset to 0.5°. When the projection deviation angle of the viewing direction vector in the first feedback area is greater than or equal to the first threshold, it is determined that the first feedback area does not meet the first constraint condition.

[0075] The duration of the gaze direction vector in the first feedback area refers to the duration during which the "gaze point" mapped by the current gaze direction vector on the two-dimensional display interface is within the first feedback area, that is, the relative stillness time of the user's eyeball in the first feedback area. This application embodiment does not impose specific restrictions on the timing method of this duration.

[0076] The second threshold is a preset minimum duration used to constrain the dwell time of the line direction vector in the first feedback region. For example, it can be preset to 100-200 milliseconds (ms). When the dwell time of the line direction vector in the first feedback region is less than the second threshold, it is determined that the first feedback region does not meet the first constraint condition.

[0077] In some alternative implementations, the user hand point cloud data includes multiple hand point cloud data arranged in chronological order.

[0078] In this embodiment, the user's hand point cloud data includes hand point cloud data from multiple moments ordered in chronological order, and the hand point cloud data at each moment includes the three-dimensional spatial coordinates related to the user's fingers at the current moment. , This represents the straight-line distance (i.e., depth value) between the point and the sensor. This represents the screen plane projection coordinates corresponding to that point. The hand point cloud data is the raw measurement result generated pixel by pixel or feature point by feature point for the user's finger at each time.

[0079] It should be noted that, in this embodiment of the application, since the user's eye image data and the user's hand point cloud data are collected simultaneously, the eye image at each moment is associated with the hand point cloud data at the current moment. This embodiment of the application does not impose specific restrictions on the construction method of the association between the eye image and the hand point cloud data at each moment.

[0080] Figure 5 This is a schematic diagram illustrating the process of determining the target region in an embodiment of this application; as shown Figure 5 As shown, determining whether the first region is a target region based on the user's hand point cloud data includes: Step 301: Starting from the second moment associated with the first region, calculate the first cosine value of the angle between the hand movement direction vector and the target direction vector at each moment, and obtain the first cosine value at one or more moments; wherein, the second moment is the moment when the time of dwell time of the user's gaze direction vector in the first region is started, the hand movement direction vector at each moment is the direction vector from the center point of each moment to the center point of the first moment, and the target direction vector is the direction vector from the center point of the first moment to the center point of the first region.

[0081] In this embodiment, after calculating the first two-dimensional position information of the center point of the hand point cloud data at each time, starting from the first time associated with the first region, the first cosine value of the angle between the hand movement direction vector and the target direction vector at each subsequent time is calculated to obtain the first cosine value of one or more times after the second time.

[0082] In this embodiment, the second moment is the moment when the time for timing the duration of the user's gaze direction vector's stay in the first region begins, that is, the moment when the "gaze point" mapped by the user's gaze direction vector in the display interface begins to enter the first region, which is also the moment when the time for timing the relative stillness of the determined relatively still region begins. One or more moments refer to all moments included from the first moment until the end of the stay time in the first region.

[0083] The direction vector of hand movement at each moment is the direction vector from the center point of the (finger) at each moment after the second moment to the center point of the (finger) at the second moment. The direction vector of the center point at the second moment represents the direction vector of the finger at the initial moment in the first region.

[0084] For example, the hand movement direction vector at two consecutive moments (frame t and frame t+1). and The calculation process can be designed as follows:

[0085] (6) in, Two-dimensional position information representing the center point of the (finger) at the second moment; and These represent the two-dimensional position information of the (finger) center point in frame t and frame t+1, respectively.

[0086] The target direction vector is the direction vector from the center point of the (finger) at the second moment to the center point of the first region. For example, the target direction vector... The calculation process can be designed as follows: (7) in, Two-dimensional position information representing the center point of the (finger) at the second moment; This represents the two-dimensional position information of the center point of the first region, and the position information of the four vertices of the first region are as follows: In this case, the two-dimensional location information of the center point of the first region .

[0087] Furthermore, based on the hand movement direction vectors at two consecutive time points (frame t and frame t+1)... and and target direction vector Calculate the first cosine value of the hand movement direction vector and the target direction vector in frame t. And calculate the first cosine value of the hand movement direction vector and the target direction vector in the (t+1)th frame. The specific process can be designed as follows:

[0088] (8) Step 302: Based on the first cosine value of each time point in one or more time points, determine the first trend value of each time point. The first trend value represents the degree to which the fourth time point in two adjacent time points approaches the first region relative to the third time point. The fourth time point is the next time point in the time sequence of the third time point.

[0089] In this embodiment, after determining the first cosine value of one or more times after the first time, the first trend value of each time is determined based on the first cosine value of each time, so as to determine whether the first region is the target region in the subsequent determination.

[0090] In this embodiment, the first trend value represents the degree of approach of the fourth moment relative to the third moment towards the first region between two adjacent moments. The fourth moment is the next moment after the third moment in the time sequence. That is, the first trend value represents the degree of approach of the next moment relative to the current moment. It is a discrimination index used to quantify whether the user's finger movement at the next moment is stably approaching the first region relative to the current moment.

[0091] Figure 6 This is a schematic diagram of the process for determining the first trend value in an embodiment of this application; as shown below. Figure 6 As shown, determining the first trend value at each time point based on the first cosine value at each of the one or more time points includes: Step 401: Determine the first two-dimensional position information corresponding to the center point of the hand point cloud data at multiple time points.

[0092] In this embodiment, the first two-dimensional position information of the finger center point at that moment is calculated based on the three-dimensional spatial coordinates of each point included in the hand point cloud data at the current moment.

[0093] In some optional implementations, determining the first two-dimensional position information corresponding to the center point of the user's hand point cloud data at multiple times includes: performing two-dimensional transformation processing on the user's hand point cloud data at the multiple times based on preset camera parameters to obtain a set of two-dimensional position information corresponding to the user's hand point cloud data at each time; and calculating the position information corresponding to the center point of the user's hand point cloud data at each time based on the set of two-dimensional position information corresponding to the user's hand point cloud data at each time.

[0094] In this embodiment, the specific calculation process of the first and second position information of the finger center point at each moment is as follows: according to the preset camera parameters, the three-dimensional spatial coordinates of each point in the user's hand point cloud data at each moment are transformed into two dimensions to obtain the two-dimensional position information set corresponding to the user's hand point cloud data at each moment. The two-dimensional position information set includes the two-dimensional position information of each point in the user's hand point cloud data. Based on the two-dimensional position information of each point in the user's hand point cloud data, the position information corresponding to the center point of the user's hand point cloud data at the current moment is calculated to obtain the first two-dimensional position information of the finger center point at each moment.

[0095] In this embodiment, camera parameters are used to convert the three-dimensional spatial coordinates of the user's hand point cloud data in the world coordinate system into two-dimensional position information in the screen coordinate system, that is, to realize two-dimensional coordinate transformation. The camera parameters may specifically include a camera intrinsic parameter matrix and a camera extrinsic parameter matrix. The camera intrinsic parameter matrix is ​​a matrix used to describe the internal geometric characteristics of the camera's optical imaging system, and may specifically include parameters such as focal length and principal point position information. The camera extrinsic parameter matrix is ​​a transformation matrix representing the transformation from the world coordinate system to the camera coordinate system, used to convert the camera's spatial position and orientation into a mathematical matrix that can be processed by a computer, and may specifically include a rotation matrix and / or a translation vector.

[0096] For example, using the camera intrinsic parameter matrix and camera intrinsic parameter matrix ( For camera rotation parameters, (For camera translation parameters), the three-dimensional spatial coordinates of each point in the user's hand point cloud data. Convert to two-dimensional location information The process can be designed as follows: (9) Furthermore, based on the two-dimensional position information of each point in the user's hand point cloud data after two-dimensional transformation... Calculate the center point The specific process can be designed as follows:

[0097] (10) The current user hand point cloud data includes n vertices, and the two-dimensional position information of the i-th vertex is: .

[0098] Step 402: Based on the target location information of the center point of the first region and the first two-dimensional location information of the center point at each of the one or more time moments, determine the first distance information from the center point of the first region at each time moment to the center point of the first region.

[0099] In this embodiment, the specific process of calculating the first trend value at each moment is as follows: First, based on the target position information of the center point of the first region and the first two-dimensional position information of the (finger) center point at each moment after the second moment, the first distance information from the (finger) center point to the center point of the first region at each moment is calculated.

[0100] In this embodiment, the first distance information is used to represent the change in distance between the center point of the (finger) and the center point of the first region at the current moment, that is, the change in distance between the user's finger and the first region at the current moment.

[0101] For example, based on the two-dimensional position information of the (finger) center point at two consecutive time points (frame t and frame t+1). and Calculate the center point of the (finger) and the target position information in frame t. First distance information And the center point and target position information of the (finger) in frame t+1. First distance information The specific process can be designed as follows: (11) Step 403: Based on the first cosine value of each of one or more time moments, calculate the angle between the user's hand movement direction vector and the target direction vector at each time moment to obtain the first angle corresponding to each time moment.

[0102] In this embodiment, based on the first cosine value of the angle between the user's hand movement direction vector and the target direction vector at each time after the second time, the angle between the user's hand movement direction vector and the target direction vector at each time is calculated using the inverse cosine function, thus obtaining the first angle corresponding to each time.

[0103] Step 404: When the first included angle corresponding to the third time moment and the first included angle corresponding to the fourth time moment satisfy the first preset condition, calculate the first trend value of the fourth time moment based on the first distance information and the first cosine value associated with the third time moment and the first distance information and the first cosine value associated with the fourth time moment. The first preset condition is that the first included angle corresponding to each of the two adjacent times moment moments is within the preset included angle range. When the first included angle corresponding to the third time moment and the first included angle corresponding to the fourth time moment do not satisfy the first preset condition, set the first trend value of the fourth time moment to 0.

[0104] In this embodiment, the first included angle of each time in the adjacent time after the second time is filtered using the first preset condition.

[0105] In this embodiment, the first preset condition is that the first included angle corresponding to each of two adjacent moments is within a preset included angle range. For example, for the first included angle... The first preset condition can be set to That is, the first angle between the third and fourth moments in adjacent moments is greater than or equal to 0 degrees and less than 90 degrees.

[0106] In this embodiment, for the third and fourth times among adjacent times that satisfy the first preset condition, the first trend value of the fourth time is calculated based on the first distance information and the first cosine value associated with the third time and the first distance information and the first cosine value associated with the fourth time; for the third and fourth times among adjacent times that satisfy the first preset condition, the first trend value of the fourth time is directly set to 0.

[0107] For example, the first preset condition is set as follows: For example, the first trend value of frame t+1. The calculation process can be designed as follows: (12) in, and These represent the first distance information in frame t and frame (t+1), respectively. and Let represent the first cosine values ​​of the t-th frame and the (t+1)-th frame, respectively.

[0108] Step 303: Based on the first trend value at each time point, determine whether the first region is the target region.

[0109] In this embodiment, after obtaining the first trend value at each time after the second time, it is determined whether the user's finger remains inclined towards the first region after the second time based on the first trend value at each time, that is, whether the first region is the target region.

[0110] In some optional implementations, determining whether the first region is a target region based on the first trend value at each time step includes: determining a first number of times when the first trend value is greater than 0 based on the first trend value at each of the one or more times step steps; and determining the first region as the target region when the first number is greater than or equal to a third threshold.

[0111] In this embodiment, the first number of times when the first trend value is greater than 0 is counted among all the first trend values ​​at all times. When the first number is greater than or equal to the third threshold, the first region is determined as the target region, that is, the first region is determined as the target region that needs further interaction.

[0112] For example, for n time points ( The first trend value of ) The third threshold can be preset to 80%. That is, when the first trend value When the first quantity is greater than or equal to 80% of the total number n, it represents the first trend value. If the probability exceeds 80%, the first region can be identified as the target region, thus providing a basis for the next step of interaction.

[0113] Based on the above embodiments, this application also provides an interactive device. Figure 7 This is a schematic diagram of the composition structure of the interactive device provided in the embodiments of this application; as shown below. Figure 7 As shown, the device includes: an acquisition unit 51, a processing unit 52, a display unit 53, and a collection unit 54; wherein, The acquisition unit 51 is used to acquire user input information; The processing unit 52 is used to generate initial output text based on the user input information; The display unit 53 is used to control the display of the initial output text on the display interface; The acquisition unit 54 is used to acquire user eye image data and user hand point cloud data; The processing unit 52 is further configured to determine a first region in the display interface based on the user's eye image data, determine target text information in the initial output text corresponding to the first region, wherein the first region represents a potential interactive region; determine whether the first region is a target region based on the user's hand point cloud data, wherein the target region represents an area to be interacted with; and generate interactive information related to the target text information if the first region is determined to be a target region.

[0114] In one optional embodiment of this application, the user's eye image data includes multiple eye images ordered chronologically. The processing unit 52 is used to extract features from the multiple eye images respectively to obtain eye movement feature information corresponding to each eye image. The eye movement feature information includes first position information and second position information related to the user's eyeball. The first position information represents pupil position information, and the second position information represents Pulchin spot position information. Based on the first position information and second position information corresponding to each eye image in the multiple eye images, a first region in the display interface is determined.

[0115] In an optional embodiment of this application, the processing unit 52 is configured to determine a first gaze direction vector of the user at a first moment based on first position information and second position information corresponding to a first eye image; the first eye image is any one of the plurality of eye images, and the first moment is the time corresponding to the first eye image; based on the first position information corresponding to each of the plurality of eye images, determine one or more feedback regions in the display interface; based on the gaze direction vector at each of the plurality of moments, determine a first region satisfying a first constraint condition from the one or more feedback regions, wherein the first constraint condition includes: the angle between the gaze direction vector and the center direction vector of the first feedback region is less than a first threshold, and the duration of the gaze direction vector in the first feedback region is greater than or equal to a second threshold, and the first feedback region is any one of the one or more feedback regions.

[0116] In one optional embodiment of this application, the user's hand point cloud data includes multiple hand point cloud data ordered in chronological order. The processing unit 52 is configured to calculate, starting from a second moment associated with the first region, the first cosine value of the angle between the hand movement direction vector and the target direction vector at each moment, to obtain one or more first cosine values ​​at each moment; wherein, the second moment is the moment when the duration of the user's gaze direction vector remaining in the first region is started, the hand movement direction vector at each moment is the direction vector from the center point of each moment to the center point of the second moment, and the target direction vector is the direction vector from the center point of the second moment to the center point of the first region; based on the first cosine values ​​of each of the one or more moments, a first trend value is determined for each moment, the first trend value representing the degree of approach of the fourth moment relative to the third moment to the first region among two adjacent moments, the fourth moment being the next moment in the chronological order of the third moment; based on the first trend values ​​of each moment, it is determined whether the first region is a target region.

[0117] In one optional embodiment of this application, the processing unit 52 is configured to: determine first two-dimensional position information corresponding to the center point of the user's hand point cloud data at multiple time moments; determine first distance information from the center point of the first region to the center point of the first region at each time moment based on the target position information of the center point of the first region and the first two-dimensional position information of the center point of each of the one or more time moments; calculate the angle between the user's hand movement direction vector and the target direction vector at each time moment based on the first cosine value of each of the one or more time moments to obtain the first angle corresponding to each time moment; when the first angle corresponding to the third time moment and the first angle corresponding to the fourth time moment satisfy a first preset condition, calculate the first trend value of the fourth time moment based on the first distance information and the first cosine value associated with the third time moment and the first distance information and the first cosine value associated with the fourth time moment, wherein the first preset condition is that the first angle corresponding to each of the two adjacent time moments is within a preset angle range; and when the first angle corresponding to the third time moment and the first angle corresponding to the fourth time moment do not satisfy the first preset condition, set the first trend value of the fourth time moment to 0.

[0118] In one optional embodiment of this application, the processing unit 52 is configured to determine a first number of times when the first trend value is greater than 0 based on the first trend value of each of the one or more times, and when the first number is greater than or equal to a third threshold, determine the first region as the target region.

[0119] In one optional embodiment of this application, the processing unit 52 is configured to perform two-dimensional transformation processing on the user hand point cloud data at the multiple time points based on preset camera parameters to obtain a set of two-dimensional position information corresponding to the user hand point cloud data at each time point; and calculate the position information corresponding to the center point of the user hand point cloud data at each time point based on the set of two-dimensional position information corresponding to the user hand point cloud data at each time point.

[0120] In this application, the acquisition unit 51, processing unit 52, display unit 53 and acquisition unit 54 in the device can be implemented by a central processing unit (CPU), digital signal processor (DSP), microcontroller unit (MCU) or field-programmable gate array (FPGA) in practical applications.

[0121] It should be noted that the above-described interactive device is only illustrated by the division of the program modules. In practical applications, the processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the interactive device and interactive method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0122] This application also provides a computer device. Figure 8 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application; as shown below. Figure 8 As shown, the computer device includes a memory 62, a processor 61, and a computer program stored on the memory 62 and executable on the processor 61.

[0123] Optionally, the computer device also includes at least one communication component 64. The various components in the computer device can be coupled together via a bus system 63. It is understood that the bus system 63 is used to implement communication between these components. In addition to a data bus, the bus system 63 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 8 The general designated all buses as Bus System 63.

[0124] It is understood that memory 62 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 62 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0125] The methods disclosed in the embodiments of this application can be applied to processor 61, or implemented by processor 61. Processor 61 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 61 or by instructions in the form of software. The processor 61 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 61 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 62. Processor 61 reads the information in memory 62 and completes the steps of the aforementioned method in combination with its hardware.

[0126] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0127] This application also provides a computer-readable storage medium on which a computer program is stored.

[0128] Optionally, the computer-readable storage medium can be applied to the interactive device of the present application embodiment; then when the program is executed by the processor, it implements the steps of the interactive method of the present application embodiment.

[0129] This application also provides a computer program product, including a computer program that can be executed by a processor 61 of an electronic device to complete the steps of the interaction method described in this application.

[0130] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0131] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0132] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0134] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0136] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0138] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An interaction method, characterized in that, The method includes: Obtain user input information, generate initial output text based on the user input information, and control the display of the initial output text on the display interface; Acquire user eye image data and user hand point cloud data; Based on the user's eye image data, a first region in the display interface is determined, and target text information corresponding to the first region in the initial output text is determined. The first region represents a potential interactive area. Based on the user's hand point cloud data, it is determined whether the first region is a target region, where the target region represents the region to be interacted with; If the first region is determined to be the target region, interactive information related to the target text information is generated.

2. The method of claim 1, wherein, The user eye image data includes multiple eye images ordered chronologically. Determining the first region in the display interface based on the user eye image data includes: Feature extraction is performed on the multiple eye images to obtain eye movement feature information corresponding to each eye image. The eye movement feature information includes first position information and second position information related to the user's eyeball. The first position information represents the pupil position information, and the second position information represents the Pulchin spot position information. Based on the first position information and the second position information corresponding to each eye image in the plurality of eye images, a first region in the display interface is determined.

3. The method according to claim 2, characterized in that, The step of determining the first region in the display interface based on the first and second position information corresponding to each eye image in the plurality of eye images includes: Based on the first position information and the second position information corresponding to the first eye image, the first gaze direction vector of the user at the first moment is determined; the first eye image is any one of the plurality of eye images, and the first moment is the time corresponding to the first eye image; Based on the first position information corresponding to each of the plurality of eye images, one or more feedback areas in the display interface are determined. Based on the gaze direction vector at each of multiple time points, a first region satisfying a first constraint condition is determined from the one or more feedback regions. The first constraint condition includes: the angle between the gaze direction vector and the center direction vector of the first feedback region is less than a first threshold, and the dwell time of the gaze direction vector in the first feedback region is greater than or equal to a second threshold. The first feedback region is any one of the one or more feedback regions.

4. The method according to any one of claims 1 to 3, characterized in that, The user's hand point cloud data includes multiple hand point cloud data arranged in chronological order. The step of determining whether the first region is the target region based on the user's hand point cloud data includes: Starting from the second moment associated with the first region, calculate the first cosine value of the angle between the hand movement direction vector and the target direction vector at each moment to obtain one or more first cosine values; wherein, the second moment is the moment when the duration of the user's gaze direction vector staying in the first region is started, the hand movement direction vector at each moment is the direction vector from the center point of each moment to the center point of the second moment, and the target direction vector is the direction vector from the center point of the second moment to the center point of the first region; Based on the first cosine value of each of the one or more times, a first trend value is determined for each time. The first trend value represents the degree to which the fourth time is closer to the first region than the third time in two adjacent times. The fourth time is the next time in the time sequence of the third time. Based on the first trend value at each time point, determine whether the first region is the target region.

5. The method according to claim 4, characterized in that, Determining the first trend value at each time point based on the first cosine value at each of the one or more time points includes: The first two-dimensional position information corresponding to the center point of the user's hand point cloud data at multiple time points is determined respectively; Based on the target location information of the center point of the first region, and the first two-dimensional location information of the center point at each of the one or more time moments, the first distance information from the center point to the center point of the first region at each time moment is determined. Based on the first cosine value of each of the one or more time moments, calculate the angle between the user's hand movement direction vector and the target direction vector at each time moment to obtain the first angle corresponding to each time moment; When the first included angle corresponding to the third time and the first included angle corresponding to the fourth time satisfy the first preset condition, the first trend value of the fourth time is calculated based on the first distance information and the first cosine value associated with the third time and the first distance information and the first cosine value associated with the fourth time. The first preset condition is that the first included angle corresponding to each of the two adjacent times is within the preset included angle range. When the first included angle corresponding to the third time point and the first included angle corresponding to the fourth time point do not satisfy the first preset condition, the first trend value of the fourth time point is set to 0.

6. The method according to claim 4, characterized in that, The step of determining whether the first region is a target region based on the first trend value at each time point includes: Based on the first trend value of each of the one or more time points, a first number of time points with the first trend value greater than 0 is determined. When the first number is greater than or equal to a third threshold, the first region is determined to be the target region.

7. The method according to claim 5, characterized in that, The step of determining the first two-dimensional position information corresponding to the center point of the user's hand point cloud data at multiple time points includes: Based on preset camera parameters, the user hand point cloud data at multiple time points are subjected to two-dimensional transformation processing to obtain a set of two-dimensional position information corresponding to the user hand point cloud data at each time point. Based on the set of two-dimensional location information corresponding to the user's hand point cloud data at each time point, the location information corresponding to the center point of the user's hand point cloud data at each time point is calculated.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

Images