A presentation interaction method, apparatus, device, and storage medium

By combining facial 3D physical coordinates and gaze vectors with microphone array sound source localization and large language models, the problem of binding fuzzy semantic commands in existing technologies has been solved, enabling precise interaction in complex exhibition hall environments and improving the accuracy of robot responses.

CN122837642BActive Publication Date: 2026-11-17DIGITAL HUAXIA (SHENZHEN) TECHNOLOGY CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611341112.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-09-01
Publication Date
2026-11-17
Estimated Expiration
2046-09-01

AI Technical Summary

Technical Problem

Existing speech recognition systems cannot bind ambiguous semantic commands to specific two-dimensional pixel areas on large screens in complex exhibition hall environments, resulting in the inability to accurately and uniquely locate the target object pointed to by the audience, causing interaction interruptions or response errors.

Method used

By filtering facial 3D physical coordinates and gaze vectors, combined with microphone array sound source localization and large language models, fuzzy semantics are bound to screen content to identify the target object and generate accurate answer text.

Benefits of technology

It enables precise binding of ambiguous semantics with specific screen content in high-density information environments, improving the accuracy and naturalness of robot interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122837642B_ABST
    Figure CN122837642B_ABST
Patent Text Reader

Abstract

This application discloses a speech interaction method, apparatus, device, and storage medium, relating to the field of speech interaction technology. The method includes: screening candidate audience members and identifying the speaker; if the speaker's gaze vector falls within the three-dimensional coordinate range of the screen, then identifying the speaker as the target object; determining the two-dimensional coordinates of the target object's gaze point on the screen based on the three-dimensional physical intersection of the target object's gaze vector and the three-dimensional coordinate range of the screen; determining the interactive content block into which the gaze point's two-dimensional coordinates fall as the target block; replacing ambiguous spatial pronouns in the target object's speech commands with text content associated with the target block, and sending the replaced speech commands as prompts to a large language model, which then generates answer text based on the content of the target block. This method can bind ambiguous semantics to specific screen content, making accurate and specific content responses, and improving interaction accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of presentation interaction technology, and in particular to a presentation interaction method, apparatus, device and storage medium. Background Technology

[0002] With the rapid development of intelligent robots, digital exhibition halls, and smart education, automated explanation robots and automated document playback systems have been widely used in various speeches, product launches, and exhibition booth guidance scenarios. However, in complex exhibition hall environments, when multiple charts / content blocks are displayed on a large screen, visitors often use vague spatial pronouns (such as "Tell me about this") to make voice inquiries. Existing speech recognition systems lack spatial directionality perception capabilities and cannot accurately bind vague semantic commands to specific two-dimensional pixel areas on the large screen. This makes it impossible to accurately and uniquely locate the target object pointed to by the visitor, which can easily lead to interaction interruptions or response errors.

[0003] Therefore, how to solve the above-mentioned technical defects has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a speech interaction method, device, equipment, and storage medium that can bind ambiguous semantics to specific screen content, make accurate and exclusive content responses, and improve the accuracy of interaction.

[0005] To address the aforementioned technical problems, this application provides a presentation interaction method, comprising: When the audience issues a voice command, the audience whose three-dimensional physical coordinates of their face relative to the absolute origin fall within the sound source localization range of the microphone array during the time period when the voice command is issued are selected as candidate audiences. If the standard deviation of the lip aspect ratio sequence of the candidate audience exceeds the preset action threshold, then the candidate audience is determined to be the vocal subject. If the gaze vector of the sound-emitting object falls within the three-dimensional coordinate range of the screen, then the sound-emitting object is determined to be the target object; Based on the three-dimensional physical intersection of the target object's gaze vector and the screen's three-dimensional coordinate range, determine the two-dimensional coordinates of the target object's gaze point on the screen; The interactive content block into which the two-dimensional coordinates of the gaze point fall is determined as the target block; The fuzzy spatial pronouns in the voice command of the target object are replaced according to the text content associated with the target block, and the replaced voice command is sent as a prompt word to the large language model so that the large language model can generate the answer text according to the content of the target block.

[0006] In some embodiments, it also includes: Obtain the three-dimensional physical coordinates of each audience member's face relative to the absolute origin; Based on the three-dimensional physical coordinates, viewers within the line-of-sight determination range are selected to obtain the target viewers; Obtain preset facial key points of the target audience, and determine the gaze vector of the target audience based on the preset facial key points; Based on the gaze vector and the three-dimensional coordinate range of the screen, determine the number of viewers whose gaze is deviating from the screen; The percentage of viewers whose eyes were off-screen was counted out of the total target audience. If the ratio reaches a preset threshold, a page-turning command is sent and a corresponding voice explanation is triggered.

[0007] In some embodiments, it also includes: The robot reads the answer text aloud; wherein the robot turns to the physical spatial location corresponding to the center point of the target block and alternates its gaze between the target block and the target object.

[0008] In some embodiments, determining the gaze vector of the target audience based on the preset facial key points includes: The absolute perspective error of the yaw angle and the absolute perspective error of the pitch angle are determined based on the preset facial key points. The original yaw angle is corrected based on the absolute perspective error of the yaw angle, and the original pitch angle is corrected based on the absolute perspective error of the pitch angle. The line-of-sight vector is determined based on the corrected yaw and pitch angles.

[0009] In some embodiments, determining the number of target viewers whose gaze deviates from the screen based on the gaze vector and the three-dimensional coordinate range of the screen includes: The target number of viewers whose gaze deviates from the screen for more than a preset time threshold is determined based on the gaze vector and the three-dimensional coordinate range of the screen.

[0010] In some embodiments, if the ratio reaches a preset threshold, sending a page-turning command and triggering corresponding voice narration includes: If the duration for which the ratio reaches the preset threshold reaches the preset duration, a page-turning command is sent and the corresponding voice explanation is triggered.

[0011] In some embodiments, it also includes: During the initialization setup phase, the line-of-sight determination range is determined.

[0012] To address the aforementioned technical problems, this application also provides a presentation interaction device, comprising: The candidate audience screening unit is used to screen audiences whose three-dimensional physical coordinates of their faces relative to the absolute origin fall within the sound source localization range of the microphone array during the time period when the audience issues a voice command. The object determination unit is used to determine the candidate audience as the vocal object if the rolling standard deviation of the lip aspect ratio sequence of the candidate audience exceeds a preset action threshold. The target object determination unit is used to determine the sound-emitting object as a target object if the line-of-sight vector of the sound-emitting object falls within the three-dimensional coordinate range of the screen. The two-dimensional coordinate determination unit for the gaze point is used to determine the two-dimensional coordinates of the gaze point of the target object on the screen based on the three-dimensional physical intersection of the gaze vector of the target object and the three-dimensional coordinate range of the screen. The target block determination unit is used to determine the interactive content block into which the two-dimensional coordinates of the gaze point fall as the target block; The replacement unit is used to replace the fuzzy spatial pronouns in the voice command of the target object according to the text content associated with the target block, and send the replaced voice command as a prompt word to the large language model so that the large language model can generate the answer text according to the content of the target block.

[0013] To address the aforementioned technical problems, this application also provides an electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the presentation interaction method as described above.

[0014] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the presentation interaction method described above.

[0015] The speech interaction method provided in this application includes: when a viewer issues a voice command, selecting viewers whose three-dimensional physical coordinates of their face relative to the absolute origin fall within the sound source localization range of the microphone array during the time period of the voice command as candidate viewers; if the standard deviation of the lip aspect ratio sequence of the candidate viewer exceeds a preset action threshold, then the candidate viewer is determined to be the voice-generating object; if the gaze vector of the voice-generating object falls within the three-dimensional coordinate range of the screen, then the voice-generating object is determined to be the target object; based on the three-dimensional physical intersection of the gaze vector of the target object and the three-dimensional coordinate range of the screen, the two-dimensional coordinates of the gaze point of the target object on the screen are determined; the interactive content block into which the two-dimensional coordinates of the gaze point fall is determined to be the target block; the fuzzy spatial pronouns in the voice command of the target object are replaced according to the text content associated with the target block, and the replaced voice command is sent as a prompt word to a large language model, so that the large language model can generate answer text based on the content of the target block.

[0016] As can be seen, this application solves the speaker's non-contact 3D gaze vector and the absolute plane equation of the large screen simultaneously, and combines this with a 2D pixel dimensionality reduction mapping algorithm to accurately and uniquely pinpoint the specific content block (e.g., a specific data curve, comparison chart, or core data panel) currently being gazed at by the speaker at both the geometric and display levels. This enables the robot to bind ambiguous semantics to specific screen content and make accurate, personalized content responses, significantly improving the robot's interaction accuracy in high-density information environments.

[0017] The presentation interaction device, equipment, and computer-readable storage medium provided in this application all have the aforementioned technical effects. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the prior art and embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a presentation interaction method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a spatial directional explanation scene based on the fusion of gaze and semantics, provided in an embodiment of this application. Figure 3 This is a schematic diagram of key points of the mouth provided in an embodiment of this application; Figure 4 A flowchart illustrating another presentation interaction method provided in an embodiment of this application; Figure 5 A schematic diagram of a dynamic speech control scenario based on group gaze deviation provided in an embodiment of this application; Figure 6 This is a schematic diagram of a speech interaction system provided in an embodiment of this application; Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The core of this application is to provide a speech interaction method, device, equipment, and storage medium that can bind ambiguous semantics to specific screen content, make accurate and exclusive content responses, and improve the accuracy of interaction.

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a presentation interaction method provided in an embodiment of this application. (Refer to...) Figure 1 As shown, the method includes: S101: When a viewer issues a voice command, viewers whose three-dimensional physical coordinates of their face relative to the absolute origin fall within the sound source localization range of the microphone array during the time period when the voice command is issued are selected as candidate viewers. S102: If the standard deviation of the lip aspect ratio sequence of the candidate audience exceeds the preset action threshold, then the candidate audience is determined to be the vocal subject. S103: If the gaze vector of the sound-emitting object falls within the three-dimensional coordinate range of the screen, then the sound-emitting object is determined to be the target object; S104: Determine the two-dimensional coordinates of the point where the target object's gaze falls on the screen based on the three-dimensional physical intersection of the target object's gaze vector and the three-dimensional coordinate range of the screen; S105: Determine the interactive content block into which the two-dimensional coordinates of the gaze point fall as the target block; S106: Replace the fuzzy spatial pronouns in the voice command of the target object according to the text content associated with the target block, and send the replaced voice command as a prompt word to the large language model so that the large language model can generate the answer text according to the content of the target block.

[0023] This embodiment aims to solve the semantic ambiguity problem that arises when viewers ask questions about information displayed on a large screen.

[0024] refer to Figure 2 As shown in the spatial directional explanation scenario based on gaze and semantic fusion, under continuous system monitoring, the depth camera tracks the facial key points of all viewers within the gaze determination range in real time (reference). Figure 3 As shown, let the left and right corners of the mouth be... , The three pairs of vertical points of the upper and lower lips are to ), and continuously calculate the lip aspect ratio (MAR) for each frame: ( , Corresponding to A / B respectively, to Corresponding to C, E, G, D, F, H respectively.

[0025] .

[0026] The system maintains a dynamic circular data buffer of length N frames for each viewer (e.g., recording data from the past 1.0 second), and updates and temporarily stores the arithmetic mean of their MAR sequence in real time. With rolling standard deviation This serves as a record of the recent lip-opening and closing status of all candidates.

[0027] When a viewer issues a vague voice command (such as "tell me a detailed explanation of this"), the microphone array instantly calculates the horizontal angle range from which the sound is coming through localization of sound source (DOA).

[0028] To eliminate interference from irrelevant people conversing in the environment, the system combines depth vision with buffered lip-sync feature streams for spatiotemporal backtracking cross-validation: the system extracts data within the time period of the voice command, first filtering out the absolute three-dimensional coordinates. Candidate faces falling within the microphone DOA angle range; then, the standard deviation of lip roll at that time was retrieved for these candidates. When a candidate's If the preset vocal action threshold is exceeded, the system initially determines that the user is a target who is speaking. In addition, the system requires that the viewer's gaze be within the screen in order to be ultimately confirmed as the only target with a valid interactive intent.

[0029] To verify the aforementioned line-of-sight conditions and obtain precise landing points, the system performs line-of-sight vector calculation and spatial ray collision on the initially identified sound-emitting object. This stage involves face affine alignment, regression angle calculation based on the Multi-Bin mechanism, and dynamic spatial disparity compensation to generate the line-of-sight vector. And solving the three-dimensional physical intersection points by combining the absolute plane equations of the large screen. The processing flow and mathematical calculations can be referred to the above embodiments, and will not be repeated here. The system determines the obtained intersection points. If the target falls within the preset physical three-dimensional boundary of the large screen, and the conditions are met, then the final target locking is officially completed, that is, the target object is determined.

[0030] Obtain the three-dimensional physical intersection of the target object Then, screen intersection dimensionality reduction and pixel coordinate mapping, as well as semantic content matching and spatial pointing control are further performed.

[0031] Screen intersection dimensionality reduction and pixel coordinate mapping: Let the three-dimensional coordinates of the top left corner of the large screen be... Its horizontal and vertical unit vectors are Physical length and width are and The display resolution is .

[0032] System calculates intersection points Relative to the screen origin physical space offset vector By using the dot product Projecting the image onto a partial plane of the screen yields the physical horizontal and vertical distances: ; .

[0033] Using a proportional affine mapping, it is transformed into two-dimensional pixel coordinates of the current display interface on the large screen. : ; .

[0034] Semantic content matching and spatial orientation control: To achieve precise interaction and dynamic responses to screen content, during the initial configuration phase, the system requires manual pre-defined block marking and text binding for specific interactive content displayed on the large screen: Manual block delineation and content binding: During the system configuration phase, operators manually delineate the interactive content blocks on the display interface. The system records the pixel bounding box of each block at the current 2D screen resolution, denoted as its diagonal coordinates. At the same time, the corresponding text content or semantic tags are bound to this block. (For example, a detailed data description or topic name for the chart), construct a global pixel coordinate-text content mapping association table.

[0035] Pixel collision and target matching: The system iterates through the current preset mapping association table in real time, matching the two-dimensional pixel coordinates... Collision detection is performed with the pre-defined bounding boxes of each block. Collision detection is performed if and only if the condition is met. and At that time, the system determined that the coordinates of the two-dimensional pixel specifically fell into the first... Each content block.

[0036] Semantic reference resolution and dynamic solution generation: After identifying the target block, the system extracts the text content associated with that block. The system performs semantic referencing resolution, replacing ambiguous spatial pronouns in the audience's spoken instructions. For example, "Give me a detailed explanation of this" is automatically replaced with "Give me a detailed explanation of the revenue data for each region in 2023." The system then sends this semantically complete instruction as a prompt to the Large Language Model (LLM). The LLM dynamically generates targeted answer text based on the block content in real time, thus avoiding the limitations of pre-set, rigid explanation words.

[0037] This embodiment solves the speaker's non-contact 3D gaze vector and the absolute plane equation of the large screen simultaneously, and combines this with a 2D pixel dimensionality reduction mapping algorithm to accurately and uniquely pinpoint the specific content block (such as a specific data curve, comparison chart, or core data panel) currently being gazed at by the speaker at both the geometric and display levels. This enables the robot to bind ambiguous semantics to specific screen content and make accurate physical turns and specific content responses, significantly improving the robot's interaction accuracy in high-density information environments.

[0038] Existing document playback systems typically use fixed-duration page turning or rely on manual control by the speaker using a physical remote control or page turner. These systems cannot perceive the audience's attention span and information reception level in real time, leading to a disconnect between the presentation pace and the audience's actual comprehension. For example, when the page content is simple and the audience has finished reading, the system remains ineffective; when the content is complex and the audience is still thinking, the system automatically turns the page. To address this technical deficiency, [reference needed]. Figure 4 As shown, in some embodiments, the presentation interaction method provided in this application further includes: S201: Obtain the three-dimensional physical coordinates of each audience member's face relative to the absolute origin.

[0039] S202: Based on the three-dimensional physical coordinates, select the audience members within the line of sight to obtain the target audience.

[0040] The execution subject of this application embodiment is a speech interaction system. This system pre-establishes a three-dimensional spatial coordinate system for the current venue. This three-dimensional spatial coordinate system has its origin (i.e., spatial coordinate system) at the camera on the robot body. The positive X-axis points horizontally to the right, the positive Y-axis points vertically downwards, and the positive Z-axis points directly in front of the camera (i.e., deep in physical space).

[0041] refer to Figure 5 As shown, a depth camera is used to capture the face of each viewer relative to the absolute origin. Three-dimensional physical coordinates The system filters out all viewers currently within the line of sight. Viewers within the line of sight are referred to as target viewers.

[0042] In some embodiments, it also includes: During the initialization setup phase, the line-of-sight determination range is determined.

[0043] During the initialization setup phase, complete the following two initialization settings: The first is to set the line-of-sight determination range: with the origin as the reference point. Based on the effective depth of field and field of view (FOV) of the camera, a three-dimensional spatial bounding box or sector is defined as the gaze determination range for identification and processing. Gait data is only collected and analyzed when the viewer's physical location is within this gaze determination range. Personnel activities outside this range are filtered out. The second step is large-screen spatial mapping: calculating and recording the relative physical coordinates and planar equations of the large screen in three-dimensional space with this camera as the origin, establishing the spatial relative positional relationship between the robot and the large screen.

[0044] S203: Obtain the preset facial key points of the target audience, and determine the gaze vector of the target audience based on the preset facial key points.

[0045] S204: Determine the number of viewers whose gazes are off-screen based on the gaze vector and the three-dimensional coordinate range of the screen.

[0046] The camera continuously captures preset facial key points of the target audience. Using a gaze estimation algorithm, the gaze vector of the target audience is extracted, and it is determined in real time whether the gaze vector falls within the preset three-dimensional coordinate range of the screen, thereby determining the number of audience members whose gaze is off-screen.

[0047] In some embodiments, determining the gaze vector of the target audience based on the preset facial key points includes: The absolute perspective error of the yaw angle and the absolute perspective error of the pitch angle are determined based on the preset facial key points. The original yaw angle is corrected based on the absolute perspective error of the yaw angle, and the original pitch angle is corrected based on the absolute perspective error of the pitch angle. The line-of-sight vector is determined based on the corrected yaw and pitch angles.

[0048] In the gaze estimation algorithm, the visual detection model locates the face in real time and captures 68 facial key points, including the centers of the eyes. Using the line connecting the centers of the eyes as a horizontal reference, the rotation angle and scaling scale are calculated to construct a two-dimensional affine transformation matrix, mapping the positions of the eyes to a preset standard pixel coordinate space. By applying affine transformations (including rotation, scaling, and translation), the non-rigid deformation of the original face image is eliminated, ensuring that the image input to the gaze estimation model has scale invariance. This significantly reduces the model's sensitivity to head tilt and improves the robustness of the solution.

[0049] The aligned face tensor is input into the gaze estimation algorithm model. The gaze estimation algorithm model employs a classification-to-regression multi-bin mechanism: the continuous angle space is divided into several discrete intervals (e.g., 90) by a span (e.g., 4 degrees), and the probability distribution of each interval is output independently. The system uses the Softmax function and the probability expectation formula to perform a weighted summation, mapping the discrete probabilities to continuous regression angles. .

[0050] Angles directly output from the model will exhibit distorted perspective due to the perspective principle of wide-angle lenses. Perspective principle refers to the geometric laws of central projection (e.g., objects appear larger when closer and smaller when farther away) that govern the projection of three-dimensional physical space onto a two-dimensional plane. In camera optical systems, when three-dimensional spatial coordinates are projected onto a two-dimensional pixel plane, a non-linear mapping based on depth of field and distance from the optical axis is generated.

[0051] To eliminate this optical error, the system extracts the horizontal and vertical coordinate pixel offsets (position scale factor) of the face center from the absolute center of the image (i.e., the camera's optical axis). The system utilizes the Pythagorean theorem ( The actual deviation radius of the center point of the face from the optical axis on the two-dimensional image plane was accurately calculated. Simultaneously, based on the pixel width and height of the current image, the system calculates the maximum physical field of view radius from the center of the image to the diagonal edge of the image. .

[0052] The system uses the above parameters to calculate the perspective error angle value. Due to the physical characteristics of the camera's wide-angle lens, the system sets a maximum edge distortion angle. (For example ,Right now (Radians). The system measures the coordinate offset of the face. Divide by the maximum field of view respectively The normalized deviation ratios for the lateral and longitudinal directions are obtained, and multiplied by the maximum distortion angle to calculate the absolute perspective errors of the yaw angle (left-right direction) and pitch angle (up-down direction). and : ; .

[0053] Yaw: In three-dimensional attitude estimation, it refers to the horizontal left and right rotation angle of the target's head or line of sight around the vertical axis (i.e., the direction of human movement such as looking left and right).

[0054] Pitch: In three-dimensional spatial pose estimation, it refers to the vertical rotation angle of the target's head or line of sight around the horizontal axis (i.e., the direction of human head-up and head-down movements).

[0055] The system starts from the original yaw angle and pitch angle In the middle, directly subtract this perspective error angle in reverse to obtain the corrected angle (represented as an angle with compensation properties). and ): ; .

[0056] By using the above-mentioned direct linear projection compensation formula, the perspective illusion caused by the deviation of the face position from the central optical axis is eliminated from a purely mathematical perspective, thus truly restoring the absolute physical angle in which the viewer's head and eyeballs actually point in three-dimensional physical space.

[0057] The three-dimensional line-of-sight direction vector is constructed. In the three-dimensional Cartesian coordinate system established in this embodiment, the spatial direction is defined as follows: with the optical center of the depth camera as the absolute origin. The positive X-axis points horizontally to the right, the positive Y-axis points vertically downwards, and the positive Z-axis points directly in front of the camera (i.e., deep in physical space).

[0058] The system utilizes the geometric transformation principle from spherical coordinates to Cartesian coordinates to calculate the three-dimensional unit direction vector, i.e., the gaze vector, representing the absolute gaze direction. .

[0059] ; ; .

[0060] With a fixed reference point Starting from the three-dimensional unit direction vector, construct the parametric equation of the line in the line-of-sight space ( ). (Length of the ray extending forward): .

[0061] In the above formula, X f Y f Z f For fixed reference points The coordinates in the X, Y, and Z directions.

[0062] Substitute the above line-space parametric equations into the large-screen absolute plane equations entered during the initialization phase. In the process of solving for parameters : .

[0063] Solving Then, substituting these values ​​back into the line-of-sight parametric equation, the coordinates of the three-dimensional physical intersection point between the line of sight and the large screen plane are obtained. .

[0064] Once the audio presentation of a single-page PPT is complete, the system enters monitoring mode. It extracts the three-dimensional physical coordinates of each audience member from the target audience. This is then compared with the physical 3D boundary of the screen. 3D physical intersection coordinates. If the view falls within the physical three-dimensional boundaries of the screen, it is considered gaze; otherwise, it is considered deviation.

[0065] S205: Calculate the percentage of viewers whose eyes are off the screen out of the total target audience.

[0066] S206: If the ratio reaches a preset threshold, a page-turning command is sent and a corresponding voice explanation is triggered.

[0067] The system continuously monitors the percentage of viewers whose gaze is currently off-screen relative to the total number of target viewers. If this percentage reaches a preset threshold, a page-turning command is sent and corresponding audio narration is triggered.

[0068] Unlike traditional methods that use timed or manual page turning, the interactive presentation method provided in this embodiment automatically determines the timing of page turning and turns the page automatically. Specifically, it filters out viewers within the line of sight based on the three-dimensional physical coordinates of the audience's face relative to the absolute origin, thus obtaining the target audience. It acquires preset facial key points of the target audience and determines the gaze vector of the target audience based on the preset facial key points. Based on the gaze vector and the three-dimensional coordinate range of the screen, it determines the number of viewers whose gaze is off-screen. When the proportion of viewers whose gaze is off-screen to the total number of target viewers reaches a preset threshold, that is, when it is determined that most people have finished reading or lost interest, the page is automatically turned. This can effectively eliminate invalid waiting or interruptions in the presentation process, making the entire presentation and guided tour process more compact and efficient.

[0069] In some embodiments, determining the number of target viewers whose gaze deviates from the screen based on the gaze vector and the three-dimensional coordinate range of the screen includes: The target number of viewers whose gaze deviates from the screen for more than a preset time threshold is determined based on the gaze vector and the three-dimensional coordinate range of the screen.

[0070] The system assigns an independent individual time window (e.g., 1.5 seconds) to each target audience member to prevent misjudging fleeting actions such as blinking or brief note-taking. When a target audience member's gaze falls outside the physical three-dimensional boundaries of the screen, the system begins timing them individually. If their gaze returns to the screen midway, the timer resets to zero. Only when the gaze continuously deviates from the screen for more than 1.5 seconds, exceeding the preset time threshold, is the system officially marked as having deviated. After filtering out all momentary movement interference from target audience members, the system calculates the proportion of deviated target audience members out of the total number of target audience members in real time.

[0071] In some embodiments, if the ratio reaches a preset threshold, sending a page-turning command and triggering corresponding voice narration includes: If the duration for which the ratio reaches the preset threshold reaches the preset duration, a page-turning command is sent and the corresponding voice explanation is triggered.

[0072] When the obtained proportion reaches a preset threshold, the system opens a group confirmation time window (e.g., 1.0 second). If the proportion remains at the preset threshold during this period, i.e., within the preset duration, the confirmation information is obtained, and the system automatically sends a next page instruction (page turning instruction) to the large screen, while simultaneously triggering the robot to seamlessly continue the next segment of voice narration.

[0073] In some embodiments, it also includes: The robot reads the answer text aloud; wherein the robot turns to the physical spatial location corresponding to the center point of the target block and alternates its gaze between the target block and the target object.

[0074] The robot uses text-to-speech (TTS) technology to read aloud the dynamically generated solution from the large model, while simultaneously controlling servo motors to precisely steer towards the target area. The physical spatial location corresponding to the center point is determined, and the eye contact between the target area and the audience asking the question is naturally alternated to complete high-precision human-computer interaction.

[0075] The subject of the presentation interaction method provided in the above embodiments of this application is a presentation interaction system. (Refer to...) Figure 6 As shown, this interactive presentation system consists of three core modules and several key components operating in tandem. The three core modules include a multimodal perception module, a hollow processing module, and an execution module.

[0076] The multimodal perception module, serving as the system's front-end data acquisition unit, comprises two main components: a depth camera and a microphone array. The depth camera not only captures two-dimensional images to extract key facial features and dynamic lip movements from the viewer, but also simultaneously acquires high-precision three-dimensional depth (Z-axis) data, providing physical distance mapping. The microphone array is responsible for omnidirectional listening and capturing the viewer's natural speech commands, while simultaneously using a sound source localization (DOA) algorithm to calculate the three-dimensional coordinates of the sound source.

[0077] The central control processing module, serving as the decision-making hub of the system, integrates a spatial modeling unit, a core gaze estimation algorithm, a 3D geometric collision detector, and an intent recognition and question-answering module. It is responsible for processing the data collected by the depth camera and microphone array and issuing action strategies.

[0078] The execution module is responsible for converting central control commands into physical and digital outputs. This module includes two execution components: a large screen controller and the robot itself. The large screen controller is responsible for interfacing with the digital display terminal to execute precise document page turning and UI block responses. The robot itself is equipped with multi-degree-of-freedom servo motors and a directional voice broadcast array, which is responsible for completing anthropomorphic eye-following steering and exclusive voice narration in physical space.

[0079] This system can be applied to dynamic speech control scenarios based on group gaze deviation and to large-screen content directionality explanation scenarios based on gaze and semantic fusion.

[0080] Dynamic speech control based on group gaze deviation often occurs in large digital exhibition halls, tiered smart classrooms, or large product launches where people are densely packed and the event space is open. In such group broadcasting environments, the presenter system often faces the challenge of scattered audience attention and difficulty in maintaining a unified focus. The system uses depth cameras to monitor the proportion of the audience's gaze and deviation from the large screen in real time within a defined gaze range. It automatically filters out background noise and interference from irrelevant personnel, accurately assesses the audience's information reception level, and adaptively adjusts the playback rhythm of the large screen text, effectively eliminating the ineffective waiting and lecture disruption caused by traditional fixed-frequency page turning in large venues.

[0081] The large-screen content-oriented explanation scenario based on gaze and semantic fusion targets multimedia interactive environments with text and image display needs, such as command centers displaying multiple data dashboards or interactive museum guide booths filled with complex charts. In such environments, viewers, facing multiple target blocks presented simultaneously on the large screen, are accustomed to using vague spatial pronouns like "this data" or "that chart" to ask questions. Faced with such complex environments with unclear multi-target referents, the system introduces a multimodal large model to perform real-time layout recognition and segmentation of the dynamic content on the large screen. It then performs dimensionality reduction mapping and semantic binding between the questioner's high-precision 3D gaze point and the sound source location captured by the microphone array, accurately pinpointing the specific screen text and image block the speaker is currently looking at. This enables highly accurate spatially oriented interactive answers, greatly improving the accuracy and naturalness of interaction in noisy, multi-visual interference environments. Whether for group presentations or individual Q&A sessions, the system must first complete the digital mapping of the physical environment and effective interactive areas.

[0082] The central control processing module pre-establishes a three-dimensional spatial coordinate system for the current site. This coordinate system uses the camera on the robot as its origin (i.e., spatial coordinate system). The positive X-axis points horizontally to the right, the positive Y-axis points vertically downwards, and the positive Z-axis points directly in front of the camera (i.e., deep in physical space).

[0083] Complete the following two initialization settings: 1. Set the line-of-sight determination range: Using the camera origin as a reference, define a three-dimensional bounding box or sector within the system based on the camera's effective depth of field and field of view (FOV). This will serve as the "line-of-sight determination range" for system recognition and processing. The system stipulates that line-of-sight data will only be collected and analyzed when the viewer's physical location is within this determination range; any movement outside this range will be filtered out by the system. 2. Large screen spatial mapping: Calculate and input the relative physical coordinates and planar equations of the large screen in the three-dimensional space with this camera as the origin, establishing the spatial relative positional relationship between the robot and the large screen.

[0084] After completing the above initialization and activating the multimodal sensing module (camera, microphone array), the system enters a continuous environmental monitoring state based on this coordinate system.

[0085] In a dynamic speech control scenario based on group gaze deviation, the robot itself acts as the speaker, dynamically controlling the playback rhythm of the large screen by sensing the group's attention within the effective range.

[0086] When the robot is used in conjunction with a large screen for explanation, the multimodal perception module uses a depth camera to capture the position of each face relative to the absolute origin. Three-dimensional physical coordinates The system filters out all viewers currently within the gaze detection range. The camera continuously captures the facial features of these viewers within this effective range. Using a gaze estimation algorithm, it extracts their gaze vectors and determines in real time whether these gaze vectors fall within the preset 3D coordinate range of the large screen.

[0087] Once the audio presentation of a single PPT slide is complete, the system enters monitoring mode. The central control processing module extracts the physical intersection point corresponding to each audience member in the group. The system compares the coordinates with the physical 3D boundaries of the screen: if the coordinates fall within the boundaries, it is considered gaze; otherwise, it is considered deviation. The system continuously calculates the proportion of viewers whose gazes are currently deviating from the screen relative to the total number of target viewers. Each viewer is assigned an independent individual time window (e.g., 1.5 seconds) to prevent misjudgment of momentary actions such as blinking or brief note-taking. When a viewer's gaze point falls outside the physical boundaries of the screen, the system starts timing them individually; if the gaze returns to the screen midway, the timer is reset to zero. Only when the gaze is continuously deviating from the screen for more than 1.5 seconds is the system officially marked as having deviated. After filtering out momentary movement interference from everyone, the system continuously calculates the proportion of viewers who have deviated relative to the total number of target viewers. When this proportion reaches a preset threshold (e.g., 70%), the system opens a 1.0-second group confirmation time window. If this high deviation proportion persists during this period, the confirmation information is obtained, the central control processing module automatically sends the next page instruction to the screen, and simultaneously triggers the robot to seamlessly continue the next segment of voice narration.

[0088] In scenarios where spatial directional explanations are based on the fusion of gaze and semantics, the depth camera tracks the facial key points of all viewers within the gaze range in real time and continuously calculates the lip aspect ratio (MAR) for each frame while the system is continuously monitoring.

[0089] The system maintains a dynamic circular data buffer of length N frames for each viewer (e.g., recording data from the past 1.0 second), and updates and temporarily stores the arithmetic mean of their MAR sequence in real time. The rolling standard deviation is used as a record of the recent mouth opening and closing status of all candidates.

[0090] When an audience member issues a vague voice command, the microphone array instantly calculates the horizontal angle range of the sound source using Direct Origin Analysis (DOA). To eliminate interference from irrelevant people conversing in the environment, the system combines depth vision with buffered lip-sync feature streams for spatiotemporal backtracking cross-validation: the system extracts data within the time period of the voice command, first filtering out absolute three-dimensional coordinates. Candidate faces falling within the microphone DOA angle range; then, the standard deviation of lip roll at that time was retrieved for these candidates. When a particular candidate's When the preset vocal action threshold is exceeded, the system initially determines that the user is the target who is speaking. In addition, the system requires that the viewer's gaze be within the screen in order to be ultimately confirmed as the sole target with valid interactive intent.

[0091] Obtain the three-dimensional physical intersection of the target object Then, perform the following operations: 1) Dimensionality reduction of screen intersection points and pixel coordinate mapping: Let the three-dimensional coordinates of the top left corner of the large screen be... Its horizontal and vertical unit vectors are Physical length and width are and The display resolution is .

[0092] System calculates intersection points Relative to the screen origin physical space offset vector By using the dot product Projecting the image onto a partial plane of the screen yields the physical horizontal and vertical distances: ; .

[0093] Then, using a proportional affine mapping, it was transformed into two-dimensional pixel coordinates of the current display interface on the large screen. : ; .

[0094] To achieve precise interaction and dynamic responses to screen content, during the initial configuration phase, the system requires manual pre-defined block marking and text binding for specific interactive content displayed on the large screen: Manual block delineation and content binding: During the system configuration phase, operators manually delineate the interactive content blocks on the display interface. The system records the pixel bounding box of each block at the current 2D screen resolution, denoted as its diagonal coordinates. At the same time, the corresponding text content or semantic tags are bound to this block. (For example, a detailed data description or topic name for the chart), construct a global pixel coordinate-text content mapping association table.

[0095] Pixel collision and target matching: The system's central control module iterates through the currently preset mapping association table in real time, and matches the previously calculated two-dimensional coordinates of the viewer's line of sight. Collision detection is performed with the pre-defined bounding boxes of each block. Collision detection is performed if and only if the condition is met. and At that time, the system clearly determined that the coordinates of the two-dimensional pixel specifically fell into the first... Each content block.

[0096] Semantic reference resolution and dynamic solution generation: After identifying the target block, the system extracts the text content associated with that block. The system performs semantic referencing resolution, replacing ambiguous spatial pronouns in the audience's spoken instructions. For example, "Give me a detailed explanation of this" is automatically replaced with "Give me a detailed explanation of the revenue data for each region in 2023." The system then sends this semantically complete instruction as a prompt to the Large Language Model (LLM). The LLM dynamically generates targeted answer text based on the block content in real time, thus avoiding the limitations of pre-set, rigid explanation words.

[0097] The robot uses text-to-speech (TTS) technology to read aloud the dynamically generated solution from the large model, while simultaneously controlling the servo motors to precisely steer the block. The physical spatial location corresponding to the center point, and the natural alternation of eye contact between the content area and the audience asking questions, completes high-precision human-computer interaction.

[0098] In summary, the speech interaction method provided in this application solves simultaneously the speaker's non-contact three-dimensional gaze vector and the absolute plane equation of the large screen, and combines this with a two-dimensional pixel dimensionality reduction mapping algorithm to accurately and uniquely pinpoint the specific content block (e.g., a specific data curve, comparison chart, or core data panel) currently being gazed at by the speaker at both the geometric and display levels. This enables the robot to bind ambiguous semantics to specific screen content and make accurate, personalized content responses, significantly improving the robot's interaction accuracy in high-density information environments. Furthermore, the interactive presentation method provided in this application automatically determines the timing of page turning and automatically turns the page. Specifically, it filters out viewers within the line of sight based on the three-dimensional physical coordinates of the audience's face relative to the absolute origin, obtains the target audience, acquires the preset facial key points of the target audience, and determines the gaze vector of the target audience based on the preset facial key points. Based on the gaze vector and the three-dimensional coordinate range of the screen, it determines the number of viewers whose gaze is off-screen. When the proportion of viewers whose gaze is off-screen to the total number of target viewers reaches a preset threshold, that is, when it is determined that most people have finished reading or lost interest, the page is automatically turned. This can effectively eliminate invalid waiting or interruptions in the presentation process, making the entire presentation and guided tour process more compact and efficient.

[0099] This application also provides a presentation interaction device, which can be referred to in conjunction with the method described above. The device includes: The candidate audience screening unit is used to screen audiences whose three-dimensional physical coordinates of their faces relative to the absolute origin fall within the sound source localization range of the microphone array during the time period when the audience issues a voice command. The object determination unit is used to determine the candidate audience as the vocal object if the rolling standard deviation of the lip aspect ratio sequence of the candidate audience exceeds a preset action threshold. The target object determination unit is used to determine the sound-emitting object as a target object if the line-of-sight vector of the sound-emitting object falls within the three-dimensional coordinate range of the screen. The two-dimensional coordinate determination unit for the gaze point is used to determine the two-dimensional coordinates of the gaze point of the target object on the screen based on the three-dimensional physical intersection of the gaze vector of the target object and the three-dimensional coordinate range of the screen. The target block determination unit is used to determine the interactive content block into which the two-dimensional coordinates of the gaze point fall as the target block; The replacement unit is used to replace the fuzzy spatial pronouns in the voice command of the target object according to the text content associated with the target block, and send the replaced voice command as a prompt word to the large language model so that the large language model can generate the answer text according to the content of the target block.

[0100] In some embodiments, it also includes: The first acquisition unit is used to acquire the three-dimensional physical coordinates of each audience member's face relative to the absolute origin. The filtering unit is used to filter out the audience members within the line of sight determination range based on the three-dimensional physical coordinates, thereby obtaining the target audience. The second acquisition unit is used to acquire preset facial key points of the target audience and determine the gaze vector of the target audience based on the preset facial key points. The determining unit is used to determine the number of viewers whose gazes deviate from the screen based on the gaze vector and the three-dimensional coordinate range of the screen. The statistics unit is used to count the proportion of viewers whose eyes are off the screen out of the total target audience. The sending unit is used to send a page-turning command and trigger corresponding voice explanation if the ratio reaches a preset threshold.

[0101] In some embodiments, it also includes: A broadcasting unit is used to broadcast the answer text via a robot; wherein the robot turns to the physical spatial location corresponding to the center point of the target block and alternates its gaze between the target block and the target object.

[0102] In some embodiments, the second acquisition unit includes: An absolute perspective error determination unit is used to determine the absolute perspective error of the yaw angle and the absolute perspective error of the pitch angle based on the preset facial key points. The correction unit is used to correct the original yaw angle based on the absolute perspective error of the yaw angle, and to correct the original pitch angle based on the absolute perspective error of the pitch angle. A line-of-sight vector determination unit is used to determine the line-of-sight vector based on the corrected yaw angle and pitch angle.

[0103] In some embodiments, the determining unit is configured to: The target number of viewers whose gaze deviates from the screen for more than a preset time threshold is determined based on the gaze vector and the three-dimensional coordinate range of the screen.

[0104] In some embodiments, the sending unit is used for: If the duration for which the ratio reaches the preset threshold reaches the preset duration, a page-turning command is sent and the corresponding voice explanation is triggered.

[0105] In some embodiments, it also includes: The line-of-sight determination range unit is used to determine the line-of-sight determination range during the initialization setting phase.

[0106] This application also provides an electronic device, referenced... Figure 7As shown, the device includes a memory 1 and a processor 2.

[0107] Memory 1 is used to store computer programs; Processor 2 is used to execute computer programs to perform the following steps: When a viewer issues a voice command, viewers whose facial 3D physical coordinates relative to the absolute origin fall within the microphone array's sound source localization range during the time period of the voice command are selected as candidate viewers. If the standard deviation of the lip aspect ratio sequence of the candidate viewer exceeds a preset action threshold, the candidate viewer is determined to be the voice source. If the gaze vector of the voice source falls within the screen's 3D coordinate range, the voice source is determined to be the target object. Based on the 3D physical intersection of the target object's gaze vector and the screen's 3D coordinate range, the 2D coordinates of the target object's gaze landing point on the screen are determined. The interactive content block into which the 2D coordinates of the gaze landing point fall is determined to be the target block. The ambiguous spatial pronouns in the target object's voice command are replaced according to the text content associated with the target block, and the replaced voice command is sent as a prompt word to the large language model, which then generates answer text based on the content of the target block.

[0108] For a description of the equipment provided in this application, please refer to the above method embodiments; further details will not be provided here.

[0109] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps: When a viewer issues a voice command, viewers whose facial 3D physical coordinates relative to the absolute origin fall within the microphone array's sound source localization range during the time period of the voice command are selected as candidate viewers. If the standard deviation of the lip aspect ratio sequence of the candidate viewer exceeds a preset action threshold, the candidate viewer is determined to be the voice source. If the gaze vector of the voice source falls within the screen's 3D coordinate range, the voice source is determined to be the target object. Based on the 3D physical intersection of the target object's gaze vector and the screen's 3D coordinate range, the 2D coordinates of the target object's gaze landing point on the screen are determined. The interactive content block into which the 2D coordinates of the gaze landing point fall is determined to be the target block. The ambiguous spatial pronouns in the target object's voice command are replaced according to the text content associated with the target block, and the replaced voice command is sent as a prompt word to the large language model, which then generates answer text based on the content of the target block.

[0110] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0112] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant details can be found in the method section.

[0113] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0115] The presentation interaction method, apparatus, device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A presentation interaction method, characterized in that, include: When the audience issues a voice command, the audience whose three-dimensional physical coordinates of their face relative to the absolute origin fall within the sound source localization range of the microphone array during the time period when the voice command is issued are selected as candidate audiences. If the standard deviation of the lip aspect ratio sequence of the candidate audience exceeds the preset action threshold, then the candidate audience is determined to be the vocal subject. If the gaze vector of the sound-emitting object falls within the three-dimensional coordinate range of the screen, then the sound-emitting object is determined to be the target object; Based on the three-dimensional physical intersection of the target object's gaze vector and the screen's three-dimensional coordinate range, determine the two-dimensional coordinates of the target object's gaze point on the screen; The interactive content block into which the two-dimensional coordinates of the gaze point fall is determined as the target block. Specifically, during the initialization configuration phase, each interactive content block is delineated, the pixel bounding box of each block at the current two-dimensional screen resolution is recorded, and corresponding text content is bound to each block, constructing a mapping association table between pixel coordinates and text content. Collision detection is performed between the two-dimensional coordinates of the gaze point and the bounding boxes of each block. When the horizontal coordinate of the two-dimensional coordinates of the gaze point falls within the horizontal range of the block's bounding box and the vertical coordinate of the two-dimensional coordinates of the gaze point falls within the vertical range of the block's bounding box, the interactive content block corresponding to the two-dimensional coordinates of the gaze point falling within the block's bounding box is determined as the target block. The fuzzy spatial pronouns in the voice command of the target object are replaced according to the text content associated with the target block, and the replaced voice command is sent as a prompt word to the large language model so that the large language model can generate the answer text according to the content of the target block. The robot reads the answer text aloud; wherein the robot turns to the physical spatial location corresponding to the center point of the target block and alternates its gaze between the target block and the target object.

2. The presentation interaction method according to claim 1, characterized in that, Also includes: Obtain the three-dimensional physical coordinates of each audience member's face relative to the absolute origin; Based on the three-dimensional physical coordinates, viewers within the line-of-sight determination range are selected to obtain the target viewers; Obtain preset facial key points of the target audience, and determine the gaze vector of the target audience based on the preset facial key points; Based on the gaze vector and the three-dimensional coordinate range of the screen, determine the number of viewers whose gaze is deviating from the screen; The percentage of viewers whose eyes were off-screen was counted out of the total target audience. If the ratio reaches a preset threshold, a page-turning command is sent and a corresponding voice explanation is triggered.

3. The presentation interaction method according to claim 2, characterized in that, Determining the gaze vector of the target audience based on the preset facial key points includes: The absolute perspective error of the yaw angle and the absolute perspective error of the pitch angle are determined based on the preset facial key points. The original yaw angle is corrected based on the absolute perspective error of the yaw angle, and the original pitch angle is corrected based on the absolute perspective error of the pitch angle. The line-of-sight vector is determined based on the corrected yaw and pitch angles.

4. The presentation interaction method according to claim 2, characterized in that, Based on the gaze vector and the three-dimensional coordinate range of the screen, the number of viewers whose gazes deviate from the screen includes: The number of viewers whose gaze deviates continuously from the screen for more than a preset time threshold is determined based on the gaze vector and the three-dimensional coordinate range of the screen.

5. The presentation interaction method according to claim 2, characterized in that, If the ratio reaches a preset threshold, a page-turning command is sent and a corresponding voice explanation is triggered, including: If the duration for which the ratio reaches the preset threshold reaches the preset duration, a page-turning command is sent and the corresponding voice explanation is triggered.

6. The presentation interaction method according to claim 2, characterized in that, Also includes: During the initialization setup phase, the line-of-sight determination range is determined.

7. A speech interaction device, characterized in that, include: The candidate audience screening unit is used to screen audiences whose three-dimensional physical coordinates of their faces relative to the absolute origin fall within the sound source localization range of the microphone array during the time period when the audience issues a voice command. The object determination unit is used to determine the candidate audience as the vocal object if the rolling standard deviation of the lip aspect ratio sequence of the candidate audience exceeds a preset action threshold. The target object determination unit is used to determine the sound-emitting object as a target object if the line-of-sight vector of the sound-emitting object falls within the three-dimensional coordinate range of the screen. The two-dimensional coordinate determination unit for the gaze point is used to determine the two-dimensional coordinates of the gaze point of the target object on the screen based on the three-dimensional physical intersection of the gaze vector of the target object and the three-dimensional coordinate range of the screen. The target block determination unit is used to determine the interactive content block into which the two-dimensional coordinates of the gaze point fall as the target block. Specifically, during the initialization configuration phase, each interactive content block is delineated, the pixel bounding box of each block at the current two-dimensional screen resolution is recorded, and corresponding text content is bound to each block, constructing a mapping association table between pixel coordinates and text content. Collision detection is performed between the two-dimensional coordinates of the gaze point and the bounding boxes of each block. When the horizontal coordinate of the two-dimensional coordinates of the gaze point falls within the horizontal range of the block bounding box and the vertical coordinate of the two-dimensional coordinates of the gaze point falls within the vertical range of the block bounding box, the interactive content block corresponding to the two-dimensional coordinates of the gaze point falling within the block bounding box is determined as the target block. The replacement unit is used to replace the fuzzy spatial pronouns in the voice command of the target object according to the text content associated with the target block, and send the replaced voice command as a prompt word to the large language model so that the large language model can generate the answer text according to the content of the target block; A broadcasting unit is used to broadcast the answer text via a robot; wherein the robot turns to the physical spatial location corresponding to the center point of the target block and alternates its gaze between the target block and the target object.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the presentation interaction method as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the presentation interaction method as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Eye gaze for spoken language understanding in multi-modal conversational interactions

    CN107077201A

  • Attention shifting of a robot in a group conversation using audio-visual perception based speaker localization

    US20210097995A1