User gesture-guided interaction method and apparatus, and electronic device

Abstract

The present application relates to the technical field of human-computer interaction. Disclosed are a user gesture-guided interaction method and apparatus, and an electronic device. The method comprises: acquiring a gesture of a user; when it is identified on the basis of the gesture of the user that the user has an intent to perform guided interaction with screen content, displaying, at the position of the screen content pointed to by the gesture on a screen, a cursor corresponding to the user; acquiring a movement state of a hand of the user who has the intent to perform guided interaction, and the type of the screen content indicated by the cursor; on the basis of the movement state of the hand of the user, controlling the representation form of the cursor corresponding to the user on the screen; and on the basis of the type of the screen content, controlling the representation mode of the cursor corresponding to the user on the screen. The present application eliminates the stability and experience risks caused by insufficient physical devices and connection problems; a guidance action of a user can be immediately reflected on a screen, such that instant feedback and response are provided, thereby improving the accuracy of guidance; and the user can directly use a gesture to perform guidance, and thus the operation is convenient and the operation efficiency is high.

Description

User gesture guidance interaction method, device and electronic device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on February 7, 2024, with application number 202410174068.5, and invention name “User gesture guided interaction method, device and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of human-computer interaction technology, and in particular to a user gesture-guided interaction method, device, and electronic device. Background Art

[0004] In daily office work, using smart display devices to initiate multi-person discussions or meetings has become an essential part of corporate life. These meetings and discussions, focusing on specific content, are a high-frequency activity, allowing participants to quickly focus on a specific area of ​​the screen. Currently, pointing to the content displayed on the screen is primarily accomplished through laser pointers, mouse pointers, or finger taps to guide participants' attention.

[0005] However, guidance methods based on physical devices such as laser pens and mice are mainly used by a single user. In terms of quantity, they cannot meet the guidance needs of multiple people. They are also prone to unstable experience risks such as missing laser pens and cumbersome computer connections to smart display devices. The human finger pointing method includes pointing at the screen but not touching the screen. In this case, the direction of human guidance may be different from the actual screen content you want to correspond to. If you point to the content with your finger at close range or directly touch the screen to point to the content, although it can meet the requirements of accurate target content indication, this method requires the user to move to the front of the screen, which is not convenient and has low operation efficiency. Summary of the Invention

[0006] The main technical problem solved by the implementation methods of this application is how to improve the stability and accuracy of user guidance screen content and provide a convenient and efficient user experience.

[0007] In order to solve the above technical problems, a technical solution adopted by the embodiment of the present application is: to provide a user gesture guidance interaction method, including: obtaining the user's gesture; when the user generates a guidance interaction intention for the screen content according to the user's gesture, displaying the corresponding user's cursor for the position of the screen content pointed by the gesture on the screen; obtaining the movement state of the user's hand that generates the guidance interaction intention, and the type of screen content indicated by the cursor; according to the movement state of the user's hand, controlling the expression form of the user's corresponding cursor on the screen; according to the type of the screen content, controlling the expression mode of the user's corresponding cursor on the screen. Among them, by obtaining the user's gesture to achieve screen content guidance, there is no need to rely on laser pointers, mice or other physical devices, thereby eliminating the stability and experience risks caused by insufficient number of physical devices and connection problems; the user can directly point to the target content through gestures, and the cursor corresponding to the target content is displayed on the screen, that is, the user's guidance action can be immediately reflected on the screen, providing instant feedback and response, thereby improving the accuracy of the guidance; the user can use gestures to guide at his seat or position, which provides a more convenient operation method and high operation efficiency.

[0008] Optionally, when there are multiple users generating the guiding interaction intention, displaying the cursors of the corresponding users at the positions on the screen to which the gestures are directed includes: setting a guiding identifier for each of the multiple users according to the order in which the guiding interaction intentions were triggered; and displaying the cursors of the corresponding users, respectively, at the positions on the screen to which the gestures of the multiple users are directed, according to the guiding identifiers. Displaying the cursors differently can improve the usability and user experience of the multi-user interactive environment and promote collaboration and participation among users.

[0009] Optionally, controlling the appearance of the cursor corresponding to the user on the screen according to the motion state of the hand includes: when it is determined according to the motion state of the hand that the user's hand remains motionless for a preset time, setting the cursor corresponding to the user to a stationary state on the screen; when it is determined according to the motion state of the hand that the user's hand is moving, moving the cursor corresponding to the user following the hand's motion trajectory, and presenting a fading ray extending backward with a circle as the center on the screen. Among them, by dynamically adjusting the cursor according to the motion state of the hand, an interactive experience that is more in line with user behavior can be provided, and the user can perceive the changes in the cursor more intuitively, making the operation more natural and smooth. In addition, the fading ray extending backward from the center of the cursor can enhance the user's perception and understanding of the hand's motion trajectory. This visualization effect can help the user better track and predict the movement path of the cursor, thereby improving the accuracy and efficiency of the operation.

[0010] Optionally, the method further includes: when multiple cursors in the stationary state exist at the same position on the screen, or when the movement trajectories of multiple cursors on the screen intersect, controlling the multiple cursors on the screen to be presented in an overlapping state with transparency. Presenting multiple cursors in an overlapping state with transparency can more clearly demonstrate their positional relationship and overlapping state, providing more intuitive visual feedback, thereby helping users understand and handle multi-cursor interaction and reducing potential confusion and misoperation.

[0011] Optionally, controlling the on-screen display mode of the user's corresponding cursor based on the type of screen content includes: identifying the type of content on the screen; switching the corresponding user's cursor display mode based on the identified content type; and providing cursor display mode setting options to enable the user to adjust the cursor display mode. Controlling the user's cursor display mode based on the screen content type can support content comprehension and navigation, and provide personalized choices, thereby helping to improve interaction efficiency, accuracy, and user satisfaction.

[0012] Optionally, the method further includes: when the user's gesture is recognized as indicating the user's intention to mark the screen content, determining a marking start point and a marking end point corresponding to the screen content based on the user's gesture; and highlighting the content between the marking start point and the marking end point on the screen. Determining the marking range based on the user's gesture and highlighting it can provide a more intuitive, accurate, and visual way to guide the user's attention and interaction.

[0013] Optionally, when there are multiple users generating the marking intention, the method further includes: obtaining guidance identifiers of the multiple users; setting a highlighting state corresponding to the guidance identifiers, wherein different guidance identifiers correspond to different highlighting states; obtaining the marking content corresponding to the marking intentions of the multiple users; and highlighting the corresponding marking content according to the highlighting state. Distinguishing and highlighting the marking intentions of multiple users can promote multi-user collaboration and communication, provide visual information display and navigation, enhance the personalized experience, and reduce confusion and conflict.

[0014] Optionally, the method further includes: when the user's intention to delete the mark is recognized based on the user's gesture, determining the marked area to be deleted based on the user's gesture; and recognizing the user's deletion action to delete the highlighted display corresponding to the marked area. Recognizing the user's intention to delete the mark based on the gesture, determining the marked area to be deleted based on the gesture, and executing the deletion action to cancel the highlighted display can provide an efficient, intuitive, and flexible way to manage and clear marks, thereby improving the efficiency and accuracy of user operations.

[0015] To solve the above technical problems, a technical solution adopted by the embodiment of the present application is to provide a user gesture guidance interaction device, comprising: a gesture acquisition module for acquiring a user's gesture; a cursor display module for, when the user generates a guidance interaction intention for the screen content according to the user's gesture, displaying a corresponding user's cursor for the position of the screen content pointed by the gesture on the screen; an information acquisition module for acquiring the motion state of the user's hand that generates the guidance interaction intention and the type of screen content indicated by the cursor; a first interaction control module for controlling the display form of the user's corresponding cursor on the screen according to the motion state of the hand; and a second interaction control module for controlling the display mode of the user's corresponding cursor on the screen according to the type of the screen content. In particular, by acquiring the user's gesture to achieve screen content guidance, there is no need to rely on a laser pointer, mouse or other physical devices, thereby eliminating the stability and experience risks caused by insufficient number of physical devices and connection problems; the user can directly point to the target content through gestures, and the cursor corresponding to the target content is displayed on the screen, that is, the user's guidance action can be immediately reflected on the screen, providing instant feedback and response, thereby improving the accuracy of guidance; the user can use gestures to guide at his or her seat or position, which provides a more convenient operation method and high operation efficiency.

[0016] In order to solve the above technical problems, a technical solution adopted in the embodiment of the present application is: to provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the user gesture guidance interaction method as described above. The electronic device realizes screen content guidance by obtaining the user's gestures, without relying on a laser pen, mouse or other physical devices, thereby eliminating the stability and experience risks caused by insufficient number of physical devices and connection problems; the user can directly point to the target content through gestures, and the cursor corresponding to the target content is displayed on the screen, that is, the user's guidance action can be immediately reflected on the screen, providing instant feedback and response, thereby improving the accuracy of the guidance; the user can use gestures for guidance at his seat or position, which provides a more convenient operation method and high operation efficiency.

[0017] In order to solve the above technical problems, a technical solution adopted in the embodiment of the present application is: providing a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the user gesture guidance interaction method as described above.

[0018] In order to solve the above technical problems, a technical solution adopted in the embodiment of the present application is: to provide a computer program product, wherein the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by an electronic device, the electronic device executes the user gesture guidance interaction method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] One or more embodiments are exemplarily illustrated by corresponding drawings, which do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0020] FIG1 is a flow chart of a user gesture guidance interaction method provided by an embodiment of the present application;

[0021] FIG2 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0022] FIG3 is a schematic diagram of the display forms of the cursor corresponding to the static state and the moving state of the hand provided in an embodiment of the present application;

[0023] FIG4 is a schematic diagram of the display of the overlapping of multiple cursors in a static state and the encounter of the moving trajectories of multiple cursors in a moving state provided by an embodiment of the present application;

[0024] FIG5 is a schematic diagram of an embodiment of the present application showing that the content type is identified as text and the cursor is displayed as a standard fading rectangle highlighting the text;

[0025] FIG6 is a schematic diagram of an embodiment of the present application showing that the content type is identified as a table and the cursor highlights a single table;

[0026] FIG7 is a schematic diagram of an embodiment of the present application showing that the content type is identified as a picture or a video and the cursor is displayed in a magnifying glass mode;

[0027] FIG8 is a schematic diagram of triggering a marking instruction and marking screen content by pinching two fingers in an embodiment of the present application;

[0028] FIG9 is a schematic structural diagram of a user gesture guidance interaction device provided in an embodiment of the present application;

[0029] FIG10 is a schematic diagram of the hardware structure of an electronic device for executing a user gesture guidance interaction method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other and are all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematic and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the device schematic or the flow chart. Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not used to limit the present application.

[0031] When using smart display devices in corporate offices to conduct multi-person discussions or meetings, guiding participants to quickly focus on specific content on the screen becomes a key requirement. However, existing guidance solutions have some problems. Guidance methods using physical devices such as laser pointers and mice are suitable for single users and cannot meet the needs of multi-person participation. In addition, their interaction methods are not natural, and there are often user experience risks such as laser pointer omissions or cumbersome connection with smart display devices. Another traditional guidance method is to use human finger pointing, which has a natural interaction method and does not limit the number of people participating in the guidance. However, there is currently a lack of direct connection between human guidance and the content of smart display devices, resulting in errors between the guidance direction and the target content. Sometimes people even need to leave their seats and walk towards the smart display device, pointing at the content with their fingers at close range to achieve precise guidance. This method is not intuitive, convenient, and inefficient.

[0032] To address the aforementioned issues, embodiments of the present application provide an improved solution based on gesture recognition technology. This solution obtains user gesture information, analyzes and recognizes the user's gesture, and determines whether the user intends to interact with screen content. Upon recognizing the user's intention, a cursor corresponding to the user is displayed based on the location of the screen content indicated by the gesture. The user's hand motion state, such as the speed, direction, and hand position of the gesture, indicating the user's intention, is then acquired. Simultaneously, the type of screen content indicated by the cursor, such as text, image, or multimedia, is also acquired. Based on the hand motion state, the on-screen display of the corresponding user's cursor is controlled. This display may include cursor shape, color, transparency, or other visual effects to provide feedback and enhance the user interaction experience. The on-screen display mode of the corresponding user's cursor is adjusted based on the type of screen content. Different types of content can provide different cursor styles or interaction methods, thereby better guiding the user's attention and actions. Through the above improvements, the user gesture interaction method of embodiments of the present application can more accurately identify the user's intention to interact with the screen and display the corresponding cursor in real time. Furthermore, based on the hand motion state and the type of screen content, a cursor display mode and pattern that better meets the user's expectations is provided. This interactive method can improve the guidance efficiency and experience in multi-person scenarios, allowing participants to interact and collaborate with smart display devices more intuitively.

[0033] The user gesture guidance interaction method is described below through specific embodiments.

[0034] Please refer to Figure 1, which is a flow chart of a user gesture guidance interaction method provided by an embodiment of the present application. The method includes:

[0035] S11. Obtain user gestures.

[0036] Acquiring a user's gesture may include: identifying a human body in the current application scenario; assigning the identified human body to a human body bounding box; further processing each human body bounding box to obtain a face expansion module and a hand expansion module; and further analyzing the hand based on the hand expansion module to obtain user gesture information. The face expansion module is used to detect and identify a face within the human body bounding box to obtain facial position and orientation information, and the hand expansion module is used to identify and expand the hand within the human body bounding box.

[0037] The application scenario can be a multi-person discussion or meeting scenario. In the current scene, multiple participants are included, and a visual recognition algorithm can be used to detect and identify human bodies in the entire scene. This can be achieved by using devices such as cameras or depth sensors to obtain images or depth information in the scene, and applying a multi-person human detection algorithm to detect and identify multiple human bodies. Once multiple human bodies are detected, a human tracking algorithm can be used to track the movement trajectory of each human body and assign a unique identifier or ID to each human body, so that different participants can be distinguished and identified in subsequent processing.

[0038] Assigning a body bounding box to each detected person creates a bounding box for each detected person, defining their location and extent within the image or video. Each body bounding box is further processed, for example, by applying a face detection algorithm to locate and identify the facial region. This algorithm can determine the location of a face by analyzing features within an image or video, thus determining a face expansion module. Based on this determined face expansion module, further facial information processing can be performed, including facial feature extraction, expression analysis, emotion recognition, and face recognition, as well as determining facial orientation.

[0039] For each human body bounding box, a hand detection algorithm can be used to locate and identify the hand area, thereby determining a hand expansion module. The hand detection algorithm can determine the hand's position by analyzing hand features in an image or video. Based on this determined hand expansion module, gesture recognition, finger tracking, and gesture interaction can be performed.

[0040] Based on the hand expansion module, further analysis of the hand to obtain user gesture information may include: using a hand key point detection algorithm, such as a convolutional neural network or regression model, to process the detected hand area and locate the positions of key points of the hand. For example, key points of the hand include key locations such as the finger joints and the palm. Based on the positions of the hand key points, a gesture recognition algorithm is applied to analyze and recognize the gestures. The gesture recognition algorithm can identify different hand gestures by analyzing the spatial relationships and dynamic changes between the hand key points. These gestures may include index finger pointing forward, thumb raised, palm open, fist clenched, and heart gestures. The specific implementation method and algorithm selection for gesture recognition can vary depending on the specific application scenario and requirements. For example, gesture recognition algorithms may include deep learning-based convolutional neural networks, hidden Markov models, Kalman filters, etc. By combining the above steps, user gesture information can be obtained from the hand expansion module and further applied to human-computer interaction.

[0041] The captured user gestures can be for all users in the current application scenario, or for a target user, such as a conference host. User gestures can be captured automatically and in real time online. In some cases, the user can proactively trigger gesture capture. For example, the system can provide a gesture-guided interaction feature, where the user initiates gesture recognition and interaction through specific gestures. This approach increases user engagement and control, and helps the system accurately recognize user gestures.

[0042] Among them, the hardware devices involved in the process of obtaining user gestures may include cameras, sensors, computing devices, etc. The camera is used to capture images or videos of the user and send them to the computing device for related processing. The camera can be a front-facing camera built into the computing device or an externally connected camera. Sensors can also be used to obtain user gestures. For example, sensor gloves, wristbands or wrist sensors used for gesture recognition can measure information such as hand movement, posture and pressure. These sensors can communicate with computing devices via Bluetooth or other wireless connections. Computing devices include but are not limited to personal computers, smartphones and tablets, embedded devices, dedicated computing devices, etc. In this embodiment, the computing device is also connected to a display device, and the obtained user gestures are used to guide the screen content of the display device and generate interaction. The display device can specifically be a large-screen display device.

[0043] S12. When it is recognized based on the user's gesture that the user intends to interact with the screen content, a cursor corresponding to the user is displayed at a position on the screen where the screen content is pointed to by the gesture.

[0044] Guidance interaction intention refers to the user's intention to use gestures to guide, control or operate the screen content. Identifying whether the user has a guidance interaction intention for the screen content based on the user's gestures can include: determining whether the user's face is facing the screen based on the face expansion module obtained above, which can be achieved by detecting facial feature points or applying a deep learning model to perform facial posture estimation; then determining whether there are one or more fingertips pointing to the screen on the hand based on the user's gestures obtained above, which can be achieved through hand key point detection and gesture direction estimation; when both of the above two conditions are met, that is, the face is facing the screen and there are fingers pointing to the screen, then detecting again whether the duration of the two conditions is greater than a preset time, such as more than 2 seconds, it can be determined that the user has a guidance interaction intention.

[0045] Other methods can also be used to determine whether the user has a guiding interaction intention. For example, if it is detected that the user's finger is pointing at the screen and the user is in a speaking state, and this state lasts for more than a preset time, it can be considered that the user has a guiding interaction intention.

[0046] The user with the guiding interaction intention can be one user or multiple users. For example, in the application scenario shown in Figure 2, there are two users with the guiding interaction intention. Their respective hand extension modules are marked with a dotted box and given their own identifiers or IDs (such as 1, 2, 3, etc.).

[0047] After determining that the user has a directed interaction intent, obtaining the position of the screen content on the screen that the user's gesture is pointing to may include: obtaining an image of the user's hand based on the hand expansion module determined above, obtaining hand key point information based on the hand image; calculating a spatial ray pointing to the finger based on the hand key point information; calculating the position of the interaction ray in space based on the spatial ray pointing to the finger and combining it with the intrinsic and extrinsic parameters of the camera; mapping the spatial position of the interaction ray to the screen coordinate system using screen parameters and camera parameters, and calculating the intersection of the interaction ray and the screen; and determining the position of the screen content on the screen that the user's gesture is pointing to based on the intersection of the interaction ray and the screen. The hand key point information may include finger positions, joint positions, etc.; the intrinsic and extrinsic parameters of the camera may include the camera's projection matrix and camera pose; and screen parameters may include screen resolution, screen size, and the relative position between the screen and camera. The specific implementation method may vary depending on the application scenario and specific requirements. In actual applications, appropriate computer vision, gesture recognition, and geometric calculation algorithms can be selected based on the specific situation, and appropriate parameter adjustments and coordinate mapping can be performed to accurately obtain the position of the screen content pointed to by the gesture on the screen.

[0048] After determining the position of the screen content pointed to by the user's gesture on the screen, a cursor is displayed at that position. The cursor can be a visual representation, such as a symbol, icon or shape that appears on the screen to indicate the user's current operating position or focus. On a computer interface, a cursor is generally used to identify a location where a user can interact, such as a text insertion point in a text editor, a mouse pointer in an operating system, etc. The cursor of an embodiment of the present application can imitate the form of a cursor on the computer interface. After determining the position of the screen content pointed to by the user's gesture on the screen, a cursor can be displayed at that position so that the user can accurately locate and perform further interaction. This cursor can be a small icon, symbol or other visual indicator, which can vary depending on the design style of a specific application or operating system.

[0049] In some embodiments, when there are multiple users generating the guidance interaction intention, displaying the corresponding user's cursor at the position on the screen of the screen content pointed to by the gesture includes: setting a guidance identifier for each of the multiple users based on the order in which the guidance interaction intention was triggered; and displaying the corresponding user's cursor at the position on the screen of the screen content pointed to by the gestures of the multiple users based on the guidance identifier. The guidance identifier can be a unique symbol, color, number, or other distinguishable identifier used to distinguish different users. For example, in Figure 2, the numbers "1" and "2" are used as guidance identifiers to distinguish two users with guidance interaction intentions. The guidance identifier can be set based on the order in which the guidance interaction intention was triggered. When a user triggers the guidance interaction intention, the triggering order is recorded, and a unique guidance identifier is assigned to each user based on the triggering order. The guidance identifier can be represented by a number, letter, color, icon, or other unique identifier. The corresponding user's cursor is displayed on the screen and displayed separately based on the guidance identifier. The cursor display order can be determined based on the triggering order, for example, different cursors are displayed in sequence according to the triggering order. When a new user triggers the guidance interaction intention, a new guidance identifier can be assigned, and the corresponding cursor is displayed on the screen.

[0050] In this embodiment, differentiated cursor display can improve the usability and user experience of a multi-user interactive environment, promoting collaboration and participation among users. Each user's cursor can be displayed using a different color, shape, or other visual features to help users quickly identify their own cursor. This personalized display method can effectively reduce user confusion and misoperation, improving user interaction efficiency. By assigning a different cursor display to each user, each user's operation can be clearly displayed, helping users better distinguish each other's input, promoting collaboration, communication, and participation. Users can also accurately observe other users' cursors and operations, thereby better understanding and responding to their behavior. Specific cursor display effects can also provide immediate visual feedback.

[0051] S13. Acquire the motion state of the hand of the user who generates the guiding interaction intention, and the type of screen content indicated by the cursor.

[0052] Hand motion states can be categorized as static and dynamic. In the static state, the user's hand maintains a relatively fixed position and posture relative to the screen or other reference points, with no noticeable hand movements or gestures. This state may indicate that the user is waiting, observing, or preparing for the next step. In the dynamic state, the user's hand performs noticeable movements or gestures, which may include finger movements, gesture formation, hand rotation, etc. The user's hand motions can be used to perform various interactive operations, such as selecting, dragging, clicking, or marking objects on the screen.

[0053] The types of screen content may include different types of elements such as text, tables, pictures or videos. Optionally, the types of screen content may also include other interactive elements such as buttons, menus, sliders, check boxes, drop-down lists, etc.

[0054] When obtaining the hand movement state of the user who generates the guiding interaction intention, computer vision technology or sensor technology can be used to extract hand-related features from images, videos or sensor data, and use appropriate algorithms for analysis and identification. When obtaining the type of screen content indicated by the cursor, a screenshot of the position of the cursor on the screen can be taken, and then the screenshot can be analyzed and processed using computer vision technology to determine the type of content indicated by the cursor. Target detection algorithms or deep learning models can also be used to locate and identify specific elements in the screenshot. For details, please refer to the records of related technologies.

[0055] S14. Control the display form of the cursor corresponding to the user on the screen according to the movement state of the hand.

[0056] The cursor on the screen can be displayed in various forms of visual effects, such as animation, light effects, color changes, etc., to present specific information, status or behavior.

[0057] Among them, controlling the expression form of the cursor corresponding to the user on the screen according to the movement state of the hand includes: when it is determined according to the movement state of the hand that the user's hand remains motionless for a preset time, setting the cursor corresponding to the user to a static state on the screen; when it is determined according to the movement state of the hand that the user's hand is moving, moving the cursor corresponding to the user following the hand movement trajectory, and presenting a fading ray extending backward with a circle as the center on the screen.

[0058] Among them, when the cursor is set to a stationary state on the screen, the cursor can be displayed as a customized icon or symbol. The cursor can also have some special effects, such as halo, flashing or particle effects. In addition to the form of the cursor itself, other visual indicators can be used to enhance the visibility and status of the cursor, for example, drawing a border, shadow or glow effect around the cursor. When it is determined that the user's hand is moving based on the movement state of the user's hand, the cursor will move along the movement trajectory of the hand and present a special effect on the screen, that is, a fading ray extending backward with the circle as the center, specifically a fading ray extending backward with the cursor position as the center. The ray can be a translucent graphic, similar to a radial light or tail, which can provide a visual feedback to help the user perceive the trajectory and speed of the hand movement more clearly. For example, please refer to Figure 3, which shows the expression form of the cursor corresponding to the stationary state and the moving state of the hand.

[0059] In this embodiment, by dynamically adjusting the cursor based on hand motion, an interactive experience more tailored to user behavior can be provided. Users can more intuitively perceive cursor changes, making operations more natural and smooth. Furthermore, the fading rays extending backward from the center of the cursor enhance the user's perception and understanding of hand motion. This visualization effect helps users better track and predict the cursor's movement path, improving operational accuracy and efficiency.

[0060] It should be noted that in addition to the above-mentioned expression of fading rays extending backward with the circle as the center, it can also be displayed in other forms. For example, it can be achieved by drawing continuous lines or dotted lines, showing the path of the hand movement from the starting position to the current cursor position. The color of the cursor can also be changed according to the speed of the hand movement or the position of the hand. For example, when the hand moves faster, the cursor can turn red or other eye-catching colors to remind the user of the intensity or speed of the hand movement. By setting the expression of the cursor on the screen according to the movement state of the hand, the visibility and interactivity can be enhanced, and personalized and differentiated design can be achieved.

[0061] In some embodiments, when multiple cursors in the stationary state are located at the same position on the screen, or when the movement trajectories of multiple cursors on the screen intersect, as shown in FIG4 , which illustrates the appearance of multiple cursors in the stationary state overlapping and the movement trajectories of multiple cursors in the moving state intersecting, the multiple cursors on the screen can be controlled to be presented in an overlapping state with transparency. Presenting multiple cursors in an overlapping state with transparency can more clearly demonstrate their positional relationship and overlapping state, providing more intuitive visual feedback, thereby helping users understand and handle multi-cursor interaction and reducing potential confusion and misoperation.

[0062] S15. Control the display mode of the cursor corresponding to the user on the screen according to the type of the screen content.

[0063] Controlling the display mode of the cursor corresponding to the user on the screen according to the type of the screen content specifically includes: identifying the content type on the screen; switching the cursor display mode of the corresponding user according to the identified content type; and providing setting options for the cursor display mode to enable the user to adjust the cursor display mode.

[0064] The system can identify different types of content on the screen, such as text, images, buttons, links, etc. When the content type is identified, the system can switch the cursor display mode according to the identified content type. Different content types may require different cursor displays to provide a more appropriate and effective user experience. For example, please refer to Figure 5. When the content type is identified as text, the cursor is presented as a standard fading rectangle with highlighted text. Please refer to Figure 6. When the content type is identified as a table, the cursor is highlighted in units of a single table. Please refer to Figure 7. When the content type is identified as a picture or video, the cursor is presented in magnifying glass mode, and the edge of the magnifying glass displays the color of the cursor corresponding to different guide marks.

[0065] Optionally, before controlling the display mode of the cursor corresponding to the user on the screen according to the type of the screen content, the user can trigger to turn on the "smart cursor" mode. In this "smart cursor" mode, the cursor display mode of the corresponding user can be switched according to the recognized content type.

[0066] By controlling cursor behavior based on content type and introducing a user-triggered "smart cursor" mode, you can improve legibility, enhance interactivity, personalize settings, and provide guidance and guidance.

[0067] The system may also provide a cursor display mode setting option, which can be displayed on the screen, allowing users to customize and adjust the cursor display. This may include adjusting the cursor shape, color, animation effect, or other related attributes. Users can also adjust the cursor display mode based on specific gestures based on the displayed setting options.

[0068] The user gesture guidance interaction method provided in the embodiment of the present application realizes screen content guidance by obtaining the user's gestures, without relying on laser pens, mice or other physical devices, thereby eliminating the stability and experience risks brought about by insufficient number of physical devices and connection problems; the user can directly point to the target content through gestures, and the cursor corresponding to the target content is displayed on the screen, that is, the user's guidance action can be immediately reflected on the screen, providing instant feedback and response, thereby improving the accuracy of the guidance; the user can use gestures for guidance at his or her seat or position, which provides a more convenient operation method and high operation efficiency.

[0069] In some embodiments, the screen content can also be marked according to the user's gesture. Specifically, when the user's intention to mark the screen content is recognized according to the user's gesture, the marking starting point and marking end point corresponding to the screen content are determined according to the user's gesture; the content between the marking starting point and the marking end point is highlighted on the screen. For example, please refer to Figure 8, the marking instruction is triggered by pinching with two fingers, the marking starting point is when the user performs the finger pinching action, and the end point is when the user recognizes that the user's fingers are not pinching with two fingers. Among them, highlighting is a way of highlighting, which can attract the user's attention and highlight an object or area through different visual effects. For example, bright colors (such as yellow, red) or colors that contrast greatly with the background color (such as black or white) can be used for highlighting. You can also use flashing, fade-in and fade-out, flashing or moving effects to attract the user's attention and highlight related content.

[0070] The tagging instruction can be triggered based on a captured finger gesture, and processed by a computing device that is in communication with a display device. The computing device then controls the tagging operation on the display device based on the tagging instruction. The tagging instruction can also be sent from the computing device to the display device, which responds to the tagging instruction and tags the content displayed on the screen. The method of this embodiment can be used to implement interactive tagging, annotation, or other forms of content manipulation to provide a richer and more intuitive user experience.

[0071] In some embodiments, when multiple users generate the marking intent, the method further includes: obtaining guidance identifiers for the multiple users; setting a highlighting state corresponding to the guidance identifiers, wherein different guidance identifiers correspond to different highlighting states; obtaining the marking content corresponding to the marking intents of the multiple users; and highlighting the corresponding marking content according to the highlighting state. This embodiment distinguishes and highlights the marking intents of multiple users, which can promote multi-user collaboration and communication, provide visual information display and navigation, enhance the personalized experience, and reduce confusion and conflict.

[0072] In some embodiments, the method further includes: upon recognizing a user's intention to delete a mark based on a gesture, determining a marked area to be deleted based on the user's gesture; and recognizing the user's deletion action to delete the highlighted display corresponding to the marked area. This embodiment recognizes the user's intention to delete a mark based on a gesture, determines the marked area to be deleted based on the gesture, and performs a deletion action to cancel the highlighted display. This provides an efficient, intuitive, and flexible way to manage and clear marks, improving the efficiency and accuracy of user operations.

[0073] Please refer to FIG9 , which is a schematic diagram of the structure of a user gesture guidance interaction device provided in an embodiment of the present application. The user gesture guidance interaction device 20 includes:

[0074] A gesture acquisition module 21 is used to acquire the user's gesture; a cursor display module 22 is used to display the corresponding user's cursor according to the position of the screen content pointed to by the gesture when the user's gesture is recognized to have a guiding interaction intention for the screen content; an information acquisition module 23 is used to acquire the movement state of the user's hand that has generated the guiding interaction intention, and the type of screen content indicated by the cursor; a first interaction control module 24 is used to control the expression form of the cursor corresponding to the user on the screen according to the movement state of the hand; a second interaction control module 25 is used to control the expression mode of the cursor corresponding to the user on the screen according to the type of the screen content.

[0075] It should be noted that the aforementioned user gesture guidance interaction device can execute the user gesture guidance interaction method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the method. For technical details not fully described in the embodiments of the user gesture guidance interaction device, please refer to the user gesture guidance interaction method provided in the embodiments of this application.

[0076] FIG10 is a schematic diagram of the hardware structure of an electronic device 30 for executing a user gesture guidance interaction method according to an embodiment of the present application. As shown in FIG10 , the electronic device 30 includes one or more processors 31 and a memory 32. FIG10 uses one processor 31 as an example. The processor 31 and the memory 32 may be connected via a bus or other means. FIG10 uses a bus connection as an example.

[0077] Memory 32, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the program instructions / modules corresponding to the user gesture guidance interaction method in the embodiments of the present application. Processor 31 executes the non-volatile software programs, instructions, and modules stored in memory 32 to execute various functional applications and data processing of the electronic device, thereby implementing the user gesture guidance interaction method in the above-mentioned method embodiment.

[0078] The memory 32 may include a program storage area and a data storage area. The program storage area may store an operating system and applications required for at least one function; the data storage area may store data generated based on the use of the user gesture-guided interaction device. Furthermore, the memory 32 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 32 may optionally include a memory remotely located relative to the processor 31. Such remote memory may be connected to the user gesture-guided interaction device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0079] The one or more modules are stored in the memory 32 and, when executed by the one or more processors 31 , perform the user gesture guidance interaction method in any of the above method embodiments.

[0080] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0081] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to: personal computers, smart phones and tablet computers, embedded devices, special-purpose computing devices, etc.

[0082] An embodiment of the present application provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by one or more processors, such as a processor 31 in Figure 10, so that the one or more processors can execute the user gesture guidance interaction method in any of the above method embodiments.

[0083] An embodiment of the present application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by the electronic device, the electronic device is enabled to execute the user gesture guidance interaction method in any of the above method embodiments.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0085] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course by hardware. Those skilled in the art can understand that all or part of the processes in the above embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Based on the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the present application as described above. For the sake of simplicity, they are not provided in detail. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A user gesture guidance interaction method, characterized in that: include: Get the user's gesture; When the user's intention to interact with the screen content is recognized based on the user's gesture, a cursor corresponding to the user is displayed at the position of the screen content pointed by the gesture on the screen; Acquiring the motion state of the hand of the user who generates the guiding interaction intention, and the type of screen content indicated by the cursor; Controlling the display of the cursor corresponding to the user on the screen according to the movement state of the hand; According to the type of the screen content, the display mode of the cursor corresponding to the user on the screen is controlled.

2. The method according to claim 1, characterized in that When there are multiple users generating the guiding interaction intention, displaying the cursor of the corresponding user at the position of the screen content pointed by the gesture on the screen includes: Setting a guidance identifier for each of the multiple users according to the order in which the guidance interaction intention is triggered; According to the guide marks, at the positions on the screen at which the gestures of the multiple users point to the screen contents, the cursors of the corresponding users are displayed respectively.

3. The method according to claim 1, characterized in that The step of controlling the display of the cursor corresponding to the user on the screen according to the motion state of the hand includes: When it is determined based on the motion state of the hand that the user's hand remains motionless within a preset time, setting the cursor corresponding to the user on the screen to a stationary state; When the user's hand movement is determined according to the movement state of the hand, the cursor corresponding to the user is moved along the hand movement trajectory, and a fading ray extending backward with a circle as the center is presented on the screen.

4. The method according to claim 3, characterized in that The method further comprises: When there are multiple cursors in the stationary state at the same position on the screen, or when the movement tracks of multiple cursors on the screen meet, the multiple cursors on the screen are controlled to be presented in an overlapping state with transparency.

5. The method according to claim 1, wherein The controlling the display mode of the cursor corresponding to the user on the screen according to the type of the screen content includes: Identify the type of content on the screen; According to the recognized content type, switch the cursor display mode of the corresponding user; A setting option for the cursor display mode is provided to enable the user to adjust the cursor display mode.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: When the user's intention to mark the screen content is recognized according to the user's gesture, a marking start point and a marking end point corresponding to the screen content are determined according to the user's gesture; The content between the marked starting point and the marked end point is highlighted on the screen.

7. The method according to claim 6, characterized in that When there are multiple users generating the marking intention, the method further includes: Get guidance signs for multiple users; Setting the highlight display state corresponding to the guide mark, wherein different guide marks correspond to different highlight display states; Obtaining the marking contents corresponding to the marking intentions of the multiple users; According to the highlighted display state, the corresponding marked content is highlighted.

8. The method according to claim 6, characterized in that The method further comprises: When it is recognized according to the user's gesture that the user intends to delete the mark, determining the mark area to be deleted according to the user's gesture; A deletion action of the user is identified to delete the highlight corresponding to the marked area.

9. A user gesture guidance interaction device, characterized in that: include: Gesture acquisition module, used to obtain user gestures; A cursor display module, configured to display a cursor corresponding to the user at a position on the screen where the gesture points to the screen content when the user's intention to interact with the screen content is recognized based on the gesture; An information acquisition module, configured to acquire the motion state of the hand of the user generating the guiding interaction intention, and the type of screen content indicated by the cursor; A first interaction control module is used to control the appearance of the cursor corresponding to the user on the screen according to the movement state of the hand; The second interaction control module is used to control the display mode of the cursor corresponding to the user on the screen according to the type of the screen content.

10. An electronic device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the user gesture guidance interaction method described in any one of claims 1 to 8.

11. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the user gesture guidance interaction method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by an electronic device, the electronic device executes the user gesture guidance interaction method according to any one of claims 1 to 8.

Images