Interaction method and device executed by electronic equipment, electronic equipment, storage medium, system and computer program product
By displaying virtual objects on smart TVs and using millimeter-wave radar to detect user gestures and object contact, air-touchscreen or mouse operations are achieved, solving the compatibility and user experience issues of smart TV interaction methods and providing an efficient and convenient interaction solution.
Patent Information
- Application Number
- CN202411002667.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-22
AI Technical Summary
Due to viewing distance, smart TVs cannot use touch screen interaction. Existing technologies such as gesture control, air mouse remote control, and touch projection have problems such as high learning cost, poor adaptability, and poor user experience.
By displaying virtual objects on a smart TV, millimeter-wave radar is used to detect the user's hand status and contact with the object surface, recognizing gestures and actions to enable air touchscreen or mouse operation. A built-in human-computer interaction module maps and executes control commands.
It achieves an efficient and convenient interactive experience without the need for auxiliary equipment, adapts to users' visual needs, reduces learning costs, and improves the interactive efficiency and user satisfaction of smart TVs.
Smart Images

Figure CN119002693A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of human-computer interaction technology, and in particular to an interaction method, device, electronic device, storage medium, system and computer program product executed by an electronic device. Background Art
[0002] The biggest difference between smart TVs and mobile devices is the interaction method. Due to the viewing distance, smart TVs cannot use touch screen interaction, but are controlled remotely. As the functions of smart TVs increase, the interactive experience has become a development bottleneck. For example, it is very troublesome to enter a password on a smart TV through a remote control.
[0003] In order to meet market demand, many more natural and convenient human-computer interaction methods have emerged, such as gesture control, flying mouse remote control, touch projection, etc. However, these interaction methods either increase the user's learning cost, or only support limited interaction scenarios, or rely on special physical devices and cannot truly meet the increasingly complex interaction needs. Summary of the invention
[0004] The interactive method, apparatus, electronic device, storage medium, system and computer program product performed by an electronic device provided by the exemplary embodiments of the present disclosure can at least solve the above technical problems and other technical problems not mentioned above.
[0005] According to one aspect of the present disclosure, there is provided an interaction method performed by an electronic device, comprising: in a first operation mode of the electronic device, displaying a virtual object corresponding to a user's hand on a screen of the electronic device; detecting a state of the user's hand, wherein the state of the hand includes a movement of the hand and a contact between the hand and a surface of an object; determining a corresponding control instruction based on the detected state of the hand; and controlling the virtual object according to the control instruction to perform interaction between the user and the electronic device.
[0006] Optionally, determining the corresponding control instruction based on the detected state of the hand includes: determining the part of the hand that contacts the surface of the object based on the detected contact between the hand and the surface of the object; when it is determined that a preset part of the hand contacts the surface of the object, determining the control instruction corresponding to the movement of the hand based on the detected movement of the hand.
[0007] Optionally, when the preset part in contact with the surface of the object includes a first part, the control instruction corresponding to the hand movement based on the detected hand movement includes at least one of the following items: when the index finger is detected clicking on the surface of the object, the corresponding control instruction is determined to be a first click instruction; when the middle finger is detected clicking on the surface of the object, the corresponding control instruction is determined to be a second click instruction; when the index finger is detected sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; when the index finger is detected moving relative to the user on the surface of the object, the corresponding control instruction is determined to be moving the position of the virtual object displayed on the screen.
[0008] Optionally, when the preset part in contact with the surface of the object includes a second part, the control instruction corresponding to the hand movement based on the detected hand movement includes at least one of the following items: when the hand is detected sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; when the finger is detected clicking on the surface of the object, the corresponding control instruction is determined to be a third clicking instruction; when the hand is detected moving relative to the user, the corresponding control instruction is determined to be moving the position of the virtual object displayed on the screen.
[0009] Optionally, the state of the hand also includes the distance between the hand and the electronic device; wherein, determining the corresponding control instruction based on the detected state of the hand includes: based on the detected distance between the hand and the electronic device, determining that the corresponding control instruction is to adjust the size of the virtual object displayed on the screen.
[0010] Optionally, the method of determining the contact position between the hand and the surface of the object based on the detected contact between the hand and the surface of the object includes: detecting the point cloud of the hand and objects within a preset range in a preset coordinate system in real time by a millimeter wave radar; determining the object that has an overlapping portion with the point cloud of the hand as the object currently in contact with the hand; and identifying a set of hand parts in the overlapping portion to determine the contact position between the hand and the surface of the object.
[0011] According to another aspect of the present disclosure, an interaction device executed by an electronic device is also provided, including: an object display module, configured to: display a virtual object corresponding to a user's hand on a screen of the electronic device in a first operation mode of the electronic device; a state detection module, configured to: detect the state of the user's hand, wherein the state of the hand includes the movement of the hand and the contact between the hand and the surface of an object; an instruction determination module, configured to: determine a corresponding control instruction based on the detected state of the hand; and an interaction execution module, configured to: control the virtual object according to the control instruction to execute interaction between the user and the electronic device.
[0012] Optionally, the instruction determination module is configured to: determine the part of the hand that contacts the surface of the object based on the detected contact between the hand and the surface of the object; when it is determined that a preset part of the hand contacts the surface of the object, determine a control instruction corresponding to the movement of the hand based on the detected movement of the hand.
[0013] Optionally, when the preset part in contact with the surface of the object includes a first part, the instruction determination module is configured as at least one of the following items: when the index finger is detected clicking on the surface of the object, the corresponding control instruction is determined to be a first click instruction; when the middle finger is detected clicking on the surface of the object, the corresponding control instruction is determined to be a second click instruction; when the index finger is detected sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; when the index finger is detected moving relative to the user on the surface of the object, the corresponding control instruction is determined to be the position of the virtual object displayed on the mobile screen.
[0014] Optionally, when the preset part in contact with the surface of the object includes a second part, the instruction determination module is configured as at least one of the following items: when a hand is detected sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; when a finger is detected clicking on the surface of the object, the corresponding control instruction is determined to be a third clicking instruction; when a hand is detected moving relative to the user, the corresponding control instruction is determined to be moving the position of the virtual object displayed on the screen.
[0015] Optionally, the state of the hand also includes the distance between the hand and the electronic device; wherein the instruction determination module is configured to: based on the detected distance between the hand and the electronic device, determine that the corresponding control instruction is to adjust the size of the virtual object displayed on the screen.
[0016] Optionally, the instruction determination module is configured to: detect in real time through a millimeter wave radar the point cloud of the hand and objects within a preset range in a preset coordinate system; determine the object with overlapping parts with the point cloud of the hand as the object currently in contact with the hand; and identify the set of hand parts in the overlapping parts to determine the part where the hand contacts the surface of the object.
[0017] According to another aspect of an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute any of the interaction methods described above and performed by the electronic device.
[0018] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute any of the above-described interaction methods performed by an electronic device.
[0019] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute any of the above-described interaction methods performed by an electronic device.
[0020] According to another aspect of an embodiment of the present disclosure, a computer program product is further provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the interaction method performed by an electronic device as described in any one of the above items is implemented.
[0021] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0022] According to the interactive method, device, electronic device, storage medium, system and computer program product performed by the electronic device disclosed in the present invention, users can interact with the electronic device through the air through body parts such as hands without relying on any auxiliary physical devices, and the interactive experience is consistent with the traditional touch screen or mouse interaction method, which can significantly improve the user's interactive experience on electronic devices, especially electronic devices that require air interaction.
[0023] In addition, the control instructions corresponding to the user's gesture will only be recognized when a specific preset part touches the surface of an object, which can prevent accidental touches and improve interaction efficiency.
[0024] In addition, when the wrist or other parts touch the surface of an object, the user's hand operation on the surface of the object can be simulated as a mouse operation to control the electronic device remotely, with high interaction efficiency and convenient operation.
[0025] In addition, when the elbow or other hand touches the surface of an object, the movement of the user's hand and the contact with the surface of the object can be simulated as a touch screen operation to control the electronic device remotely, with high interaction efficiency and convenient operation.
[0026] In addition, the learning cost is low, and users do not need to learn specific gestures. Any user who is accustomed to touch screen or mouse interaction can quickly get started. Moreover, the interaction process is easy, and the user's line of sight does not need to frequently switch between auxiliary devices such as projection areas and screens of electronic devices such as smart TVs. There is also no need to make specific gestures in the air. The user can operate electronic devices such as smart TVs on the surface of the object just like touching the screen or operating a mouse.
[0027] In addition, by detecting the distance between the user and the electronic device and adjusting the display size of the virtual object accordingly, it can adapt to the user's visual needs, improve interaction efficiency, and optimize the interaction experience.
[0028] In addition, millimeter-wave radar is used to detect the dynamic situation of the user and surrounding objects, and whether the point cloud overlaps is used to determine whether the user's body parts are in contact with an object and which parts are in contact, which can achieve the corresponding purpose efficiently, conveniently, quickly and with low consumption.
[0029] In addition, the size requirement for the physical plane of the contacting object is relatively low, and users can complete the interaction with any surface of an object within the recognizable range of the millimeter-wave radar. During the interaction process, users can move their position at will and reselect the physical plane for interaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0031] Figure 1 A schematic diagram showing a method of inputting a password through a remote control on a smart TV in the related art;
[0032] Figure 2 A schematic diagram showing 21 key points used to recognize gestures in the related art;
[0033] Figure 3 A schematic diagram showing a related art method of identifying gestures through key points;
[0034] Figure 4 A schematic diagram showing interaction with a smart TV through gestures in the related art;
[0035] Figure 5 A schematic diagram showing interaction with a smart TV through an air mouse and a remote controller in the related art;
[0036] Figure 6 A schematic diagram showing the interaction between a touch projection and a smart TV in the related art;
[0037] Figure 7 A scene diagram for smart TV interaction in an exemplary embodiment of the present disclosure is shown;
[0038] Figure 8 A flowchart showing an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure;
[0039] Fig. 9 A schematic diagram showing a flow chart of performing gesture recognition in an exemplary embodiment of the present disclosure;
[0040] Fig.10 A flowchart showing a method of adjusting the size of a virtual object in an exemplary embodiment of the present disclosure is shown;
[0041] Fig.11A schematic diagram showing a framework of an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure;
[0042] Fig.12 A schematic flow chart showing an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure;
[0043] Fig.13 A schematic diagram showing a flow chart of implementing an air touch screen operation by applying an interactive method executed by an electronic device in an exemplary embodiment of the present disclosure;
[0044] Fig.14 A schematic diagram showing an example of an electronic device implementing an air touch screen operation using an interactive method in an exemplary embodiment of the present disclosure;
[0045] Fig.15 A schematic diagram showing a flow chart of implementing air mouse operation by applying an interactive method executed by an electronic device in an exemplary embodiment of the present disclosure;
[0046] Fig.16 A schematic diagram showing an example of an air mouse operation implemented by an electronic device using an interactive method in an exemplary embodiment of the present disclosure;
[0047] Fig.17 A schematic diagram showing how to adjust the size of a virtual object in an exemplary embodiment of the present disclosure;
[0048] Fig.18 A schematic diagram showing an application scenario of an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure;
[0049] Fig.19 A block diagram showing an interaction device executed by an electronic device in an exemplary embodiment of the present disclosure;
[0050] Fig. 20 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0051] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.
[0053] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0054] Figure 1 A schematic diagram showing a method of inputting a password through a remote control on a smart TV in the related art.
[0055] Reference Figure 1 Since the remote control is small in size and has fewer buttons, whether you use the remote control to move the selection box in the direction to select in the keyboard displayed on the smart TV to achieve input, or directly input through the keyboard on the remote control, you need to rely on the remote control as an auxiliary physical device, and the process is slow and cumbersome, and cannot achieve efficient and convenient results.
[0056] In addition, gesture recognition is a research and development field in computer science and language technology that involves the recognition and interpretation of human gestures. Most of the gesture recognition technologies in related technologies are based on machine learning algorithms, which can be used as Figure 2 As shown, 21 key points of the hand are marked in the image, that is, the coordinates of each key point in the image can be given.
[0057] Figure 2 A schematic diagram showing 21 key points for identifying gestures in the related art. The positions, numbers, and definitions of the 21 key points of the gesture (for example, "0" represents the wrist (WRIST)) can be referred to in Figure 2 .
[0058] Figure 3 A schematic diagram showing a related art for identifying gestures by key points is shown, referring to Figure 3,Related technologies can infer the current gesture by calculation based on the coordinates of the ,key points of the hand, for example, Figure 3 From top to bottom and from left to right, they can represent "OK", "1", "YEAH", "SPIDERMAN", "FIST", "FIVE", "THREE", "ROCK", etc., and then gesture-based smart TV control commands can be designed to provide users with a more natural and intuitive interactive experience.
[0059] Figure 4 A schematic diagram showing the interaction with a smart TV through gestures in the related art, referring to Figure 4 , the corresponding interactive control of the smart TV can be achieved through the following gestures in sequence: for example, "play / pause" can be achieved by "raising your palm", "increase the volume" can be achieved by "raising your index finger", "lower the volume" can be achieved by "thumbs down", "mute / unmute" can be achieved by "fist", "return" can be achieved by "thumbs left", and "go to the next page" can be achieved by "thumbs up".
[0060] However, the interaction method based on gesture recognition still has the following problems:
[0061] (1) Increased learning cost, users need to learn specific gestures;
[0062] (2) During the interaction process, users need to make specific gestures in mid-air, which is not convenient.
[0063] In related technologies, an air mouse (also known as an air flying mouse) is generally based on a wireless mouse or a physical remote control, with the addition of a gyroscope, UWB (Ultra-Wideband) and other modules, so that it can sense changes in direction and speed. Users only need to move in the air and tap the remote control to move and click the cursor on the smart TV.
[0064] Figure 5 A schematic diagram showing interaction with a smart TV through an air mouse and a remote controller in the related art.
[0065] Reference Figure 5 (a) is a schematic diagram of interacting with a smart TV through an air mouse, and a mouse arrow 502 corresponding to a remote control 501 (air mouse) can be displayed on the screen of the smart TV; Figure 5 (b) is a schematic diagram of interacting with a smart TV via a remote control. The remote control 503 can be pointed at the screen of the smart TV, and a pointing mark 504 corresponding to the direction of the remote control 503 can be displayed on the screen.
[0066] However, this interactive method still has the following problems:
[0067] (1) It cannot fully adapt to the interactive needs of smart TVs because the mouse was originally designed for computers and requires users to watch the screen closely, while the viewing distance of smart TVs is relatively far;
[0068] (2) A physical handheld device is still required, and the interactive experience is not much different from that of a traditional physical remote control.
[0069] Figure 6 A schematic diagram showing interaction with a smart TV through touch projection in the related art.
[0070] Reference Figure 6 , touch projection usually uses infrared or laser technology to realize the touch function, and generally includes a projection module 601 and a sensor module 602. The projection module can project the content of the smart TV onto a physical plane, for example, plane 603, and the sensor module 602 can sense the gestures and operations of the user in the projection area 604, and convert them into signals that can be recognized by the smart TV, thereby realizing touch screen-like interaction to improve the efficiency and interactivity of the interaction.
[0071] However, this interactive method still has the following problems:
[0072] (1) The user's line of sight needs to frequently switch between the projection area and the smart TV screen, which makes the interaction process cumbersome and the user experience poor;
[0073] (2) To ensure the effect, the projection area cannot be too small, otherwise the projected content cannot be seen clearly, which will limit the usage scenarios.
[0074] Research has found that most users are more accustomed to the mouse interaction method on personal computers or the touch screen interaction method on mobile devices. If these two interaction methods can be applied to the interaction process of electronic devices such as smart TVs, the user's interaction experience will be greatly improved. At the same time, the viewing distance of smart TVs is farther than that of personal computers and mobile devices, and they can usually only be operated remotely. Therefore, the new interaction method disclosed in the present invention can be used to achieve remote interaction with smart TVs by touch screen or mouse.
[0075] In order to solve the above problems, the present disclosure provides an interaction method, device, electronic device, storage medium, system and computer program product performed by an electronic device, which can enable users to interact with electronic devices through the air through body parts such as hands without relying on any auxiliary physical devices, and the interaction experience is consistent with the traditional touch screen or mouse interaction method, which can significantly improve the user's interaction experience on electronic devices, especially electronic devices that require air interaction.
[0076] Next, we will refer to Figures 7 to 19 The interactive method, apparatus, electronic device, storage medium, system and computer program product performed by an electronic device of the present disclosure are specifically described.
[0077] The interaction method performed by an electronic device described in the exemplary embodiments of the present disclosure can be applied to a variety of scenarios that require interaction with a screen (for example, a smart TV, a car central control screen, etc.). The following mainly describes the scenario of application to smart TV interaction as an example.
[0078] Figure 7 A scene diagram applied to smart TV interaction in an exemplary embodiment of the present disclosure is shown.
[0079] Reference Figure 7 A smart TV that uses the interactive method according to an exemplary embodiment of the present disclosure can detect the state of the user's hand to obtain control instructions from the user. The user can directly control the smart TV through hand movements (such as various gestures) and the contact between the hand and the surface of an object (such as a desktop, etc.).
[0080] Figure 8 A flow chart showing an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure.
[0081] Reference Figure 8 , in step 801, in a first operation mode of the electronic device, a virtual object corresponding to a user's hand is displayed on a screen of the electronic device.
[0082] According to an exemplary embodiment of the present disclosure, the execution subject of each step of the present disclosure may be an electronic device with a display function, such as a smart TV, which can interact with a user. The user may issue an instruction to turn on the unassisted device operation mode to enter the first operation mode. For example, the user may enter the first operation mode by operating the remote control of the smart TV, or may enter the first operation mode through voice control, and the present disclosure may not limit this. After entering the first operation mode, the user may be prompted that the first operation mode has been entered by displaying a virtual object on the screen. The virtual object can be used to indicate the screen position corresponding to the position of the user's hand and the state of the user's hand. The initial position of the display virtual object may be a preset position, such as the center of the screen, etc. The display form includes but is not limited to a mouse cursor, a virtual palm, a highlight selection box, and other virtual icons for prompting the visual focus, or other possible forms.
[0083] In step 802, the state of the user's hand may be detected, wherein the state of the hand may include the movement of the hand and the contact between the hand and the surface of the object. The movement of the hand may include the moving speed, direction, gesture, etc. of the hand. The object that the user's hand contacts may be any object that is currently in surface contact with the user's hand, such as a desktop, a dedicated touch pad, a touch ball, etc. The contact between the hand and the surface of the object may include whether the hand contacts the object, the size of the contact area, the contact position, etc.
[0084] According to the exemplary embodiments of the present disclosure, human body sensing and gesture recognition technology may be based on millimeter wave radar, machine vision, wearable devices, etc., and more advanced technologies may appear in the future. The present disclosure may not limit which human body sensing and gesture recognition technology is used to detect the state of the user's hand.
[0085] According to an exemplary embodiment of the present disclosure, a millimeter-wave radar module can be built into a smart TV to detect the state of the user's hand. Millimeter-wave radar is a radar that works in the millimeter-wave band and can measure the relative distance, speed and orientation of the object being measured. It can be widely used in human body detection, gesture recognition and other fields. For example, a millimeter-wave radar with an operating frequency of 76-81GHz (corresponding to a wavelength of about 4mm) can detect movements of a few tenths of a millimeter. Millimeter-wave radar can sense the surrounding environment by emitting millimeter waves and receiving their reflected signals. These reflected signals can be converted into a set of discrete point clouds, each of which can represent the position and attributes of a reflection source, and can include the three-dimensional coordinates, moving speed, and moving direction of the object.
[0086] In step 803, a corresponding control instruction may be determined based on the detected state of the hand.
[0087] According to an exemplary embodiment of the present disclosure, a control instruction for a smart TV may be determined according to the detected motion of a hand and the contact between the hand and the surface of an object.
[0088] According to an exemplary embodiment of the present disclosure, determining a corresponding control instruction based on the detected state of a hand may include but is not limited to the following steps: the part of the hand in contact with the surface of an object may be determined based on the detected contact between the hand and the surface of the object; when it is determined that a preset part of the hand is in contact with the surface of the object, a control instruction corresponding to the movement of the hand may be determined based on the detected movement of the hand.
[0089] According to an exemplary embodiment of the present disclosure, the preset part of the hand may be, for example, a wrist, an elbow, etc. When it is determined that the preset part of the hand is in contact with the surface of an object, a corresponding control instruction may be determined according to the movement of the hand to prevent accidental touches and improve control efficiency.
[0090] According to an exemplary embodiment of the present disclosure, the contact part between the hand and the surface of the object is determined based on the detected contact between the hand and the surface of the object, which may include but is not limited to the following steps: the point cloud of the hand and the object within a preset range in a preset coordinate system may be detected in real time by a millimeter wave radar; the object with overlapping parts with the point cloud of the hand may be determined as the object currently in contact with the hand; and the set of hand parts in the overlapping parts may be identified to determine the contact part between the hand and the surface of the object.
[0091] According to an exemplary embodiment of the present disclosure, when detecting the motion of a user's hand and the contact between the hand and the surface of an object in real time, the following steps may be adopted:
[0092] (1) The millimeter wave radar module can be used to detect the point cloud of the user's hand in a preset coordinate system in real time;
[0093] (2) It is possible to determine whether the point cloud of a certain object overlaps with the point cloud of the user's hand based on the point cloud. The specific method is to traverse the three-dimensional coordinates of each point in the point cloud and determine whether the three-dimensional coordinates of the point cloud of another object are consistent with the three-dimensional coordinates or the distance is less than a certain threshold. If so, step (3) can be executed, otherwise return to step (1);
[0094] (3) The user gesture and the set of user hand parts corresponding to the above-mentioned overlapping parts can be identified by a gesture recognition algorithm based on millimeter wave radar. The user gesture here may include but is not limited to clicking, sliding up, down, left, and right, etc., and the set of user hand parts here may include but is not limited to the index finger, middle finger, wrist, elbow, etc.;
[0095] (4) It can be determined whether the user gesture and the set of user hand parts meet the preset conditions. If so, step (5) can be executed, otherwise, it returns to step (1). The preset conditions here can be used to prevent user misoperation. For example, a possible preset condition is that the set of user hand parts needs to include the wrist, that is, the user's operation action will be recognized only when the user's wrist touches the object;
[0096] (5) The above-mentioned gestures and the set of parts of the user's hand can be mapped to corresponding operation actions according to a preset mapping relationship. The operation actions here may include but are not limited to index finger clicking, middle finger clicking, up, down, left, and right sliding, etc. The mapping relationship here may refer to the mapping relationship between the above-mentioned gestures and the set of parts of the user's hand and the operation actions. For example, if the gesture is clicking, and the set of parts of the user's hand includes the index finger, then the corresponding operation action may be index finger clicking.
[0097] Fig. 9 A schematic diagram showing a flow chart of gesture recognition in an exemplary embodiment of the present disclosure.
[0098] Reference Fig. 9According to an exemplary embodiment of the present disclosure, a method for recognizing a user gesture by using a millimeter wave radar may be as follows:
[0099] [1] Use two-dimensional fast Fourier transform to analyze the distance and Doppler of the scene:
[0100] A two-dimensional fast Fourier transform (2D-FFT) operation can be performed on the analog to digital converter (ADC) data of each receiving channel to obtain a two-dimensional fast Fourier transform matrix (2D-FFT matrix) including a range dimension (Range) and a Doppler dimension (Doppler);
[0101] [2] Non-coherent accumulation of 2D-FFTs across multiple channels to create a range-Doppler heatmap:
[0102] The two-dimensional fast Fourier transform matrix of each receiving channel can be non-coherently accumulated to obtain a range-Doppler heat map;
[0103] [3] Multiple features can be extracted from the range-Doppler heat map, and each feature generates a time series; specifically, the time series can be obtained according to the frame index;
[0104] [4] The extracted features are then used for gesture recognition:
[0105] The features extracted within a specific time window can be classified by a machine learning algorithm to output a corresponding gesture category. For example, the features extracted within a specific time window can be sent to a machine learning algorithm such as an artificial neural network (ANN).
[0106] According to an exemplary embodiment of the present disclosure, the state of the hand may also include the distance between the hand and the electronic device; wherein, determining the corresponding control instruction based on the detected state of the hand may include but is not limited to the following steps: based on the detected distance between the hand and the electronic device, determining the corresponding control instruction to adjust the size of the virtual object displayed on the screen.
[0107] According to an exemplary embodiment of the present disclosure, the millimeter wave radar module can detect the three-dimensional coordinates of the user's hand and the smart TV in a preset coordinate system with the user's location as the origin in real time, and the distance between the user's hand and the smart TV can be calculated based on the three-dimensional coordinates. For example, if the coordinates of the smart TV are (x 0 ,y 0,z 0 ), the user's hand coordinates are (x 1 ,y 1 ,z 1 ), then the distance can be The distance between the user's hand and the smart TV can represent the distance between the user and the smart TV. When the distance is large, the virtual objects displayed on the screen can be enlarged to lower the visual threshold. When the distance is small, the virtual objects displayed on the screen can be reduced to avoid blocking the picture.
[0108] According to an exemplary embodiment of the present disclosure, the millimeter wave radar module can detect the three-dimensional coordinates of the user's hand in a preset coordinate system with the user's position as the origin in real time, and calculate the moving direction and moving distance of the user's hand according to the change of the three-dimensional coordinates. 1 ,y 1 ,z 1 ) moves to coordinate (x 2 ,y 2 ,z 2 ), then the moving vector can be expressed as (x 2 -x 1 ,y 2 -y 1 ,z 2 -z 1 )=(a,b,c), the moving distance can be expressed as The moving direction can be expressed as
[0109] According to an exemplary embodiment of the present disclosure, it is possible to determine whether a user's hand is in contact with a surface of an object and whether the contact position satisfies a preset condition, and to perform different interactive operations according to different determination results.
[0110] According to an exemplary embodiment of the present disclosure, when the preset part in contact with the surface of an object includes a first part, a control instruction corresponding to the movement of the hand is determined based on the detected movement of the hand, which may include but is not limited to one of the following items: when the index finger is detected clicking on the surface of the object, the corresponding control instruction can be determined to be a first click instruction; when the middle finger is detected clicking on the surface of the object, the corresponding control instruction can be determined to be a second click instruction; when the index finger is detected sliding on the surface of the object, the corresponding control instruction can be determined to slide the current page; when the index finger is detected moving relative to the user on the surface of the object, the corresponding control instruction can be determined to move the position of the virtual object displayed on the screen.
[0111] According to an exemplary embodiment of the present disclosure, the first part can be the wrist or other parts. Taking the first part as the wrist as an example, when the preset part in contact with the surface of the object includes the wrist, the movement of the user's hand in contact with the surface of the object can be simulated as operating the mouse on the smart TV through the air; the first click instruction can simulate the left mouse button being clicked, which is used to implement basic operations such as clicking, selecting, and dragging; the second click instruction can simulate the right mouse button being clicked, which is used to open menus, view properties, shortcuts, and other operations; the operation of the mouse wheel can be simulated by sliding the index finger up and down on the surface of the object, and the corresponding control instruction can be determined as sliding the current page.
[0112] According to an exemplary embodiment of the present disclosure, when the preset part in contact with the surface of an object includes a second part, a control instruction corresponding to the movement of the hand is determined based on the detected movement of the hand, which may include but is not limited to one of the following items: when the hand is detected sliding on the surface of the object, the corresponding control instruction can be determined to be sliding the current page; when the finger is detected clicking on the surface of the object, the corresponding control instruction can be determined to be a third clicking instruction; when the hand is detected moving relative to the user, the corresponding control instruction can be determined to be moving the position of a virtual object displayed on the screen.
[0113] According to an exemplary embodiment of the present disclosure, the second part can be an elbow or other parts. Taking the second part as an elbow as an example, when the preset part in contact with the surface of the object includes the elbow, the movement of the user's hand in contact with the surface of the object can be simulated as a touch-screen operation on a smart TV through the air; the third click instruction can simulate the user's finger clicking on an element on the screen by touching the screen; at this time, the user can control the position of the virtual object displayed on the screen by moving his hand in mid-air.
[0114] It should be understood that the above mapping relationship between the user's hand movements and control instructions is only an example, and the movements, control instructions and their mapping relationships according to the embodiments of the present disclosure are not limited to this. In addition, new movements and control instructions can be adjusted or added according to user needs. In addition, the part where the user's hand contacts the surface of the object is not limited to the wrist or elbow, but can also be other parts of the hand such as fingers.
[0115] Return to reference Figure 8 In step 804, the virtual object may be controlled according to the control instruction to perform interaction between the user and the electronic device.
[0116] According to an exemplary embodiment of the present disclosure, a virtual object can be controlled according to a control instruction to perform interactions between a user and an electronic device, such as element clicking, element selection, page sliding, page zooming, etc. Here, an element may refer to an icon, a control, etc. displayed on an electronic device, and a page may refer to a displayed home screen page, an application page, etc. of the electronic device. It should be understood that the type of interaction is not limited thereto.
[0117] According to an exemplary embodiment of the present disclosure, a human-computer interaction module can be built into a smart TV to execute control instructions to interact with a user. The human-computer interaction module can dynamically execute corresponding control instructions for the smart TV according to a preset mapping relationship based on the operation of the user's hand on the surface of an object (the object currently in contact with the user's hand), where the mapping relationship refers to the mapping relationship between the operation of the user's hand on the surface of an object and the control instruction for the smart TV.
[0118] According to an exemplary embodiment of the present disclosure, the size of the virtual object can be dynamically adjusted according to the real-time detection result, the virtual object can be moved, the corresponding virtual object animation effect can be displayed, and the corresponding control instructions for the smart TV can be executed.
[0119] For example, when the user clicks the surface of an object with the index finger, the smart TV can execute a control instruction to click the current element; when the user slides left and right on the surface of an object, the smart TV can execute a control instruction to slide the current page left and right.
[0120] For example, the position of the virtual object on the smart TV screen can be dynamically updated according to the preset mapping relationship based on the moving direction and moving distance of the user's hand relative to the user through the human-computer interaction module. The mapping relationship here can refer to the mapping relationship between the moving direction and moving distance of the user's hand relative to the user and the moving direction and moving distance of the virtual object on the smart TV screen. For example, if the user's hand moves left and right relative to the user, the virtual object can move left and right relative to the screen. If the user's hand moves forward and backward relative to the user, the virtual object can move up and down relative to the screen. A possible mapping relationship of the moving distance is that if the moving distance of the user's hand in the corresponding moving direction is x, and the preset sensitivity coefficient is a, the moving distance of the virtual object relative to the screen can be x*a.
[0121] According to an exemplary embodiment of the present disclosure, it is possible to map the movement of a user's hand to the corresponding position movement of a virtual object displayed on an electronic device.
[0122] According to an exemplary embodiment of the present disclosure, the human-computer interaction module can dynamically display the animation effect of the corresponding virtual object according to the preset mapping relationship based on the operation action of the user's hand on the surface of a certain object. The mapping relationship here can refer to the mapping relationship between the operation action of the user's hand on the surface of a certain object and the preset animation effect of the virtual object; for example, when the user's hand performs an index finger click operation on the surface of the object, the virtual object on the screen can display the animation effect of the index finger clicking; when the user's hand performs a left-right sliding operation on the surface of the object, the virtual object on the screen can display the left-right sliding animation effect.
[0123] According to an exemplary embodiment of the present disclosure, the operation motion of the user's hand on the surface of an object may be mapped to a corresponding animation effect of a virtual object and a corresponding interactive operation on an electronic device.
[0124] Fig.10 A flow chart of adjusting the size of a virtual object in an exemplary embodiment of the present disclosure is shown.
[0125] Reference Fig.10 According to an exemplary embodiment of the present disclosure, a human hand can be detected in real time, and the size of a virtual object can be adjusted according to the real-time detection result.
[0126] For example, the human-computer interaction module can dynamically adjust the size of the virtual object according to the preset mapping relationship based on the distance between the user's hand and the smart TV, so that the user can still get a good interactive experience when he is far away or close to the smart TV. The mapping relationship here can be a mapping relationship between the distance value and the size of the virtual object, or a mapping relationship between the distance range and the size of the virtual object. For example, assuming that the preset standard distance is X 0 , preset the initial size of the virtual object to Y 0 , the current distance is X, the current virtual object size is Y, then: Y = Y 0 *(X / X 0 ).
[0127] According to an exemplary embodiment of the present disclosure, the size of a virtual object on a smart TV may be dynamically adjusted according to the distance between a user's hand and an electronic device.
[0128] Fig.11 A schematic framework diagram showing an interaction method performed by an electronic device in an exemplary embodiment of the present disclosure is shown.
[0129] Fig.12 A flowchart of an interaction method executed by an electronic device in an exemplary embodiment of the present disclosure is shown.
[0130] Reference Fig.11 According to an exemplary embodiment of the present disclosure, a millimeter wave radar module and a human-computer interaction module may be built into a smart TV, referring to Fig.12After turning on the first operation mode, that is, the operation mode without auxiliary equipment, the distance between the user's hand and the smart TV, the movement relative to the user, and the operation action on the surface of a certain object can be detected in real time (through the millimeter wave radar module); the size of the virtual object can be dynamically adjusted according to the real-time detection results (through the human-computer interaction module), the virtual object can be moved, the corresponding virtual object animation effect can be displayed, and the corresponding control instructions for the smart TV can be executed.
[0131] Fig.13 A schematic diagram showing a flow chart of an exemplary embodiment of the present disclosure for implementing an air touch screen operation using an interactive method executed by an electronic device.
[0132] Reference Fig.13 According to an exemplary embodiment of the present disclosure, a user can perform touch-screen operations on a smart TV screen through the air with his hands, so that an interactive experience consistent with touch-screen interaction can be obtained without touching the screen. The specific steps may be as follows: a millimeter-wave radar module and a human-computer interaction module may be built into the smart TV; after the user turns on the smart TV and turns on the touch-screen interaction mode (a type of interaction mode without auxiliary devices), a virtual hand may be displayed on the smart TV screen, and the initial position may be located in the exact center of the smart TV screen; the millimeter-wave radar module may detect in real time the movement of the user's hand relative to the user and the operation on the surface of an object; when the user's elbow contacts an object, the human-computer interaction module may execute corresponding control instructions for the smart TV based on the real-time detection results, for example: when the user's hand slides left and right on the surface of the current object, the smart TV may execute a control instruction to slide the current page left and right; when the user's hand slides up and down on the surface of the current object, the smart TV may execute a control instruction to slide the current page up and down; Fig.14 A schematic diagram showing an example embodiment of the present disclosure in which an interactive method performed by an electronic device is used to implement an air touch screen operation, with reference to Fig.14 In (a), when the user's finger clicks on the surface of the current object, the smart TV can execute the control command of clicking the current element. When the user's elbow touches an object, the human-computer interaction module can move the virtual hand according to the real-time detection results, for example: Fig.14 In (b), when the user's hand moves left and right relative to the user, the virtual hand can move left and right relative to the screen; when the user's hand moves forward and backward relative to the user, the virtual hand can move up and down relative to the screen, that is, in touch screen mode, the user's hand can move in the air to control the movement of the virtual hand on the screen.
[0133] Fig.15 A schematic diagram of a flow chart showing an example embodiment of the present disclosure in which an interactive method executed by an electronic device is used to implement air mouse operation.
[0134] Reference Fig.15According to an exemplary embodiment of the present disclosure, a user can perform mouse operations on a smart TV screen through the air with his hands, so that an interactive experience consistent with mouse interaction can be obtained without relying on a physical mouse. The specific steps are as follows: a millimeter-wave radar module and a human-computer interaction module can be built into the smart TV; after the user turns on the smart TV and turns on the mouse interaction mode (a type of the first operating mode without auxiliary device interaction), the mouse cursor can be displayed on the smart TV screen, and the initial position can be located in the center of the smart TV screen. The millimeter-wave radar module can detect in real time the movement of the user's hand relative to the user and the operating actions on the surface of an object; when the user's wrist contacts an object, the human-computer interaction module can execute corresponding control instructions for the smart TV according to the real-time detection results, for example: Fig.16 A schematic diagram showing an example of an embodiment of the present disclosure in which an interaction method performed by an electronic device is used to implement an air mouse operation, with reference to Fig.16 In (a), when the user's index finger clicks the surface of the current object, the smart TV can execute the control command of clicking the left mouse button; when the user's middle finger clicks the surface of the current object, the smart TV can execute the control command of clicking the right mouse button; when the user's index finger slides up and down on the surface of the current object, the smart TV can execute the mouse wheel rolling operation. When the user's wrist touches an object, the human-computer interaction module can move the mouse cursor according to the real-time detection results, for example: refer to Fig.16 In (b), when the user's index finger moves left and right relative to the user on the current object surface, the mouse cursor can move left and right relative to the screen; when the user's index finger moves forward and backward relative to the user on the current object surface, the mouse cursor can move up and down relative to the screen.
[0135] According to an exemplary embodiment of the present disclosure, the size of a virtual object can be dynamically adjusted according to the distance between a user's hand and a smart TV. The specific steps may be as follows: a millimeter wave radar module and a human-computer interaction module may be built into the smart TV; after the user turns on the smart TV and starts the first operation mode without auxiliary devices, a virtual object may be displayed on the smart TV screen, and the initial size may be a default size; the distance between the user's hand and the smart TV may be detected in real time by the millimeter wave radar module; the size of the virtual object may be dynamically adjusted according to the real-time detection result by the human-computer interaction module, for example, Fig.17 A schematic diagram showing how to adjust the size of a virtual object in an exemplary embodiment of the present disclosure, with reference to Fig.17 In (a), when the user is far away from the smart TV, a larger virtual object can be displayed; refer to Fig.17 In (b), when the user is closer to the smart TV, a smaller virtual object can be displayed.
[0136] According to an exemplary embodiment of the present disclosure, a new interaction method performed by an electronic device can be proposed based on millimeter wave radar technology to achieve air interaction with the electronic device in the manner of a touch screen or a mouse.
[0137] Fig.18 A schematic diagram showing an application scenario of an interaction method executed by an electronic device in an exemplary embodiment of the present disclosure is shown.
[0138] According to the exemplary embodiments of the present disclosure, as a new air-to-air interaction method, the present disclosure can be applied to any scenario requiring air-to-air interaction, and its implementation steps can be basically the same. Fig.18 In (a), the present disclosure can be applied to inputting passwords in a touch screen manner on a smart TV, and can also be applied to zooming in and out of photos in an electronic device by touch screen manner, referring to Fig.18 (b) in the figure can also be used to remotely operate the in-vehicle central control screen on the car steering wheel.
[0139] In addition, the application scenarios of the interaction method performed by the electronic device in the exemplary embodiment of the present disclosure may also include but are not limited to digital signage, projectors, vehicle-mounted central control screens, VR (Virtual Reality) / AR (Augmented Reality) / MR (Mixed Reality) glasses, and any other scenarios that require remote interaction.
[0140] The interaction method performed by an electronic device proposed in the exemplary embodiments of the present disclosure can enable users to interact with electronic devices through the air without relying on any auxiliary physical devices, and the interaction experience is consistent with the traditional touch screen or mouse interaction method, which can significantly improve the user's interaction experience on electronic devices, especially electronic devices that require air interaction.
[0141] Compared with the interaction method based on gesture recognition in the related art, the exemplary embodiments of the present disclosure have the following advantages: low learning cost, users do not need to learn specific gestures, and any user who is accustomed to touch screen or mouse interaction can quickly get started; the interaction process is easier, users do not need to make specific gestures in the air, they only need to operate on the surface of the object like touching the screen or operating the mouse.
[0142] Compared with the air mouse-based interaction method in the related art, the exemplary embodiment of the present disclosure has the following advantages: it can fully adapt to the interaction requirements of smart TVs, and users can more easily identify the mouse cursor on the smart TV screen; it has less reliance on physical devices and does not require any physical devices to be held.
[0143] Compared with the touch-projection-based interaction method in the related art, the exemplary embodiments of the present disclosure have the following advantages: the interaction process is easier, and the user's line of sight does not need to frequently switch between the projection area and the electronic device screen; the requirement for the size of the physical plane is low, and the user can complete the interaction on any surface of an object within the identifiable range of the millimeter-wave radar, and during the interaction process, the user can move the position at will and reselect the physical plane for interaction.
[0144] Fig.19 A block diagram showing an interaction device executed by an electronic device in an exemplary embodiment of the present disclosure.
[0145] Reference Fig.19 An exemplary embodiment of the present disclosure further provides an interaction device 1900 executed by an electronic device, which may include but is not limited to an object display module 1901, a state detection module 1902, an instruction determination module 1903, and an interaction execution module 1904.
[0146] The object display module 1901 can display a virtual object corresponding to the user's hand on the screen of the electronic device in the first operation mode of the electronic device.
[0147] The state detection module 1902 can detect the state of the user's hand, wherein the state of the hand includes the movement of the hand and the contact between the hand and the surface of the object.
[0148] The instruction determination module 1903 can determine the corresponding control instruction based on the detected state of the hand.
[0149] The interaction execution module 1904 can control the virtual object according to the control instruction to execute the interaction between the user and the electronic device.
[0150] According to an exemplary embodiment of the present disclosure, the instruction determination module 1903 can determine the part of the hand that contacts the surface of an object based on the detected contact between the hand and the surface of the object; when it is determined that a preset part of the hand contacts the surface of the object, a control instruction corresponding to the movement of the hand is determined based on the detected movement of the hand.
[0151] According to an exemplary embodiment of the present disclosure, when the preset part in contact with the surface of an object includes a first part, the instruction determination module 1903 can execute at least one of the following items including but not limited to: when the index finger is detected clicking on the surface of the object, the corresponding control instruction is determined to be a first click instruction; when the middle finger is detected clicking on the surface of the object, the corresponding control instruction is determined to be a second click instruction; when the index finger is detected sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; when the index finger is detected moving relative to the user on the surface of the object, the corresponding control instruction is determined to be moving the position of the virtual object displayed on the screen.
[0152] According to an exemplary embodiment of the present disclosure, when the preset part in contact with the surface of the object includes a second part, the instruction determination module 1903 can execute at least one of the following items including but not limited to: when a hand is detected sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; when a finger is detected clicking on the surface of the object, the corresponding control instruction is determined to be a third clicking instruction; when a hand is detected moving relative to the user, the corresponding control instruction is determined to be moving the position of a virtual object displayed on the screen.
[0153] According to an exemplary embodiment of the present disclosure, the state of the hand may also include the distance between the hand and the electronic device; wherein the instruction determination module 1903 may determine the corresponding control instruction to adjust the size of the virtual object displayed on the screen based on the detected distance between the hand and the electronic device.
[0154] According to an exemplary embodiment of the present disclosure, the instruction determination module 1903 can detect the point cloud of the hand and objects within a preset range in a preset coordinate system in real time through a millimeter wave radar; determine the object with overlapping parts with the point cloud of the hand as the object currently in contact with the hand; and identify the set of hand parts in the overlapping parts to determine the part where the hand contacts the surface of the object.
[0155] It is understandable that in the exemplary embodiment of the interactive device 1900 executed by the electronic device, the specific implementation process is roughly the same as the exemplary embodiment of the interactive method executed by the electronic device, and will not be repeated here. The interactive device 1900 executed by the electronic device can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these devices may also be uniformly executed by components in physical entity devices (for example, processors, clients, or servers, etc.).
[0156] Fig. 20 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.
[0157] Reference Fig. 20 The electronic device 2000 includes at least one memory 2001 and at least one processor 2002, wherein the at least one memory 2001 stores a set of computer executable instructions. When the computer executable instruction set is executed by the at least one processor 2002, an interaction method performed by the electronic device according to an exemplary embodiment of the present disclosure is executed.
[0158] As an example, the electronic device 2000 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 2000 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above-mentioned instruction (or instruction set) individually or in combination. The electronic device 2000 may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
[0159] In the electronic device 2000, the processor 2002 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0160] The processor 2002 may execute instructions or codes stored in the memory 2001, wherein the memory 2001 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.
[0161] The memory 2001 may be integrated with the processor 2002, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 2001 may include a separate device, such as an external disk drive, a storage array, or any other storage device that may be used by a database system. The memory 2001 and the processor 2002 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 2002 can read files stored in the memory.
[0162] In addition, the electronic device 2000 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 2000 may be connected to each other via a bus and / or a network.
[0163] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the above-mentioned interaction method performed by the electronic device.
[0164] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to execute additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.
[0165] Another embodiment of the present disclosure relates to a system including at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the above-mentioned interaction method performed by the electronic device.
[0166] It should be noted that the system according to the exemplary embodiments of the present disclosure can completely rely on the execution of computer programs or instructions to realize corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., lib library) to realize the corresponding functions.
[0167] On the other hand, when the above-mentioned system is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.
[0168] According to an exemplary embodiment of the present disclosure, the storage device may be integrated with the computing device, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage device may include an independent device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage device and the computing device may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the computing device can read instructions stored in the storage device.
[0169] Another embodiment of the present disclosure relates to a computer program product, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the interactive method performed by an electronic device as described in any one of the above items is implemented.
[0170] According to the interactive method, apparatus, electronic device, storage medium, system and computer program product performed by the electronic device provided by the present disclosure, the user can interact with the target device through the air through body parts such as hands without relying on any auxiliary physical devices, and the interactive experience is consistent with the traditional touch screen or mouse interaction method, which can significantly improve the user's interactive experience on the target device, especially the target device that requires air interaction.
[0171] In addition, the control instructions corresponding to the user's gesture will only be recognized when a specific preset part touches the surface of an object, which can prevent accidental touches and improve interaction efficiency.
[0172] In addition, when the wrist or other parts touch the surface of an object, the user's hand operation on the surface of the object can be simulated as a mouse operation to control the electronic device remotely, with high interaction efficiency and convenient operation.
[0173] In addition, when the elbow or other hand touches the surface of an object, the movement of the user's hand and the contact with the surface of the object can be simulated as a touch screen operation to control the electronic device remotely, with high interaction efficiency and convenient operation.
[0174] In addition, the learning cost is low, and users do not need to learn specific gestures. Any user who is accustomed to touch screen or mouse interaction can quickly get started. Moreover, the interaction process is easy, and the user's line of sight does not need to frequently switch between auxiliary devices such as projection areas and screens of electronic devices such as smart TVs. There is also no need to make specific gestures in the air. The user can operate electronic devices such as smart TVs on the surface of the object just like touching the screen or operating a mouse.
[0175] In addition, by detecting the distance between the user and the electronic device and adjusting the display size of the virtual object accordingly, it can adapt to the user's visual needs, improve interaction efficiency, and optimize the interaction experience.
[0176] In addition, millimeter-wave radar is used to detect the dynamic situation of the user and surrounding objects, and whether the point cloud overlaps is used to determine whether the user's body parts are in contact with an object and which parts are in contact, which can achieve the corresponding purpose efficiently, conveniently, quickly and with low consumption.
[0177] In addition, the size requirement for the physical plane of the contacting object is relatively low, and users can complete the interaction with any surface of an object within the recognizable range of the millimeter-wave radar. During the interaction process, users can move their position at will and reselect the physical plane for interaction.
[0178] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.
Claims
1. An interactive method performed by an electronic device, It is characterized in that include: In a first operation mode of the electronic device, displaying a virtual object corresponding to a user's hand on a screen of the electronic device; Detecting the state of the user's hand, wherein the state of the hand includes the movement of the hand and the contact between the hand and the surface of the object; Determine a corresponding control instruction based on the detected state of the hand; The virtual object is controlled according to the control instruction to perform interaction between the user and the electronic device.
2. The interactive method performed by an electronic device according to claim 1, It is characterized in that The determining of the corresponding control instruction based on the detected state of the hand comprises: Determine the contact position between the hand and the surface of the object based on the detected contact between the hand and the surface of the object; When it is determined that a preset part of the hand is in contact with the surface of the object, a control instruction corresponding to the movement of the hand is determined based on the detected movement of the hand.
3. The interactive method performed by an electronic device as claimed in claim 2, It is characterized in that When the preset part in contact with the surface of the object includes the first part, the determining of the control instruction corresponding to the hand motion condition based on the detected hand motion condition includes at least one of the following items: When it is detected that the index finger clicks the surface of the object, determining the corresponding control instruction as the first click instruction; When it is detected that the middle finger clicks the surface of the object, determining the corresponding control instruction as a second click instruction; When the index finger is detected sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; When it is detected that the index finger moves on the surface of the object relative to the user, the corresponding control instruction is determined to be the position of the virtual object displayed on the moving screen.
4. The interactive method performed by an electronic device as claimed in claim 2, It is characterized in that When the preset part in contact with the surface of the object includes the second part, the determining of the control instruction corresponding to the hand motion condition based on the detected hand motion condition includes at least one of the following items: When it is detected that the hand is sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; When it is detected that the finger clicks the surface of the object, determining that the corresponding control instruction is a third click instruction; When it is detected that the hand moves relative to the user, the corresponding control instruction is determined to move the position of the virtual object displayed on the screen.
5. The interactive method performed by an electronic device according to claim 1, It is characterized in that The state of the hand also includes the distance between the hand and the electronic device; Wherein, determining the corresponding control instruction based on the detected state of the hand includes: Based on the detected distance between the hand and the electronic device, the corresponding control instruction is determined to adjust the size of the virtual object displayed on the screen.
6. The interactive method performed by an electronic device as claimed in claim 2, It is characterized in that The step of determining the contact position between the hand and the surface of the object based on the detected contact situation between the hand and the surface of the object comprises: The millimeter wave radar is used to detect the point cloud of the hand and objects within the preset range in the preset coordinate system in real time; The object that overlaps with the point cloud of the hand is determined as the object that the hand is currently in contact with; The set of hand parts in the overlapping portion is identified to determine the part where the hand contacts the surface of the object.
7. An interactive device executed by an electronic device, It is characterized in that include: an object display module, configured to: display a virtual object corresponding to a user's hand on a screen of the electronic device in a first operation mode of the electronic device; The state detection module is configured to: detect the state of the user's hand, wherein the state of the hand includes the movement of the hand and the contact between the hand and the surface of the object; The instruction determination module is configured to: determine a corresponding control instruction based on the detected state of the hand; The interaction execution module is configured to: control the virtual object according to the control instruction to execute the interaction between the user and the electronic device.
8. The interactive device executed by an electronic device as claimed in claim 7, It is characterized in that The instruction determination module is configured to: Determine the contact position between the hand and the surface of the object based on the detected contact between the hand and the surface of the object; When it is determined that a preset part of the hand is in contact with the surface of the object, a control instruction corresponding to the movement of the hand is determined based on the detected movement of the hand.
9. The interactive device executed by an electronic device as claimed in claim 8, It is characterized in that When the preset part in contact with the surface of the object includes the first part, the instruction determination module is configured to perform at least one of the following items: When it is detected that the index finger clicks the surface of the object, determining the corresponding control instruction as the first click instruction; When it is detected that the middle finger clicks the surface of the object, determining the corresponding control instruction as a second click instruction; When the index finger is detected sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; When it is detected that the index finger moves on the surface of the object relative to the user, the corresponding control instruction is determined to be the position of the virtual object displayed on the moving screen.
10. The interactive device executed by an electronic device according to claim 8, It is characterized in that When the preset part in contact with the surface of the object includes the second part, the instruction determination module is configured to perform at least one of the following items: When it is detected that the hand is sliding on the surface of the object, the corresponding control instruction is determined to be sliding the current page; When it is detected that the finger clicks the surface of the object, determining that the corresponding control instruction is a third click instruction; When it is detected that the hand moves relative to the user, the corresponding control instruction is determined to move the position of the virtual object displayed on the screen.
11. The interactive device executed by an electronic device according to claim 7, It is characterized in that The state of the hand also includes the distance between the hand and the electronic device; Wherein, the instruction determination module is configured as follows: Based on the detected distance between the hand and the electronic device, the corresponding control instruction is determined to adjust the size of the virtual object displayed on the screen.
12. The interactive device executed by an electronic device according to claim 8, It is characterized in that The instruction determination module is configured to: The millimeter wave radar is used to detect the point cloud of the hand and objects within the preset range in the preset coordinate system in real time; The object that overlaps with the point cloud of the hand is determined as the object that the hand is currently in contact with; The set of hand parts in the overlapping portion is identified to determine the part where the hand contacts the surface of the object.
13. An electronic device, It is characterized in that include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, they prompt the at least one processor to execute the interaction method performed by the electronic device as described in any one of claims 1 to 6.
14. A computer-readable storage medium storing instructions, It is characterized in that When the instructions are executed by at least one processor, the at least one processor is prompted to execute the interaction method performed by an electronic device as described in any one of claims 1 to 6.
15. A system comprising at least one computing device and at least one storage device storing instructions, It is characterized in that When the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the interaction method performed by an electronic device as described in any one of claims 1 to 6.
16. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instructions are executed by a processor, the interaction method performed by an electronic device as described in any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Point cloud-based finger pinching interaction method, head-mounted display device, interaction system and storage medium
CN120949947A
Point cloud-based pinch interaction method, head-mounted device, interaction system and storage medium
CN120949947B
