Gesture interaction method and device, electronic equipment and storage medium

Abstract

The invention relates to a gesture interaction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring an image containing a gesture of a user; identifying a gesture in the image, and determining a spatial position of the gesture; and determining a prompt voice corresponding to the gesture, carrying out sound image positioning on the prompt voice according to the spatial position of the gesture, and outputting the sound image. Through the method, a listener can feel what gestures are made and what positions the gestures are made, and intelligence is high.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a gesture interaction method and device, an electronic device, and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, the ways of human-computer interaction have become more and more diversified, such as the emergence of contactless gesture interaction. During gesture interaction, electronic devices (such as virtual reality devices) require gesture recognition to perform some convenient operations. However, in many cases, users cannot know the gesture recognition status, resulting in poor user experience and poor intelligence of electronic devices. Summary of the invention

[0003] The present invention provides a gesture interaction method and device, an electronic device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a gesture interaction method is provided, including:

[0005] Get an image containing the user's gesture;

[0006] recognizing a gesture in the image and determining a spatial position of the gesture;

[0007] The prompt voice corresponding to the gesture is determined, and the prompt voice is localized according to the spatial position of the gesture and then outputted.

[0008] In some embodiments, the identifying a gesture in the image includes:

[0009] Recognize the finger shape in the image, and / or recognize the gesture as left hand or right hand;

[0010] The determining the prompt voice corresponding to the gesture includes:

[0011] The prompt voice corresponding to the gesture is determined based on at least one of the recognition result of the left hand or the right hand and the recognition result of the finger shape.

[0012] In some embodiments, the image includes a plurality of gestures; determining the prompt voice corresponding to the gesture, and outputting the prompt voice after performing sound and image localization according to the spatial position of the gesture, comprises:

[0013] For each gesture recognized, the prompt voice corresponding to the gesture is determined, and the prompt voice is output after sound and image positioning according to the spatial position of the gesture; wherein, the prompt voices for different gestures are different, and the sound and image orientations of the prompt voices for gestures at different spatial positions are different.

[0014] In some embodiments, the outputting the prompt voice after performing sound and image localization according to the spatial position of the gesture includes:

[0015] According to the spatial position of the gesture, the prompt voice is localized using a head-related transfer function and then output.

[0016] In some embodiments, the spatial position of the gesture includes: the distance of the gesture relative to the electronic device that collects the image and / or the azimuth of the gesture relative to the electronic device;

[0017] The method of outputting the prompt voice after performing sound image localization based on the spatial position of the gesture by using a head-related transfer function comprises:

[0018] Converting the prompt voice into a frequency domain signal;

[0019] Based on the distance and / or azimuth of the gesture relative to the electronic device, the frequency domain signal is rendered using the head-related transfer function to obtain a rendered frequency domain signal;

[0020] Convert the rendered frequency domain signal into a time domain signal and output it.

[0021] In some embodiments, determining the spatial position of the gesture includes:

[0022] The spatial position of the gesture is determined based on preset calibration parameters of a camera in an electronic device that captures the image and the recognized position of the gesture in the image.

[0023] In some embodiments, the gesture interaction method is applied to the electronic device, and the electronic device includes a virtual reality device.

[0024] According to a second aspect of an embodiment of the present disclosure, a gesture interaction device is provided, the device comprising:

[0025] An acquisition module configured to acquire an image containing a user's gesture;

[0026] a recognition module configured to recognize a gesture in the image and determine a spatial position of the gesture;

[0027] The sound and image localization module is configured to determine the prompt voice corresponding to the gesture, and output the prompt voice after sound and image localization according to the spatial position of the gesture.

[0028] In some embodiments, the recognition module is further configured to recognize the finger shape in the image, and / or recognize the gesture as being of the left hand or the right hand; the sound and image localization module is further configured to determine the prompt voice corresponding to the gesture based on the recognition result of the left hand or the right hand, and at least one of the recognition results of the finger shape.

[0029] In some embodiments, the image includes multiple gestures; the sound and image localization module is further configured to determine the prompt voice corresponding to each recognized gesture, and output the prompt voice after sound and image localization according to the spatial position of the gesture; wherein the prompt voices for different gestures are different, and the sound and image orientations of the prompt voices for gestures in different spatial positions are different.

[0030] In some embodiments, the sound and image localization module is further configured to perform sound and image localization on the prompt voice using a head-related transfer function according to the spatial position of the gesture and then output the result.

[0031] In some embodiments, the spatial position of the gesture includes: the distance of the gesture relative to the electronic device that collects the image and / or the azimuth of the gesture relative to the electronic device; the sound and image localization module is further configured to convert the prompt voice into a frequency domain signal; based on the distance and / or azimuth of the gesture relative to the electronic device that collects the image, the frequency domain signal is rendered using the head-related transfer function to obtain a rendered frequency domain signal; the rendered frequency domain signal is converted into a time domain signal and output.

[0032] In some embodiments, the recognition module is further configured to determine the spatial position of the gesture based on preset calibration parameters of a camera in the electronic device that captures the image and the recognized position of the gesture in the image.

[0033] In some embodiments, the gesture interaction method is applied to the electronic device, and the electronic device includes a virtual reality device.

[0034] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0035] processor;

[0036] a memory for storing processor-executable instructions;

[0037] Wherein, the processor is configured to execute the gesture interaction method as described in the first aspect above.

[0038] According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0039] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the gesture interaction method as described in the first aspect above.

[0040] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0041] In the embodiments of the present disclosure, after the electronic device determines the gesture and spatial position in the image, it performs audio-visual orientation of the prompt voice corresponding to the gesture based on the spatial position and then outputs it, so that the listener can feel what kind of gesture is made and at what position the gesture is made. In addition, based on the prompt voice corresponding to the gesture, it is also convenient for the listener to understand whether it is an erroneous gesture, and the intelligence is relatively high.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0044] Figure 1 It is a flow chart of a gesture interaction method shown in an embodiment of the present disclosure.

[0045] Figure 2 This is an example diagram of a gesture in an embodiment of the present disclosure.

[0046] Figure 3 An example coordinate diagram provided by an embodiment of the present disclosure is established based on an electronic device that collects images.

[0047] Figure 4 Shown is an example diagram of a head-mounted glasses in an embodiment of the present disclosure.

[0048] Figure 5 This is an example diagram of a voice prompt output after sound and image localization in an embodiment of the present disclosure.

[0049] Figure 6 This is an example diagram of a gesture interaction device in an embodiment of the present disclosure.

[0050] Figure 7 It is a block diagram of a device according to an exemplary embodiment. DETAILED DESCRIPTION

[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0052] Figure 1 is a flow chart of a gesture interaction method shown in an embodiment of the present disclosure, such as Figure 1 As shown, the following steps are included:

[0053] S11, acquiring an image including a user's gesture;

[0054] S12, recognizing a gesture in the image, and determining a spatial position of the gesture;

[0055] S13: Determine a prompt voice corresponding to the gesture, and output the prompt voice after performing sound and image localization on the prompt voice according to the spatial position of the gesture.

[0056] In some embodiments of the present disclosure, the gesture interaction method can be applied to electronic devices that support gesture interaction, such as user equipment (UE), mobile phones, tablet computers, personal digital assistants (PDAs), vehicle-mounted devices, virtual reality devices, etc. The electronic device may include a camera, which captures an image through the camera, identifies the gesture in the image and the spatial position of the gesture, and then outputs a gesture prompt voice according to the spatial position of the gesture. Among them, the camera in the electronic device may include one or more; in addition, the electronic device may also include a depth camera such as TOF (Time of Flight).

[0057] In other embodiments of the present disclosure, the gesture interaction method can be applied to other devices associated with the electronic device supporting gesture interaction, such as a server. After the electronic device captures an image through a camera, the image is sent to the server, and the server recognizes the gesture included in the image and sends it to the electronic device. The electronic device determines the spatial position of the gesture and determines the corresponding prompt voice according to the gesture, and then performs sound and image localization on the prompt voice and outputs it. In the embodiments of the present disclosure, the application control method is applied to an electronic device for gesture interaction as an example for explanation.

[0058] In step S11, the electronic device acquires an image containing a user's gesture, which is an image that can be captured within the acquisition range of the camera of the electronic device, and the gesture may be an air gesture. In the image containing the user's gesture, the gesture includes the finger shape, which may include a dynamic gesture or a static gesture, wherein a dynamic gesture refers to a gesture whose finger shape changes over time, for example, it may include a sliding gesture or a clicking gesture; a static gesture refers to a gesture whose finger shape is fixed, such as a "1" gesture or a heart gesture, etc.; in addition, in the disclosed embodiment, the gesture may also include a left hand or a right hand.

[0059] In the embodiment of the present disclosure, the image acquired by the electronic device may be a single image or an image sequence including multiple images, wherein the image sequence includes images acquired in chronological order. The electronic device may recognize gestures based on one or more images acquired within a preset time period. In addition, in the embodiment of the present disclosure, the image may include multiple gestures, and the multiple gestures may be gestures of the left and right hands of the same user, or gestures of the hands of different users.

[0060] Figure 2 is an example diagram of a gesture in an embodiment of the present disclosure, such as Figure 2 As shown, L1 is a dynamic gesture of sliding to the right, L2 is a dynamic gesture of clicking twice in a row with a single finger, and L3 is a static gesture of opening the palm.

[0061] In step S12, the electronic device recognizes a gesture in the image, including recognizing the shape of fingers in the image, and / or recognizing the gesture as a left hand or a right hand.

[0062] In some embodiments, the electronic device may first detect the hand, and when the hand is detected, further determine the left / right hand, and the finger shape. When detecting the hand, for example, the characteristics of the hand skin color may be used to detect the hand based on the fact that the color range of the hand is relatively uniform and clustered; an edge-based gesture detection method may also be used to calculate the outline of the image in the image using an edge detection operator to determine whether it matches the outline of the hand based on the outline. When determining the left / right hand, and the finger shape, the left and right hands may be distinguished based on the outline of the hand, and then a template matching method may be further used to compare the detected hand with different gestures in the template library, and the gesture with the highest matching degree may be determined as the finger shape of the hand included in the image; of course, the template matching method may also be used directly to distinguish the left / right hand and the finger shape.

[0063] It should be noted that if the electronic device acquires a single image, the gesture recognized based on the single image can be understood as a static gesture, and the electronic device can determine the corresponding prompt voice based only on the static gesture. If the electronic device acquires multiple images, the electronic device can determine whether it is a static gesture or a dynamic gesture by comparing the differences between the gestures recognized in each image.

[0064] In other embodiments, an image containing a user's gesture (one or more images within a preset time) may also be input into a gesture recognition model to determine the gesture included in the image, wherein the gesture recognition model may be trained based on a deep learning network, for example, based on training sample data including different gesture images and label values, a network such as a convolutional neural network (CNN) or a deep learning network (DNN) is trained and adjusted to obtain a gesture recognition model, wherein the label value is a preset gesture recognition result. In the disclosed embodiment, the gesture recognition model may recognize one or more gestures in an image, and may recognize left and right hands, and / or finger shapes.

[0065] In the disclosed embodiment, the electronic device further determines a spatial position of the gesture, where the spatial position includes a distance between the gesture and the electronic device that captures the image and / or an azimuth angle between the gesture and the electronic device that captures the image.

[0066] In some embodiments, the electronic device may determine the relative position between the electronic device carried by the user making the gesture and the electronic device collecting the image, and determine the relative position as the spatial position of the gesture. For example, the electronic device collecting the image may determine the distance between the electronic device carried by the user making the gesture based on a distance sensor (such as a TOF sensor), and may determine the azimuth of the electronic device carried by the user making the gesture relative to the electronic device collecting the image based on the sending and receiving of sound wave signals between the electronic devices.

[0067] In other embodiments, the electronic device may also determine the spatial position of the gesture based on the image and calibration parameters of the electronic device that captures the image.

[0068] In step S13, the electronic device determines the prompt voice corresponding to the gesture. In some embodiments, the electronic device determines the prompt voice corresponding to the gesture based on at least one of the recognition result of the left hand or the right hand and the recognition result of the finger shape. For example, the prompt voices of the left hand and the right hand may be different; the prompt voices of different finger shapes may also be different. It can be understood that by determining the prompt voice based on a variety of methods, it is convenient for the listener to understand the specific form of the gesture, and the intelligence is high.

[0069] In the disclosed embodiment, the electronic device also outputs the prompt voice after performing sound and image localization according to the spatial position of the gesture. The prompt voice for each gesture may be processed and output using relevant processing functions for sound and image localization, such as Head Related Transfer Function (HRTF), Head Related Impulse Response (HRIR), etc. In the disclosed embodiment, the prompt voice is output after performing sound and image localization so that the listener can feel a sense of hearing with a sense of picture, that is, the listener can distinguish the spatial position of the gesture from the output sound.

[0070] In the related art, gesture recognition has no sound prompt or a simple sound prompt without any position directionality, resulting in the user being unable to know whether he has made a gesture, what kind of gesture he has made, and where he has made the gesture.

[0071] In this regard, in the embodiment of the present disclosure, after the electronic device determines the gesture and spatial position in the image, it performs audio-visual orientation of the prompt voice corresponding to the gesture based on the spatial position and then outputs it, so that the listener can feel what kind of gesture is made and at what position the gesture is made. In addition, based on the prompt voice corresponding to the gesture, it is also convenient for the listener to understand whether it is an erroneous gesture, and the intelligence is relatively high.

[0072] In some embodiments, the image includes a plurality of gestures; determining the prompt voice corresponding to the gesture, and outputting the prompt voice after performing sound and image localization according to the spatial position of the gesture, comprises:

[0073] For each gesture recognized, the prompt voice corresponding to the gesture is determined, and the prompt voice is output after sound and image positioning according to the spatial position of the gesture; wherein, the prompt voices for different gestures are different, and the sound and image orientations of the prompt voices for gestures at different spatial positions are different.

[0074] As mentioned above, the image may include multiple gestures. In the embodiment of the present disclosure, for each recognized gesture, the electronic device outputs prompt voices with different audio and video directions, and the prompt voices for different gestures are also different, so that the listener can clearly know the control direction and type of each gesture, and the intelligence is relatively high.

[0075] In some embodiments, the outputting the prompt voice after performing sound and image localization according to the spatial position of the gesture includes:

[0076] According to the spatial position of the gesture, the prompt voice is localized using a head-related transfer function and then output.

[0077] In the embodiment of the present disclosure, since the head-related transfer function describes the transmission process of sound waves from the sound source to the two ears, it is a function model established by simulating the human physiological structure (such as the head, auricle, and torso, etc.) and the human perception of sound. Therefore, sound and image positioning is performed based on the head-related transfer function, so that the prompt voice output can provide the listener with a better listening experience.

[0078] The following formula is the expression of head-related transfer function:

[0079]

[0080]

[0081] Among them, H L is the transfer function corresponding to the left ear, H R is the transfer function corresponding to the right ear; the angular frequency f of the sound wave refers to the angular frequency of the phonon when it propagates in a solid, liquid or gas. The magnitude of the angular frequency depends on the physical properties of the medium in which the phonon propagates; r is the distance from the sound source to the center of the head. When r is greater than 1.2 meters, H L and H R Basically, it has nothing to do with r; θ is the horizontal azimuth angle of the sound source relative to the center of the head; is the elevation angle of the sound source relative to the center of the head; P0 is the complex sound pressure at the center of the head when the head does not exist; P L and P R They are the complex sound pressures produced by the sound source at the left and right ears of the listener, respectively.

[0082] It should be noted that, in the embodiment of the present disclosure, the head-related transfer function can be a function applicable to the listener himself, or can be a universal function. The embodiment of the present disclosure uses the head-related transfer function to perform sound image localization, and the solution is simple and effective.

[0083] In some embodiments, the spatial position of the gesture includes: the distance of the gesture relative to the electronic device that collects the image and / or the azimuth of the gesture relative to the electronic device;

[0084] The method of outputting the prompt voice after performing sound image localization based on the spatial position of the gesture by using a head-related transfer function comprises:

[0085] Converting the prompt voice into a frequency domain signal;

[0086] Based on the distance and / or azimuth of the gesture relative to the electronic device, the frequency domain signal is rendered using the head-related transfer function to obtain a rendered frequency domain signal;

[0087] Convert the rendered frequency domain signal into a time domain signal and output it.

[0088] In the disclosed embodiment, the spatial position of the gesture includes the distance of the gesture relative to the electronic device that captures the image and / or the azimuth of the gesture relative to the electronic device, and the azimuth may include the horizontal azimuth and / or elevation of the gesture relative to the electronic device.

[0089] The head-related transfer function processes speech in the frequency domain. In the disclosed embodiment, the electronic device converts the prompt speech of the gesture into a frequency domain signal. For example, the prompt speech can be converted into the frequency domain by Fourier transform. Subsequently, the electronic device substitutes the distance and / or azimuth between the gesture and the electronic device that collects the image based on the aforementioned formula (1) and / or formula (2) to obtain a rendered frequency domain signal, and then converts the frequency domain signal into a time domain signal output, for example, by inverse Fourier transform.

[0090] It should be noted that if the electronic device outputs the prompt voice after binaural cross-localization based on formula (1) and formula (2), the listener can feel the stereo surround of binaural. In addition, if the spatial position of the gesture includes one of the distance or azimuth, the unknown parameter can be set to a preset value. For example, when the spatial position only includes the horizontal azimuth, the distance and elevation can be set to preset values. Due to the difference in horizontal azimuth, different sound image localization effects can still be obtained based on the head-related transfer function.

[0091] As mentioned above, the electronic device can determine the spatial position of the gesture based on the image and the calibration parameters of the electronic device that captures the image. In some embodiments, for example, when the electronic device that captures the image is an electronic device that includes a binocular camera, after the electronic device captures the first image and the second image that include the gesture through the binocular camera, the azimuth and distance of the gesture relative to the electronic device that captures the image can be determined based on the position of the same gesture in different images.

[0092] In some other embodiments, determining the gesture in the image and the spatial position of the gesture includes:

[0093] recognizing an image in the gesture;

[0094] The spatial position of the gesture is determined based on preset calibration parameters of a camera in an electronic device that captures the image and the recognized position of the gesture in the image.

[0095] In the disclosed embodiment, the electronic device may recognize one or more gestures in the image based on the aforementioned method. It should be noted that after the gesture is recognized, the position of the gesture in the image can be known. After the electronic device determines the position of the gesture in the image, it can determine the spatial position of the gesture in combination with the preset calibration parameters of the camera.

[0096] Exemplarily, when the electronic device that captures images is a monocular camera, the electronic device can determine the position of the gesture in the camera coordinate system based on the position of the gesture in the image and the internal parameters of the electronic device that captures the image, thereby obtaining the distance and azimuth of the gesture relative to the electronic device that captures the image.

[0097] Figure 3 An example coordinate diagram provided by an embodiment of the present disclosure is established based on an electronic device that collects images, such as Figure 3 As shown, it is assumed that the orientation of the electronic device is the y-axis, the vertical direction is the z-axis, and the x-axis is perpendicular to the z-axis and the y-axis. After determining the position (x, y, z) value of the gesture in the camera coordinate system based on the internal parameters of the electronic device, the distance, horizontal azimuth angle, and elevation angle of the gesture relative to the electronic device that captures the image can be calculated based on the following formula (3):

[0098] x=rcosθsinφ; y=rsinθsinφ; z=rcosφ (3)

[0099] Wherein, r is the distance between the gesture and the electronic device that collects the image; θ is the angle between the y-axis and the x-axis, that is, the horizontal azimuth; φ is the angle between the z-axis and the xy plane, that is, the elevation angle.

[0100] It can be understood that in the embodiment of the present disclosure, the electronic device determines the spatial position of the gesture by combining the image and the preset calibration parameters of the camera in the electronic device that captures the image. The solution is simple and effective.

[0101] In some embodiments, the gesture interaction method is applied to the electronic device, and the electronic device includes a virtual reality device.

[0102] In an embodiment of the present disclosure, the gesture interaction method is applied to a virtual reality device, such as head-mounted glasses. Figure 4 FIG. 1 is an example diagram of a head-mounted glasses according to an embodiment of the present disclosure. Figure 4 As shown, the head-mounted glasses include three cameras A1, A2 and A3, and the spatial position of the gesture can be located through the head-mounted glasses themselves.

[0103] In addition, in the embodiment of the present disclosure, the virtual reality device further includes a speaker, and the user of the virtual reality device can have a better listening experience based on the sound and image localization prompt sound output by the speaker. Figure 5 This is an example diagram of a prompt voice output after sound and image localization in an embodiment of the present disclosure. The user's ears can recognize the type and direction of the gesture based on the prompt sound after sound and image localization.

[0104] Since virtual reality devices require gesture recognition to perform some convenient operations, inaccurate gesture recognition or no feedback from gesture recognition often makes it difficult for users of virtual reality devices to use them. Therefore, the method of the disclosed embodiment can bring a better experience to users when applied to virtual reality devices, and can improve the intelligence of virtual reality devices.

[0105] It should be noted that in the above method of the specific implementation mode, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0106] Figure 6 is an example diagram of a gesture interaction device in an embodiment of the present disclosure, referring to Figure 6 , the device comprises:

[0107] An acquisition module 101 is configured to acquire an image including a user's gesture;

[0108] A recognition module 102, configured to recognize a gesture in the image and determine a spatial position of the gesture;

[0109] The sound and image localization module 103 is configured to determine the prompt voice corresponding to the gesture, and output the prompt voice after sound and image localization according to the spatial position of the gesture.

[0110] In some embodiments, the recognition module 102 is further configured to recognize the finger shape in the image, and / or recognize the gesture as being of the left hand or the right hand; the sound and image localization module is further configured to determine the prompt voice corresponding to the gesture based on at least one of the recognition results of the left hand or the right hand, and the recognition results of the finger shape.

[0111] In some embodiments, the image includes multiple gestures; the sound and image localization module 103 is further configured to determine the prompt voice corresponding to each recognized gesture, and output the prompt voice after sound and image localization according to the spatial position of the gesture; wherein the prompt voices for different gestures are different, and the sound and image orientations of the prompt voices for gestures at different spatial positions are different.

[0112] In some embodiments, the sound image localization module 103 is further configured to perform sound image localization on the prompt voice using a head related transfer function according to the spatial position of the gesture and then output the result.

[0113] In some embodiments, the spatial position of the gesture includes: the distance of the gesture relative to the electronic device that collects the image and / or the azimuth of the gesture relative to the electronic device; the sound and image localization module 103 is also configured to convert the prompt voice into a frequency domain signal; based on the distance and / or azimuth of the gesture relative to the electronic device that collects the image, the frequency domain signal is rendered using the head-related transfer function to obtain a rendered frequency domain signal; the rendered frequency domain signal is converted into a time domain signal and output.

[0114] In some embodiments, the recognition module 102 is further configured to determine the spatial position of the gesture based on preset calibration parameters of a camera in the electronic device that captures the image and the recognized position of the gesture in the image.

[0115] In some embodiments, the gesture interaction method is applied to the electronic device, and the electronic device includes a virtual reality device.

[0116] about Figure 6 The specific manner in which each module performs operations in the device in the illustrated embodiment has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0117] Figure 7 8 is a block diagram of a device according to an exemplary embodiment. For example, the apparatus 800 (device) may be a smart phone, a virtual reality device, etc.

[0118] Reference Figure 7 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0119] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0120] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0121] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with determining, managing, and distributing power for the device 800.

[0122] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0123] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0124] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0125] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0126] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0127] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0128] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0129] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the aforementioned gesture interaction method.

[0130] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims above.

[0131] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A gesture interaction method, characterized in that: The method comprises: Get an image containing the user's gesture; recognizing a gesture in the image and determining a spatial position of the gesture; The prompt voice corresponding to the gesture is determined, and the prompt voice is localized according to the spatial position of the gesture and then outputted.

2. The method according to claim 1, characterized in that The identifying the gesture in the image comprises: Recognize the finger shape in the image, and / or recognize the gesture as left hand or right hand; The determining the prompt voice corresponding to the gesture includes: The prompt voice corresponding to the gesture is determined based on at least one of the recognition result of the left hand or the right hand and the recognition result of the finger shape.

3. The method according to claim 1 or 2, characterized in that: The image includes a plurality of gestures; the determining the prompt voice corresponding to the gesture, and outputting the prompt voice after performing sound and image localization according to the spatial position of the gesture, comprises: For each gesture recognized, the prompt voice corresponding to the gesture is determined, and the prompt voice is output after sound and image positioning according to the spatial position of the gesture; wherein, the prompt voices for different gestures are different, and the sound and image orientations of the prompt voices for gestures at different spatial positions are different.

4. The method according to claim 1, characterized in that: The outputting of the prompt voice after performing sound and image localization according to the spatial position of the gesture comprises: According to the spatial position of the gesture, the prompt voice is localized using a head-related transfer function and then output.

5. The method according to claim 4, characterized in that The spatial position of the gesture includes: the distance of the gesture relative to the electronic device that collects the image and / or the azimuth of the gesture relative to the electronic device; The method of outputting the prompt voice after performing sound image localization based on the spatial position of the gesture by using a head-related transfer function comprises: Converting the prompt voice into a frequency domain signal; Based on the distance and / or azimuth of the gesture relative to the electronic device, the frequency domain signal is rendered using the head-related transfer function to obtain a rendered frequency domain signal; Convert the rendered frequency domain signal into a time domain signal and output it.

6. The method according to claim 1, characterized in that Determining the spatial position of the gesture includes: The spatial position of the gesture is determined based on preset calibration parameters of a camera in an electronic device that captures the image and the recognized position of the gesture in the image.

7. The method according to claim 6, characterized in that The gesture interaction method is applied to the electronic device, and the electronic device includes a virtual reality device.

8. A gesture interaction device, characterized in that: The device comprises: An acquisition module configured to acquire an image containing a user's gesture; a recognition module configured to recognize a gesture in the image and determine a spatial position of the gesture; The sound and image localization module is configured to determine the prompt voice corresponding to the gesture, and output the prompt voice after sound and image localization according to the spatial position of the gesture.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the gesture interaction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor in an electronic device, the electronic device is enabled to execute the gesture interaction method as described in any one of claims 1 to 7.

Images