Interactive method and related device
The virtual hand model interacts with the virtual pet through electronic devices by identifying user gestures and actions, solving the problems of high cost of raising pets and difficulty in management, providing low-cost, widely applicable interactive methods, and enhancing the user experience.
Patent Information
- Application Number
- CN202111131758.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-26
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-09-26
AI Technical Summary
Raising pets takes a lot of time and money, and improper management is prone to pet injuries. The existing interactive methods are single, costly and have low sense of participation.
Identify the user's gestures through electronic devices, control the interaction between the virtual hand model and the virtual pet, realize the interaction between the user and the virtual pet, and use deep learning algorithms and computer graphics technology to generate highly realistic 2D or 3D pets and hands. Model.
Without adding additional equipment, a wider range of applicable scenarios, simple operations and low-cost interaction methods are provided to enhance the user experience and meet the needs of users' interaction with pets.
Smart Images

Figure CN113986093B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an interactive method and related devices. Background Art
[0002] With the development of urbanization and the change of people's way of thinking, keeping pets is a common phenomenon in today's society. Various pets such as cats and dogs can not only fill the emptiness in your heart, but also add a lot of fun to your life.
[0003] However, keeping pets also brings many problems: on the one hand, keeping pets takes a lot of time and money, and the cost is high, such as the need to vaccinate pets frequently and the need to be responsible for the pets' eating, drinking, defecation, etc. On the other hand, improper pet management can easily lead to pet injuries.
[0004] Therefore, how to achieve interaction between users and virtual pets is a problem that those skilled in the art are currently studying. Summary of the invention
[0005] Embodiments of the present application provide an interactive method and related devices. Through some embodiments of the present application, an electronic device can control a virtual hand model and a virtual pet according to the user's gestures, and effectively realize the interaction between the user and the virtual pet through the virtual hand model.
[0006] In a first aspect, an embodiment of the present application provides an interactive method, the method comprising:
[0007] In response to a first user operation, the electronic device displays a first interface, the first interface including a first virtual pet and a first virtual hand model;
[0008] The electronic device acquires one or more frames of images, and recognizes a first gesture action according to the one or more frames of images;
[0009] The electronic device displays the first virtual hand model as the first gesture action, and displays the first virtual pet as a first interactive action, and the first interactive action and the first gesture action are used for the first virtual pet to interact with the first virtual hand model.
[0010] It is understandable that the interactive method shown in the present application is performed by an electronic device, which can be referred to as an interactive device. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a desktop computer, and the like.
[0011] Exemplarily, the electronic device may include a display screen, which is used to display the first interface. The first user operation can be understood as a user operation such as a click operation, a slide operation, and a long press operation, and the first user operation is used to trigger the electronic device to display the first interface.
[0012] Exemplarily, the first virtual pet can be understood as a two-dimensional (2D) or three-dimensional (3D) pet model generated by an electronic device using computer technology. Specifically, the type of the first virtual pet can be a kitten, a puppy, a bird, etc., which is not limited in this application. The first virtual hand model can be understood as a 2D or 3D hand model generated by an electronic device using computer technology.
[0013] Exemplarily, the electronic device may analyze the acquired one or more frames of images through a gesture posture estimation algorithm, identify a first gesture action corresponding to the one or more frames of images, and then control the first virtual pet and the first virtual hand model according to the first gesture action, thereby effectively realizing the interaction between the user and the virtual pet through the first virtual hand model.
[0014] In a possible implementation, the electronic device recognizes a first gesture action according to the one or more frames of images, including:
[0015] When the electronic device acquires a frame of image, the electronic device identifies a plurality of gesture key points corresponding to the frame of image from the frame of image; the plurality of gesture key points corresponding to the frame of image correspond to finger positions and wrist positions of the hand;
[0016] The electronic device recognizes the first gesture action according to the gesture key points corresponding to the frame of image.
[0017] In a possible implementation, the electronic device recognizes a first gesture action according to the one or more frames of images, including:
[0018] When the electronic device acquires multiple frames of images, the electronic device identifies multiple gesture key points corresponding to the first image from a first image, where the first image is one frame of the multiple frames of images; the multiple gesture key points corresponding to the first image correspond to finger positions and wrist positions of the hand;
[0019] The electronic device recognizes the first gesture action according to the gesture key points corresponding to each frame of the multiple frames of images; the positions of the multiple gesture key points corresponding to at least two frames of images in the multiple frames of images are different.
[0020] In one possible implementation, the multiple gesture key points include 21 gesture key points, gesture key point 0 represents the point closest to the wrist; gesture key point 1 to gesture key point 4 represent points corresponding to the thumb of the palm; gesture key point 5 to gesture key point 8 represent points corresponding to the index finger of the palm; gesture key point 9 to gesture key point 12 represent points corresponding to the middle finger of the palm; gesture key point 13 to gesture key point 16 represent points corresponding to the ring finger of the palm; gesture key point 17 to gesture key point 20 represent points corresponding to the little finger of the palm.
[0021] In a possible implementation, the initial position of the first virtual hand model corresponds to any of the following parts of the first virtual pet: the head of the first virtual pet, the tail of the first virtual pet, and the feet of the first virtual pet.
[0022] In a possible implementation manner, the electronic device displays the first virtual hand model as the first gesture action, including:
[0023] The electronic device displays the first virtual hand model as a gesture action in a database that has the highest matching degree with the first gesture action, and the database includes at least two gesture actions.
[0024] In a possible implementation, the first gesture action includes any one or more of the following: waving, stroking, patting, and clenching a fist; the first interactive action includes any one or more of the following: shaking head, wagging tail, sticking out tongue, waving, and lying down.
[0025] In a second aspect, an embodiment of the present application provides an interactive device, the device comprising:
[0026] A response unit, configured to respond to a first user operation; and control the display unit to display a first interface, the first interface including a first virtual pet and a first virtual hand model;
[0027] An acquisition unit, used for acquiring one or more frames of images;
[0028] The recognition unit is used to recognize a first gesture action according to the one or more frames of images; and control the display unit to display the first virtual hand model as the first gesture action, and to display the first virtual pet as a first interactive action, wherein the first interactive action and the first gesture action are used for the first virtual pet to interact with the first virtual hand model.
[0029] In a possible implementation, the recognition unit is specifically used to, when acquiring a frame of image, recognize multiple gesture key points corresponding to the frame of image from the frame of image; the multiple gesture key points corresponding to the frame of image correspond to the finger positions and wrist positions of the hand;
[0030] The recognition unit is specifically used to recognize the first gesture action according to the gesture key points corresponding to the frame image.
[0031] In a possible implementation, the recognition unit is specifically used to recognize multiple gesture key points corresponding to the first image from a first image when acquiring multiple frames of images, where the first image is one frame of the multiple frames of images; the multiple gesture key points corresponding to the first image correspond to finger positions and wrist positions of the hand;
[0032] The recognition unit is specifically used to recognize the first gesture action according to the gesture key points corresponding to each frame of the multiple frames of images; the positions of the multiple gesture key points corresponding to at least two frames of the multiple frames of images are different.
[0033] In one possible implementation, the multiple gesture key points include 21 gesture key points, gesture key point 0 represents the point closest to the wrist; gesture key point 1 to gesture key point 4 represent points corresponding to the thumb of the palm; gesture key point 5 to gesture key point 8 represent points corresponding to the index finger of the palm; gesture key point 9 to gesture key point 12 represent points corresponding to the middle finger of the palm; gesture key point 13 to gesture key point 16 represent points corresponding to the ring finger of the palm; gesture key point 17 to gesture key point 20 represent points corresponding to the little finger of the palm.
[0034] In a possible implementation, the initial position of the first virtual hand model corresponds to any of the following parts of the first virtual pet: the head of the first virtual pet, the tail of the first virtual pet, and the feet of the first virtual pet.
[0035] In a possible implementation, the display unit is specifically configured to display the first virtual hand model as a gesture action in a database that has the highest matching degree with the first gesture action, and the database includes at least two gesture actions.
[0036] In a possible implementation, the first gesture action includes any one or more of the following: waving, stroking, patting, and clenching a fist; the first interactive action includes any one or more of the following: shaking head, wagging tail, sticking out tongue, waving, and lying down.
[0037] In a third aspect, an embodiment of the present application discloses an electronic device, comprising: a processor and a memory, wherein the memory stores a computer program, and the processor calls the computer program stored in the memory to execute a method as in the first aspect or any possible implementation of the first aspect.
[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on one or more processors, the method in the first aspect or any possible implementation of the first aspect is executed.
[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes program instructions, and when the program instructions are executed by a processor, the processor executes a method as in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the background technology.
[0041] Figure 1 It is a flowchart of an interactive method provided in an embodiment of the present application;
[0042] Figure 2 is a schematic diagram showing a first virtual pet and a first virtual hand model provided by an embodiment of the present application;
[0043] Figure 3 is a schematic diagram of recognizing a first gesture action provided by an embodiment of the present application;
[0044] Figure 4 is another schematic diagram of recognizing a first gesture action provided by an embodiment of the present application;
[0045] Figure 5 is a structural schematic diagram of an interactive device provided in an embodiment of the present application;
[0046] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "the", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear contrary indication in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items. The terms "first" and "second" in the specification, claims and drawings of the present application, etc. are used to distinguish different objects, rather than to describe a specific order.
[0048] With the development of urbanization, the physical distance between people has shortened, but the distance in their hearts has gradually increased. In order to fill the inner emptiness, various small animals such as cats and dogs have entered thousands of households. However, raising pets such as cats and dogs also brings many problems: on the one hand, raising pets takes a lot of time and money, and the cost is high. For example, pets need to be vaccinated frequently and they need to be responsible for their eating, drinking, defecating and urinating. On the other hand, improper pet management can easily lead to pet injuries.
[0049] At present, the wave of deep learning is sweeping all walks of life. Algorithms based on deep learning are far superior to traditional algorithms in terms of algorithm accuracy and execution speed, making a variety of application scenarios possible. In addition, the development of computer graphics is also rapid, and highly realistic and high-degree-of-freedom models can be easily created. Therefore, in order to meet the desire of users who are not convenient to keep pets but want to interact with pets, deep learning algorithms, computer graphics and other technologies can be used to allow users to interact with virtual pets through electronic devices.
[0050] For example, the popular Tom Cat application enables the user to interact with Tom Cat by responding to the user clicking the screen. However, the operation function of this solution is single and the sense of participation is low.
[0051] Some products interact with pets by remotely controlling electronic devices at a distance. However, the above solutions are expensive, difficult to find after the pet moves around, and also have single functions and low sense of participation.
[0052] Some products use smart wearable gloves or sensors (such as Google's Soli sensor) to capture gestures and interact with virtual pets through gestures. However, the above solutions require additional equipment, which is also expensive and difficult to be accepted by users.
[0053] Based on the above problems, the present application provides an interactive method and related devices. Through some embodiments of the present application, an electronic device can control a virtual hand model and a virtual pet according to the user's gestures, and effectively realize the interaction between the user and the virtual pet through the virtual hand model. The interactive method shown in the present application can be performed by an electronic device, and the electronic device can be called an interactive device. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a desktop computer, and other devices.
[0054] See also Figure 1 , Figure 1 is a flow chart of an interactive method provided in an embodiment of the present application. Figure 1 As shown, the interactive method includes:
[0055] 101: In response to a first user operation, the electronic device displays a first interface, where the first interface includes a first virtual pet and a first virtual hand model.
[0056] Exemplarily, the first user operation can be understood as a user operation such as a click operation, a slide operation, and a long press operation, and the first user operation is used to trigger the electronic device to display the first interface. If the electronic device may include a display screen, the electronic device receives the first user operation through the display screen, and in response to the first user operation, the display screen displays the first interface.
[0057] In an embodiment of the present application, the first virtual pet can be understood as a two-dimensional (2D) or three-dimensional (3D) pet model generated by an electronic device using computer technology. Specifically, the type of the first virtual pet can be a kitten, a puppy, a bird, etc., which is not limited in the present application. The first virtual hand model can be understood as a 2D or 3D hand model generated by an electronic device using computer technology. Exemplarily, the first virtual pet is a 3D model, and the first virtual hand model is also a 3D model; or, the first virtual pet is a 3D model, and the first virtual hand model is a 2D model, which is not limited in the present application.
[0058] For example, see Figure 2 , Figure 2 This is a schematic diagram of an interface for displaying a first virtual pet and a first virtual hand model provided in an embodiment of the present application.
[0059] Optionally, the initial position of the first virtual hand model corresponds to any of the following parts of the first virtual pet: the head of the first virtual pet, the tail of the first virtual pet, and the feet of the first virtual pet.
[0060] like Figure 2 As shown, in response to a click operation on the "Pet House" application, the electronic device jumps from interface 21 to interface 22. The click operation on the "Pet House" application can be understood as the first user operation, and interface 22 can be understood as the first interface. Interface 22 displays a virtual cat model 222 and a virtual hand model 221. The virtual cat model 222 can be understood as the first virtual pet, and the virtual hand model 221 can be understood as the first virtual hand model. In addition, Figure 2 The virtual hand model 221 in the figure may correspond to the head of the virtual cat model 222 , that is, the virtual hand model 221 is located above the head of the virtual cat model 222 .
[0061] 102: The electronic device acquires one or more image frames.
[0062] In an embodiment of the present application, the one or more frames of images may be images acquired by an electronic device in real time; they may also be images stored in a memory of an electronic device; they may also be images received from other electronic devices, and the other electronic devices may be mobile phones, tablet computers, and servers, etc. The embodiment of the present application does not limit how the electronic device obtains the one or more frames of images. For example, when an electronic device obtains one frame of image, the electronic device may obtain a photo or a picture, etc. For another example, when an electronic device obtains multiple frames of images, the electronic device may obtain multiple photos or multiple pictures, or the electronic device may obtain a video stream corresponding to the multiple frames of images, that is, the multiple frames of images may include a video.
[0063] Exemplarily, the electronic device may include a camera, which collects video in real time and obtains one or more frames of images from the video collected in real time.
[0064] Exemplarily, the electronic device may include a memory, which can store images or videos in application software such as a camera, social software, or game software, and the electronic device directly obtains the one or more frames of images from the memory.
[0065] 103: The electronic device recognizes a first gesture action according to the one or more frames of images.
[0066] In some embodiments, when the electronic device acquires a frame of image, the electronic device identifies a plurality of gesture key points corresponding to the frame of image from the frame of image; the plurality of gesture key points corresponding to the frame of image correspond to finger positions and wrist positions of the hand;
[0067] The electronic device recognizes the first gesture action according to the gesture key points corresponding to the one frame of image.
[0068] For example, see Figure 3 , Figure 3 This is a schematic diagram of recognizing a first gesture action provided in an embodiment of the present application.
[0069] Figure 3 The portion 301 in the figure can be understood as the above-mentioned frame of image, and the image 301 includes a palm.
[0070] According to the joint positions of the palm, for example, a normal palm has 5 fingers and 14 joints, electronic devices can use multiple key points to represent a palm. Figure 3 In part 302 of the image 301, the electronic device recognizes 21 gesture key points, which are numbered from 0 to 20, and the 21 gesture key points correspond to the palm in the image 301. Among them, the gesture key point numbered 0 represents the point closest to the wrist; the gesture key points numbered 1 to 4 represent the points corresponding to the thumb of the palm; the gesture key points numbered 5 to 8 represent the points corresponding to the index finger of the palm; the gesture key points numbered 9 to 12 represent the points corresponding to the middle finger of the palm; the gesture key points numbered 13 to 16 represent the points corresponding to the ring finger of the palm; and the gesture key points numbered 17 to 20 represent the points corresponding to the little finger of the palm.
[0071] Optionally, the electronic device may record the position information of 21 gesture key points. For example, the electronic device may use the gesture key point closest to the wrist as a reference point, such as Figure 3 The gesture key point numbered 0 in the image is then recorded with respect to the position information of other gesture key points relative to the reference point.
[0072] In some embodiments, the electronic device can recognize gestures by calculating the angle between the vectors composed of the key points of the hand of the fingers. For example, the gesture key points numbered 0 to 2 are used as the first vector, and the gesture key points numbered 2 to 4 are used as the second vector, and the first angle between the first vector and the second vector is calculated. When the first angle is greater than or equal to the first threshold, the electronic device recognizes that the thumb of the palm in image 301 is in a bending motion; when the first angle is less than the first threshold, the electronic device recognizes that the thumb of the palm in image 301 is in an extended motion. It is understandable that the actions of other fingers can be determined by similar methods, which will not be repeated here.
[0073] It can be understood that the above-mentioned first threshold is an angle value, which can be 30 degrees, 35 degrees, etc., and this application does not limit this.
[0074] In other embodiments, the electronic device can calculate the angle between the vectors formed by the key points of the hand of the finger, and directly use the angle as the movement of the finger. For example, the gesture key points numbered 0 to 2 are used as the first vector, and the gesture key points numbered 2 to 4 are used as the second vector. If the angle between the first vector and the second vector is calculated to be 30 degrees, then the bending degree of the thumb is 35 degrees.
[0075] After determining the movement of each finger, the electronic device can obtain the gesture movement of the palm in the image 301 by combining the movements of each finger. Figure 3 In part 303 , the electronic device determines the gesture action of the palm in image 301 through the relative positions of each gesture key point.
[0076] In some embodiments, when the electronic device acquires multiple frames of images, the electronic device identifies multiple gesture key points corresponding to the first image from a first image, where the first image is one frame of the multiple frames of images; the multiple gesture key points corresponding to the first image correspond to finger positions and wrist positions of the hand;
[0077] The electronic device recognizes the first gesture action according to the gesture key points corresponding to each frame of the multiple frames of images; the positions of the multiple gesture key points corresponding to at least two frames of the multiple frames of images are different.
[0078] For example, see Figure 4 , Figure 4 is another schematic diagram of recognizing a first gesture action provided by an embodiment of the present application. Figure 4 The 401 part in the figure can be understood as a multi-frame image. The 4011 part can be understood as the first image, that is, the image 4011 is a frame image in the multi-frame image 401. Figure 4 As shown, the electronic device processes each frame of the image (such as Figure 4 The first frame image, the second frame image to the Nth frame image) are recognized to obtain the gesture key points corresponding to each frame image, and then the gesture action of each frame image is recognized. For details, please refer to Figure 3 The description of the corresponding embodiments is not repeated here. Afterwards, the electronic device can calculate the position change matrix of the gesture key points in two adjacent frames of images, calculate the distance change matrix of the gesture key points based on the position coordinate change matrix of the gesture key points, and connect the distance change matrices calculated from multiple adjacent frames of images in series as the features of the multi-frame image 401. Finally, the electronic device can use a classifier to classify the multi-frame image 401 and identify the gesture actions corresponding to the multi-frame image 401.
[0079] In the embodiment of the present application, the gesture action is determined by using a distance variation matrix corresponding to gesture key points of adjacent images in multiple frames of images, which can improve the accuracy of gesture recognition and thus enhance the user experience.
[0080] 104: The electronic device displays the first virtual hand model as the first gesture action, and displays the first virtual pet as a first interaction action, where the first interaction action and the first gesture action are used for the first virtual pet to interact with the first virtual hand model.
[0081] After the electronic device recognizes the first gesture, the first virtual hand model is displayed as the first gesture. Accordingly, the first virtual pet interacts with the first virtual hand model with the first interactive action.
[0082] In some embodiments, the first gesture action includes any one or more of the following: waving, stroking, patting, and clenching a fist; the first interactive action includes any one or more of the following: shaking head, wagging tail, sticking out tongue, waving, and lying down.
[0083] For example, when the electronic device recognizes that the first gesture action is “stroking”, the electronic device may Figure 2 The hand model 221 in the figure is displayed as a "stroking" state; the action of the virtual cat model 222 is set to be a tail wagging. Alternatively, when the electronic device recognizes that the first gesture action is "patting", the electronic device can Figure 2 The hand model 221 in the video is displayed as a “flapping” state; the action of the virtual cat model 222 is set to “cat claw retraction” to achieve interaction between the hand model 221 and the virtual cat model 222.
[0084] In summary, through the interactive method provided in the embodiment of the present application, the electronic device can recognize the user's gestures to control the virtual hand model and the virtual pet without connecting to other electronic devices, thereby effectively realizing the interaction between the user and the virtual pet through the virtual hand model. In addition, since no additional devices need to be connected, the application scenarios are wider, the operation is simpler, and the cost is lower.
[0085] In some embodiments, Figure 2 The virtual hand model 221 and the virtual cat model 222 shown in the figure respond to a second user operation of the user on the virtual hand model 221, such as a sliding operation, a clicking operation, etc. The electronic device moves the position of the virtual hand model 221 according to the second user operation, so that the interaction between the virtual cat model 222 and the virtual hand model 221 is more realistic, thereby enhancing the user experience.
[0086] In some other embodiments, the electronic device displays the first virtual hand model as the first gesture action, including:
[0087] The electronic device displays the first virtual hand model as a gesture action in a database that has the highest matching degree with the first gesture action, and the database includes at least two gesture actions.
[0088] Specifically, the electronic device stores common gestures in a database, such as stroking, waving, and clenching fists. After the electronic device recognizes the first gesture through the one or more frames of images, the electronic device calculates the similarity between the first gesture and the gestures in the database, and displays the first virtual hand model as the gesture in the database with the highest matching degree with the first gesture, so that the movements displayed by the first virtual hand model are more flexible and realistic, thereby improving the user experience.
[0089] In some further embodiments, each gesture action in the database corresponds to one or more interaction actions; and the first interaction action is one of the one or more interaction actions.
[0090] Specifically, the gesture actions stored in the database by the electronic device correspond to one or more interactive actions. It is understandable that the gesture actions and interactive actions can be matched based on experience. For example, the gesture action is "stroking", and the corresponding interactive actions can be "wagging the tail", "squinting" or "slowly shaking the head", etc. The gesture action is "patting", and the corresponding interactive actions can be "cat claw retraction", "crying" or "backing off", etc., so that the interaction between the first virtual hand model and the first virtual pet is more flexible and realistic, thereby improving the user experience.
[0091] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.
[0092] See also Figure 5 , Figure 5 is a schematic diagram of the structure of an interactive device provided in an embodiment of the present application. The interactive device 50 is used to execute the above-mentioned interactive method (such as Figure 1 The method shown in the figure) is included in the protection scope of the present application as long as the device can implement the interactive method provided by the present application. For example, the interactive device 50 can be a mobile phone, a desktop computer, a portable notebook, etc., which is not limited in the present embodiment. Figure 5 As shown, the interactive device 50 includes a response unit 501, a display unit 502, an acquisition unit 503 and an identification unit 504. The description of each unit is as follows:
[0093] The response unit 501 is used to respond to a first user operation; and control the display unit 502 to display a first interface, the first interface including a first virtual pet and a first virtual hand model;
[0094] An acquisition unit 503 is used to acquire one or more frames of images;
[0095] The recognition unit 504 is used to recognize a first gesture action according to the one or more frames of images; and control the display unit 502 to display the first virtual hand model as the first gesture action, and to display the first virtual pet as a first interactive action, wherein the first interactive action and the first gesture action are used for the first virtual pet to interact with the first virtual hand model.
[0096] Optionally, the recognition unit 504 is specifically configured to, when acquiring a frame of image, recognize a plurality of gesture key points corresponding to the frame of image from the frame of image; the plurality of gesture key points corresponding to the frame of image correspond to finger positions and wrist positions of the hand;
[0097] The recognition unit 504 is specifically configured to recognize the first gesture action according to the gesture key points corresponding to the one frame of image.
[0098] Optionally, the recognition unit 504 is specifically configured to recognize, from a first image, a plurality of gesture key points corresponding to the first image when acquiring multiple frames of images, the first image being one frame of the multiple frames of images; the plurality of gesture key points corresponding to the first image correspond to finger positions and wrist positions of the hand;
[0099] The recognition unit 504 is specifically configured to recognize the first gesture action according to the gesture key points corresponding to each frame of the multiple frames of images; the positions of the multiple gesture key points corresponding to at least two frames of the multiple frames of images are different.
[0100] Optionally, the multiple gesture key points include 21 gesture key points, gesture key point 0 represents the point closest to the wrist; gesture key point 1 to gesture key point 4 represent points corresponding to the thumb of the palm; gesture key point 5 to gesture key point 8 represent points corresponding to the index finger of the palm; gesture key point 9 to gesture key point 12 represent points corresponding to the middle finger of the palm; gesture key point 13 to gesture key point 16 represent points corresponding to the ring finger of the palm; gesture key point 17 to gesture key point 20 represent points corresponding to the little finger of the palm.
[0101] Optionally, the initial position of the first virtual hand model corresponds to any of the following parts of the first virtual pet: the head of the first virtual pet, the tail of the first virtual pet, and the feet of the first virtual pet.
[0102] Optionally, the display unit 502 is specifically configured to display the first virtual hand model as a gesture action in a database that has the highest matching degree with the first gesture action, and the database includes at least two gesture actions.
[0103] Optionally, the first gesture action includes any one or more of the following: waving, stroking, patting, and clenching a fist; the first interactive action includes any one or more of the following: shaking head, wagging tail, sticking out tongue, waving, and lying down.
[0104] See also Figure 6 , Figure 6 60 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 60 can be used to implement the above-mentioned interaction method. Exemplarily, the electronic device 60 can be a mobile phone, a desktop computer, a portable notebook or other devices.
[0105] like Figure 6 As shown, the electronic device 60 includes a memory 601 and a processor 602. Optionally, the electronic device 60 may also include a communication interface 603 and a bus 604; further optionally, the electronic device 60 may also include a display screen 605. The memory 601, the processor 602, the communication interface 603 and the display screen 605 are connected to each other through the bus 604.
[0106] The memory 601 is used to provide a storage space, and the storage space can store data such as an operating system and a computer program. The memory 601 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM).
[0107] The processor 602 is a module for performing arithmetic operations and logical operations, and may be a combination of one or more of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), or a microprocessor unit (MPU). In addition, a computer program is stored in the memory 601, and the processor 602 may call the computer program stored in the memory 601 to execute the corresponding method.
[0108] The display screen 605 is used to implement the display function of the electronic device 60. For example, the display screen 605 can be used to display pictures or videos.
[0109] In the present application embodiment, Figure 6 The electronic device 60 shown has a display screen 605, which is used to receive a first user operation;
[0110] The processor 602 is used to respond to the first user operation and control the display screen 605 to display a first interface, where the first interface includes a first virtual pet and a first virtual hand model;
[0111] The processor 602 is further configured to acquire one or more frames of images;
[0112] The processor 602 is also used to recognize a first gesture action based on the one or more frames of images; and control the display screen 605 to display the first virtual hand model as the first gesture action, and to display the first virtual pet as a first interactive action, wherein the first interactive action and the first gesture action are used for the first virtual pet and the first virtual hand model to interact.
[0113] It can be understood that the processor 602 can be used to implement the functions of the response unit 501, the acquisition unit 503 and the identification unit 504 in the interactive device 50, and the display screen 605 can be used to implement the function of the display unit 502 in the interactive device 50.
[0114] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed on one or more processors, Figures 1 to 4 The method shown.
[0115] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the above claims.
Claims
1. An interactive method, characterized in that: The method comprises: In response to a first user operation, the electronic device displays a first interface, wherein the first interface includes a first virtual pet and a first virtual hand model; The electronic device acquires one or more frames of images, and recognizes a first gesture action according to the one or more frames of images; The electronic device displays the first virtual hand model as a gesture action in a database that has the highest matching degree with the first gesture action, wherein the database includes at least two gesture actions, and displays the first virtual pet as a first interaction action, wherein the first interaction action and the first gesture action are used for the first virtual pet to interact with the first virtual hand model; Wherein, when the electronic device acquires multiple frames of images, the electronic device recognizes a first gesture action according to the one or more frames of images, including: The electronic device recognizes each frame of image to obtain a gesture key point corresponding to each frame of image, wherein the gesture key point corresponding to each frame of image corresponds to a finger position and a wrist position of a hand; The electronic device calculates a position change matrix of gesture key points in two adjacent frames of images, calculates a distance change matrix of the gesture key points based on the position change matrix, connects the distance change matrices calculated from multiple adjacent frames of images in series as features of the multiple frames of images, and classifies the multiple frames of images using a classifier based on the features to identify a first gesture action corresponding to the multiple frames of images.
2. The method according to claim 1, characterized in that: In the case where the electronic device acquires a frame of image, the electronic device recognizes a first gesture action according to the frame or frames of image, including: The electronic device recognizes a plurality of gesture key points corresponding to the frame of image from the frame of image; the plurality of gesture key points corresponding to the frame of image correspond to finger positions and wrist positions of the hand; The electronic device recognizes the first gesture action according to the gesture key points corresponding to the frame of image.
3. The method according to claim 1, characterized in that At least two frames of images in the multiple frames of images correspond to different positions of gesture key points.
4. The method according to claim 2 or 3, characterized in that: The multiple gesture key points include 21 gesture key points, gesture key point 0 represents the point closest to the wrist; gesture key point 1 to gesture key point 4 represent points corresponding to the thumb of the palm; gesture key point 5 to gesture key point 8 represent points corresponding to the index finger of the palm; gesture key point 9 to gesture key point 12 represent points corresponding to the middle finger of the palm; gesture key point 13 to gesture key point 16 represent points corresponding to the ring finger of the palm; gesture key point 17 to gesture key point 20 represent points corresponding to the little finger of the palm.
5. The method according to claim 4, characterized in that The initial position of the first virtual hand model corresponds to any one of the following parts of the first virtual pet: the head of the first virtual pet, the tail of the first virtual pet, and the feet of the first virtual pet.
6. The method according to claim 1, characterized in that The first gesture action includes any one or more of the following: waving, stroking, patting, and clenching a fist; the first interactive action includes any one or more of the following: shaking head, wagging tail, sticking out tongue, waving, and lying down.
7. An interactive device, characterized in that: The device comprises a unit for executing the method according to any one of claims 1 to 6, the device comprising: A response unit, configured to respond to a first user operation; and control the display unit to display a first interface, wherein the first interface includes a first virtual pet and a first virtual hand model; An acquisition unit, used for acquiring one or more frames of images; The recognition unit is used to recognize a first gesture action according to the one or more frames of images; and control the display unit to display the first virtual hand model as the first gesture action, and to display the first virtual pet as a first interactive action, wherein the first interactive action and the first gesture action are used for the first virtual pet to interact with the first virtual hand model.
8. An electronic device, characterized in that: include: A processor and a memory, wherein a computer program is stored in the memory, and the processor calls the computer program stored in the memory to execute the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on one or more processors, the method according to any one of claims 1 to 6 is executed.
Citation Information
Patent Citations
Spatial interaction in augmented reality
CN107077169A
Virtual scene interaction method and device and electronic equipment
CN110568929A
Gesture recognition method and device, computer equipment and storage medium
CN112904994A
Method for controlling virtual pet and intelligent projection equipment
CN113313836A
Method and apparatus for controlling electronic device based on gesture
US20210191611A1
