Gesture interaction method and device, equipment, medium and product

By installing an AI camera in an electronic device, collecting hand images from multiple angles and communicating with the whiteboard program, the problem of frequent movement of the main speaker is solved, and the remote operation of the whiteboard program is realized, and the work efficiency is improved.

CN118982866APending Publication Date: 2024-11-19SHENZHEN HONGHE INNOVATION INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411039512.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In teaching, meeting and other scenarios, the speaker needs to frequently move to the whiteboard program to operate, resulting in reduced work efficiency.

Method used

By installing an AI camera in an electronic device, it collects hand images from multiple angles, performs gesture recognition, and establishes communication with the whiteboard program to achieve remote gesture interaction, avoiding the user's need to move to the device to operate.

Benefits of technology

It improves user's work efficiency and realizes remote operation of whiteboard programs, so that users can operate interactively without personally arriving at the device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982866A_ABST
    Figure CN118982866A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a gesture interaction method and device, equipment, a medium and a product, and relates to the technical field of man-machine interaction. According to the embodiment of the invention, the AI camera is installed in the electronic equipment, and the communication between the AI camera and the whiteboard program is established, so that the AI camera can perform gesture recognition on the hand images of at least two angles collected in the shooting area to obtain the target gesture and the target instruction corresponding to the target gesture; and the target instruction is sent to the whiteboard program, so that the whiteboard program can execute the corresponding operation based on the target instruction, the response to the remote gesture is realized, a user does not need to move to the front of the electronic equipment, and the working efficiency of the user is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a gesture interaction method, device, equipment, medium and product. Background Art

[0002] In teaching, meeting and other scenarios, whiteboard programs are usually used, and users can interact with the whiteboard program through their hands, stylus, etc.

[0003] Taking offline teaching as an example, the lecturer may interact with the audience during the explanation. When interacting with the audience, the lecturer usually moves to the audience's position to improve the interaction effect. When the lecturer needs to operate the whiteboard program during the interaction with the audience, the lecturer needs to walk in front of the screen and operate the whiteboard program with his hands or stylus, which causes the lecturer to move back and forth, reducing the lecturer's work efficiency. Summary of the invention

[0004] The embodiments of the present application provide a gesture interaction method, device, equipment, medium and product, which can remotely interact with a whiteboard program through gestures, thereby improving the user's work efficiency.

[0005] In a first aspect, an embodiment of the present application provides a gesture interaction method, which is applied to an artificial intelligence AI camera, wherein the AI ​​camera is arranged on an electronic device, a whiteboard program is also installed in the electronic device, and the AI ​​camera communicates with the whiteboard program;

[0006] The method includes:

[0007] Collect hand images from at least two angles within the shooting area of ​​the AI ​​camera;

[0008] Perform gesture recognition on each hand image to obtain the target gesture;

[0009] Matching the target gesture with the instruction table to obtain the target instruction corresponding to the target gesture, the instruction table is used to store the corresponding relationship between the gesture and the instruction;

[0010] The target instruction is sent to the whiteboard program, so that the whiteboard program responds to the target instruction and performs an operation corresponding to the target gesture.

[0011] In a second aspect, an embodiment of the present application provides a gesture interaction device, which is applied to an artificial intelligence AI camera, wherein the AI ​​camera is arranged on an electronic device, and a whiteboard program is also installed in the electronic device, and the AI ​​camera communicates with the whiteboard program;

[0012] The device includes:

[0013] A collection module, used to collect hand images at at least two angles within the shooting area of ​​the AI ​​camera;

[0014] A recognition module is used to perform gesture recognition on each hand image to obtain a target gesture;

[0015] A matching module, used to match the target gesture with the instruction table to obtain a target instruction corresponding to the target gesture, and the instruction table is used to store the corresponding relationship between the gesture and the instruction;

[0016] The sending module is used to send the target instruction to the whiteboard program, so that the whiteboard program responds to the target instruction and performs an operation corresponding to the target gesture.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0018] processor;

[0019] a memory for storing computer program instructions;

[0020] When the computer program instructions are executed by the processor, the method according to the first aspect is implemented.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0022] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0023] The embodiment of the present application collects hand images at at least two angles within the shooting area of ​​the AI ​​camera; performs gesture recognition on each hand image to obtain a target gesture; matches the target gesture with the instruction table to obtain a target instruction corresponding to the target gesture, and the instruction table is used to store the correspondence between gestures and instructions; and sends the target instruction to the whiteboard program so that the whiteboard program responds to the target instruction and performs an operation corresponding to the target gesture. That is, the present application installs an AI camera in an electronic device and establishes communication between the AI ​​camera and the whiteboard program, so that the AI ​​camera can perform gesture recognition on hand images at at least two angles collected within the shooting area, obtain the target gesture and the target instruction corresponding to the target gesture, and send the target instruction to the whiteboard program, thereby, the whiteboard program can perform the corresponding operation based on the target instruction, realize the response to the remote gesture, and the user does not need to move to the front of the electronic device, thereby improving the user's work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of a scene of a gesture interaction method provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of a gesture interaction method provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of an interface of a whiteboard program provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the interface of another whiteboard program provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the interface of another whiteboard program provided in an embodiment of the present application;

[0030] Figure 6 A flowchart of another gesture interaction method provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of activating a gesture recognition function of an AI camera provided in an embodiment of the present application;

[0032] Figure 8 A structural diagram of a gesture interaction device provided in an embodiment of the present application;

[0033] Fig. 9 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present application and are not configured to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0035] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0036] As mentioned above, currently the interaction between users and whiteboard programs is mainly achieved by touching the screen of the device where the whiteboard program is located. However, in scenarios such as teaching and meetings, the speaker usually interacts with the audience, so that the speaker is away from the electronic device. In this case, if the speaker needs to operate the whiteboard program, he needs to return to the front of the electronic device, causing the speaker to walk back and forth, reducing the speaker's work efficiency.

[0037] To this end, the embodiments of the present application provide a gesture interaction method, apparatus, device, medium and product, which allow users to remotely interact with a whiteboard program, thereby improving user work efficiency.

[0038] The present application installs an AI camera in an electronic device and establishes communication between the AI ​​camera and a whiteboard program, so that the AI ​​camera can perform gesture recognition on hand images collected at least two angles within a shooting area, obtain a target gesture and a target instruction corresponding to the target gesture, and send the target instruction to the whiteboard program. As a result, the whiteboard program can perform corresponding operations based on the target instruction, thereby realizing response to remote gestures without the user having to move to the front of the electronic device, thereby improving the user's work efficiency.

[0039] Figure 1 A scene diagram of a gesture interaction method provided in an embodiment of the present application, the scene may include an electronic device 10, an AI camera 101 is provided in the outer frame of the electronic device 10, and a whiteboard program 102 is also installed in the electronic device 10. For the convenience of description, Figure 1 The whiteboard program 102 is separated from the electronic device 10 . When actually used, the whiteboard program 102 is installed in the electronic device 10 as an application.

[0040] The AI ​​camera 101 communicates with the whiteboard program 102. The AI ​​camera 101 can capture images, recognize gestures in the images, obtain instructions corresponding to the gestures, and send the instructions to the whiteboard program 102, so that the whiteboard program 102 performs corresponding operations based on the instructions. As a result, the user does not need to contact the electronic device 10 to operate the whiteboard program 102, thereby improving the user's work efficiency.

[0041] The following describes the gesture interaction method, device, equipment, medium and product provided in the embodiments of the present application in combination with the above scenarios and specific embodiments.

[0042] Figure 2 A flowchart of a gesture interaction method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the gesture interaction method may include the following steps:

[0043] S210: Collect hand images from at least two angles within the shooting area of ​​the AI ​​camera.

[0044] S220: Perform gesture recognition on each hand image to obtain a target gesture.

[0045] S230: Match the target gesture with the instruction table to obtain a target instruction corresponding to the target gesture.

[0046] The instruction table is used to store the corresponding relationship between gestures and instructions.

[0047] S240: Send the target instruction to the whiteboard program, so that the whiteboard program responds to the target instruction and performs an operation corresponding to the target gesture.

[0048] The embodiment of the present application realizes remote operation of the whiteboard program through the interaction between the AI ​​camera and the whiteboard program, thereby improving the user's work efficiency.

[0049] The above steps are described in detail below:

[0050] In S210, the size of the shooting area is related to the camera. The camera can only capture the user's image when the user is within the shooting area of ​​the camera. When the user is at a viewing angle or distance that the camera cannot detect, the camera cannot capture the user's image. For example, in an embodiment of the present application, when the user or the user's hand is within the shooting area of ​​the AI ​​camera, the AI ​​camera can capture the user's hand image.

[0051] For example, the effective shooting area of ​​the AI ​​camera is 8 meters, that is, when the user is within 8 meters in front of the AI ​​camera, the AI ​​camera can capture the user's hand image. Exemplarily, the AI ​​camera can be a rotatable camera, for example, it can rotate between 0°-360° to adapt to more scenarios.

[0052] The embodiment of the present application can use an AI camera to collect hand images at multiple angles, and use the hand images at multiple angles to recognize gestures, which can improve the accuracy of gesture recognition results, thereby enabling the whiteboard program to respond more accurately.

[0053] In S220, illustratively, the target gesture can be obtained by template matching. For example, the hand image at each angle can be matched with the reference hand image, and the target gesture can be determined based on the matching result. Taking the acquisition of hand images at three angles as an example, if there are at least two hand images that match the reference hand image, and the reference hand images matched by the at least two hand images are the same image, the gesture to which the reference hand image belongs can be determined as the target gesture. If at most only one hand image matches the reference hand image, the AI ​​camera needs to re-acquire the hand image and then re-match it to ensure the accuracy of the recognition result.

[0054] Exemplarily, the target gesture may also be determined by a model. For example, hand images at various angles may be input into a gesture recognition model, and the gesture recognition model outputs a gesture recognition result to obtain the target gesture. The gesture recognition model may be a deep learning model.

[0055] Of course, the target gesture corresponding to the hand image may also be identified by other methods, which are not specifically limited in the embodiments of the present application.

[0056] In some embodiments, if the gesture corresponding to the hand image is not recognized within the preset time, the process may return to S210 and re-collect the hand image. The preset time may be set according to actual needs, for example, 5 seconds.

[0057] In S230, the instruction table is used to store the correspondence between gestures and instructions, and the correspondence between gestures and instructions can be set or updated according to actual needs.

[0058] For example, the "fist" gesture can correspond to the command "competitive", which means turning on the eraser function of the whiteboard program. Another example is that the "palm" gesture can correspond to the command "palms", which means turning on the function of moving the canvas. Another example is that the "scissor hand" gesture can correspond to the command "unanimous", which means turning on the select all function.

[0059] After the target gesture corresponding to the hand image is identified, the target gesture can be matched with the gestures in the instruction table to obtain the target instruction corresponding to the target gesture.

[0060] In S240, after the target instruction is determined, the AI ​​camera can send the target instruction to the whiteboard program. After receiving the target instruction, the whiteboard program can start the corresponding function and return a feedback instruction to the AI ​​camera indicating that the function has been started. After receiving the feedback instruction from the whiteboard program, the AI ​​camera can follow and scan the gesture in real time, and synchronize the data to the whiteboard program in real time, and the whiteboard program responds and performs the corresponding operation. The operation may include at least one of the following: move the cursor left, move the cursor right, move the canvas, write, erase, select, and confirm. In actual application, other operations may also be included, which are not limited in the embodiments of the present application.

[0061] For example, after the AI ​​camera sends a target instruction to the whiteboard program, it can start timing. If the feedback instruction returned by the whiteboard program is not received within the preset time, the AI ​​camera can resend the target instruction to the whiteboard program. After reaching the preset number of times and still not receiving the feedback instruction returned by the whiteboard program, a prompt message can be output to prompt the user that the AI ​​camera and the whiteboard program are communicating abnormally. The preset time here can be set to 2s, for example. The prompt message can be output in the form of text, voice, etc.

[0062] In an embodiment of the present application, for example, the target instruction is "pinc", which represents drawing with a pen tool. The AI ​​camera sends the target instruction to the whiteboard program, and after receiving the feedback instruction from the whiteboard program, the AI ​​camera can follow and scan the gesture in real time, and synchronize the scan data to the whiteboard program in real time. At this time, pen lines drawn synchronously with the user's gesture can appear on the whiteboard canvas.

[0063] For example, Figure 3 As shown, it is assumed that the cursor is located at the position of the second tool button 301 of the bottom toolbar at this time, such as Figure 4 As shown, after a left gesture 401 is made remotely, the AI ​​camera determines that the target instruction corresponding to the gesture is "left", which means moving the cursor to the left. After the AI ​​camera sends the target instruction to the whiteboard program, the whiteboard program can move the cursor of the bottom toolbar to the left. The cursor is now located at the position of the first tool button 402.

[0064] exist Figure 3 On the basis of Figure 5As shown, assuming that the user remotely makes a right gesture 501, the AI ​​camera determines that the target instruction corresponding to the gesture is "right", which means moving the cursor to the right. After the AI ​​camera sends the target instruction to the whiteboard program, the whiteboard program can move the cursor of the bottom toolbar to the right. The cursor is now located at the position of the third tool button 502.

[0065] The process of starting other instructions is similar. For example, when the user remotely makes a writing gesture, the whiteboard program can quickly start the writing function, and then the user can write in the whiteboard program.

[0066] For another example, when a user remotely makes a gesture to move the canvas, the whiteboard program can quickly start the function of moving the canvas. At this time, the user can drag and move the canvas as needed.

[0067] For another example, when the user remotely makes a palm gesture, the whiteboard program can quickly start the eraser function. At this time, the user can erase the handwriting, graphics and other contents on the whiteboard program. Among them, when the AI ​​camera recognizes that the gesture is a palm, it can determine whether it is a palm by identifying the area of ​​the gesture. For example, if it is detected that the width of the gesture is greater than or equal to a1, and the height of the gesture is greater than or equal to a2, it can be determined that the gesture is a palm. Exemplarily, a1=5cm, a2=3cm. For example, if the AI ​​camera detects that the width of a gesture = 18cm≥5cm, and the height = 15cm≥3cm, it can be determined that the gesture is a palm.

[0068] For another example, when the user remotely makes a selection gesture, the whiteboard program can quickly start the selection function. At this time, the user can select the handwriting, graphics and other contents on the whiteboard program.

[0069] This application utilizes the interaction between the AI ​​camera and the whiteboard program so that the whiteboard program can perform corresponding operations based on the target instructions sent by the AI ​​camera, thereby realizing response to remote gestures without the user having to move in front of the electronic device, thereby improving the user's work efficiency.

[0070] In some embodiments, if the whiteboard program receives multiple target instructions, it may respond to them in sequence according to the time of receipt, or it may only respond to the target instruction received first.

[0071] Figure 6 A flowchart of another gesture interaction method provided in an embodiment of the present application, Figure 6 and Figure 2 The difference is that Figure 2 The S220 in can be replaced by Figure 6 S2201-S2203 in.

[0072] In S2201, candidate gestures corresponding to each hand image are determined based on the similarity between the target hand image in each hand image and each reference hand image in the image library.

[0073] The target hand image here may be part or all of each hand image, that is, the embodiment of the present application may determine the candidate gestures corresponding to each hand image based on the similarity between a partial hand image and a reference hand image, or may determine the candidate gestures corresponding to each hand image based on the similarity between the entire hand image and the reference hand image.

[0074] When the target hand image is a partial hand image, illustratively, a partial hand image may be randomly selected from each hand image as the target hand image.

[0075] The image library is used to store at least one reference hand image of at least one angle corresponding to at least one reference gesture. In the embodiment of the present application, the target hand image can be matched with each reference hand image in the image library to obtain the similarity between the target hand image and each reference hand image, and the candidate gesture corresponding to each hand image can be determined based on the similarity. The similarity determination process is not limited in the embodiment of the present application.

[0076] For example, a reference gesture corresponding to a reference hand image having a similarity greater than a similarity threshold may be determined as a candidate gesture.

[0077] In S2202, the candidate gesture is matched with the gestures in the instruction table to obtain the gesture type of the candidate gesture.

[0078] After the candidate gesture is determined, the candidate gesture can be matched with the gestures in the instruction table to determine the gesture type to which the candidate gesture belongs. The instruction table is also used to store the correspondence between gestures and gesture types.

[0079] The gesture types here may include, for example, continuous gestures and discontinuous gestures, wherein a continuous gesture means that the gesture needs to have a continuous motion track, and the duration of the continuous motion track can be set according to actual needs, for example, can be set to 100ms.

[0080] A non-continuous gesture means that the gesture does not need to have a continuous motion trajectory, that is, after the whiteboard program starts the corresponding function, after the AI ​​camera recognizes the gesture corresponding to the function again, the whiteboard program can respond to the gesture and perform the corresponding operation.

[0081] In S2203, the target gesture of each hand image is determined according to the gesture type.

[0082] After the gesture type of the candidate gesture is determined, the target gesture can be determined according to the gesture type. For example, if the gesture type is a discontinuous gesture, the candidate gesture can be directly determined as the target gesture.

[0083] If the gesture type is a continuous gesture, it is necessary to further determine whether the candidate gesture moves continuously within the preset time length. If the candidate gesture moves continuously within the preset time length, the candidate gesture can be determined as the target gesture. If the candidate gesture does not move continuously within the preset time length, it is necessary to re-time and determine whether the candidate gesture moves continuously within the preset time length. The preset time length here can be set to 1s.

[0084] The embodiment of the present application can first determine the candidate gestures according to the similarity between the target hand image and the reference hand image, then determine the gesture type based on the candidate gestures, and then determine the target gesture according to the gesture type, thereby improving the accuracy of the gesture recognition results and enabling the whiteboard program to respond more accurately to meet the needs of users.

[0085] In some embodiments, the above S2201 may include the following steps:

[0086] Selecting a hand image from each hand image as a first hand image, and determining a first similarity between the first hand image and each reference image in the image library;

[0087] When the maximum similarity among the first similarities is greater than or equal to the similarity threshold, the first hand image is determined as the target hand image, and the reference gesture to which the reference hand image corresponding to the maximum similarity belongs is determined as the candidate gesture of each hand image;

[0088] When the maximum similarity among the first similarities is less than the similarity threshold, each hand image is determined as a target hand image; and based on the similarity between each target hand image and each reference hand image in the image library, the candidate gesture corresponding to each hand image is determined.

[0089] Exemplarily, a hand image may be randomly selected from each hand image as the first hand image, and a candidate gesture may be determined according to a first similarity between the first hand image and each reference hand image.

[0090] For example, the maximum first similarity can be determined from the first similarities. If the maximum first similarity is greater than or equal to the similarity threshold, the reference gesture corresponding to the reference hand image corresponding to the maximum first similarity can be directly determined as the candidate gesture without calculating the similarity between other hand images and the reference hand image, which simplifies the operation and improves the response speed of the gesture.

[0091] If the maximum first similarity is still smaller than the similarity threshold, it is necessary to continue to determine the similarities between the remaining hand images and the reference hand images, and determine the candidate gestures based on the similarities of all the hand images.

[0092] In this case, illustratively, for each target hand image, the maximum similarity between the target hand image and each reference hand image may be determined;

[0093] When the reference hand images corresponding to the maximum similarities are the same, determining a weighted sum of the maximum similarities;

[0094] When the weighted sum is greater than or equal to the similarity threshold, the reference gesture to which the same reference hand image belongs is determined as a candidate gesture of each hand image;

[0095] When the weighted sum is less than the similarity threshold, the operation of collecting the hand image is returned.

[0096] Exemplarily, after determining the maximum similarities of each hand image, it can be further determined whether the reference hand images corresponding to each maximum similarity belong to the same reference gesture. If they belong to the same reference gesture, the candidate gestures can be determined based on each maximum similarity.

[0097] For example, the weighted sum of each maximum similarity can be calculated. The weight corresponding to each maximum similarity can be determined according to the scenario, requirements, etc., or can be determined according to the positional relationship between the AI ​​camera and the hand image. For example, the hand image facing the AI ​​camera has the largest weight, and the weights of the other hand images are smaller. The sum of the weights of each maximum similarity is 1.

[0098] Exemplarily, if the weighted sum is greater than or equal to the similarity threshold, the reference gesture to which the same reference hand image belongs can be determined as the candidate gesture of each hand image; otherwise, the operation of capturing hand images is returned to recapture the hand images, and the above process is re-executed.

[0099] The embodiment of the present application can first randomly select a first hand image from each hand image. If the maximum similarity between the first hand image and each reference hand image is greater than or equal to the similarity threshold, the candidate gesture can be determined directly based on the first hand image. Otherwise, the candidate gesture can be further determined in combination with the similarity between other hand images and the reference hand images, thereby improving the accuracy of the candidate gesture determination results and further improving the accuracy of the response.

[0100] Taking the gesture type including a continuous gesture as an example, illustratively, the above S2203 may include the following steps:

[0101] Determine whether the motion trajectory of the candidate gesture is continuous within a preset time length;

[0102] When it is determined that the movement trajectory of the candidate gesture is continuous within a preset time period, the candidate gesture is determined as a target gesture;

[0103] If it is determined that the motion trajectory of the candidate gesture within the preset time length is discontinuous, the operation returns to executing the operation of determining whether the motion trajectory of the candidate gesture within the preset time length is continuous.

[0104] Exemplarily, when it is determined that the gesture type of the candidate gesture is a continuous gesture, it can be further determined whether the motion trajectory of the candidate gesture is continuous within a preset time length. If it is continuous, the candidate gesture can be determined as the target gesture. Otherwise, it is necessary to continue to determine whether the motion trajectory of the candidate gesture is continuous within the preset time length.

[0105] That is, for continuous gestures, the embodiment of the present application will only determine the candidate gesture as the target gesture when it is determined that its motion trajectory is continuous within a preset time length, and send corresponding instructions to the whiteboard program. This avoids the occurrence of false touches and improves the accuracy of gesture recognition results.

[0106] In some embodiments, before S210, the gesture interaction method may further include the following steps:

[0107] Determine whether there is a preset gesture within the shooting area of ​​the AI ​​camera;

[0108] When there is a preset gesture in the shooting area and the gesture state corresponding to the preset gesture meets the activation conditions, the gesture recognition function of the AI ​​camera is activated.

[0109] The preset gesture is used to activate the gesture recognition function of the AI ​​camera. In the embodiment of the present application, the AI ​​camera can perform the above gesture recognition operation only after the gesture recognition function of the AI ​​camera is activated. Otherwise, it can be used as an ordinary camera and only used to collect images. Thus, it can meet the needs of more scenarios. The preset gesture can be set according to the needs.

[0110] The gesture state satisfies the activation condition, which means that the gesture state remains unchanged within a preset time period, or the gesture state changes as required within a preset time period.

[0111] Figure 7 A schematic diagram of activating the gesture recognition function of an AI camera is exemplified, such as Figure 7 As shown, there are four gesture states, corresponding to state 701, state 702, state 703 and state 704 respectively. State 701, state 702, state 703 and state 704 may be the same or different. Figure 7 Take the case where all states are the same as an example.

[0112] If the gesture state changes from state 701 to state 704 within the preset time, it can be determined that the gesture recognition function of the AI ​​camera is activated successfully, and the subsequent gesture recognition process can be executed. If the gesture state does not change from state 701 to state 704 within the preset time, or the state is interrupted, it is necessary to re-detect state 701 to state 704. The preset time here can be set to 500ms, for example.

[0113] That is, the embodiment of the present application will only activate the gesture recognition function of the AI ​​camera in combination with the change of the gesture state of the preset gesture when the gesture recognition function of the AI ​​camera is needed. This can reduce the power consumption of the AI ​​camera and enable the AI ​​camera to adapt to more scenarios and meet more user needs.

[0114] The embodiment of the present application utilizes an AI camera to capture the user's remote gestures, and through interaction with the whiteboard program, the whiteboard program can remotely respond to the gestures and perform corresponding operations without the user having to move to the front of the electronic device, thereby greatly improving the user's work efficiency.

[0115] Based on the same inventive concept, the present application embodiment also provides a gesture interaction device. Figure 8 The gesture interaction device provided in the embodiment of the present application is described in detail.

[0116] Figure 8 This is a structural diagram of a gesture interaction device provided in an embodiment of the present application. Figure 8 As shown, the gesture interaction device 800 may include:

[0117] The acquisition module 801 is used to acquire hand images at at least two angles within the shooting area of ​​the AI ​​camera;

[0118] The recognition module 802 is used to perform gesture recognition on each hand image to obtain a target gesture;

[0119] A matching module 803 is used to match the target gesture with the instruction table to obtain a target instruction corresponding to the target gesture, and the instruction table is used to store the corresponding relationship between the gesture and the instruction;

[0120] The sending module 804 is used to send the target instruction to the whiteboard program, so that the whiteboard program responds to the target instruction and performs an operation corresponding to the target gesture.

[0121] The embodiment of the present application collects hand images at at least two angles within the shooting area of ​​the AI ​​camera; performs gesture recognition on each hand image to obtain a target gesture; matches the target gesture with the instruction table to obtain a target instruction corresponding to the target gesture, and the instruction table is used to store the correspondence between gestures and instructions; and sends the target instruction to the whiteboard program so that the whiteboard program responds to the target instruction and performs an operation corresponding to the target gesture. That is, the present application installs an AI camera in an electronic device and establishes communication between the AI ​​camera and the whiteboard program, so that the AI ​​camera can perform gesture recognition on hand images at at least two angles collected within the shooting area, obtain the target gesture and the target instruction corresponding to the target gesture, and send the target instruction to the whiteboard program, thereby, the whiteboard program can perform the corresponding operation based on the target instruction, realize the response to the remote gesture, and the user does not need to move to the front of the electronic device, thereby improving the user's work efficiency.

[0122] In some embodiments, the identification module 802 is specifically configured to:

[0123] Determining a candidate gesture corresponding to each hand image according to a similarity between a target hand image in each hand image and each reference hand image in an image library, wherein the image library is used to store a reference hand image of at least one angle corresponding to at least one reference gesture;

[0124] Matching the candidate gesture with the gestures in the instruction table to obtain the gesture type of the candidate gesture, the instruction table is also used to store the corresponding relationship between the gesture and the gesture type;

[0125] According to the gesture type, the target gesture of each hand image is determined.

[0126] In some embodiments, the identification module 802 is specifically configured to:

[0127] Selecting a hand image from each hand image as a first hand image, and determining a first similarity between the first hand image and each reference image in the image library;

[0128] When the maximum similarity among the first similarities is greater than or equal to the similarity threshold, the first hand image is determined as the target hand image, and the reference gesture to which the reference hand image corresponding to the maximum similarity belongs is determined as the candidate gesture of each hand image;

[0129] When the maximum similarity among the first similarities is less than the similarity threshold, each hand image is determined as a target hand image; and based on the similarity between each target hand image and each reference hand image in the image library, the candidate gesture corresponding to each hand image is determined.

[0130] In some embodiments, the identification module 802 is specifically configured to:

[0131] For each target hand image, determining the maximum similarity between the target hand image and each reference hand image;

[0132] When the reference hand images corresponding to the maximum similarities are the same, determining a weighted sum of the maximum similarities;

[0133] When the weighted sum is greater than or equal to the similarity threshold, the reference gesture to which the same reference hand image belongs is determined as a candidate gesture of each hand image;

[0134] When the weighted sum is less than the similarity threshold, the operation of collecting the hand image is returned.

[0135] In some embodiments, the gesture type includes a continuous gesture;

[0136] The identification module 802 is specifically used for:

[0137] Determine whether the motion trajectory of the candidate gesture is continuous within a preset time length;

[0138] When it is determined that the movement trajectory of the candidate gesture is continuous within a preset time period, the candidate gesture is determined as a target gesture;

[0139] If it is determined that the motion trajectory of the candidate gesture within the preset time length is discontinuous, the operation returns to executing the operation of determining whether the motion trajectory of the candidate gesture within the preset time length is continuous.

[0140] In some embodiments, the gesture interaction device 800 may further include:

[0141] A judgment module, used for judging whether there is a preset gesture in the shooting area of ​​the AI ​​camera before the acquisition module 801 acquires hand images of at least two angles in the shooting area of ​​the AI ​​camera;

[0142] The activation module is used to activate the gesture recognition function of the AI ​​camera when there is a preset gesture in the shooting area and the gesture state corresponding to the preset gesture meets the activation condition.

[0143] In some embodiments, the operation corresponding to the target gesture includes at least one of the following:

[0144] Move cursor left, move cursor right, move canvas, write, erase, select, confirm.

[0145] Figure 8 Each module in the device shown has the function of realizing Figure 2 and Figure 6 The functions of each step in the process can achieve the corresponding technical effects, so for the sake of brevity, they will not be described in detail here.

[0146] Based on the same inventive concept, the embodiment of the present application further provides an electronic device, which may be a large-screen device. Fig. 9 The electronic device provided in the embodiments of the present application is described in detail.

[0147] like Fig. 9 As shown, the electronic device 900 may include a processor 910 and a memory 920 for storing computer program instructions.

[0148] The processor 910 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0149] The memory 920 may include a large capacity memory for data or instructions. For example, but not limitation, the memory 920 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 920 may include a removable or non-removable (or fixed) medium, or the memory 920 is a non-volatile solid-state memory. In one example, the memory 920 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0150] The processor 910 reads and executes the computer program instructions stored in the memory 920 to implement Figure 2 and Figure 6 The method in the embodiment shown in the figure is achieved Figure 2 and Figure 6 The corresponding technical effects achieved by executing the method in the illustrated embodiment are not described in detail here for the sake of brevity.

[0151] In one example, the electronic device 900 may further include a communication interface 930 and a bus 940. Fig. 9 As shown, the processor 910, the memory 920, and the communication interface 930 are connected via a bus 940 and communicate with each other.

[0152] The communication interface 930 is mainly used to implement communication between various modules, devices and / or equipment in the embodiments of the present application.

[0153] Bus 940 includes hardware, software or both, and each component of electronic device 900 is coupled to each other.For example, but not limitation, bus 940 may include accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, front side bus (Front Side Bus, FSB), hypertransmission (Hyper Transport, HT) interconnection, industry standard architecture (IndustryStandard Architecture, ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 940 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0154] The electronic device can execute the gesture interaction method in the embodiment of the present application after collecting hand images from at least two angles within the shooting area of ​​the AI ​​camera, thereby realizing the combination of Figure 2 and Figure 6 The gesture interaction method described and Figure 8 Describe the gesture interaction device.

[0155] In addition, in combination with the gesture interaction method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the gesture interaction methods in the above embodiment is implemented.

[0156] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which is executed by at least one processor to implement the various processes of the above-mentioned gesture interaction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0157] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A gesture interaction method, It is characterized in that Applied to an artificial intelligence AI camera, the AI ​​camera is arranged on an electronic device, a whiteboard program is also installed in the electronic device, and the AI ​​camera communicates with the whiteboard program; The method comprises: Collect hand images from at least two angles within the shooting area of ​​the AI ​​camera; Performing gesture recognition on each of the hand images to obtain a target gesture; Matching the target gesture with an instruction table to obtain a target instruction corresponding to the target gesture, wherein the instruction table is used to store a correspondence between gestures and instructions; The target instruction is sent to the whiteboard program, so that the whiteboard program responds to the target instruction and performs an operation corresponding to the target gesture.

2. The method according to claim 1, It is characterized in that The performing gesture recognition on each of the hand images to obtain a target gesture includes: Determining a candidate gesture corresponding to each of the hand images according to a similarity between a target hand image in each of the hand images and each of the reference hand images in an image library, wherein the image library is used to store a reference hand image of at least one angle corresponding to at least one reference gesture; Matching the candidate gesture with the gestures in the instruction table to obtain the gesture type of the candidate gesture, wherein the instruction table is also used to store the correspondence between gestures and gesture types; According to the gesture type, a target gesture of each of the hand images is determined.

3. The method according to claim 2, It is characterized in that The step of determining the candidate gestures corresponding to each of the hand images according to the similarity between the target hand image in each of the hand images and each of the reference hand images in the image library includes: Selecting a hand image from each of the hand images as a first hand image, and determining a first similarity between the first hand image and each reference image in the image library; When the maximum similarity among the first similarities is greater than or equal to the similarity threshold, determining the first hand image as the target hand image, and determining the reference gesture to which the reference hand image corresponding to the maximum similarity belongs as the candidate gesture of each of the hand images; When the maximum similarity among the first similarities is less than the similarity threshold, each of the hand images is determined as the target hand image; and based on the similarity between each of the target hand images and each reference hand image in the image library, the candidate gesture corresponding to each of the hand images is determined.

4. The method according to claim 3, It is characterized in that The method further comprises: determining a candidate gesture corresponding to each of the hand images by comparing the similarity between each of the target hand images and each of the reference hand images in the image library, and comprising: For each of the target hand images, determining the maximum similarity between the target hand image and each of the reference hand images; In the case where the reference hand images corresponding to the maximum similarities are the same, determining a weighted sum of the maximum similarities; When the weighted sum is greater than or equal to the similarity threshold, determining the reference gesture to which the same reference hand image belongs as the candidate gesture of each hand image; When the weighted sum is less than the similarity threshold, the operation of acquiring the hand image is returned to execution.

5. The method according to claim 2, It is characterized in that The gesture type includes a continuous gesture; The step of determining a target gesture of each of the hand images according to the gesture type includes: Determining whether the motion trajectory of the candidate gesture is continuous within a preset time period; When it is determined that the movement trajectory of the candidate gesture is continuous within a preset time period, determining the candidate gesture as the target gesture; In the case where it is determined that the motion trajectory of the candidate gesture within the preset time length is discontinuous, the operation of returning to determine whether the motion trajectory of the candidate gesture within the preset time length is continuous is performed.

6. The method according to any one of claims 1 to 5, It is characterized in that Before collecting hand images at least at two angles in the shooting area of ​​the AI ​​camera, the method further includes: Determine whether there is a preset gesture within the shooting area of ​​the AI ​​camera; When the preset gesture exists in the shooting area and the gesture state corresponding to the preset gesture meets the activation condition, the gesture recognition function of the AI ​​camera is activated.

7. The method according to any one of claims 1 to 5, It is characterized in that The operation corresponding to the target gesture includes at least one of the following: Move cursor left, move cursor right, move canvas, write, erase, select, confirm.

8. A gesture interaction device, It is characterized in that Applied to an artificial intelligence AI camera, the AI ​​camera is arranged on an electronic device, a whiteboard program is also installed in the electronic device, and the AI ​​camera communicates with the whiteboard program; The device comprises: A collection module, used to collect hand images at at least two angles within the shooting area of ​​the AI ​​camera; A recognition module, used for performing gesture recognition on each of the hand images to obtain a target gesture; A matching module, used to match the target gesture with an instruction table to obtain a target instruction corresponding to the target gesture, wherein the instruction table is used to store the correspondence between gestures and instructions; The sending module is used to send the target instruction to the whiteboard program, so that the whiteboard program responds to the target instruction and performs an operation corresponding to the target gesture.

9. An electronic device, It is characterized in that include: processor; a memory for storing computer program instructions; When the computer program instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, It is characterized in that The method comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.