Dynamic gesture recognition method, device, equipment and storage medium based on optical flow
Through optical flow technology, the problem of existing methods relying on training data is solved, and high accuracy recognition is achieved without training data, which is suitable for a variety of electronic devices.
Patent Information
- Application Number
- CN202011389813.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-01
- Publication Date
- 2025-01-10
- Estimated Expiration
- 2040-12-01
AI Technical Summary
The existing dynamic gesture recognition methods rely on training data, resulting in a decrease in recognition accuracy and poor user experience when scene changes or when user dynamic gesture styles are not fully collected.
Using a dynamic gesture recognition method based on optical flow, a multi-frame image is obtained, the motion information of optical flow points is determined, the optical flow sparse matrix is generated, and matrix addition is performed to identify dynamic gestures without relying on training data.
Improves the recognition accuracy of dynamic gestures and enhances the user experience, suitable for various electronic devices such as mobile phones, tablets and wearable devices.
Smart Images
Figure CN112464833B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of gesture recognition, and in particular to a method, device, equipment and storage medium for dynamic gesture recognition based on optical flow. Background Art
[0002] At present, people mainly interact with electronic devices such as TVs, fans, smart phones, cameras or computers through mouse, keyboard, touch screen or remote control, but these human-computer interaction methods are not user-friendly and have great limitations, which are not enough to meet people's needs. Hands are one of the most flexible parts of the human body. Gestures are widely used in daily communication and operation. They are convenient and fast. Human-computer interaction with electronic devices can be carried out through dynamic gestures, and the recognition of dynamic gestures is particularly important. At present, dynamic gestures are mainly recognized through trained neural networks, but this recognition method depends on training data. When the scene changes or the user's dynamic gesture style is not fully collected by the training data, it will greatly affect the recognition of dynamic gestures, and the user experience is not good. Summary of the invention
[0003] The embodiments of the present application provide a method, apparatus, device and storage medium for dynamic gesture recognition based on optical flow, aiming to improve the recognition accuracy of dynamic gestures.
[0004] In a first aspect, an embodiment of the present application provides a dynamic gesture recognition method based on optical flow, comprising:
[0005] Acquire multiple frames of images captured by the shooting device, and determine motion information of each optical flow point in the image based on the multiple frames of images;
[0006] Generate an optical flow sparse matrix according to the motion information of each of the optical flow points, and perform matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix;
[0007] determining whether there is a dynamic gesture in the multiple frames of images according to the target optical flow sparse matrix, and identifying a palm image region in the multiple frames of images when there is a dynamic gesture in the multiple frames of images;
[0008] The gesture category to which the dynamic gesture belongs is determined according to the motion information of each optical flow point in the palm image area.
[0009] In a second aspect, an embodiment of the present application further provides a dynamic gesture recognition device based on optical flow, wherein the dynamic gesture recognition device based on optical flow includes an acquisition module, a determination module, a matrix generation module and a recognition module, wherein:
[0010] The acquisition module is used to acquire multiple frames of images captured by the shooting device;
[0011] The determination module is used to determine the motion information of each optical flow point in the image according to the multiple frames of images;
[0012] The matrix generation module is used to generate an optical flow sparse matrix according to the motion information of each optical flow point, and perform matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix;
[0013] The determination module is further used to determine whether there is a dynamic gesture in the multiple frames of images according to the target optical flow sparse matrix;
[0014] When there is a dynamic gesture in the multiple frames of images, identifying a palm image area in the multiple frames of images by the recognition module;
[0015] The determination module is further used to determine the gesture category to which the dynamic gesture belongs according to the motion information of each optical flow point in the palm image area.
[0016] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor, a memory, a camera, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the dynamic gesture recognition method as described above are implemented.
[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the dynamic gesture recognition method as described above are implemented.
[0018] The embodiments of the present application provide a method, device, equipment and storage medium for dynamic gesture recognition based on optical flow, which obtains multiple frames of images captured by a shooting device, and determines the motion information of each optical flow point in the image based on the multiple frames of images, then generates an optical flow sparse matrix based on the motion information of each optical flow point, and performs matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix, and finally determines whether there is a dynamic gesture in the multiple frames of images based on the target optical flow sparse matrix, and when there is a dynamic gesture in the multiple frames of images, identifies the palm image area in the multiple frames of images, and determines the gesture category to which the dynamic gesture belongs based on the motion information of each optical flow point in the palm image area, without relying on training data, and can improve the recognition accuracy of dynamic gestures. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0020] Figure 1 is a flowchart of a dynamic gesture recognition method based on optical flow provided in an embodiment of the present application;
[0021] Figure 2 is a schematic diagram of an initial optical flow sparse matrix in an embodiment of the present application;
[0022] Figure 3 is a schematic diagram of an optical flow sparse matrix in an embodiment of the present application;
[0023] Figure 4 is a schematic diagram of a target optical flow sparse matrix in an embodiment of the present application;
[0024] Figure 5 is a schematic block diagram of a dynamic gesture recognition device based on optical flow provided in an embodiment of the present application;
[0025] Figure 6 It is a schematic block diagram of the structure of an electronic device provided in an embodiment of the present application.
[0026] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0029] At present, people mainly interact with electronic devices such as TVs, fans, smart phones, cameras or computers through mouse, keyboard, touch screen or remote control, but these human-computer interaction methods are not user-friendly and have great limitations, which are not enough to meet people's needs. Hands are one of the most flexible parts of the human body. Gestures are widely used in daily communication and operation. They are convenient and fast. Human-computer interaction with electronic devices can be carried out through dynamic gestures, and the recognition of dynamic gestures is particularly important. At present, dynamic gestures are mainly recognized through trained neural networks, but this recognition method depends on training data. When the scene changes or the user's dynamic gesture style is not fully collected by the training data, it will greatly affect the recognition of dynamic gestures, and the user experience is not good.
[0030] To solve the above problems, an embodiment of the present application provides a dynamic gesture recognition method based on optical flow. The dynamic gesture recognition method can be applied to electronic devices, which may include mobile phones, tablet computers, laptops, desktop computers, personal digital assistants and wearable devices, etc. The electronic device includes a shooting device. The electronic device obtains multiple frames of images captured by the shooting device, and determines the motion information of each optical flow point in the image based on the multiple frames of images, and then generates an optical flow sparse matrix based on the motion information of each optical flow point, and performs matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix. Finally, based on the target optical flow sparse matrix, it is determined whether there is a dynamic gesture in the multiple frames of images, and when there is a dynamic gesture in the multiple frames of images, the palm image area in the multiple frames of images is identified, and the gesture category to which the dynamic gesture belongs is determined based on the motion information of each optical flow point in the palm image area. The recognition accuracy of the dynamic gesture can be improved without relying on training data.
[0031] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0032] See also Figure 1 , Figure 1 It is a flowchart of a dynamic gesture recognition method based on optical flow provided in an embodiment of the present application.
[0033] like Figure 1 As shown, the optical flow-based dynamic gesture recognition method includes steps S101 to S104.
[0034] Step S101: Acquire multiple frames of images captured by the shooting device, and determine motion information of each optical flow point in the image based on the multiple frames of images.
[0035] Exemplarily, the electronic device controls the camera to capture an image at a preset interval, and performs palm recognition on the captured image through a preset palm recognition model to obtain a palm recognition result; if the palm recognition result is that there is a palm in the image captured by the camera, the camera is controlled to capture multiple frames of images. The number of multiple frames of images, the capture interval, the number of camera devices, and the preset time can be set based on actual conditions, and the embodiments of the present application do not specifically limit this. For example, the camera captures 10 frames of images, the capture interval is 0.3 seconds, the camera includes a high-definition camera or an infrared camera, and the preset time is 1 second.
[0036] Among them, based on the assumption that the ambient light intensity of the image does not change, the luminosity of the point in the same world coordinate system in different frame images does not change. As long as the point of change in the next frame is found, the motion information of the point can be tracked. Exemplarily, based on the pyramid LK optical flow method, the motion information of each optical flow point in the image between adjacent frame images is determined according to multiple frame images, and the motion information includes the motion distance and motion direction of the optical flow point between every two adjacent frame images.
[0037] For example, the shooting device acquires 10 frames of images in a short time, which are the first image, the second image, the third image, the fourth image, the fifth image, the sixth image, the seventh image, the eighth image, the ninth image, and the tenth image acquired at time T, time T+1, time T+2, time T+3, time T+4, time T+5, time T+6, time T+7, time T+8, and time T+9, respectively. The motion information of the optical flow point between time T and time T+1 can be determined through the first image and the second image, the motion information of the optical flow point between time T+1 and time T+2 can be determined through the second image and the third image, the motion information of the optical flow point between time T+2 and time T+3 can be determined through the third image and the fourth image, the motion information of the optical flow point between time T+3 and time T+4 can be determined through the fourth image and the fifth image, and the motion information of the optical flow point between time T+4 and time T+5 can be determined through the fifth image and the sixth image.
[0038] Through the sixth image and the seventh image, the motion information of the optical flow point between the T+5 moment and the T+6 moment can be determined. Through the seventh image and the eighth image, the motion information of the optical flow point between the T+6 moment and the T+7 moment can be determined. Through the eighth image and the ninth image, the motion information of the optical flow point between the T+7 moment and the T+8 moment can be determined. Through the ninth image and the tenth image, the motion information of the optical flow point between the T+8 moment and the T+9 moment can be determined.
[0039] Step S102: Generate an optical flow sparse matrix according to the motion information of each of the optical flow points, and perform matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix.
[0040] Exemplarily, the matrix value of each optical flow point is determined according to the movement distance of each optical flow point between two adjacent frame images; the initial matrix value of each optical flow point in the preset initial optical flow sparse matrix is adjusted to the corresponding matrix value. The memory of the electronic device stores the initial optical flow sparse matrix corresponding to the optical flow points in the image captured by the shooting device. The initial matrix value of each optical flow point in the initial optical flow sparse matrix can be set based on actual conditions, and the embodiments of the present application do not specifically limit this. For example, Figure 2 As shown, the initial matrix value of each optical flow point in the initial optical flow sparse matrix is 0.
[0041] In one embodiment, according to the movement distance of each optical flow point, the target matrix value of each optical flow point in at least one initial optical flow sparse matrix can be determined by: comparing the movement distance of each optical flow point with a preset distance threshold; obtaining a first optical flow point whose movement distance is greater than or equal to the preset distance threshold and a second optical flow point whose movement distance is less than the preset distance threshold; determining the first preset value as the matrix value of the first optical flow point, and determining the second preset value as the matrix value of the second optical flow point. The first preset value and the second preset value can be set based on actual conditions, and the embodiment of the present application does not make specific limitations on this. For example, the first preset value is 1, and the second preset value is 0. Figure 1 The initial matrix value of each optical flow point in the initial optical flow sparse matrix shown is adjusted to the corresponding matrix value, and the obtained optical flow sparse matrix is as follows Figure 3 shown.
[0042] In one embodiment, according to the movement distance of each optical flow point, a target matrix value of each optical flow point in at least one initial optical flow sparse matrix can be determined by: comparing the movement distance of each optical flow point with a preset distance threshold; obtaining a first optical flow point whose movement distance is greater than or equal to the preset distance threshold and a second optical flow point whose movement distance is less than the preset distance threshold; determining the movement distance of the first optical flow point as the matrix value of the first optical flow point, and determining the second preset value as the matrix value of the second optical flow point.
[0043] Among them, the number of optical flow sparse matrices is determined according to the number of multi-frame images. Optionally, if the number of multi-frame images is N, the number of optical flow sparse matrices is N / 2. For example, 5 optical flow sparse matrices can be obtained from 10 frames of images. The first optical flow sparse matrix can be obtained through the motion information of the optical flow points between time T and time T+1, the second optical flow sparse matrix can be obtained through the motion information of the optical flow points between time T+2 and time T+3, the third optical flow sparse matrix can be obtained through the motion information of the optical flow points between time T+4 and time T+5, the fourth optical flow sparse matrix can be obtained through the motion information of the optical flow points between time T+6 and time T+7, and the fifth optical flow sparse matrix can be obtained through the motion information of the optical flow points between time T+8 and time T+9. Then, the first optical flow sparse matrix, the second optical flow sparse matrix, the third optical flow sparse matrix, the fourth optical flow sparse matrix and the fifth optical flow sparse matrix are accumulated to obtain the target optical flow sparse matrix.
[0044] Among them, the general method for calculating the area of a connected domain is to continuously traverse and sum, and then select the connected domain with the largest area. If the size of the target optical flow sparse matrix is m*n, the complexity of calculating the area of the matrix is O(m*n), and the maximum area is calculated by multiplying m by m on the basis of m*n, that is, the time complexity is O(m*m*n). The complexity is relatively high, which affects the running speed. The embodiment of the present application can calculate the matrix energy value of the target optical flow sparse matrix based on the stack, that is, the values of each row in the target optical flow sparse matrix are accumulated to obtain a one-dimensional array; the maximum continuous digital sum of the one-dimensional array is determined, and the maximum continuous digital sum of the one-dimensional array is written into an empty stack; when the stack is not empty, and the number of times h[i] that the maximum continuous digital sum of the one-dimensional array is added to the stack is less than the top of the stack, the maximum continuous digital sum stored at the top of the stack is popped out, and the popped out maximum continuous digital sum is determined as the matrix energy value of the target optical flow sparse matrix. The complexity of this part of the algorithm is O(n). Considering that the algorithm complexity of one row of the matrix is O(n), and the complexity of m rows of the matrix is O(m*n), the algorithm complexity is reduced by m times, which greatly improves the algorithm's running efficiency and speed.
[0045] Step S103: determining whether there is a dynamic gesture in the multiple-frame images according to the target optical flow sparse matrix, and identifying a palm image region in the multiple-frame images when there is a dynamic gesture in the multiple-frame images.
[0046] Exemplarily, the area of each connected domain of the target optical flow sparse matrix is determined, and the largest area is determined as the matrix energy value of the target optical flow sparse matrix; if the matrix energy value is greater than or equal to the preset matrix energy value, it is determined that there is a dynamic gesture in the multi-frame image; if the matrix energy value is less than the preset matrix energy value, it is determined that there is no dynamic gesture in the multi-frame image. The preset matrix energy value can be set based on actual conditions, and the embodiments of the present application do not make specific limitations on this. For example, the target optical flow sparse matrix is as follows Figure 4 As shown, through calculation, it is found that the target optical flow sparse matrix includes three connected domains [3,3], [1,1,2,3] and [8,8,7,9,6,5,3]. The connected domain with the largest area is [8,8,7,9,6,5,3], and the maximum area is 64. Therefore, the matrix energy value of the target optical flow sparse matrix is 64.
[0047] In one embodiment, the method for identifying the palm image area in the multiple frames of images may be: identifying the palm image area in the multiple frames of images according to a preset palm recognition model. The preset palm recognition model is a pre-trained neural network model, and its training process is specifically as follows: acquiring an image captured by a shooting device, and annotating the image to obtain sample data, wherein the sample data includes a sample image, the type of sample image, and the palm image area annotated in the sample image; iteratively training the neural network model according to a large amount of sample data until the neural network model after iterative training converges or the number of iterative training reaches a preset number, thereby obtaining a palm recognition model. It can be understood that the neural network model includes convolutional neural network models (CNN), deep neural networks (DNN) and recurrent neural networks (RNN).
[0048] Step S104: determining the gesture category to which the dynamic gesture belongs according to the motion information of each optical flow point in the palm image area.
[0049] The gesture categories to which the dynamic gestures belong include waving, sliding the hand to the left, sliding the hand to the right, sliding the hand up, sliding the hand down, and the like.
[0050] In one embodiment, the method for determining the gesture category to which a dynamic gesture belongs may be: determining a plurality of target optical flow points according to the movement distance of each optical flow point in the palm image area, wherein the movement distance of the target optical flow point is greater than or equal to a preset distance threshold; determining the optical flow motion vector of the dynamic gesture according to the motion information of each target optical flow point; determining the similarity between the optical flow motion vector and the preset optical flow motion vector corresponding to each gesture category; and determining the gesture category corresponding to the highest similarity as the gesture category to which the dynamic gesture belongs. Among them, the electronic device stores the preset optical flow motion vector corresponding to each gesture category, and the preset optical flow motion vectors of different gesture categories are different. The similarity between the optical flow motion vector and the preset optical flow motion vector can be the cosine similarity, and the preset distance threshold can be set based on the actual situation, which is not specifically limited in the embodiments of the present application.
[0051] In one embodiment, after determining the gesture category to which the dynamic gesture belongs, a control instruction corresponding to the gesture category is obtained, and according to the control instruction, the electronic device is controlled to execute the corresponding event. In the same usage scenario, different gesture categories correspond to different control instructions, and in different usage scenarios, the same gesture category may correspond to different control instructions.
[0052] For example, with respect to the movement of a selection mark in a display page of an electronic device, if the dynamic gesture is a swiping hand to the left, the selection mark displayed by the electronic device moves to the left; if the dynamic gesture is a swiping hand to the right, the selection mark displayed by the electronic device moves to the right; if the dynamic gesture is a swiping hand upward, the selection mark displayed by the electronic device moves upward; if the dynamic gesture is a swiping hand downward, the selection mark displayed by the electronic device moves downward.
[0053] For another example, for the jump control of pictures displayed on the electronic device, if the dynamic gesture is swiping the hand to the left, the picture displayed on the electronic device jumps to the previous picture; if the dynamic gesture is swiping the hand to the right, the picture displayed on the electronic device jumps to the next picture; if the dynamic gesture is swiping the hand upward, the electronic device displays the detailed information of the picture; if the dynamic gesture is swiping the hand downward, the electronic device deletes the displayed picture.
[0054] As another example, for the page turning control of the text displayed on the electronic device, if the dynamic gesture is a swiping hand to the left, the page displayed by the electronic device jumps to the previous page; if the dynamic gesture is a swiping hand to the right, the page displayed by the electronic device jumps to the next page; if the dynamic gesture is a swiping hand upward, the electronic device displays the homepage of the book; if the dynamic gesture is a swiping hand downward, the electronic device marks the currently displayed page.
[0055] The optical flow-based dynamic gesture recognition method provided in the above embodiment obtains multiple frames of images captured by a shooting device, and determines the motion information of each optical flow point in the image based on the multiple frames of images, and then generates an optical flow sparse matrix based on the motion information of each optical flow point, and performs matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix. Finally, based on the target optical flow sparse matrix, it is determined whether there is a dynamic gesture in the multiple frames of images, and when there is a dynamic gesture in the multiple frames of images, the palm image area in the multiple frames of images is identified, and the gesture category to which the dynamic gesture belongs is determined based on the motion information of each optical flow point in the palm image area. It does not need to rely on training data and can improve the recognition accuracy of dynamic gestures.
[0056] See also Figure 5 , Figure 5 It is a schematic block diagram of a dynamic gesture recognition device based on optical flow provided in an embodiment of the present application.
[0057] like Figure 5 As shown, the optical flow-based dynamic gesture recognition device 200 includes an acquisition module 210, a determination module 220, a matrix generation module 230 and a recognition module 240, wherein:
[0058] The acquisition module 210 is used to acquire multiple frames of images captured by the shooting device;
[0059] The determination module 220 is used to determine the motion information of each optical flow point in the image according to the multiple frames of images;
[0060] The matrix generation module 230 is used to generate an optical flow sparse matrix according to the motion information of each optical flow point, and perform matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix;
[0061] The determination module 220 is further configured to determine whether there is a dynamic gesture in the multiple frames of images according to the target optical flow sparse matrix;
[0062] When there is a dynamic gesture in the multiple frames of images, the palm image area in the multiple frames of images is identified by the identification module 240;
[0063] The determination module 220 is further configured to determine the gesture category to which the dynamic gesture belongs according to the motion information of each optical flow point in the palm image area.
[0064] In one embodiment, the motion information includes the motion distance of the optical flow point between every two adjacent frame images, and the matrix generation module 230 is further used to:
[0065] Determine a matrix value of each optical flow point according to a movement distance of each optical flow point between two adjacent frame images;
[0066] The initial matrix value of each optical flow point in the preset initial optical flow sparse matrix is adjusted to the corresponding matrix value.
[0067] In one embodiment, the matrix generation module 230 is further used to:
[0068] Comparing the movement distance of each of the optical flow points with a preset distance threshold;
[0069] Acquire a first optical flow point whose movement distance is greater than or equal to a preset distance threshold and a second optical flow point whose movement distance is less than the preset distance threshold;
[0070] A first preset value is determined as the matrix value of the first optical flow point, and a second preset value is determined as the matrix value of the second optical flow point.
[0071] In one embodiment, the determining module 220 is further configured to:
[0072] Determine the area of each connected domain of the target optical flow sparse matrix, and determine the largest area as the matrix energy value of the target optical flow sparse matrix;
[0073] If the matrix energy value is greater than or equal to a preset matrix energy value, determining that a dynamic gesture exists in the multiple frames of images;
[0074] If the matrix energy value is less than a preset matrix energy value, it is determined that no dynamic gesture exists in the multiple frames of images.
[0075] In one embodiment, the identification module 240 is further configured to:
[0076] According to a preset palm recognition model, palm image areas in the multiple frames of images are identified, wherein the preset palm recognition model is a pre-trained neural network model.
[0077] In one embodiment, the determination module 220 is further used to: determine a plurality of target optical flow points according to the movement distance of each optical flow point in the palm image area, wherein the movement distance of the target optical flow point is greater than or equal to a preset distance threshold;
[0078] Determining an optical flow motion vector of the dynamic gesture according to the motion information of each target optical flow point;
[0079] Determining the similarity between the optical flow motion vector and a preset optical flow motion vector corresponding to each gesture category;
[0080] The gesture category corresponding to the highest similarity is determined as the gesture category to which the dynamic gesture belongs.
[0081] In one embodiment, the dynamic gesture recognition device 200 further includes:
[0082] The acquisition module 210 is further used to acquire the control instruction corresponding to the gesture category;
[0083] The execution module is used to control the electronic device to execute a corresponding event according to the control instruction.
[0084] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module and unit can refer to the corresponding process in the aforementioned embodiment of the dynamic gesture recognition method based on optical flow, and will not be repeated here.
[0085] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 6 The electronic device shown is running.
[0086] See also Figure 6 , Figure 6 It is a schematic block diagram of the structure of an electronic device provided in an embodiment of the present application.
[0087] like Figure 6 As shown, the electronic device 300 includes a processor 310, a memory 320 and a camera 330, and the processor 310, the memory 320 and the camera 330 are connected via a bus 340, and the bus 340 is, for example, an I2C (Inter-integrated Circuit) bus.
[0088] Specifically, the memory 320 may include a storage medium and an internal memory, and the memory 320 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk. The storage medium may store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor may execute any one of the dynamic gesture recognition methods based on optical flow.
[0089] The processor 310 is used to provide computing and control capabilities to support the operation of the entire electronic device. The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any dynamic gesture recognition method based on optical flow.
[0090] It should be understood that the processor 310 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0091] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0092] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0093] Acquire multiple frames of images captured by the shooting device, and determine motion information of each optical flow point in the image based on the multiple frames of images;
[0094] Generate an optical flow sparse matrix according to the motion information of each of the optical flow points, and perform matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix;
[0095] determining whether there is a dynamic gesture in the multiple frames of images according to the target optical flow sparse matrix, and identifying a palm image region in the multiple frames of images when there is a dynamic gesture in the multiple frames of images;
[0096] The gesture category to which the dynamic gesture belongs is determined according to the motion information of each optical flow point in the palm image area.
[0097] In one embodiment, the motion information includes the motion distance of the optical flow point between every two adjacent frame images, and the processor, when generating the optical flow sparse matrix according to the motion information of each optical flow point, is used to implement:
[0098] Determine a matrix value of each optical flow point according to a movement distance of each optical flow point between two adjacent frame images;
[0099] The initial matrix value of each optical flow point in the preset initial optical flow sparse matrix is adjusted to the corresponding matrix value.
[0100] In one embodiment, when determining the matrix value of each optical flow point according to the movement distance of each optical flow point between two adjacent frame images, the processor is used to implement:
[0101] Comparing the movement distance of each of the optical flow points with a preset distance threshold;
[0102] Acquire a first optical flow point whose movement distance is greater than or equal to a preset distance threshold and a second optical flow point whose movement distance is less than the preset distance threshold;
[0103] A first preset value is determined as the matrix value of the first optical flow point, and a second preset value is determined as the matrix value of the second optical flow point.
[0104] In one embodiment, when the processor determines whether there is a dynamic gesture in the multiple frames of images according to the target optical flow sparse matrix, it is configured to implement:
[0105] Determine the area of each connected domain of the target optical flow sparse matrix, and determine the maximum area as the matrix energy value of the target optical flow sparse matrix;
[0106] If the matrix energy value is greater than or equal to a preset matrix energy value, determining that a dynamic gesture exists in the multiple frames of images;
[0107] If the matrix energy value is less than a preset matrix energy value, it is determined that no dynamic gesture exists in the multiple frames of images.
[0108] In one embodiment, when the processor realizes identifying the palm image area in the multiple frames of images, it is used to realize:
[0109] According to a preset palm recognition model, palm image areas in the multiple frames of images are identified, wherein the preset palm recognition model is a pre-trained neural network model.
[0110] In one embodiment, when determining the gesture category to which the dynamic gesture belongs according to the motion information of each optical flow point in the palm image area, the processor is used to implement:
[0111] Determine a plurality of target optical flow points according to the movement distance of each optical flow point in the palm image area, wherein the movement distance of the target optical flow point is greater than or equal to a preset distance threshold;
[0112] Determining an optical flow motion vector of the dynamic gesture according to the motion information of each target optical flow point;
[0113] Determining the similarity between the optical flow motion vector and a preset optical flow motion vector corresponding to each gesture category;
[0114] The gesture category corresponding to the highest similarity is determined as the gesture category to which the dynamic gesture belongs.
[0115] In one embodiment, after determining the gesture category to which the dynamic gesture belongs according to the motion information of each optical flow point in the palm image area, the processor is further configured to implement:
[0116] A control instruction corresponding to the gesture category is obtained, and according to the control instruction, the electronic device is controlled to execute a corresponding event.
[0117] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process in the aforementioned embodiment of the dynamic gesture recognition method based on optical flow, and will not be repeated here.
[0118] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0119] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the dynamic gesture recognition method based on optical flow in the present application.
[0120] The computer-readable storage medium may be volatile or non-volatile. The computer-readable storage medium may be an internal storage unit of the electronic device described in the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc., equipped on the electronic device.
[0121] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.
[0122] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0123] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0124] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.
[0125] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A dynamic gesture recognition method based on optical flow, characterized in that: Applied to an electronic device, the electronic device includes a photographing device, and the method includes: Acquire multiple frames of images captured by the shooting device, and determine motion information of each optical flow point in the image according to the multiple frames of images, wherein the motion information includes a motion distance of the optical flow point between every two adjacent frames of images; Generate an optical flow sparse matrix according to the movement distance of each optical flow point between each two adjacent frame images, and perform matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix; determining whether there is a dynamic gesture in the multiple frames of images according to the target optical flow sparse matrix, and identifying a palm image region in the multiple frames of images when there is a dynamic gesture in the multiple frames of images; determining the gesture category to which the dynamic gesture belongs according to the motion information of each optical flow point in the palm image area; Wherein, determining whether there is a dynamic gesture in the multiple frames of images according to the target optical flow sparse matrix includes: Determine the area of each connected domain of the target optical flow sparse matrix, and determine the largest area as the matrix energy value of the target optical flow sparse matrix; If the matrix energy value is greater than or equal to a preset matrix energy value, determining that a dynamic gesture exists in the multiple frames of images; If the matrix energy value is less than a preset matrix energy value, it is determined that no dynamic gesture exists in the multiple frames of images.
2. The dynamic gesture recognition method according to claim 1, characterized in that: The step of generating an optical flow sparse matrix according to the movement distance of each optical flow point between every two adjacent frame images comprises: Determine a matrix value of each optical flow point according to a movement distance of each optical flow point between two adjacent frame images; The initial matrix value of each optical flow point in the preset initial optical flow sparse matrix is adjusted to the corresponding matrix value to obtain the optical flow sparse matrix.
3. The dynamic gesture recognition method according to claim 2, characterized in that: Determining the matrix value of each optical flow point according to the movement distance of each optical flow point between two adjacent frame images includes: Comparing the movement distance of each of the optical flow points with a preset distance threshold; Acquire a first optical flow point whose movement distance is greater than or equal to a preset distance threshold and a second optical flow point whose movement distance is less than the preset distance threshold; A first preset value is determined as the matrix value of the first optical flow point, and a second preset value is determined as the matrix value of the second optical flow point.
4. The dynamic gesture recognition method according to claim 1, characterized in that: The identifying of the palm image area in the multiple frames of images comprises: According to a preset palm recognition model, palm image areas in the multiple frames of images are identified, wherein the preset palm recognition model is a pre-trained neural network model.
5. The dynamic gesture recognition method according to any one of claims 1 to 4, characterized in that: The step of determining the gesture category to which the dynamic gesture belongs according to the motion information of each optical flow point in the palm image area includes: Determine a plurality of target optical flow points according to the movement distance of each optical flow point in the palm image area, wherein the movement distance of the target optical flow point is greater than or equal to a preset distance threshold; Determining an optical flow motion vector of the dynamic gesture according to the motion information of each target optical flow point; Determining the similarity between the optical flow motion vector and a preset optical flow motion vector corresponding to each gesture category; The gesture category corresponding to the highest similarity is determined as the gesture category to which the dynamic gesture belongs.
6. The dynamic gesture recognition method according to any one of claims 1 to 4, characterized in that: After determining the gesture category to which the dynamic gesture belongs according to the motion information of each optical flow point in the palm image area, the method further includes: A control instruction corresponding to the gesture category is obtained, and according to the control instruction, the electronic device is controlled to execute a corresponding event.
7. A dynamic gesture recognition device based on optical flow, characterized in that: Applied to an electronic device, the electronic device includes a shooting device, the dynamic gesture recognition device includes an acquisition module, a determination module, a matrix generation module and a recognition module, wherein: The acquisition module is used to acquire multiple frames of images captured by the shooting device; The determination module is used to determine the motion information of each optical flow point in the image according to the multiple frames of images, wherein the motion information includes the motion distance of the optical flow point between every two adjacent frames of images; The matrix generation module is used to generate an optical flow sparse matrix according to the movement distance of each optical flow point between each two adjacent frame images, and perform matrix addition operation on each generated optical flow sparse matrix to obtain a target optical flow sparse matrix; The determination module is further used to determine whether there is a dynamic gesture in the multiple frames of images according to the target optical flow sparse matrix; When there is a dynamic gesture in the multiple frames of images, identifying a palm image area in the multiple frames of images by the recognition module; The determination module is further used to determine the gesture category to which the dynamic gesture belongs according to the motion information of each optical flow point in the palm image area; The determining module is further used for: Determine the area of each connected domain of the target optical flow sparse matrix, and determine the largest area as the matrix energy value of the target optical flow sparse matrix; If the matrix energy value is greater than or equal to a preset matrix energy value, determining that a dynamic gesture exists in the multiple frames of images; If the matrix energy value is less than a preset matrix energy value, it is determined that no dynamic gesture exists in the multiple frames of images.
8. An electronic device, characterized in that: The electronic device includes a processor, a memory, a camera, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the dynamic gesture recognition method as described in any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the dynamic gesture recognition method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Hand gesture recognition for virtual reality and augmented reality devices
US20180189556A1