Gesture recognition method and related products

Abstract

The embodiment of the present application discloses a gesture recognition method and related products, which comprises the following steps: acquiring multiple gesture images; acquiring position information of hand key points in each gesture image in the multiple gesture images; determining the distance of the hand in each two gesture images adjacent in time from the acquisition of the multiple gesture images according to the position information, and obtaining multiple distances corresponding to the multiple gesture images; determining a target model from a preset dynamic gesture recognition model and a static gesture recognition model according to the multiple distances; and calling the target model to perform gesture recognition on the multiple gesture images. The embodiment of the present application is beneficial to improving the accuracy of the gesture recognition result.

Description

Technical Field

[0001] This application relates to the field of image processing, specifically to a gesture recognition method and related products. Background Technology

[0002] With the development of science and technology and the widespread use of various electronic devices in society, people are paying increasing attention to improving the level and quality of human-computer interaction technology. Gestures have long been considered a natural and intuitive means of human-computer interaction; therefore, gesture recognition based on computer vision has become an important research area in human-computer interaction. Based on gesture recognition technology, users can conveniently control electronic devices through gestures. However, currently, gesture recognition typically does not differentiate between dynamic and static gesture recognition; both are recognized using the same model, leading to insufficient accuracy in the recognition results. Summary of the Invention

[0003] This application provides a gesture recognition method and related products to improve the accuracy of gesture recognition results.

[0004] In a first aspect, embodiments of this application provide a gesture recognition method, comprising the following steps:

[0005] Acquire multiple frames of gesture images;

[0006] Obtain the position information of key hand points in each frame of the multi-frame gesture images;

[0007] Based on the location information, the distance between the hands in every two adjacent frames of the multi-frame gesture images is determined, thereby obtaining multiple distances corresponding to the multi-frame gesture images.

[0008] The target model is determined from the preset dynamic gesture recognition model and static gesture recognition model based on the multiple distances;

[0009] The target model is invoked to perform gesture recognition on the multi-frame gesture images.

[0010] Secondly, embodiments of this application provide a gesture recognition device, including:

[0011] The first acquisition unit is used to acquire multiple frames of gesture images;

[0012] The second acquisition unit is used to acquire the position information of key hand points in each frame of the multi-frame gesture images;

[0013] The first determining unit is used to determine the distance between the hands in every two adjacent frames of the gesture images based on the location information, thereby obtaining multiple distances corresponding to the multi-frame gesture images.

[0014] The second determining unit is used to determine the target model from the preset dynamic gesture recognition model and static gesture recognition model based on the multiple distances;

[0015] The calling unit is used to call the target model to perform gesture recognition on the multi-frame gesture images.

[0016] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing the steps in the first aspect of embodiments of this application.

[0017] Fourthly, embodiments of this application provide a computer storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of this embodiment.

[0018] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0019] As can be seen, in this embodiment, the electronic device first acquires multiple frames of gesture images, then acquires the position information of key hand points within each frame of the gesture images, and then, based on this position information, determines the distance between the hand in every two adjacent frames of the gesture images, obtaining multiple distances. Finally, based on these multiple distances, a target model is determined from preset dynamic gesture recognition models and static gesture recognition models, and the target model is called to perform gesture recognition on the multi-frame gesture images. Therefore, the electronic device can automatically call different gesture recognition models to perform gesture recognition based on the different distances the user's hand moves; that is, different preset models are used for recognition of dynamic and static gestures, which helps to improve the accuracy of gesture recognition. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1AThis is a schematic diagram of a hardware architecture provided in an embodiment of this application;

[0022] Figure 1B This is a schematic diagram of another hardware architecture provided in an embodiment of this application;

[0023] Figure 1C This is a component example diagram of an electronic device provided in an embodiment of this application;

[0024] Figure 2 This is a flowchart illustrating a gesture recognition method provided in an embodiment of this application;

[0025] Figure 3A This is a block diagram of the functional units of a gesture recognition device provided in an embodiment of this application;

[0026] Figure 3B This is a block diagram of the functional units of another gesture recognition device provided in the embodiments of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0028] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] The embodiments of this application will now be described with reference to the accompanying drawings.

[0031] The hardware architecture of the technical solution in this application can be any of the following:

[0032] (1) Electronic equipment + image acquisition equipment architecture, such as Figure 1A As shown, the electronic device 100 is communicatively connected to the image acquisition device 200, which is used to acquire gesture images. The electronic device 100 can communicate with and acquire gesture images from the image acquisition device 200. For example, the electronic device 100 can acquire multiple frames of gesture images from the image acquisition device. Then, based on the position information of key hand points within the acquired multiple frames of gesture images, it finally determines a target gesture recognition model from a preset dynamic gesture recognition model and a static gesture recognition model, and calls this target gesture recognition model to perform gesture recognition on the acquired multiple frames of gesture images. Specifically, the technical solution of this application can be applied in in-vehicle gesture recognition scenarios. The electronic device 100 can be an in-vehicle device such as a car infotainment system installed in the vehicle, and the image acquisition device 200 can be a camera installed in the vehicle, etc.

[0033] (2) Local architecture of electronic devices, such as Figure 1B As shown, the electronic device 100 is equipped with an image acquisition device 101, which can acquire gesture images through the locally installed image acquisition device 101.

[0034] The electronic device 100 in this application can be structured as follows: Figure 1C As shown, the electronic device 100 may include a processor 110, a memory 120, a communication interface 130, and one or more programs 121, wherein the one or more programs 121 are stored in the memory 120 and configured to be executed by the processor 110, and the one or more programs 121 include instructions for performing any step in the above method embodiments.

[0035] The communication interface 130 is used to support communication between the electronic device 100 and other devices. The processor 110 may be, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, units, and circuits described in conjunction with the embodiments of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0036] The memory 120 can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0037] In a specific implementation, the processor 110 is used to execute any step performed by the electronic device in the following method embodiments, and when performing data transmission such as sending, it may selectively call the communication interface 130 to complete the corresponding operation.

[0038] It should be noted that the structural schematic diagram of the above-described electronic device 100 is merely an example, and the actual number of components included may be greater or less; no single limitation is made here. For example, the electronic device 100 may also include an image acquisition device, which can be communicatively connected to the processor 110 and the storage 120 via a communication bus. The processor 110 can acquire gesture images through the image acquisition device, and the acquired gesture images can also be stored in the storage 120. Specifically, when the processor executes the step of acquiring multiple frames of gesture images in the method embodiment of this application, it can directly acquire multiple frames of gesture images through the image acquisition device, or it can retrieve the multiple frames of gesture images acquired by the image acquisition device and stored in the storage 120 from the storage 120.

[0039] Please see Figure 2 , Figure 2 This is a flowchart illustrating a gesture recognition method provided in an embodiment of this application. This method can be applied to, for example... Figure 1A or Figure 1B On the electronic device shown, such as Figure 2 As shown, the gesture recognition method includes the following steps:

[0040] S201, The electronic device acquires multiple frames of gesture images.

[0041] In this multi-frame gesture image, the acquisition time of each frame of the gesture image is different. The multi-frame gesture images are sorted by acquisition time from early to late or from late to early. The acquisition time interval between each two adjacent frames of the gesture image is a first preset time.

[0042] In a specific implementation, the image acquisition device / or image acquisition unit can acquire one frame of gesture image at a first preset time interval and save the acquired gesture image in the image acquisition device or electronic device (a preset number of gesture images with the closest time to the current time can be set to save device storage resources). When the electronic device obtains these multiple frames of gesture images, it can directly retrieve them from the gesture images saved by the image acquisition device or electronic device. Alternatively, the electronic device can also notify the image acquisition device or acquire gesture images only when it receives a gesture recognition request.

[0043] In addition, for in-vehicle gesture recognition scenarios, for driving safety considerations, electronic devices can also obtain vehicle driving status information (such as vehicle speed) after receiving a gesture recognition request. This vehicle driving status information is used to indicate whether the vehicle is in a driving state or a stationary state. When the vehicle driving status information indicates that the vehicle is in a driving state, the identity information of the user who made the request is determined. If the user is determined to be the driver, gesture recognition is stopped and a risk warning message is output.

[0044] Specifically, a pressure detection device can be installed under each seat in the vehicle. This pressure detection device is connected to an electronic device. The electronic device obtains the pressure detection results from each pressure detection device and determines that only the driver's seat is occupied in the vehicle, thus identifying the user who made the request as the driver.

[0045] Alternatively, an image acquisition device can be installed in front of the driver's seat. This device is connected to an electronic device to capture facial images of passengers in the driver's seat. The electronic device stores user identity information, including palm print information and facial information for each user. After acquiring multiple frames of gesture images, the electronic device can acquire the facial image captured by the image acquisition device in front of the driver's seat. Then, it performs palm print recognition on the gesture images to determine the palm print information and performs facial recognition on the facial images to determine the facial information. If the palm print information and facial information belong to the same user, then the user who issued the gesture recognition request is determined to be the driver.

[0046] S202, the electronic device acquires the position information of key hand points in each frame of the multi-frame gesture image.

[0047] Among them, the key points of the hand can be the joints of the user's hand, that is, the electronic device can recognize each frame of gesture image and extract the position information of the user's hand joints in each frame of gesture image.

[0048] S203, the electronic device determines the distance between the hands in every two adjacent frames of the multi-frame gesture images based on the location information, thereby obtaining multiple distances corresponding to the multi-frame gesture images.

[0049] For example, a multi-frame gesture image may include three frames, which are ordered from earliest to latest according to the acquisition time as: image 1, image 2, and image 3. Multiple distances are determined by distance 1 based on image 1 and image 2, and distance 2 based on image 2 and image 3.

[0050] S204, the electronic device determines the target model from the preset dynamic gesture recognition model and static gesture recognition model based on the multiple distances.

[0051] Because users' hands hardly move when using static gestures, while the position of their hands usually changes continuously when using dynamic gestures, electronic devices can determine whether a user is using a dynamic or static gesture based on the distance between the hands of two adjacent gesture images captured at the same time. This allows them to then retrieve a pre-set dynamic or static gesture recognition model for gesture recognition accordingly.

[0052] The preset dynamic gesture recognition model and static gesture recognition model are trained using different training samples. Specifically, the training method for the dynamic gesture recognition model can be as follows: First, obtain multiple training samples and training labels corresponding to each training sample. Each training sample includes a set of gesture images (the number of gesture images in the set is greater than 1, for example, it can be a second preset number). The training label corresponding to each training sample can be the actual dynamic gesture category corresponding to the set of gesture images. For each training sample, perform the following operations: Input the current training sample (i.e., the set of gesture images) into the neural network to obtain the predicted dynamic gesture category corresponding to the training sample. The initial parameters of the neural network can be obtained through random initialization. Based on the predicted dynamic gesture category and the sample label (i.e., the actual dynamic gesture category) corresponding to the current training sample, adjust the parameters of the neural network to obtain the trained dynamic gesture recognition model.

[0053] The training method for static gesture recognition models is the same as that for dynamic gesture recognition models. In the training samples obtained by the static gesture recognition model, each training sample includes a frame of gesture image, and the training label corresponding to each training sample is the actual static gesture category. After inputting each training sample into the neural network, the predicted static gesture category corresponding to the training sample is obtained. Then, the parameters of the neural network can be adjusted according to the predicted static gesture category and the actual static gesture category corresponding to the current training sample to obtain the trained static gesture recognition model.

[0054] In practice, the electronic device can pre-store multiple different dynamic gesture recognition models and static gesture recognition models. Each dynamic gesture recognition model and static gesture recognition model is associated with a unique user identifier. After receiving a gesture recognition request, the electronic device can acquire multiple frames of gesture images. The gesture recognition request can carry the user identifier (the user identifier can be selected and determined by the user on the gesture recognition start interface). Before determining the target model, the electronic device can determine the dynamic gesture recognition model and static gesture recognition model that match the user identifier from the multiple pre-stored dynamic gesture recognition models and static gesture recognition models, and use them as the preset dynamic gesture recognition model and static gesture recognition model.

[0055] Furthermore, after each gesture recognition, the electronic device can use the result of this gesture recognition as sample data to train the target model used this time. In other words, by training the (dynamic and / or static) gesture recognition model corresponding to each user through the gesture recognition results, the model can be more closely matched with the user, which can further improve the accuracy of gesture recognition results.

[0056] S205, the electronic device calls the target model to perform gesture recognition on the multi-frame gesture images.

[0057] As can be seen, in this embodiment, the electronic device first acquires multiple frames of gesture images, then acquires the position information of key hand points within each frame of the gesture images, and then, based on this position information, determines the distance between the hand in every two adjacent frames of the gesture images, obtaining multiple distances. Finally, based on these multiple distances, a target model is determined from preset dynamic gesture recognition models and static gesture recognition models, and the target model is called to perform gesture recognition on the multi-frame gesture images. Therefore, the electronic device can automatically call different gesture recognition models to perform gesture recognition based on the different distances the user's hand moves; that is, different preset models are used for recognition of dynamic and static gestures, which helps to improve the accuracy of gesture recognition.

[0058] In one possible example, determining the target model from the preset dynamic gesture recognition model and static gesture recognition model based on the plurality of distances includes the following steps: if the number of distances greater than a preset threshold among the plurality of distances is greater than a first preset number, then the dynamic gesture recognition model is determined to be the target model; otherwise, the static gesture recognition model is determined to be the target model.

[0059] In practical implementation, to facilitate statistical processing by electronic devices, after determining each distance, the distance can be simplified based on its relationship with a preset threshold. For example, when the distance is greater than the preset threshold, the value of the distance is recorded as 1; when the distance is less than or equal to the preset threshold, the value of the distance is recorded as 0. To determine whether the number of distances exceeding the preset threshold is greater than a first preset number, the values ​​of multiple distances can be directly added together. If the sum is greater than the first preset number, then the number of distances exceeding the preset threshold is greater than the first preset number.

[0060] In practice, considering that users of different ages or heights may have different hand movement ranges when using the same gesture, the preset threshold can be determined based on the user identity information corresponding to the gesture recognition request. Different user identity information corresponds to different preset thresholds. The electronic device can query the correspondence between the preset user identity information and the preset threshold based on the user identity information, thereby determining the preset threshold corresponding to the user identity information.

[0061] Specifically, user identity information can be determined by the electronic device based on user identifiers. The electronic device pre-stores multiple user identifiers and the corresponding user identity information for each identifier. These user identifiers can be carried in gesture recognition requests received by the electronic device, or they can be determined by the electronic device through palmprint recognition of multiple frames of gesture images. User identity information may specifically include at least one of the following: user age and user height.

[0062] As can be seen in this example, when the number of distances greater than the preset threshold among multiple distances is greater than the first preset number, the dynamic gesture recognition model is determined as the target model; otherwise, the static gesture recognition model is determined as the target model. Judging based on multiple distances helps to improve the accuracy of model calling.

[0063] In one possible example, after determining the dynamic gesture recognition model as the target model, the method further includes the following steps: calling the dynamic gesture recognition model to perform gesture recognition on the multi-frame gesture images; if no dynamic gesture is recognized, then re-determining the static gesture recognition model as the target model.

[0064] In practical implementation, it is considered that the user's hand may move when using static gestures. For example, in a vehicle gesture recognition scenario, due to vehicle vibration and other reasons, the hand position in the gesture image captured at different times may change when the user uses a static gesture. In this case, even if the dynamic gesture recognition model is determined as the target model, calling the dynamic recognition model will not be able to recognize the dynamic gesture, thus leading to gesture recognition failure. Therefore, when the dynamic gesture model fails to recognize the dynamic gesture, the static gesture recognition model can be redefined as the target model.

[0065] When an electronic device uses a dynamic gesture recognition model to recognize gestures from multiple frames of images, the recognition results can include multiple scores. Each score corresponds to a preset dynamic gesture category. When the score for a particular dynamic gesture category meets a preset condition (e.g., the score is greater than a preset score), the electronic device can determine that dynamic gesture category as the final gesture recognition result for the multi-frame image. If the score for each dynamic gesture category does not meet the preset condition, it is considered that no dynamic gesture has been recognized.

[0066] In addition, in other embodiments, if the target model is a dynamic gesture recognition model, the electronic device can also call a static gesture recognition model at the same time when calling the dynamic gesture recognition model for gesture recognition. If no dynamic gesture is recognized, the result obtained by calling the static gesture recognition model is directly determined as the final gesture recognition result to improve the device response speed.

[0067] Furthermore, when the electronic device is an in-vehicle device, it can also obtain vehicle location information. When the vehicle is determined to be on a preset uneven road section based on the vehicle location information, both the dynamic gesture recognition model and the static gesture recognition model are called simultaneously when the target model is a dynamic model, in order to save resources.

[0068] As can be seen in this example, if the dynamic gesture recognition model is not recognized after calling the dynamic gesture recognition model, the static gesture recognition model is redefined as the target model, which helps to improve the reliability of gesture recognition.

[0069] In one possible example, when the target model is the dynamic gesture recognition model, the step of calling the target model to perform gesture recognition on the multi-frame gesture images includes the following steps: obtaining a second preset number of gesture images from the multi-frame gesture images whose acquisition time is closest to the current time, wherein the second preset number is greater than 1; and calling the dynamic gesture recognition model to perform gesture recognition on the second preset number of gesture images.

[0070] The second preset number can be equal to the number of the multi-frame gesture images. That is, in step 201, the multi-frame gesture images acquired by the electronic device are the second preset number of gesture images whose acquisition time is closest to the current time.

[0071] As can be seen in this example, since the user's gestures are not always constant when using dynamic gestures, calling the dynamic gesture recognition model to perform gesture recognition on multiple frames of gesture images is beneficial to ensuring the accuracy of the dynamic gesture recognition results when the target model is a dynamic gesture recognition model.

[0072] In one possible example, when the target model is the static gesture recognition model, the step of calling the target model to perform gesture recognition on the multi-frame gesture images includes the following steps: obtaining the gesture image whose acquisition time is closest to the current time from the multi-frame gesture images; and calling the static gesture recognition model to perform gesture recognition on the single-frame gesture image.

[0073] Since the user's gestures do not change when using static gestures, in order to save the resources of electronic devices, gesture recognition can be performed only on the gesture image that is closest to the current moment when it was captured.

[0074] In other embodiments, to improve the accuracy of gesture recognition results, the electronic device can also acquire a third preset number (greater than 1) of gesture images from multiple frames of gesture images, and call the static gesture recognition model to perform gesture recognition on each of the third preset number of gesture images, obtaining multiple gesture recognition results. Each gesture recognition result includes the category of the recognized static gesture. The electronic device can count the number of each static gesture category in the multiple gesture recognition results and determine the static gesture category with the most occurrences as the final gesture recognition result. Alternatively, the electronic device can comprehensively determine the final gesture recognition result based on the number of each static gesture category in the multiple gesture recognition results and the acquisition time of the image corresponding to each gesture recognition result. Specifically, a time coefficient is determined according to the order of gesture image acquisition time, and the value used for counting the gesture recognition result of that frame of gesture image is determined based on the time coefficient of each frame of gesture image. Among them, the larger the time coefficient, the larger the value used for counting. Depending on the actual needs, the earlier the acquisition time, the larger the time coefficient, or the later the acquisition time, the larger the time coefficient.

[0075] Specifically, the time coefficient can be directly used as the quantity for statistics. For example, if the third preset quantity is 3, three frames of gesture images are acquired: image A, image B, and image C. Image A is acquired first, followed by image B, and image C is acquired last. The gesture recognition results of images A and B are category 1, and the gesture recognition result of image C is category 2. If the time coefficient of the first acquisition is set to 0, the time coefficient of the middle acquisition is set to 1, and the time coefficient of the last acquisition is set to 2, after statistics, the quantity of category 1 is 1, the quantity of category 2 is 2, so the final recognition result is category 2.

[0076] As can be seen, in this example, when the target model is a static gesture recognition model, the electronic device only performs gesture recognition on a single frame of gesture image, which helps to reduce the resource consumption of the electronic device.

[0077] In one possible example, the location information includes coordinate information; the step of determining the distance between the hands in every two adjacent frames of the multi-frame gesture images based on the location information includes the following steps: performing the following operations for each frame of the multi-frame gesture images: determining the coordinate information of the center point of the hand in the current frame gesture image based on the coordinate information of the key points of the hand in the current frame gesture image;

[0078] For each pair of gesture images, the following operations are performed: Based on the coordinate information of key hand points in each of the current two gesture images, determine the mean and standard deviation of the coordinates; based on the mean and standard deviation, standardize the coordinate information of the hand center point in each of the current two gesture images; based on the standardized coordinate information of the hand center point in each of the current two gesture images, determine the distance between the hands in the current two gesture images. Note that there can be multiple key hand points in each gesture image, but only one hand center point. Determining the distance between the hands in two gesture images using the coordinate information of the hand center point helps reduce the processing load of the electronic device, thereby improving its processing efficiency.

[0079] Specifically, the electronic device can calculate the distance between the hand's center points in each of the two current gesture images based on the coordinates of the hand's center point in each frame of the standardized gesture images, and determine this distance as the distance of the hand in the two current gesture images. For example, the two current gesture images include a first image and a second image, and the coordinates of the hand's center point in the standardized first image are (x...). a y a The coordinates of the center point of the hand in the standardized second image are (x...). b y b If the distance between the hands in the current two frames of the gesture image is calculated, then...

[0080] In practice, considering that determining the hand movement distance based on the coordinates of the hand's center point in the gesture image before standardization (i.e., the original coordinates without standardization), even if the actual distance the user's hand moves is the same, the distance reflected in the gesture image will differ due to variations in the distance between the user's hand and the image acquisition device, or due to differences in the image size (e.g., length and width). In other words, the hand distance determined using the original coordinates of the hand's center point will also differ. Furthermore, if the distance of the user's hand is determined using the original coordinates of the hand's center point, differences in the distance between the hand and the image acquisition device will interfere with the final determined distance, and the determined distance will not be resistant to image scaling.

[0081] Therefore, instead of directly using the original coordinates of the hand's center point in each image to calculate the hand distance, we determine the mean and standard deviation of the coordinates by using the coordinate information of the hand's key points and center point in each of the two gesture images. We then standardize the original coordinates of the hand's center point. Since the mean and variance of the coordinates characterize the dispersion of the hand's key points in the two gesture images, the dispersion of the hand's key points remains consistent regardless of how the distance between the user's hand and the image acquisition device changes or how the acquired gesture image is scaled. Therefore, by standardizing the original coordinate information using the mean and variance, the obtained hand distance has the ability to resist image scaling and can also eliminate interference caused by different distances between the hand and the image acquisition device.

[0082] As can be seen, in this example, the coordinate information for determining the distance between the hands in two frames of gesture images is the coordinate information of the hand's center point after standardization processing using the coordinate mean and standard deviation. Since the coordinate standard deviation and coordinate mean are used to characterize the discreteness of the key points of the hands in two frames of gesture images, even if the distance between the user's hand and the image acquisition device / image acquisition apparatus changes, or the image size of the gesture image changes, the discreteness of the key points of the hands in the gesture image will not change, which helps to improve the reliability of the determined hand distance.

[0083] In one possible example, the coordinate information includes a first coordinate value and a second coordinate value, and the coordinate mean and coordinate standard deviation include: the first mean and the first standard deviation of the first coordinate values ​​of the hand key points in each of the current two-frame gesture images, and the second mean and the second standard deviation of the second coordinate values ​​of the hand key points in each of the current two-frame gesture images;

[0084] The standardization process for the coordinate information of the hand center point in each of the two current gesture images, based on the coordinate mean and the coordinate standard deviation, includes the following steps: For the hand center point in each of the two current gesture images, the following operation is performed: The standardized coordinate information of the current hand center point is determined using the following formula:

[0085]

[0086] in, The first coordinate value of the current hand center point after standardization. The second coordinate value of the current hand center point after standardization is given, x is the first coordinate value of the current hand center point before standardization, y is the second coordinate value of the current hand center point before standardization, μx is the first mean, σx is the first standard deviation, μy is the second mean, and σy is the second standard deviation.

[0087] For example, taking the current two-frame gesture image, including the first image and the second image, as an example, if the first image includes k hand key points, the coordinate information of these k hand key points is as follows:

[0088] (x1, y1), (x2, y2)...(x k-1 y k-1 ), (x k y k );

[0089] The second image includes nk key points for the hand, and the coordinate information of these nk key points is as follows:

[0090] (x k+1 y k+1 ), (x k+2 y k+2 ...(x) n y n );

[0091] Suppose that the coordinates of the center point of the hand in the first image, determined based on the coordinates of k key hand points, are P. C1 (x C1 y C1 Based on the coordinate information of nk key hand points, the coordinate information of the center point of the hand in the second image is P. C2 (x C2 y C2 ).

[0092] The first standard deviation is determined based on the coordinate information of the aforementioned k key hand points and nk key hand points. Second standard deviation First mean Second mean

[0093] Because the coordinates of the hand's center point in the first image after standardization are...

[0094] The coordinates of the center point of the hand in the standardized second image are as follows:

[0095] By substituting the specific values ​​of the first standard deviation σx, the second standard deviation σy, the first mean μx, and the second mean μy into the coordinate information of the two hand center points after the above standardization process, the distance between the hands in the current two frames of gesture images can be determined:

[0096]

[0097] Using the method described above for determining hand distance based on the coordinates of the hand's center point after standardization, even if the distance between the user's hand and the image acquisition device changes, or the image size is scaled, the coordinates of the hand's center point (including the first and second coordinates) in the first two frames of the gesture image after standardization will change to a multiple of 'a', for example, P. C1 (x C1 y C1 The coordinate information changes to (ax) C1 ,ay C1 ), P C2 (x C2 y C2 The coordinate information changes to (ax) C2 ,ay C2 );

[0098] At this point, the first standard deviation becomes aσx, the second standard deviation becomes aσy, the first mean remains aμx, and the second mean remains aμy;

[0099] That is, after the coordinate information is changed to a multiple of the original, the coordinate information of the hand center point after standardization is as follows:

[0100]

[0101] That is, after the coordinate information changes to a multiple of 'a', the distance to the hand can be obtained:

[0102]

[0103] In other words, after standardization, the determined hand distance remains the same even if the distance between the user's hand and the image acquisition device / appearance acquisition unit changes, or if the image size is scaled.

[0104] As can be seen, in this example, the coordinate information of the standardized hand center point determined by the preset formula is not affected by the different distances of the user's hand from the image acquisition device / image acquisition unit, nor by the scaling of the image size, which helps to improve the reliability of the hand distance determined based on this coordinate information.

[0105] This application can divide electronic devices into functional units based on the above method examples. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0106] Figure 3A This is a functional unit block diagram of a gesture recognition device provided in an embodiment of this application. The gesture recognition device 30 can be applied to, for example... Figure 1A or Figure 1B In the illustrated electronic device, the gesture recognition device 30 includes:

[0107] The first acquisition unit 301 is used to acquire multiple frames of gesture images;

[0108] The second acquisition unit 302 is used to acquire the position information of key hand points in each frame of the multi-frame gesture image;

[0109] The first determining unit 303 is used to determine the distance between the hands in every two adjacent frames of the gesture images based on the location information, thereby obtaining multiple distances corresponding to the multi-frame gesture images.

[0110] The second determining unit 304 is used to determine the target model from the preset dynamic gesture recognition model and static gesture recognition model based on the multiple distances;

[0111] The calling unit 305 is used to call the target model to perform gesture recognition on the multi-frame gesture images.

[0112] In one possible example, the second determining unit 304 is specifically used to: if the number of distances greater than a preset threshold among the plurality of distances is greater than a first preset number, then determine the dynamic gesture recognition model as the target model; otherwise, determine the static gesture recognition model as the target model.

[0113] In one possible example, the gesture recognition device 30 further includes: a third determining unit, configured to, after determining the dynamic gesture recognition model as the target model, call the dynamic gesture recognition model to perform gesture recognition on the multi-frame gesture images; if no dynamic gesture is recognized, then re-determine the static gesture recognition model as the target model.

[0114] In one possible example, when the target model is the dynamic gesture recognition model, the calling unit 305 is specifically used to: obtain a second preset number of gesture images from the multi-frame gesture images whose acquisition time is closest to the current time, the second preset number being greater than 1; and call the dynamic gesture recognition model to perform gesture recognition on the second preset number of gesture images.

[0115] In one possible example, when the target model is the static gesture recognition model, the calling unit 305 is specifically used to: obtain the gesture image whose acquisition time is closest to the current time from the multi-frame gesture images; and call the static gesture recognition model to perform gesture recognition on the gesture image of the frame.

[0116] In one possible example, the location information includes coordinate information; the first determining unit 303 is specifically configured to: perform the following operations for each frame of the multi-frame gesture images: determine the coordinate information of the hand center point in the current frame gesture image based on the coordinate information of the hand key points in the current frame gesture image; perform the following operations for every two frames of gesture images: determine the coordinate mean and coordinate standard deviation based on the coordinate information of the hand key points in each frame gesture image in the current two frames gesture images; standardize the coordinate information of the hand center point in each frame gesture image in the current two frames gesture images based on the coordinate mean and the coordinate standard deviation; and determine the distance of the hand in the current two frames gesture images based on the standardized coordinate information of the hand center point in each frame gesture image in the current two frames gesture images.

[0117] In one possible example, the coordinate information includes a first coordinate value and a second coordinate value, and the coordinate mean and coordinate standard deviation include: the first mean and first standard deviation of the first coordinate values ​​of the hand key points in each frame of the current two-frame gesture images, and the second mean and second standard deviation of the second coordinate values ​​of the hand key points in each frame of the current two-frame gesture images; in terms of standardizing the coordinate information of the hand center point in each frame of the current two-frame gesture images according to the coordinate mean and the coordinate standard deviation, the first determining unit 303 is specifically used to: perform the following operation on the hand center point in each frame of the current two-frame gesture images: determine the standardized coordinate information of the current hand center point using the following formula:

[0118]

[0119] in, The first coordinate value of the current hand center point after standardization. The second coordinate value of the current hand center point after standardization is given, x is the first coordinate value of the current hand center point before standardization, y is the second coordinate value of the current hand center point before standardization, μx is the first mean, σx is the first standard deviation, μy is the second mean, and σy is the second standard deviation.

[0120] When using integrated units, the functional unit composition block diagram of the gesture recognition device provided in this application embodiment is as follows: Figure 3B As shown. In Figure 3B The gesture recognition device includes a processing module 310 and a communication module 311. The processing module 310 controls and manages the actions of the gesture recognition device, for example, the steps executed by the first acquisition unit 301, the second acquisition unit 302, the first determination unit 303, the second determination unit 304, and the calling unit 305, and / or other processes for executing the techniques described herein. The communication module 311 supports interaction between the gesture recognition device and other devices. Figure 3B As shown, the gesture recognition device may also include a storage module 312, which is used to store the program code and data of the gesture recognition device.

[0121] The processing module 310 can be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 311 can be a transceiver, RF circuitry, or a communication interface, etc. The storage module 312 can be a memory.

[0122] All relevant content in each scenario involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here. All of the above gesture recognition devices can perform the above... Figure 2 The steps performed by the electronic device in the gesture recognition method shown.

[0123] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.

[0124] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may include an electronic device.

[0125] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0126] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0128] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0131] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0132] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A gesture recognition method, characterized by, The method comprises the following steps: acquiring a plurality of gesture images; acquiring position information of hand key points in each gesture image in the plurality of gesture images; determining distances of a hand in each two gesture images adjacent in time from the plurality of gesture images according to the position information, to obtain a plurality of distances corresponding to the plurality of gesture images; determining a target model from a preset dynamic gesture recognition model and a static gesture recognition model according to the plurality of distances; calling the target model to perform gesture recognition on the plurality of gesture images; wherein the determining the target model from the preset dynamic gesture recognition model and the static gesture recognition model according to the plurality of distances comprises the following steps: if a number of distances greater than a preset threshold in the plurality of distances is greater than a first preset number, determining that the dynamic gesture recognition model is the target model; otherwise, determining that the static gesture recognition model is the target model.

2. The method of claim 1, after the determining that the dynamic gesture recognition model is the target model, the method further comprises the following steps: calling the dynamic gesture recognition model to perform gesture recognition on the plurality of gesture images; if a dynamic gesture is not recognized, re-determining the static gesture recognition model as the target model.

3. The method of claim 1 or 2, in a case where the target model is the dynamic gesture recognition model, the calling the target model to perform gesture recognition on the plurality of gesture images comprises the following steps: acquiring a second preset number of gesture images closest to a current time in time from the plurality of gesture images, the second preset number being greater than 1; calling the dynamic gesture recognition model to perform gesture recognition on the second preset number of gesture images.

4. The method of claim 1 or 2, in a case where the target model is the static gesture recognition model, the calling the target model to perform gesture recognition on the plurality of gesture images comprises the following steps: acquiring one gesture image closest to a current time in time from the plurality of gesture images; calling the static gesture recognition model to perform gesture recognition on the one gesture image.

5. The method of claim 1 or 2, the position information comprises coordinate information; and the determining distances of a hand in each two gesture images adjacent in time from the plurality of gesture images according to the position information comprises the following steps: for each gesture image in the plurality of gesture images, performing the following operations: determining coordinate information of a hand center point in the current gesture image according to coordinate information of hand key points in the current gesture image; for each two gesture images, performing the following operations: determining a coordinate mean and a coordinate standard deviation according to coordinate information of hand key points in each gesture image in the current two gesture images; performing standardization processing on coordinate information of a hand center point in each gesture image in the current two gesture images according to the coordinate mean and the coordinate standard deviation. According to the coordinate information of the hand center point in each of the two gesture images after the standardization processing, the distance of the hand in the two gesture images is determined.

6. The method of claim 5, the coordinate information comprising a first coordinate value and a second coordinate value, the coordinate mean and coordinate standard deviation comprising: The first mean and the first coordinate standard deviation of the first coordinate value of the hand key point in each of the two gesture images, and the second mean and the second standard deviation of the second coordinate value of the hand key point in each of the two gesture images; The standardization processing of the coordinate information of the hand center point in each of the two gesture images according to the coordinate mean and the coordinate standard deviation includes the following steps: The following operations are performed on the hand center point in each of the two gesture images: The coordinate information of the current hand center point after the standardization processing is determined by the following formula: wherein, is a first coordinate value of the current hand center point after normalization processing, is a second coordinate value of the current hand center point after normalization processing, x is a first coordinate value of the current hand center point before normalization processing, and y is a second coordinate value of the current hand center point before normalization processing, is the first mean value, is the first standard deviation, is the second mean value, is the second standard deviation.

7. A gesture recognition apparatus, characterized by It includes: A first acquisition unit is configured to acquire a plurality of gesture images. A second acquisition unit is configured to acquire position information of a hand key point in each of the plurality of gesture images. A first determination unit is configured to determine a distance of a hand in each of two gesture images adjacent in time according to the position information, and obtain a plurality of distances corresponding to the plurality of gesture images. A second determination unit is configured to determine a target model from a preset dynamic gesture recognition model and a static gesture recognition model according to the plurality of distances. A calling unit is configured to call the target model to perform gesture recognition on the plurality of gesture images. The second determination unit is specifically configured to: if the number of distances greater than a preset threshold in the plurality of distances is greater than a first preset number, determine that the dynamic gesture recognition model is the target model; otherwise, determine that the static gesture recognition model is the target model.

8. An electronic device, comprising: A processor, a memory, a communication interface, and one or more programs are included, the one or more programs are stored in the memory and are configured to be executed by the processor, and the programs include instructions for performing steps in the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer program for electronic data exchange is stored, wherein the computer program causes a computer to perform steps in the method of any one of claims 1-6.

Images