A toy robot point reading control method based on gesture recognition

Abstract

The present application discloses a toy robot point reading control method based on gesture recognition, which relates to the field of image recognition technology. The toy robot point reading control method based on gesture recognition includes: obtaining multiple time-series gesture recognition images; obtaining multiple super-pixel blocks based on each time-series gesture recognition image; obtaining gesture information coefficients corresponding to each super-pixel block based on multiple super-pixel blocks; obtaining multiple connected domains from multiple super-pixel blocks based on the gesture information coefficients; obtaining a first connected domain based on multiple connected domains; and controlling the toy robot based on the current gesture recognition image. Compared with the prior art, the segmentation process of the present application reduces the interference of complex lighting and background, avoids the advance entry of hand features, and avoids the situation where segmentation cannot be recognized under different imaging conditions, that is, it improves the recognition accuracy under complex backgrounds and reduces the time consumption of gesture recognition.

Description

Technical Field

[0001] The present application relates to the technical field of image recognition, and in particular to a toy robot point reading control method based on gesture recognition. Background Art

[0002] The application of gesture recognition technology in toy robots is an innovative direction in the field of human-computer interaction (HCI). It uses computer vision and machine learning algorithms to enable robots to understand and respond to human gestures, thereby achieving a more natural and intuitive control method. As an entertainment and educational tool for children or adults, toy robots can greatly improve user experience, increase interactivity and fun by integrating gesture recognition technology.

[0003] The current robot gesture recognition mainly relies on computer vision and machine learning algorithms, which recognize and analyze interactive gestures by training deep learning models for gesture recognition, and further control the robot to execute corresponding instructions. However, for toy robots, the processor performance they carry is usually low, and directly segmenting and recognizing the collected images may cause the processing time to be too long. In addition, direct recognition may cause misrecognition or missed recognition in complex backgrounds. Summary of the invention

[0004] The purpose of this application is to provide a toy robot point reading control method based on gesture recognition, so as to solve the technical problem of how to accurately recognize interactive gestures in application scenarios where processor performance is limited.

[0005] To achieve the above objectives, this application provides the following technical solutions:

[0006] A toy robot point reading control method based on gesture recognition, comprising:

[0007] Acquire multiple time-sequential gesture recognition images;

[0008] Based on each temporal gesture recognition image, a plurality of super pixel blocks are obtained;

[0009] Based on multiple superpixel blocks, obtaining gesture information coefficients corresponding to each superpixel block one by one; the gesture information coefficients are at least used to indicate the probability that a gesture exists in the corresponding superpixel block;

[0010] Based on the gesture information coefficient, a plurality of connected domains are obtained from a plurality of superpixel blocks;

[0011] Based on the multiple connected domains, a first connected domain is obtained; the first connected domain is a connected domain having the highest average value of gesture information coefficients of each superpixel block in each connected domain;

[0012] Based on the first connected domain, obtaining a current gesture recognition image;

[0013] The toy robot is controlled based on the current gesture recognition image.

[0014] As a specific solution in the technical solution of the present application, the method of obtaining gesture information coefficients corresponding to each superpixel block based on multiple superpixel blocks includes:

[0015] Based on multiple superpixel blocks, a first superpixel block and a first sequential gesture recognition image are acquired; wherein the first superpixel block is any one of the superpixel blocks; and the first sequential gesture recognition image is a sequential gesture recognition image corresponding to the first superpixel block;

[0016] Based on the first superpixel block, obtaining first centroid coordinates;

[0017] Based on the first time-series gesture recognition image, acquiring the image center coordinates;

[0018] Acquire a first distance based on the first centroid coordinates and the image center coordinates;

[0019] Based on the first distance, gesture information coefficients corresponding to each superpixel block are obtained.

[0020] As a specific solution in the technical solution of the present application, the step of obtaining gesture information coefficients corresponding to each superpixel block based on the first distance includes:

[0021] Based on each of the time-sequence gesture recognition images, acquiring a plurality of second time-sequence gesture recognition images within a preset time and whose time sequence is after the first time-sequence gesture recognition image;

[0022] Based on a plurality of second sequential gesture recognition images, obtaining second super pixel blocks corresponding to each second sequential gesture recognition image one by one; the second super pixel block is a super pixel block having the highest structural similarity with the first super pixel block in the corresponding second sequential gesture recognition image;

[0023] Based on the plurality of second super pixel blocks, obtaining second centroid coordinates corresponding one-to-one to each second super pixel block;

[0024] Based on the first centroid coordinate and each second centroid coordinate, a plurality of line segments are acquired; the starting point of each line segment is the first centroid coordinate, and the end point is any one of the second centroid coordinates;

[0025] Based on multiple line segments, multiple slopes are obtained;

[0026] Based on multiple slopes, obtain the slope variance;

[0027] Based on the slope variance and the first distance, a first coefficient is obtained; the first coefficient is at least used to indicate the probability that the object corresponding to the first super pixel block is a non-linear motion object;

[0028] Based on the first coefficient, gesture information coefficients corresponding to each superpixel block are obtained.

[0029] As a specific solution in the technical solution of the present application, the calculation formula for obtaining the first coefficient based on the slope variance and the first distance is as follows:

[0030] ;

[0031] in, represents the first coefficient; represents the first distance; represents the slope variance; Represents a mapping function used to map the value in the brackets to the interval [0,1]; b is a positive number greater than 0.

[0032] As a specific solution in the technical solution of the present application, the step of obtaining gesture information coefficients corresponding to each superpixel block based on the first coefficients includes:

[0033] Based on the first time-series gesture recognition image, obtaining a plurality of third super pixel blocks; the third super pixel block is any super pixel block in the first time-series gesture recognition image;

[0034] Arrange the third super pixel blocks from large to small according to the first coefficients;

[0035] Based on each third superpixel block, an average variance and a fourth superpixel block are obtained; the average variance is the average value of the variances of pixel values ​​in each third superpixel block; the fourth superpixel block is any third superpixel block that is ranked first and has a corresponding pixel value variance greater than the average variance;

[0036] Based on the fourth superpixel block, obtaining a first pixel mean and a first variance; the first pixel mean is an average value of R channel pixel values ​​in each pixel point in the fourth superpixel block; the first variance is a variance of V channel pixel values ​​in each pixel point in the fourth superpixel block;

[0037] Obtaining a second coefficient based on the first pixel mean and the first variance;

[0038] Based on the second coefficient, gesture information coefficients corresponding to each superpixel block are obtained.

[0039] As a specific solution in the technical solution of the present application, the step of obtaining gesture information coefficients corresponding to each superpixel block based on the second coefficients includes:

[0040] Obtaining a plurality of texture pixel points from the fourth super pixel block based on an edge algorithm;

[0041] Based on each texture pixel point, a plurality of second variances are obtained; the second variances are variances of the coordinates of each texture pixel point;

[0042] Based on the second variance and the second coefficient, obtaining a third coefficient;

[0043] Based on the third coefficient, gesture information coefficients corresponding to each superpixel block are obtained.

[0044] As a specific solution in the technical solution of the present application, the calculation formula for obtaining the third coefficient based on the second variance and the second coefficient is as follows:

[0045] ;

[0046] in, represents the third coefficient; represents the first pixel mean; represents the first variance; represents the second variance.

[0047] As a specific solution in the technical solution of the present application, the step of obtaining gesture information coefficients corresponding to each superpixel block based on the third coefficient includes:

[0048] Based on each third coefficient, obtaining a coefficient average value; the coefficient average value is the average value of the maximum value and the minimum value of each third coefficient;

[0049] Based on the first super pixel block, obtaining a fixed coefficient; the fixed coefficient is a third coefficient corresponding to the first super pixel block;

[0050] Based on the coefficient average and the fixed coefficient, a gesture information coefficient corresponding to the first superpixel block is obtained.

[0051] As a specific solution in the technical solution of the present application, the calculation formula for obtaining the gesture information coefficient corresponding to the first super pixel block based on the coefficient average value and the fixed coefficient is as follows:

[0052] ;

[0053] in, represents a gesture information coefficient corresponding to the first superpixel block; represents the coefficient mean; represents a fixed coefficient of the first superpixel block; Indicates the absolute value. A positive number greater than 0.

[0054] As a specific solution in the technical solution of the present application, the method of obtaining multiple connected domains from multiple superpixel blocks based on the gesture information coefficient includes:

[0055] Based on each superpixel block, a plurality of fifth superpixel blocks are obtained; the fifth superpixel blocks are superpixel blocks whose sequence numbers are in the top 50% of the gesture information coefficients in each superpixel block, which are sorted from large to small;

[0056] Based on the plurality of fifth superpixel blocks, obtaining a plurality of connected domains;

[0057] The step of acquiring a first connected domain based on the multiple connected domains includes:

[0058] Based on each connected domain, a plurality of second connected domains are obtained; the second connected domain is any connected domain having the fourth super pixel block in each connected domain;

[0059] Based on the multiple second connected domains, a third connected domain and a fourth connected domain are obtained; the third connected domain is any connected domain among the second connected domains; the fourth connected domain is any connected domain among the second connected domains that is different from the third connected domain;

[0060] Based on the third connected domain and the fourth connected domain, super pixel blocks within a range enclosed by tangent lines are selected for combination until the first connected domain is obtained.

[0061] Compared with the prior art, the beneficial effects of this application are:

[0062] After acquiring multiple frames of sequential gesture recognition images and dividing them into blocks, the present application uses the frame difference method combined with the characteristics of hand movement to analyze the key hand area (i.e., the fourth superpixel block), and then evaluates the image features of the palm and back of the hand, evaluates the palm and back of the hand features contained in other superpixel blocks, and compares them with the image features of the fourth superpixel block, extracts the hand information area (i.e., the first connected domain) in the sequential gesture recognition image, and inputs the recognition model to output the point reading control information. Compared with the prior art, the segmentation process reduces the interference of complex lighting and background, avoids the advance entry of hand features, and avoids the situation where segmentation cannot be recognized under different imaging conditions, which improves the recognition accuracy under complex backgrounds and reduces the time consumption of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flow chart of a toy robot point reading control method based on gesture recognition proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] The terms "first", "second", etc. in the description of the embodiments of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. For example, the first temporal gesture recognition image and the second temporal gesture recognition image proposed below are different images. It should be understood that the images used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules that appear in the embodiments of the present application is only a logical division. There may be other division methods when implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in the embodiments of the present application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0066] In order to solve the technical problems in the prior art that the segmentation and recognition of the collected images by the existing toy robot may cause too long processing time or may cause misrecognition or missed recognition, the present application proposes an embodiment of a toy robot point reading control method based on gesture recognition, such as Figure 1 As shown, the toy robot point reading control method based on gesture recognition includes steps S01 to S07.

[0067] Step S01: Acquire a plurality of time-series gesture recognition images.

[0068] It should be clear that when the toy robot is in standby mode, the sensor for detecting gesture movements (e.g., infrared sensor) usually operates in low power consumption mode, and the sensor collects data at a lower frequency, but can still capture the initial movement of the gesture. Once the sensor detects the initial movement of the gesture (e.g., the start of a specific gesture or a significant change in movement), the robot will wake up from the standby state, increase the sampling rate and processing power of the sensor, and enter the full-function recognition mode. In the full-function recognition mode, multiple frames of gesture images (also known as time-series gesture recognition images) are continuously collected by a high-resolution industrial camera, and the image format is RGB image; the HSV channel information of the image is obtained by calculating the HSV pixel values ​​of different pixels. That is to say, in this embodiment, obtaining multiple time-series gesture recognition images is a mature technology and will not be described in detail here. It should be noted that in this embodiment, multiple time-series gesture recognition images can be directly acquired by the camera or read from the memory.

[0069] Step S02: Based on each time-series gesture recognition image, a plurality of super pixel blocks are obtained.

[0070] It should be clear that for the sequential gesture recognition images collected by the toy robot, the screen often contains multiple objects, and these objects may have different image texture features. When actually using gestures to control the toy robot, the hand usually does not have a background part similar to its texture features, so the image can be segmented and preprocessed by combining the texture features of different regions. Superpixel segmentation is used to divide the sequential gesture recognition image into multiple superpixel blocks. Superpixel blocks are small areas composed of a series of pixels with adjacent positions and similar features such as color, brightness, and texture. Most of these small areas retain effective information for further image segmentation and generally do not destroy the boundary information of objects in the image. After obtaining the image blocks using local image features, the image information of different areas is compared in combination with the image features of the hand. For the hand area, superpixel segmentation can usually obtain the complete hand boundary, reducing the probability of losing the hand boundary when segmenting directly through hand image feature analysis.

[0071] It should be noted that superpixel segmentation of images is a mature technology and will not be described in detail here.

[0072] Step S03: Based on multiple super pixel blocks, obtain gesture information coefficients corresponding to each super pixel block.

[0073] It should be clear that in the gesture recognition process, after the motion is detected by the action detection sensor, the camera on the toy robot is used to collect the sequential gesture recognition image. When the user makes a key gesture, the hand is usually in motion, that is, the position of the pixel points related to the gesture in the image is constantly changing over time. Therefore, the probability of the hand area can be evaluated by obtaining the difference pixels in different sequential gesture recognition images and combining the characteristics of the gesture formation process. For a certain frame of sequential gesture recognition image, the frame difference pixel points in the current frame and the previous frame sequential gesture recognition image are marked by the frame difference method. These pixel points represent objects whose positions have changed in time between the previous frame sequential gesture recognition image and the current frame sequential gesture recognition image, and these pixel points are marked as moving pixel points. The connected domain information of the moving pixel points is obtained, and the labels of the moving pixel points that form the connected domain composed of a single pixel point are removed. The connected domains of different moving pixels are close in spatial position, usually representing the motion information of the same object, and the superpixel blocks corresponding to these connected domains can usually also represent the same object. Multiple superpixel blocks involved in a connected domain are merged. However, for the images captured by the toy robot, the moving objects in the captured image may also contain other objects. Therefore, it is also necessary to analyze the hand information contained in the moving pixels.

[0074] It should be noted that the embodiments of the present application form a connected domain by merging each superpixel block, and each connected domain can correspond to an object. Merging each superpixel block to form a connected domain is a mature technology and will not be described here. It is easy to understand that, as can be seen from the foregoing, after superpixel segmentation, multiple superpixel blocks can be obtained, some of which are superpixel blocks of the user's hands, and some are not. In the application scenario of interactive gesture recognition, it is only necessary to obtain the superpixel blocks related to the user's gesture. That is to say, in this embodiment, the gesture information coefficient is at least used to represent the probability of having a gesture in the corresponding superpixel block.

[0075] In an embodiment of the present application, any suitable coefficient may be used as the gesture information coefficient of each superpixel block. For example, in an embodiment of the present application, step S03, based on multiple superpixel blocks, obtains gesture information coefficients corresponding to each superpixel block one by one, including steps S08 to S12.

[0076] Step S08: Based on the multiple super pixel blocks, a first super pixel block and a first temporal gesture recognition image are acquired.

[0077] It should be understood that, in this embodiment, the first super pixel block is any one of the super pixel blocks; and the first temporal gesture recognition image is a temporal gesture recognition image corresponding to the first super pixel block.

[0078] It should be noted that in the process of controlling a toy robot using gestures, the user points his hand at the camera on the toy machine to make gesture information, which is consistent with the general use of visual detection. In other words, when controlling a toy robot, the hand usually appears in the center of the picture, that is, the superpixel block corresponding to the gesture must be located at the center of the corresponding sequential gesture recognition image. Even if the gesture information is not correctly recognized, the toy robot will usually readjust the hand position to align it with the camera area. Based on this, the probability of whether each superpixel block is a user gesture can be determined by whether the superpixel block is located at the center of the corresponding sequential gesture recognition image.

[0079] Step S09: Based on the first superpixel block, obtain the first centroid coordinates.

[0080] It should be clear that based on a superpixel block, obtaining the corresponding centroid coordinates is a mature technology and will not be described in detail here.

[0081] Step S10: based on the first time-series gesture recognition image, obtaining the image center coordinates.

[0082] It should be clear that in the application scenario of gesture recognition, the center coordinates of each sequential gesture recognition image are consistent, and obtaining the center coordinates of a sequential gesture recognition image is also a mature technology, which will not be elaborated here.

[0083] Step S11: Acquire a first distance based on the first centroid coordinates and the image center coordinates.

[0084] It should be clear that obtaining the distance corresponding to the two coordinate points based on the two coordinate points is also a mature technology and will not be described in detail here.

[0085] Step S12: Based on the first distance, obtain gesture information coefficients corresponding to each super pixel block.

[0086] It should be clear that in the embodiment of the present application, the first distance can be used as a gesture information coefficient. It is easy to understand that if the first distance is smaller, the probability that the information contained in the super pixel block corresponding to the first distance is gesture information is greater.

[0087] It should be noted that since it may be difficult to meet the robustness of the algorithm by evaluating the hand area only through the consistent detection features of visual detection, it is also necessary to combine the characteristics of the gesture formation process. For example, in the process of making gestures, due to the limitations of joints, the motion trajectory of gesture movements is usually non-linear. Therefore, the gesture formation process needs to be analyzed in combination with multiple time-series gesture recognition images. If the motion trajectory of a superpixel block is a straight line, the superpixel block is most likely unrelated to the gesture; if the motion trajectory of a superpixel block is a non-straight line, the superpixel block is most likely related to the gesture. In an embodiment of the present application, in order to further improve the accuracy of the gesture information coefficient, in one embodiment of the present application, step S12, based on the first distance, obtains the gesture information coefficient corresponding to each superpixel block one by one, including steps S13 to S20.

[0088] Step S13: based on each of the time-sequence gesture recognition images, a plurality of second time-sequence gesture recognition images within a preset time and whose time sequence is after the first time-sequence gesture recognition image are acquired.

[0089] It should be noted that, in this embodiment, the preset time can be set according to the requirements, for example, the preset time can be 1 second or 2 seconds, etc. The second sequential gesture recognition images are adjacent in time sequence, and the time sequence of each second sequential gesture recognition image is located after the first sequential gesture recognition image.

[0090] Step S14: based on the plurality of second temporal gesture recognition images, obtaining second super pixel blocks corresponding to the respective second temporal gesture recognition images.

[0091] It should be clear that, in the process of the user making a gesture, the position of each finger will change in a short period of time, but the morphological characteristics of each finger itself will not change significantly, so the second super-pixel block corresponding to the second temporal gesture recognition image and the first super-pixel block can be determined by the similarity between each second temporal gesture recognition image and the first super-pixel block. That is, in this embodiment, the second super-pixel block is the super-pixel block with the highest structural similarity to the first super-pixel block in the corresponding second temporal gesture recognition image.

[0092] Step S15: Based on the multiple second super pixel blocks, obtain the second centroid coordinates corresponding to each second super pixel block one by one.

[0093] It should be clear that based on a superpixel block, obtaining the corresponding centroid coordinates is a mature technology and will not be described in detail here.

[0094] Step S16: Based on the first centroid coordinates and each second centroid coordinates, obtain multiple line segments.

[0095] As can be seen from the foregoing, in the process of making gestures, the movement of each finger must be non-linear (that is, each finger rotates around the joint, etc.). In this embodiment, the starting point of each line segment is the first centroid coordinate, and the end point is any second centroid coordinate among the second centroid coordinates. If the movement of each finger is definitely non-linear, the slopes of each line segment must be inconsistent.

[0096] Step S17: Based on the multiple line segments, obtain multiple slopes.

[0097] It should be clear that obtaining the slope corresponding to a line segment based on the line segment is a mature technology and will not be elaborated here.

[0098] Step S18: Based on the multiple slopes, obtain the slope variance.

[0099] It should be clear that obtaining the variance of certain values ​​based on these values ​​is a mature technology and will not be described in detail here.

[0100] Step S19: Obtain a first coefficient based on the slope variance and the first distance.

[0101] It should be clear that, in this embodiment, the first coefficient is at least used to represent the probability that the object corresponding to the first super pixel block is a non-linear moving object. In an embodiment of the present application, the first coefficient can be the value corresponding to the slope variance minus the value corresponding to the first distance, or the first coefficient is the value corresponding to the slope variance divided by the value corresponding to the first distance. In a specific embodiment of the present application, step S19, based on the slope variance and the first distance, obtains the calculation formula of the first coefficient as follows:

[0102] ;

[0103] in, represents the first coefficient; represents the first distance; represents the slope variance; Represents a mapping function used to map the value in the brackets to the interval [0,1]; b is a positive number greater than 0.

[0104] As we can see from the previous article, if the slope variance The larger the first distance is, the more likely the corresponding superpixel block is the superpixel block of the user's hand; if the first distance is smaller, the more likely the corresponding superpixel block is the superpixel block of the user's hand. That is to say, in this embodiment, if the first coefficient The larger the value is, the more likely the corresponding superpixel block is the superpixel block of the user's hand.

[0105] Step S20: Based on the first coefficients, a gesture information coefficient corresponding to each superpixel block is obtained.

[0106] It should be clear that in this embodiment, the first coefficient can be directly used as the gesture information coefficient. It should be noted that for the hand area in the time-series gesture recognition image, the time of adjacent frames is short, and the area involved in the moving pixel point may not be the complete hand area. Since the image features of the front and back sides of the hand have certain differences, and the front and back of the hand cannot be predicted in advance, if it is necessary to accurately understand whether the corresponding super-pixel block is the super-pixel block of the user's hand, it is necessary to combine further comparative analysis. It is easy to understand that although the color of the hand skin usually has a more consistent tone, it is difficult to judge the hand area in the image by preset data in application scenarios with different skin colors and different lighting conditions. In order to obtain the gesture information coefficient that accurately characterizes each super-pixel block, in one embodiment of the present application, step S20, based on the first coefficient, obtains the gesture information coefficient corresponding to each super-pixel block one by one, including steps S21 to S26.

[0107] Step S21: acquiring a plurality of third super pixel blocks based on the first temporal gesture recognition image.

[0108] It should be clear that, in this embodiment, the third super pixel block is any super pixel block in the first temporal gesture recognition image. That is, in the embodiment of the present application, the third super pixel block can be any super pixel block in the first temporal gesture recognition image.

[0109] Step S22: Arrange the third super pixel blocks from large to small according to the first coefficients.

[0110] As can be seen from the foregoing, the larger the first coefficient is, the more likely the corresponding third superpixel block is to be the superpixel block corresponding to the user's hand. In this embodiment, the third superpixel blocks are arranged from large to small according to the first coefficient, that is, the superpixel blocks arranged in the front are more likely to be the superpixel blocks corresponding to the user's hand.

[0111] Step S23: Based on each third super pixel block, obtain the average variance and the fourth super pixel block.

[0112] It should be noted that the texture and color of the front (i.e. palm) and back (i.e. back) of the human hand are somewhat different. Generally speaking, the color of the palm is redder than that of the back, and the number and distribution discreteness of the palm texture of the palm image are much higher than those of the back area. To evaluate the hand information of each superpixel block, the R channel pixel value and internal texture features of the fourth superpixel block can be compared and analyzed.

[0113] In this embodiment, the average variance is the average value of the variances of the pixel values ​​in each third superpixel block, and the fourth superpixel block is any third superpixel block that is ranked first and has a corresponding pixel value variance greater than the average variance. That is, in this embodiment, the pixel value difference corresponding to each pixel point in the fourth superpixel block is large, and the first coefficient corresponding to the fourth superpixel block is ranked first, that is, the fourth superpixel block is likely to include both the palm and the back of the hand.

[0114] Step S24: Based on the fourth super pixel block, obtain a first pixel mean and a first variance.

[0115] It should be clear that, in the present embodiment, the first pixel mean is the average value of the R channel pixel values ​​in each pixel point in the fourth super pixel block. The first variance is the variance of the V channel pixel values ​​in each pixel point in the fourth super pixel block. Based on multiple values, obtaining the average value of these values ​​is a mature technology, which will not be described in detail here. Based on multiple values, obtaining the variance of these values ​​is also a mature technology, which will not be described in detail here.

[0116] Step S25: Obtain a second coefficient based on the first pixel mean and the first variance.

[0117] It should be clear that in the embodiment of the present application, the second coefficient may be the product of the value corresponding to the first pixel mean and the value corresponding to the first variance. The second coefficient may also be the sum of the value corresponding to the first pixel mean and the value corresponding to the first variance.

[0118] Step S26: Based on the second coefficient, obtain the gesture information coefficient corresponding to each super pixel block one by one.

[0119] Specifically, step S26, based on the second coefficient, obtains the gesture information coefficient corresponding to each super pixel block one by one, including steps S27 to S30.

[0120] Step S27: obtaining a plurality of texture pixel points from the fourth super pixel block based on an edge algorithm.

[0121] It should be clear that obtaining multiple texture pixel points from a super pixel block based on edge algorithm is a mature technology and will not be described in detail here.

[0122] Step S28: obtaining a plurality of second variances based on each texture pixel point.

[0123] In this embodiment, the second variance is the variance of the coordinates of each texture pixel point.

[0124] Step S29: obtaining a third coefficient based on the second variance and the second coefficient;

[0125] It should be clear that the third coefficient in the present application can be the product of the second variance and the second coefficient, or can be the sum of the second variance and the second coefficient. In a specific embodiment of the present application, step S29, based on the second variance and the second coefficient, obtains the calculation formula of the third coefficient as follows:

[0126] ;

[0127] in, represents the third coefficient; represents the first pixel mean; represents the first variance; represents the second variance.

[0128] Step S30: Based on the third coefficient, obtain the gesture information coefficient corresponding to each super pixel block one by one.

[0129] It should be noted that for superpixel blocks that represent more palm information, The value is larger, while the back of the hand information The value is smaller. Select the maximum and minimum midpoint As the dividing point between the palm and the back of the hand, Divided into two intervals, The larger one is the palm area. The smaller one is the back of hand interval. That is to say, after obtaining the hand information feature (i.e., the third coefficient), the amount of hand information contained in each superpixel block can be obtained by comparing the third coefficients of other superpixel blocks. Specifically, step S30, based on the third coefficient, obtains the gesture information coefficient corresponding to each superpixel block, including steps S31 to S33.

[0130] Step S31: Based on each third coefficient, obtain the coefficient average value.

[0131] It should be noted that, in this embodiment, the coefficient average value is the average value of the maximum value and the minimum value of each third coefficient. Based on two values, obtaining the average value is a mature technology and will not be described in detail here.

[0132] Step S32: Obtain fixed coefficients based on the first superpixel block.

[0133] It should be clear that the fixed coefficient is the third coefficient corresponding to the first super pixel block. Based on the foregoing, it can be seen that obtaining the third coefficient corresponding to the first super pixel block is as described in step S27 to step S30, which will not be repeated here.

[0134] Step S33: Based on the coefficient average and the fixed coefficient, obtain the gesture information coefficient corresponding to the first super pixel block.

[0135] It should be clear that the closer the fixed coefficient is to the coefficient average, the more hand information the first superpixel block contains. In an embodiment of the present application, the gesture information coefficient may be the difference between the coefficient average and the fixed coefficient. In another embodiment of the present application, in step S33, based on the coefficient average and the fixed coefficient, the calculation formula for obtaining the gesture information coefficient corresponding to the first superpixel block is as follows:

[0136] ;

[0137] in, represents a gesture information coefficient corresponding to the first superpixel block; represents the coefficient mean; represents a fixed coefficient of the first superpixel block; Indicates the absolute value. A positive number greater than 0.

[0138] It should be noted that, in this embodiment, if the fixed coefficient The coefficient mean The closer, the The smaller it is, the smaller the gesture information coefficient is. The larger the gesture information coefficient is, the The larger it is, the more hand information the corresponding superpixel block contains.

[0139] Step S04: Based on the gesture information coefficient, multiple connected domains are obtained from multiple superpixel blocks.

[0140] It should be clear that the hand area usually occupies a smaller screen in the temporal gesture recognition image. In order to further compress the input amount of the recognition model, it is necessary to select a suitable temporal gesture recognition image according to the amount of hand information of different superpixel blocks. In an embodiment of the present application, in order to obtain a suitable temporal gesture recognition image, step S04, based on the gesture information coefficient, obtains multiple connected domains from multiple superpixel blocks, including steps S34 and S35.

[0141] Step S34: Based on each super pixel block, obtain multiple fifth super pixel blocks.

[0142] It should be clear that the fifth superpixel block is a superpixel block whose sequence number is in the top 50% of the gesture information coefficients in each superpixel block. It is easy to understand that, as can be seen from the foregoing, the superpixel block with a larger gesture information coefficient contains more hand information.

[0143] Step S35: Based on the multiple fifth superpixel blocks, obtain multiple connected domains.

[0144] As can be seen from the foregoing, since the fifth superpixel block contains more hand information, the hand information in the connected domain obtained based on multiple fifth superpixel blocks is also more. Based on superpixel blocks, obtaining a connected domain is also a mature technology and will not be described in detail here.

[0145] Step S05: Based on multiple connected domains, a first connected domain is obtained.

[0146] It should be clear that in this embodiment, the first connected domain is the connected domain with the highest average value of gesture information coefficients of each superpixel block in each connected domain. Specifically, step S05, based on multiple connected domains, obtains the first connected domain, including steps S36 to S38.

[0147] Step S36: Based on each connected domain, obtain multiple second connected domains.

[0148] In this embodiment, the second connected domain is any connected domain having the fourth super pixel block in each connected domain. As can be seen from the foregoing, the fourth super pixel block is the super pixel block that is most likely to contain hand information in each time-series gesture recognition image. If the corresponding connected domain has the fourth super pixel block, it means that the connected domain is likely to be the connected domain corresponding to the user's hand.

[0149] Step S37: based on the plurality of second connected domains, obtaining a third connected domain and a fourth connected domain.

[0150] In this embodiment, the third connected domain is any one of the second connected domains, and the fourth connected domain is any connected domain in the second connected domains that is different from the third connected domain. In order to further reduce the number of connected domains, it is also necessary to select the first connected domain from the third connected domain and the fourth connected domain.

[0151] Step S38: Based on the third connected domain and the fourth connected domain, super pixel blocks within the range enclosed by the tangent are selected for combination until the first connected domain is obtained. The first connected domain finally selected is the connected domain containing the most user hand information, which can represent the user's specific gesture and greatly reduce the amount of subsequent data processing.

[0152] Step S06: Based on the first connected domain, a current gesture recognition image is acquired.

[0153] It should be noted that obtaining the corresponding current gesture recognition image based on the connected domain is a mature technology and will not be described in detail here.

[0154] Step S07: Controlling the toy robot based on the current gesture recognition image.

[0155] It should be clear that, based on the current gesture recognition image, controlling the toy robot is also a mature technology and will not be elaborated here.

[0156] It should be clear that the embodiment of the gesture recognition-based toy robot point-reading control method proposed in the present application obtains multiple frames of sequential gesture recognition images and divides them into blocks, then uses the frame difference method in combination with the characteristics of hand movement to analyze the key hand area (i.e., the fourth super-pixel block), and then evaluates the image features of the palm and back of the hand, evaluates the palm and back of the hand features contained in other super-pixel blocks, and compares them with the image features of the fourth super-pixel block, extracts the hand information area (i.e., the first connected domain) in the sequential gesture recognition image, and inputs it into the recognition model, and outputs the point-reading control information. Compared with the prior art, the segmentation process reduces the interference of complex lighting and background, avoids the advance entry of hand features, and avoids the situation where segmentation cannot be recognized under different imaging conditions, that is, improves the recognition accuracy under complex backgrounds and reduces the time consumption of gesture recognition.

[0157] It should be clear that the computer-readable storage media in this application include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technology, read-only compact disc read-only memory, digital versatile disc or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves.

[0158] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the methods, devices and equipment described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0160] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0161] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0163] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0164] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disk), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0165] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles of the present application.

Claims

1. A toy robot point reading control method based on gesture recognition, characterized in that: include: Acquire multiple time-sequential gesture recognition images; Based on each temporal gesture recognition image, a plurality of super pixel blocks are obtained; Based on multiple superpixel blocks, obtaining gesture information coefficients corresponding to each superpixel block one by one; the gesture information coefficients are at least used to indicate the probability that a gesture exists in the corresponding superpixel block; Based on the gesture information coefficient, a plurality of connected domains are obtained from a plurality of superpixel blocks; Based on the multiple connected domains, obtaining a first connected domain; The first connected domain is a connected domain in which the average value of gesture information coefficients of each superpixel block in each connected domain is the highest; Based on the first connected domain, obtaining a current gesture recognition image; Based on the current gesture recognition image, controlling the toy robot; The step of obtaining gesture information coefficients corresponding to each superpixel block based on multiple superpixel blocks includes: Based on multiple superpixel blocks, a first superpixel block and a first sequential gesture recognition image are acquired; wherein the first superpixel block is any one of the superpixel blocks; and the first sequential gesture recognition image is a sequential gesture recognition image corresponding to the first superpixel block; Based on the first superpixel block, obtaining first centroid coordinates; Based on the first time-series gesture recognition image, acquiring the image center coordinates; Acquire a first distance based on the first centroid coordinates and the image center coordinates; Based on the first distance, obtaining gesture information coefficients corresponding to each superpixel block one by one; The acquiring, based on the first distance, gesture information coefficients corresponding to each superpixel block one by one includes: Based on each of the time-sequence gesture recognition images, acquiring a plurality of second time-sequence gesture recognition images within a preset time and whose time sequence is after the first time-sequence gesture recognition image; Based on a plurality of second sequential gesture recognition images, obtaining second super pixel blocks corresponding to each second sequential gesture recognition image one by one; the second super pixel block is a super pixel block having the highest structural similarity with the first super pixel block in the corresponding second sequential gesture recognition image; Based on the plurality of second super pixel blocks, obtaining second centroid coordinates corresponding one-to-one to each second super pixel block; Based on the first centroid coordinate and each second centroid coordinate, a plurality of line segments are acquired; the starting point of each line segment is the first centroid coordinate, and the end point is any one of the second centroid coordinates; Based on multiple line segments, multiple slopes are obtained; Based on multiple slopes, obtain the slope variance; Based on the slope variance and the first distance, a first coefficient is obtained; the first coefficient is at least used to indicate the probability that the object corresponding to the first super pixel block is a non-linear motion object; Based on the first coefficient, a gesture information coefficient corresponding to each superpixel block is obtained; the calculation formula for obtaining the first coefficient based on the slope variance and the first distance is as follows: Wherein, G represents the first coefficient; d represents the first distance; represents the slope variance; exp() represents the mapping function used to map the value in the brackets to the range of [0,1]; b is a positive number; The step of obtaining gesture information coefficients corresponding to each superpixel block based on the first coefficients includes: Based on the first time-series gesture recognition image, obtaining a plurality of third super pixel blocks; the third super pixel block is any super pixel block in the first time-series gesture recognition image; Arrange the third super pixel blocks from large to small according to the first coefficients; Based on each third superpixel block, an average variance and a fourth superpixel block are obtained; the average variance is the average value of the variances of pixel values ​​in each third superpixel block; the fourth superpixel block is any third superpixel block that is ranked first and has a corresponding pixel value variance greater than the average variance; Based on the fourth superpixel block, obtaining a first pixel mean and a first variance; the first pixel mean is an average value of R channel pixel values ​​in each pixel point in the fourth superpixel block; the first variance is a variance of V channel pixel values ​​in each pixel point in the fourth superpixel block; Obtaining a second coefficient based on the first pixel mean and the first variance; Based on the second coefficient, obtaining gesture information coefficients corresponding to each superpixel block one by one; The step of obtaining gesture information coefficients corresponding to each superpixel block based on the second coefficients includes: Obtaining a plurality of texture pixel points from the fourth super pixel block based on an edge algorithm; Based on each texture pixel point, a plurality of second variances are obtained; the second variances are variances of the coordinates of each texture pixel point; Based on the second variance and the second coefficient, obtaining a third coefficient; Based on the third coefficient, obtaining gesture information coefficients corresponding to each superpixel block one by one; The calculation formula for obtaining the third coefficient based on the second variance and the second coefficient is as follows: Wherein, η represents the third coefficient; represents the first pixel mean; represents the first variance; represents the second variance; The step of obtaining gesture information coefficients corresponding to each superpixel block based on the third coefficient includes: Based on each third coefficient, obtaining a coefficient average value; the coefficient average value is the average value of the maximum value and the minimum value of each third coefficient; Based on the first super pixel block, obtaining a fixed coefficient; the fixed coefficient is a third coefficient corresponding to the first super pixel block; Based on the coefficient average and the fixed coefficient, obtaining a gesture information coefficient corresponding to the first superpixel block; The calculation formula for obtaining the gesture information coefficient corresponding to the first super pixel block based on the coefficient average value and the fixed coefficient is as follows: Wherein, H represents the gesture information coefficient corresponding to the first superpixel block; represents the average value of the coefficient; η1 represents the fixed coefficient of the first superpixel block; || represents the absolute value; a is a positive number; Acquiring a second coefficient based on the first pixel mean and the first variance includes: A value obtained by adding or multiplying the first pixel mean and the first variance is used as a second coefficient.

2. The toy robot point reading control method based on gesture recognition according to claim 1 is characterized in that: The method of obtaining a plurality of connected domains from a plurality of superpixel blocks based on the gesture information coefficient includes: Based on each superpixel block, a plurality of fifth superpixel blocks are obtained; the fifth superpixel blocks are superpixel blocks whose sequence numbers are in the top 50% of the gesture information coefficients in each superpixel block, which are sorted from large to small; Based on the plurality of fifth superpixel blocks, obtaining a plurality of connected domains; The step of acquiring a first connected domain based on the multiple connected domains includes: Based on each connected domain, a plurality of second connected domains are obtained; the second connected domain is any connected domain having the fourth super pixel block in each connected domain; Based on the multiple second connected domains, a third connected domain and a fourth connected domain are obtained; the third connected domain is any connected domain among the second connected domains; the fourth connected domain is any connected domain among the second connected domains that is different from the third connected domain; Based on the third connected domain and the fourth connected domain, super pixel blocks within a range enclosed by tangent lines are selected for combination until the first connected domain is obtained.

Images