Method, device, electronic device and storage medium for determining change speed of controlled object

Abstract

The present disclosure provides a method, device, electronic device and storage medium for determining the change speed of a controlled object. The method includes: acquiring an image frame sequence of a target hand; determining the key point coordinates of the target hand according to the image frame sequence, thereby determining the reference size of this frame image according to the key point coordinates and determining the first reference speed and the second reference speed of the target hand in the image according to any two frames of images; and then determining the change speed of the controlled object under the control of the target hand at the target moment in the process of using continuous air gesture control according to the reference size, the first reference speed and the second reference speed. Since the moving speed of the target hand is calculated with reference to the size of the target hand in the image, it is achieved that when the actual moving speed of the target hand is the same when the target hand is at different distances, the change effect of the controlled object under the control of the target hand is the same, so as to ensure the same control effect on the controlled object.

Description

Technical Field

[0001] The present disclosure relates to the field of human-computer interaction technology, and in particular to a method, device, electronic device and storage medium for determining a change speed of a controlled object. Background Art

[0002] In the related art, human-computer interaction using air gestures is usually divided into switch-type control and continuous control. Continuous control is to control the controlled object by continuous movement of gestures. However, when the moving speed of the operator in the image is proportional to the distance between the operator and the gesture capture device, the change in the distance between the operator and the gesture capture device affects the operation speed of the controlled object, thereby causing instability in the control of the controlled object. Summary of the invention

[0003] In view of this, the purpose of the present disclosure is to provide a method, device, electronic device and storage medium for determining the change speed of a controlled object.

[0004] Based on the above purpose, the present disclosure provides a method for determining a change speed of a controlled object, comprising:

[0005] Acquire a sequence of image frames of the target hand;

[0006] Determining the key point coordinates of the target hand according to the image frame sequence;

[0007] Determine a reference size of the image according to the key point coordinates, and determine a first reference speed and a second reference speed of the target hand according to any two frames of images;

[0008] The changing speed of the controlled object under the control of the target hand at the target time is determined according to the reference size, the first reference speed and the second reference speed.

[0009] In a possible implementation, the key point coordinates include: coordinates corresponding to the base of the index finger and the base of the little finger of the target hand;

[0010] The step of determining the reference size of the image according to the key point coordinates further comprises:

[0011] Acquire the coordinates corresponding to the base of the index finger and the coordinates corresponding to the base of the little finger according to the image;

[0012] The distance between the base of the index finger and the base of the little finger is determined according to the corresponding coordinates of the base of the index finger and the corresponding coordinates of the base of the little finger, and the distance is used as the reference size.

[0013] In a possible implementation, the key point coordinates include: coordinates corresponding to the base of the palm of the target hand;

[0014] The method of determining the reference size of the image according to the key point coordinates and determining the first reference speed and the second reference speed of the target hand according to any two frames of images further includes:

[0015] Acquire the image at a first moment before the target moment and the image at the target moment;

[0016] Determine the corresponding coordinates of the base of the palm of the target hand at the first moment according to the image at the first moment;

[0017] Determining the corresponding coordinates of the base of the palm of the target hand at the target moment according to the image at the target moment, and determining the time difference between the first moment and the target moment; and

[0018] The first reference speed and the second reference speed are determined according to the corresponding coordinates of the base of the palm of the target hand at the first moment, the corresponding coordinates of the base of the palm of the target hand at the target moment, and the time difference.

[0019] In a possible implementation, the first reference speed and the second reference speed are determined according to the coordinates corresponding to the base of the palm of the target hand at the first moment, the coordinates corresponding to the base of the palm of the target hand at the target moment, and the time difference, and the calculation formula is:

[0020]

[0021]

[0022] Among them, v k|x represents the first reference speed, v k|y represents the second reference speed, x k represents the horizontal coordinate corresponding to the palm base of the target hand at the first moment, x k-1 The horizontal coordinate corresponding to the palm base of the target hand at the target time, y k represents the ordinate corresponding to the palm base of the target hand at the first moment, y k-1 The ordinate corresponding to the palm base of the target hand at the target moment, Δ t represents the time difference.

[0023] In a possible implementation, the change speed includes: a first speed and a second speed of the controlled object; the reference size includes a target reference size corresponding to the image at the target moment;

[0024] The method of determining the changing speed of the controlled object under the control of the target hand at the target time according to the reference size, the first reference speed and the second reference speed further includes:

[0025] Obtaining the target reference size;

[0026] determining the first speed according to the target reference size and the first reference speed; and

[0027] The second speed is determined according to the target reference size and the second reference speed, and the calculation formula is:

[0028]

[0029]

[0030] Among them, v c|x represents the first speed, v c|y represents the second speed, m represents a constant, and standardSize1 represents the target standard size.

[0031] In a possible implementation, the changing speed includes: a first speed and a second speed of the controlled object;

[0032] The method of determining the changing speed of the controlled object under the control of the target hand at the target time according to the reference size, the first reference speed and the second reference speed further includes:

[0033] Determine an average reference size according to an average value of the reference sizes of each frame of image in the image frame sequence;

[0034] determining the first speed according to the average reference size and the first reference speed; and

[0035] The second speed is determined based on the average reference size and the second reference speed.

[0036] In a possible implementation manner, determining the key point coordinates of the target hand according to the image frame sequence further includes:

[0037] Get the key point detection model;

[0038] The image frame sequence is input into the key point detection model to determine the key point coordinates of the target hand.

[0039] Based on the same inventive concept, the present disclosure provides a device for determining a change speed of a controlled object, comprising:

[0040] an image acquisition module configured to acquire a sequence of image frames of a target hand;

[0041] A first determination module is configured to determine the key point coordinates of the target hand according to the image frame sequence;

[0042] A second determination module is configured to determine a reference size of the image according to the key point coordinates, and to determine a first reference speed and a second reference speed of the target hand according to any two frames of images;

[0043] The third determination module is configured to determine a changing speed of the controlled object under the control of the target hand at a target moment according to the reference size, the first reference speed and the second reference speed.

[0044] In a possible implementation, the key point coordinates include: coordinates corresponding to the base of the index finger and the base of the little finger of the target hand;

[0045] The second determination module is further configured to obtain the corresponding coordinates of the base of the index finger and the corresponding coordinates of the base of the little finger according to the image; determine the distance between the base of the index finger and the base of the little finger according to the corresponding coordinates of the base of the index finger and the corresponding coordinates of the base of the little finger, and use the distance as the reference size.

[0046] In a possible implementation, the key point coordinates include: coordinates corresponding to the base of the palm of the target hand; the second determination module is further configured to obtain the image at a first moment before the target moment and the image at the target moment; determine the coordinates corresponding to the base of the palm of the target hand at the first moment according to the image at the first moment; determine the coordinates corresponding to the base of the palm of the target hand at the target moment according to the image at the target moment, and determine the time difference between the first moment and the target moment; and determine the first reference speed and the second reference speed according to the coordinates corresponding to the base of the palm of the target hand at the first moment, the coordinates corresponding to the base of the palm of the target hand at the target moment and the time difference.

[0047] In a possible implementation, the change speed includes: a first speed and a second speed of the controlled object; the reference size includes a target reference size corresponding to the image at the target time; the third determination module is further configured to obtain the target reference size; determine the first speed according to the reference size and the first reference speed; and determine the second speed according to the reference size and the second reference speed, and the calculation formula is:

[0048]

[0049]

[0050] Among them, v c|x represents the first speed, v c|y represents the second speed, m represents a constant, and standardSize1 represents the target standard size.

[0051] In one possible implementation, the changing speed includes: a first speed and a second speed of the controlled object; the third determination module is further configured to determine an average reference size based on an average value of the reference size of each frame image in the image frame sequence; determine the first speed based on the target reference size and the first reference speed; and determine the second speed based on the target reference size and the second reference speed.

[0052] In a possible implementation, the first determination module is further configured to obtain a key point detection model; and input the image frame sequence into the key point detection model to determine the key point coordinates of the target hand.

[0053] Based on the same inventive concept, the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method as described above when executing the program.

[0054] Based on the same inventive concept, the present disclosure provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the above method.

[0055] From the above, it can be seen that the method, device, electronic device and storage medium for determining the change speed of the controlled object provided by the present disclosure obtain an image frame sequence of the target hand; determine the key point coordinates of the target hand according to the image frame sequence, thereby determining the reference size of this frame image according to the key point coordinates and determining the first reference speed and the second reference speed of the target hand in the image according to any two frames of images; and then determine the change speed of the controlled object under the control of the target hand at the target moment in the process of using continuous air gesture control according to the reference size, the first reference speed and the second reference speed. Since the moving speed of the target hand is calculated with the size of the target hand in the image as a reference, no matter how far the target hand is from the gesture capture device and the size of the target hand in the image, as long as the actual moving speed of the target hand is the same, the change speed of the operating hand in the image is the same, thereby achieving the same change effect of the controlled object under the control of the target hand when the actual moving speed of the target hand is the same when the target hand is at different distances, so as to ensure the same control effect on the controlled object. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 A schematic diagram showing an application scenario of the method for determining the change speed of a controlled object provided by an embodiment of the present disclosure;

[0058] Figure 2 An exemplary flow chart of a method for determining a change speed of a controlled object provided by an embodiment of the present disclosure is shown;

[0059] Figure 3 An exemplary schematic diagram showing joint points of a target hand according to an embodiment of the present disclosure is shown;

[0060] Figure 4 An exemplary numbering diagram of joint points of a target hand according to an embodiment of the present disclosure is shown;

[0061] Figure 5 An exemplary structural diagram of a device for determining a change speed of a controlled object provided by an embodiment of the present disclosure is shown;

[0062] Figure 6 A schematic diagram of an exemplary structure of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0064] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0065] As described in the background technology section, with the development of intelligent interaction technology, air gesture control technology is increasingly being used in various fields, such as vehicle-mounted systems, smart homes, VR content interaction, and other fields. The uses of air gesture control are generally divided into two types: switch-type control and continuous control. Among them, switch-type control, for example, can be turning on / off the screen, turning on / off recording, etc., while continuous control, for example, can be adjusting the volume, moving the cursor, etc. For switch-type control, only static specific gestures are needed to complete the control function, but continuous control requires obtaining information on the speed of gesture changes.

[0066] In the remote gesture control system, the two-dimensional camera is a common gesture capture device for obtaining gesture information. The applicant has found through research that when the target hand moves at a certain speed, the moving speed of the target hand in the image is inversely proportional to the distance of the target hand from the camera. If continuous control is performed based on the moving speed of the target hand in the image, it will lead to the problem that the controlled object changes quickly when the target hand is close to the camera, and the controlled object changes slowly when the target hand is far from the camera.

[0067] For this reason, an embodiment of the present disclosure provides a method for determining the changing speed of a controlled object, comprising: acquiring an image frame sequence of a target hand; determining the key point coordinates of the target hand based on the image frame sequence; determining a reference size of the image based on the key point coordinates, and determining a first reference speed and a second reference speed of the target hand based on any two frames of images; determining the changing speed of the controlled object under the control of the target hand at the target moment based on the reference size, the first reference speed and the second reference speed.

[0068] The method, device, electronic device and storage medium for determining the speed of change of the controlled object provided by the embodiments of the present disclosure obtain the image frame sequence of the target hand; determine the key point coordinates of the target hand according to the image frame sequence, thereby determining the reference size of this frame image according to the key point coordinates and determining the first reference speed and the second reference speed of the target hand in the image according to any two frames of images; and then determine the change speed of the controlled object under the control of the target hand at the target moment in the process of using continuous air gesture control according to the reference size, the first reference speed and the second reference speed. Since the moving speed of the target hand is calculated with the size of the target hand in the image as a reference, no matter how far the target hand is from the gesture capture device and the size of the target hand in the image, as long as the actual moving speed of the target hand is the same, the change speed of the operating hand in the image is the same, thereby achieving the same change effect of the controlled object under the control of the target hand when the actual moving speed of the target hand is the same when the target hand is at different distances, so as to ensure the same control effect on the controlled object.

[0069] refer to Figure 1, which is a schematic diagram of an application scenario of a method for determining the speed of change of a controlled object provided according to an embodiment of the present disclosure. The application scenario includes a terminal device 101, a server 102, and a data storage system 103. Among them, the terminal device 101, the server 102 and the data storage system 103 can be connected through a wired or wireless communication network. The terminal device 101 includes but is not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA) or other electronic devices capable of realizing the above functions. The server 102 and the data storage system 103 can both be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0070] The server 102 is used to provide a service for determining the changing speed of a controlled object to a user of the terminal device 101. The terminal device 101 can communicate with the server 102. The terminal device 101 obtains the user's control gesture image and sends the user's control gesture image to the server 102. The server 102 processes the user's control gesture image and obtains the changing speed of the controlled object controlled by the target hand at the target moment by determining the reference size, the first reference speed and the second reference speed according to the image, so that the terminal device 101 can be applied to air gesture control scenarios, such as vehicle-mounted systems, smart homes, VR content interaction and other scenarios.

[0071] A large amount of training sample data is stored in the data storage system 103, and the source of the training sample data includes but is not limited to existing databases, data crawled from the Internet, or data uploaded by users when using the client. The server 102 can also continuously optimize the controlled object change speed determination technology based on the newly added training sample data.

[0072] Combine the following Figure 1 The exemplary controlled object change speed determination method provided by the embodiment of the present disclosure is described by using the application scenario. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principle of the present disclosure, and the implementation of the present disclosure is not limited in this respect. On the contrary, the implementation of the present disclosure can be applied to any applicable scenario.

[0073] refer to Figure 2 The method for determining the change speed of a controlled object provided by the embodiment of the present disclosure includes:

[0074] S202: Acquire a sequence of image frames of the target hand.

[0075] The present disclosure is applied to fields related to air gesture control. Devices that implement air gesture control are usually externally or internally connected to a two-dimensional camera. These two-dimensional cameras usually have sensors that can capture the control gestures of a target hand. The gesture capture device obtains an image frame sequence containing the target hand in real time through the two-dimensional camera, and obtains multiple frames of images of the target hand, for example, obtaining 30 frames of images to determine the image frame sequence.

[0076] In some embodiments, the acquired image containing the target hand may contain multiple hands. In this case, the target hand can be identified, for example, the hand that appears in the two-dimensional camera for the longest time is captured as the target hand, or before the controlled object is controlled, the target hand makes a corresponding starting gesture to activate the gesture control function. The two-dimensional camera uses the hand that first makes the starting gesture as the target hand and captures subsequent images of it.

[0077] S204: Determine the key point coordinates of the target hand according to the image frame sequence.

[0078] After determining the image frame sequence of the target hand, a target detection model can be constructed, and the acquired image frame sequence can be input into the target detection model to obtain the coordinate information of the key points.

[0079] In some implementations, the key point detection model can be obtained and the key point coordinates of the target hand can be determined by inputting the obtained image frame sequence into the key point detection model. Specifically, the key point detection model can be constructed based on a deep neural network model (DNN) framework. It should be noted that the method for constructing the key point detection model can include the following steps:

[0080] Using a two-dimensional camera to acquire an image frame sequence, the image frame sequence may include a target hand, and the image frame sequence is divided into a training set and a test set; wherein the training set may include training sample data and labels, and the labels are used to identify key point coordinate information of the target hand;

[0081] Obtain any deep learning model with object detection function, input the training set into the deep learning model to train the model, and thus obtain a key point detection model;

[0082] The trained key point detection model is tested using the test set to obtain an optimized key point detection model, and the key point coordinate information of the target hand is detected and extracted using the final key point detection model.

[0083] In some implementations, key points of the target hand may be identified based on a specific position of the target hand, such as the joints of the target hand. Figure 3, the key points of the target hand are marked according to the joints of the target hand, so that the joint point legend of the target hand can be obtained, specifically including: dividing the index finger from the fingertip to the root into 4 joint points, dividing the middle finger from the fingertip to the root into 4 joint points, dividing the ring finger from the fingertip to the root into 4 joint points, dividing the little finger from the fingertip to the root into 4 joint points, dividing the thumb from the fingertip to the root into 3 joint points, the joint points at the thenar muscle and the joint points at the base of the palm.

[0084] In order to simplify the target hand and each joint point, we can get the following Figure 4 The joint numbering diagram shown in the figure shows that each joint point is numbered sequentially from the joint point at the base of the palm to the tip of the little finger. Figure 4 , the joint point at the base of the palm is numbered 0, the joint point at the thenar muscle is numbered 1, and the three joint points from the base to the fingertip of the thumb are numbered 2, 3, and 4 respectively. Similarly, all the joint points corresponding to the index finger, middle finger, ring finger, and little finger can be numbered.

[0085] It should be noted that each joint point can correspond to unique coordinate information according to its number. For example, the coordinate information of the joint point at the base of the palm can be expressed as P0 (x0, y0).

[0086] In some embodiments, the key point coordinates may further include the joint point coordinates and the center point coordinates of the target hand, and the joint point coordinates may be selected from the four relatively stable joint points 5, 9, 13, and 17 of the target hand joint points, which correspond to the joint points corresponding to the base of the index finger, the base of the middle finger, the base of the ring finger, and the base of the little finger, respectively. The key point coordinates of the target hand are determined based on the image, and the joint point coordinates and the center point coordinates of the target hand can be further determined based on the image, and the center point coordinates correspond to the coordinates corresponding to the joint points at the base of the palm.

[0087] S206: Determine a reference size of the image according to the key point coordinates, and determine a first reference speed and a second reference speed of the target hand according to any two frames of images.

[0088] In this embodiment, the reference size of the image can be determined according to the key point coordinates, which is used to establish a connection between each image, and further the first reference speed and the second reference speed of the target hand are determined according to any two frames of images to determine the speed of change of the target hand in the image. Among them, two adjacent frames of images can be selected, and images separated by multiple frames can also be selected. Since the acquisition speed of each frame is fast enough, the position of the target hand can be determined in the image. In the case where the time difference between any two frames of images can be determined, the first reference speed and the second reference speed can be determined by selecting any two frames of images.

[0089] In some embodiments, the reference size can be determined based on the corresponding coordinates of the base of the index finger and the base of the little finger obtained from the image, that is, the distance between the joint point 5 and the joint point 17 is used as the reference size of the image. It should be noted that the reference size can also be calculated based on the distance between other joint points, and the joint point 5 and the joint point 17 are located near the center of the target hand and are basically on the same horizontal line. At the same time, the width of the target hand can be roughly marked. Therefore, in this embodiment, the distance between the joint point 5 and the joint point 17 is selected as the reference size.

[0090] Specifically, the coordinate information corresponding to the joint point 5 and the joint point 17 can be obtained, that is, P5 (x5, y5) and P 17 (x 17 ,y 17 ), the formula for calculating the reference size is as follows:

[0091]

[0092] Among them, standardSize represents the reference size, x5 represents the horizontal coordinate corresponding to the base of the index finger, y5 represents the vertical coordinate corresponding to the base of the index finger, and x 17 Indicates the horizontal coordinate corresponding to the base of the little finger, y 17 Indicates the vertical coordinate corresponding to the base of the little finger.

[0093] In some embodiments, in order to determine the speed of change of the target hand in the image, an image at the first moment before the target moment and an image at the target moment may be acquired; the center point coordinates at the first moment are determined based on the image at the first moment, the center point coordinates at the target moment are determined based on the image at the target moment, and the time difference between the first moment and the target moment is determined; and then the first reference speed and the second reference speed are determined based on the center point coordinates at the first moment, the center point coordinates at the target moment, and the time difference. Specifically, images at the k-1 moment and the k moment are acquired respectively, and the center point coordinates of the target hand at the k-1 moment are determined to be zero. k (x k ,y k ), the coordinate of the center point of the target hand at time k is O k-1 (x k-1 ,y k-1 ), define the time difference between time k-1 and time k as Δ t The k moment and the k-1 moment can be two non-adjacent moments, and the time difference can be determined according to the number of frames between them. The speed of change of the target hand in the image can be calculated according to the displacement and time difference of the target hand center point between the first moment and the target moment. The specific calculation formula is as follows:

[0094]

[0095]

[0096] Among them, v k|x represents the first reference velocity, v k|y represents the second reference speed, x k represents the horizontal coordinate of the center point at the first moment (that is, the horizontal coordinate corresponding to the palm base of the target hand), x k-1 represents the horizontal coordinate of the center point at the target time (that is, the horizontal coordinate corresponding to the palm base of the target hand), y k represents the ordinate of the center point at the first moment (i.e., the ordinate corresponding to the palm base of the target hand), y k-1 represents the ordinate of the center point at the target time (i.e., the ordinate corresponding to the palm base of the target hand), Δ t It should be noted that the first reference speed is the change speed of the target hand decomposed along the horizontal axis in the image, and the second reference speed is the change speed of the target hand decomposed along the vertical axis in the image.

[0097] S208: Determine a change speed of the controlled object under the control of the target hand at the target moment according to the reference size, the first reference speed and the second reference speed.

[0098] In this embodiment, after determining the reference size of the image and the change speed of the target hand in the image, a proportional relationship can be established to determine the change speed of the controlled object according to the change speed of the target hand in the image.

[0099] In some embodiments, during the control process using continuous air gestures, the speed at which the controlled object is controlled by the target hand at the target moment can be divided into a first speed and a second speed, corresponding to the first reference speed and the second reference speed, respectively. For example, the first speed can be the speed at which the target hand changes along the X-axis, and the second speed can be the speed at which the target hand changes along the Y-axis. When the target hand moves only along the X-axis or the Y-axis, the speed at which the change occurs can be the first speed or the second speed. When the target hand moves not only along the X-axis or the Y-axis, the speed at which the first speed and the second speed change occurs can be taken as the speed at which the change occurs through a speed synthesis formula. Furthermore, the first speed can be determined based on the acquired target reference size and the first reference speed, and the calculation formula is as follows:

[0100]

[0101] Among them, v c|xrepresents the first speed, m represents a constant, and standardSize1 represents the target reference size. The constant m can be selected according to the control requirements. For example, when executing a movement instruction, the target hand's moving speed needs to be the same as the controlled object's changing speed, then m can be 1; when executing a volume control instruction, the larger the value of m, the faster the controlled object can change even if the target hand's moving speed is slower. m can be adjusted according to the requirements of the controlled object's changing speed. The reference size includes the target reference size, where the target reference size can be the reference size corresponding to the image at the target moment.

[0102] The second speed can be determined according to the target reference size and the second reference speed, and the calculation formula is as follows:

[0103]

[0104] Among them, v c|y Indicates the second speed.

[0105] The speed synthesis formula is as follows:

[0106]

[0107] In some embodiments, the reference size used to calculate the first speed and the second speed may be the target reference size corresponding to the current frame image acquired at the target moment, or may be an average reference size determined based on the average value of the reference sizes of each frame image in the image frame sequence.

[0108] Specifically, the first speed may be determined according to the obtained average reference size and the first reference speed, and the calculation formula is as follows:

[0109]

[0110] Among them, standardSize2 represents the average reference size.

[0111] The second speed can be determined according to the average reference size and the second reference speed, and the calculation formula is as follows:

[0112]

[0113] The speed synthesis formula is as follows:

[0114]

[0115] When the speed of acquiring the image frame sequence is fast enough, the first speed and the second speed can be determined by combining the average standard size standardSize2 with the first reference speed and the second reference speed, so as to determine the change speed of the controlled object according to the composite speed. When there is no obvious position difference between the images of each frame in the image frame sequence, the first speed and the second speed can be determined by combining the target standard size standardSize1 with the first reference speed and the second reference speed, so as to determine the change speed of the controlled object according to the composite speed.

[0116] It should be noted that the change speed of the controlled object can be calculated as the change speed ratio of the target hand in the image, regardless of the distance of the target hand from the two-dimensional camera, regardless of the size of the target hand in the image, as long as the actual moving speed of the target hand is the same, the change speed of the controlled object finally calculated is the same. Thus, the control effect of the target hand on the controlled object is not affected by the distance of the target hand from the two-dimensional camera and the size of the target hand in the image.

[0117] From the above, it can be seen that the method, device, electronic device and storage medium for determining the change speed of the controlled object provided by the present disclosure obtain an image frame sequence of the target hand; determine the key point coordinates of the target hand according to the image frame sequence, thereby determining the reference size of this frame image according to the key point coordinates and determining the first reference speed and the second reference speed of the target hand in the image according to any two frames of images; and then determine the change speed of the controlled object under the control of the target hand at the target moment in the process of using continuous air gesture control according to the reference size, the first reference speed and the second reference speed. Since the moving speed of the target hand is calculated with the size of the target hand in the image as a reference, no matter how far the target hand is from the gesture capture device and the size of the target hand in the image, as long as the actual moving speed of the target hand is the same, the change speed of the operating hand in the image is the same, thereby achieving the same change effect of the controlled object under the control of the target hand when the actual moving speed of the target hand is the same when the target hand is at different distances, so as to ensure the same control effect on the controlled object.

[0118] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0119] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0120] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure further provides a device for determining the change speed of a controlled object.

[0121] refer to Figure 5 The device for determining the change speed of the controlled object comprises: an image acquisition module, a first determination module, a second determination module and a third determination module; wherein,

[0122] an image acquisition module configured to acquire a sequence of image frames of a target hand;

[0123] A first determination module is configured to determine the key point coordinates of the target hand according to the image frame sequence;

[0124] A second determination module is configured to determine a reference size of the image according to the key point coordinates, and to determine a first reference speed and a second reference speed of the target hand according to any two frames of images;

[0125] The third determination module is configured to determine a changing speed of the controlled object under the control of the target hand at a target moment according to the reference size, the first reference speed and the second reference speed.

[0126] In some embodiments, the key point coordinates include: coordinates corresponding to the base of the index finger and the base of the little finger of the target hand;

[0127] The second determination module is further configured to obtain the corresponding coordinates of the base of the index finger and the corresponding coordinates of the base of the little finger according to the image; determine the distance between the base of the index finger and the base of the little finger according to the corresponding coordinates of the base of the index finger and the corresponding coordinates of the base of the little finger, and use the distance as the reference size.

[0128] In some embodiments, the key point coordinates include: the corresponding coordinates of the base of the palm of the target hand; the second determination module is further configured to obtain the image at a first moment before the target moment and the image at the target moment; determine the corresponding coordinates of the base of the palm of the target hand at the first moment according to the image at the first moment; determine the corresponding coordinates of the base of the palm of the target hand at the target moment according to the image at the target moment, and determine the time difference between the first moment and the target moment; and determine the first reference speed and the second reference speed according to the corresponding coordinates of the base of the palm of the target hand at the first moment, the corresponding coordinates of the base of the palm of the target hand at the target moment and the time difference.

[0129] In some implementations, the first reference speed and the second reference speed are determined according to the coordinates corresponding to the base of the palm of the target hand at the first moment, the coordinates corresponding to the base of the palm of the target hand at the target moment, and the time difference, and the calculation formula is:

[0130]

[0131]

[0132] Among them, v k|x represents the first reference speed, v k|y represents the second reference speed, x k represents the horizontal coordinate corresponding to the palm base of the target hand at the first moment, x k-1 The horizontal coordinate corresponding to the palm base of the target hand at the target time, y k represents the ordinate corresponding to the palm base of the target hand at the first moment, y k -1 represents the ordinate corresponding to the palm base of the target hand at the target moment, Δ t represents the time difference.

[0133] In some embodiments, the change speed includes: a first speed and a second speed of the controlled object; the reference size includes a target reference size corresponding to the image at the target time; the third determination module is further configured to obtain the target reference size; determine the first speed according to the reference size and the first reference speed; and determine the second speed according to the reference size and the second reference speed, and the calculation formula is:

[0134]

[0135]

[0136] Among them, v c|x represents the first speed, v c|yrepresents the second speed, m represents a constant, and standardSize1 represents the target standard size.

[0137] In some embodiments, the changing speed includes: a first speed and a second speed of the controlled object; the third determination module is further configured to determine an average reference size based on an average value of the reference size of each frame image in the image frame sequence; determine the first speed based on the target reference size and the first reference speed; and determine the second speed based on the target reference size and the second reference speed.

[0138] In some embodiments, the first determination module is further configured to obtain a key point detection model; and input the image frame sequence into the key point detection model to determine the key point coordinates of the target hand.

[0139] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0140] The device of the above embodiment is used to implement the corresponding method for determining the change speed of the controlled object in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0141] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for determining the changing speed of the controlled object described in any of the above-mentioned embodiments is implemented. Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 610, a memory 620, an input / output interface 630, a communication interface 640, and a bus 650. The processor 610, the memory 620, the input / output interface 630, and the communication interface 640 are connected to each other through the bus 650 in the device.

[0142] The processor 610 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0143] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 620 and called and executed by the processor 610.

[0144] The input / output interface 630 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0145] The communication interface 640 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0146] The bus 650 comprises a pathway for transmitting information between the various components of the device (eg, the processor 610, the memory 620, the input / output interface 630, and the communication interface 640).

[0147] It should be noted that, although the above device only shows the processor 610, the memory 620, the input / output interface 630, the communication interface 640 and the bus 650, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0148] The electronic device of the above embodiment is used to implement the corresponding method for determining the change speed of the controlled object in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0149] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method for determining the changing speed of the controlled object as described in any of the above embodiments.

[0150] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0151] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method for determining the change speed of the controlled object as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0152] It should be noted that the embodiments of the present disclosure may be further described in the following manner:

[0153] A method for determining a change speed of a controlled object, comprising:

[0154] Acquire a sequence of image frames of the target hand;

[0155] Determining the key point coordinates of the target hand according to the image frame sequence;

[0156] Determine a reference size of the image according to the key point coordinates, and determine a first reference speed and a second reference speed of the target hand according to any two frames of images;

[0157] The changing speed of the controlled object under the control of the target hand at the target time is determined according to the reference size, the first reference speed and the second reference speed.

[0158] Optional,

[0159] The key point coordinates include: coordinates corresponding to the base of the index finger and the base of the little finger of the target hand;

[0160] The step of determining the reference size of the image according to the key point coordinates further comprises:

[0161] Acquire the coordinates corresponding to the base of the index finger and the coordinates corresponding to the base of the little finger according to the image;

[0162] The distance between the base of the index finger and the base of the little finger is determined according to the corresponding coordinates of the base of the index finger and the corresponding coordinates of the base of the little finger, and the distance is used as the reference size.

[0163] Optional,

[0164] The key point coordinates include: the corresponding coordinates of the palm base of the target hand;

[0165] The determining of the reference size of the image and the first reference speed and the second reference speed of the target hand according to the key point coordinates further comprises:

[0166] Acquire the image at a first moment before the target moment and the image at the target moment;

[0167] Determine the corresponding coordinates of the base of the palm of the target hand at the first moment according to the image at the first moment;

[0168] Determining the corresponding coordinates of the base of the palm of the target hand at the target moment according to the image at the target moment, and determining the time difference between the first moment and the target moment; and

[0169] The first reference speed and the second reference speed are determined according to the corresponding coordinates of the base of the palm of the target hand at the first moment, the corresponding coordinates of the base of the palm of the target hand at the target moment, and the time difference.

[0170] Optional,

[0171] The first reference speed and the second reference speed are determined according to the corresponding coordinates of the palm base of the target hand at the first moment, the corresponding coordinates of the palm base of the target hand at the target moment, and the time difference, and the calculation formula is:

[0172]

[0173]

[0174] Among them, v k|x represents the first reference speed, v k|y represents the second reference speed, x k represents the horizontal coordinate corresponding to the palm base of the target hand at the first moment, x k-1 The horizontal coordinate corresponding to the palm base of the target hand at the target time, y k represents the ordinate corresponding to the palm base of the target hand at the first moment, y k-1 The ordinate corresponding to the palm base of the target hand at the target moment, Δ t represents the time difference.

[0175] Optionally, the change speed includes: a first speed and a second speed of the controlled object; the reference size includes a target reference size corresponding to the image at the target moment;

[0176] The method of determining the changing speed of the controlled object under the control of the target hand at the target time according to the reference size, the first reference speed and the second reference speed further includes:

[0177] Obtaining the target reference size;

[0178] determining the first speed according to the target reference size and the first reference speed; and

[0179] The second speed is determined according to the target reference size and the second reference speed, and the calculation formula is:

[0180]

[0181]

[0182] Among them, v c|x represents the first speed, v c|y represents the second speed, m represents a constant, and standardSize1 represents the target standard size.

[0183] Optionally, the changing speed includes: a first speed and a second speed of the controlled object;

[0184] The method of determining the changing speed of the controlled object under the control of the target hand at the target time according to the reference size, the first reference speed and the second reference speed further includes:

[0185] Determine an average reference size according to an average value of the reference sizes of each frame of image in the image frame sequence;

[0186] determining the first speed according to the target reference size and the first reference speed; and

[0187] The second speed is determined based on the target reference size and the second reference speed.

[0188] Optionally, determining the key point coordinates of the target hand according to the image frame sequence further includes:

[0189] Get the key point detection model;

[0190] The image frame sequence is input into the key point detection model to determine the key point coordinates of the target hand.

[0191] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0192] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0193] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0194] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for determining the change speed of a controlled object, wherein: include: Obtain a sequence of image frames of the target hand; Determining the key point coordinates of the target hand according to the image frame sequence; wherein the key point coordinates include the joint point coordinates of the target hand; Determine a reference size of the image according to the key point coordinates, and determine a first reference speed and a second reference speed of the target hand according to any two frames of images; Determining a change speed of the controlled object under the control of the target hand at a target time according to the reference size, the first reference speed and the second reference speed; Determining the reference size of the image according to the key point coordinates includes: Determine the reference size of the image according to the distance between any two relative stable joint point coordinates; The change speed includes: a first speed and a second speed of the controlled object; the reference size includes a target reference size corresponding to the image at the target time; The method of determining the changing speed of the controlled object under the control of the target hand at the target time according to the reference size, the first reference speed and the second reference speed further includes: Obtaining the target reference size; determining the first speed according to the target reference size and the first reference speed; and The second speed is determined according to the target reference size and the second reference speed, and the calculation formula is: Among them, v c|x represents the first speed, v ic|y represents the second speed, m represents a constant, standardSize1 represents the target standard size, v k|x represents the first reference speed, v k|y represents the second reference speed.

2. The method according to claim 1, wherein: The key point coordinates include: coordinates corresponding to the base of the index finger and the base of the little finger of the target hand; The step of determining the reference size of the image according to the key point coordinates further comprises: Acquire the coordinates corresponding to the base of the index finger and the coordinates corresponding to the base of the little finger according to the image; The distance between the base of the index finger and the base of the little finger is determined according to the corresponding coordinates of the base of the index finger and the corresponding coordinates of the base of the little finger, and the distance is used as the reference size.

3. The method according to claim 1, wherein: The key point coordinates include: the corresponding coordinates of the palm base of the target hand; The method of determining the reference size of the image according to the key point coordinates and determining the first reference speed and the second reference speed of the target hand according to any two frames of images further includes: Acquire the image at a first moment before the target moment and the image at the target moment; Determine the corresponding coordinates of the base of the palm of the target hand at the first moment according to the image at the first moment; Determining the corresponding coordinates of the base of the palm of the target hand at the target moment according to the image at the target moment, and determining the time difference between the first moment and the target moment; and The first reference speed and the second reference speed are determined according to the corresponding coordinates of the base of the palm of the target hand at the first moment, the corresponding coordinates of the base of the palm of the target hand at the target moment, and the time difference.

4. The method according to claim 3, wherein: The first reference speed and the second reference speed are determined according to the corresponding coordinates of the palm base of the target hand at the first moment, the corresponding coordinates of the palm base of the target hand at the target moment, and the time difference, and the calculation formula is: Among them, v k|x represents the first reference speed, v k|y represents the second reference speed, x k represents the horizontal coordinate corresponding to the palm base of the target hand at the first moment, x k-1 The horizontal coordinate corresponding to the palm base of the target hand at the target time, y k represents the ordinate corresponding to the palm base of the target hand at the first moment, y k-1 The ordinate corresponding to the palm base of the target hand at the target moment, Δ t represents the time difference.

5. The method according to claim 1, wherein: The change speed includes: a first speed and a second speed of the controlled object; The method of determining the changing speed of the controlled object under the control of the target hand at the target time according to the reference size, the first reference speed and the second reference speed further includes: Determine an average reference size according to an average value of the reference sizes of each frame of image in the image frame sequence; determining the first speed according to the average reference size and the first reference speed; and The second speed is determined based on the average reference size and the second reference speed.

6. The method according to claim 1, wherein: The determining the key point coordinates of the target hand according to the image frame sequence further comprises: Get the key point detection model; The image frame sequence is input into the key point detection model to determine the key point coordinates of the target hand.

7. A device for determining the speed of change of a controlled object, wherein: include: an image acquisition module configured to acquire a sequence of image frames of a target hand; A first determination module is configured to determine the key point coordinates of the target hand according to the image frame sequence; wherein the key point coordinates include the joint point coordinates of the target hand; A second determination module is configured to determine a reference size of the image according to the key point coordinates, and to determine a first reference speed and a second reference speed of the target hand according to any two frames of images; A third determination module is configured to determine a change speed of the controlled object under the control of the target hand at a target time according to the reference size, the first reference speed and the second reference speed; Determining the reference size of the image according to the key point coordinates includes: Determine the reference size of the image according to the distance between any two relative stable joint point coordinates; The change speed includes: a first speed and a second speed of the controlled object; the reference size includes a target reference size corresponding to the image at the target time; The method of determining the changing speed of the controlled object under the control of the target hand at the target time according to the reference size, the first reference speed and the second reference speed further includes: Obtaining the target reference size; determining the first speed according to the target reference size and the first reference speed; and The second speed is determined according to the target reference size and the second reference speed, and the calculation formula is: Among them, v c|x represents the first speed, v c|y represents the second speed, m represents a constant, and standardSize1 represents the target standard size.

8. An electronic device, wherein: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 6.

Images