Method for detecting and controlling hand gestures in virtual studio system

Abstract

The present invention relates to a method for detecting and controlling gestures in a virtual broadcasting system in the field of artificial intelligence. Step S100: palm detection stage. First, detect whether a palm appears in an input image. If not, continue to detect the next input image until a palm is detected. Step S200: palm tracking stage. When a palm is detected in the input image, track the palm for a certain period of time. If the palm can be successfully tracked for a certain period of time, enter the next stage; if the tracking fails, return to the palm detection stage. Step S300: optical flow calculation stage. Perform optical flow calculation on the input image, and collect and analyze the results of the optical flow calculation. Step S400: gesture identification stage. Collect and analyze the results of the optical flow calculation to identify the gesture. Step S500: gesture control stage. According to the gesture identified in the previous stage and in combination with the corresponding relationship between the gesture and the system operation, the virtual broadcasting system is controlled.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for detecting and controlling gestures in a virtual studio system. Background Art

[0002] With the continuous development of modern education, in order to meet the requirements of quality education, open and interactive multimedia video teaching has been gradually applied to teaching practice. The establishment of a digital campus TV station can promote students' ability to explore independently and innovate learning, and can improve students' language expression ability, appearance and demeanor, broaden students' horizons, and establish an interactive video network teaching platform between teachers and students.

[0003] As an important means of real-time TV program production, the virtual studio system based on blue box matting technology has been widely used in various TV program productions. The virtual studio system obtains the foreground image by the camera and obtains the motion parameters of the camera through the tracking system; the graphic workstation generates the background signal and mask signal of the three-dimensional virtual scene in real time according to the motion parameters of the camera; the camera video signal, the three-dimensional virtual scene signal and the mask signal are matted and synthesized in the matting synthesizer and output to the recording device. The virtual studio system can automatically track the position, pan, pitch and focus and zoom parameters of the camera, and change the three-dimensional virtual scene in real time according to these parameters, so that the change of the three-dimensional virtual scene is synchronized and consistent with the movement of the camera, ensuring the correct perspective relationship between the foreground and the background. The host in the foreground looks completely immersed in the three-dimensional virtual scene generated by the computer, with a realistic TV studio effect. However, the existing virtual studio system cannot recognize the host's gestures and respond to different gestures. In view of the shortcomings of the prior art, the present invention discloses a method for detecting and controlling gestures in a virtual studio system to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to solve the above defects and provide a method for detecting and controlling gestures in a virtual broadcasting system to solve the problems mentioned in the above background technology.

[0005] The object of the present invention is achieved in the following ways:

[0006] The method for detecting and controlling gestures in a virtual studio system comprises the following steps:

[0007] Step S100: palm detection stage, first detect whether there is a palm in the input image, if not, continue to detect the next input image until a palm is detected;

[0008] Step S200: palm tracking stage, when a palm is detected in the input image, the palm is tracked for a certain period of time. If the palm is successfully tracked for a certain period of time, the next stage is entered; if the tracking fails, the palm detection stage is returned;

[0009] Step S300: Optical flow calculation stage, performing optical flow calculation on the input image, and collecting and analyzing the results of the optical flow calculation;

[0010] Step S400: gesture identification stage, collecting and analyzing the results through optical flow calculation to identify the gesture action;

[0011] Step S500: gesture control stage, according to the gesture action identified in the previous stage and in combination with the corresponding relationship between the gesture action and the system operation, the virtual broadcasting system is controlled.

[0012] Furthermore, in the above description, in step S100 , the hierarchical Adaboost technique is used to detect whether a palm appears in the input image.

[0013] Furthermore, in the above description, before executing step S100 , the hierarchical Adaboost technology needs to be trained with a specific object set so that the hierarchical Adaboost technology can determine whether the specific object exists in the input image.

[0014] Further in the above description, the specific object may be a face, a fist, or a palm, so as to adapt to different application scenarios.

[0015] Further in the above description, the training method of the hierarchical Adaboost technology includes the following steps:

[0016] Step T1: Let O represent a specific object, train Adaboost to determine whether O appears in an image I, and assume that the specific features of object O form a set F = {f1,f2,···,f T}, for each feature f t Define a classifier h t , that is, h t Only based on the feature f t To determine whether there is object O in image I, it is called h t is a weak classifier of object O, let S = {(x1,y1),(x2,y2),····,(x m ,y m )) is a sample set collected to train the classifier, where x i ∈X is the sample space, y i ∈Y={-1,+1} is the label space, +1 and -1 represent that a sample x is "yes" or "not" the specific object, respectively. Let Dt represents the distribution of relative importance among samples at time t. At the beginning, let D1(i) = 1 / m; i = 1, ···, m, indicating that all samples have the same importance;

[0017] Step T2: When using classifier h t To judge a sample x in S i If h t (x i )≠y i Indicates h t Misjudged x i category, then the function 1[h t (x i )≠y i ] indicates that when h t (x i )≠y i When , the value is 1; otherwise, the value is 0. Define the classifier h t The classification error for all samples in S is Let α t The feature f t The weight is defined as α t =ln((1-ε t ) / ε t ) / 2, so when ε t When α increases, t will become smaller, and the distribution of sample importance D t+1 It can be updated according to the following formula, so that h t Misclassified samples, their importance for the next h t+1 will increase, otherwise it will decrease. Where Z t is used for regularization. The above process is repeated for all weak classifiers. Finally, the strong classifier H(x) is formed by combining all weak classifiers. The mathematical table is

[0018] Step T3: Feature set F = {f1,f2,···,f T} can be divided into groups F k ,k=1,··,K, then for each group F k Find a strong classifier H k (x) When performing classification, the hierarchical Adaboost performs H k (x), k = 1, · · , K, only when the sample x passes the classification judgment of the previous stage will the judgment of the subsequent stage be continued. Therefore, in order to speed up the execution of hierarchical Adaboost, the feature clustering F k,k=1,··,K design uses simple features to prioritize clustering, that is, |F k |<|F k+1 |,k=1,··,K-1 method, so that the front-end classifier H can be executed k (x).

[0019] Further in the above description, in step S300, the optical flow calculation is performed only within a limited range near the palm in the input image.

[0020] Further in the above description, the optical flow calculation in step S300 is to calculate the movement of the object presented on the image when it moves in three-dimensional space. The optical flow calculation needs to be based on the following two assumptions:

[0021] Assumption 1: The brightness of the surface of an object remains constant when moving over a short distance;

[0022] Assumption 2: Adjacent image points in the image move similarly. Let f(x,y,t) represent the grayscale value of the image point (x,y) at time t. Expand it with Taylor series: where f x ,f y ,f t is the partial differential of f with respect to x, y, and t respectively. Assume that the image point near (x, y) is translated by a small distance in the time interval dt. According to the above assumption 1: the brightness of the object surface remains unchanged when moving over a short distance, that is, f(x+dx,y+dy,t+dt)=f(x,y,t), so equation (1) can be written as If dx, dy, dt are small enough, will approach zero, so equation (2) can be rewritten as f x dx+f y dy+f t dt=0.----(3), after sorting, we can get The speed That is the optical flow of the image point (x, y);

[0023] According to the above formula, we can calculate the moving speed (i.e., optical flow) of all image points. However, in order to satisfy the second assumption that adjacent image points in the image move similarly, it is also necessary to perform smooth motion on the optical flow of the image points. This can be done by finding (u, v) to minimize the following objective equation: in is the square of the partial differential of u and v. The above optimization problem can be solved by the Lagrange multiplier method: in is the average value of u, is the average value of v,

[0024] Further in the above description, the gesture identification in step S400 is based on the optical flow vector calculated from the collected input image, which is used to determine the gesture action after statistical analysis. Let o(x, y, t) represent the optical flow vector calculated at the image point (x, y) at time t, and average the optical flow vectors of the image points within a limited range H×W near the palm. That is, the vector of the palm movement at time t. Assuming that the time period of the palm waving is T, the average optical flow during this period is Gesture recognition is to judge the gesture action based on O.

[0025] Further in the above description, the gesture actions in step S400 include left swipe, right swipe, downward swipe, upward swipe, merging both hands together, and separating both hands.

[0026] Further to the above description, after the gesture recognition is completed, it can be applied to control the operation of specific functions of the virtual studio system according to the correspondence between different gestures and system operations. Gesture control will be divided into two control modes, corresponding to the following software versions respectively:

[0027] Version 1: Controls the entry and exit logic of 2D and 3D components in a 3D scene. Swipe left and right respectively correspond to the entry and exit of the background window. Swipe down corresponds to the change of 3D scene. Swipe up corresponds to the weather forecast or data report. Putting both hands together corresponds to adding particle effects. Splitting both hands corresponds to turning off particle effects. Particle effects refer to 3D components that simulate water, fire, fog, gas, and fireworks.

[0028] Version 2: Controls the entry and exit, rotation logic, and anatomical display of 3D components in the 3D scene. A left swipe gesture corresponds to a left-rotating component, a right swipe gesture corresponds to a right-rotating component, an up swipe gesture corresponds to an up-rotating component, a down swipe gesture corresponds to a down-rotating component, merging both hands corresponds to merging the anatomical diagram, and separating both hands corresponds to the anatomical component.

[0029] Beneficial effects of the present invention: When the system is used, it needs to be combined with an external blue box facility. The result of gesture recognition will be used to control the virtual studio system. At the beginning of the system, the input image is first detected to see if there is a palm. If there is a palm, the palm is circled with a red circle. Then the palm is tracked in the subsequent input image to ensure its existence. After ensuring the existence of the palm, the optical flow is calculated in the next input image. Gesture recognition is to judge the gesture action based on the average optical flow calculated during the waving period of the palm. After detecting a gesture action, the virtual studio system is controlled by referring to the criteria between the pre-defined gesture action and the operation of the virtual studio system. In this way, the purpose of switching virtual scenes by gestures is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a method for detecting and controlling gestures in a virtual broadcasting system of the present invention;

[0031] Figure 2 A schematic diagram of the direction of gestures in the method for detecting and controlling gestures in a virtual broadcasting system of the present invention;

[0032] Figure 3 A schematic diagram of gesture actions in a method for detecting and controlling gesture actions in a virtual broadcasting system of the present invention; DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0034] In this embodiment, refer to Figure 1-Figure 3 The method for detecting and controlling gestures in a virtual studio system specifically implemented includes the following steps:

[0035] Step S100: palm detection stage, first detect whether there is a palm in the input image, if not, continue to detect the next input image until a palm is detected;

[0036] Step S200: palm tracking stage, when a palm is detected in the input image, the palm is tracked for a certain period of time. If the palm is successfully tracked for a certain period of time, the next stage is entered; if the tracking fails, the palm detection stage is returned;

[0037] Step S300: Optical flow calculation stage, performing optical flow calculation on the input image, and collecting and analyzing the results of the optical flow calculation;

[0038] Step S400: gesture identification stage, collecting and analyzing the results through optical flow calculation to identify the gesture action;

[0039] Step S500: gesture control stage, according to the gesture action identified in the previous stage and in combination with the corresponding relationship between the gesture action and the system operation, the virtual broadcasting system is controlled.

[0040] When this system is used, it needs to be combined with external blue box facilities. The result of gesture recognition will be used to control the virtual studio system. At the beginning of the system, the input image is first detected to see if there is a palm. If there is a palm, the palm is circled with a red circle. Then the palm is tracked in the subsequent input image to ensure its existence. After ensuring the existence of the palm, the optical flow is calculated in the next input image. Gesture recognition is to judge the gesture action based on the average optical flow calculated during the waving period of the palm. After detecting a gesture action, the virtual studio system is controlled by referring to the pre-defined gesture action and the criteria between the operation of the virtual studio system. In order to achieve the purpose of switching virtual scenes through gestures.

[0041] In this embodiment, in step S100, the hierarchical Adaboost technology is used to detect whether a palm appears in the input image. Before executing step S100, the hierarchical Adaboost technology needs to be trained with a specific object set so that the hierarchical Adaboost technology can determine whether a specific object exists in the input image. The specific object can be a face, a fist, or a palm, so as to adapt to different application scenarios. The training method of the hierarchical Adaboost technology includes the following steps:

[0042] Step T1: Let O represent a specific object, train Adaboost to determine whether O appears in an image I, and assume that the specific features of object O form a set F = {f1,f2,···,f T}, for each feature f t Define a classifier h t , that is, h t Only based on the feature f t To determine whether there is object O in image I, it is called h t is a weak classifier of object O, let S = {(x1,y1),(x2,y2),····,(x m ,y m )) is a sample set collected to train the classifier, where x i ∈X is the sample space, y i ∈Y={-1,+1} is the label space, +1 and -1 represent that a sample x is "yes" or "not" the specific object, respectively. Let D t represents the distribution of relative importance among samples at time t. At the beginning, let D1(i) = 1 / m; i = 1, ···, m, indicating that all samples have the same importance;

[0043] Step T2: When using classifier h t To judge a sample x in S i If h t (xi )≠y i Indicates h t Misjudged x i category, then the function 1[h t (x i )≠y i ] indicates that when h t (x i )≠y i When , the value is 1; otherwise, the value is 0. Define the classifier h t The classification error for all samples in S is Let α t The feature f t The weight is defined as α t =ln((1-ε t ) / ε t ) / 2, so when ε t When α increases, t will become smaller, and the distribution of sample importance D t+1 It can be updated according to the following formula, so that h t Misclassified samples, their importance for the next h t+1 will increase, otherwise it will decrease. Where Z t is used for regularization. The above process is repeated for all weak classifiers. Finally, the strong classifier H(x) is formed by combining all weak classifiers. The mathematical table is

[0044] Step T3: Feature set F = {f1,f2,···,f T} can be divided into groups F k ,k=1,··,K, then for each group F k Find a strong classifier H k (x) When performing classification, the hierarchical Adaboost performs H k (x), k = 1, · · , K, only when the sample x passes the classification judgment of the previous stage will the judgment of the subsequent stage be continued. Therefore, in order to speed up the execution of hierarchical Adaboost, the feature clustering F k ,k=1,··,K design uses simple features to prioritize clustering, that is, |F k |<|F k+1 |,k=1,··,K-1 method, so that the front-end classifier H can be executed k (x).

[0045] In this embodiment, in step S300, optical flow calculation is performed only within a limited range near the palm in the input image. The optical flow calculation in step S300 calculates the movement of the object presented on the image when it moves in three-dimensional space. The optical flow calculation needs to be based on the following two assumptions:

[0046] Assumption 1: The brightness of the surface of an object remains constant when moving over a short distance;

[0047] Assumption 2: Adjacent image points in the image move similarly. Let f(x,y,t) represent the grayscale value of the image point (x,y) at time t. Expand it with Taylor series: where f x ,f y ,f t is the partial differential of f with respect to x, y, and t respectively. Assume that the image point near (x, y) is translated by a small distance in the time interval dt. According to the above assumption 1: the brightness of the object surface remains unchanged when moving over a short distance, that is, f(x+dx,y+dy,t+dt)=f(x,y,t), so equation (1) can be written as If dx, dy, dt are small enough, will approach zero, so equation (2) can be rewritten as f x dx+f y dy+f t dt=0.----(3), after sorting, we can get The speed That is the optical flow of the image point (x, y);

[0048] According to the above formula, we can calculate the moving speed (i.e., optical flow) of all image points. However, in order to satisfy the second assumption that adjacent image points in the image move similarly, it is also necessary to perform smooth motion on the optical flow of the image points. This can be done by finding (u, v) to minimize the following objective equation: in is the square of the partial differential of u and v. The above optimization problem can be solved by the Lagrange multiplier method: in is the average value of u, is the average value of v,

[0049] In this embodiment, the gesture identification in step S400 is based on the optical flow vectors calculated from the collected input images, which are used to determine the gesture action after statistical analysis. Let o(x, y, t) represent the optical flow vector calculated at the image point (x, y) at time t, and average the optical flow vectors of the image points within a limited range H×W near the palm. That is, the vector of the palm movement at time t. Assuming that the time period of the palm waving is T, the average optical flow during this period is Gesture recognition is to determine the gesture action according to O. In step S400, the gesture actions include left swipe, right swipe, downward swipe, upward swipe, merging both hands and separating both hands. After the gesture action is recognized, it can be applied to control the operation of specific functions of the virtual studio system according to the corresponding relationship between different gesture actions and system operations. Gesture control will be divided into two control modes, corresponding to the following software versions respectively:

[0050] Version 1: Controls the entry and exit logic of 2D and 3D components in a 3D scene. Swipe left and right respectively correspond to the entry and exit of the background window. Swipe down corresponds to the change of 3D scene. Swipe up corresponds to the weather forecast or data report. Putting both hands together corresponds to adding particle effects. Splitting both hands corresponds to turning off particle effects. Particle effects refer to 3D components that simulate water, fire, fog, gas, and fireworks.

[0051] Version 2: Controls the entry and exit, rotation logic, and anatomical display of 3D components in the 3D scene. A left swipe gesture corresponds to a left-rotating component, a right swipe gesture corresponds to a right-rotating component, an up swipe gesture corresponds to an up-rotating component, a down swipe gesture corresponds to a down-rotating component, merging both hands corresponds to merging the anatomical diagram, and separating both hands corresponds to the anatomical component.

[0052] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention is disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent changes and modifications made to the above embodiments according to the technology of the present invention are within the scope of the technical solution of the present invention without departing from the content of the technical solution of the present invention.

Claims

1. A method for detecting and controlling gestures in a virtual studio system, characterized in that The following steps are involved: Step S100: palm detection stage, first detect whether there is a palm in the input image, if not, continue to detect the next input image until a palm is detected; Step S200: palm tracking stage, when a palm is detected in the input image, the palm is tracked for a certain period of time. If the palm can be successfully tracked for a certain period of time, the next stage is entered; If tracking fails, it returns to the palm detection stage; Step S300: Optical flow calculation stage, performing optical flow calculation on the input image, and collecting and analyzing the results of the optical flow calculation; Step S400: gesture identification stage, collecting and analyzing the results through optical flow calculation to identify the gesture action; Step S500: gesture control stage, according to the gesture action identified in the previous stage, and in combination with the corresponding relationship between the gesture action and the system operation, the virtual studio system is controlled; The optical flow calculation in step S300 is to calculate the movement of the object presented on the image when it moves in three-dimensional space. The optical flow calculation needs to be based on the following two assumptions: Assumption 1: The brightness of the surface of an object remains constant when moving over a short distance; Assumption 2: Adjacent image points in the image move similarly. Let f(x,y,t) represent the grayscale value of the image point (x,y) at time t. Expand it with Taylor series: where f x ,f y ,f t is the partial differential of f with respect to x, y, and t respectively. Assume that the image point near (x, y) is translated by a small distance in the time interval dt. According to the above assumption 1: the brightness of the object surface remains unchanged when moving over a short distance, that is, f(x+dx,y+dy,t+dt)=f(x,y,t), so equation (1) can be written as If dx, dy, dt are small enough, will approach zero, so equation (2) can be rewritten as f x dx+f y dy+f t dt=0.----(3), after sorting, we can get The speed That is the optical flow of the image point (x, y); According to the above formula, we can calculate the moving speed of all image points, that is, the optical flow. However, in order to meet the second assumption: the adjacent image points in the image move similarly, it is also necessary to perform smooth movements on the optical flow of the image points. This can be done by finding (u, v) to minimize the following objective equation, in is the square of the partial differential of u and v. The above optimization problem can be solved by the Lagrange multiplier method: in is the average value of u, is the average value of v, The gesture identification in step S400 is based on the optical flow vectors calculated from the collected input images, which are used to determine the gesture action after statistical analysis. Let o(x, y, t) represent the optical flow vector calculated at the image point (x, y) at time t, and average the optical flow vectors of the image points within a limited range H×W near the palm. That is, the vector of the palm movement at time t. Assuming that the time period of the palm waving is T, the average optical flow during this period is Gesture recognition is to judge the gesture action based on O.

2. The method for detecting and controlling gestures in a virtual broadcasting system according to claim 1, characterized in that: In step S100 , a hierarchical Adaboost technique is used to detect whether a palm appears in the input image.

3. The method for detecting and controlling gestures in a virtual broadcasting system according to claim 2, characterized in that: Before executing step S100 , the hierarchical Adaboost technology needs to be trained with a specific object set so that the hierarchical Adaboost technology can determine whether there is a specific object in the input image.

4. The method for detecting and controlling gestures in a virtual broadcasting system according to claim 3, characterized in that: The specific object may be a human face, a fist, or a palm.

5. The method for detecting and controlling gestures in a virtual broadcasting system according to claim 3, characterized in that: The training method of the hierarchical Adaboost technique comprises the following steps: Step T1: Let O represent a specific object, train Adaboost to determine whether O appears in an image I, and assume that the specific features of object O form a set F = {f1,f2,···,f T }, for each feature f t Define a classifier h t , that is, h t Only based on the feature f t To determine whether there is object O in image I, it is called h t is a weak classifier of object O, let S = {(x1,y1),(x2,y2),····,(x m ,y m )) is a sample set collected to train the classifier, where xi∈X is the sample space, y i ∈Y={-1,+1} is the label space, +1 and -1 represent that a sample x is "yes" or "not" the specific object, respectively. Let D t represents the distribution of relative importance among samples at time t. At the beginning, let D1(i) = 1 / m; i = 1, ···, m, indicating that all samples have the same importance; Step T2: When using classifier h t To judge a sample x in S i If h t (x i )≠y i Indicates h t Misjudged x i category, then the function 1[h t (x i )≠y i ] indicates that when h t (x i )≠y i When , the value is 1; otherwise, the value is 0. Define the classifier h t The classification error for all samples in S is Let α t The feature f t The weight is defined as α t =ln((1-ε t ) / ε t ) / 2, so when ε t When α increases, t will become smaller, and the distribution of sample importance D t+1 It can be updated according to the following formula, so that h t Misclassified samples, their importance for the next h t+1 will increase, otherwise it will decrease. Where Z t It is used for regularization. The above process is repeated for all weak classifiers. The strong classifier H(x) is formed by combining all weak classifiers. The mathematical table is Step T3: Feature set F = {f1,f2,···,f T } can be divided into groups F k ,k=1,··,K, then for each group F k Find a strong classifier H k (x) When performing classification, the hierarchical Adaboost performs H k (x), k = 1, · · , K, only when the sample x passes the classification judgment of the previous stage will the judgment of the subsequent stage be continued. Therefore, in order to speed up the execution of hierarchical Adaboost, the feature clustering F k ,k=1,··,K design uses simple features to prioritize clustering, that is, |F k |<|F k+1 |,k=1,··,K-1 method, so that the front-end classifier H can be executed k (x).

6. The method for detecting and controlling gestures in a virtual broadcasting system according to claim 1, characterized in that: In step S300 , optical flow calculation is performed only within a limited range near the palm in the input image.

7. The method for detecting and controlling gestures in a virtual studio system according to claim 1, characterized in that: The gesture actions in step S400 include left swipe, right swipe, downward swipe, upward swipe, merging both hands together, and separating both hands.

8. The method for detecting and controlling gestures in a virtual studio system according to claim 7, characterized in that: After the gesture recognition is completed, it can be used to control the operation of specific functions of the virtual studio system according to the corresponding relationship between different gestures and system operations. Gesture control will be divided into two control modes, corresponding to the following software versions: Version 1: Controls the entry and exit logic of 2D and 3D components in a 3D scene. Swipe left and right respectively correspond to the entry and exit of the background window. Swipe down corresponds to the change of 3D scene. Swipe up corresponds to the weather forecast or data report. Putting both hands together corresponds to adding particle effects. Splitting both hands corresponds to turning off particle effects. Particle effects refer to 3D components that simulate water, fire, fog, gas, and fireworks. Version 2: Controls the entry and exit, rotation logic, and anatomical display of 3D components in the 3D scene. A left swipe gesture corresponds to a left-rotating component, a right swipe gesture corresponds to a right-rotating component, an up swipe gesture corresponds to an up-rotating component, a down swipe gesture corresponds to a down-rotating component, merging both hands corresponds to merging the anatomical diagram, and separating both hands corresponds to the anatomical component.

Images