Human-computer interaction method, system, device and medium

Abstract

The present invention discloses a human-computer interaction method, system, device, and medium, which relate to the field of computer vision recognition technology. The method comprises: obtaining an action image of a user pointing at an interactive object; recognizing the user's face and hand in the action image to obtain a facial image and a hand image; extracting feature values ​​of the facial image within a coordinate system with a control device as the origin, calibrating the horizontal and vertical angles of the facial image based on the feature values ​​to determine the user's line of sight; establishing a hidden Markov model (HMM) of the hand image's pointing phase features to determine the user's finger pointing; using the intersection of the user's line of sight and the finger pointing as the user input focus, and controlling the interactive object to perform a response action based on the user input focus to achieve human-computer interaction. The present invention improves the accuracy and recognition speed of human-computer interaction.

Description

Technical Field

[0001] The present invention relates to the field of computer vision recognition technology, and in particular to a human-computer interaction method, system, device and medium. Background Art

[0002] Human-Computer Interaction (HCI) is a technology that studies the relationship between humans and computers and their interactions. Its research goal is to utilize all possible information channels for human-computer communication and to improve the naturalness and efficiency of interaction.

[0003] Baluja and Pomerleau proposed a method for inferring a user's gaze point on a computer screen from an image of the human eye. This method uses an image of the eye as input to a neural network to infer the user's gaze position on the screen. The method involves capturing an image of the user's eye using a high-definition camera, ensuring high clarity to accurately capture details such as the pupil and iris. The preprocessed eye image is then fed into the neural network. During the inference phase, the input eye image is converted into a series of feature maps, which are then decoded into the coordinates of the gaze point on the screen.

[0004] The above-mentioned existing technology has a drawback: during human-computer interaction, the user's gaze point on the computer screen is used as a means of determining interactive operation through eye images. This requires the user to be close to the camera and not to turn their head too far. If the user's head turns too far, misjudgment can easily occur, resulting in incorrect interactive operation and failure to achieve the desired interactive effect. Summary of the Invention

[0005] Based on this, it is necessary to provide a human-computer interaction method, system, device and medium to address the above technical problems.

[0006] An embodiment of the present invention provides a human-computer interaction method, including:

[0007] Get the action image when the user points to the interactive object;

[0008] Recognize the user's face and hands on the action image to obtain the user's face image and hand image respectively;

[0009] Extracting feature values ​​of the facial image within a three-dimensional coordinate system established with the control device as the origin, and calibrating the horizontal and vertical angles of the facial image based on the feature values ​​to determine the user's line of sight;

[0010] Extract the pointing phase features of the hand image, including the start, hold, and end phases, build a hidden Markov model (HMM) for each pointing phase feature, and perform optimal path selection on all the hidden Markov models (HMMs) to determine the user's finger pointing;

[0011] The intersection of the user's line of sight and the finger's pointing direction is used as the user input focus, and the interactive object is controlled to perform a response action according to the user input focus to achieve human-computer interaction.

[0012] Optionally, recognizing the face of the user in the action image specifically includes:

[0013] In the RGB space, the stimulus value of the captured image color is:

[0014] ,

[0015] in, f ( l ) represents the relative spectral power distribution of color light perceived by the human eye. r ( l ), g ( l )and b ( l ) are all CIE1964XYZ spectral stimulus values;

[0016] According to the constancy of the human eye in perceiving color, the influence of brightness in representing skin color is removed to obtain the component value of skin color. The calculation formula of the component value is:

[0017] ,

[0018] in, R 、 G and B is the stimulus value of the image color, r The red component, g It is a green ingredient. b It is the blue component;

[0019] Normalize the skin color component values ​​and convert the RGB space into r g Space, skin color r g It is a two-dimensional normal distribution in space;

[0020] Separate the user's face from the background image and perform grayscale processing to obtain a grayscale image;

[0021] Locate facial feature points and extract facial parameters in the grayscale image to obtain the user's facial image.

[0022] Optionally, calibrating the horizontal angle and vertical angle of the user's facial image according to the feature values ​​of the facial image specifically includes:

[0023] Divide the user's facial image vertically and horizontally according to the number of nodes to obtain a standard coding set representing the horizontal and vertical angles of the facial image. Y=[ Y 1 ，Y 2 ，Y 3 ，Y 4 ，Y 5 ]; Y 1 The horizontal angle and vertical angle division results of the facial image at the first node, Y 2 The horizontal angle and vertical angle division results of the facial image at the second node are: Y 3 The horizontal angle and vertical angle division results of the facial image at the third node, Y 4 The horizontal angle and vertical angle division results of the facial image at the fourth node are: Y 5 The horizontal angle and vertical angle division results of the facial image at the fifth node;

[0024] Extract eigenvalues ​​from the facial image and normalize the eigenvalues ​​to obtain the input vector set X = ( x 1 ， x 2 ，… ，x m ) T ;

[0025] Input the input vector into the Gaussian RBF neural network GRBF to obtain the output vector set y =[ y 1 , y 2 , y 3 , y 4 , y 5 , the formula is:

[0026] ,

[0027] in, j 、 m =1, 2, 3, … 、 n , n is the output vector dimension, yj is the first Gaussian RBF neural network GRBF corresponding to the input vector j The output vector of the output node, b j is the basis function width of the network hidden layer nodes; x p For the P input vectors, p= 1, 2, 3, … 、 P , P is the total number of input vectors; i =1, 2, 3, … 、 h,h is the number of hidden layer units, w ij is the connection weight from the hidden layer to the output layer, c i is the center of the network hidden layer node, s is the variance of the basis function;

[0028] Determine confidence based on standard encoding and output vector β Confidence β It is used to characterize the degree of matching between the output vector and the standard encoding. The formula is:

[0029] ,

[0030] in, Y j is an element in the standard encoding set, n and j is a positive integer;

[0031] If the confidence β Greater than or equal to the set threshold α , then the division result corresponding to the standard coding is the horizontal angle and vertical angle of the user's facial image.

[0032] Optionally, all hidden Markov models (HMMs) are subjected to optimal path selection, which specifically includes:

[0033] Construct an observation sequence based on each pointing phase feature of the hand image O = O 1 O 2 O 3 ……O t , ( O t =( oh pan,t Oh, ohtilt,t )),in, oh pan,t is the translational angular velocity of the pointing stage feature at time point t, oh tilt,t is the vertical angular velocity corresponding to the pointing phase characteristic at time point t;

[0034] Build the model i ( A , B ,π), where π is the initial state probability vector, A is the state transition probability matrix, B Observation probability matrix;

[0035] Based on the observation sequence O and model i Construct a hidden Markov model HMM, whose expression P ( O | i )for:

[0036] ,

[0037] in, O is the observation sequence, O 1. O 2. …O r is the value in the observation sequence, Q is the best path, q 1 、 q 2 、 … 、 q r is the value in the best path, b q1 、 b q2 、 … 、 b qr is the value in the observation probability matrix, a q1 、 a q2 、 … 、 a qr is the value in the state transition probability matrix, r =1, 2, 3, … 、 t ;

[0038] Use model i Parameters ( A , B ,π) representsP ( O|θ ),make α ( P ( O|θ )) / ath =0, and the solution is used as the re-estimation formula of each parameter;

[0039] According to the re-estimation formula of each parameter, the P ( O|θ ), obtain the optimal path, and determine the user's finger pointing angle through the optimal path.

[0040] An embodiment of the present invention further provides a human-computer interaction system, comprising:

[0041] An image acquisition module, used to acquire an action image when a user points to an interactive object;

[0042] An image recognition module is used to recognize the user's face and hands in the action image to obtain the user's face image and hand image respectively;

[0043] A facial processing module is used to extract feature values ​​of the facial image within a three-dimensional coordinate system established with the control device as the origin, and to calibrate the horizontal and vertical angles of the facial image based on the feature values ​​to determine the user's line of sight;

[0044] The hand processing module is used to extract the pointing phase features of the hand image, including the start, hold, and end phases, build a hidden Markov model (HMM) for each pointing phase feature, and perform optimal path selection on all the hidden Markov models (HMMs) to determine the user's finger pointing;

[0045] The response module is used to take the intersection of the user's line of sight and the finger's pointing as the user input focus, and control the interactive object to perform a response action according to the user input focus to achieve human-computer interaction.

[0046] An embodiment of the present invention further provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned human-computer interaction method when executing the computer program.

[0047] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned human-computer interaction method are implemented.

[0048] The human-computer interaction method, system, device, and medium provided by the embodiments of the present invention have the following beneficial effects compared to the prior art:

[0049] In existing human-computer interaction technologies, eye images are used to determine the user's gaze point on the computer screen. This requires the user to be close to the camera and not to turn their head too far. Excessive head deflection can easily lead to misjudgment, resulting in incorrect interaction and failure to achieve the desired interaction effect.

[0050] The present invention identifies the user's face and hands on the action image to obtain the user's facial image and hand image respectively, determines the user's line of sight angle based on the facial image, determines the user's finger pointing angle based on the hand image, and takes the intersection of the extended lines of the user's line of sight angle and the finger pointing angle as the line of sight focus; combines the line of sight direction and the finger pointing direction to determine the user's input focus, and determines the user's input focus by combining the line of sight angle and the finger pointing angle, avoiding the use of a single discrimination method, solving the problem of easy misjudgment when using eye images as a discrimination method in the prior art, thereby improving the accuracy and recognition speed of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A head-hand relationship cylindrical coordinate system for a human-computer interaction method provided in one embodiment;

[0052] Figure 2 A head-hand relationship diagram of a pointing action of a human-computer interaction method provided in one embodiment;

[0053] Figure 3 A diagram showing the working principle of a human-computer interaction method provided in one embodiment;

[0054] Figure 4 A schematic flow chart of a human-computer interaction method provided in one embodiment;

[0055] Figure 5 A neural network model diagram of a human-computer interaction method provided in one embodiment;

[0056] Figure 6 A three-state hidden Markov state set topology diagram of a human-computer interaction method provided in one embodiment;

[0057] Figure 7 A screen positioning effect diagram of a human-computer interaction method provided in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] 1. Principle Description

[0060] like Figure 1 As shown, the facial center point after face recognition is taken as the coordinate origin, the direction of sight and finger pointing are in x 、 The projection on the z-axis rotates 360° and forms a cylindrical coordinate system with the vertical y-axis.

[0061] Δ i =( i Head - i Hand ),

[0062] Δ f =( f Head - f Hand ) .

[0063] in, i Head and i Hand Indicates the horizontal angle between the line of sight and the finger pointing, f Head and f Hand Indicates the vertical angle between the line of sight and the finger pointing, Dth and Df Indicates the horizontal and vertical angle difference between the two.

[0064] like Figure 2 As shown in the figure, it is found that the two are basically fixed values ​​during the pointing maintenance phase. In other words, when a person points at an object, the position and angle relationship between the head and hand remain fixed. Therefore, this relationship can be applied to human-computer interaction systems to quickly locate the user's pointing focus.

[0065] In one embodiment, a human-computer interaction method is provided, the method comprising:

[0066] 1. The working principle is as follows: Figure 3 and Figure 4 As shown:

[0067] (1) The user is at an appropriate distance from the interactive object (screen, robot, other device, etc.), and the control device obtains the action image of the user pointing at the interactive object. The action image contains the image of the entire process of the user pointing at the interactive object. Based on the preprocessing of the image, the user's face and hand in the action image are recognized to obtain the user's facial image and hand image respectively. In the three-dimensional coordinate system established with the control device as the origin, the feature value of the facial image is extracted, and the horizontal angle and vertical angle of the facial image are calibrated according to the feature value of the facial image to determine the user's line of sight direction. The hand image is extracted, including the pointing stage features of the start, hold and end, and a hidden Markov model HMM of each pointing stage feature is established. All hidden Markov models HMM are used to select the best path to determine the user's finger pointing. The intersection of the user's line of sight direction and the finger pointing is used as the user input focus. According to the user input focus, the interactive object is controlled to perform a response action to achieve human-computer interaction.

[0068] (2) A Gaussian RBF neural network (Generalized Radial Basis Function, GRBF), which is more suitable for handling highly nonlinear problems, is used to calibrate the orientation of a person's face. The relevant eigenvalues ​​are extracted from the processed facial grayscale image and input into a previously learned and trained neural network. Through comparison and result evaluation, the horizontal and vertical angles of the face are calibrated, thereby determining the direction of sight.

[0069] (3) Based on the characteristics of finger pointing, the Hidden Markov Model (HMM) is used to determine the finger pointing angle. During the interaction, the Hidden Markov Model (HMM) is used to separate this stage and determine the finger pointing.

[0070] (4) When the gaze direction and finger pointing are determined, the intersection of the user's gaze direction and finger pointing is used as the user input focus. The interactive object is controlled to perform a response action according to the user input focus to achieve human-computer interaction.

[0071] 2. Eye direction recognition

[0072] 2.1 Facial Image Acquisition

[0073] According to colorimetry theory, in the RGB space, the stimulus value of the captured image color can be calculated by the following formula:

[0074] .

[0075] in, f ( l ) represents the relative spectral power distribution of color light perceived by the human eye. r ( l ), g ( l ), b ( l ) are all CIE1964XYZ spectral stimulation values, and the integration range is the visible light band, generally 380nm-780nm.

[0076] The brightness of the captured image has a significant impact on the accuracy of recognition. Therefore, based on the constancy of the human eye in perceiving color, the influence of brightness on skin color is removed to obtain the component value of skin color. The calculation formula of the component value is:

[0077] .

[0078] in, R 、 G and B is the stimulus value of the image color, r The red component, g It is a green ingredient. b It is the blue component.

[0079] By normalizing the skin color component values, the RGB space can be converted into r g Space, skin color r g It has a two-dimensional normal distribution in space.

[0080] After distinguishing the user's face from the background image, grayscale processing is performed to obtain a grayscale image. Facial feature points are located in the grayscale image and facial parameters are extracted to obtain the user's facial image.

[0081] 2.1.1 Model Selection

[0082] Since facial orientation recognition is a highly nonlinear problem, Gaussian RBF neural network GRBF is used. Figure 5 As shown, mhn Gaussian RBF neural network GRBF structure, m is the input vector dimension, h is the number of hidden layer units, n is the output vector dimension. X = ( x 1 ， x 2 ，… ，x m ) T is the input vector set of the network, consisting of signal source nodes.

[0083] The hidden layer uses a nonlinear optimization strategy to adjust the parameters of the activation function.h Depends on the problem being described. The radial basis function chooses the Gaussian function, which is:

[0084] .

[0085] in, c i is the center of the network hidden layer node, s is the variance of the basis function, x p For the p input vectors, p= 1, 2, 3 …P , P is the total number of input vectors.

[0086] Since it is difficult to find the specific angle of the facial direction, we can divide the vertical angle and the horizontal angle. Divide the user's facial image vertically and horizontally according to the number of nodes. There are 25 possibilities in total, which can be represented by a standard code consisting of 5 bits. The standard code set representing the horizontal and vertical angles of the facial image is obtained. Y= [ Y1, Y2, Y3, Y4, Y5 ]. Y1 The horizontal angle and vertical angle division results of the facial image at the first node, Y2 The horizontal angle and vertical angle division results of the facial image at the second node are: Y3 The horizontal angle and vertical angle division results of the facial image at the third node, Y4 The horizontal angle and vertical angle division results of the facial image at the fourth node are: Y5 The horizontal angle and vertical angle division results of the facial image at the fifth node are shown in Table 1.

[0087] Table 1 Standard coding of facial orientation

[0088]

[0089] Extract eigenvalues ​​from the facial image and normalize the eigenvalues ​​to obtain the input vector set X = ( x 1 ， x 2 ，… ，x m ) T .

[0090] Input the input vector into the Gaussian RBF neural network GRBF to obtain the output vector set y =[ y1 , y2 ,y3 , y4 , y5 ], and its formula is:

[0091] .

[0092] in, j 、 m =1, 2, 3, … 、 n , n is the output vector dimension, y j is the first Gaussian RBF neural network GRBF corresponding to the input vector j The output vector of the output node, b j is the basis function width of the network hidden layer nodes; x p For the P input vectors, p= 1, 2, 3, … 、 P , P is the total number of input vectors; i =1, 2, 3, … 、 h,h is the number of hidden layer units, w ij is the connection weight from the hidden layer to the output layer, c i is the center of the network hidden layer node, s is the variance of the basis function.

[0093] set up d is the expected output value of the input vector, then the variance of the basis function s It can be expressed as:

[0094] .

[0095] in, c i is the center of the network hidden layer node, y j is the first Gaussian RBF neural network GRBF corresponding to the input vector j The output vector of the output node, s is the variance of the basis function.

[0096] 2.1.2 Results

[0097] Get confidence based on standard encoding and output vector β Confidence β It is used to characterize the degree of matching between the output vector and the standard encoding. The formula is:

[0098] .

[0099] in, Y j is an element in the standard encoding set, n and j Is a positive integer.

[0100] If the confidence β Greater than or equal to the set threshold α , then the division result corresponding to the standard coding is the horizontal angle and vertical angle of the user's facial image; otherwise, it is the facial orientation that the user's facial image corresponding to the standard coding cannot recognize.

[0101] 2.1.3 Gaussian RBF Neural Network (GRBF)

[0102] There are three parameters that need to be solved for the Gaussian RBF neural network GRBF: the center c of the basis function, the variance s i and the weights from the hidden layer to the output layer.

[0103] A. Obtaining the basis function center c based on the K-means clustering method

[0104] ①Network initialization. Random selection h training samples as cluster centers c i ( i =1, 2,…, h )

[0105] ② Group the input training sample set according to the nearest neighbor rule x p with the center c i The Euclidean distance between x p Assign to each cluster set d p ( p =1,2, … , P )middle.

[0106] ③Re-adjust the cluster center. Calculate each cluster set d p The average value of the training samples in the new cluster center c i , if the new cluster center does not change, then the obtained c i That is the final basis function center of the GRBF neural network cOtherwise, return to ② and enter the next round of center solution.

[0107] B. Solving for variance s i

[0108] .

[0109] in, c max is the maximum distance between the selected centers, i =1,2, … , h .

[0110] C. Calculate the weights between the hidden layer and the output layer w

[0111] The connection weights of neurons from the hidden layer to the output layer are directly calculated using the least squares method. The calculation formula is as follows:

[0112] .

[0113] in, p =1, 2, ..., P , P is the total number of input vectors; i =1, 2,…, h.

[0114] 2.2 Finger Pointing Recognition

[0115] The pointing process of the hand is divided into three pointing phase features, namely start, hold and end. For these three pointing phase features, a respective hidden Markov model HMM is established, and its state set topology is as follows: Figure 6 As shown, all hidden Markov models (HMMs) are used to select the best path to determine the pointing angle of the user's finger.

[0116] Construct an observation sequence based on each pointing phase feature of the hand image O = O 1 O 2 O 3 ……O t , ( O t =( oh pan,t Oh, oh tilt,t )).in, oh pan,t is the translational angular velocity of the pointing stage feature at time point t, ohtilt,t is the vertical angular velocity corresponding to the pointing phase characteristic at time point t.

[0117] Build the model i ( A , B ,π), where π is the initial state probability vector, A is the state transition probability matrix, B Observation probability matrix.

[0118] Based on the observation sequence O and model i Construct a hidden Markov model HMM, whose expression P ( O | i )for:

[0119] .

[0120] in, O is the observation sequence, O 1. O 2. …、O r is the value in the observation sequence, Q is the best path, q 1 、 q 2 、 … 、 q r is the value in the best path, b q1 、 b q2 、 … 、 b qr is the value in the observation probability matrix, a q1 、 a q2 、 … 、 a qr is the value in the state transition probability matrix, r =1, 2, 3, … 、 t .

[0121] Then, choose the best path Q = q 1 q 2 q 3 …q tThe Segmental K-means algorithm based on Viterbi decoding algorithm is used. The basic idea is to use the model i Parameters ( A , B ,π) represents P ( O|θ ),make α ( P ( O|θ )) / ath =0, and the solution is used as the re-estimation formula of each parameter. According to the re-estimation formula of each parameter, maximize P ( O|θ ), obtain the optimal path, and determine the user's finger pointing angle through the optimal path.

[0122] 3. Experimental results and analysis

[0123] The computer used in the experiment had a main frequency of 3.0 GHz and 2 GB of memory. The image acquisition device was a standard camera, and the captured image size was 352 × 288. After processing, the user's image was converted into mouse control signals to control the computer. The visual processing speed in the experiment reached 15 frames per second. For screen positioning, the computer displayed a 10 × 10 table image in full screen. The experimenter pointed to each cell in the table one by one to test the accuracy of the screen positioning.

[0124] Table 2 lists the test data of the system's screen positioning accuracy under different lighting conditions when the experimenter is 3 m away from the camera.

[0125] When the lighting conditions are good, the present invention can better locate the screen and recognize the user's gestures. When the lighting conditions are poor, the grayscale of the image will be affected, resulting in a decrease in the accuracy of human eye detection and hand detection, thereby affecting the accuracy of the system.

[0126] Table 2 Screen positioning test

[0127]

[0128] Figure 7 The test data of the system's screen positioning and gesture recognition accuracy at different distances from the experimenter to the camera under good lighting conditions are given. Due to the limitation of the resolution of the collected image, when the distance from the user to the camera increases, the accuracy of the system's screen positioning and gesture recognition decreases to a certain extent. Figure 7 It can be seen that compared with traditional gaze tracking and gesture recognition technologies, the present invention has great advantages in long-distance human-computer interaction.

[0129] Based on the same inventive concept, the present invention also provides a human-computer interaction system, which includes:

[0130] An image acquisition module, used to acquire an action image when a user points to an interactive object;

[0131] An image recognition module is used to recognize the user's face and hands in the action image to obtain the user's face image and hand image respectively;

[0132] A facial processing module is used to extract feature values ​​of the facial image within a three-dimensional coordinate system established with the control device as the origin, and to calibrate the horizontal and vertical angles of the facial image based on the feature values ​​to determine the user's line of sight;

[0133] The hand processing module is used to extract the pointing phase features of the hand image, including the start, hold, and end phases, build a hidden Markov model (HMM) for each pointing phase feature, and perform optimal path selection on all the hidden Markov models (HMMs) to determine the user's finger pointing;

[0134] The response module is used to take the intersection of the user's line of sight and the finger's pointing as the user input focus, and control the interactive object to perform a response action according to the user input focus to achieve human-computer interaction.

[0135] Furthermore, the present invention also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the human-computer interaction method. Specific implementation methods can be found in the method embodiments and will not be repeated here.

[0136] Furthermore, the present invention also provides a storage medium storing a computer program, such as a memory containing instructions, which are executable by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device. When executed by the processor, this computer program can implement the steps in the embodiments of the human-computer interaction method. Specific implementation methods can be found in the method embodiments and are not further described here.

[0137] The above embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A human-computer interaction method, characterized in that: include: Get the action image when the user points to the interactive object; Recognize the user's face and hands on the action image to obtain the user's face image and hand image respectively; Extracting feature values ​​of the facial image within a three-dimensional coordinate system established with the control device as the origin, and calibrating the horizontal and vertical angles of the facial image based on the feature values ​​to determine the user's line of sight; Extract the pointing phase features of the hand image, including the start, hold, and end phases, build a hidden Markov model (HMM) for each pointing phase feature, and perform optimal path selection on all the hidden Markov models (HMMs) to determine the user's finger pointing; The intersection of the user's line of sight and the finger's direction is used as the user input focus, and the interactive object is controlled to perform a response action according to the user input focus to achieve human-computer interaction; The step of calibrating the horizontal angle and the vertical angle of the facial image according to the characteristic values ​​of the facial image specifically includes: Divide the facial image vertically and horizontally according to the number of nodes to obtain a standard coding set Y = [Y1, Y2, Y3, Y4, Y5] representing the horizontal angle and vertical angle of the facial image; Y1 is the horizontal angle and vertical angle division result of the facial image at the first node, Y2 is the horizontal angle and vertical angle division result of the facial image at the second node, Y3 is the horizontal angle and vertical angle division result of the facial image at the third node, Y4 is the horizontal angle and vertical angle division result of the facial image at the fourth node, and Y5 is the horizontal angle and vertical angle division result of the facial image at the fifth node; Extract eigenvalues ​​from the facial image, normalize the eigenvalues, and obtain the input vector set X = (x1, x2, ..., x m ) T ; Input the input vector into the Gaussian RBF neural network GRBF and obtain the output vector set y = [y1, y2, y3, y4, y5]; Among them, y j is the output vector of the j-th output node of the Gaussian RBF neural network GRBF corresponding to the input vector, j = 1, 2, 3, ..., n, n is the output vector dimension; A confidence level β is determined based on the standard code and the output vector; the confidence level β is used to characterize the degree of match between the output vector and the standard code; if the confidence level β is greater than or equal to a set threshold α, the division result corresponding to the standard code is the horizontal angle and the vertical angle of the facial image.

2. A human-computer interaction method according to claim 1, characterized in that: The recognition of the user's face on the action image specifically includes: In the RGB space, the stimulus value of the captured image color is: in, Represents the relative spectral power distribution of color light perceived by the human eye. r(λ), g(λ), and b(λ) are all CIE1964XYZ spectral stimulus values. According to the constancy of the human eye in perceiving color, the influence of brightness in representing skin color is removed to obtain the component value of skin color. The calculation formula of the component value is: Among them, R, G, and B are the stimulus values ​​of the image color, r is the red component, g is the green component, and b is the blue component; Normalize the skin color component value and convert the RGB space into r g Space, skin color in r g It is a two-dimensional normal distribution in space; Separate the user's face from the background image and perform grayscale processing to obtain a grayscale image; Locate facial feature points and extract facial parameters in the grayscale image to obtain the user's facial image.

3. The human-computer interaction method according to claim 1, wherein: The input vector is input into the Gaussian RBF neural network GRBF to obtain the output vector set y=[y1, y2, y3, y4, y5], which is formulated as follows: Among them, b j is the basis function width of the network hidden layer nodes; x p is the Pth input vector, p=1, 2, 3, ..., P, P is the total number of input vectors; i=1, 2, 3, ..., h, h is the number of hidden layer units, w ij is the connection weight from the hidden layer to the output layer, c i is the center of the network hidden layer node, σ is the variance of the basis function; The confidence level β is determined based on the standard code and the output vector. The confidence level β is used to characterize the degree of matching between the output vector and the standard code, and its formula is: Among them, Y j is an element in the standard encoding set, and n and j are positive integers.

4. A human-computer interaction system, characterized in that: include: An image acquisition module, used to acquire an action image when a user points to an interactive object; An image recognition module is used to recognize the user's face and hands in the action image to obtain the user's face image and hand image respectively; A facial processing module is used to extract feature values ​​of the facial image within a three-dimensional coordinate system established with the control device as the origin, and to calibrate the horizontal and vertical angles of the facial image based on the feature values ​​to determine the user's line of sight; The step of calibrating the horizontal angle and the vertical angle of the facial image according to the characteristic values ​​of the facial image specifically includes: Divide the facial image vertically and horizontally according to the number of nodes to obtain a standard coding set Y = [Y1, Y2, Y3, Y4, Y5] representing the horizontal angle and vertical angle of the facial image; Y1 is the horizontal angle and vertical angle division result of the facial image at the first node, Y2 is the horizontal angle and vertical angle division result of the facial image at the second node, Y3 is the horizontal angle and vertical angle division result of the facial image at the third node, Y4 is the horizontal angle and vertical angle division result of the facial image at the fourth node, and Y5 is the horizontal angle and vertical angle division result of the facial image at the fifth node; Extract eigenvalues ​​from the facial image, normalize the eigenvalues, and obtain the input vector set X = (x1, x2, ..., x m ) T ; Input the input vector into the Gaussian RBF neural network GRBF and obtain the output vector set y = [y1, y2, y3, y4, y5]; Among them, y j is the output vector of the j-th output node of the Gaussian RBF neural network GRBF corresponding to the input vector, j = 1, 2, 3, ..., n, n is the output vector dimension; Determining a confidence level β based on the standard code and the output vector; the confidence level β is used to characterize the degree of match between the output vector and the standard code; if the confidence level β is greater than or equal to a set threshold α, the division result corresponding to the standard code is the horizontal angle and the vertical angle of the facial image; The hand processing module is used to extract the pointing phase features of the hand image, including the start, hold, and end phases, build a hidden Markov model (HMM) for each pointing phase feature, and perform optimal path selection on all the hidden Markov models (HMMs) to determine the user's finger pointing; The response module is used to take the intersection of the user's line of sight and the finger's pointing as the user input focus, and control the interactive object to perform a response action according to the user input focus to achieve human-computer interaction.

5. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the human-computer interaction method according to any one of claims 1 to 3 are implemented.

6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the human-computer interaction method according to any one of claims 1 to 3 are implemented.

Images