Human-computer interaction method, system, equipment and medium

By identifying and processing the user's facial and hand movements, combining the line of sight and finger pointing angles, and determining the input focus, the problem of misjudgment of eye images in the prior art is solved, and more accurate and efficient human-computer interaction is achieved.

CN119045668AActive Publication Date: 2024-11-29SHIJIAZHUANG UNIVERSITY
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411235489.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-11-29
Estimated Expiration
2044-09-04

AI Technical Summary

Technical Problem

In the prior art, when human-computer interaction, there is a problem of misjudgment of the user's gaze point through eye images, especially when the user's head deflects too large, it is easy to lead to erroneous interaction operations, and the expected interaction effect cannot be achieved.

Method used

By obtaining user's motion images, identifying face and hands, extracting feature values ​​of face and hands, establishing a three-dimensional coordinate system, determining the line of sight and finger pointing angle, combining the two to determine the input focus, using Gaussian RBF neural network and Hidmaer The Kov model is processed to achieve accurate interactive operation.

Benefits of technology

It improves the accuracy and recognition speed of human-computer interaction, avoids the problem of misjudgment of a single discrimination method, and enhances the naturalness and efficiency of interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045668A_ABST
    Figure CN119045668A_ABST
Patent Text Reader

Abstract

The invention discloses a man-machine interaction method, system and device and a medium, and relates to the technical field of computer visual recognition. Comprises: acquiring an action image when a user points to an interactive object; identifying the face and the hand of the user on the action image to obtain a face image and a hand image; in a coordinate system taking the control equipment as an original point, extracting a characteristic value of the face image, and calibrating a horizontal angle and a vertical angle of the face image according to the characteristic value so as to determine the sight line direction of the user; establishing an HMM (Hidden Markov Model) of hand image pointing stage features to determine finger pointing of the user; and taking an intersection point of the sight line direction of the user and finger pointing as a user input focus, and controlling the interactive object to execute a response action according to the user input focus so as to realize man-machine interaction. According to the invention, the accuracy and recognition speed of man-machine interaction are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision recognition technology, and in particular to a human-computer interaction method, system, device and medium. Background Art

[0002] Human-Computer Interaction (HCI) is the technology that studies the relationship between humans and computers and their interactions. Its research purpose is to utilize all possible information channels for human-computer communication and to improve the naturalness and efficiency of the interaction.

[0003] In the prior art, Baluja and Pomerleau proposed a method to infer the user's gaze point on the computer screen from the image of the human eye, and input the human eye image into the neural network to infer the gaze position of the eye on the computer screen. The specific process includes: capturing the user's eye image through a high-definition camera to ensure high image clarity so as to accurately capture details such as the pupil and iris. The pre-processed eye image is input into the neural network. In the inference stage of the neural network, the input eye image is converted into a series of feature maps, which are further decoded into the coordinates of the gaze point on the screen.

[0004] The defects of the above-mentioned prior art are: during human-computer interaction, the user's gaze point on the computer screen uses the eye image as a means of judging the interactive operation, requiring the user to be close to the camera and not to deflect the head too much. When the customer's head deflects too much, it is easy to misjudge, resulting in erroneous interactive operations, and failing to achieve the expected interactive effect. Summary of the invention

[0005] Based on this, it is necessary to provide a human-computer interaction method, system, device and medium to address the above technical issues.

[0006] An embodiment of the present invention provides a human-computer interaction method, including:

[0007] Get the action image when the user points to the interactive object;

[0008] Recognize the face and hand of the user in the action image to obtain the face image and hand image of the user respectively;

[0009] In a three-dimensional coordinate system established with the control device as the origin, extracting feature values ​​of the facial image, and calibrating the horizontal angle and vertical angle of the facial image according to the feature values ​​of the facial image to determine the sight direction of the user;

[0010] Extract the pointing stage features of the hand image including the start, hold and end, establish the hidden Markov model HMM of each pointing stage feature, and select the best path for all the hidden Markov models HMM to determine the finger pointing of the user;

[0011] The intersection of the user's line of sight and the finger's pointing direction is taken as the user input focus, and the interactive object is controlled to perform a response action according to the user input focus to achieve human-computer interaction.

[0012] Optionally, recognizing the face of the user in the action image specifically includes:

[0013] In the RGB space, the stimulus value of the captured image color is:

[0014]

[0015] in, Represents the relative spectral power distribution of color light perceived by the human eye. r(λ), g(λ) and b(λ) are all CIE1964XYZ spectral stimulus values.

[0016] According to the constancy of human eyes in perceiving colors, the influence of brightness in characterizing skin color is removed to obtain the component value of skin color. The calculation formula of the component value is:

[0017]

[0018] Among them, R, G and B are the stimulus values ​​of the image color, r is the red component, g is the green component, and b is the blue component;

[0019] Normalize the skin color component values ​​and convert the RGB space into r g Space, skin color in r g It is normally distributed in two dimensions in space;

[0020] The user's face is distinguished from the background image and grayscale processed to obtain a grayscale image;

[0021] The facial feature points are located in the grayscale image and the facial parameters are extracted to obtain the user's facial image.

[0022] Optionally, calibrating the horizontal angle and the vertical angle of the user's facial image according to the feature value of the facial image specifically includes:

[0023] Divide the user's facial image in the vertical direction and the horizontal direction according to the number of nodes to obtain a standard coding set Y=[Y1, Y2, Y3, Y4, Y5] representing the horizontal angle and vertical angle of the facial image; Y1 is the horizontal angle and vertical angle division result of the facial image at the first node, Y2 is the horizontal angle and vertical angle division result of the facial image at the second node, Y3 is the horizontal angle and vertical angle division result of the facial image at the third node, Y4 is the horizontal angle and vertical angle division result of the facial image at the fourth node, and Y5 is the horizontal angle and vertical angle division result of the facial image at the fifth node;

[0024] Extract eigenvalues ​​from the facial image and normalize them to obtain the input vector set X = (x1, x2, ..., x m ) T ;

[0025] Input the input vector into the Gaussian RBF neural network GRBF, and get the output vector set y = [y1, y2, y3, y4, y5], the formula is:

[0026]

[0027] Where j, m = 1, 2, 3, ..., n, n is the output vector dimension, y j is the output vector of the jth output node of the Gaussian RBF neural network GRBF corresponding to the input vector, b j is the basis function width of the network hidden layer nodes; x p is the Pth input vector, p = 1, 2, 3, ..., P, P is the total number of input vectors; i = 1, 2, 3, ..., h, h is the number of hidden layer units, w ij is the connection weight from the hidden layer to the output layer, c i is the center of the network hidden layer nodes, σ is the variance of the basis function;

[0028] The confidence β is determined based on the standard encoding and the output vector; the confidence β is used to characterize the matching degree between the output vector and the standard encoding, and its formula is:

[0029]

[0030] Among them, y j is the output vector, Y j is the standard encoding, n and j are positive integers;

[0031] If the confidence β is greater than or equal to the set threshold α, the division result corresponding to the standard coding is the horizontal angle and vertical angle of the user's facial image.

[0032] Optionally, all hidden Markov models HMMs are subjected to optimal path selection, which specifically includes:

[0033] Based on the features of each pointing stage of the hand image, an observation value sequence O=O1O2O3...O is constructed t , (O t =(ω pan ,t,ω tilt,t )), where ω pan,t is the translation angular velocity of the pointing stage feature corresponding to time point t, ω tilt,t is the vertical angular velocity corresponding to the pointing phase characteristic at time point t;

[0034] Construct the model θ(A, B, π), where π is the initial state probability vector, A is the state transition probability matrix, and B is the observation probability matrix;

[0035] Based on the observation sequence O and the model θ, a hidden Markov model HMM is constructed, and its expression P(O|θ) is:

[0036]

[0037] Among them, O is the observation value sequence, O1, O2, ...O r is the value in the observation sequence, Q is the best path, q1, q2, …, q r is the value in the best path, b q1 、b q2 , …, b qr is the value in the observation probability matrix, a q1 、a q2 , …, a qr is the value in the state transition probability matrix, r = 1, 2, 3, ..., t;

[0038] Denote P(O|θ) by the parameters (A, B, π) of model θ, let α(P(O|θ)) / αθ=0, and use the solution as the re-estimation formula of each parameter;

[0039] The optimal path is obtained by maximizing P(O|θ) according to the re-estimation formula of each parameter, and the pointing angle of the user's finger is determined through the optimal path.

[0040] The embodiment of the present invention further provides a human-computer interaction system, comprising:

[0041] An image acquisition module, used to acquire an action image when a user points to an interactive object;

[0042] An image recognition module, used to recognize the face and hand of the user in the action image, and obtain the face image and hand image of the user respectively;

[0043] A facial processing module, used to extract feature values ​​of the facial image in a three-dimensional coordinate system established with the control device as the origin, and calibrate the horizontal angle and vertical angle of the facial image according to the feature values ​​of the facial image to determine the user's line of sight direction;

[0044] The hand processing module is used to extract the pointing stage features of the hand image including the start, hold and end, establish the hidden Markov model HMM of each pointing stage feature, and select the best path for all the hidden Markov models HMM to determine the finger pointing of the user;

[0045] The response module is used to take the intersection of the user's sight direction and the finger pointing as the user input focus, and control the interactive object to perform a response action according to the user input focus to achieve human-computer interaction.

[0046] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned human-computer interaction method when executing the computer program.

[0047] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned human-computer interaction method are implemented.

[0048] Compared with the prior art, the human-computer interaction method, system, device and medium provided by the embodiments of the present invention have the following beneficial effects:

[0049] In the existing technology, when interacting with a computer, the user's gaze point on the computer screen is used as a means of judging the interactive operation through the eye image, which requires the user to be close to the camera and not to turn the head too much. When the customer's head turns too much, it is easy to misjudge, resulting in incorrect interactive operations and failing to achieve the expected interactive effect.

[0050] The present invention recognizes the face and hands of the user in the action image to obtain the facial image and hand image of the user respectively, determines the user's sight angle based on the facial image, determines the user's finger pointing angle based on the hand image, and takes the intersection of the extended lines of the user's sight angle and the finger pointing angle as the sight focus; determines the user input focus in combination with the sight direction and the finger pointing direction, and determines the user input focus by combining the sight angle and the finger pointing angle, avoiding the use of a single discrimination means, and solving the problem of easy misjudgment when using eye images as a discrimination means in the prior art, thereby improving the accuracy and recognition speed of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A head-hand relationship cylindrical coordinate system of a human-computer interaction method provided in one embodiment;

[0052] Figure 2 A head-hand relationship diagram of a pointing action of a human-computer interaction method provided in one embodiment;

[0053] Figure 3 A working principle diagram of a human-computer interaction method provided in one embodiment;

[0054] Figure 4 A schematic diagram of a flow chart of a human-computer interaction method provided in an embodiment;

[0055] Figure 5 A neural network model diagram of a human-computer interaction method provided in an embodiment;

[0056] Figure 6 A three-state hidden Markov state set topology diagram of a human-computer interaction method provided in an embodiment;

[0057] Figure 7 A screen positioning effect diagram of a human-computer interaction method provided in an embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] 1. Principle description

[0060] like Figure 1 As shown in FIG. 1 , the facial center point after face recognition is taken as the coordinate origin, and the projection of the sight direction and finger pointing on the x-axis and the z-axis is rotated 360° to form a cylindrical coordinate system with the vertical y-axis.

[0061] Δθ=(θ Head -θ Hand ),

[0062]

[0063] Among them, θ Head and θ Hand Indicates the horizontal angle between the line of sight and the finger pointing, and represents the vertical angle between the line of sight and the finger pointing, Δθ and Indicates the horizontal and vertical angle difference between the two.

[0064] like Figure 2As shown in the figure, it is found that the two are basically fixed values ​​in the pointing holding stage, that is, when the human body points to an object, the position and angle relationship of the head and hand are fixed. Therefore, this relationship between the two can be applied to the human-computer interaction system to quickly locate the user's pointing focus.

[0065] In one embodiment, a human-computer interaction method is provided, the method comprising:

[0066] 1. The working principle is as follows: Figure 3 and Figure 4 As shown:

[0067] (1) When the user is at an appropriate distance from an interactive object (screen, robot, other device, etc.), the control device obtains an action image of the user pointing at the interactive object. The action image includes an image of the entire process of the user pointing at the interactive object. Based on the preprocessing of the image, the user's face and hand in the action image are recognized to obtain the user's face image and hand image respectively. In a three-dimensional coordinate system established with the control device as the origin, the feature value of the facial image is extracted, and the horizontal angle and vertical angle of the facial image are calibrated according to the feature value of the facial image to determine the user's line of sight direction. The hand image is extracted including the pointing stage features of the start, hold and end, and a hidden Markov model HMM of each pointing stage feature is established. All hidden Markov models HMM are used to select the best path to determine the user's finger pointing. The intersection of the user's line of sight direction and the finger pointing is taken as the user input focus, and the interactive object is controlled to perform a response action according to the user input focus to achieve human-computer interaction.

[0068] (2) A Gaussian RBF neural network (Generalized Radial Basis Function, GRBF) is used to calibrate the orientation of a person's face, which is more suitable for dealing with highly nonlinear problems. The relevant eigenvalues ​​are extracted from the processed facial grayscale image and input into a neural network that has been learned and trained in advance. Through comparison and result evaluation, the horizontal and vertical angles of the person's face are calibrated, thereby determining the direction of sight.

[0069] (3) According to the characteristics of finger pointing, the hidden Markov model (HMM) is used to determine the finger pointing angle. During the interaction, the hidden Markov model HMM is used to separate this stage and determine the finger pointing.

[0070] (4) When the gaze direction and finger pointing are determined, the intersection of the user's gaze direction and finger pointing is used as the user input focus. The interactive object is controlled to perform a response action according to the user input focus to achieve human-computer interaction.

[0071] 2. Eye direction recognition

[0072] 2.1 Facial Image Acquisition

[0073] According to the colorimetric theory, in the RGB space, the stimulus value of the captured image color can be calculated by the following formula:

[0074]

[0075] in, Represents the relative spectral power distribution of color light perceived by the human eye. r(λ), g(λ), and b(λ) are all CIE1964XYZ spectral stimulus values. The integration range is the visible light band, generally 380nm-780nm.

[0076] The brightness of the captured image has a great influence on the accuracy of recognition. Therefore, based on the constancy of the human eye in perceiving color, the influence of brightness in representing skin color is removed to obtain the component value of skin color. The calculation formula of the component value is:

[0077]

[0078] Among them, R, G and B are the stimulus values ​​of the image color, r is the red component, g is the green component, and b is the blue component.

[0079] By normalizing the skin color component values, the RGB space can be converted into r g Space, skin color in r g It has a two-dimensional normal distribution in space.

[0080] After distinguishing the user's face from the background image, grayscale processing is performed to obtain a grayscale image. In the grayscale image, facial feature points are located and facial parameters are extracted to obtain the user's facial image.

[0081] 2.1.1 Model selection

[0082] Since facial orientation recognition is a highly nonlinear problem, Gaussian RBF neural network GRBF is used. Figure 5 As shown, it is a Gaussian RBF neural network GRBF with mhn structure, m is the input vector dimension, h is the number of hidden layer units, and n is the output vector dimension. m ) T is the input vector set of the network, consisting of signal source nodes.

[0083] The hidden layer uses a nonlinear optimization strategy to adjust the parameters of the activation function. The number of hidden layer units h is determined by the described problem. The radial basis function selects the Gaussian function, that is:

[0084]

[0085] Among them, c i is the center of the hidden layer node of the network, σ is the variance of the basis function, x p is the pth input vector, p=1, 2, 3…P, P is the total number of input vectors.

[0086] Since it is difficult to find the specific angle of the facial direction, the vertical angle and the horizontal angle can be divided. The user's facial image is divided vertically and horizontally according to the number of nodes. There are 25 possibilities in total, which can be represented by a standard code consisting of 5-bit codes. The standard code set Y = [Y1, Y2, Y3, Y4, Y5] representing the horizontal angle and vertical angle of the facial image is obtained. Y1 is the horizontal angle and vertical angle division result of the facial image at the first node, Y2 is the horizontal angle and vertical angle division result of the facial image at the second node, Y3 is the horizontal angle and vertical angle division result of the facial image at the third node, Y4 is the horizontal angle and vertical angle division result of the facial image at the fourth node, and Y5 is the horizontal angle and vertical angle division result of the facial image at the fifth node. The standard coding scheme is shown in Table 1:

[0087] Table 1 Standard coding of facial orientation

[0088]

[0089] Extract eigenvalues ​​from the facial image and normalize them to obtain the input vector set X = (x1, x2, ..., x m ) T .

[0090] Input the input vector into the Gaussian RBF neural network GRBF, and get the output vector set y = [y1, y2, y3, y4, y5], the formula is:

[0091]

[0092] Where j, m = 1, 2, 3, ..., n, n is the output vector dimension, y j is the output vector of the jth output node of the Gaussian RBF neural network GRBF corresponding to the input vector, b j is the basis function width of the network hidden layer nodes; x p is the Pth input vector, p = 1, 2, 3, ..., P, P is the total number of input vectors; i = 1, 2, 3, ..., h, h is the number of hidden layer units, w ij is the connection weight from the hidden layer to the output layer, c i is the center of the hidden layer nodes of the network, and σ is the variance of the basis function.

[0093] Let d be the expected output value of the input vector, then the variance σ of the basis function can be expressed as:

[0094]

[0095] Among them, c i is the center of the network hidden layer node, y j is the output vector of the jth output node of the Gaussian RBF neural network GRBF corresponding to the input vector, and σ is the variance of the basis function.

[0096] 2.1.2 Results

[0097] Based on the standard encoding and the output vector, the confidence β is obtained; the confidence β is used to characterize the matching degree between the output vector and the standard encoding, and its formula is:

[0098]

[0099] Among them, y j is the output vector, Y j is the standard encoding, n and j are positive integers.

[0100] If the confidence β is greater than or equal to the set threshold α, the division result corresponding to the standard coding is the horizontal angle and vertical angle of the user's facial image; otherwise, it is the facial orientation that cannot be recognized by the user's facial image corresponding to the standard coding.

[0101] 2.1.3 Gaussian RBF Neural Network GRBF

[0102] There are three parameters that need to be solved for the Gaussian RBF neural network GRBF: the center c of the basis function, the variance σ i And the weights from the hidden layer to the output layer.

[0103] A. Obtaining the basis function center c based on the K-means clustering method

[0104] ① Network initialization. Randomly select h training samples as cluster centers c i (i=1,2,…,h)

[0105] ② Group the input training sample set according to the nearest neighbor rule according to x p With center c i The Euclidean distance between p Assigned to each cluster set δ p (p=1, 2, …, P).

[0106] ③Re-adjust the cluster center. Calculate each cluster set δ p The average value of the training samples in the new cluster center c i, if the new cluster center does not change, the obtained c i This is the final basis function center c of the GRBF neural network. Otherwise, return to ② and enter the next round of center solution.

[0107] B. Solving for the variance σ i

[0108]

[0109] Among them, c max is the maximum distance between the selected centers, i = 1, 2, …, h.

[0110] C. Calculate the weight w between the hidden layer and the output layer

[0111] The connection weights of neurons from the hidden layer to the output layer are directly calculated using the least squares method. The calculation formula is as follows:

[0112]

[0113] Wherein, p=1, 2, ..., P, P is the total number of input vectors; i=1, 2, ..., h.

[0114] 2.2 Finger pointing recognition

[0115] The hand pointing process is divided into three pointing phase features, namely, start, hold and end. For these three pointing phase features, respective hidden Markov models (HMMs) are established, and their state set topology is as follows: Figure 6 As shown, all hidden Markov models HMMs are used to select the best path to determine the pointing angle of the user's finger.

[0116] Based on the features of each pointing stage of the hand image, an observation value sequence O=O1O2O3...O is constructed t , (O t =(ω pan,t ,ω tilt,t )). Among them, ω pan,t is the translation angular velocity of the pointing stage feature corresponding to time point t, ω tilt,t is the vertical angular velocity corresponding to the pointing stage characteristics at time point t.

[0117] Construct the model θ(A, B, π), where π is the initial state probability vector, A is the state transition probability matrix, and B is the observation probability matrix.

[0118] Based on the observation sequence O and the model θ, a hidden Markov model HMM is constructed, and its expression P(O|θ) is:

[0119]

[0120] Among them, O is the observation value sequence, O1, O2, ..., O r is the value in the observation sequence, Q is the best path, q1, q2, …, q r is the value in the best path, b q1 、b q2 , …, b qr is the value in the observation probability matrix, a q1 、a q2 , …, a qr is the value in the state transition probability matrix, r = 1, 2, 3, …, t.

[0121] Then, choose the best path Q = q1q2q3…q t The Segmental K-means algorithm based on the Viterbi decoding algorithm is used. The basic idea is: use the parameters (A, B, π) of the model θ to represent P(O|θ), let α(P(O|θ)) / αθ=0, and use the solution as the re-estimation formula of each parameter. According to the re-estimation formula of each parameter, P(O|θ) is maximized to obtain the optimal path, and the user's finger pointing angle is determined through the optimal path.

[0122] 3. Experimental results and analysis

[0123] The main frequency of the computer used in the experiment is 3.0GHz, the memory is 2GB, the image acquisition device is an ordinary camera, and the size of the acquired image is 352×288. After processing, the user image will be converted into a mouse control signal to control the computer. The speed of the visual processing part in the experiment can reach 15 frames / s. In screen positioning, the computer displays an image of a 10×10 table in full screen, and the experimenter points to each cell in the table in turn to test the accuracy of screen positioning.

[0124] Table 2 lists the test data of the system's screen positioning accuracy under different lighting conditions when the experimenter is 3m away from the camera.

[0125] When the light conditions are good, the present invention can better locate the screen and recognize the user's gestures. When the light conditions are poor, the grayscale of the image will be affected, resulting in a decrease in the accuracy of human eye detection and hand detection, thereby affecting the accuracy of the system.

[0126] Table 2 Screen positioning test

[0127]

[0128] Figure 7The test data of the screen positioning and gesture recognition accuracy of the system at different distances from the experimenter to the camera under good lighting conditions are given. Due to the limitation of the resolution of the collected image, when the distance from the user to the camera increases, the accuracy of the screen positioning and gesture recognition of the system decreases to a certain extent. Figure 7 It can be seen that compared with traditional eye tracking and gesture recognition technologies, the present invention has great advantages in long-distance human-computer interaction.

[0129] Based on the same inventive concept, the present invention also provides a human-computer interaction system, which includes:

[0130] An image acquisition module, used to acquire an action image when a user points to an interactive object;

[0131] An image recognition module, used to recognize the face and hand of the user in the action image, and obtain the face image and hand image of the user respectively;

[0132] A facial processing module, used to extract feature values ​​of the facial image in a three-dimensional coordinate system established with the control device as the origin, and calibrate the horizontal angle and vertical angle of the facial image according to the feature values ​​of the facial image to determine the user's line of sight direction;

[0133] The hand processing module is used to extract the pointing stage features of the hand image including the start, hold and end, establish the hidden Markov model HMM of each pointing stage feature, and select the best path for all the hidden Markov models HMM to determine the finger pointing of the user;

[0134] The response module is used to take the intersection of the user's sight direction and the finger pointing as the user input focus, and control the interactive object to perform a response action according to the user input focus to achieve human-computer interaction.

[0135] Furthermore, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the human-computer interaction method when executing the computer program. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0136] Furthermore, the present invention also provides a storage medium on which a computer program is stored, such as a memory containing instructions, and the instructions can be executed by a processor of a computer device to complete the above method. For example, a non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. When the computer program is executed by the processor, the steps in the human-computer interaction method embodiment can be implemented. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0137] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A human-computer interaction method, It is characterized in that include: Get the action image when the user points to the interactive object; Recognize the face and hand of the user in the action image to obtain the face image and hand image of the user respectively; In a three-dimensional coordinate system established with the control device as the origin, extracting feature values ​​of the facial image, and calibrating the horizontal angle and vertical angle of the facial image according to the feature values ​​of the facial image to determine the sight direction of the user; Extract the pointing stage features of the hand image including the start, hold and end, establish the hidden Markov model HMM of each pointing stage feature, and select the best path for all the hidden Markov models HMM to determine the finger pointing of the user; The intersection of the user's line of sight and the finger's pointing direction is taken as the user input focus, and the interactive object is controlled to perform a response action according to the user input focus to achieve human-computer interaction.

2. A human-computer interaction method as claimed in claim 1, It is characterized in that The recognition of the user's face on the action image specifically includes: In the RGB space, the stimulus value of the captured image color is: in, Represents the relative spectral power distribution of color light perceived by the human eye. r(λ), g(λ) and b(λ) are all CIE1964XYZ spectral stimulus values. According to the constancy of human eyes in perceiving colors, the influence of brightness in characterizing skin color is removed to obtain the component value of skin color. The calculation formula of the component value is: Among them, R, G and B are the stimulus values ​​of the image color, r is the red component, g is the green component, and b is the blue component; Normalize the skin color component values ​​and convert the RGB space into r g Space, skin color in r g It is normally distributed in two dimensions in space; The user's face is distinguished from the background image and grayscale processed to obtain a grayscale image; The facial feature points are located in the grayscale image and the facial parameters are extracted to obtain the user's facial image.

3. A human-computer interaction method as claimed in claim 1, It is characterized in that The step of calibrating the horizontal angle and the vertical angle of the user's facial image according to the characteristic value of the facial image specifically includes: Divide the user's facial image in the vertical direction and the horizontal direction according to the number of nodes to obtain a standard coding set Y=[Y1, Y2, Y3, Y4, Y5] representing the horizontal angle and vertical angle of the facial image; Y1 is the horizontal angle and vertical angle division result of the facial image at the first node, Y2 is the horizontal angle and vertical angle division result of the facial image at the second node, Y3 is the horizontal angle and vertical angle division result of the facial image at the third node, Y4 is the horizontal angle and vertical angle division result of the facial image at the fourth node, and Y5 is the horizontal angle and vertical angle division result of the facial image at the fifth node; Extract eigenvalues ​​from the facial image and normalize the eigenvalues ​​to obtain the input vector set X = (x 1 , x 2 , …, x m ) T ; Input the input vector into the Gaussian RBF neural network GRBF, and get the output vector set y = [y1, y2, y3, y4, y5], the formula is: Where j, m = 1, 2, 3, ..., n, n is the output vector dimension, y j is the output vector of the jth output node of the Gaussian RBF neural network GRBF corresponding to the input vector, b j is the basis function width of the network hidden layer nodes; x p is the Pth input vector, p = 1, 2, 3, ..., P, P is the total number of input vectors; i = 1, 2, 3, ..., h, h is the number of hidden layer units, w ij is the connection weight from the hidden layer to the output layer, c i is the center of the network hidden layer nodes, σ is the variance of the basis function; The confidence β is determined based on the standard encoding and the output vector; the confidence β is used to characterize the matching degree between the output vector and the standard encoding, and its formula is: Among them, y j is the output vector, Y j is the standard encoding, n and j are positive integers; If the confidence β is greater than or equal to the set threshold α, the division result corresponding to the standard coding is the horizontal angle and vertical angle of the user's facial image.

4. A human-computer interaction method as claimed in claim 1, It is characterized in that The optimal path selection of all hidden Markov models HMMs specifically includes: Based on the features of each pointing stage of the hand image, an observation sequence O=O is constructed 1 O 2 O 3 ...O t , (O t =(ω pan,t ,ω tilt,t )), where ω pan,t is the translation angular velocity of the pointing stage feature corresponding to time point t, ω tilt,t is the vertical angular velocity corresponding to the pointing phase characteristic at time point t; Construct the model θ(A, B, π), where π is the initial state probability vector, A is the state transition probability matrix, and B is the observation probability matrix; Based on the observation sequence O and the model θ, a hidden Markov model HMM is constructed, and its expression P(O|θ) is: Among them, O is the observation value sequence, O 1 , O 2 ,…O r is the value in the observation sequence, Q is the best path, q 1 ,q 2 ,…,q r is the value in the best path, b q1 、b q2 , …, b qr is the value in the observation probability matrix, a q1 、a q2 , …, a qr is the value in the state transition probability matrix, r = 1, 2, 3, ..., t; Denote P(O|θ) by the parameters (A, B, π) of model θ, let α(P(O|θ)) / αθ=0, and use the solution as the re-estimation formula of each parameter; The optimal path is obtained by maximizing P(O|θ) according to the re-estimation formula of each parameter, and the pointing angle of the user's finger is determined through the optimal path.

5. A human-computer interaction system, It is characterized in that include: An image acquisition module, used to acquire an action image when a user points to an interactive object; An image recognition module, used to recognize the face and hand of the user in the action image, and obtain the face image and hand image of the user respectively; A facial processing module, used to extract feature values ​​of the facial image in a three-dimensional coordinate system established with the control device as the origin, and calibrate the horizontal angle and vertical angle of the facial image according to the feature values ​​of the facial image to determine the user's line of sight direction; The hand processing module is used to extract the pointing stage features of the hand image including the start, hold and end, establish the hidden Markov model HMM of each pointing stage feature, and select the best path for all the hidden Markov models HMM to determine the finger pointing of the user; The response module is used to take the intersection of the user's sight direction and the finger pointing as the user input focus, and control the interactive object to perform a response action according to the user input focus to achieve human-computer interaction.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, It is characterized in that When the processor executes the computer program, the steps of the human-computer interaction method according to any one of claims 1 to 4 are implemented.

7. A storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the human-computer interaction method according to any one of claims 1 to 4 are implemented.

Citation Information

Patent Citations

  • Method for recognizing human eye state based on built-in type hidden Markov model

    CN101299234A

  • Device and method for selecting and operating objects in a non-touch manner

    CN102270035A

  • Dynamic gesture sequence real-time recognition method, system and device

    CN108256421A

  • Man-machine interaction system and method

    CN108845668A

  • Automatic driving assistance system and method using eye movement tracking and gesture recognition technologies

    CN111931579A