Man-machine interaction method and intelligent AI assistant using same

Abstract

The invention provides a man-machine interaction method and an intelligent AI assistant using the method. The man-machine interaction method comprises the following steps: detecting pixel coordinates of face images of one or more persons in an image in real time; estimating the distance between the face of the one or more persons and the monocular camera; using a trained convolutional neural network model or a self-trained Transform model to extract feature vectors of a face image within a preset distance from the monocular camera; calculating the similarity between the feature vector of the face image within the preset distance and the stored face feature vector; and comparing the calculated similarity degree with a preset threshold value to judge whether the person within the preset distance appears before or not. According to the embodiment of the invention, the monocular camera is utilized to accurately and quickly obtain the face position and the face features, so that the virtual image and a person can interact more vividly.

Description

Technical Field

[0001] The present disclosure relates to image processing technology, and more specifically, to a method for human-computer interaction and an intelligent AI assistant using the method. Background Art

[0002] In order to achieve deep interaction between the virtual image and an unspecified object (for example, a passerby) through eye contact, gestures, language, etc., it is necessary to identify the interactive object and obtain the location information of the interactive object. During the recognition process, the interactive object may leave the screen and return. At this time, the system will identify it as a new user and cannot continue the previous communication content. In a multi-person scenario, due to the influence of each person's posture, angle, distance from the screen, etc., it may not be possible to accurately judge the interactive object, and it is also impossible to accurately perform intelligent interaction.

[0003] In the process of obtaining the position information of the interactive object, binocular or multi-camera simultaneous detection can be used to calculate the three-dimensional information of the target. However, this method is costly. First, the time synchronization of multiple cameras is difficult to control, and the amount of calculation is huge, making it difficult to detect in real time at a frame rate of 30 frames per second; second, different cameras have non-negligible quantization errors in the detection position of the same object to be detected, resulting in a larger error in the three-dimensional position coordinates obtained in the subsequent calculation.

[0004] If a monocular camera is used to determine the position of the interactive object, the current traditional algorithm can only use the projective geometry method to determine the direction of the face's epipolar line in theory, and cannot determine the depth information, that is, the distance information between the interactive object and the virtual person, so it cannot provide realistic interaction, for example, it cannot provide a realistic eye contact experience for the virtual person's eye interaction. In addition, the existing technical route of using a monocular camera to determine depth information still has the following problems. If the edge blur of the object is modeled according to the different blur levels of objects at different distances to infer the depth information of the object, this method has high requirements for image resolution. In the application scenario of face detection, the image blur caused by insufficient resolution will drastically affect the estimation of the depth information of the face from the camera. If the depth information is inferred by the change of the surface shadow of the object, this method is only applicable to estimating the relative depth on the same continuous surface, and cannot be used in the application scenario of estimating the distance between the face and the camera. Other methods, such as inferring depth information from the occlusion relationship of the object and inferring depth information from the haze phenomenon caused by atmospheric scattering, have imposed strong constraints on the category and nature of the objects in the image, and the scene is very limited, and cannot be used to estimate the distance of the face. Summary of the invention

[0005] The present disclosure provides a method for human-computer interaction and an intelligent AI assistant using the method, which helps to realize intelligent human-computer interaction and achieve a more vivid interactive experience.

[0006] In order to solve at least one of the above-mentioned technical problems, according to the first aspect of the present disclosure, a human-computer interaction method is provided, which includes real-time detection of pixel coordinates of facial images of one or more persons in an image, wherein the detected image is from a monocular camera; estimating the distance between the faces of one or more persons and the monocular camera; using a trained convolutional neural network model or a self-trained Transformer model to extract feature vectors of facial images within a preset distance from the monocular camera; calculating the similarity between the feature vectors of the facial images within the preset distance and the stored facial feature vectors; and comparing the calculated similarity with a preset threshold to determine whether the faces within the preset distance have appeared before.

[0007] The embodiments of the present disclosure can use a monocular camera to obtain the distance between the human head and the camera more accurately and quickly, especially the depth information, through the above method. The above method is applicable to situations where one or more people appear in the image. In the case of multiple people, the distance between each person's head and the camera can be obtained, thereby distinguishing the position of each person and more accurately judging the object of interaction. By distinguishing the position of the interactive object, the virtual image can also change its position within the range of the screen, thereby achieving a more natural interaction, while also ensuring the communication of details such as eye contact.

[0008] According to one aspect of the present disclosure, optionally, if it is determined that a person within the preset distance has not appeared based on the comparison between the calculated similarity and the preset threshold, the currently acquired voiceprint feature is compared with the stored voiceprint feature; if a stored voiceprint feature matching it is found in the stored voiceprint features, it is determined that a face within the preset distance has appeared, and the corresponding stored face feature vector is retrieved based on the correspondence between the stored voiceprint feature and the stored face feature vector; and the similarity between the retrieved stored face feature vector and the feature vector of the face image within the preset distance is calculated. If it is determined that they are the same person based on the comparison result of the similarity with the preset threshold, the retrieved stored face feature vector is replaced by averaging the face feature vector within the preset distance and the retrieved stored face feature vector; if it is determined that they are not the same person based on the comparison result of the similarity with the preset threshold, the retrieved stored face feature vector is replaced by the face feature vector within the preset distance.

[0009] In order to increase the accuracy of recognition, the voiceprint features can be used to confirm whether the current interactive object has appeared before. For example, after judging that the current interactive object has not appeared before through the facial feature vector, if the voiceprint features of the current person are collected, further confirmation can be made through comparison of the voiceprint features, thereby increasing the accuracy of the judgment. If the voiceprint features do not match the stored voiceprint features, then it is judged that the current interactive object has not appeared, and a new interaction is started. If the current interactive object is judged to have appeared through the voiceprint features, the feature vector extracted from the current face can be compared with the stored facial feature vector corresponding to the voiceprint features to determine whether the facial feature vector stored in the database needs to be replaced.

[0010] According to one aspect of the present disclosure, optionally, the estimating the distance between one or more faces and the monocular camera includes estimating the distance between the faces and the monocular camera based on the pixel coordinates of the one or more faces, the occupied pixel size and the prior size of the human head, and the human-computer interaction method also includes determining the spatial three-dimensional coordinates of the faces based on the intrinsic parameter matrix of the monocular camera and the distance between one or more faces and the monocular camera, which is represented by the lateral and longitudinal offset distances of the center point of the face relative to the main optical axis of the monocular camera and the distance between the center point of the face and the center of the monocular camera.

[0011] According to one aspect of the present disclosure, optionally, detecting the pixel coordinates of the human head in the image includes detecting the human head using a target detection algorithm to obtain a rectangular frame of the head.

[0012] According to one aspect of the present disclosure, optionally, estimating the distance between the faces of one or more persons and the monocular camera includes using a Depth Vision Transformer model to obtain a depth map corresponding to the face image to estimate the distance between the face and the monocular camera, and the human-computer interaction method also includes determining the spatial three-dimensional coordinates of the face based on the intrinsic parameter matrix of the monocular camera, the distance between the faces of one or more persons and the monocular camera, and the pixel coordinates of the face.

[0013] According to one aspect of the present disclosure, optionally, using a self-trained Transformer model to extract a feature vector of a face image within a preset distance from the monocular camera includes: extracting a preliminary feature F0 from the input face image data Y via a convolutional layer: Among them, H and W represent the height and width of the input face image, C represents the number of channels, represents the image vector space, H' and W' represent the height and width of the output feature map after the convolution layer, C' represents the number of channels of the feature map, Represents the output feature vector space; split the image block and flatten it to get Where N is the number of image blocks, P is the size of the image block, F0′ is the feature vector after the image block is flattened, It is the feature vector space after the image is flattened into blocks; the self-trained Transformer encoder is used to extract the features of F0′ to obtain the feature vector of each image block and the global feature vector.

[0014] According to one aspect of the present disclosure, optionally, when a self-trained Transformer model is used to extract a feature vector of a face within a preset distance from the monocular camera, the global feature is saved as a face feature vector in a face feature database. Optionally, the method for human-computer interaction may also include extracting a current face feature vector using a trained convolutional neural network model; calculating the degree of similarity between the current face feature vector and the stored face feature vector; and comparing the calculated degree of similarity with a preset threshold to determine whether the current face has appeared before.

[0015] The embodiments of the present disclosure can further enhance the interaction effect and user experience by recognizing facial features of the face. Through the above facial feature recognition, it can be determined whether the interactive object has appeared before. If it appears again, the virtual image can continue the previous communication with the interactive object, so as to achieve the continuity of the interactive content.

[0016] Optionally, the method for human-computer interaction may further include recognizing the lip contour of a person's face, and determining whether the person is speaking based on the lip contour.

[0017] When there are multiple people within the camera's lens range, the object that the virtual image is interacting with can be determined by recognizing lip shapes.

[0018] Optionally, when voice information of the interactive object is detected, the voiceprint features of the voice information are stored, and the corresponding relationship between the voiceprint features and the facial feature vector of the interactive object is stored.

[0019] Optionally, the dimension of the facial feature vector is at least 128.

[0020] Optionally, the method for human-computer interaction may also include using a general data set to train the model parameters of the convolutional neural network to obtain a pre-trained model, and using a self-made data set to continue training the model parameters of the trained convolutional neural network to obtain a convolutional neural network model, wherein the images of the self-made data set are collected from actual application scenarios.

[0021] Using a self-made data set to continue training the model parameters of the trained convolutional neural network can make the trained data more compatible with the application scenario and improve the accuracy of face recognition. Using a high-definition camera to obtain data and training the Transformer model with a self-made data set can obtain a facial feature vector of at least 256 dimensions.

[0022] According to the second aspect of the present disclosure, an embodiment of the present disclosure provides a device for human-computer interaction, which includes a monocular camera for collecting human head and / or facial information, a microphone for collecting voice; a processor; and a memory on which a computer program is stored, wherein when the program is executed by the processor, the head positioning method described in the embodiment of the first aspect is implemented.

[0023] According to a third aspect of the present disclosure, an embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the human-computer interaction method as described in the embodiment of the first aspect is implemented.

[0024] According to the fourth aspect of the present disclosure, an embodiment of the present disclosure provides an intelligent AI assistant, which includes a screen for displaying a virtual image; a monocular camera for collecting human head and / or facial information; a microphone for collecting the voices of passers-by; a processor; and a memory storing one or more computer programs, wherein the one or more computer programs include instructions, and when the instructions are executed by the one or more processors, the human-computer interaction method as described in the embodiment of the first aspect is executed.

[0025] The embodiments of the present disclosure can realize an intelligent AI assistant that interacts with passers-by. It can determine the position of passers-by through a monocular camera, so as to have personalized communication with passers-by by eye contact. At the same time, in a multi-person scenario, the object of interaction can be accurately determined by judging the position and / or voiceprint of the passers-by. Adding face recognition technology and / or voiceprint recognition technology can avoid identifying passers-by who have appeared as new users, and continue the previous communication with the object of interaction, realizing a more accurate intelligent dialogue experience.

[0026] It is not necessary to achieve all the advantages described above at the same time for any device implementing the present disclosure. Other features and advantages of the present disclosure will be described in the subsequent description embodiments, and partly become apparent from the description embodiments, or be understood by implementing the present disclosure. The purposes and advantages of the embodiments of the present disclosure can be realized and obtained by the structures indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0028] Figure 1 is a schematic flow chart of face positioning according to an embodiment of the present disclosure;

[0029] Figure 2 is a schematic flow chart of face positioning according to another embodiment of the present disclosure;

[0030] Figure 3 is a schematic flow chart of interactive object recognition according to an embodiment of the present disclosure;

[0031] Figure 4 is another schematic flow chart of interactive object recognition according to an embodiment of the present disclosure;

[0032] Figure 5 is a structural schematic diagram of a human-computer interaction device according to an embodiment of the present disclosure;

[0033] Figure 6 It is a structural diagram of an intelligent AI assistant using a human-computer interaction device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. The various different embodiments can be combined with each other to form other embodiments not shown in the following description. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0035] Unless otherwise defined, the technical terms or scientific terms used herein shall have the usual meanings understood by persons of ordinary skill in the field to which the present disclosure belongs. The words "first", "second" and similar words used in the patent application specification and claims of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not necessarily indicate a quantity restriction. Words such as "include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0036] Figure 1 A schematic flowchart of face positioning according to an embodiment of the present disclosure is shown. The method 100 includes steps S110-S140.

[0037] In step S110, the camera is calibrated with internal parameters using Zhang's calibration method to obtain a matrix P. The matrix P is used to establish a mapping from the three-dimensional coordinates of the space point to the two-dimensional coordinates of the image pixel point. The mapping method is expressed by the following formula:

[0038] x=PX

[0039] Where X represents the three-dimensional coordinates of the spatial point, X = [xyz]; P is the matrix of the camera, x is the two-dimensional homogeneous coordinates of the image pixel, x = [c x c y 1]. The matrix P has the following form:

[0040]

[0041] Where f represents the focal length of the camera, p x and p y They represent the principal point offset of the camera, that is, the horizontal and vertical coordinates of the pixel point projected from the world point on the central axis of the camera to the two-dimensional image.

[0042] The camera disclosed herein is a high-precision camera, and preferably the precision can exceed 4 million pixels.

[0043] In step S120, for an image containing a head, a deep learning method is used to detect the position of the head in the two-dimensional image. For example, a deep learning framework for object detection (e.g., yolo-v5) can be used to perform deep learning training on a large-scale human head data set. The obtained pre-trained model can detect human head targets in a two-dimensional image at a frame rate of more than 30 frames per second, and the detection has high real-time performance, high accuracy, high recall rate, and temporal stability. In addition, other algorithms such as SSD (Single Shot Detection) can also be used.

[0044] The detection result can be returned in the form of a rectangular frame. According to the rectangular frame, the pixel coordinates c of the center of the face on the image can be obtained. x c y and the size of the head in pixels in the image h box The center of the face can be represented by the face center point, and the pixel size of the head can be represented by the size of the rectangular frame.

[0045] In step S130, according to the imaging principle, the distance z between the face (with the center of the face as the head position, set as the starting point) and the center of the camera can be estimated and calculated using the following formula:

[0046]

[0047] Where v is the image distance, h head is the height of the head from top to bottom. To simplify the calculation, since the face is generally far away from the camera, the image is approximately at one focal length, so the image distance v can be approximately equal to the focal length f. The estimation formula is:

[0048]

[0049] h head It can be determined based on prior knowledge of the normal size of a human head, for example, h head It can be set to a fixed value, or it can be adjusted based on other information of the interaction object, such as gender, age, etc.

[0050] Here, z means the z coordinate of the three-dimensional coordinates of the spatial point represented by the face.

[0051] In step S140, the three-dimensional coordinates of the face point are determined based on the camera parameters and the estimated distance of the face from the camera. Starting from the camera matrix P, the x-coordinate and y-coordinate are derived from the z-coordinate, and the derivation formula is as follows:

[0052]

[0053] Wherein, x is the actual distance that the face (for example, the head position is represented by the center point of the face) is offset to the right compared with the main optical axis of the camera, and y is the actual distance that the face is offset upward compared with the main optical axis of the camera.

[0054] This detection method can detect and track the heads of multiple people in the picture at the same time, and use the pixel size information of each face to predict the three-dimensional position of each face at the same time. After integrating all the position information, the passerby closest to the three-dimensional virtual person can be selected as the interaction object. By using the above method to locate the head position, only one camera is needed to quickly determine the three-dimensional coordinates of the face in real time, reducing costs, simplifying calculations while ensuring the accuracy of positioning, and realizing real-time interaction between virtual images and humans, obtaining a more natural and realistic interaction effect.

[0055] In order to achieve a more intelligent interaction effect, Figure 1 Based on the embodiment shown, further implementations can be performed Figure 3 The method 200 shown. In order to identify whether the object of the current interaction has appeared before and whether to continue the previous communication content, the facial features of the person can be further recognized.

[0056] Figure 2 A schematic flow chart of a face location method according to another embodiment of the present disclosure is shown. In step S110A, the camera is calibrated with internal parameters, which can be used to correct the image and eliminate image distortion. The internal parameter calibration can be performed in an existing manner, not limited to the Zhang calibration method.

[0057] Step S120A may use a method similar to step S120 to detect the face in the image and determine the position of the face. Step S120A obtains the size of the rectangular frame of the face image (eg, length and width information) and the pixel coordinate information of the face.

[0058] However, Figure 1 Unlike the embodiment of the present invention, the face coordinates and size calculated in step S120A are not directly involved in the calculation of the distance between the face and the camera, but are used to determine the position of the face image in the entire image.

[0059] In step 130A, the Depth Vision Transformer (DPT) model is used to estimate the distance of the face from the camera. First, in the encoder, the input face image is divided into fixed-size image blocks, each image block is regarded as a token, and each block is linearly mapped to a high-dimensional feature vector. These tokens are processed through a multi-layer self-attention mechanism to generate feature representations with global context information, and then these tokens are recombined into image representations of different resolutions. In the decoder, the feature representations of different resolutions are gradually fused using a residual convolution module and upsampled to the final output resolution to convert the feature map into a depth map.

[0060] By training on a high-resolution face dataset, the process can better capture the global and local information of the face in the high-definition image, and obtain a more accurate depth estimate. The distance between the face and the camera can be set to the average of the depth values ​​of the depth map or to the depth value of the pixel in the middle of the pattern.

[0061] In step 140A, the spatial three-dimensional coordinates of the face are obtained based on the distance between the face and the camera obtained in step 130A, the parameters obtained by the camera internal parameter calibration, and the pixel coordinates of the face.

[0062] By adopting the above method, the distance between the face or head of a person and the camera can be determined when using a monocular camera, and the object of interaction can be estimated by the distance, thereby realizing the interaction between the virtual image and the human being.

[0063] exist Figure 3 In step S210, a self-trained convolutional neural network model or a self-trained Transformer model (hereinafter referred to as Face TF) is used to extract the current facial feature vector. For example, a deep convolutional neural network (CNN) model can be used to extract at least 128-dimensional facial feature vectors of all people, map the head image to at least 128-dimensional space (such as Euclidean space), and save it to a facial feature database. In order to improve the accuracy of face recognition, the convolutional neural network model can use a self-made face data set to train the model through transfer learning based on the pre-trained model during training. The data of the self-made face data set can be collected from the scenarios in which the interactive device is used. After training the model using the self-made data set, a larger distance can be obtained for two different faces, which is conducive to better distinguishing faces.

[0064] In another example, high-resolution data can be collected through a high-definition monocular camera. By extracting facial feature vectors using FaceTF, at least 256-dimensional facial feature vectors can be obtained, which can improve the accuracy of face recognition. Due to the lack of suitable open source datasets for high-resolution images, self-made datasets can be used to train the Face TF model for specific interactive device scenarios.

[0065] The following describes the steps of obtaining the facial feature vector in a face image through the Face TF model.

[0066] First, the input face image data Y is passed through the convolutional layer to extract the preliminary feature F0, that is,

[0067]

[0068] Among them, H and W represent the height and width of the input face image, C represents the number of channels, represents the image vector space, H' and W' represent the height and width of the output feature map after the convolution layer, C' represents the number of channels of the feature map, Represents the output feature vector space.

[0069] The image blocks are split and flattened to obtain Where N is the number of image blocks, P is the size of the image block, F0′ is the feature vector after the image block is flattened, is the feature vector space after the image is flattened. Then, each block will be mapped to D dimensions after flattening. Each block is added with a position code, which represents the position information. Feature extraction is performed through the Transformer encoder to obtain D is the predetermined dimension, is the D-dimensional feature vector space converted from the flattened vector space, F f It is a feature vector in the D-dimensional feature vector space, which represents the features of each image block.

[0070] The Transformer encoder is used to obtain the features of each image block and the global features. The initial value of the global features is h. After passing through the encoder, the output of the global features is obtained as f = Transformer (h). The global features are saved as face feature vectors in the face feature database.

[0071] exist Figure 1 and Figure 2 In the method described, whether to store the corresponding facial feature vector can be determined based on the distance between the camera and the face. For passers-by at a long distance, their facial features may not be extracted and stored.

[0072] In step S220, the similarity between the feature vector of the face in the current image and the feature vector of the stored face image is calculated. Optionally, the spatial distance between the current face feature vector x and the stored face feature vector y can be calculated as a measure of the similarity. The distance d(x, y) between two points x and y is calculated as follows, where n is the spatial dimension:

[0073]

[0074] Spatial distance is directly related to image similarity. The spatial distance in different images of the same person is small, and the images of different people have a large distance in space. By setting a threshold for the spatial distance, we can determine whether two people in different images are the same person. In addition to the Euclidean distance, other algorithms can also be used to calculate the spatial distance between facial feature vectors. For example, data normalization can be used, with 1 as the threshold. If the value is less than 1, it is considered to be the same person. The smaller the value, the greater the probability of the same person; if it is greater than or equal to 1, it is considered to be different people. The larger the value, the greater the probability of different people.

[0075] In addition, the measurement result of the similarity between the current facial feature vector x and the stored facial feature vector y may also be obtained by cosine similarity calculation or other methods.

[0076] Optionally, the similarity between the current face feature vector x and the stored face feature vector y can be calculated in the following way: Where x·y represents the dot product, and ‖x‖ and ‖y‖ represent the norms of vectors x and y.

[0077] In step S230, it is determined whether the similarity between the feature vector of the face in the current image and the feature vector of the stored face image is less than a threshold.

[0078] If the extracted face is considered to be the same person as the one previously stored (S250), the previous interaction or conversation can be continued. If the extracted face is considered to be different from the one previously stored (S260), the current person can be treated as a new user and his / her facial feature vector (step 262) and voiceprint features of the voice information (if any) can be stored, or further confirmation can be performed (see Figure 4 ).

[0079] Figure 4 Shown in Figure 3Based on the method shown, a method 300 is provided for further determining whether the current person has appeared by obtaining voiceprint features. This can avoid the problem of misjudgment due to situations such as the face of the person not facing the camera. In step S310, the voice of the current speaker is obtained through a microphone or other device, and the voiceprint features are extracted from the voice, and compared with the stored voiceprint features to determine whether there are matching voiceprint features in the stored voiceprint features (S320). The voiceprint recognition algorithm can adopt, for example, the ECAPA-TDNN model. After training, the classification accuracy of the voiceprint model can reach above 0.9608, and the accuracy of pairwise comparison can reach above 0.99980.

[0080] If there is no matching voiceprint, the feature vector of the face is stored in the face database, the voiceprint feature is stored in the voiceprint database and the corresponding relationship between them is stored (S330). If there is a matching voiceprint feature, it is determined that there has been previous interaction with the current person (S340). The face feature vector corresponding to the matching voiceprint feature is retrieved from the stored database and the face feature vector is extracted from the image of the current person (S350). In step S360, the degree of similarity between the retrieved face feature vector and the extracted face feature vector of the current person is compared with a threshold value. If it is lower than a predetermined threshold value, the stored face feature vector is replaced with the mean of the extracted face feature vector and the stored face feature vector (S370), otherwise the previously stored face feature vector can be replaced with the extracted face feature vector of the current person (S380).

[0081] If it is determined at S320 that there is no matching voiceprint, the process may be repeated by reacquiring the image and / or sound. Figure 3 and / or Figure 4 step to further confirm the accuracy of the judgment (because the acquired images or sounds may be more disturbed due to the movement of people).

[0082] Figure 4 An example is in Figure 3 Based on the facial recognition steps shown in the figure, the voiceprint feature comparison is used for further confirmation. Figure 3 If there is a problem with the facial image obtained in the steps shown, resulting in an error in judgment (for example, the image is not a frontal image, the expression changes greatly, the image is not clear enough, etc.), you can also re-confirm it through the voiceprint feature to avoid identifying the same person as a new user and being unable to continue the previous interaction. Figure 4 In an embodiment, a frontal image of a person may be obtained for comparison, and if the voiceprint features match but the feature vector of the image does not match, the facial feature database may be updated with the facial feature vector of the frontal image for subsequent facial recognition.

[0083] In order to better determine the object of interaction in a multi-person scenario, lip shape feature recognition can also be added. For example, at the beginning of communication, if the microphone detects input, the current speaker is determined by the lip shape features of the face. For example, the Dlib algorithm can be used to detect facial key points. This method can be used with Figure 1 After locating the human head and generating three-dimensional coordinates, the lip shape feature is further detected to determine the object being interacted with among multiple people. This step can also be combined with Figure 3 In the embodiment shown, the recognition is performed after the face recognition of the person. The recognition of lip shape features can be performed in other suitable or necessary situations.

[0084] Figure 5 A schematic diagram of the structure of a human-computer interaction device according to an embodiment of the present disclosure is shown. The human-computer interaction device 400 may include a processor 410, a monocular camera 420, a display 430, a speaker 440, a microphone 450, a memory 460, a power module 470 and a communication module 480. It is understood that the schematic diagram of the structure of the embodiment of the present disclosure does not constitute a specific limitation on the above-mentioned device. In other embodiments of the present disclosure, the device may include more or fewer components, or split / combine certain components, or arrange the components differently. The illustrated components may be implemented with hardware, software, or a combination of hardware and software. For example, some of the functions of the processor 410 and the memory 460 described below may also be implemented by the processor and / or memory in the remote device, that is, a part of the processor 410 and the memory 460 may be included in the remote device.

[0085] The processor 410 may include one or more processing units. For example, the processor may include one or more of a central processing unit, a graphics processor, a digital signal processor, an image signal processor, a neural network processor, etc., and may be an independent device or an integrated processor, which can be used to read and execute instructions and generate control signals. The processor 410 may also include a memory for storing instructions and data. The processor can process images at a frame rate of 30, so as to perform real-time reasoning and prediction using the deployed depth model, and determine the position of the head coordinates through subsequent matrix operations. The processor 410 can implement one or more of the methods described in the above embodiments by executing instructions.

[0086] The monocular camera 420 has the function of capturing external images, and its frame rate is generally not less than 30 frames per second to ensure the real-time and continuity of the detection. Optionally, the monocular camera 420 has a field of view of at least 120 degrees, which can ensure that the camera has a sufficiently wide viewing range.

[0087] The display 430 may include one or more display panels. In order to display a life-size virtual image, the display 430 may be set large enough. The virtual image may move around and change position within the range of the screen, thereby interacting with passers-by more naturally.

[0088] The speaker 440 can transmit the sound of the virtual image, and the microphone 450 is used to collect the voice signal of the interactive object. The microphone 450 can be a separate component, or can be combined with the camera 420 or other devices.

[0089] The memory 460 can store computer executable program codes, including instructions. The memory can also store one or more of image data, video data, audio data, a facial feature vector data set extracted from an image, a voiceprint data set, a correspondence between a facial feature vector and voiceprint data, and lip shape data. The memory can include a high-speed random access memory, a non-volatile memory, and the like.

[0090] The power module 470 can supply power to various parts of the device and can also provide charge and discharge management (if a battery is used).

[0091] The communication module 480 may include wired and / or wireless communication solutions. The communication module 480 may send the collected data to a cloud server or a remote server for processing, thereby reducing the computing power and data storage requirements of the field equipment.

[0092] When using Transformer to extract image features in the above method, the amount of calculation will be very large, and an image processor is generally used as the processor 410. In order to optimize and improve the calculation speed, one or more of the following methods can be used to improve the real-time performance and accuracy of face detection and recognition.

[0093] First, mixed precision training can be used to use FP16 (16-bit floating point) and FP32 (32-bit floating point) mixed calculations to improve training speed and reduce video memory usage while maintaining model accuracy. Through mixed precision training, the model disclosed in this disclosure can significantly improve calculation speed while maintaining accuracy, especially in real-time human-computer interaction scenarios, such as face detection and depth estimation, which helps reduce latency and improve response speed.

[0094] Optionally, dynamic quantization can be used to convert some calculations into a low-precision format (such as INT8), which mainly includes using the quantization tools provided by the deep learning framework to convert some layers of the model into a low-precision format, calibrating the quantized model using a calibration data set to ensure numerical accuracy, and verifying the performance of the quantized model to ensure that the accuracy loss is within an acceptable range. The use of dynamic quantization can significantly accelerate the model of the present disclosure in the inference stage, especially in edge devices or resource-constrained environments, helping to reduce hardware requirements and improve the deployment flexibility and cost-effectiveness of the system.

[0095] Alternatively, using CUDA / cuDNN acceleration can make the present disclosure more efficient in utilizing hardware resources, especially when processing high-resolution images and real-time feature extraction, significantly improving the real-time processing capability and throughput of the system.

[0096] Figure 6 An intelligent AI assistant 50 is shown, which is based on Figure 5 The human-computer interaction device is implemented. The monocular camera 51 of the intelligent AI assistant collects image information of the interactive object. The three-dimensional coordinates of the head of the interactive object 54 are determined by executing the stored instructions. The processor of the intelligent AI assistant 50 controls the virtual image 53 displayed by the display 52 to interact or communicate with the interactive object. Figure 6 The specific working modes and structures of the monocular camera 51 and the display 52 are respectively Figure 5 The camera 420 and the display 430 are the same and will not be described in detail. The intelligent AI assistant 50 can independently implement one or more of the methods described in the above embodiments, or it can implement some instructions through the processor of the remote server. The intelligent AI assistant can detect and track the human head through the above embodiments, guide the virtual image to interact with people through eye contact, gestures and / or language, and enhance the user experience. For example, the intelligent AI assistant 50 can be used as a smart billboard to increase the experience and dissemination effect of advertising, and can also be used in public places such as banks to assist or replace manual services.

[0097] The above description is merely an exemplary embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure. The protection scope of the present disclosure is determined by the appended claims.

Claims

1. A human-computer interaction method, characterized in that include: Detecting pixel coordinates of facial images of one or more persons in an image in real time, wherein the detected image is from a monocular camera; estimating the distance between the face of the one or more persons and the monocular camera; Extracting feature vectors of facial images within a preset distance from the monocular camera using a trained convolutional neural network model or a self-trained Transformer model; Calculating the similarity between the feature vector of the face image within a preset distance and the stored face feature vector; and The calculated similarity is compared with a preset threshold to determine whether the person within the preset distance has appeared before.

2. The human-computer interaction method according to claim 1, characterized in that Also includes: If it is determined that the person within the preset distance has not appeared according to the comparison between the calculated similarity and the preset threshold, the currently acquired voiceprint feature is compared with the stored voiceprint feature; If a stored voiceprint feature matching the stored voiceprint feature is found in the stored voiceprint features, it is determined that a person within the preset distance has appeared, and the corresponding stored facial feature vector is retrieved according to the correspondence between the stored voiceprint feature and the stored facial feature vector; and Calculate the degree of similarity between the retrieved stored facial feature vector and the feature vector of the facial image within the preset distance; if it is determined that they are the same person based on the comparison result of the degree of similarity and the preset threshold, replace the retrieved stored facial feature vector with the average of the facial feature vector within the preset distance and the retrieved stored facial feature vector; if it is determined that they are not the same person based on the comparison result of the degree of similarity and the preset threshold, replace the retrieved stored facial feature vector with the facial feature vector within the preset distance.

3. The human-computer interaction method according to claim 1, characterized in that The estimating the distance between the faces of one or more persons and the monocular camera includes estimating the distance between the faces and the monocular camera based on the pixel coordinates of the faces, the pixel size occupied and the prior size of the human head. The human-computer interaction method also includes determining the spatial three-dimensional coordinates of the faces based on the intrinsic parameter matrix of the monocular camera and the distance between the faces of one or more persons and the monocular camera, which is represented by the lateral and longitudinal offset distances of the center point of the face relative to the main optical axis of the monocular camera and the distance between the center point of the face and the center of the monocular camera.

4. The human-computer interaction method according to claim 1, characterized in that The estimating the distance between the faces of one or more persons and the monocular camera includes using a Depth Vision Transformer model to obtain a depth map corresponding to a face image to estimate the distance between the face and the monocular camera. The human-computer interaction method also includes determining the spatial three-dimensional coordinates of the face based on an intrinsic parameter matrix of the monocular camera, the distance between the faces of one or more persons and the monocular camera, and the pixel coordinates of the face.

5. The human-computer interaction method according to claim 1, characterized in that Using a self-trained Transformer model to extract a feature vector of a face image within a preset distance from the monocular camera includes: The input face image data Y is passed through the convolution layer to extract the preliminary feature F0: Among them, H and W represent the height and width of the input face image, C represents the number of channels, represents the image vector space, H' and W' represent the height and width of the output feature map after the convolution layer, C' represents the number of channels of the feature map, Represents the output feature vector space; The image blocks are split and flattened to obtain Where N is the number of image blocks, P is the size of the image block, F0′ is the feature vector after the image block is flattened, It is the feature vector space after the image is flattened; A self-trained Transformer encoder is used to extract the features of F0′ to obtain the feature vector of each image block and the global feature vector.

6. The human-computer interaction method according to claim 1, characterized in that When a self-trained Transformer model is used to extract a feature vector of a face within a preset distance from the monocular camera, the global feature vector is saved as a face feature vector in a face feature database.

7. The human-computer interaction method according to claim 2, characterized in that When the voice information of the interactive object is detected, the voiceprint features of the voice information are stored, and the corresponding relationship between the voiceprint features and the facial feature vector of the interactive object is stored.

8. The human-computer interaction method according to claim 1, characterized in that The method includes using a general data set to train the model parameters of a convolutional neural network to obtain a pre-trained model, and using a self-made data set to continue training the model parameters of the trained convolutional neural network to obtain a convolutional neural network model, wherein the images of the self-made data set are collected from actual application scenarios.

9. An intelligent AI assistant, characterized in that It includes a screen for displaying a virtual image; a camera for collecting information about a person's head and / or face; and a microphone for collecting a person's voice. processor; and a memory storing one or more computer programs, wherein the one or more computer programs include instructions, and when the instructions are executed by the one or more processors, the human-computer interaction method according to any one of claims 1 to 8 is executed.

10. The intelligent AI assistant according to claim 9, characterized in that The processor is an image processor, and one or more of mixed precision training, dynamic quantization or CUDA / cuDNN acceleration is used to optimize the calculation of the image processor.

Images