Intelligent health-care accompanying robot with face tracking function

Abstract

The invention relates to the field of intelligent tracking, and particularly discloses an intelligent health-care accompanying robot with a face tracking function, which performs image preprocessing and face feature analysis on a first user face image and a second user face image by using an image analysis and processing technology based on computer vision. In this way, screen pose data is intelligently adjusted according to a fine-grained global semantic compensation representation between the first user facial feature and the second user facial feature. The robot can intelligently adjust the screen posture by collecting the user face image in real time and analyzing the posture of the user face image, it is ensured that the robot is located at the optimal watching position of the user all the time, and the operation difficulty and visual fatigue are greatly reduced. In addition, the face orientation of the user can be tracked and adapted in real time through an intelligent automatic control mode, and more natural and smoother interaction experience is provided.

Description

Technical Field

[0001] The present application relates to the field of intelligent tracking, and more specifically, to an intelligent health care companion robot with facial tracking function. Background Art

[0002] With the advent of an aging society, smart health care companion robots have gradually become an indispensable part of the lives of the elderly. These robots not only provide daily entertainment and social interaction, but also can monitor health conditions, remind people to take medicine, make emergency calls and other important functions. In order to better serve the elderly user group, robots need to have a high degree of humanized interaction capabilities.

[0003] However, traditional intelligent health care companion robots usually have a fixed screen position and cannot automatically adjust the position and angle of the displayed content according to the user's actual posture. For elderly people with limited mobility or decreased vision, this design may cause great inconvenience in use, increase the difficulty of operation and visual fatigue. In addition, traditional robots lack facial tracking functions and cannot respond to changes in the user's facial orientation in real time, resulting in a poor interactive experience.

[0004] Therefore, an optimized intelligent health care companion robot with facial tracking function is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] According to one aspect of the present application, there is provided an intelligent health care companion robot with a facial tracking function, comprising: a facial image acquisition module, used to acquire a first user facial image and a second user facial image captured by a camera of the intelligent health care companion robot; a facial image processing module, used to preprocess the first user facial image and the second user facial image and extract facial features to obtain the first user facial features and the second user facial features respectively; a facial feature difference encoding module, used to perform feature global semantic encoding on the first user facial features and the second user facial features to obtain user facial part and posture semantic difference encoding features, wherein the facial feature difference encoding module comprises: a user facial feature local fusion unit, used to perform fine-grained local semantic fusion on the first user facial features and the second user facial features to obtain a set of first and second user facial fine-grained local shared semantic features; a user facial feature semantic compensation interaction unit, used to perform semantic compensation interactive aggregation on the set of first and second user facial fine-grained local shared semantic features to obtain the user facial part and posture semantic difference encoding features; a facial tracking instruction adjustment module, used to obtain a facial tracking instruction for representing screen posture adjustment data according to the user facial part and posture semantic difference encoding features.

[0007] Compared with the prior art, the present application provides an intelligent health care companion robot with a facial tracking function, which collects the facial image of the first user and the facial image of the second user through the camera of the intelligent health care companion robot, and uses computer vision-based image analysis and processing technology to perform image preprocessing and facial feature analysis on the facial image of the first user and the facial image of the second user, so as to intelligently adjust the screen posture data according to the fine-grained global semantic compensation representation between the facial features of the first user and the facial features of the second user. By collecting the user's facial image in real time and analyzing its posture, the robot can intelligently adjust the screen posture to ensure that it is always in the best viewing position for the user, greatly reducing the difficulty of operation and visual fatigue. And through intelligent automatic control methods, it can track and adapt to the user's facial orientation in real time, providing a more natural and smooth interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0009] Figure 1 This is a block diagram of an intelligent health care companion robot with facial tracking function according to an embodiment of the present application.

[0010] Figure 2 This is a data flow diagram of an intelligent health care companion robot with facial tracking function according to an embodiment of the present application.

[0011] Figure 3 This is a block diagram of a facial image processing module in an intelligent healthcare companion robot with facial tracking function according to an embodiment of the present application.

[0012] Figure 4 This is a block diagram of a facial feature difference encoding module in an intelligent healthcare companion robot with facial tracking function according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0014] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0015] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0016] It is worth noting that in this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0017] With the advent of an aging society, smart health care companion robots have gradually become an indispensable part of the lives of the elderly. These robots not only provide daily entertainment and social interaction, but also can monitor health conditions, remind people to take medicine, make emergency calls and other important functions. In order to better serve the elderly user group, robots need to have a high degree of humanized interaction capabilities.

[0018] However, traditional intelligent health care companion robots usually have a fixed screen position and cannot automatically adjust the position and angle of the displayed content according to the user's actual posture. For elderly people with limited mobility or decreased vision, this design may cause great inconvenience in use, increase the difficulty of operation and visual fatigue. In addition, traditional robots lack facial tracking functions and cannot respond to changes in the user's facial orientation in real time, resulting in a poor interactive experience.

[0019] Therefore, in response to the above technical problems, the present application proposes an intelligent health care companion robot with a facial tracking function, which collects the facial images of the first user and the second user through the camera of the intelligent health care companion robot, and uses computer vision-based image analysis and processing technology to perform image preprocessing and facial feature analysis on the first user facial image and the second user facial image, so as to intelligently adjust the screen posture data according to the fine-grained global semantic compensation representation between the facial features of the first user and the second user. By collecting the user's facial image in real time and analyzing its posture, the robot can intelligently adjust the screen posture to ensure that it is always in the best viewing position for the user, greatly reducing the difficulty of operation and visual fatigue. And through intelligent automatic control methods, it can track and adapt to the user's facial orientation in real time, providing a more natural and smooth interactive experience.

[0020] Figure 1 This is a block diagram of an intelligent health care companion robot with facial tracking function according to an embodiment of the present application. Figure 2 : is a data flow diagram of an intelligent health care companion robot with a facial tracking function according to an embodiment of the present application. Specifically, Figure 1 and Figure 2 As shown, the intelligent health care companion robot 100 with facial tracking function according to the embodiment of the present application includes: a facial image acquisition module 110, used to acquire a first user facial image and a second user facial image captured by a camera of the intelligent health care companion robot; a facial image processing module 120, used to preprocess and extract facial features of the first user facial image and the second user facial image respectively to obtain first user facial features and second user facial features; a facial feature difference encoding module 130, used to perform feature global semantic encoding of the first user facial features and the second user facial features to obtain user facial posture semantic difference encoding features; a facial tracking instruction adjustment module 140, used to obtain a facial tracking instruction for representing screen posture adjustment data according to the user facial posture semantic difference encoding features.

[0021] In the embodiment of the present application, the facial image acquisition module 110 is used to acquire the facial image of the first user and the facial image of the second user captured by the camera of the intelligent health care companion robot. In particular, there is a predetermined time interval between the first user facial image and the second user facial image. It should be understood that the first user facial image refers to the initial facial image of the user captured by the robot at a certain point in time, which serves as a reference benchmark and records the facial features, expressions, and position of the user at a specific moment. The second user facial image refers to the facial image of the same user captured by the robot again after a predetermined time interval with the first image acquisition. This time interval can be flexibly set according to the specific application scenario, such as a few seconds or minutes. Therefore, in order to enable the robot to track the user's position change, in the technical solution of the present application, the first user facial image and the second user facial image captured by the camera of the intelligent health care companion robot are obtained, so that it can ensure that its screen is always facing the user, which is very important for scenes that require long-term interaction or companionship, such as elderly or patient care.

[0022] In order to ensure that the user's facial information can be accurately captured, the robot is equipped with a high-resolution camera, which is placed in a position where the angle can be flexibly adjusted to cover the largest possible spatial range. When the robot is in working state, it will start the camera and aim it at the expected user area. The camera parameters (such as focal length, aperture, etc.) have been pre-optimized to ensure clear images under various lighting conditions. In addition, considering the possible impact of different environments, such as changes in indoor light or background complexity, the system also has built-in automatic exposure and white balance adjustment functions, which help maintain stable image quality.

[0023] Once the camera is ready, the next step is to decide when and how to trigger the image acquisition action. In this link, the choice of time interval is crucial. If the interval is too short, the difference between two consecutive pictures may be very small, thus wasting computing resources; on the contrary, if the interval is too long, important changes in the user's position may be missed, affecting the tracking effect. Therefore, a reasonable acquisition cycle is usually set according to the needs of the actual application scenario. For the elderly group with slower movements, a time interval of several seconds may be appropriate, while if it is used to monitor children or other fast-moving objects, it needs to be shortened to milliseconds. It is worth noting that the "first user facial image" here refers to the initial facial image captured by the robot at a specific moment. It serves as a reference to record the user's facial features, expressions, and position at a fixed point. The "second user facial image" is the facial image of the same user acquired again after a certain time difference from the first acquisition, which is used to reflect any displacement or posture changes of the user during this period.

[0024] After the time interval is determined, the system will periodically activate the camera to shoot according to the predetermined rules. Each time it is triggered, a series of standardized operation procedures will be executed: first, the camera device is initialized, its connection status is checked to see if it is normal, and the correct resolution and other necessary parameters are set; then a command is issued to open the camera shutter, complete an exposure process, and finally generate a digital image file. Since the entire process involves real-time data transmission, it is necessary to ensure that the communication link is stable and reliable to avoid image loss due to network delays or packet loss.

[0025] In the entire image acquisition process, in addition to paying attention to technical factors, privacy protection is also a topic that cannot be ignored. Given that personal facial information is sensitive data, developers have taken a series of strict measures to protect user rights from infringement. On the one hand, all image data is limited to internal use and is strictly prohibited from being leaked; on the other hand, unnecessary storage is minimized without compromising functionality, and only enough data copies are retained for the current task. At the same time, users will be reminded to understand and agree to the relevant terms to ensure that their right to know is fully respected. In this way, the intelligent health care companion robot can efficiently obtain high-quality user facial images while taking into account security and user experience, providing a solid foundation for realizing intelligent facial tracking.

[0026] In the embodiment of the present application, the facial image processing module 120 is used to preprocess and extract facial features from the first user facial image and the second user facial image respectively to obtain first user facial features and second user facial features. Figure 3 FIG. 1 is a block diagram of a facial image processing module in an intelligent health care companion robot with a facial tracking function according to an embodiment of the present application. Specifically, Figure 3 As shown, the facial image processing module 120 includes: a facial image preprocessing unit 121, used to input the first user facial image and the second user facial image into an image preprocessing module to obtain a preprocessed first user facial image and a preprocessed second user facial image; a facial image feature extraction unit 122, used to input the preprocessed first user facial image and the preprocessed second user facial image into a face feature extractor based on a Mobile-Former model respectively to obtain a first user facial feature map as the first user facial feature and a second user facial feature map as the second user facial feature.

[0027] Specifically, the facial image preprocessing unit 121 is used to input the first user facial image and the second user facial image into an image preprocessing module to obtain a preprocessed first user facial image and a preprocessed second user facial image. Accordingly, considering that different lighting conditions, shooting angles or environmental factors may cause the quality of the original user facial images to be uneven, and the images collected in the actual environment often contain various types of noise (such as Gaussian noise, salt and pepper noise, etc.), based on this, in order to convert these images into a unified standard format, ensure the consistency of subsequent analysis, and effectively reduce noise interference and improve image quality, in the technical solution of the present application, the first user facial image and the second user facial image are input into the image preprocessing module to resize and grayscale the original images to obtain the preprocessed first user facial image and the preprocessed second user facial image.

[0028] Specifically, after the first user's facial image and the second user's facial image are captured by the camera, they will first be sent to the image preprocessing module. This process begins with the standardization of image resolution. Since the images taken by different devices have different sizes, if they are not unified, it may cause waste of computing resources or result deviation in subsequent analysis. Therefore, the system will adjust each input image to a fixed size according to a preset standard (usually based on the display resolution of the target platform). Doing so not only helps to increase processing speed, but also ensures consistency of output. The next step is color space conversion, that is, converting color images into grayscale images. Although color information is very important for some application scenarios, in facial recognition tasks, brightness changes often reflect facial structural characteristics better than color differences. By removing color components, the complexity of the problem can be simplified and the amount of data can be reduced, thereby speeding up the processing process.

[0029] After completing the basic resizing and grayscale, the next step is to optimize the image quality. This step is mainly aimed at eliminating noise interference and improving visual effects. In real life, due to factors such as unstable lighting conditions and sensor characteristics, the original images obtained often contain various types of noise, such as Gaussian noise, salt and pepper noise, etc. If these noises are not removed, the effect of feature extraction will be seriously affected. To this end, the preprocessing module uses a series of advanced filtering algorithms to purify the image. For example, the bilateral filter can smooth the internal area of ​​the image while maintaining edge details; while the adaptive median filter is good at suppressing impulse noise. In addition to noise reduction, contrast enhancement is also a link that cannot be ignored. Through means such as histogram equalization or gamma correction, both the dark and bright parts of the image can be better presented, thereby highlighting facial features.

[0030] In addition to the above routine processing, the preprocessing module also performs some specific tasks to further optimize the image. One of the key points is to locate and crop the part containing the face. Although modern computer vision technology is very mature, there may still be a risk of false detection or multiple detection when facing complex backgrounds. In order to avoid this, the system will use an efficient detection algorithm to determine the approximate location of the face on the basis of preliminary processing, and then draw a rectangular frame around the area. This process can not only narrow the scope of subsequent analysis and reduce computational costs, but also effectively eliminate interference caused by irrelevant elements. It is worth noting that the cropping here is not a simple geometric cut, but is combined with information from posture estimation to ensure that even faces shot at non-frontal angles can be captured correctly.

[0031] After completing all the above steps, the output of the preprocessing module is the optimized facial image of the first user. This image has been adjusted to a standard size, converted to grayscale format, noise interference removed, contrast enhanced, and the part containing the face accurately cropped out. More importantly, through multi-scale analysis, each pixel in the image carries rich local semantic information, which is fully prepared for subsequent facial feature extraction and analysis. In this way, the intelligent health care companion robot can obtain high-quality facial images as input, providing accurate data for subsequent processing.

[0032] Specifically, the facial image feature extraction unit 122 is used to input the preprocessed first user facial image and the preprocessed second user facial image into a facial feature extractor based on the Mobile-Former model to obtain a first user facial feature map as the first user facial feature and a second user facial feature map as the second user facial feature. It should be understood that, considering that the first user facial image and the second user facial image both contain facial feature information of different levels and scales, such as the overall shape of the face, the local positions of the facial features, etc. Therefore, in the technical solution of the present application, the preprocessed first user facial image and the preprocessed second user facial image are respectively input into a facial feature extractor based on the Mobile-Former model to capture and extract features of different scales in the user facial images, and obtain the first user facial feature map and the second user facial feature map. It is understandable that the Mobile-Former model is a hybrid model that combines the advantages of the convolutional neural network (CNN) and the Transformer architecture. It combines the global modeling capabilities of the Transformer and the local feature capture capabilities of the convolutional neural network (CNN), while maintaining computational efficiency and model lightweight, aiming to provide efficient visual processing capabilities for mobile devices and other resource-constrained environments. Therefore, compared with traditional CNN, Mobile-Former can better capture local features of the face, such as the specific morphological changes of key parts such as the eyes, nose, and mouth, which helps to more accurately identify the user's facial features, especially in the case of expression changes or subtle movements.

[0033] In particular, the processing process of the first user facial image after preprocessing is taken as an example for specific explanation: when the first user facial image after preprocessing enters the Mobile-Former model, it will first pass through a series of lightweight convolutional layers. The main task of these convolutional layers is to capture low-level features in the image, such as basic visual elements such as edges and textures. In order to ensure computational efficiency, depthwise separable convolution is used here to replace the traditional standard convolution operation. This method can effectively extract local features without significantly increasing the number of parameters. Each convolutional layer is followed by batch normalization and ReLU activation functions to accelerate the training process and improve the generalization ability of the model. In addition, in order to better retain spatial information, the stride of the convolutional layer is usually set to 1, and the padding technique is used to keep the output size unchanged.

[0034] As the feature map becomes smaller, the network enters the local-global interaction module. At this stage, Mobile-Former introduces the Self-Attention Mechanism to facilitate the encoding of long-distance dependencies. LGIM contains two paths: one is a conventional convolution path that focuses on capturing close spatial relationships; the other is a path based on the self-attention mechanism to understand broader contextual information. These two paths do not work in isolation, but collaborate with each other through the Multi-Head Mechanism to ultimately output a fused feature representation. This design enables the model to focus on fine-grained local details while also understanding the overall view of the entire face. For facial recognition tasks, such a structure helps to accurately capture the specific morphological changes of key parts such as eyes, nose, and mouth.

[0035] Going further into the network, Mobile-Former begins to rely more on the Transformer component. The structure of this part is designed to better simulate the behavior of the self-attention mechanism, thereby enhancing the model's ability to understand complex patterns. Each multi-head self-attention layer is followed by a feedforward network, which consists of two linear mappings with a layer of activation function GELU in between. This combination allows the model to learn more abstract concepts, not just simple spatial position relationships. At the same time, residual connections are added to each layer to prevent the gradient vanishing problem and promote information flow. Even when the network is very deep, the residual connection can maintain an effective backpropagation path, which is crucial for stabilizing the training process.

[0036] In the last few stages, Mobile-Former returns to a design concept that is more CNN-like. At this point, the model mainly relies on techniques such as dense connections and pyramid pooling to enhance feature expression. Dense connections break the bottleneck effect in information transmission and promote cross-layer feature reuse; while pyramid pooling aggregates global context information based on receptive fields of different scales, providing rich semantic descriptions for subsequent classification or other tasks. Finally, after a series of complex processing, Mobile-Former generates a set of highly concentrated and meaningful features, namely the first user facial feature map. These features not only contain the overall shape information of the face, but also record in detail the specific position and morphological changes of the facial features, laying a solid foundation for achieving accurate facial tracking.

[0037] In the entire processing flow, the Mobile-Former model cleverly combines the advantages of CNN and Transformer, ensuring the effective capture of local features and achieving good modeling of global information. Especially in the face of application scenarios with limited computing resources such as mobile devices, Mobile-Former, with its unique structural design and optimization strategy, can significantly reduce energy consumption while maintaining high performance, providing users with a smooth and natural face recognition experience.

[0038] In an embodiment of the present application, the facial feature difference encoding module 130 is used to perform feature global semantic encoding on the first user facial features and the second user facial features to obtain semantic difference encoding features of user facial posture. Accordingly, it is considered that the first user facial features and the second user facial features respectively express the facial information of the user at two different time points. Specifically, the first user facial features refer to the original feature state of the user's face at a specific time point or under certain circumstances. The second user facial features capture the dynamic changes in the face over time. Therefore, in order to more accurately capture the subtle differences in the user's facial features between the two time points, so as to improve the subsequent precise adjustment of the screen posture, in the technical solution of the present application, the feature global semantic encoding technology is used to perform semantic difference compensation analysis on the first user facial features and the second user facial features. In this way, the understanding of the relationship between the two features is enhanced through the global semantic mechanism, and the key areas that have changed in the two observations can be highlighted, thereby providing more valuable information for the adjustment of the screen posture of the intelligent health companion robot.

[0039] Figure 4 FIG. 1 is a block diagram of a facial feature difference encoding module in an intelligent health care companion robot with a facial tracking function according to an embodiment of the present application. Specifically, Figure 4 As shown, the facial feature difference encoding module 130 includes: a user facial feature local fusion unit 131, which is used to perform fine-grained local semantic fusion on the first user facial feature and the second user facial feature to obtain a set of fine-grained local shared semantic features of the first and second user faces; a user facial feature semantic compensation interaction unit 132, which is used to perform semantic compensation interaction aggregation on the set of fine-grained local shared semantic features of the first and second user faces to obtain the user facial posture semantic difference encoding features.

[0040] Specifically, in an embodiment of the present application, the user facial feature local fusion unit 131 includes: a facial feature decomposition unit, which is used to perform feature decomposition on the first user facial feature map and the second user facial feature map to obtain a set of first user facial semantic local feature matrices and a set of second user facial semantic local feature matrices; a facial feature shared semantic extraction subunit, which is used to perform twin shared semantic extraction on each corresponding group of first user facial semantic local feature matrices and second user facial semantic local feature matrices in the set of the first user facial semantic local feature matrices and the set of the second user facial semantic local feature matrices to obtain a set of first and second user facial fine-grained local shared semantic feature matrices as a set of first and second user facial fine-grained local shared semantic features.

[0041] More specifically, in the embodiment of the present application, the facial feature decomposition unit is used to: perform feature decomposition on the first user facial feature map and the second user facial feature map along the channel dimension of the first user facial feature map and the second user facial feature map to obtain a set of the first user facial semantic local feature matrix and a set of the second user facial semantic local feature matrix. The process is expressed by the formula: ;in, and are respectively the first user facial feature map and the second user facial feature map, To perform feature dissociation on the feature map, and are the first, second, and third facial semantic local feature matrices of the first user, respectively. and The first user's facial semantic local feature matrix, and are the first, second, and third facial semantic local feature matrices of the second user respectively. and The second user's facial semantic local feature matrix.

[0042] It should be understood that feature dissociation is performed on the first user facial feature map and the second user facial feature map along the channel dimension of the first user facial feature map and the second user facial feature map to decompose the information in each feature map into multiple semantic local features, each of which focuses on capturing a specific type of facial feature (such as features of local areas such as eyes, nose, mouth, etc.), to obtain a set of the first user facial semantic local feature matrix and a set of the second user facial semantic local feature matrix, thereby achieving more refined feature expression.

[0043] More specifically, in an embodiment of the present application, the facial feature shared semantic extraction subunit is used to perform twin shared semantic extraction on each corresponding first user facial semantic local feature matrix and second user facial semantic local feature matrix in the set of the first user facial semantic local feature matrix and the set of the second user facial semantic local feature matrix to obtain a set of first and second user facial fine-grained local shared semantic feature matrices. The process is expressed by the formula: ;in, is the first user's facial semantic local feature matrix in the set The first user's facial semantic local feature matrix, is the first in the set of the second user's facial semantic local feature matrix The second user's facial semantic local feature matrix, and They are point-by-point addition, point-by-point multiplication, and point-by-point subtraction. For cascade processing, is point convolutional coding, yes function, yes and A fine-grained local shared semantic feature matrix between the first and second user faces.

[0044] Accordingly, twin shared semantics extraction is performed on each set of corresponding first user facial semantic local feature matrices and second user facial semantic local feature matrices to obtain a set of first and second user facial fine-grained local shared semantic feature matrices. That is, by using the twin network structure, two inputs (i.e., facial semantic local feature matrices of the first user and the second user) can be processed simultaneously, and the two inputs are mapped through the same weights in the shared network layer to learn a shared semantic space in which the same or similar facial features of users from different channels will have similar representations.

[0045] Specifically, in an embodiment of the present application, the user facial feature semantic compensation interaction unit 132 includes: a user facial feature semantic compensation calculation subunit, which is used to calculate the semantic compensation features of each first and second user facial fine-grained local shared semantic feature matrix in the set of the first and second user facial fine-grained local shared semantic feature matrices to obtain a set of first and second user facial compensation text semantic coding feature matrices; a user facial feature semantic enhancement fusion subunit, which is used to perform semantic enhancement fusion on the set of the first and second user facial fine-grained local shared semantic feature matrices according to the set of the first and second user facial compensation text semantic coding feature matrices to obtain a user facial posture semantic difference coding feature map as the user facial posture semantic difference coding feature.

[0046] More specifically, in the embodiment of the present application, the user facial feature semantic compensation calculation subunit is used to: perform semantic compensation decoding based on the large language model on each first and second user facial fine-grained local shared semantic feature matrix in the set of the first and second user facial fine-grained local shared semantic feature matrices to obtain a set of first and second user facial semantic compensation text descriptions, and the process is expressed by the formula: ;in, yes and The fine-grained local shared semantic feature matrix between the first and second user faces, For semantic compensation decoding operation, for The corresponding semantically compensated text descriptions of the first and second user's faces.

[0047] The first and second user facial semantic compensation text descriptions in the set of the first and second user facial semantic compensation text descriptions are respectively subjected to text convolution semantic coding to obtain a set of the first and second user facial semantic compensation text semantic coding feature matrices. The process is expressed by the formula: ;in, for The corresponding semantically compensated text descriptions of the first and second user's faces, Convolutional coding of text, for The corresponding first and second user facial compensation text semantic encoding feature matrices.

[0048] It should be understood that if only common semantic features are emphasized in feature interaction, other key features, such as complementary features, may be ignored. These complementary features carry some unique information. Although they may not be significant in a single feature, they can supplement more context and details when the features are fused, thereby enhancing the comprehensiveness and accuracy of the final feature representation. Therefore, the semantic compensation decoding based on the large language model is performed on each of the fine-grained local shared semantic feature matrices of the first and second users' faces. By using the large language model, the complex facial semantic features can be converted into natural language text that is easy to understand and interpret, and a set of semantic compensation text descriptions of the first and second users' faces is obtained. In particular, the large language model can automatically supplement the missing information or refine the ambiguous expression according to the context, thereby generating a more complete and accurate text description. For example, if a facial feature only shows a slight change, the large language model can infer a more detailed explanation based on other related features.

[0049] Correspondingly, text convolution semantic coding is performed on each of the first and second user facial semantic compensation text descriptions to extract deeper semantic information and perform feature alignment, thereby obtaining a set of first and second user facial compensation text semantic coding feature matrices.

[0050] More specifically, in an embodiment of the present application, the user facial feature semantic enhancement fusion subunit is used to: perform fine-grained semantic interaction compensation on each corresponding first and second user facial fine-grained local shared semantic feature matrix and first and second user facial compensation text semantic encoding feature matrix in the set of the first and second user facial fine-grained local shared semantic feature matrix and the set of the first and second user facial compensation text semantic encoding feature matrix to obtain a set of first and second user facial fine-grained local interaction semantic enhancement feature matrices, and the process is expressed by the formula: ;in, yes and The fine-grained local shared semantic feature matrix between the first and second user faces, for The corresponding first and second user face compensation text semantic encoding feature matrix, for The transposed matrix of is matrix multiplication, is the The scale is the matrix width multiplied by the matrix height, yes function, and are weighting coefficients, is the first and second user's facial fine-grained local interaction semantic enhancement feature matrix in the set A first and second user facial fine-grained local interaction semantic enhancement feature matrix.

[0051] The set of the first and second user facial fine-grained local interaction semantic reinforcement feature matrices is aggregated along the channel dimension to obtain the user facial pose semantic difference encoding feature map, and the process is expressed by the formula: ;in, and The first, second, and third sets of the fine-grained local interaction semantic enhancement feature matrices of the first and second user's faces are respectively and The first and second user's facial fine-grained local interaction semantic enhancement feature matrix, is the number of feature matrices in the set of the first and second user facial fine-grained local interaction semantics enhanced feature matrices, To aggregate a collection of feature matrices along the channel dimension, It is the semantic difference encoding feature map of the user's facial part and pose.

[0052] It should be understood that fine-grained semantic interaction compensation is performed on each set of corresponding first and second user facial fine-grained local shared semantic feature matrices and first and second user facial compensated text semantic encoding feature matrices to obtain a set of first and second user facial fine-grained local interactive semantic reinforcement feature matrices. That is, visual features and text descriptions can often provide complementary information. For example, some subtle changes in facial expressions may not be easily perceived in the image, but are clearly stated in the text description.

[0053] Finally, the set of the first and second user facial fine-grained local interaction semantic reinforcement feature matrices is aggregated along the channel dimension to obtain the user facial position and posture semantic difference encoding feature map. In this way, facial feature information in different areas and at different levels is integrated together, so that the robot can understand the user's facial position and posture changes more accurately, and thus intelligently adjust the screen posture to adapt to the user's current state and needs.

[0054] In an embodiment of the present application, the facial tracking instruction adjustment module 140 is used to obtain a facial tracking instruction for representing screen posture adjustment data according to the semantic difference coding feature of the user's facial posture. Specifically, in an embodiment of the present application, the facial tracking instruction adjustment module is used to: input the semantic difference coding feature map of the user's facial posture into a decoder-based facial tracking module to obtain the facial tracking instruction, and the facial tracking instruction includes screen posture adjustment data. That is, the semantic difference coding feature of the user's facial posture obtained by performing feature global encoding using the first user facial feature map and the second user facial feature map is decoded and processed to intelligently adjust the screen posture data. By real-time acquisition of the user's facial image and analysis of its posture, the robot can intelligently adjust the screen posture to ensure that it is always in the user's best viewing position, greatly reducing the difficulty of operation and visual fatigue. And the user's facial orientation can be tracked in real time to provide a more natural and smooth interactive experience.

[0055] It should be understood that the facial tracking instructions are essentially a guide to the robot's hardware actions, and the most critical part is the screen posture adjustment data. The screen posture adjustment data determines how the robot moves its display screen to ensure the best viewing angle. This includes but is not limited to the screen's up and down rotation angles (elevation angle, depression angle), horizontal rotation angles (left and right turn), and possible vertical or horizontal translation distances. In addition, considering that different users may have different sitting or standing heights, even for elderly people sitting in wheelchairs, the height of the screen needs to be adjusted accordingly. Therefore, the screen posture adjustment data is a multi-dimensional set of parameters that accurately defines all the necessary steps for the screen to transform from its current position to a new position.

[0056] Specifically, the user face position and posture semantic difference encoding feature map is input into the decoder-based face tracking module, which can be achieved in the following way: first, the user face position and posture semantic difference encoding feature map is provided as input to the decoder. This feature map contains rich facial position, posture and expression information, which is the basis for the next step of analysis.

[0057] The decoder is usually part of a deep learning model, such as a deconvolution layer or a fully connected layer in a convolutional neural network (CNN), which is used to reconstruct high-level abstract features into specific outputs. The decoder can be designed as a multi-task learning framework that can not only predict the location of facial key points, but also estimate the posture angles of the head (such as pitch, yaw, and roll), and recognize changes in facial expressions. Use a pre-trained posture estimation model to determine the angle of the user's head relative to the device. This step is critical for calculating accurate screen posture adjustment because different head postures affect the choice of the best viewing angle. By detecting facial key points (such as eyes, nose, mouth, etc.), the user's facial center and its position in space are further accurately located. This is very important to ensure that the screen content is always within the user's optimal viewing range.

[0058] In the actual implementation process, after the decoder receives data from the semantic difference encoding feature map of the user's facial posture, it will start a series of complex algorithm operations. First, it needs to identify the change trend of the current user's face relative to the initial set position of the robot. For example, if the user turns his head to the left, the decoder needs to understand that this is a signal that the left tilt angle needs to be increased; if the user looks up, it means that the screen should increase its elevation angle appropriately. Next, the decoder will evaluate the degree of this change to determine the specific adjustment amount. This process is not a simple linear mapping, but involves nonlinear calculations, because there is a complex correspondence between the change in facial posture and the ideal screen position, and other factors such as ambient light and the distance between the user and the robot need to be considered.

[0059] Once the decoder has completed the above analysis, it will generate a set of specific screen posture adjustment data. These data are then passed to the robot's motion control system, which is responsible for performing the actual physical adjustments. For example, if the decoder determines that the screen needs to be rotated 15 degrees to the left and lowered 2 cm, the motion control system will drive the motor or other mechanical device to achieve such a change.

[0060] That is, the generated screen posture adjustment instructions will be sent to the control system, which is responsible for actually changing the physical position or parameter settings of the screen. This may involve the operation of a robotic arm, motor, or other automated equipment, or it may be limited to software-level parameter adjustments, such as changes in display settings. Such adjustments can greatly improve the quality of human-computer interaction, allowing users to enjoy a more natural and comfortable experience. In this way, not only the interactive performance of the intelligent health care companion robot is improved, but also more humane services are provided for the elderly and other people with limited mobility, significantly improving their quality of life.

[0061] Preferably, inputting the user face pose semantic difference coded feature map into a decoder-based face tracking module to obtain a face tracking instruction comprises: determining a feature mean of the user face pose semantic difference coded feature map and characteristic variance , and the feature mean Divide by the feature mean With the characteristic variance The sum of the semantic difference encoding probability values ​​of the user's facial position and pose is obtained .

[0062] The feature mean Multiply by the user face position posture semantic difference coding probability value to obtain the user face position posture semantic difference coding statistical field value .

[0063] Subtract one from the user face pose semantic difference coding statistical field value and divide it by the user face pose semantic difference coding statistical field value to obtain the user face pose semantic difference coding partial probability value .

[0064] Calculate the user face position semantic difference coding feature map with the user face position semantic difference coding partial probability value as the power function of the exponent , and multiplied by the partial probability value of the semantic difference encoding of the user's facial part posture to obtain the microscopic representation feature map of the semantic difference encoding of the user's facial part posture ,in, represents the semantic difference encoding feature map of the user's facial pose, Represents dot product.

[0065] After multiplying the user face pose semantic difference coding feature map by the user face pose semantic difference coding partial probability value, an exponential function with a natural constant as the base is calculated to obtain the user face pose semantic difference coding macro mapping feature map .

[0066] After calculating the logarithm of the user face pose semantic difference coding micro-representation feature map with base 2, the weighted sum of the feature map is performed with the user face pose semantic difference coding macro-mapping feature map to obtain an optimized user face pose semantic difference coding feature map. ,in, and represents different weighted hyperparameters, Indicates point addition.

[0067] The optimized user face pose semantic difference encoding feature map is input into a decoder-based face tracking module to obtain a face tracking instruction.

[0068] Here, considering that the first user facial feature map and the second user facial feature map respectively represent the image semantic features of the preprocessed first user facial image and the preprocessed second user facial image, when performing feature enhancement interactive coding based on cross-domain fine-grained semantic compensation, due to the cross-domain fine-grained semantic compensation differences caused by the differences in the distribution of image semantic features, the user facial part pose semantic difference coding feature map will have a probability convergence and divergence based on different feature enhancement interactivity, thereby affecting the accuracy of the facial tracking instructions obtained by the input decoder-based facial tracking module.

[0069] Based on this, the partial low-order derivative of the statistical distribution field corresponding to the user facial pose semantic difference coding feature map is used as the non-overlapping macro-feature representation behavior patch of the user facial pose semantic difference coding feature map, and the different macro-behavior patch organization space under the non-isotropic backbone structure of the user facial pose semantic difference coding feature map is used to enhance the dynamic sensitivity of the long-series micro-complex information distribution of the user facial pose semantic difference coding feature map to the generation of probabilistic macro-representation behavior, thereby achieving the common agility of convergence of the probability convergent and divergent feature set of the user facial pose semantic difference coding feature map to the probability density distribution space, and improving the accuracy of the facial tracking instruction obtained by inputting the user facial pose semantic difference coding feature map into the decoder-based facial tracking module.

[0070] In summary, the intelligent health care companion robot 100 with facial tracking function based on the embodiment of the present application is explained, which collects the facial image of the first user and the facial image of the second user by the camera of the intelligent health care companion robot, and uses the image analysis and processing technology based on computer vision to perform image preprocessing and facial feature analysis on the first user facial image and the second user facial image, so as to intelligently adjust the screen posture data according to the fine-grained global semantic compensation representation between the facial features of the first user and the second user. By collecting the user's facial image in real time and analyzing its posture, the robot can intelligently adjust the screen posture to ensure that it is always in the best viewing position for the user, greatly reducing the difficulty of operation and visual fatigue. And through the intelligent automatic control method, it can track and adapt to the user's facial orientation in real time, providing a more natural and smooth interactive experience.

[0071] As described above, the intelligent health care companion robot 100 with facial tracking function according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an intelligent algorithm with facial tracking function. In one possible implementation, the intelligent health care companion robot 100 with facial tracking function according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the intelligent health care companion robot 100 with facial tracking function can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the intelligent health care companion robot 100 with facial tracking function can also be one of the many hardware modules of the wireless terminal.

[0072] Alternatively, in another example, the smart health care companion robot 100 with facial tracking function and the wireless terminal may also be separate devices, and the smart health care companion robot 100 with facial tracking function may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0073] The above descriptions of various implementations of the present disclosure are exemplary and not exhaustive. The disclosure is not limited to the disclosed implementations, and many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The terms used herein are selected to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various practical methods disclosed herein.

Claims

1. An intelligent health care companion robot with facial tracking function, characterized in that: include: A facial image acquisition module, used to acquire a first user facial image and a second user facial image captured by a camera of the intelligent health care companion robot; a facial image processing module, used to preprocess the first user facial image and the second user facial image and extract facial features to obtain first user facial features and second user facial features; A facial feature difference encoding module is used to perform global semantic encoding of the first user facial features and the second user facial features to obtain user facial posture semantic difference encoding features, wherein the facial feature difference encoding module includes: a user facial feature local fusion unit, used to perform fine-grained local semantic fusion of the first user facial features and the second user facial features to obtain a set of fine-grained local shared semantic features of the first and second user faces; a user facial feature semantic compensation interaction unit, used to perform semantic compensation interaction aggregation on the set of fine-grained local shared semantic features of the first and second user faces to obtain the user facial posture semantic difference encoding features; a facial tracking instruction adjustment module, used to obtain a facial tracking instruction for representing screen posture adjustment data based on the user facial posture semantic difference encoding features.

2. The intelligent health care companion robot with facial tracking function according to claim 1, characterized in that: There is a predetermined time interval between the first user facial image and the second user facial image.

3. The intelligent health care companion robot with facial tracking function according to claim 2, characterized in that: The facial image processing module includes: a facial image preprocessing unit, which is used to input the first user facial image and the second user facial image into the image preprocessing module to obtain a preprocessed first user facial image and a preprocessed second user facial image; a facial image feature extraction unit, which is used to input the preprocessed first user facial image and the preprocessed second user facial image into a face feature extractor based on the Mobile-Former model respectively to obtain a first user facial feature map as the first user facial feature and a second user facial feature map as the second user facial feature.

4. The intelligent health care companion robot with facial tracking function according to claim 3, characterized in that: The user facial feature local fusion unit includes: a facial feature decomposition unit, which is used to perform feature decomposition on the first user facial feature map and the second user facial feature map to obtain a set of first user facial semantic local feature matrices and a set of second user facial semantic local feature matrices; a facial feature shared semantic extraction subunit, which is used to perform twin shared semantic extraction on each corresponding group of first user facial semantic local feature matrices and second user facial semantic local feature matrices in the set of the first user facial semantic local feature matrices and the set of the second user facial semantic local feature matrices to obtain a set of first and second user facial fine-grained local shared semantic feature matrices as a set of the first and second user facial fine-grained local shared semantic features.

5. The intelligent health care companion robot with facial tracking function according to claim 4, characterized in that: The facial feature decomposition unit is used to perform feature decomposition on the first user facial feature map and the second user facial feature map along the channel dimension of the first user facial feature map and the second user facial feature map to obtain a set of the first user facial semantic local feature matrices and a set of the second user facial semantic local feature matrices.

6. The intelligent health care companion robot with facial tracking function according to claim 5, characterized in that: The user facial feature semantic compensation interaction unit includes: a user facial feature semantic compensation calculation subunit, which is used to calculate the semantic compensation features of each first and second user facial fine-grained local shared semantic feature matrix in the set of the first and second user facial fine-grained local shared semantic feature matrices to obtain a set of first and second user facial compensation text semantic coding feature matrices; a user facial feature semantic enhancement fusion subunit, which is used to perform semantic enhancement fusion on the set of the first and second user facial fine-grained local shared semantic feature matrices according to the set of the first and second user facial compensation text semantic coding feature matrices to obtain a user facial part posture semantic difference coding feature map as the user facial part posture semantic difference coding feature.

7. The intelligent health care companion robot with facial tracking function according to claim 6, characterized in that: The user facial feature semantic compensation calculation subunit is used to: perform semantic compensation decoding based on a large language model on each first and second user facial fine-grained local shared semantic feature matrix in the set of the first and second user facial fine-grained local shared semantic feature matrices to obtain a set of first and second user facial semantic compensation text descriptions; perform text convolution semantic encoding on each first and second user facial semantic compensation text description in the set of the first and second user facial semantic compensation text descriptions to obtain a set of the first and second user facial compensation text semantic encoding feature matrices.

8. The intelligent health care companion robot with facial tracking function according to claim 7, characterized in that: The user facial feature semantic enhancement fusion subunit is used to: perform fine-grained semantic interaction compensation on each corresponding group of first and second user facial fine-grained local shared semantic feature matrices and first and second user facial compensation text semantic coding feature matrices in the set of the first and second user facial fine-grained local shared semantic feature matrices and the set of the first and second user facial compensation text semantic coding feature matrices to obtain a set of first and second user facial fine-grained local interaction semantic enhancement feature matrices; aggregate the set of the first and second user facial fine-grained local interaction semantic enhancement feature matrices along the channel dimension to obtain the user facial posture semantic difference coding feature map.

9. The intelligent health care companion robot with facial tracking function according to claim 8, characterized in that: The facial tracking instruction adjustment module is used to: input the user face posture semantic difference encoding feature map into the decoder-based facial tracking module to obtain the facial tracking instruction, and the facial tracking instruction includes screen posture adjustment data.

Images