Intelligent robot with scene perception capability and method

Abstract

The invention relates to the technical field of intelligent robots, and particularly discloses an intelligent robot with scene perception capability and a method, and the intelligent robot with scene perception capability uses a camera of the robot to collect a pose monitoring video of a target old-age object. Meanwhile, a communication module of the robot is used for receiving a time queue of the pulse data, collected by the intelligent wearable device, of the target elderly object; then, based on the perceived data, the intelligent robot can identify the current situation and adjust the behavior of the intelligent robot according to the current situation so as to select a more appropriate behavior strategy. Therefore, different requirements of the old people in the scene can be predicted through the sensing information of the intelligent robot for the scene, so that personalized service content is customized, for example, when it is detected that the old people are resting, the volume can be reduced to avoid disturbing, and more personalized and intelligent health care service support is provided.

Description

Technical Field

[0001] The present application relates to the field of intelligent robot technology, and more specifically, to an intelligent robot and method with scene perception capability. Background Art

[0002] In the field of health care services, elderly care and companion robots play an increasingly important role. As the global population aging trend intensifies, more and more elderly people need assistance and support in their daily lives. Traditional manual care methods are not only costly, but also difficult to provide 24-hour uninterrupted service. The development of robotics technology provides a new way to solve this problem. They can provide round-the-clock services, reduce the workload of caregivers, and provide safer, more comfortable and personalized care for the elderly.

[0003] However, although traditional care and companion robots have improved the quality of life of the elderly to a certain extent, they still have some significant limitations. For example, existing robots can usually only perform pre-programmed tasks and cannot flexibly adjust their behavior according to changes in the surrounding environment, which means that they cannot adapt well to each elderly person's unique daily living habits or emergencies. In addition, due to algorithmic and technical limitations, these devices often do not have sufficient autonomy and learning ability, and have difficulty making complex decisions or predicting user needs, which limits their effectiveness as companion robots.

[0004] Therefore, an intelligent robot with scene perception capability is desired. Summary of the invention

[0005] The present application provides an intelligent robot and method with scene perception capabilities, which can predict the different needs of the elderly in the scene through the intelligent robot's perception information of the scene, thereby customizing personalized service content. For example, when it detects that the elderly are resting, it will lower the volume to avoid disturbing them, thereby providing more personalized and intelligent health care service support.

[0006] In a first aspect, an intelligent robot with scene perception capability is provided, comprising:

[0007] A posture monitoring video acquisition module, which is used to use the robot's camera to collect posture monitoring videos of target elderly objects;

[0008] A pulse time series data acquisition module, used for receiving the time queue of the pulse data of the target elderly object collected from the smart wearable device through the communication module of the robot;

[0009] A behavior pattern feature extraction module, used for extracting features based on the behavior pattern of the elderly from the posture monitoring video to obtain semantic coding features of the behavior pattern of the elderly;

[0010] A pulse time series feature extraction module, used for performing pulse time series encoding on the time queue of the pulse data to obtain pulse time series semantic encoding features;

[0011] A scene multimodal joint perception module, used for inputting the semantic coding features of the elderly behavior pattern and the semantic coding features of the pulse timing into a cross-modal scene sensor to obtain a scene multimodal fine-grained joint perception representation, wherein the scene multimodal joint perception module includes: a feature interaction optimization unit, used for performing feature interaction optimization processing based on external knowledge modulation on the semantic coding features of the elderly behavior pattern and the semantic coding features of the pulse timing to obtain an external knowledge optimized scene multimodal fine-grained feature interaction representation; a fine-grained modulation optimization coding unit, used for performing semantic association coding based on fine-grained modulation optimization on the semantic coding features of the elderly behavior pattern and the semantic coding features of the pulse timing to obtain the scene multimodal fine-grained joint perception representation based on the external knowledge optimized scene multimodal fine-grained feature interaction representation;

[0012] The scene type judgment and behavior adjustment module is used to determine the scene type label and specify the behavior strategy of the robot based on the multimodal fine-grained joint perception representation of the scene.

[0013] In a second aspect, a control method for an intelligent robot with scene perception capability is provided, comprising:

[0014] Use the robot's camera to collect posture monitoring video of the target elderly object;

[0015] Receiving, through the communication module of the robot, a time queue of the pulse data of the target elderly subject collected from a smart wearable device;

[0016] Extracting features based on the behavior patterns of the elderly from the posture monitoring video to obtain semantic coding features of the behavior patterns of the elderly;

[0017] Performing pulse timing coding on the time queue of the pulse data to obtain pulse timing semantic coding features;

[0018] Inputting the semantic coding features of the elderly behavior pattern and the pulse timing semantic coding features into a cross-modal scene sensor to obtain a multi-modal fine-grained joint perception representation of the scene, including: performing feature interaction optimization processing based on external knowledge modulation on the semantic coding features of the elderly behavior pattern and the pulse timing semantic coding features to obtain an external knowledge optimized multi-modal fine-grained feature interaction representation of the scene; based on the external knowledge optimized multi-modal fine-grained feature interaction representation of the scene, performing semantic association coding based on fine-grained modulation optimization on the semantic coding features of the elderly behavior pattern and the pulse timing semantic coding features to obtain the multi-modal fine-grained joint perception representation of the scene;

[0019] Based on the multimodal fine-grained joint perception representation of the scene, a scene type label is determined, and a behavior strategy of the robot is specified.

[0020] The present application provides an intelligent robot and method with scene perception capability, which uses the robot's camera to collect posture monitoring video of the target elderly object. At the same time, the robot's communication module receives the time queue of the pulse data of the target elderly object collected by the intelligent wearable device. Then, based on the perceived data, the intelligent robot can identify the current situation and adjust its behavior accordingly to select a more appropriate behavior strategy. In this way, the intelligent robot can predict the different needs of the elderly in the scene through its perception information of the scene, thereby customizing personalized service content. For example, when it detects that the elderly are resting, it will lower the volume to avoid disturbing them, thereby providing more personalized and intelligent health care service support. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0022] Figure 1 A schematic block diagram of an intelligent robot with scene perception capability according to an embodiment of the present application.

[0023] Figure 2 Schematic diagram of data flow of an intelligent robot with scene perception capability according to an embodiment of the present application.

[0024] Figure 3 This is a schematic block diagram of a scene multimodal joint perception module in an intelligent robot with scene perception capability according to an embodiment of the present application.

[0025] Figure 4 This is a schematic block diagram of a feature interaction optimization unit in an intelligent robot with scene perception capability according to an embodiment of the present application.

[0026] Figure 5 This is a schematic block diagram of a fine-grained modulation optimization coding unit in an intelligent robot with scene perception capability according to an embodiment of the present application.

[0027] Figure 6 This is a schematic block diagram of a control method for an intelligent robot with scene perception capability according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0029] Intelligent robots with scene perception capabilities are an important innovation in the field of health care services. They combine advanced technologies in multiple fields such as computer vision, deep learning, sensor technology, and artificial intelligence. Such robots can capture environmental images through built-in or external cameras, and use advanced image processing algorithms to analyze this information to identify the posture and behavior patterns of the elderly (such as walking, sitting, lying down, etc.). At the same time, with the help of connection with smart wearable devices, it can monitor the vital signs of the elderly in real time, such as the temporal changes of the pulse. Then, based on the perceived data, the intelligent robot can identify the current situation and adjust its behavior accordingly to select a more appropriate behavior strategy. In this way, the intelligent robot can predict the different needs of the elderly in the scene through the perception information of the scene, so as to customize personalized service content. For example, when it detects that the elderly are resting, it will lower the volume to avoid disturbing them, so as to provide more personalized and intelligent health care service support.

[0030] Figure 1 A schematic block diagram of an intelligent robot with scene perception capability according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of an intelligent robot with scene perception capability according to an embodiment of the present application. Figure 1 and Figure 2As shown, the intelligent robot 1 with scene perception capability includes: a posture monitoring video acquisition module 10, which is used to use the robot's camera to collect the posture monitoring video of the target elderly object; a pulse timing data acquisition module 20, which is used to receive the time queue of the pulse data of the target elderly object collected from the intelligent wearable device through the robot's communication module; a behavior pattern feature extraction module 30, which is used to extract features based on the elderly behavior pattern of the posture monitoring video to obtain the semantic coding features of the elderly behavior pattern; a pulse timing feature extraction module 40, which is used to perform pulse timing coding on the time queue of the pulse data to obtain the pulse timing semantic coding features; a scene multimodal joint perception module 50, which is used to input the semantic coding features of the elderly behavior pattern and the pulse timing semantic coding features into a cross-modal scene sensor to obtain a scene multimodal fine-grained joint perception representation; a scene type judgment and behavior adjustment module 60, which is used to determine the scene type label based on the scene multimodal fine-grained joint perception representation, and specify the robot's behavior strategy.

[0031] Exemplarily, in the posture monitoring video acquisition module 10, the posture monitoring video of the target elderly object is collected by the camera of the robot. It should be understood that the posture monitoring video of the target elderly object is collected by the camera of the robot, so as to realize the recognition and analysis of the behavior pattern of the elderly, so as to provide more personalized and intelligent health care service support. Specifically, through the video information captured by the camera, the intelligent robot can identify the behavior patterns of the elderly, such as walking, sitting, lying down and other actions. This information is very important for understanding the current needs of the elderly. For example, when it is detected that the elderly are resting, the robot will automatically adjust the behavior strategy, such as lowering the volume to avoid disturbing. Monitor abnormal situations, such as falls or long periods of inactivity, so as to take timely measures or sound alarms to ensure the safety of the elderly. Learn and adapt to the living habits of the elderly. Over time, the robot can accumulate data and understand the laws of the daily life of the elderly, so as to better predict their needs and provide corresponding help.

[0032] In one embodiment, the robot is equipped with a high-resolution camera to ensure that the various subtle movements of the elderly can be clearly captured. The camera may include a wide-angle lens to cover a larger observation range; it may also include a depth perception camera (such as an RGB-D camera) to obtain three-dimensional spatial information, so as to more accurately understand the distance and position relationship of objects. In order to fully capture the movements of the elderly, in other embodiments, multiple cameras may be set to shoot from different angles. This helps to overcome the problem of limited field of view of a single camera, especially in complex environments to ensure all-round behavior tracking of the elderly. Some advanced versions of the camera also have autofocus and electronic / optical zoom capabilities, which can adjust the focal length without changing the physical position, so that the robot can focus on a specific area or object and improve the accuracy of detail capture. Considering that the living environment of the elderly may be low in light, the camera should have good low-light performance and be able to maintain high image quality in dim conditions.

[0033] Once started, the camera will begin to continuously record the surrounding dynamics to form a stable video stream. These video streams will be transmitted to the robot's central processing unit or other dedicated hardware accelerator for preliminary processing. The received raw video frames will first undergo a series of preprocessing steps, such as denoising, color correction, etc., to improve image quality. In other examples of this application, image enhancement technology is also applied to highlight the information of key parts (such as human body contours) to facilitate subsequent feature extraction. In order to ensure the security of the data, appropriate encryption measures should be taken during the transmission of the video stream from the camera to the processing unit to prevent unauthorized access. At the same time, local storage and cloud backup are used to balance the immediate availability and long-term preservation requirements of the data. In this way, on the one hand, video clips within a certain period of time can be retained locally for immediate backtracking; on the other hand, the processed data can also be uploaded to the cloud platform for long-term archiving and further analysis.

[0034] Exemplarily, in the pulse timing data acquisition module 20, the time queue of the pulse data of the target elderly object collected by the smart wearable device is received through the communication module of the robot. It should be understood that relying solely on visual information (such as behavioral patterns captured by a camera) may not be enough to fully understand the state of the elderly. For example, the elderly appear to be resting, but their abnormally elevated heart rate may indicate anxiety or other health problems. By combining physiological signals such as heart rate from smart wearable devices, the robot can more accurately judge the current situation and make a more appropriate response. : Smart wearable devices can continuously collect vital sign data such as pulse, blood pressure, etc. These data are crucial for long-term tracking of the health of the elderly, and help to detect potential health risks early, such as the occurrence of major diseases such as heart attacks or strokes, so as to take preventive measures to reduce the chance of disease.

[0035] In one embodiment, the smart wearable device is usually connected to the robot through a short-range wireless communication protocol such as Bluetooth Low Energy (BLE), Wi-Fi, or Zigbee. These technologies have the characteristics of low latency and low energy consumption, and are suitable for continuous transmission of a small number of data packets for a long time, such as heart rate readings. To ensure compatibility and stability, robot manufacturers may develop a dedicated application programming interface (API) or software development kit (SDK) to simplify integration with different brands and models of smart wearable devices.

[0036] In one embodiment, the time queue of the pulse data of the target elderly object collected by the smart wearable device is received by the communication module of the robot, including: first, the robot searches for nearby smart wearable devices through its built-in communication module, and tries to pair with them to establish a secure connection. This process generally involves identity authentication to ensure that only authorized devices can send data to the robot. Once a stable connection is established, the smart wearable device will regularly push the latest pulse data to the robot. These data are usually organized into a time series format, that is, each data point is timestamped to form a so-called "time queue". This not only records the specific values, but also retains the time order in which they occur, which is convenient for subsequent analysis. The received data will be temporarily stored in the local memory of the robot and may be preliminarily processed (such as filtering, averaging, etc.) to remove noise interference. Afterwards, these data can be saved to a persistent storage medium for future reference, or directly sent to a cross-modal scene sensor for real-time analysis. All data transmitted from the smart wearable device to the robot should be encrypted to ensure that sensitive information will not be leaked even if it is intercepted during network transmission. The use of standard encryption protocols such as TLS / SSL can effectively ensure the security of data.

[0037] Exemplarily, in the behavior pattern feature extraction module 30, the posture monitoring video is subjected to feature extraction based on the behavior pattern of the elderly to obtain the semantic coding features of the behavior pattern of the elderly. It should be understood that in order to capture the behavior pattern semantics of the target elderly object presented in the posture monitoring video, so as to perform effective scene recognition to adjust the behavior strategy of the robot, in one embodiment of the technical solution of the present application, the behavior pattern feature extraction module is used to: input the posture monitoring video into the elderly behavior pattern feature extractor based on the CNN-RNN combination model to obtain the elderly behavior pattern semantic coding feature vector as the elderly behavior pattern semantic coding feature. It should be understood that the CNN-RNN combination model is to extract the features of the posture monitoring video through the combination of CNN and RNN, wherein CNN is particularly good at processing image data and can extract spatial features related to human body posture from continuous video frames. Through multi-layer convolution operations, it can capture local information at different scales, such as joint positions, directions of body parts, etc., which are the basis for judging the posture of the elderly (such as standing, sitting, lying down). At the same time, CNN can effectively reduce the redundant information in the video frame and retain the most important features, providing a more compact and meaningful representation for subsequent processing. RNN is very suitable for processing sequence data with time dependence. For behavioral patterns, action is not a static state presented in a video frame, but a process that changes over time. Therefore, using RNN can help us understand the evolution of behavior over a period of time, such as changing from "sitting" to "standing up", so as to more accurately and comprehensively capture the semantic characteristics of the behavior pattern of the elderly. Therefore, the elderly behavior pattern feature extractor based on the CNN-RNN combined model combines the spatial feature extraction capability of CNN and the time series modeling capability of RNN, and can extract high-quality spatial features on each frame, and on this basis, construct the temporal dynamic features of the entire action sequence. This enables the robot to more accurately identify complex continuous actions and posture time sequence semantics, such as walking, lying down, etc., rather than just a single posture, which helps the robot understand the actions and intentions of the elderly, and provides a basis for subsequent robot behavior strategy adjustments.

[0038] Exemplarily, in the pulse timing feature extraction module 40, the time queue of the pulse data is pulse-timing encoded to obtain pulse timing semantic coding features. In one embodiment, the pulse timing feature extraction module is used to: input the time queue of the pulse data into the pulse timing encoder based on the forward LSTM model to obtain the pulse timing semantic coding feature vector as the pulse timing semantic coding feature. It should be understood that the time queue of the pulse data is input into the pulse timing encoder based on the forward LSTM model for feature mining to extract the timing semantic feature information of the elderly's pulse data, thereby obtaining the pulse timing semantic coding feature vector. It is worth mentioning that the trained forward LSTM model can learn the semantic meaning behind the pulse timing data, for example, an accelerated heart rate may mean exercise or anxiety, while a slow heart rate may be a resting state. This semantic level encoding is very helpful for identifying the behavior patterns of the elderly and whether they are in a sleeping state.

[0039] Exemplarily, in the scene multimodal joint perception module 50, the semantic coding features of the elderly behavior pattern and the pulse timing semantic coding features are input into the cross-modal scene sensor to obtain the scene multimodal fine-grained joint perception representation. It should be understood that since the semantic coding feature vector of the elderly behavior pattern and the pulse timing semantic coding feature vector respectively contain the semantic features of the elderly behavior pattern and the timing semantic features of the physiological state, there is a complex relationship between the elderly behavior pattern (such as walking, sitting, lying down, etc.) and the physiological state (such as heart rate changes). For example, a sudden increase in heart rate may be because the elderly are exercising, or it may be due to anxiety or discomfort, and a slow heart rate may be because the elderly are resting or in a sick state. Therefore, in order to be able to capture these subtle relationships more accurately, so as to help the robot better understand the current situation to make more appropriate behavioral strategy adjustments, in the technical solution of the present application, the semantic coding features of the elderly behavior pattern and the pulse timing semantic coding features are further input into the cross-modal scene sensor for processing to obtain the scene multimodal fine-grained joint perception representation. The processing of the cross-modal scene sensor is to optimize the fine-grained interaction between the semantic features of the elderly's behavior patterns and the temporal semantic features of their physiological states by integrating external knowledge, thereby improving the performance of the model on specific tasks. When the robot can accurately identify the association between the elderly's behavior and physiological state, it can respond more appropriately. For example, when it is detected that the elderly's behavior pattern is lying down and the heart rate changes slowly, it may be that the elderly are resting. At this time, the intelligent robot will lower the volume to avoid disturbing them.

[0040] In one embodiment, Figure 3As shown, the scene multimodal joint perception module 50 includes: a feature interaction optimization unit 51, which is used to perform feature interaction optimization processing on the semantic coding features of the elderly behavior pattern and the semantic coding features of the pulse timing based on external knowledge modulation to obtain an external knowledge optimized scene multimodal fine-grained feature interaction representation; a fine-grained modulation optimization coding unit 52, which is used to optimize the scene multimodal fine-grained feature interaction representation based on the external knowledge optimization, and perform semantic association coding based on fine-grained modulation optimization on the semantic coding features of the elderly behavior pattern and the semantic coding features of the pulse timing to obtain the scene multimodal fine-grained joint perception representation.

[0041] In one embodiment, Figure 4 As shown, the feature interaction optimization unit 51 includes: a fine-grained feature interaction processing subunit 511, which is used to perform fine-grained feature interaction processing on the semantic encoding feature vector of the elderly behavior pattern and the semantic encoding feature vector of the pulse timing to obtain a scene multimodal fine-grained feature interaction matrix; an external knowledge optimization subunit 512, which is used to input the scene multimodal fine-grained feature interaction matrix into an external knowledge-based attention unit to obtain an external knowledge-optimized scene multimodal fine-grained feature interaction matrix as the external knowledge-optimized scene multimodal fine-grained feature interaction representation.

[0042] Exemplarily, in the fine-grained feature interaction processing subunit 511, the semantic coding feature vector of the elderly behavior pattern and the semantic coding feature vector of the pulse time sequence are subjected to fine-grained feature interaction processing to obtain a scene multimodal fine-grained feature interaction matrix. Specifically, the process can be expressed as follows:

[0043]

[0044] Among them, V1 and V2 are the semantic coding feature vector of the elderly behavior pattern and the semantic coding feature vector of the pulse time sequence, respectively. is matrix multiplication, (·) T is the matrix transpose, M p is the scene multimodal fine-grained feature interaction matrix.

[0045] By performing fine-grained feature interaction processing on the semantic encoding feature vectors of the elderly's behavior patterns and the pulse time series semantic encoding feature vectors to obtain the scene multimodal fine-grained feature interaction matrix, this process greatly enhances the depth and accuracy of the intelligent robot's understanding of the current scene. Specifically, it can capture the complex relationship between the elderly's behavior patterns and physiological states. For example, in daily life, changes in heart rate not only reflect the level of physical activity, but may also indicate mood swings or health problems. When the elderly's heart rate suddenly speeds up, this may be a normal reaction caused by exercise, or it may be a sign of anxiety, pain or other discomfort. Through fine-grained feature interaction processing, the robot can combine visually observed behaviors (such as whether strenuous activities are being carried out) and heart rate data from smart wearable devices to more accurately judge the elderly's true state. If an abnormal heart rate is detected and there is no corresponding physical activity as an explanation, the system may trigger further health check reminders or notify caregivers to pay attention. At the same time, fine-grained feature interaction processing can also improve the learning ability and generalization performance of the model. By deeply mining the associations between behavior patterns and physiological signals at different time points, the robot can not only better understand the current situation, but also accumulate experience and gradually optimize its prediction ability and decision-making logic. Over time, you will learn to recognize more types of combined patterns, allowing you to respond appropriately when faced with new or unseen situations.

[0046] Exemplarily, in the external knowledge optimization subunit 512, the scene multimodal fine-grained feature interaction matrix is ​​input into the attention unit based on external knowledge to obtain the external knowledge optimized scene multimodal fine-grained feature interaction matrix as the external knowledge optimized scene multimodal fine-grained feature interaction representation. Specifically, the process can be expressed as:

[0047]

[0048] Among them, M p is the scene multimodal fine-grained feature interaction matrix, M k and M v represents the first external knowledge attention learnable memory parameter matrix and the second external knowledge attention learnable memory parameter matrix, norm(·) represents the normalization function, M y Optimizing scene multimodal fine-grained feature interaction matrix for external knowledge.

[0049] It should be understood that in actual application scenarios, such as the field of health care services, intelligent robots must not only recognize the current actions of the elderly (such as walking, sitting, and lying down), but also understand the emotions or health conditions that may be hidden behind these behaviors. Simply relying on the behavioral patterns captured by the camera or the heart rate changes recorded by the smart wearable device often cannot fully and accurately reflect the real needs of the elderly. Therefore, introducing external knowledge (such as medical literature, expert advice, historical cases, etc.) and optimizing them through attention units can significantly improve the expressiveness and adaptability of the model. When the robot receives a scene multimodal fine-grained feature interaction matrix, this matrix contains the interaction information between the semantic encoding feature vectors of the behavioral patterns extracted from video surveillance and the semantic encoding feature vectors of the heart rate time series from the smart wearable device. However, this raw interaction information may not be enough to reveal all potential associations. By inputting this matrix into the attention unit based on external knowledge, the robot can use the pre-accumulated expertise to guide the analysis process and find those subtle but crucial connections. For example, certain specific heart rate change patterns are often accompanied by certain behaviors or emotional states (such as increased heart rate when anxious), and these patterns may be summarized from a large amount of clinical data. With the help of external knowledge, the robot can better identify these patterns and adjust its behavior strategies accordingly to provide more considerate services to the elderly. The attention unit allows the model to dynamically assign weights to different features according to task requirements, which means it can focus more on the information points that are most critical to the current situation.

[0050] In one embodiment, Figure 5 As shown, the fine-grained modulation optimization coding unit 52 includes: a fine-grained modulation optimization subunit 521 for optimizing the behavior pattern of the elderly, which is used to perform fine-grained modulation optimization on the semantic coding feature vector of the behavior pattern of the elderly based on the external knowledge optimization scene multimodal fine-grained feature interaction matrix to obtain an optimized semantic coding feature vector of the behavior pattern of the elderly; a pulse timing semantic fine-grained modulation optimization subunit 522 for performing fine-grained modulation optimization on the pulse timing semantic coding feature vector based on the external knowledge optimization scene multimodal fine-grained feature interaction matrix to obtain an optimized pulse timing semantic coding feature vector; a scene multimodal semantic association coding subunit 523 for performing semantic association coding on the optimized semantic coding feature vector of the behavior pattern of the elderly and the optimized semantic coding feature vector of the pulse timing to obtain a scene multimodal fine-grained joint perception coding vector as the scene multimodal fine-grained joint perception representation.

[0051] In one embodiment, the elderly behavior pattern fine-grained modulation optimization subunit 521 is used to: perform a linear transformation on the elderly behavior pattern semantic encoding feature vector to obtain the elderly behavior pattern semantic query feature vector and the elderly behavior pattern semantic value feature vector; use the external knowledge optimization scenario multimodal fine-grained feature interaction matrix as the key matrix, input the elderly behavior pattern semantic query feature vector, the elderly behavior pattern semantic value feature vector and the key matrix into the Transformer-based fine-grained modulation module to obtain the optimized elderly behavior pattern semantic encoding feature vector. Specifically, the process can be expressed as:

[0052]

[0053] Among them, V1 is the semantic encoding feature vector of the elderly behavior pattern, M y Optimize the scene multimodal fine-grained feature interaction matrix for external knowledge, W 1q and b 1q They represent the weight matrix and bias vector of semantic query of elderly behavior patterns, respectively. 1v and b 1v They represent the semantic value weight matrix of the elderly behavior pattern and the semantic value bias vector of the elderly behavior pattern, V 1q and V 1v are the semantic query feature vector of the elderly behavior pattern and the semantic value feature vector of the elderly behavior pattern, respectively, and d is M y The scale is the matrix width multiplied by the matrix height, softmax(·) represents the normalized exponential function, V ′1 The optimized semantic encoding feature vector of the elderly behavior pattern.

[0054] It should be understood that, firstly, the query feature vector (Query) and value feature vector (Value) of the elderly behavior pattern are obtained through linear transformation. These two vectors represent different aspects of the behavior pattern: the query feature vector is used to find correlations with other features, while the value feature vector carries specific information about the behavior. This separation helps to analyze the relationship between each behavior element more carefully, so that the robot can understand the elderly's behavioral background and intentions more deeply. For example, when the elderly's "lying down" action is detected, the query feature can help determine whether this action is accompanied by other related behaviors or physiological changes (such as slowing heart rate), while the value feature provides specific details about "lying down" itself (such as posture, duration, etc.). This lays a solid foundation for subsequent situational judgment. Secondly, the scene multimodal fine-grained feature interaction matrix optimized by external knowledge is introduced as the key matrix (Key), and it works together with the above query and value feature vectors in the fine-grained modulation module based on the Transformer structure. The key here is that external knowledge is not limited to static data sets, but also includes dynamically updated professional advice, medical literature, and historical cases. These rich external resources can guide the model to better understand the meaning behind specific behavior patterns. For example, when a sudden increase in the heart rate of an elderly person is detected, if combined with relevant knowledge of cardiology, the robot can more accurately assess the risk level of this situation and take appropriate measures accordingly. In this way, the robot is not just observing the surface behavior, but is able to understand its underlying meaning at a deeper level, thereby making more reasonable decisions. The fine-grained modulation module based on the Transformer structure allows the model to dynamically assign attention weights to different feature combinations according to task requirements. This means that it can focus on those information points that are most critical to the current situation judgment, while ignoring the less important parts.

[0055] In one embodiment, the pulse timing semantic fine-grained modulation optimization subunit 522 is used to: perform a linear transformation on the pulse timing semantic coding feature vector to obtain a pulse timing semantic query feature vector and a pulse timing semantic value feature vector; use the external knowledge optimization scenario multimodal fine-grained feature interaction matrix as a key matrix, input the pulse timing semantic query feature vector, the pulse timing semantic value feature vector and the key matrix into the Transformer-based fine-grained modulation module to obtain the optimized pulse timing semantic coding feature vector. Specifically, the process can be expressed as:

[0056]

[0057] Wherein, V2 is the pulse time sequence semantic coding feature vector, M y Optimize the scene multimodal fine-grained feature interaction matrix for external knowledge, W2q and b 2q They represent the pulse time series semantic query weight matrix and pulse time series semantic query bias vector, W 2v and b 2v Respectively represent the pulse time series semantic value weight matrix and pulse time series semantic value bias vector, V 2q and V 2v are the pulse time series semantic query feature vector and the pulse time series semantic value feature vector, V ′2 The optimized pulse timing semantic encoding feature vector.

[0058] It should be understood that the pulse time series semantic query feature vector (Query) and value feature vector (Value) are obtained through linear transformation. These two vectors represent different aspects of heart rate changes: the query feature vector is used to find correlations with other features, while the value feature carries the specific information of the time series. This separation helps to analyze the heart rate fluctuations at each time point and its relationship with other behavioral patterns or physiological indicators in more detail. For example, when the heart rate of the elderly is detected to be suddenly accelerated, the query feature can help determine whether this change is accompanied by other related behaviors or physiological phenomena (such as increased physical activity, emotional excitement, etc.), while the value feature provides specific details about the heart rate change (such as rate, duration, etc.). This lays a solid foundation for subsequent situational judgment. The scene multimodal fine-grained feature interaction matrix optimized by external knowledge is introduced as the key matrix (Key), and acts together with the above query and value feature vectors in the fine-grained modulation module based on the Transformer structure. The key here is that external knowledge is not limited to static data sets, but also includes dynamically updated professional advice, medical literature, and historical cases. These rich external resources can guide the model to better understand the meaning behind specific heart rate changes. For example, when an abnormally high heart rate is detected in an elderly person, if combined with relevant knowledge of cardiology, the robot can more accurately assess the risk level of this situation and take appropriate measures accordingly. In this way, the robot is not only monitoring changes in heart rate, but is able to understand its potential meaning at a deeper level, so as to make more reasonable decisions. The fine-grained modulation module based on the Transformer structure allows the model to dynamically assign attention weights to different feature combinations according to task requirements. This means that it can focus on those information points that are most critical to the current situation judgment, while ignoring the less important parts.

[0059] In one embodiment, the scene multimodal semantic association coding subunit 523 is used to calculate the location point division between the optimized semantic coding feature vector of the elderly behavior pattern and the optimized semantic coding feature vector of the pulse timing to obtain the scene multimodal fine-grained joint perception coding vector. Specifically, the process can be expressed as:

[0060]

[0061] Among them, V ′1 is the optimized semantic encoding feature vector of the elderly behavior pattern, V ′2 is the optimized pulse timing semantic coding feature vector, and V is the multimodal fine-grained joint perception coding vector of the scene.

[0062] It should be understood that point division by position is an effective mathematical operation method, which can reveal the relative relationship between two feature vectors at the same time point while maintaining the independence of each feature dimension. For the elderly, their behavior patterns (such as walking, sitting, lying down) and physiological states (such as heart rate changes) are mutually influential and closely related. For example, when the elderly are physically active, their heart rate usually increases accordingly; while when they are resting or sleeping, their heart rate tends to be stable. Through this point division by position operation, the robot can more accurately capture the dynamic relationship between these behaviors and physiological states, and then understand the specific situation of the elderly. This helps the robot identify those situations that appear normal on the surface but may actually pose health risks, such as abnormally high heart rate without obvious physical activity as an explanation, indicating that there may be anxiety or other potential problems. Combining the optimized semantic encoding feature vector of the elderly's behavior pattern and the optimized semantic encoding feature vector of the pulse time series, and generating a scene multimodal fine-grained joint perception encoding vector by point division by position, the robot can better consider multiple sources of information and improve its accuracy in judging complex situations. Specifically, this method can help the robot distinguish the subtle differences behind similar behavior patterns. For example, the same "sitting" posture, but with a higher heart rate, may mean that the elderly are nervous or uncomfortable, while a lower heart rate may indicate a relaxed or quiet state. In this way, the robot can make more accurate judgments in the face of complex daily life, adjust its behavior strategies accordingly, and provide more personalized service content.

[0063] Specifically, in the cross-modal scene sensor, the design of each step aims to improve the expressiveness of the model by deeply mining the fine-grained interactive relationship between the semantic features of the elderly's behavioral patterns and the temporal semantic features of the physiological state and combining external knowledge. This process not only strengthens the interaction between features, but also makes full use of external knowledge to guide model training, thereby significantly improving the performance of the model in dealing with complex scene recognition and behavioral decision-making tasks. It is worth mentioning that by using external knowledge (such as medical literature, expert advice, historical cases, etc.), the fine-grained feature interaction representation can be optimized through the attention mechanism, making the model's understanding and application of specific fields more accurate. For example, knowing what types of activities are usually accompanied by certain heart rate changes can help robots predict the needs of the elderly more accurately. This can not only improve the intelligent robot's ability to understand and respond to complex situations, but also provide more intimate and personalized health care services for each elderly person. This approach not only ensures the quality of life of the elderly, but also provides them with more security and personalized support.

[0064] Exemplarily, in the scene type judgment and behavior adjustment module 60, based on the multimodal fine-grained joint perception representation of the scene, the scene type label is determined, and the behavior strategy of the robot is specified. In one embodiment, the scene type judgment and behavior adjustment module is used to: input the multimodal fine-grained joint perception encoding vector of the scene into a classifier-based scene recognizer to obtain a scene recognition result, and the scene recognition result is used to represent the scene type label; based on the scene recognition result, the behavior strategy of the robot is specified. That is to say, the multimodal fine-grained joint perception feature information about the scene is used to perform scene recognition classification to obtain the scene type label. Then, based on the scene recognition result, the behavior strategy of the robot is specified. In this way, the different needs of the elderly in the scene can be predicted through the perception information of the intelligent robot on the scene, so as to customize personalized service content. For example, when it is detected that the elderly are resting, it will lower the volume to avoid disturbing them, so as to provide more personalized and intelligent health care service support.

[0065] In a specific embodiment of the present application, the classifier-based scene recognizer is a support vector machine (SVM), and the model is trained and verified before actual application. By collecting and labeling a large amount of sample data in different situations, it is ensured that various typical elderly life scenes are covered, such as resting, exercising, dining, watching TV, etc. Each sample should contain corresponding posture monitoring video and pulse data and corresponding scene type labels. Use these data sets to train the model, and continuously optimize the model parameters through cross-validation until satisfactory performance indicators such as accuracy and recall are achieved. After training, the performance of the model is evaluated on an independent test set to ensure that it has good generalization ability and can perform well on new data that has not been seen. Once the trained classifier is deployed to the robot system, it can receive data from sensors in real time, generate new scene multimodal fine-grained joint perception encoding vectors, and output scene recognition results based on this, thereby specifying the behavior strategy of the robot. For example, if it detects that the elderly person's heart rate is gradually slowing down and the body is in a still state, the classifier may determine it as a "rest" scene, and the robot will automatically lower the volume to avoid disturbing them. When the heart rate speeds up and there are obvious signs of physical activity, the classifier may identify it as an "exercise" scene, and the robot can remind the elderly to replenish water appropriately or suggest taking a break.

[0066] According to the scene type label output by the classifier, the robot can dynamically adjust its behavior strategy to adapt to the current situational needs. This adjustment is not only to follow the predefined rules, but also involves the application of adaptive learning mechanisms. The robot uses machine learning algorithms to continuously observe the reactions of the elderly and gradually optimize its own behavior. For example, if a certain behavior (such as playing a certain song) is found to be particularly popular with the elderly, the robot can repeat this behavior in similar situations to better meet the needs of the elderly. At the same time, no matter what the situation, the robot should always put the safety of the elderly first. For example, when abnormal heart rate changes or other potential danger signals are detected, the robot should immediately take measures to notify medical staff or initiate emergency plans.

[0067] Preferably, passing the scene multimodal fine-grained joint perceptual coding vector through a classifier-based scene recognizer to obtain a scene recognition result comprises:

[0068] The distance between each pair of eigenvalues ​​of the scene multimodal fine-grained joint perceptual coding vector is calculated, such as the L2 distance, and the square root of the distance is taken to obtain the scene multimodal fine-grained joint perceptual coding distance representation matrix, that is,

[0069]

[0070] Where V represents the multi-modal fine-grained joint perceptual encoding vector of the scene, vi and v j represents the i-th and j-th eigenvalues ​​of the multimodal fine-grained joint perceptual encoding vector of the scene, d(v i ,v j ) represents the distance between the i-th and j-th eigenvalues ​​of the multimodal fine-grained joint perceptual encoding vector of the scene, D i,j Represents the eigenvalue of the (i, j) position in the scene multimodal fine-grained joint perceptual encoding distance representation matrix;

[0071] The scene multimodal fine-grained joint perceptual coding autocorrelation matrix of the scene multimodal fine-grained joint perceptual coding vector is obtained, that is, Wherein, V represents the multimodal fine-grained joint perception encoding vector of the scene, T represents the transpose of the vector, represents matrix multiplication, D′ represents the scene multimodal fine-grained joint perception encoding autocorrelation matrix;

[0072] The scene multimodal fine-grained joint perceptual coding vector is matrix-multiplied with the scene multimodal fine-grained joint perceptual coding distance representation matrix to obtain a scene multimodal fine-grained joint perceptual coding primary mapping vector, that is, Wherein, V represents the scene multimodal fine-grained joint perception coding vector, D represents the scene multimodal fine-grained joint perception coding distance representation matrix, V ′ Represents the first-level mapping vector of the scene multi-modal fine-grained joint perception encoding;

[0073] The scene multimodal fine-grained joint perceptual coding first-level mapping vector is matrix-multiplied with the matrix product of the scene multimodal fine-grained joint perceptual coding distance representation matrix and the scene multimodal fine-grained joint perceptual coding autocorrelation matrix to obtain a scene multimodal fine-grained joint perceptual coding multi-level mapping vector Among them, V ′ represents the first-level mapping vector of the scene multimodal fine-grained joint perceptual coding, D represents the distance representation matrix of the scene multimodal fine-grained joint perceptual coding, D′ represents the autocorrelation matrix of the scene multimodal fine-grained joint perceptual coding, and V″ represents the multi-level mapping vector of the scene multimodal fine-grained joint perceptual coding;

[0074] The scene multimodal fine-grained joint perceptual coding multi-level mapping vector is interpolated with the scene multimodal fine-grained joint perceptual coding associated eigenvector (if the eigenvalue is insufficient, interpolation or zero padding) composed of the eigenvalue of the scene multimodal fine-grained joint perceptual coding autocorrelation matrix to obtain an optimized scene multimodal fine-grained joint perceptual coding vector;

[0075] The optimized scene multimodal fine-grained joint perceptual coding vector is passed through a classifier-based scene recognizer to obtain a scene recognition result.

[0076] Considering that the semantic coding feature vector of the elderly behavior pattern and the semantic coding feature vector of the pulse timing represent the semantic coding features of the elderly behavior pattern and the semantic association features of the elderly pulse timing respectively, when performing cross-modal scene perception based on external knowledge modulation, the difference in modulation modes of the external knowledge for the semantic coding feature vector of the elderly behavior pattern and the semantic coding feature vector of the pulse timing will cause the scene multimodal fine-grained joint perception coding vector obtained by fine-grained semantic interaction to have instance decision missing, thereby affecting the accuracy of the scene recognition result obtained by the classifier-based scene recognizer.

[0077] Therefore, through the linear target mapping representation based on the similarity distance representation matrix of the scene multimodal fine-grained joint perception coding vector, the self-correlation of the scene multimodal fine-grained joint perception coding vector is instantiated to perform a secondary target mapping representation based on a multi-level distribution hierarchy, and the association mismatch negative influence factor is compensated by the association fusion kernel bias to improve the instance determination degree of the eigenvalue of the scene multimodal fine-grained joint perception coding vector under the similarity restriction, that is, the significance of the eigenvalue as an instance for classification regression judgment, and improve the accuracy of the scene recognition result obtained by the scene multimodal fine-grained joint perception coding vector through the classifier-based scene recognizer. In this way, the different needs of the elderly in the scene can be predicted through the perception information of the intelligent robot on the scene, so as to customize personalized behavior strategies to provide more personalized and intelligent health care service support.

[0078] In summary, according to the embodiment of the present application, an intelligent robot with scene perception capability is explained, which uses the robot's camera to collect the position monitoring video of the target elderly object. At the same time, the robot's communication module receives the time queue of the pulse data of the target elderly object collected by the intelligent wearable device. Then, based on the perceived data, the intelligent robot can identify the current situation and adjust its behavior accordingly to select a more appropriate behavior strategy. In this way, the intelligent robot can predict the different needs of the elderly in the scene through the perception information of the scene, so as to customize personalized service content. For example, when it detects that the elderly are resting, it will lower the volume to avoid disturbing them, so as to provide more personalized and intelligent health care service support.

[0079] Figure 6 Schematic diagram of a control method for an intelligent robot with scene perception capability according to an embodiment of the present application. Figure 6As shown, the control method of the intelligent robot with scene perception capability includes: S1, using the robot's camera to collect the position monitoring video of the target elderly object; S2, receiving the time queue of the pulse data of the target elderly object collected by the intelligent wearable device through the communication module of the robot; S3, performing feature extraction based on the elderly behavior pattern of the position monitoring video to obtain the semantic coding feature of the elderly behavior pattern; S4, performing pulse timing coding on the time queue of the pulse data to obtain the pulse timing semantic coding feature; S5, inputting the semantic coding feature of the elderly behavior pattern and the pulse timing semantic coding feature into the cross-modal scene sensor to obtain the multi-modal fine-grained joint perception representation of the scene; S6, determining the scene type label based on the multi-modal fine-grained joint perception representation of the scene, and specifying the behavior strategy of the robot.

[0080] In one embodiment, the semantic coding features of the elderly behavior patterns and the semantic coding features of the pulse timing are input into a cross-modal scene sensor to obtain a multi-modal fine-grained joint perception representation of the scene, including: performing feature interaction optimization processing based on external knowledge modulation on the semantic coding features of the elderly behavior patterns and the semantic coding features of the pulse timing to obtain an external knowledge optimized scene multi-modal fine-grained feature interaction representation; based on the external knowledge optimized scene multi-modal fine-grained feature interaction representation, performing semantic association encoding based on fine-grained modulation optimization on the semantic coding features of the elderly behavior patterns and the semantic coding features of the pulse timing to obtain the multi-modal fine-grained joint perception representation of the scene.

[0081] Here, those skilled in the art will appreciate that the specific operations of each step in the control method of the intelligent robot with scene perception capability have been described in the above reference. Figures 1 to 5 It has been introduced in detail in the description of the intelligent robot with scene perception capability, and therefore, its repeated description will be omitted.

[0082] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0083] It should be understood that the specific examples in this article are only intended to help those skilled in the art better understand the embodiments of the present application, rather than to limit the scope of the embodiments of the present application.

[0084] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0085] It should also be understood that the various implementation methods described in this specification can be implemented individually or in combination, and the embodiments of the present application are not limited to this.

[0086] Unless otherwise stated, all technical and scientific terms used in the embodiments of the present application are the same as the meanings generally understood by those skilled in the art of the technical field of the present application. The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the scope of the present application. The term "and / or" used in the present application includes any and all combinations of one or more related listed items. The singular forms of "a kind of", "above" and "the" used in the embodiments of the present application and the appended claims are also intended to include majority forms, unless the context clearly indicates other meanings. In addition, the terms "first", "second" etc. are only used for description purposes and cannot be understood as indicating or suggesting relative importance.

[0087] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0090] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0091] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An intelligent robot with scene perception capability, characterized in that: include: A posture monitoring video acquisition module, which is used to use the robot's camera to collect posture monitoring videos of target elderly objects; A pulse time series data acquisition module, used for receiving the time queue of the pulse data of the target elderly object collected from the smart wearable device through the communication module of the robot; A behavior pattern feature extraction module is used to extract features based on the behavior pattern of the elderly from the posture monitoring video to obtain semantic coding features of the behavior pattern of the elderly; A pulse time series feature extraction module, used for performing pulse time series encoding on the time queue of the pulse data to obtain pulse time series semantic encoding features; A scene multimodal joint perception module, used for inputting the semantic coding features of the elderly behavior pattern and the semantic coding features of the pulse timing into a cross-modal scene sensor to obtain a scene multimodal fine-grained joint perception representation, wherein the scene multimodal joint perception module includes: a feature interaction optimization unit, used for performing feature interaction optimization processing based on external knowledge modulation on the semantic coding features of the elderly behavior pattern and the semantic coding features of the pulse timing to obtain an external knowledge optimized scene multimodal fine-grained feature interaction representation; a fine-grained modulation optimization coding unit, used for performing semantic association coding based on fine-grained modulation optimization on the semantic coding features of the elderly behavior pattern and the semantic coding features of the pulse timing to obtain the scene multimodal fine-grained joint perception representation based on the external knowledge optimized scene multimodal fine-grained feature interaction representation; The scene type judgment and behavior adjustment module is used to determine the scene type label and specify the behavior strategy of the robot based on the multimodal fine-grained joint perception representation of the scene.

2. The intelligent robot with scene perception capability according to claim 1, characterized in that: The behavior pattern feature extraction module is used to: input the posture monitoring video into the elderly behavior pattern feature extractor based on the CNN-RNN combination model to obtain the elderly behavior pattern semantic coding feature vector as the elderly behavior pattern semantic coding feature.

3. The intelligent robot with scene perception capability according to claim 2, characterized in that: The pulse timing feature extraction module is used to: input the time queue of the pulse data into a pulse timing encoder based on a forward LSTM model to obtain a pulse timing semantic coding feature vector as the pulse timing semantic coding feature.

4. The intelligent robot with scene perception capability according to claim 3, characterized in that: The feature interaction optimization unit comprises: A fine-grained feature interaction processing subunit, used for performing fine-grained feature interaction processing on the semantic coding feature vector of the elderly behavior pattern and the semantic coding feature vector of the pulse time sequence to obtain a scene multimodal fine-grained feature interaction matrix; The external knowledge optimization subunit is used to input the scene multimodal fine-grained feature interaction matrix into the external knowledge-based attention unit to obtain the external knowledge-optimized scene multimodal fine-grained feature interaction matrix as the external knowledge-optimized scene multimodal fine-grained feature interaction representation.

5. The intelligent robot with scene perception capability according to claim 4, characterized in that: The fine-grained modulation optimization coding unit includes: The fine-grained modulation optimization subunit of the elderly behavior pattern is used to perform fine-grained modulation optimization on the semantic encoding feature vector of the elderly behavior pattern based on the multimodal fine-grained feature interaction matrix of the external knowledge optimization scenario to obtain an optimized semantic encoding feature vector of the elderly behavior pattern; A pulse timing semantic fine-grained modulation optimization subunit is used to perform fine-grained modulation optimization on the pulse timing semantic coding feature vector based on the external knowledge optimization scenario multimodal fine-grained feature interaction matrix to obtain an optimized pulse timing semantic coding feature vector; The scene multimodal semantic association coding subunit is used to perform semantic association coding on the optimized semantic coding feature vector of the elderly behavior pattern and the optimized semantic coding feature vector of the pulse timing to obtain a scene multimodal fine-grained joint perception coding vector as the scene multimodal fine-grained joint perception representation.

6. The intelligent robot with scene perception capability according to claim 5, characterized in that: The fine-grained modulation optimization subunit of the elderly behavior pattern is used to: Performing a linear transformation on the semantic encoding feature vector of the elderly behavior pattern to obtain a semantic query feature vector of the elderly behavior pattern and a semantic value feature vector of the elderly behavior pattern; The external knowledge optimization scenario multimodal fine-grained feature interaction matrix is ​​used as a key matrix, and the elderly behavior pattern semantic query feature vector, the elderly behavior pattern semantic value feature vector and the key matrix are input into a fine-grained modulation module based on the Transformer structure to obtain the optimized elderly behavior pattern semantic encoding feature vector.

7. The intelligent robot with scene perception capability according to claim 6, characterized in that: The pulse timing semantic fine-grained modulation optimization subunit is used to: Performing a linear transformation on the pulse time series semantic coding feature vector to obtain a pulse time series semantic query feature vector and a pulse time series semantic value feature vector; The external knowledge optimization scenario multimodal fine-grained feature interaction matrix is ​​used as a key matrix, and the pulse timing semantic query feature vector, the pulse timing semantic value feature vector and the key matrix are input into the Transformer-based fine-grained modulation module to obtain the optimized pulse timing semantic encoding feature vector.

8. The intelligent robot with scene perception capability according to claim 7, characterized in that: The scene multimodal semantic association coding subunit is used to calculate the point-by-point division between the optimized semantic coding feature vector of the elderly behavior pattern and the optimized semantic coding feature vector of the pulse timing to obtain the scene multimodal fine-grained joint perception coding vector.

9. The intelligent robot with scene perception capability according to claim 8, characterized in that: The scenario type determination and behavior adjustment module is used to: Inputting the scene multimodal fine-grained joint perceptual coding vector into a classifier-based scene recognizer to obtain a scene recognition result, wherein the scene recognition result is used to represent a scene type label; Based on the scene recognition result, a behavior strategy of the robot is specified.

10. A control method for an intelligent robot with scene perception capability, characterized in that: include: Use the robot's camera to collect posture monitoring video of the target elderly object; Receiving, through the communication module of the robot, a time queue of the pulse data of the target elderly subject collected from the smart wearable device; Extracting features based on the behavior patterns of the elderly from the posture monitoring video to obtain semantic coding features of the behavior patterns of the elderly; Performing pulse timing coding on the time queue of the pulse data to obtain pulse timing semantic coding features; Inputting the semantic coding features of the elderly behavior pattern and the pulse timing semantic coding features into a cross-modal scene sensor to obtain a multi-modal fine-grained joint perception representation of the scene, including: performing feature interaction optimization processing based on external knowledge modulation on the semantic coding features of the elderly behavior pattern and the pulse timing semantic coding features to obtain an external knowledge optimized multi-modal fine-grained feature interaction representation of the scene; based on the external knowledge optimized multi-modal fine-grained feature interaction representation of the scene, performing semantic association coding based on fine-grained modulation optimization on the semantic coding features of the elderly behavior pattern and the pulse timing semantic coding features to obtain the multi-modal fine-grained joint perception representation of the scene; Based on the multimodal fine-grained joint perception representation of the scene, a scene type label is determined, and a behavior strategy of the robot is specified.

Images