Multi-modal fusion life perception method and system

Abstract

The invention discloses a multi-modal fusion life perception method and system, and the method comprises the steps: firstly, obtaining multi-modal sensor information through a plurality of different types of sensors, and enabling the sensors to be deployed according to the life habit information and life scene information of a user; then, generating a first sensing model and a second sensing model according to the deployment positions and types of the sensors; inputting the information of the multi-modal sensor into the first sensing model to obtain a preliminary feature and a preliminary prediction result; and inputting the information and the preliminary features into a second perception model to obtain a secondary prediction result. And then, according to the correlation degree of the preliminary prediction result and the secondary prediction result, generating early warning information. According to the embodiment of the invention, by fusing the information of the multi-mode sensor, the living state of the user can be comprehensively perceived and monitored, and by dual prediction, the behavior state of the user, particularly emergency situations such as falling down and the like, can be more accurately judged, so that an early warning mechanism is triggered in time.

Description

Technical Field

[0001] The embodiments of the present application relate to but are not limited to the field of home smart elderly care technology, and in particular to a multi-modal fusion life perception method and system. Background Art

[0002] The perception of the living conditions of elderly people living alone is one of the hot spots and difficulties that every child or the elderly group in society is concerned about. Due to the decline in the physical functions of the elderly, the possibility of falling is greatly increased, and the injuries that may result from falling, such as fractures, soft tissue contusions, and psychological trauma, cannot be ignored. If the user who falls cannot get timely help, his injuries may be further aggravated. In order to solve this problem, traditional monitoring methods often rely on a single sensor or device, but this method has great limitations. For example, it may not be able to fully capture the user's living conditions and behavioral characteristics, and may not be able to accurately judge sudden unexpected behaviors such as falls. Therefore, there is an urgent need for a more advanced, accurate and comprehensive method to perceive and monitor the user's living conditions. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiment of the present application provides a multi-modal fusion life perception method, which can fully perceive and monitor the user's life status and trigger an early warning mechanism in a timely manner.

[0005] In a first aspect, an embodiment of the present application provides a multimodal fusion life perception method, including:

[0006] Acquiring multimodal sensor information through multiple different types of sensors, wherein the sensors are deployed according to the user's living habit information and life scene information;

[0007] generating a first perception model and a second perception model according to the deployment location and type of the sensor, wherein the computational complexity of the first perception model is lower than the computational complexity of the second perception model;

[0008] Inputting the multimodal sensor information into the first perception model to obtain preliminary features and preliminary prediction results;

[0009] Inputting the multimodal sensor information and the preliminary features into the second perception model to obtain a secondary prediction result;

[0010] Generate warning information according to the correlation degree between the preliminary prediction result and the secondary prediction result.

[0011] In combination with the first aspect, in an embodiment of the present application, the multimodal sensor information includes life behavior status data and continuous life vital signs data, the life behavior status data is provided by a specified image, and the continuous life vital signs data is provided by a specified point cloud data.

[0012] In combination with the first aspect, in an embodiment of the present application, the multimodal sensor information is input into the first perception model to obtain preliminary features and preliminary prediction results, including: denoising and normalizing the life behavior state data and the continuous life vital signs data to obtain first multimodal behavior state data, and inputting the first multimodal behavior state data into the first perception model; extracting multimodal state features from the first multimodal behavior state data, and fusing the multimodal state features to obtain preliminary features; and obtaining preliminary prediction results based on the state feature representation information.

[0013] In combination with the first aspect, in an embodiment of the present application, the obtaining of multimodal sensor information includes: when the preliminary prediction result characterizes that the target user perceives normally, obtaining first life behavior status data of the target user; when the preliminary prediction result characterizes that the target user perceives abnormally, obtaining second life behavior status data of the target user, and generating preliminary warning information based on the second life behavior status data; the first life behavior status data is provided by a first image, the second life behavior status data is provided by a second image, and the resolution of the second image is greater than the resolution of the first image.

[0014] In combination with the first aspect, in one embodiment of the present application, the multimodal sensor information and the preliminary features are input into the second perception model to obtain a secondary prediction result, including: combining the life behavior state data and the continuous life vital signs data after denoising and normalization, and the preliminary features to form second multimodal behavior state data; inputting the second multimodal behavior state data into the second perception model for prediction to obtain a secondary prediction result.

[0015] In combination with the first aspect, in an embodiment of the present application, the warning information is generated according to the degree of correlation between the preliminary prediction result and the secondary prediction result, including: when the preliminary prediction result and the secondary prediction result both show abnormalities, the warning information is first abnormal warning information; when the secondary prediction result shows abnormality and the preliminary prediction result shows normal, the warning information is second abnormal warning information; when the secondary prediction result shows normal and the preliminary prediction result shows abnormality, the warning information is third abnormal warning information; the abnormality intensity of the first abnormal warning information is greater than the abnormality intensity of the second abnormal warning information, and the abnormality intensity of the second abnormal warning information is greater than the abnormality intensity of the third abnormal warning information.

[0016] In combination with the first aspect, in an embodiment of the present application, generating a first perception model and a second perception model according to the deployment position and type of the sensor includes: constructing an initial first perception model and an initial second perception model according to the deployment position and type of the sensor; performing model training on the initial first perception model to obtain a first perception model, and performing model training on the initial second perception model to obtain a second perception model.

[0017] In the second aspect, an embodiment of the present application provides a multimodal fusion life perception system, including a terminal, a server and multiple sensors, wherein the multiple sensors, the terminal and the server communicate with each other; the multiple sensors are deployed according to life habit information and life scene information, the terminal is used to obtain multimodal sensor information, construct an initial first perception model according to the deployment position and type of the sensor, perform model training on the initial first perception model to obtain a first perception model, input the multimodal sensor information into the first perception model to obtain preliminary features and preliminary prediction results; the server is used to obtain multimodal sensor information and the preliminary features, construct an initial second perception model according to the deployment position and type of the sensor, perform model training on the initial second perception model to obtain a second perception model, input the multimodal sensor information and the preliminary features into the second perception model to obtain a secondary prediction result; and generate warning information according to the degree of correlation between the preliminary prediction result and the secondary prediction result.

[0018] In combination with the second aspect, in an embodiment of the present application, the types of sensors include color cameras, depth cameras, millimeter wave radars, and infrared cameras.

[0019] In combination with the second aspect, in an embodiment of the present application, the system is provided with a communication module for sending information to a designated user according to the early warning information.

[0020] The embodiment of the present application provides a multimodal fusion life perception method. First, multimodal sensor information is obtained through multiple different types of sensors, wherein these sensors are deployed according to the user's living habits and life scene information. This customized configuration can collect the user's life status and behavior feature data in an all-round and multi-angle manner, effectively avoiding the monitoring blind spots caused by the viewing angle or function limitation of a single sensor, and ensuring the comprehensive perception of the user's life status. Then, according to the deployment location and type of the sensor, a first perception model and a second perception model are generated. The multimodal sensor information is input into the first perception model to obtain preliminary features and preliminary prediction results; and then this information and the preliminary features are input into the second perception model together to obtain the secondary prediction results. Subsequently, according to the degree of correlation between the preliminary prediction results and the secondary prediction results, an early warning information is generated. This double verification mechanism significantly improves the reliability of the early warning system. Due to the real-time nature of the multimodal sensor information, the early warning information can be quickly conveyed to the emergency contact to ensure that the user can get rescue in the first time. In contrast, the traditional single sensor method may lead to inaccurate recognition of sudden unexpected behaviors such as falls due to insufficient data or misjudgment. The embodiment of the present application, by fusing multimodal sensor information and combining preliminary features with secondary prediction results, can more accurately judge the user's behavioral status, especially emergency situations such as falls, thereby triggering an early warning mechanism in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a multimodal fusion life perception method provided in an embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of the basic framework of the multimodal fusion life perception method provided in the embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of a sensor arrangement scheme for a home sensing space provided by a specific example of the present application;

[0024] Figure 4 is a specific flow chart of step 130 provided in an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of a high-resolution enabling strategy for a color camera provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of a judgment and analysis mechanism for abnormal situations provided in an embodiment of the present application;

[0027] Figure 7 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] It should be noted that, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order from that in the flowchart. The terms "first", "second", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions that can be implemented in this application, so they have no technical substantive significance. Any modification of the structure, change of the proportional relationship or adjustment of the size, without affecting the effects that can be produced by this application and the purposes that can be achieved, should still fall within the scope of the technical content disclosed in this application. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of narration, and are not used to limit the scope of implementation of this application. The change or adjustment of the relative relationship should also be regarded as the scope of implementation of this application without substantial change of the technical content.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0031] The perception of the living conditions of single users is the focus and difficulty of their children and the elderly in society. The elderly spend most of their time at home, and in a single-living environment, once an accident occurs, they often find it difficult to get timely help, which has become a major problem that endangers the healthy life of the elderly. The various living conditions of users at home, such as walking, sitting, bathing, cooking, and going to the toilet, not only involve individual behavioral characteristics and physiological characteristics, but more importantly, it is necessary to be able to monitor individual sudden unexpected behaviors, especially falls. Falls are a common accident for the elderly and one of the most harmful accidents. Due to the decline in physical function of the elderly, the possibility of falls is greatly increased, and the injuries that may be caused by falls, such as fractures, soft tissue contusions, and psychological trauma, cannot be ignored. If the user who falls cannot get timely help, his injuries may be further aggravated. To solve this problem, traditional monitoring methods often rely on a single sensor or device, but this method has great limitations. For example, it may not be able to fully capture the user's living conditions and behavioral characteristics, and may not be able to accurately judge sudden unexpected behaviors such as falls. Therefore, a more advanced, accurate and comprehensive method is urgently needed to perceive and monitor the user's living conditions.

[0032] In view of this, the embodiment of the present application provides a multimodal fusion life perception method and life perception system. First, multimodal sensor information is obtained through multiple different types of sensors, wherein these sensors are deployed according to the user's living habits and life scene information. This customized configuration can collect the user's life status and behavior feature data in an all-round and multi-angle manner, effectively avoiding the monitoring blind spots caused by the viewing angle or function limitation of a single sensor, and ensuring the comprehensive perception of the user's life status. Then, according to the deployment location and type of the sensor, a first perception model and a second perception model are generated. The multimodal sensor information is input into the first perception model to obtain preliminary features and preliminary prediction results; and then this information and the preliminary features are input into the second perception model together to obtain the secondary prediction results. Subsequently, according to the degree of correlation between the preliminary prediction results and the secondary prediction results, an early warning message is generated. This double verification mechanism significantly improves the reliability of the early warning system. Due to the real-time nature of the multimodal sensor information, the early warning information can be quickly conveyed to the emergency contact to ensure that the user can get rescue in the first time. In contrast, the traditional single sensor method may lead to inaccurate recognition of sudden unexpected behaviors such as falls due to insufficient data or misjudgment. The embodiment of the present application, by fusing multimodal sensor information and combining preliminary features with secondary prediction results, can more accurately judge the user's behavioral status, especially emergency situations such as falls, thereby triggering an early warning mechanism in a timely manner.

[0033] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0034] Reference Figure 1 , Figure 1 1 is a flow chart of a multimodal fusion life perception method provided in an embodiment of the present application. The method includes but is not limited to steps 110 to 150.

[0035] Step 110: acquiring multimodal sensor information through a plurality of different types of sensors, wherein the sensors are deployed according to the user's living habit information and life scene information;

[0036] Step 120: Generate a first perception model and a second perception model according to the deployment location and type of the sensor;

[0037] Step 130: Input the multimodal sensor information into the first perception model to obtain preliminary features and preliminary prediction results;

[0038] Step 140: input the multimodal sensor information and the preliminary features into a second perception model to obtain a secondary prediction result;

[0039] Step 150: Generate warning information based on the degree of correlation between the preliminary prediction result and the secondary prediction result.

[0040] Steps 110 to 150 are described in detail below.

[0041] In a feasible embodiment, usually, the user's life status perception information covers a wide range of behavioral characteristics and physiological characteristics. Behavioral characteristics mainly include activities in the home, such as walking, sitting, lying, personal hygiene habits (bathing), cooking and going to the toilet, and other daily behaviors; while physiological characteristics mainly include core physiological indicators such as body temperature, respiratory rate, heart rate rhythm, sleep quality and physical activity.

[0042] In a feasible embodiment, in order to accurately and efficiently obtain the user's life state perception information, a variety of sensors can be flexibly deployed according to the user's life habit information and life scene information, including but not limited to color cameras, depth cameras, millimeter wave radar devices, and infrared cameras. Specifically, for the user's dynamic gait characteristics (for example, walking mode in the residence) and its static posture characteristics (such as posture when sitting or lying), a color camera can be deployed to periodically capture images to record and analyze these characteristics in detail. At the same time, the addition of a depth camera can further enrich the information dimension of the color image, and more accurately evaluate the gait parameters and static posture characteristics of the elderly by providing depth data. The millimeter wave radar device, as an advanced detection device operating in the 30 to 300 GHz frequency band, can accurately measure the distance, speed and direction of the target object and other key information by transmitting and receiving millimeter wave signals, and its high-resolution characteristics enable it to identify small targets. More importantly, since human life activities (such as movement, breathing, heartbeat) will cause changes in the reflected echo sequence, the millimeter wave radar can accurately capture and analyze these life parameters. Combined with advanced point cloud human imaging algorithms, millimeter wave radar can achieve comprehensive coverage of breathing, heartbeat, sleep status, body activity and multi-target monitoring. Infrared cameras can capture infrared radiation that is invisible to the human eye. It is understandable that any object above thermodynamic zero will radiate infrared rays. When the ambient temperature is lower than the normal human body temperature (about 37 degrees Celsius), the human skin will release obvious infrared radiation energy, and its intensity and distribution directly reflect the temperature state of the object. When abnormal lesions occur in a part of the human body, it is often accompanied by subtle changes in the temperature field. By capturing these changes, infrared cameras can quickly and clearly reveal the temperature differences caused by different causes, providing important clues for health monitoring.

[0043] In a feasible embodiment, multimodal sensor information (i.e., life status perception information) is comprehensively acquired through multiple types of sensors deployed at multiple locations, covering the user's life behavior status data and continuous life sign data. The life behavior status data records the user's dynamic gait characteristics and static posture characteristics. This information is mainly provided by specific image sequences captured by color cameras and depth cameras, which can intuitively show the user's activity patterns and body changes in daily life. At the same time, the continuous life sign data records the user's key physiological indicators such as movement, breathing, and heartbeat. These data can be obtained through the precise point cloud data of the millimeter wave radar device.

[0044] In a feasible embodiment, both the first perception model and the second perception model can be network models obtained based on the construction of a convolutional neural network. Specifically, the first perception model is a lightweight model, which is more streamlined in computational complexity than the second perception model (i.e., a large multimodal model), which means that it requires fewer computing resources and has a faster processing speed. In practical applications, the first perception model can be deployed on a local terminal, which can quickly respond to and process information from multimodal sensors, make preliminary and rapid judgments, and give concise decision analysis. The second perception model, due to its large computational amount and stronger analytical ability, can be deployed on a server. It can further conduct in-depth analysis and confirmation based on the preliminary judgment of the first perception model, or supplement and improve the perception information. Such a deployment strategy not only optimizes the allocation of computing resources, but also ensures the efficiency and accuracy of data processing, providing users with a more intelligent and accurate home life experience.

[0045] In a feasible embodiment, the first perception model and the second perception model can be constructed according to the deployment location and type of the sensor. The specific process is as follows: First, according to the deployment location and type of the sensor, the initial first perception model (i.e., lightweight model) and the second perception model (i.e., large model) are constructed. Both models have the ability to quickly respond to and process information from multimodal sensors, and can provide corresponding prediction results. It is worth mentioning that the initial second perception model can also integrate the preliminary features from the initial first perception model to give a more accurate prediction. Subsequently, the initial first perception model is deeply trained using the relevant training data set, and the model parameters are optimized with the help of an algorithm to ensure that it can accurately extract valuable information from the sensor data and give corresponding perception results. After this series of training, the optimized first perception model can be obtained. Similarly, the initial second perception model is also comprehensively trained to obtain a second perception model with high performance. Through these two trained perception models, accurate processing and analysis of different sensor data can be achieved, providing users with more intelligent and accurate perception services.

[0046] The following is an explanation of the process of generating the first perception model and the second perception model according to the deployment location and type of the sensor using a specific example.

[0047] Assume that in a smart home environment, a color camera and an infrared camera are installed in the living room. The color camera is used to capture the user's facial features and activity trajectory, while the infrared camera is used to monitor indoor temperature changes and human thermal radiation to detect whether abnormal body temperature occurs. A depth camera and millimeter-wave radar are deployed in the bedroom. The depth camera is used to accurately measure the spatial layout and furniture position in the bedroom, while the millimeter-wave radar can monitor the vital signs of the person in the bed, such as breathing and heartbeat. For the color camera in the living room, a lightweight model (i.e., the first perception model) can be generated, which can quickly process the low-resolution and high-resolution RGB images of the color camera, identify the facial features of the user in the area, and preliminarily judge their activity status (such as walking, still, etc.). For the infrared camera, the lightweight model can analyze the temperature data and preliminarily judge whether abnormal body temperature occurs. On the server, a large model (i.e., the second perception model) is generated for the color camera and infrared camera in the living room. This model can receive the preliminary features transmitted by the lightweight model and combine it with its own high-resolution images and temperature data for more in-depth analysis. For example, it can more accurately identify the identity of users in the area, analyze their behavior patterns, and accurately detect the specific location and degree of abnormal body temperature. For the depth camera and millimeter-wave radar in the bedroom, corresponding lightweight models and large models can also be generated respectively. The lightweight model can quickly process the vital signs data of the depth image and the millimeter-wave radar to perform preliminary anomaly detection; while the large model can combine preliminary features, vital signs data and depth images for more in-depth sleep monitoring and anomaly detection. In this way, according to the deployment location and type of the sensor, a perception model suitable for different scenarios and needs can be generated to achieve perception and monitoring of the user's living status.

[0048] See also Figure 2 , Figure 2 It is a basic framework diagram of the multimodal fusion life perception method provided by the embodiment of the present application. Under this framework, a variety of sensors are first deployed in a living space, including color cameras, depth cameras, millimeter wave radars and infrared cameras. The multimodal sensor information captured and provided by these sensors specifically covers color images, depth images, point cloud data and infrared images. Subsequently, the multimodal information is deeply processed and analyzed using convolutional neural networks (first perception model and second perception model). Each network makes an independent decision output for its corresponding modal data. On this basis, the decision results from different modalities are further integrated, and through comprehensive judgment, information about the recognition of the elderly's physical state is finally output. This method not only makes full use of the complementary advantages of multimodal data, but also through the powerful processing power of convolutional neural networks, it realizes the accurate extraction and output of life status perception information, and provides strong support for subsequent monitoring, early warning and response.

[0049] In a feasible embodiment, based on the user's living habits information and specific life scene characteristics, the deployment location and type selection of sensors can reflect differentiated strategies and priorities. According to the specific needs of different spatial areas in the home environment, the configuration of sensors can be flexibly adjusted to ensure that each area can obtain the most appropriate monitoring method, thereby providing users with more accurate and comprehensive home life status perception services. Figure 3 As shown in the figure, taking the elderly's home life scene as an example, in the living room area, color cameras, millimeter-wave radars and depth cameras can be deployed to capture and record the elderly's activities and action details in all directions; in the bedroom, infrared cameras and millimeter-wave radars are configured to effectively monitor the elderly's sleep status and physiological characteristics even at night to ensure the continuity and accuracy of monitoring; in the kitchen, the combination of infrared cameras and color cameras can more accurately identify the elderly's cooking behavior and food processing process, and improve the meticulousness of monitoring; as for the toilet space, considering privacy protection and practicality, millimeter-wave radar can be used for monitoring. For the rich multimodal sensor information obtained in each space, convolutional neural networks (first perception model and second perception model) are further used for in-depth processing and analysis. When multiple sensors are deployed in a certain area, the decision results from different modes can be integrated, and the physical status information of the elderly in the area can be accurately output through comprehensive judgment. Finally, the physical status information of the elderly generated in all areas will be integrated to form a complete record of home physical status indicators, and abnormal detection and notification will be performed based on these indicator records to ensure that the elderly can receive timely and effective care.

[0050] In a feasible embodiment, the first perception model is a lightweight model that can perform a series of operations after receiving information from the multimodal sensor, including but not limited to data preprocessing, modal feature extraction or preliminary recognition analysis, so as to quickly give simple conclusions such as decision analysis or abnormal alarm. Figure 4 As shown, step 130 involves a process of inputting multimodal sensor information into a first perception model to obtain preliminary features and preliminary prediction results, including but not limited to steps 410 to 430.

[0051] Step 410: De-noising and normalizing the life behavior state data and the continuous life vital sign data to obtain first multimodal behavior state data, and inputting the first multimodal behavior state data into a first perception model;

[0052] Step 420: extracting multimodal state features from the first multimodal behavior state data, and fusing the multimodal state features to obtain preliminary features;

[0053] Step 430: Obtain preliminary prediction results based on preliminary feature representation information.

[0054] In a feasible embodiment, in step 410, the purpose of denoising is to remove outliers or noise signals in the data caused by factors such as sensor noise and environmental interference to improve the accuracy and reliability of the data. Normalization aims to convert data of different dimensions and scales to the same scale range for model processing and comparison. Normalization helps to speed up the training of the model and improve the predictive performance of the model. The first multimodal behavior state data is obtained, which has been pre-processed (denoising and normalization processing, etc.) and is suitable as the input of the first perception model.

[0055] In a feasible embodiment, in step 420, key information or features are extracted from the data of each modality using machine learning or deep learning algorithms. These features can reflect the user's life behavior status and physical sign information. The extracted multimodal features are fused to form more comprehensive and rich preliminary features. Among them, the fusion strategy may include weighted average, maximum value selection, feature splicing or attention mechanism in deep learning, etc.

[0056] In a feasible embodiment, in step 430, a preliminary prediction result can be derived based on the preliminary feature representation information. For example, if the preliminary feature clearly shows that the user is currently in a falling posture, then the preliminary prediction result will be determined as an abnormal state. This prediction result is based on the in-depth analysis and accurate identification of the user's behavior state data, and is intended to provide users with timely safety warnings or take corresponding assistance measures.

[0057] In a feasible embodiment, through the processing of step 410 to step 430, the first perception model can receive multimodal sensor information, perform data preprocessing, feature extraction and fusion, and finally obtain preliminary features and preliminary prediction results. These results can provide strong support for subsequent decision analysis, abnormal alarm or further processing. At the same time, since the first perception model usually adopts a lightweight design, it can reduce computational complexity and resource consumption while ensuring performance.

[0058] In a feasible embodiment, in order to fully ensure the security of user privacy and the effective recording of emergency events, the embodiment of the present application also provides a high-resolution activation strategy for a color camera, such as Figure 5As shown in the figure, in normal mode, when the preliminary prediction results show a normal state, indicating that the target user is in a normal activity state, the color camera will use image downsampling technology to generate low-resolution images for basic body feature recognition, including preliminary recognition of dynamic gait and static posture. This approach ensures the appropriate display of information while maintaining user privacy, which is convenient for intuitive display on the terminal device or sending status notifications to the associated children, so that they can understand the user profile at any time. However, once the preliminary prediction results indicate that the target user may be in an abnormal state, such as encountering an illegal intrusion by a stranger, or an abnormal displacement of an object when no one is present, the color camera will immediately switch to a high-resolution image data stream mode. This switch is designed to accurately capture and display the details of the abnormal event, providing a crucial visual basis for subsequent in-depth analysis and confirmation of the nature of the event. Then, the alarm mechanism is synchronously activated on multiple terminals (such as smart devices such as mobile phones) for abnormal notification, and the captured high-resolution images are securely stored in the local device or cloud server to prepare for possible subsequent investigations and evidence preservation. For this scenario, the specific collection process of multimodal sensor information in step 110 covers at least the following aspects: first, based on the preliminary prediction of the first perception model, if the target user is in a normal state, a first data set reflecting his or her life behavior state is collected and processed, and the data set mainly comes from the first image after resolution reduction processing; on the contrary, if the preliminary prediction shows that the target user may be abnormal, a more detailed second data set is collected and processed to generate preliminary warning information. This data set comes from a high-resolution second image, and its resolution is significantly higher than that of the first image, so as to provide more delicate and accurate visual details.

[0059] In a feasible embodiment, the specific steps of inputting multimodal sensor information and preliminary features into the second perception model to obtain more accurate secondary prediction results include: first, denoising and normalizing the life behavior state data and continuous life vital signs data to improve the quality and consistency of the data; then, integrating these processed data with the preliminary features to form the second multimodal behavior state data. This data set not only covers the details of the user's life behavior, but also incorporates the preliminary feature analysis results. Next, the second multimodal behavior state data is input into the second perception model. Based on advanced algorithms and deep learning techniques, the model can comprehensively analyze multiple data types to more accurately predict the user's status. It is worth noting that the secondary prediction results are often more accurate and reliable than the preliminary prediction results, providing strong support for real-time monitoring and early warning of user status.

[0060] In a feasible embodiment, in the process of generating warning information according to the degree of association between the preliminary prediction result and the secondary prediction result, when both the preliminary prediction result and the secondary prediction result show abnormality, the warning information is the first abnormal warning information; when the secondary prediction result shows abnormality and the preliminary prediction result shows normality, the warning information is the second abnormal warning information; when the secondary prediction result shows normality and the preliminary prediction result shows abnormality, the warning information is the third abnormal warning information. Among them, the abnormal intensity of the first abnormal warning information is greater than the abnormal intensity of the second abnormal warning information, and the abnormal intensity of the second abnormal warning information is greater than the abnormal intensity of the third abnormal warning information. Further, strong abnormal, abnormal and weak abnormal processing strategies can be formulated according to the warning information to ensure that the target population associated with the target user, such as children, relatives and friends, can quickly and accurately know the abnormal situation that the target user may have. Specifically, when the prediction results of the two perception models both show abnormality, it is determined to be a strong abnormality, and an emergency notification can be immediately sent to the target population online. If only the secondary prediction shows abnormality and the preliminary prediction is normal, it is determined to be a general abnormality, and a text message notification can be sent first, and if it is not confirmed within the specified time, a reminder will be given again. On the contrary, if the secondary prediction is normal but the initial prediction is abnormal, it is judged as a weak abnormality and only one SMS notification can be sent to reduce unnecessary disturbances.

[0061] In a feasible embodiment, in the collaborative strategy of the first perception model and the second perception model, the judgment and analysis mechanism for abnormal situations is as follows: Figure 6As shown in the figure, the first perception model can efficiently process multimodal sensor information (such as images from color cameras, point clouds from millimeter-wave radars, etc.), perform data preprocessing, modal feature extraction and preliminary identification and analysis tasks, and quickly give preliminary prediction results (such as decision analysis or abnormal alarm). However, its preliminary prediction results still need to be combined with the in-depth analysis results of the second perception model to confirm or supplement the perception analysis to ensure the accuracy and completeness of the prediction results. Specifically, when both the first perception model and the second perception model detect anomalies, the system immediately determines it as a strong anomaly and immediately notifies the user's children, relatives and friends and other key contacts through the background online method. If the second perception model determines it to be abnormal and the first perception model determines it to be normal, the system will regard it as a general abnormality, send a text message to notify the relevant contacts through the background, and require confirmation within the specified time. If the confirmation is not received within the specified time, the system will send a text message reminder again to ensure that the abnormal situation receives timely attention. On the contrary, if the second perception model determines it to be normal and the first perception model determines it to be abnormal, the system will regard it as a weak abnormality and only send a text message once through the background to notify the relevant contacts. This hierarchical response strategy not only ensures timely notification of abnormal situations, but also avoids unnecessary interference to users due to frequent false alarms. In addition, when both the first perception model and the second perception model are judged to be normal, it means that the user is in a normal and healthy state, and the system does not need to make any notification operations. Through this collaborative strategy and intelligent judgment and analysis mechanism, the system can provide users with more accurate and efficient life perception services, ensuring that users can receive timely and effective attention and help when encountering emergencies.

[0062] The following is an application scenario to illustrate the multimodal fusion life perception method of the present application.

[0063] In the application scenario of elderly health monitoring in smart homes, various types of sensors are deployed to monitor the health status and living habits of the elderly at home. These sensors include: Color camera: used to capture the daily activities of the elderly, such as walking, sitting, lying down, and facial expression changes. Depth camera: Generates a three-dimensional image by measuring the distance between the object and the camera, which helps to more accurately identify the elderly's movements and postures. Millimeter wave radar device: Through the human body imaging algorithm of the millimeter wave radar point cloud, it can realize the monitoring of breathing, heartbeat, sleep, movement and multiple targets. Infrared camera: Using infrared radiation imaging, it can capture the body temperature distribution and activity of the elderly in a completely dark environment, which helps to discover potential fall risks. These sensors are carefully deployed according to the elderly's living habits (such as activity time, activity area) and living scenes (such as bedroom, living room, kitchen) to ensure that the elderly's life information can be collected comprehensively and accurately. Based on the deployment location and type of sensors, machine learning technology can be used to generate two perception models: the first perception model and the second perception model. The first perception model mainly focuses on extracting preliminary features and preliminary prediction results from the information obtained from multiple types of sensors, such as basic action recognition analysis of the elderly. The second perception model combines the information of multiple types of sensors and the preliminary features of the first perception model to perform deeper fusion and prediction. Specifically, after the multimodal sensor information is input into the first perception model, the model can automatically identify preliminary features such as the dynamic gait and static posture of the elderly, and make preliminary predictions based on these features, such as judging whether the elderly are in a normal state of activity. Then, the multimodal sensor information and preliminary features are input into the second perception model for secondary prediction. The second perception model can comprehensively consider information from more dimensions, such as the movement trajectory of the elderly, changes in body temperature, etc., so as to obtain more accurate and comprehensive prediction results, such as judging whether the elderly have problems such as falling risk or poor sleep quality. Finally, according to the degree of correlation between the preliminary prediction results and the secondary prediction results, corresponding warning information can be generated. For example, if the prediction results of both models show that the elderly are at risk of falling, a strong abnormal warning will be triggered immediately, and an emergency notification will be sent to children or relatives and friends through terminal programs such as mobile phone APP. If only the second perception model predicts an abnormality and the first perception model does not find an abnormality, an abnormal warning will be sent and confirmation will be required within a specified time. If the prediction results of the two models are inconsistent, a weak abnormal warning will be sent for further observation and monitoring. Through such a multimodal fusion life perception method, the health status of the elderly at home can be monitored more comprehensively and accurately, providing strong protection for their safety.

[0064] In addition, the embodiment of the present application also provides a multimodal fusion life perception system, which integrates a terminal, a server and a variety of sensors, and realizes seamless communication among the three. According to the user's living habit information and life scene information, after arranging multiple types of sensors in each independent scene at home, according to the type and deployment location of the multiple types of sensors, the corresponding first perception model and the second perception model can be constructed, wherein the first perception model is deployed in the terminal and the second perception model is deployed in the server. It should be noted that both perception models can be constructed based on advanced neural network architecture. Specifically, the first perception model adopts a lightweight design idea to maximize the computing efficiency; and the second perception model, as a multimodal large model, has a more powerful analysis ability. After the two models are trained and adjusted respectively, a model with better performance can be obtained. In actual operation, the terminal will first collect multimodal information from various sensors. Subsequently, the first perception model can quickly and accurately parse these multimodal information, extract key features, and give preliminary prediction results based on this. At the same time, the server is also synchronously receiving these multimodal information. The second perception model can not only receive the preliminary feature information from the first perception model, but also integrate the multimodal information it has obtained to conduct a more detailed and in-depth secondary prediction analysis, thereby further improving the accuracy of the prediction results. Through the system's built-in communication module, relevant notifications can be accurately pushed to designated users. In this process, the system can intelligently generate early warning information based on the degree of correlation between the preliminary prediction results and the secondary prediction results. This mechanism enables the system to issue an alarm in a timely manner when the user's living status is abnormal, ensuring the safety and health of the user.

[0065] In a feasible embodiment, the types of sensors used include color cameras, depth cameras, millimeter wave radars and infrared cameras to build a comprehensive life perception system. Among them, the color camera is mainly responsible for capturing the user's gait characteristics in low-resolution mode (i.e., the single-frame image resolution does not exceed 480*640, with RGB three-color channels), which covers the posture and behavior pattern when walking. In addition, it will periodically record the user's static characteristics, especially the posture when falling. Once the system detects that a stranger breaks in or an abnormal situation occurs, the color camera will immediately switch to high-resolution mode (HD or UHD) to clearly capture and record the illegal behavior of strangers. The depth camera complements the color camera. It focuses on obtaining depth images and can accurately measure the distance from objects in the real world to the camera, thereby providing real physical size information. This function not only enriches the depth information missing from the color camera, but also further enhances the detection accuracy of the user's gait characteristics and static posture characteristics. The millimeter wave radar uses the point cloud human imaging algorithm to achieve accurate capture of breathing, heartbeat, sleep status, movement and multi-target monitoring. Infrared cameras can promptly detect and clearly display temperature changes caused by various reasons in the human body. It is worth emphasizing that the modal information collected by these sensors can be automatically uploaded to the server and terminal according to the preset time period (for example, every 10 seconds, 20 seconds, etc.) for real-time online perception and analysis. This mechanism ensures that the system can monitor the user's living conditions around the clock and without blind spots, and when any sudden unexpected situation is detected (such as falls, abnormal body temperature, abnormal heart rate or breathing during sleep, etc.), the user's children, relatives, friends and other key contacts are immediately notified through calls, text messages and other communication methods so that they can take action as soon as possible to ensure the user's safety and health.

[0066] It should be noted that since the multimodal fusion life perception system of this embodiment can realize the multimodal fusion life perception method of the previous embodiment, the multimodal fusion life perception system of this embodiment and the multimodal fusion life perception method of the previous embodiment have the same technical principles and the same beneficial effects. In order to avoid repetition, they will not be repeated here.

[0067] Reference Figure 7 The embodiment of the present application further discloses an electronic device, the electronic device 700 comprising:

[0068] at least one processor 701;

[0069] At least one memory 702, used to store at least one program;

[0070] When at least one program is executed by at least one processor 701, the multimodal fusion life perception method as described above is implemented.

[0071] An embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program executable by a processor. When the computer program executable by the processor is executed by the processor, it is used to implement the multimodal fusion life perception method as described above.

[0072] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-modal fusion life perception method, characterized in that: include: Acquiring multimodal sensor information through multiple different types of sensors, wherein the sensors are deployed according to the user's living habit information and life scene information; generating a first perception model and a second perception model according to the deployment location and type of the sensor, wherein the computational complexity of the first perception model is lower than the computational complexity of the second perception model; Inputting the multimodal sensor information into the first perception model to obtain preliminary features and preliminary prediction results; Inputting the multimodal sensor information and the preliminary features into the second perception model to obtain a secondary prediction result; Generate warning information according to the correlation degree between the preliminary prediction result and the secondary prediction result.

2. According to the multimodal fusion life perception method of claim 1, it is characterized in that: The multimodal sensor information includes life behavior status data and continuous life sign data, wherein the life behavior status data is provided by a specified image, and the continuous life sign data is provided by a specified point cloud data.

3. The multimodal fusion life perception method according to claim 2 is characterized in that: The step of inputting the multimodal sensor information into the first perception model to obtain preliminary features and preliminary prediction results includes: denoising and normalizing the life behavior state data and the continuous life vital signs data to obtain first multimodal behavior state data, and inputting the first multimodal behavior state data into the first perception model; extracting multimodal state features from the first multimodal behavior state data, and fusing the multimodal state features to obtain preliminary features; and obtaining preliminary prediction results based on the state feature representation information.

4. The multimodal fusion life perception method according to claim 3 is characterized in that: The obtaining of multimodal sensor information includes: when the preliminary prediction result indicates that the target user's perception is normal, obtaining first life behavior status data of the target user; when the preliminary prediction result indicates that the target user's perception is abnormal, obtaining second life behavior status data of the target user, and generating preliminary warning information based on the second life behavior status data; the first life behavior status data is provided by a first image, and the second life behavior status data is provided by a second image, and the resolution of the second image is greater than the resolution of the first image.

5. The multimodal fusion life perception method according to claim 2 is characterized in that: The step of inputting the multimodal sensor information and the preliminary features into the second perception model to obtain a secondary prediction result includes: combining the life behavior state data and the continuous life vital signs data after denoising and normalization, and the preliminary features to form second multimodal behavior state data; and inputting the second multimodal behavior state data into the second perception model for prediction to obtain a secondary prediction result.

6. The multimodal fusion life perception method according to claim 1, characterized in that: The generating of warning information according to the degree of correlation between the preliminary prediction result and the secondary prediction result includes: when the preliminary prediction result and the secondary prediction result both show abnormalities, the warning information is first abnormal warning information; when the secondary prediction result shows abnormality and the preliminary prediction result shows normality, the warning information is second abnormal warning information; when the secondary prediction result shows normality and the preliminary prediction result shows abnormality, the warning information is third abnormal warning information; the abnormality intensity of the first abnormal warning information is greater than the abnormality intensity of the second abnormal warning information, and the abnormality intensity of the second abnormal warning information is greater than the abnormality intensity of the third abnormal warning information.

7. The multimodal fusion life perception method according to claim 1, characterized in that: The generating of the first perception model and the second perception model according to the deployment position and type of the sensor includes: constructing an initial first perception model and an initial second perception model according to the deployment position and type of the sensor; performing model training on the initial first perception model to obtain the first perception model, and performing model training on the initial second perception model to obtain the second perception model.

8. A multi-modal fusion life perception system, characterized in that: It includes a terminal, a server and multiple sensors, and the multiple sensors, the terminal and the server communicate with each other; the multiple sensors are deployed according to life habit information and life scene information, the terminal is used to obtain multimodal sensor information, an initial first perception model is constructed according to the deployment location and type of the sensor, the initial first perception model is trained to obtain a first perception model, the multimodal sensor information is input into the first perception model, and preliminary features and preliminary prediction results are obtained; the server is used to obtain multimodal sensor information and the preliminary features, an initial second perception model is constructed according to the deployment location and type of the sensor, the initial second perception model is trained to obtain a second perception model, the multimodal sensor information and the preliminary features are input into the second perception model to obtain a secondary prediction result; and early warning information is generated according to the degree of correlation between the preliminary prediction result and the secondary prediction result.

9. The multi-modal fusion life perception system according to claim 8, characterized in that: The types of sensors include color cameras, depth cameras, millimeter wave radars, and infrared cameras.

10. The multi-modal fusion life perception system according to claim 9, characterized in that: The system is provided with a communication module for sending information to designated users according to the early warning information.

Images