A method, device, equipment and medium for intelligent infant monitoring based on multimodal monitoring

Abstract

The present application relates to the field of medical monitoring technology and discloses a method, device, equipment and medium for intelligent monitoring of infants based on multimodal monitoring. The method comprises: obtaining vibration physiological image data of a target object; extracting signal components from the vibration physiological image data to obtain a vibration vital sign signal of the target object; performing a joint vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object to obtain a physiological vital sign signal of the target object; obtaining a physiological behavior signal of the target object; performing a multimodal abnormality analysis on the target object based on the physiological sign signal and the physiological behavior signal to generate monitoring measures for the target object. The beneficial effect is that the physiological signs of the target object are jointly analyzed using multidimensional optical information, which overcomes the bottleneck of a single monitoring method in physiological sign monitoring, improves the accuracy of the vital sign analysis results, and thereby improves the monitoring effect of the target object.

Description

Technical Field

[0001] The present application relates to the field of medical monitoring technology, and in particular to a method, device, equipment and medium for intelligent infant monitoring based on multimodal monitoring. Background Art

[0002] In the field of medical monitoring technology, infants are common monitoring subjects. Due to their fragile physiological functions and susceptibility to external environmental influences, infants' physiological states often have large fluctuations. Therefore, long-term and continuous health monitoring of infants is required to reduce the probability of health and safety accidents. In related technologies, the monitoring process usually monitors the vital signs and health status of the monitored subjects in real time, and conducts risk analysis based on the monitoring data to discover potential health risks of the monitored subjects. On this basis, early warnings are also issued based on the analysis results to enable timely intervention in the health status of the monitored subjects, ensure that the vital signs of the monitored subjects are in a stable state, and reduce the impact of potential health risks on the monitored subjects. However, in related technologies, the accuracy of risk analysis and warnings based on monitoring data still needs to be improved. Summary of the Invention

[0003] The present application provides an intelligent infant monitoring method, device, equipment and medium based on multimodal monitoring, which uses multi-dimensional optical information to jointly analyze the physiological signs of the target object, overcomes the bottleneck of a single monitoring method in physiological sign monitoring, improves the accuracy of the sign analysis results, and thus improves the monitoring effect of the target object and ensures the life safety of the target object.

[0004] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0005] In a first aspect, an embodiment of the present application provides a method for intelligent monitoring of infants based on multimodal monitoring, the method comprising:

[0006] Acquiring vibration physiological image data of a target object; wherein the vibration physiological image data is wave optical information representing physiological vibration of the target object;

[0007] Extracting signal components from the vibration physiological image data to obtain a vibration sign signal of the target object;

[0008] performing a combined vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object to obtain a physiological vital sign signal of the target object; wherein the visual physiological image data is visual imaging information representing the vital sign signal of the target object;

[0009] Acquire physiological behavior signals of the target object; perform multimodal abnormality analysis on the target object based on the physiological sign signals and the physiological behavior signals, and generate monitoring measures for the target object.

[0010] The embodiment of the present application proposes an intelligent infant monitoring method based on multimodal monitoring, which obtains wave optical information and visual imaging information containing the vital sign signals of the target object, and performs a joint analysis based on the wave optical information and visual imaging information to obtain the physiological sign signals of the target object; performs abnormal analysis on the target object based on multimodal signals such as the physiological sign signals and physiological behavior signals of the target object, and generates corresponding monitoring measures to intervene in the health status of the target object according to the monitoring measures, ensuring that the physiological signs of the target object are in a stable state and protecting the life safety of the target object. Compared with the relevant technology, the present application uses multiple optical means to acquire data from the target object, and performs a joint analysis based on the acquired multidimensional optical information to determine the physiological sign signals of the target object, thereby overcoming the bottleneck of a single monitoring means in physiological sign monitoring, improving the accuracy of the physiological sign signals, and thus improving the monitoring effect of the target object. In addition, the present application also obtains the physiological sign signals of the target object through an all-optical fusion architecture, reducing the risk of electromagnetic interference and improving the stability and accuracy of the obtained signal.

[0011] Optionally, performing a joint vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object to obtain the physiological vital sign signal of the target object includes:

[0012] Performing a physical sign feature analysis on the target object based on the visual physiological image data to obtain a physiological physical sign feature of the target object; wherein the physiological physical sign feature is a signal feature of a visual physical sign signal of the target object, and the visual physical sign signal is a physical sign signal of the target object obtained from the visual physiological image data;

[0013] Feature matching is performed on the vibration sign signal according to the physiological sign feature, and the physiological sign signal is extracted from the vibration sign signal.

[0014] Optionally, performing a physical sign feature analysis on the target object based on the visual physiological image data to obtain the physiological sign features of the target object includes:

[0015] Performing signal component analysis on the visual physiological image data to extract the visual sign signal;

[0016] Signal feature extraction is performed on the visual sign signal to obtain the physiological sign feature.

[0017] Optionally, the vibration sign signal includes multiple sign signals of the target object; and extracting signal components from the vibration physiological image data to obtain the vibration sign signal of the target object includes:

[0018] Performing phase recovery on the vibration physiological image data to obtain a composite phase signal of the target object; wherein the composite phase signal is a fusion signal of the phase signals corresponding to the multiple vital sign signals;

[0019] Performing frequency separation on the composite phase signal to obtain a vital sign phase signal corresponding to any one of the multiple vital sign signals;

[0020] Perform signal reconstruction according to the vital sign phase signals to obtain corresponding vital sign signals;

[0021] The vibration sign signal is obtained according to all the sign signals.

[0022] Optionally, performing phase recovery according to the vibration physiological image data to obtain a composite phase signal of the target object includes:

[0023] performing phase extraction on the vibration physiological image data to obtain an initial phase signal of the target object;

[0024] Phase unwrapping is performed on the initial phase signal to obtain the composite phase signal.

[0025] Optionally, acquiring vibration physiological image data of the target object includes:

[0026] Using an optical sensing device to collect physiological vibrations of the target object to obtain a physiological vibration signal of the target object;

[0027] Signal imaging is performed according to the physiological vibration signal to obtain the vibration physiological image data.

[0028] Optionally, the physiological behavior signal includes behavior signals of the target object in multiple modalities; performing multimodal abnormality analysis on the target object based on the physiological sign signal and the physiological behavior signal to generate monitoring measures for the target object includes:

[0029] Performing abnormality analysis on the physiological sign signal and the physiological behavior signal respectively to obtain abnormality analysis results of the target object; wherein the abnormality analysis results indicate the number and type of modalities in which the target object has abnormal physical signs;

[0030] The monitoring measures are generated according to the abnormality analysis results.

[0031] In a second aspect, an embodiment of the present application provides an intelligent infant monitoring device based on multimodal monitoring, the device comprising:

[0032] A vibration feature acquisition module, configured to acquire vibration physiological image data of a target object; wherein the vibration physiological image data is wave optical information representing physiological vibrations of the target object;

[0033] a vibration signal extraction module, configured to extract signal components from the vibration physiological image data to obtain a vibration vital sign signal of the target object;

[0034] a combined vital sign analysis module, configured to perform a combined vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object, to obtain a physiological vital sign signal of the target object; wherein the visual physiological image data is visual imaging information representing the vital sign signal of the target object;

[0035] The modal abnormality analysis module is used to obtain the physiological behavior signal of the target object; perform multimodal abnormality analysis on the target object according to the physiological sign signal and the physiological behavior signal, and generate monitoring measures for the target object.

[0036] In a third aspect, an embodiment of the present application provides a computer device comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the method described in any one of the above embodiments by executing the computer instructions.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute any one of the methods in the above embodiments.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute any one of the methods described in the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 A diagram showing the steps of a multimodal monitoring method for intelligent infant monitoring according to an embodiment of the present application;

[0041] Figure 2 A diagram showing the steps for obtaining physiological sign signals in an embodiment of the present application;

[0042] Figure 3 A diagram showing the steps for obtaining physiological sign characteristics in an embodiment of the present application;

[0043] Figure 4 A diagram showing the steps for obtaining vibration sign characteristics in an embodiment of the present application;

[0044] Figure 5 A diagram showing the steps for obtaining a composite phase signal in an embodiment of the present application;

[0045] Figure 6 A diagram showing the steps for obtaining vibration physiological image data in an embodiment of the present application;

[0046] Figure 7a This is a schematic structural diagram of the photoelectric sensing device in an embodiment of the present application;

[0047] Figure 7b This is a schematic diagram of the use of the photoelectric sensing device in the embodiment of the present application;

[0048] Figure 8 A diagram showing steps for generating monitoring measures in an embodiment of the present application;

[0049] Figure 9 A module diagram of an intelligent infant monitoring device based on multimodal monitoring provided in an embodiment of the present application;

[0050] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0052] In the field of medical monitoring technology, infants are common monitoring subjects. Due to their fragile physiological functions and susceptibility to external environmental influences, infants' physiological states often fluctuate greatly. Therefore, long-term and continuous health monitoring of infants is necessary to reduce the probability of health and safety accidents. In related technologies, the monitoring process usually involves real-time monitoring of the subject's vital signs and health status, and risk analysis based on the monitoring data to identify potential health risks of the subject. On this basis, early warning is also issued based on the analysis results to enable timely intervention in the subject's health state, ensuring that the subject's vital signs are in a stable state and reducing the impact of potential health risks on the subject.

[0053] When real-time monitoring of the monitored object is carried out, the physiological signs of the monitored object are generally obtained through physiological signal monitoring equipment or optical monitoring technology. Among them, the physiological signal monitoring equipment can be any one or more of the devices such as electrocardiogram monitoring equipment, blood oxygen saturation monitor, blood pressure monitoring equipment, body temperature monitoring equipment or respiratory monitoring equipment. These devices need to establish a physical connection with the monitored object and collect signals on the surface of the monitored object's body through electrodes, sensor patches or other sensing devices to obtain the monitored object's physiological signs. The above-mentioned sensing devices may cause discomfort to the monitored object during long-term health monitoring. In severe cases, they may cause allergies or skin damage to the monitored object, which in turn affects the health of the monitored object. At the same time, the sensing device connected to the monitored object may also restrict the movement of the monitored object, affect the freedom of movement of the monitored object, and even affect the sleep quality and comfort of the monitored object.

[0054] In the technical solution of obtaining the physiological signs of the monitored object through optical monitoring technology, the optical monitoring technology can be pulse oximetry (SpO2) monitoring, photoplethysmography (PPG), infrared thermal imaging or spectral imaging. To illustrate one of the optical monitoring technologies, taking photoplethysmography as an example, this technology mainly illuminates the skin surface of the monitored object and obtains the reflected light from the skin surface through an optical sensor, thereby analyzing the blood flow under the skin surface and monitoring the blood flow changes in real time for calculating the monitored object's pulse or heart rate and other physiological signs. It can be understood that optical monitoring technology has the advantage of being non-invasive, which can reduce the discomfort and infection risk of the monitored object during the monitoring process.

[0055] However, in reality, the subject is bedridden during monitoring, and their skin may be covered by thick blankets and other objects. This can block the light from the optical monitoring technology, preventing it from effectively reaching the subject's skin, affecting data acquisition during the monitoring process. Similarly, some subjects cannot maintain the same posture for long periods of time during monitoring. When the subject changes posture, the angle and path of the optical monitoring technology's light hitting the subject's skin surface may change, significantly affecting the accuracy and stability of the monitoring data.

[0056] Furthermore, in some special environments with poor lighting conditions, such as low-light environments or at night, the use of optical monitoring technology to obtain data may result in poor data quality due to insufficient lighting. Therefore, it is necessary to use visible light to supplement the illumination to improve data quality. However, supplementary illumination at night can easily affect the sleep quality of the monitored subject, thereby disrupting or even changing the subject's sleep cycle, and negatively affecting the subject's health.

[0057] Based on the above problems, the present application provides an intelligent infant monitoring method, device, equipment and medium based on multimodal monitoring, the method including: obtaining vibration physiological image data of the target object; extracting signal components of the vibration physiological image data to obtain the vibration vital sign signal of the target object; performing a joint vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object to obtain the physiological vital sign signal of the target object; obtaining the physiological behavior signal of the target object; performing multimodal abnormality analysis on the target object based on the physiological sign signal and the physiological behavior signal, and generating monitoring measures for the target object.

[0058] The intelligent infant monitoring method based on multimodal monitoring provided in the present application obtains wave optical information and visual imaging information containing the vital sign signals of the target object, and performs a joint analysis based on the wave optical information and the visual imaging information to obtain the physiological sign signals of the target object; the target object is analyzed for abnormalities based on multimodal signals such as the physiological sign signals and physiological behavior signals of the target object, and corresponding monitoring measures are generated to intervene in the health status of the target object according to the monitoring measures, to ensure that the physiological signs of the target object are in a stable state, and to protect the life safety of the target object.

[0059] Compared to related technologies, this application utilizes multiple optical methods to acquire data from the target object and performs a joint analysis based on the acquired multi-dimensional optical information to determine the target object's physiological sign signals. This overcomes the bottleneck of single monitoring methods in physiological sign monitoring, improves the accuracy of physiological sign signals, and thus enhances the monitoring effect of the target object. In addition, this application also obtains the target object's physiological sign signals through an all-optical fusion architecture, reducing the risk of electromagnetic interference and improving the stability and accuracy of the obtained signals.

[0060] The intelligent infant monitoring method based on multimodal monitoring provided in this specification can be applied to subjects requiring real-time vital sign monitoring, such as infants aged 0 to 1 year, including newborns within 28 days of birth. It is understood that the intelligent infant monitoring method based on multimodal monitoring provided in this specification can be applied to different monitoring environments, such as in a hospital's neonatal department to monitor newborns, or in a home environment to monitor physiological signs such as an infant's respiration or heart rate, and provide early warning of abnormal conditions based on the monitoring results.

[0061] According to an embodiment of the present application, an embodiment of an intelligent infant monitoring method based on multimodal monitoring is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0062] In this embodiment, a method for intelligent monitoring of infants based on multimodal monitoring is provided, which can be used for the above-mentioned monitoring objects that need real-time vital sign monitoring. Figure 1 As shown, the method includes:

[0063] S100. Acquire vibration physiological image data of a target object; wherein the vibration physiological image data is wave optical information representing physiological vibration of the target object.

[0064] S200. Extract signal components from the vibration physiological image data to obtain a vibration sign signal of the target object.

[0065] S300. Perform a joint vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object to obtain the physiological vital sign signal of the target object; wherein the visual physiological image data is visual imaging information representing the vital sign signal of the target object.

[0066] S400. Acquire physiological behavior signals of the target object; perform multimodal abnormality analysis on the target object based on the physiological sign signals and physiological behavior signals, and generate monitoring measures for the target object.

[0067] The vibration physiological image data is wave optical information that characterizes the physiological vibrations of the target object. Wave optical imaging is performed based on the target object's own physiological vibrations to generate corresponding wave optical image data, which serves as the vibration physiological image data. The vibration physiological image data can include interference fringe image sequences obtained by dual-wavelength interference and image data such as scattering spectra obtained by dynamic light scattering. By extracting signal components from the vibration physiological image data, a vibration vital sign signal can be obtained. The vibration vital sign signal can be an independent signal that reflects the target object's physiological state and vital sign information, including but not limited to signals such as respiratory signals, heartbeat signals, pulse signals, and blood pressure signals.

[0068] Visual physiological image data is visual imaging information representing the target subject's vital signs. Optical imaging technology is used to capture images of the target subject's specific skin surface, generating an image containing target feature data as the visual physiological image data. The target feature data can represent the target subject's physiological signs. It is understood that the method for generating visual physiological image data can be any optical monitoring technology. By performing feature analysis on the visual physiological image data, multiple vital signs signals can be obtained to reflect the target subject's physiological state and vital sign information.

[0069] Physiological sign signals can be obtained through combined vital sign analysis and accurately represent the target subject's physiological signs, including but not limited to respiratory signals, heartbeat signals, pulse signals, and blood pressure signals. Physiological behavioral signals can represent the target subject's physical activities during monitoring, reflecting the target subject's behavioral patterns and reflecting the target subject's health status from different modalities.

[0070] Specifically, wave optical imaging is performed based on the physiological vibrations of the target object to obtain vibration physiological image data of the target object. The vibration physiological image data represents the physiological vibrations of the target object, which are generated by the target object's physiological activities, including heartbeat, breathing, or blood flow. Therefore, the physiological vibration signal extracted from the vibration physiological image data can be a composite signal containing multiple types of vital sign signals of the target object. Based on the physiological vibration signal, multiple vibration vital sign signals of the target object can be obtained, each representing a different physiological sign of the target object.

[0071] Furthermore, visual physiological image data of the target object is acquired, and feature analysis and signal extraction are performed on the visual physiological image data to obtain a visual vital sign signal. It is understood that, depending on the method and location of acquiring the visual physiological image data, the visual vital sign signal may also be a composite signal containing multiple types of vital sign signals of the target object. Signal characteristics of various types of vital sign signals can be obtained based on the visual vital sign signal, including but not limited to frequency, amplitude, and phase. Based on the vibration vital sign signal and the obtained signal characteristics, a combined vital sign analysis is performed on the target object to obtain the target object's physiological vital sign signal.

[0072] It should be noted that a single optical monitoring method is extremely susceptible to interference from the environment and external factors in actual situations, and cannot accurately represent the physiological signs of the target object. In addition, the monitoring data obtained based on a single optical monitoring method can often only one-sidedly represent the physiological signs of the target object in a certain aspect. In the case that there are significant individual differences among the target objects, the obtained physiological sign signal may cause a significant deviation from the actual signal, affecting the accuracy of the physiological sign signal. Compared with the related art, the present application uses multiple optical means to acquire data from the target object and obtain image data of the target object in multiple optical dimensions, thereby overcoming the bottleneck of a single monitoring method in physiological sign monitoring, and can combine image data in multiple optical dimensions to achieve a full range of physiological sign analysis of the target object, thereby improving the monitoring accuracy and monitoring effect, and thus improving the monitoring effect of the target object.

[0073] In some embodiments, the method for acquiring visual physiological image data may be infrared thermal imaging, and the visual physiological image data obtained thereby may represent the body temperature distribution of the target object. The process of combined vital signs analysis may include: transforming the vibration vital signs signal into the frequency domain, where the vibration vital signs signal appears as a plurality of signal peaks corresponding to different frequencies; extracting features of the vibration vital signs signal to obtain vibration signal features, including the frequency and peak value corresponding to the signal peak; obtaining the temperature distribution features of the target object based on the visual physiological image data, including the temperature distribution and distribution gradient of the target object; establishing a vital signs feature vector of the target object at the current moment based on the vibration signal features and the temperature distribution features; comparing the vital signs feature vectors at different moments, extracting the vital signs feature vectors whose errors exceed a preset error threshold as abnormal feature vectors; performing abnormal analysis on the target object based on the vibration signal features and the temperature distribution features in the abnormal feature vector to determine the specific location where the abnormality of the target object occurs.

[0074] It is understood that physiological vibrations are generated at different locations on the target object due to physiological activity, and the frequency and amplitude of these physiological vibrations are constant, resulting in a constant vibration signal characteristic. That is, the peak value of the signal peak corresponding to each frequency is fixed. These physiological activities can be, for example, blood flow, which forms subcutaneous microcirculation beneath the target object's skin to maintain the target object's normal physiological activity. When the subcutaneous microcirculation at a certain location on the target object's body becomes abnormal, the physiological vibration at that abnormal location deviates from the normal physiological vibration, causing an error in the vibration signal characteristics in the vital sign feature vector. Based on the error in the vibration signal characteristics, the physiological vibration characteristics of the abnormal location can be determined. Simultaneously, the specific location on the target object where the abnormality occurs can also be determined based on the temperature distribution characteristics in the corresponding vital sign feature vector. Therefore, by combining the temperature distribution characteristics of the target object at this time with the physiological sign signal at the abnormal location, a specific abnormality analysis can be performed on the target object, thereby generating targeted monitoring measures and improving the monitoring effect for the target object.

[0075] Furthermore, the target subject's behavior is monitored to obtain the target subject's physiological and behavioral signals. Based on the physiological and behavioral signals, a multimodal abnormality analysis is performed on the target subject. When an abnormality in the target subject's health status is detected, the specific abnormality of the target subject can be analyzed based on the physiological and behavioral signals. Targeted monitoring measures are generated based on the specific abnormality to provide timely intervention for the target subject, ensuring that the target subject's vital signs remain stable and reducing the impact of the abnormality on the target subject.

[0076] The intelligent infant monitoring method based on multimodal monitoring provided in this embodiment obtains wave optical information and visual imaging information containing the vital sign signals of the target object, and performs a joint analysis based on the wave optical information and the visual imaging information to obtain the physiological sign signals of the target object; the target object is analyzed for abnormalities based on multimodal signals such as the physiological sign signals and physiological behavior signals of the target object, and corresponding monitoring measures are generated to intervene in the health status of the target object according to the monitoring measures, thereby ensuring that the physiological signs of the target object are in a stable state and protecting the life safety of the target object.

[0077] Compared to related technologies, this application utilizes multiple optical methods to acquire data from the target object and performs a joint analysis based on the acquired multi-dimensional optical information to determine the target object's physiological sign signals. This overcomes the bottleneck of single monitoring methods in physiological sign monitoring, improves the accuracy of physiological sign signals, and thus enhances the monitoring effect of the target object. In addition, this application also obtains the target object's physiological sign signals through an all-optical fusion architecture, reducing the risk of electromagnetic interference and improving the stability and accuracy of the obtained signals.

[0078] Reference Figure 2 As shown, as an embodiment of the present application, a joint vital sign analysis is performed on the target object based on the vibration vital sign signal and the visual physiological image data of the target object to obtain the physiological vital sign signal of the target object, including:

[0079] S310. Perform a physical sign feature analysis on the target object based on the visual physiological image data to obtain the physiological physical sign features of the target object; wherein the physiological physical sign features are signal features of the visual physical sign signals of the target object, and the visual physical sign signals are physical sign signals of the target object obtained from the visual physiological image data.

[0080] S320. Perform feature matching on the vibration sign signal according to the physiological sign feature, and extract the physiological sign signal from the vibration sign signal.

[0081] Specifically, optical imaging technology is used to capture an image of a specific skin surface of a target subject, generating an image containing target feature data as visual physiological image data. Visual vital signs signals can be derived from the visual physiological image data. It will be appreciated that, depending on the method and location of acquiring the visual physiological image data, the visual vital signs signal can be a composite signal containing multiple types of vital signs signals of the target subject. Therefore, by performing feature analysis on the visual vital signs signal based on the types and corresponding characteristics of the different vital signs signals, signal features corresponding to the different vital signs signals can be extracted from the visual vital signs signal as physiological vital signs features.

[0082] To illustrate the process of deriving physiological sign features, optical imaging technology can be used to capture images of a target subject's face, capturing images showing the periodic changes in temperature and movement of key organs and facial color associated with physiological activity. Image analysis of these images reveals the periodic variation trends of corresponding features within the images, yielding various physiological sign signals of the target subject as physiological sign features.

[0083] Furthermore, in this embodiment, the process of performing a joint vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object includes: performing feature matching on the vibration vital sign signal according to the physiological vital sign characteristics, and extracting the physiological vital sign signal from the vibration vital sign signal.

[0084] It should be noted that when extracting signal components from vibration physiological image data, since the physiological activities of the target object are complex and multi-layered, multiple corresponding signals may be obtained for any type of vital sign signal, but in actual circumstances it is impossible to determine the true signal from multiple signals. Physiological sign features contain multiple types of signal features, and the signal features correspond to different types of vital sign signals, and the types of these vital sign signals can be the same as the types of vital sign signals contained in the vibration vital sign signal. In the case of the same type of vital sign signals, feature matching can be performed on the vital sign signals in the vibration vital sign signal based on the signal features in the physiological sign features to determine the true signal from multiple vital sign signals corresponding to any type of vital sign signal, thereby extracting the physiological sign signal and improving the accuracy of the physiological sign signal.

[0085] Reference Figure 3 As shown, as an embodiment of the present application, the target object is analyzed for physical sign characteristics based on visual physiological image data to obtain the target object's physiological sign characteristics, including:

[0086] S312. Perform signal component analysis on the visual physiological image data to extract visual sign signals.

[0087] S314. Extract signal features from the visual vital sign signal to obtain physiological vital sign features.

[0088] Specifically, the signal component analysis process may include performing image analysis on the visual physiological image data to extract visual vital signs signals. The visual vital signs signals may be composite signals containing multiple types of vital signs signals. Based on the actual characteristics of the different types of vital signs signals, the visual vital signs signals may be separated to obtain multiple independent signals corresponding to different types of physiological signs.

[0089] Furthermore, signal features are extracted from multiple independent signals in the visual vital sign signal to obtain signal features corresponding to different types of vital sign signals as physiological vital sign features. It should be noted that when the environmental conditions of the target object's environment meet the preset requirements, such as when the ambient lighting conditions are good, the credibility of the physiological vital sign features is higher than the signal features in the vibration vital sign signal. Therefore, the vibration vital sign signal can be feature matched according to the physiological vital sign features to extract the physiological vital sign signal from the vibration vital sign signal and improve the accuracy of the physiological vital sign signal. It is understandable that when the environmental conditions of the target object's environment cannot meet the preset requirements, the credibility of the physiological vital sign features is lower than the signal features in the vibration vital sign signal. In this case, the feature matching step can be ignored.

[0090] Reference Figure 4As shown, as an embodiment of the present application, the vibration sign signal includes multiple sign signals of the target object; the signal component extraction of the vibration physiological image data is performed to obtain the vibration sign signal of the target object, including:

[0091] S210. Perform phase recovery based on the vibration physiological image data to obtain a composite phase signal of the target object; wherein the composite phase signal is a fusion signal of phase signals corresponding to multiple vital sign signals.

[0092] S220. Perform frequency separation on the composite phase signal to obtain a vital sign phase signal corresponding to any vital sign signal among the multiple vital sign signals.

[0093] S230. Perform signal reconstruction according to the vital sign phase signals to obtain corresponding vital sign signals.

[0094] S240. Obtain a vibration sign signal based on all the sign signals.

[0095] Specifically, vibration physiological image data represents the target subject's various physiological vibrations, each corresponding to its own vital sign signal. These different vital sign signals are integrated into a composite vibration signal. Therefore, signal component extraction is necessary to extract these different vital sign signals from vibration physiological image data.

[0096] Furthermore, phase recovery is performed on the vibration composite signal based on the vibration physiological image data to obtain a composite phase signal of the target object. The composite phase signal can be a continuous phase distribution signal that represents the phase information of the vibration composite signal. Based on this, signal separation is performed on the composite phase signal based on the actual characteristics of different types of vital sign signals to obtain multiple independent phase signals corresponding to different types of vital sign signals as vital sign phase signals.

[0097] Exemplarily, the method of frequency separation can be frequency domain filtering separation. First, a fast Fourier transform is performed on the composite phase signal, and the composite phase signal is transformed into the frequency domain to obtain a corresponding spectrum distribution. Secondly, a corresponding bandpass filter is designed according to the actual characteristics of the vital sign signal to extract the corresponding vital sign signal from the spectrum distribution. Common vital sign signals include respiratory signals and heartbeat signals, wherein the frequency range of the respiratory signal is about 0.1Hz to 0.5Hz, and the frequency range of the heartbeat signal is about 1Hz to 2Hz. According to the above frequency range, a bandpass filter with a cutoff frequency of 0.05Hz to 0.8Hz can be designed for the respiratory signal, and a bandpass filter with a cutoff frequency of 0.8Hz to 3Hz can be designed for the heartbeat signal. Signal extraction is performed on the composite phase signal respectively to obtain the respiratory signal and the heartbeat signal. At this time, the respiratory signal and the heartbeat signal are both in the frequency domain. The respiratory signal and the heartbeat signal are respectively subjected to inverse Fourier transform and transformed back to the time domain to obtain the separated respiratory signal and heartbeat signal.

[0098] Furthermore, the optical parameters used to generate the vibration physiological image data can be pre-set. Based on the pre-set optical parameters and the vital sign phase signal, signal reconstruction is performed for each type of physiological sign to obtain the corresponding vital sign signal. After all types of vital sign signals are obtained, the vibration vital sign signal is derived from all the vital sign signals to achieve complete separation of the vibration composite signal.

[0099] Reference Figure 5 As shown, as an embodiment of the present application, phase recovery is performed based on the vibration physiological image data to obtain a composite phase signal of the target object, including:

[0100] S212. Perform phase extraction on the vibration physiological image data to obtain an initial phase signal of the target object.

[0101] S214. Perform phase unwrapping on the initial phase signal to obtain a composite phase signal.

[0102] Specifically, a phase extraction algorithm is used to extract the phase of the vibration physiological image data to obtain the initial phase corresponding to each pixel in the vibration physiological image data, and then obtain the initial phase signal of the target object. Exemplarily, the phase extraction algorithm can be based on a Fourier transform method or a phase shift interferometry method. When the vibration physiological image data is an interference fringe image sequence obtained by interference, the number of initial phase signals can be two or more, corresponding to light of different wavelengths. At this time, all the initial phase signals are synthesized into a composite phase, corresponding to the phase information of the composite phase signal.

[0103] Furthermore, the initial phase signal is phase unwrapped using a phase unwrapping algorithm to perform phase continuous correction on the initial phase signal to obtain a composite phase signal. For example, when the vibration physiological image data is an interference fringe image sequence obtained by interference, the phase unwrapping algorithm can be a path tracking method that tracks the interference fringe path in the vibration physiological image data and compares the phases of adjacent pixels. If a phase jump occurs between adjacent pixels, the phase of the pixel experiencing the phase jump is corrected to eliminate the phase jump in the initial phase signal, improve the continuity of the initial phase signal, and obtain a composite phase signal.

[0104] Reference Figure 6 As shown in one embodiment of the present application, obtaining vibration physiological image data of a target object includes:

[0105] S110. Use optical sensing equipment to collect physiological vibrations of the target object to obtain a physiological vibration signal of the target object.

[0106] S120. Perform signal imaging based on the physiological vibration signal to obtain vibration physiological image data.

[0107] Reference Figure 7a , the structure of the optical sensing device can be as follows Figure 7a As shown, the optical sensing device includes an outer layer, a sensing fiber, a laser light source, and an optoelectronic interface. The outer layer can be a medical silicone pad, which is ultrathin and flexible and can quickly respond to external vibrations. The outer layer is embedded with a sensing fiber. The sensing fiber can be a multi-array multi-core fiber, arranged in a serpentine pattern within the outer layer. This serpentine pattern improves the spatial resolution of the sensing fiber. The sensing fiber contacts an object attached to the outer layer through the outer layer. The propagation characteristics of light in the sensing fiber change in response to the object's vibration, thereby monitoring the object's vibration. The two ends of the sensing fiber are connected to the laser light source and the optoelectronic interface, respectively. The laser light source provides light input to the sensing fiber. In some embodiments, the laser light source can also be located outside the optical sensing device to reduce the complexity of the optical sensing device. The optoelectronic interface receives the light output from the sensing fiber and transmits it to an external imaging device to obtain vibration physiological image data. For example, the external imaging device can be an interferometer, in which case the vibration physiological image data can be a sequence of interference fringe images obtained by interference.

[0108] Reference Figure 7b As shown, Figure 7aThe optical sensing device shown can be laid flat on the target subject's bed, with a mattress placed above it to enhance the subject's comfort. The target subject is positioned above the mattress, corresponding to the area of ​​the optical sensing device where the sensing fiber is embedded. During monitoring, the target subject generates physiological vibrations due to physiological activity. These vibrations are transmitted through the mattress to the optical sensing device, where they are collected and detected, producing physiological vibration signals. These signals are then transmitted via an optoelectronic interface to an external imaging device, where they are imaged to produce physiological vibration image data.

[0109] Compared to related technical solutions that use physiological signal monitoring equipment or optical monitoring technology to obtain the physiological signs of the monitored subject, the optical sensing device in this application does not require direct contact with the target subject, achieving unconstrained physiological sign monitoring, improving the target subject's comfort during the monitoring process, and reducing the impact on the monitored subject's health. In addition, because this application directly collects the target subject's physiological vibrations through sensing optical fibers rather than using optical monitoring technology for visual imaging, it is not affected by light obstruction, and there is no need to provide supplemental light to the target subject during the monitoring process, reducing the impact on the target subject's sleep quality.

[0110] Reference Figure 8 As shown, as an embodiment of the present application, the physiological behavior signal includes behavioral signals of the target object in multiple modalities; multimodal abnormality analysis is performed on the target object based on the physiological sign signal and the physiological behavior signal, and monitoring measures for the target object are generated, including:

[0111] S410. Perform abnormality analysis on the physiological sign signal and the physiological behavior signal respectively to obtain abnormality analysis results of the target object; wherein the abnormality analysis results represent the number and type of modalities in which the target object has abnormal signs.

[0112] S420. Generate monitoring measures based on the abnormal analysis results.

[0113] Specifically, the target subject's behavior is monitored to obtain physiological behavioral signals. The target subject's physiological behavioral signals may include body movement signals and sound signals. Body movement signals may be conscious or unconscious body movements made by the target subject due to their health status. Whether the body movement signals are abnormal can indicate whether the target subject's health status is abnormal. To illustrate the body movement signals, if the target subject is a newborn, sudden, large, and sustained body movements may indicate an abnormality in the target subject, requiring timely intervention.

[0114] As an example, the body motion signal can be obtained by tracking the body motion of a target object using a camera. Based on the image data or video data captured by the camera, a joint point calibration algorithm is used to calibrate multiple corresponding joints, and the body motion signal is determined based on the displacement of the joints. The above-mentioned camera can use visible light or near-infrared light for data collection. Specifically, a camera using visible light can collect data from the target object during the day, and a camera using near-infrared light can collect data from the target object at night.

[0115] Similarly, sound signals can be sounds consciously or unconsciously emitted by the target subject due to their health condition. Sound signals can include normal sound signals and abnormal sound signals. For example, normal sound signals can include normal conversation or breathing sounds of the target subject, indicating that the target subject is in a normal health state. Abnormal sound signals can include groaning, etc., indicating that the target subject is in pain or discomfort, that is, an abnormality in the target subject's health state.

[0116] For example, a sound signal can be collected by a microphone array and extracted using a voiceprint feature extraction model. For example, the voiceprint feature extraction model can combine Mel-Frequency Cepstral Coefficients (MFCCs) with a Gaussian Mixture Model (GMM) and use a high-pass filter to pre-emphasize the collected signal to optimize the frequency of the sound signal, making the sound signal's spectrum flatter and compensating for suppressed high-frequency components.

[0117] Furthermore, a multimodal abnormality analysis is performed on the target subject based on the physiological sign signals and physiological behavior signals. The target subject's health status is analyzed in multiple modalities to determine the number and type of modalities in which the target subject exhibits abnormal signs. If the number and type of modalities in which the target subject exhibits abnormal signs meet preset abnormality conditions, the target subject's health status is determined to be abnormal. The specific abnormality of the target subject is analyzed based on the physiological sign signals and physiological behavior signals, and targeted monitoring measures are generated based on the specific abnormality to intervene in the target subject in a timely manner, ensure that the target subject's vital signs remain stable, and reduce the impact of the abnormality on the target subject.

[0118] Exemplarily, an abnormality analysis is performed on the target object's physiological sign signals and various physiological behavior signals to determine whether the modes corresponding to the above signals are abnormal. When the target object has a single-modal abnormality, an audible and visual alarm can be performed in combination with the alarm equipment in the target object's environment to remind medical staff to intervene in the target object. When the target object has a dual-modal abnormality, the abnormal state of the target object is pushed to the medical terminal on the basis of the audible and visual alarm to quickly notify the medical staff of the abnormal state. When the target object has a tri-modal or multi-modal abnormality, it is necessary to compare the target object's physiological sign signals at the current moment with the physiological sign signals at the historical moment. If the physiological sign signals at the current moment deviate significantly from the physiological sign signals at the historical moment, it indicates that the target object has a serious abnormal state and it is necessary to initiate emergency linkage to quickly intervene in the target object to ensure the life safety of the target object.

[0119] Accordingly, please refer to Figure 9 The present invention provides an intelligent infant monitoring device based on multimodal monitoring, the device comprising:

[0120] The vibration feature acquisition module 910 is used to acquire vibration physiological image data of the target object; wherein the vibration physiological image data is wave optical information representing the physiological vibration of the target object.

[0121] The vibration signal extraction module 920 is used to extract signal components from the vibration physiological image data to obtain the vibration vital sign signal of the target object.

[0122] The combined vital sign analysis module 930 is used to perform combined vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object to obtain the physiological vital sign signal of the target object; wherein the visual physiological image data is the visual imaging information representing the vital sign signal of the target object.

[0123] The modal abnormality analysis module 940 is used to obtain physiological behavior signals of the target object; perform multimodal abnormality analysis on the target object based on the physiological sign signals and physiological behavior signals, and generate monitoring measures for the target object.

[0124] In some optional embodiments, the combined vital signs analysis module 930 includes:

[0125] The physical sign feature analysis unit is used to perform physical sign feature analysis on the target object based on the visual physiological image data to obtain the physiological physical sign features of the target object; wherein the physiological physical sign features are the signal features of the visual physical sign signals of the target object, and the visual physical sign signals are the physical sign signals of the target object obtained from the visual physiological image data.

[0126] The body sign signal extraction unit is used to perform feature matching on the vibration body sign signal according to the physiological body sign feature, and extract the physiological body sign signal from the vibration body sign signal.

[0127] In some optional embodiments, the physical sign feature analysis unit includes:

[0128] The vital sign signal separation subunit is used to perform signal component analysis on visual physiological image data and extract visual vital sign signals.

[0129] The signal feature extraction subunit is used to extract signal features from visual sign signals to obtain physiological sign features.

[0130] In some optional embodiments, the vibration sign signal includes multiple sign signals of the target object; the vibration signal extraction module 920 includes:

[0131] The phase signal recovery unit is used to perform phase recovery based on the vibration physiological image data to obtain a composite phase signal of the target object; wherein the composite phase signal is a fusion signal of the phase signals corresponding to multiple vital sign signals.

[0132] The phase signal separation unit is used to perform frequency separation on the composite phase signal to obtain a vital sign phase signal corresponding to any one of the multiple vital sign signals.

[0133] The vibration signal reconstruction unit is used to reconstruct the signal according to the vital sign phase signal to obtain the corresponding vital sign signal.

[0134] The vital sign signal acquisition unit is used to obtain a vibration vital sign signal according to all vital sign signals.

[0135] In some optional implementations, the phase signal recovery unit includes:

[0136] The phase extraction subunit is used to perform phase extraction on the vibration physiological image data to obtain the initial phase signal of the target object.

[0137] The phase unwrapping subunit is used to perform phase unwrapping on the initial phase signal to obtain a composite phase signal.

[0138] In some optional implementations, the vibration feature acquisition module 910 includes:

[0139] The vibration collection unit is used to collect physiological vibrations of the target object using an optical sensing device to obtain the physiological vibration signal of the target object.

[0140] The signal imaging unit is used to perform signal imaging according to the physiological vibration signal to obtain vibration physiological image data.

[0141] In some optional implementations, the modal anomaly analysis module 940 includes:

[0142] The abnormality analysis unit is used to perform abnormality analysis on the physiological sign signal and the physiological behavior signal respectively to obtain the abnormality analysis results of the target object; wherein the abnormality analysis results indicate the number and type of modalities in which the target object has abnormal signs.

[0143] The measure generation unit is used to generate monitoring measures according to the abnormality analysis results.

[0144] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0145] The multimodal monitoring-based intelligent infant monitoring device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0146] See also Figure 10 , Figure 10 1 is a structural diagram of a computer device provided by an embodiment of the present application. As shown in the figure, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 10 A processor 10 is taken as an example.

[0147] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0148] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0149] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0151] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0152] The embodiments of the present application also provide a computer-readable storage medium. The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0153] An embodiment of the present application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to any embodiment of the present application.

[0154] Although the embodiments of the present application have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations shall fall within the scope defined by the appended claims.

[0155] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0156] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1The function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0161] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0162] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0163] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

[0164] Although the embodiments of the present application have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations shall fall within the scope defined by the appended claims.

Claims

1. A method for intelligent infant monitoring based on multimodal monitoring, characterized in that: The method comprises: Acquiring vibration physiological image data of a target object; wherein the vibration physiological image data is wave optical information representing physiological vibration of the target object; Extracting signal components from the vibration physiological image data to obtain a vibration sign signal of the target object; Based on the vibration vital sign signal and the visual physiological image data of the target object, a joint vital sign analysis is performed on the target object to obtain the physiological vital sign signal of the target object; wherein the visual physiological image data is visual imaging information that characterizes the vital sign signal of the target object; the process of the joint vital sign analysis includes: transforming the vibration vital sign signal into the frequency domain and performing feature extraction to obtain vibration signal features; obtaining the temperature distribution features of the target object based on the visual physiological image data; establishing a vital sign feature vector of the target object at the current moment based on the vibration signal features and the temperature distribution features; comparing the vital sign feature vectors at different moments, and extracting the vital sign feature vectors whose errors exceed a preset error threshold as abnormal feature vectors; performing abnormality analysis on the target object based on the vibration signal features and temperature distribution features in the abnormal feature vector to determine the specific location where the abnormality of the target object occurs; Acquire the physiological behavior signal of the target object; perform multimodal abnormality analysis on the target object based on the physiological sign signal of the specific location where the abnormality occurs in the target object and the physiological behavior signal, and generate monitoring measures for the target object.

2. The method according to claim 1, characterized in that The step of performing a combined vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object to obtain the physiological vital sign signal of the target object further includes: Performing a physical sign feature analysis on the target object based on the visual physiological image data to obtain a physiological physical sign feature of the target object; wherein the physiological physical sign feature is a signal feature of a visual physical sign signal of the target object, and the visual physical sign signal is a physical sign signal of the target object obtained from the visual physiological image data; Feature matching is performed on the vibration sign signal according to the physiological sign feature, and the physiological sign signal is extracted from the vibration sign signal.

3. The method according to claim 2, characterized in that The performing physical sign feature analysis on the target object based on the visual physiological image data to obtain the physiological sign features of the target object includes: Performing signal component analysis on the visual physiological image data to extract the visual sign signal; Signal feature extraction is performed on the visual sign signal to obtain the physiological sign feature.

4. The method according to claim 1, wherein The vibration sign signal includes multiple sign signals of the target object; and extracting signal components from the vibration physiological image data to obtain the vibration sign signal of the target object includes: Performing phase recovery on the vibration physiological image data to obtain a composite phase signal of the target object; wherein the composite phase signal is a fusion signal of the phase signals corresponding to the multiple vital sign signals; Performing frequency separation on the composite phase signal to obtain a vital sign phase signal corresponding to any one of the multiple vital sign signals; Perform signal reconstruction according to the vital sign phase signals to obtain corresponding vital sign signals; The vibration sign signal is obtained according to all the sign signals.

5. The method according to claim 4, characterized in that The performing phase recovery according to the vibration physiological image data to obtain the composite phase signal of the target object includes: performing phase extraction on the vibration physiological image data to obtain an initial phase signal of the target object; Phase unwrapping is performed on the initial phase signal to obtain the composite phase signal.

6. The method according to any one of claims 1 to 5, characterized in that The step of obtaining the vibration physiological image data of the target object includes: Using an optical sensing device to collect physiological vibrations of the target object to obtain a physiological vibration signal of the target object; Signal imaging is performed according to the physiological vibration signal to obtain the vibration physiological image data.

7. The method according to any one of claims 1 to 5, characterized in that The physiological behavior signal includes behavior signals of the target object in multiple modalities; performing multimodal abnormality analysis on the target object based on the physiological sign signal of the specific location where the abnormality occurs and the physiological behavior signal to generate monitoring measures for the target object includes: Performing abnormality analysis on the physiological sign signal and the physiological behavior signal of the specific location where the abnormality occurs on the target object, respectively, to obtain an abnormality analysis result of the target object; wherein the abnormality analysis result indicates the number and type of modalities in which the target object has abnormal physical signs; The monitoring measures are generated according to the abnormality analysis results.

8. An intelligent infant monitoring device based on multimodal monitoring, characterized in that: The device comprises: A vibration feature acquisition module, configured to acquire vibration physiological image data of a target object; wherein the vibration physiological image data is wave optical information representing physiological vibrations of the target object; a vibration signal extraction module, configured to extract signal components from the vibration physiological image data to obtain a vibration vital sign signal of the target object; A combined vital sign analysis module is configured to perform a combined vital sign analysis on the target object based on the vibration vital sign signal and the visual physiological image data of the target object to obtain the physiological vital sign signal of the target object; wherein the visual physiological image data is visual imaging information characterizing the vital sign signal of the target object; the combined vital sign analysis process comprises: transforming the vibration vital sign signal into the frequency domain and performing feature extraction to obtain vibration signal features; obtaining temperature distribution features of the target object based on the visual physiological image data; establishing a vital sign feature vector of the target object at the current moment based on the vibration signal features and the temperature distribution features; comparing the vital sign feature vectors at different moments, extracting the vital sign feature vectors whose errors exceed a preset error threshold as abnormal feature vectors; performing abnormality analysis on the target object based on the vibration signal features and temperature distribution features in the abnormal feature vector to determine the specific location where the abnormality occurs in the target object; The modal abnormality analysis module is used to obtain the physiological behavior signals of the target object; perform multimodal abnormality analysis on the target object based on the physiological sign signals of the specific location where the abnormality occurs and the physiological behavior signals, and generate monitoring measures for the target object.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images