Intelligent infant monitoring method, device and equipment based on multi-modal monitoring and medium

Abstract

The invention relates to the technical field of medical monitoring, and discloses an intelligent infant monitoring method and device based on multi-modal monitoring, equipment and a medium, and the method comprises the steps: obtaining vibration physiological image data of a target object; performing signal component extraction on the vibration physiological image data to obtain a vibration sign signal of the target object; according to the vibration sign signal and the visual physiological image data of the target object, carrying out combined sign analysis on the target object to obtain a physiological sign signal of the target object; acquiring a physiological behavior signal of the target object; according to the physiological sign signals and the physiological behavior signals, multi-modal anomaly analysis is conducted on the target object, and monitoring measures for the target object are generated. The method has the beneficial effects that the physiological signs of the target object are subjected to conjoint analysis by utilizing the multi-dimensional optical information, the bottleneck of a single monitoring means in physiological sign monitoring is overcome, the accuracy of a sign analysis result is improved, and the monitoring effect on the target object is further improved.

Description

Technical Field

[0001] This application relates to the technical field of medical monitoring, and particularly to an intelligent infant monitoring method, device, equipment and medium based on multi-modal monitoring. Background Art

[0002] In the technical field of medical monitoring, infants are common monitoring objects. Due to their fragile physiological functions and susceptibility to the external environment, the physiological states of infants often have large fluctuations. Therefore, long-term and continuous health monitoring of infants is required to reduce the occurrence probability of health and safety accidents. In related technologies, the monitoring process usually involves real-time monitoring of data such as the vital signs and health status of the monitored object, and risk analysis based on the monitored data to discover potential health risks of the monitored object. On this basis, early warnings are also made according to the analysis results to enable timely intervention in the health status of the monitored object, ensure that the vital signs of the monitored object are in a stable state, and reduce the impact of potential health risks on the monitored object. However, in related technologies, the accuracy of risk analysis and early warning based on monitored data still needs to be improved. Summary of the Invention

[0003] This application provides an intelligent infant monitoring method, device, equipment and medium based on multi-modal monitoring, which uses multi-dimensional optical information to jointly analyze the physiological signs of the target object, overcomes the bottleneck of single monitoring means in physiological sign monitoring, improves the accuracy of sign analysis results, and further enhances the monitoring effect on the target object, ensuring the life safety of the target object.

[0004] To achieve the above object, the main technical solutions adopted in this application include: In a first aspect, an embodiment of this application provides an intelligent infant monitoring method based on multi-modal monitoring, and the method includes: Obtain vibration physiological image data of the target object; wherein, the vibration physiological image data is the fluctuating optical information representing the physiological vibration of the target object; Extract signal components from the vibration physiological image data to obtain the vibration sign signal of the target object; According to the vibration sign signal and the visual physiological image data of the target object, perform joint sign analysis on the target object to obtain the physiological sign signal of the target object; wherein, the visual physiological image data is the visual imaging information representing the sign signal of the target object; Obtain the physiological behavior signal of the target object; perform multi-modal anomaly analysis on the target object according to the physiological sign signal and the physiological behavior signal, and generate a monitoring measure for the target object.

[0005] The infant intelligent monitoring method based on multimodal monitoring proposed in the embodiments of the present application acquires the fluctuating optical information and visual imaging information of the vital sign signals of the target object, and performs joint analysis based on the fluctuating optical information and visual imaging information to obtain the physiological sign signals of the target object; performs anomaly analysis on the target object according to multimodal signals such as the physiological sign signals and physiological behavior signals of the target object, generates corresponding monitoring measures, and intervenes in the health status of the target object according to the monitoring measures to ensure that the physiological signs of the target object are in a stable state and safeguard the life safety of the target object. Compared with the related technologies, the present application uses a variety of optical means to acquire data of the target object, and performs joint analysis based on the acquired multi-dimensional optical information to determine the physiological sign signals of the target object, thereby overcoming the bottleneck in physiological sign monitoring by a single monitoring means, improving the accuracy of the physiological sign signals, and further enhancing the monitoring effect on the target object. In addition, the present application also obtains the physiological sign signals of the target object through an all-optical fusion framework, reducing the risk of electromagnetic interference and enhancing the stability and accuracy of the obtained signals.

[0006] Optionally, the performing joint sign analysis on the target object according to the vibration sign signal and the visual physiological image data of the target object to obtain the physiological sign signal of the target object includes: Performing sign feature analysis on the target object based on the visual physiological image data to obtain the physiological sign features of the target object; wherein, the physiological sign features are the signal features of the visual sign signals of the target object, and the visual sign signals are the sign signals of the target object obtained from the visual physiological image data; Performing feature matching on the vibration sign signal according to the physiological sign features, and extracting the physiological sign signal from the vibration sign signal.

[0007] Optionally, the performing sign feature analysis on the target object based on the visual physiological image data to obtain the physiological sign features of the target object includes: Performing signal component analysis on the visual physiological image data to extract the visual sign signal; Performing signal feature extraction on the visual sign signal to obtain the physiological sign features.

[0008] Optionally, the vibration sign signal includes multiple sign signals of the target object; the performing signal component extraction on the vibration physiological image data to obtain the vibration sign signal of the target object includes: Performing phase recovery according to the vibration physiological image data to obtain the composite phase signal of the target object; wherein, the composite phase signal is a fusion signal of the phase signals corresponding to the multiple sign signals; Frequency-separate the composite phase signal to obtain the physiological-signal phase signal corresponding to any one of the multiple physiological-signal signals; Perform signal reconstruction respectively according to the physiological-signal phase signal to obtain the corresponding physiological-signal signal; Obtain the vibration physiological-signal signal according to all the physiological-signal signals.

[0009] Optionally, the obtaining the composite phase signal of the target object by performing phase recovery on the vibration physiological-image data includes: Perform phase extraction on the vibration physiological-image data to obtain the initial phase signal of the target object; Perform phase unwrapping on the initial phase signal to obtain the composite phase signal.

[0010] Optionally, the obtaining the vibration physiological-image data of the target object includes: Use an optical sensing device to collect physiological vibrations of the target object to obtain the physiological vibration signal of the target object; Perform signal imaging according to the physiological vibration signal to obtain the vibration physiological-image data.

[0011] Optionally, the physiological behavior signal includes the behavior signals of the target object in multiple modalities; the performing multimodal anomaly analysis on the target object according to the physiological-signal signal and the physiological behavior signal to generate a monitoring measure for the target object includes: Perform anomaly analysis on the physiological-signal signal and the physiological behavior signal respectively to obtain the anomaly analysis result of the target object; wherein, the anomaly analysis result represents the number and type of modalities in which the target object has physiological-signal anomalies; Generate the monitoring measure according to the anomaly analysis result.

[0012] In a second aspect, an embodiment of the present application provides an intelligent infant monitoring device based on multimodal monitoring, and the device includes: A vibration feature acquisition module, configured to acquire vibration physiological-image data of a target object; wherein, the vibration physiological-image data is the wave optical information characterizing the physiological vibrations of the target object; A vibration signal extraction module, configured to perform signal component extraction on the vibration physiological-image data to obtain the vibration physiological-signal signal of the target object; A combined physiological-signal analysis module, configured to perform combined physiological-signal analysis on the target object according to the vibration physiological-signal signal and the visual physiological-image data of the target object to obtain the physiological-signal signal of the target object; wherein, the visual physiological-image data is the visual imaging information characterizing the physiological-signal signal of the target object; A modal anomaly analysis module is used to obtain the physiological behavior signals of the target object; perform multi-modal anomaly analysis on the target object according to the physiological sign signals and the physiological behavior signals, and generate guardianship measures for the target object.

[0013] In a third aspect, an embodiment of the present application provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the method described in any one of the above embodiments.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the method described in any one of the above embodiments.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions, and the computer instructions are used to cause a computer to execute the method described in any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0017] Figure 1 It is a step diagram of the infant intelligent guardianship method based on multi-modal monitoring provided by the embodiment of the present application; Figure 2 It is a step diagram of obtaining physiological sign signals in the embodiment of the present application; Figure 3 It is a step diagram of obtaining physiological sign features in the embodiment of the present application; Figure 4 It is a step diagram of obtaining vibration sign features in the embodiment of the present application; Figure 5 It is a step diagram of obtaining a composite phase signal in the embodiment of the present application; Figure 6 It is a step diagram of obtaining vibration physiological image data in the embodiment of the present application; Figure 7a It is a structural schematic diagram of the optoelectronic sensing device in the embodiment of the present application; Figure 7b It is a usage schematic diagram of the optoelectronic sensing device in the embodiment of the present application; Figure 8It is a flowchart for generating guardianship measures in the embodiments of the present application; Figure 9 It is a module diagram of an infant intelligent guardianship device based on multimodal monitoring provided by the embodiments of the present application; Figure 10 It is a schematic structural diagram of a computer device provided by the embodiments of the present application. Detailed implementation manners

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0019] In the field of medical guardianship technology, infants are common guardianship objects. Due to their fragile physiological functions and susceptibility to the external environment, the physiological states of infants often have the problem of large fluctuations. Therefore, long-term and continuous health guardianship of infants is required to reduce the occurrence probability of health and safety accidents. In related technologies, the guardianship process usually monitors data such as the vital signs and health status of the guardianship object in real time, and performs risk analysis based on the monitored data to discover potential health risks of the guardianship object. On this basis, early warnings are also made according to the analysis results to enable timely intervention in the health status of the guardianship object, ensure that the vital signs of the guardianship object are in a stable state, and reduce the impact of potential health risks on the guardianship object.

[0020] When monitoring the guardianship object in real time, the physiological signs of the guardianship object are generally obtained through physiological signal monitoring devices or optical monitoring technologies. Among them, the physiological signal monitoring device can be any one or more of devices such as an electrocardiogram monitoring device, a blood oxygen saturation monitor, a blood pressure monitoring device, a body temperature monitoring device, or a respiration monitoring device. These devices need to establish a physical connection with the guardianship object, and collect signals on the body surface of the guardianship object through electrodes, sensor patches, or other sensing devices to obtain the physiological signs of the guardianship object. However, the above-mentioned sensing devices may cause discomfort to the guardianship object during the long-term health guardianship process. In severe cases, it may cause the guardianship object to have allergies or skin injuries, etc., thereby affecting the health of the guardianship object. At the same time, the sensing device connected to the guardianship object may also restrict the movement of the guardianship object, affect the freedom of movement of the guardianship object, and even affect the sleep quality and comfort of the guardianship object.

[0021] In the technical solution of obtaining the physiological signs of the monitored object through optical monitoring technology, the optical monitoring technology can be technologies such as pulse oximetry (SpO2) monitoring, photoplethysmography (PPG), infrared thermal imaging, or spectral imaging. Exemplarily, one of the optical monitoring technologies is described. Taking photoplethysmography as an example, this technology mainly irradiates the skin surface of the monitored object and obtains the reflected light from the skin surface through an optical sensor, so as to analyze the blood flow under the skin surface, monitor the blood flow changes in real time, and be used to calculate physiological signs such as the pulse or heart rate of the monitored object. It can be understood that the optical monitoring technology has the advantage of being non-invasive, which can reduce the discomfort and infection risk of the monitored object during the monitoring process.

[0022] However, in actual situations, the monitored object is in a lying state during the monitoring process, and the skin surface of the monitored object may be covered by objects such as thick quilts, resulting in the occlusion of the light of the optical monitoring technology and being unable to effectively irradiate the skin surface of the monitored object, affecting the data acquisition during the monitoring process. Similarly, some monitored objects cannot maintain the same posture for a long time during the monitoring process. When the monitored object changes its posture, the angle and path of the light of the optical monitoring technology irradiating on the skin surface of the monitored object may change, and these changes greatly affect the accuracy and stability of the monitoring data.

[0023] In addition, in some special environments with poor lighting conditions, such as low-light environments or at night, problems such as the decline in data quality due to insufficient lighting may occur when using optical monitoring technology to obtain data. Therefore, visible light needs to be used for supplementary lighting to improve the data quality. And nocturnal supplementary lighting is likely to affect the sleep quality of the monitored object, further interfering with and even changing the sleep cycle of the monitored object, and having a negative impact on the health status of the monitored object.

[0024] Based on the above problems, the present application provides an infant intelligent monitoring method, device, equipment, and medium based on multimodal monitoring. The method includes: obtaining vibration physiological image data of a target object; extracting signal components from the vibration physiological image data to obtain vibration sign signals of the target object; performing joint sign analysis on the target object according to the vibration sign signals and visual physiological image data of the target object to obtain physiological sign signals of the target object; obtaining physiological behavior signals of the target object; and performing multimodal anomaly analysis on the target object according to the physiological sign signals and physiological behavior signals to generate monitoring measures for the target object.

[0025] The infant intelligent monitoring method based on multimodal monitoring provided by this application acquires the fluctuating optical information and visual imaging information of the vital sign signals of the target object, and performs joint analysis based on the fluctuating optical information and visual imaging information to obtain the physiological vital sign signals of the target object; abnormal analysis is performed on the target object according to multimodal signals such as the physiological vital sign signals and physiological behavior signals of the target object, and corresponding monitoring measures are generated, so as to intervene in the health status of the target object according to the monitoring measures, ensure that the physiological vital signs of the target object are in a stable state, and guarantee the life safety of the target object.

[0026] Compared with the related technology, this application uses a variety of optical means to acquire data of the target object, and performs joint analysis based on the acquired multi-dimensional optical information to determine the physiological vital sign signals of the target object, thereby overcoming the bottleneck of single monitoring means in physiological vital sign monitoring, improving the accuracy of physiological vital sign signals, and further enhancing the monitoring effect on the target object. In addition, this application also obtains the physiological vital sign signals of the target object through an all-optical fusion framework, reducing the risk of electromagnetic interference and improving the stability and accuracy of the obtained signals.

[0027] The infant intelligent monitoring method based on multimodal monitoring provided in this specification can be applied to monitoring objects that require real-time vital sign monitoring. The above monitoring objects can be infants aged 0 to 1 year old, including newborns within 28 days after birth, etc. It can be understood that the infant intelligent monitoring method based on multimodal monitoring provided in this specification can be applied to different monitoring environments. For example, it can be applied to the neonatal department of a hospital to monitor newborns, or applied to a home environment to monitor physiological vital signs such as the breathing or heart rate of infants, and early warning of abnormal states can be carried out according to the monitoring results.

[0028] According to an embodiment of this application, an embodiment of an infant intelligent monitoring method based on multimodal monitoring is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0029] In this embodiment, an infant intelligent monitoring method based on multimodal monitoring is provided, which can be used for the above-mentioned monitoring objects that require real-time vital sign monitoring. Refer to Figure 1 As shown, the method includes: S100. Acquire the vibration physiological image data of the target object; wherein, the vibration physiological image data is the fluctuating optical information representing the physiological vibration of the target object.

[0030] S200. Extract the signal components from the vibration physiological image data to obtain the vibration vital sign signals of the target object.

[0031] S300. Perform joint sign analysis on the target object based on the vibration sign signal and the visual physiological image data of the target object to obtain the physiological sign signal of the target object; wherein, the visual physiological image data is the visual imaging information representing the sign signal of the target object.

[0032] S400. Obtain the physiological behavior signal of the target object; perform multimodal anomaly analysis on the target object based on the physiological sign signal and the physiological behavior signal, and generate a monitoring measure for the target object.

[0033] Among them, the vibration physiological image data is the wave optics information representing the physiological vibration of the target object. Wave optics imaging is performed according to the physiological vibration of the target object itself to generate corresponding wave optics image data as the vibration physiological image data. The vibration physiological image data may include an interference fringe image sequence obtained by dual-wavelength interference, and image data such as a scattering spectrum obtained by dynamic light scattering. By extracting the signal components from the vibration physiological image data, a vibration sign signal can be obtained. The vibration sign signal can be an independent signal reflecting the physiological state and sign information of the target object, including but not limited to signals such as a respiration signal, a heartbeat signal, a pulse signal, and a blood pressure signal.

[0034] The visual physiological image data is the visual imaging information representing the sign signal of the target object. An optical imaging technique is used to collect an image of a specific skin surface of the target object to generate an image containing target feature data as the visual physiological image data. The target feature data can represent the physiological signs of the target object. It can be understood that the method for generating the visual physiological image data can be any one of the optical monitoring techniques. By performing feature analysis on the visual physiological image data, multiple sign signals can be obtained to reflect the physiological state and sign information of the target object.

[0035] The physiological sign signal can be obtained through joint sign analysis and is a signal used to accurately represent the physiological signs of the target object, including but not limited to signals such as a respiration signal, a heartbeat signal, a pulse signal, and a blood pressure signal. The physiological behavior signal can represent the body activities of the target object during the monitoring process, reflecting the behavior pattern of the target object, so as to reflect the health state of the target object from different modalities.

[0036] Specifically, fluctuation optical imaging is performed according to the physiological vibration of the target object to obtain the vibration physiological image data of the target object. The vibration physiological image data characterizes the physiological vibration of the target object, and the physiological vibration of the target object is generated by the physiological activities of the target object, including heartbeat, breathing, or blood flow, etc. Therefore, the physiological vibration signal extracted from the vibration physiological image data can be a composite signal containing various types of physical sign signals of the target object. According to the physiological vibration signal, various vibration physical sign signals of the target object can be obtained, respectively representing different physiological signs of the target object.

[0037] Furthermore, the visual physiological image data of the target object is obtained, and through feature analysis and signal extraction of the visual physiological image data, the visual physical sign signal can be obtained. It can be understood that according to the acquisition method and acquisition location of the visual physiological image data, the visual physical sign signal can also be a composite signal containing various types of physical sign signals of the target object. Based on the visual physical sign signal, the signal characteristics of various types of physical sign signals can be obtained, including but not limited to frequency, amplitude, and phase, etc. According to the vibration physical sign signal and the obtained signal characteristics, joint physical sign analysis is performed on the target object to obtain the physiological physical sign signal of the target object.

[0038] It should be noted that a single optical monitoring method is extremely vulnerable to environmental and external factors in actual situations and cannot accurately represent the physiological signs of the target object. In addition, the monitoring data obtained according to a single optical monitoring method often can only one-sidedly represent the physiological signs of the target object in a certain aspect. In the case of significant individual differences in the target object, it may lead to a significant deviation between the obtained physiological physical sign signal and the actual signal, affecting the accuracy of the physiological physical sign signal. Compared with the related technologies, the present application uses multiple optical means to obtain data of the target object, obtaining the image data of the target object under multiple optical dimensions, thereby overcoming the bottleneck of a single monitoring method in physiological sign monitoring, being able to combine the image data under multiple optical dimensions to achieve all-round physiological sign analysis of the target object, improving the monitoring accuracy and monitoring effect, and further enhancing the monitoring effect on the target object.

[0039] In some embodiments, the method for obtaining visual physiological image data may be infrared thermography, and the visual physiological image data obtained thereby may represent the body temperature distribution of the target object. The process of combined sign analysis may include: transforming the vibration sign signal into the frequency domain, at which time the vibration sign signal exhibits multiple signal peaks corresponding to different frequencies; extracting features from the vibration sign signal to obtain vibration signal features, including the frequencies and peak values corresponding to the signal peaks; obtaining the temperature distribution features of the target object based on the visual physiological image data, including the temperature distribution and distribution gradient of the target object; establishing a sign feature vector of the target object at the current moment according to the vibration signal features and the temperature distribution features; comparing the sign feature vectors at different moments, and extracting the sign feature vectors with errors exceeding a preset error threshold as abnormal feature vectors; performing abnormal analysis on the target object according to the vibration signal features and temperature distribution features in the abnormal feature vectors to determine the specific location where the target object has an abnormality.

[0040] It can be understood that physiological vibrations will be generated at different positions of the target object due to physiological activities, and the frequencies and amplitudes of the physiological vibrations are certain, and the vibration signal features obtained thereby are also certain, that is, the peak values of the signal peaks corresponding to each frequency are fixed. These physiological activities may be activities such as blood flow, which form subcutaneous microcirculation under the skin of the target object to maintain the normal physiological activities of the target object. When the subcutaneous microcirculation state at a certain position of the target object is abnormal, the physiological vibration at the abnormal position deviates from the normal physiological vibration, resulting in an error in the vibration signal features in the sign feature vector. The physiological vibration features at the abnormal position can be determined according to the vibration signal features with errors. At the same time, the specific part where the target object has an abnormality can also be determined according to the temperature distribution features in the corresponding sign feature vector. Therefore, by combining the temperature distribution features of the target object at this time and the physiological sign signals at the abnormal position, specific abnormal analysis can be performed on the target object, so as to be able to generate targeted monitoring measures and improve the monitoring effect on the target object.

[0041] Furthermore, monitor the behavior of the target object to obtain the physiological behavior signal of the target object. Perform multimodal abnormal analysis on the target object according to the physiological sign signal and the physiological behavior signal. When it is detected that the health state of the target object is abnormal, the specific abnormal situation of the target object can also be analyzed according to the physiological sign signal and the physiological behavior signal, and targeted monitoring measures can be generated according to the specific abnormal situation to perform timely intervention on the target object to ensure that the vital signs of the target object are in a stable state and reduce the impact of abnormal situations on the target object.

[0042] The infant intelligent monitoring method based on multimodal monitoring provided in this embodiment acquires the fluctuating optical information and visual imaging information of the vital sign signals of the target object, and performs joint analysis based on the fluctuating optical information and visual imaging information to obtain the physiological sign signals of the target object; performs anomaly analysis on the target object according to multimodal signals such as the physiological sign signals and physiological behavior signals of the target object, generates corresponding monitoring measures, and intervenes in the health status of the target object according to the monitoring measures to ensure that the physiological signs of the target object are in a stable state and guarantee the life safety of the target object.

[0043] Compared with the related technologies, this application uses multiple optical means to acquire data of the target object, and performs joint analysis based on the acquired multi-dimensional optical information to determine the physiological sign signals of the target object, thereby overcoming the bottleneck of single monitoring means in physiological sign monitoring, improving the accuracy of physiological sign signals, and further enhancing the monitoring effect on the target object. In addition, this application also acquires the physiological sign signals of the target object through an all-optical fusion architecture, reducing the risk of electromagnetic interference and enhancing the stability and accuracy of the obtained signals.

[0044] Refer to Figure 2 As shown in the figure, as an embodiment of this application, joint sign analysis is performed on the target object according to the vibration sign signals and the visual physiological image data of the target object to obtain the physiological sign signals of the target object, including: S310. Perform sign feature analysis on the target object based on the visual physiological image data to obtain the physiological sign features of the target object; wherein, the physiological sign features are the signal features of the visual sign signals of the target object, and the visual sign signals are the sign signals of the target object obtained from the visual physiological image data.

[0045] S320. Perform feature matching on the vibration sign signals according to the physiological sign features, and extract the physiological sign signals from the vibration sign signals.

[0046] Specifically, use optical imaging technology to collect images of the specific skin surface of the target object, generate an image containing target feature data as the visual physiological image data, and the visual sign signals can be obtained according to the visual physiological image data. It can be understood that according to the acquisition method and acquisition location of the visual physiological image data, the visual sign signals can be composite signals containing various types of sign signals of the target object. Therefore, according to the types of different sign signals and their corresponding features, perform feature analysis on the visual sign signals, and the signal features corresponding to different sign signals can be extracted from the visual sign signals as the physiological sign features.

[0047] Exemplarily illustrate the process of obtaining physiological sign features. By using optical imaging technology to collect images of the face of a target object, images of the temperature and movement of the key organs of the target object changing periodically with physiological activities, and images of the facial color of the target object changing periodically with physiological activities can be obtained. Analyze the above images. According to the periodic change trend of the corresponding features in the images, various sign signals of the target object can be obtained as physiological sign features.

[0048] Further, in this embodiment, the process of performing joint sign analysis on the target object according to the vibration sign signal and the visual physiological image data of the target object includes: performing feature matching on the vibration sign signal according to the physiological sign features, and extracting the physiological sign signal from the vibration sign signal.

[0049] It should be noted that when extracting signal components from the vibration physiological image data, since the physiological activities of the target object are complex and multi-level, multiple corresponding signals may be obtained for any type of sign signal, but in actual situations, it is impossible to determine the true signal from multiple signals. The physiological sign features contain various types of signal features, and the signal features respectively correspond to different types of sign signals, and the types of these sign signals can be the same as the types of sign signals included in the vibration sign signal. In the case where the sign signal types are the same, the sign signals in the vibration sign signal can be subjected to feature matching according to the signal features in the physiological sign features, so as to determine the true signal from the multiple sign signals corresponding to any sign signal type, thereby extracting the physiological sign signal and improving the accuracy of the physiological sign signal.

[0050] Refer to Figure 3 As shown, as an embodiment of the present application, performing sign feature analysis on the target object based on the visual physiological image data to obtain the physiological sign features of the target object includes: S312. Perform signal component analysis on the visual physiological image data and extract visual sign signals.

[0051] S314. Perform signal feature extraction on the visual sign signals to obtain physiological sign features.

[0052] Specifically, the process of signal component analysis may include: performing image analysis according to the visual physiological image data and extracting visual sign signals therefrom. The visual sign signals can be composite signals containing various types of sign signals. According to the actual features of different types of sign signals, the visual sign signals can be signal-separated to obtain multiple independent signals corresponding to different types of physiological signs.

[0053] Furthermore, signal feature extraction is performed on multiple independent signals in the visual sign signal to obtain signal features corresponding to different types of sign signals, which are used as physiological sign features. It should be noted that when the environmental conditions of the environment where the target object is located meet the preset requirements, such as good environmental lighting conditions, etc., the credibility of the physiological sign features is higher than that of the signal features in the vibration sign signal. Therefore, the vibration sign signal can be feature-matched according to the physiological sign features to extract the physiological sign signal from the vibration sign signal, thereby improving the accuracy of the physiological sign signal. It can be understood that when the environmental conditions of the environment where the target object is located cannot meet the preset requirements, the credibility of the physiological sign features is lower than that of the signal features in the vibration sign signal. In this case, the feature matching step can be ignored.

[0054] Referring Figure 4 As shown, as an embodiment of the present application, the vibration sign signal includes multiple sign signals of the target object; signal component extraction is performed on the vibration physiological image data to obtain the vibration sign signal of the target object, including: S210. Perform phase recovery according to the vibration physiological image data to obtain the composite phase signal of the target object; wherein, the composite phase signal is a fusion signal of the phase signals corresponding to multiple sign signals.

[0055] S220. Perform frequency separation on the composite phase signal to obtain the sign phase signal corresponding to any one of the multiple sign signals.

[0056] S230. Perform signal reconstruction respectively according to the sign phase signal to obtain the corresponding sign signal.

[0057] S240. Obtain the vibration sign signal according to all the sign signals.

[0058] Specifically, the vibration physiological image data represents various physiological vibrations of the target object. The various physiological vibrations respectively correspond to their own sign signals, and different types of sign signals are fused with each other in the vibration physiological image data to form a vibration composite signal. Therefore, it is necessary to extract different types of sign signals from the vibration physiological image data through signal component extraction.

[0059] Furthermore, perform phase recovery on the vibration composite signal according to the vibration physiological image data to obtain the composite phase signal of the target object. The composite phase signal can be a continuous phase distribution signal, which is used to represent the phase information of the vibration composite signal. On this basis, according to the actual characteristics of different types of sign signals, perform signal separation on the composite phase signal to obtain multiple independent phase signals corresponding to different types of sign signals, which are used as sign phase signals.

[0060] Exemplarily, the method of frequency separation can be frequency-domain filtering separation. First, perform a fast Fourier transform on the composite phase signal to transform the composite phase signal into the frequency domain and obtain the corresponding spectral distribution. Secondly, design a corresponding band-pass filter according to the actual characteristics of the physiological sign signal to extract the corresponding physiological sign signal from the spectral distribution. Common physiological sign signals include respiratory signals and heartbeat signals. Among them, the frequency range of the respiratory signal is about 0.1 Hz to 0.5 Hz, and the frequency range of the heartbeat signal is about 1 Hz to 2 Hz. According to the above frequency ranges, a band-pass filter with a cut-off frequency of 0.05 Hz to 0.8 Hz can be designed for the respiratory signal, and a band-pass filter with a cut-off frequency of 0.8 Hz to 3 Hz can be designed for the heartbeat signal. The composite phase signal is respectively subjected to signal extraction to obtain the respiratory signal and the heartbeat signal. At this time, both the respiratory signal and the heartbeat signal are in the frequency domain. Perform an inverse Fourier transform on the respiratory signal and the heartbeat signal respectively to transform them back to the time domain, so as to obtain the separated respiratory signal and heartbeat signal.

[0061] Furthermore, the optical parameters for generating the vibration physiological image data can be preset. According to the preset optical parameters and the physiological sign phase signal, signal reconstruction is respectively performed on different types of physiological signs to obtain the corresponding physiological sign signals. After obtaining all types of physiological sign signals, the vibration physiological sign signal is obtained according to all the physiological sign signals to achieve the complete separation of the vibration composite signal.

[0062] Refer to Figure 5 As shown, as an embodiment of the present application, performing phase recovery on the vibration physiological image data to obtain the composite phase signal of the target object includes: S212. Perform phase extraction on the vibration physiological image data to obtain the initial phase signal of the target object.

[0063] S214. Perform phase unwrapping on the initial phase signal to obtain the composite phase signal.

[0064] Specifically, use a phase extraction algorithm to perform phase extraction on the vibration physiological image data to obtain the initial phase corresponding to each pixel point in the vibration physiological image data, and further obtain the initial phase signal of the target object. Exemplarily, the phase extraction algorithm can be a method based on Fourier transform or a phase-shifting interference method, etc. When the vibration physiological image data is an interference fringe image sequence obtained by interference, the number of initial phase signals can be two or more and respectively correspond to lights of different wavelengths. At this time, all the initial phase signals are jointly synthesized into a composite phase, corresponding to the phase information of the composite phase signal.

[0065] Further, a phase unwrapping algorithm is used to perform phase unwrapping on the initial phase signal to continuously correct the phase of the initial phase signal and obtain a composite phase signal. Exemplarily, when the vibration physiological image data is an interference fringe image sequence obtained by interference, the phase unwrapping algorithm can be a path tracking method. The interference fringe path in the vibration physiological image data is tracked, and the phases of adjacent pixel points are compared. If a phase jump occurs between adjacent pixel points, the pixel points with the phase jump are phase-corrected to eliminate the phase jump in the initial phase signal, improve the continuity of the initial phase signal, and obtain a composite phase signal.

[0066] Referring to Figure 6 As an embodiment of the present application shown, obtaining vibration physiological image data of a target object includes: S110. Using an optical sensing device to collect physiological vibrations of the target object and obtain the physiological vibration signal of the target object.

[0067] S120. Performing signal imaging based on the physiological vibration signal to obtain vibration physiological image data.

[0068] Referring to Figure 7a , the structure of the optical sensing device can be as Figure 7a shown. The optical sensing device includes an outer layer, a sensing optical fiber, a laser light source, and an optoelectronic interface. Among them, the outer layer can be a medical silicone pad, which has the characteristics of being ultra-thin and flexible and can quickly respond to external vibrations. The sensing optical fiber is embedded inside the outer layer. The sensing optical fiber can be a multi-array multi-core optical fiber and is arranged in a serpentine wiring form in the outer layer. The spatial resolution of the sensing optical fiber is improved through the serpentine wiring. The sensing optical fiber contacts the object attached to the surface of the outer layer through the outer layer, and changes the propagation characteristics of the light in the sensing optical fiber in response to the vibration of the object, thereby monitoring the vibration of the object. Both ends of the sensing optical fiber are respectively connected to the laser light source and the optoelectronic interface. The laser light source is used to provide light source input for the sensing optical fiber. In some embodiments, the laser light source can also be arranged outside the optical sensing device to reduce the complexity in the optical sensing device. The optoelectronic interface is used to receive the light source output in the sensing optical fiber and transmit the light source output to an external imaging device to obtain vibration physiological image data. Exemplarily, the external imaging device can be an interferometer, and at this time, the vibration physiological image data can be an interference fringe image sequence obtained by interference.

[0069] Referring to Figure 7b shown, as Figure 7aThe optical sensing device shown can be laid flat on the bed where the target object is located, and a mattress can be laid above the optical sensing device to improve the comfort of the target object. On this basis, the target object is located above the mattress, corresponding to the area in the optical sensing device where the sensing optical fiber is embedded. During the monitoring process, the target object generates physiological vibrations due to physiological activities. The physiological vibrations are transmitted to the optical sensing device through the mattress and collected by the optical sensing device to obtain the physiological vibration signal of the target object. The physiological vibration signal is transmitted to an external imaging device through an optoelectronic interface, and signal imaging is performed in the external imaging device to obtain vibration physiological image data.

[0070] Compared with the technical solutions in the related art that obtain the physiological signs of the monitored object through physiological signal monitoring devices or optical monitoring technologies, the optical sensing device in this application does not need to be in direct contact with the target object, realizes unconstrained physiological sign monitoring, improves the comfort of the target object during the monitoring process, and at the same time reduces the impact on the health of the monitored object. In addition, since this application directly collects the physiological vibrations of the target object through the sensing optical fiber instead of performing visual imaging through optical monitoring technologies, it will not be affected by light occlusion, and there is no need to supplement light to the target object during the monitoring process, reducing the impact on the sleep quality of the target object.

[0071] Refer to Figure 8 As shown, as an embodiment of this application, the physiological behavior signal includes the behavior signals of the target object in multiple modalities; multi-modal anomaly analysis is performed on the target object according to the physiological sign signal and the physiological behavior signal to generate monitoring measures for the target object, including: S410. Perform anomaly analysis on the physiological sign signal and the physiological behavior signal respectively to obtain the anomaly analysis result of the target object; wherein, the anomaly analysis result represents the number of modalities and the modality types in which the target object has sign anomalies.

[0072] S420. Generate monitoring measures according to the anomaly analysis result.

[0073] Specifically, the behavior of the target object is monitored to obtain the physiological behavior signal of the target object. The physiological behavior signal of the target object may include body movement signals and sound signals, etc. Among them, the body movement signal may be a conscious or unconscious body movement of the target object due to its own health status. Whether the body movement signal is abnormal can indicate whether the health status of the target object is abnormal. To illustrate the body movement signal exemplarily, in the case where the target object is a newborn, if the target object suddenly makes large and continuous body movements, it may indicate that the target object has an abnormality and needs to be intervened in time.

[0074] As an example, the body movement signal can be obtained by tracking the body movement of the target object through a camera. According to the image data or video data collected by the camera, multiple corresponding joint points are calibrated through a joint point calibration algorithm, and the body movement signal is determined according to the displacement degree of the joint points. The above camera can collect data using visible light or near-infrared light. Among them, the camera using visible light can collect data on the target object during the day, and the camera using near-infrared light can collect data on the target object at night.

[0075] Similarly, the sound signal can be the sound consciously or unconsciously emitted by the target object due to its own health status. The sound signal can include a normal sound signal and an abnormal sound signal. Exemplarily, the normal sound signal can be a sound signal such as the normal conversation or breathing sound of the target object, indicating that the target object is in a normal health state; the abnormal sound signal can include groans, etc., indicating that the target object is in pain or discomfort, that is, the health state of the target object is abnormal.

[0076] As an example, the sound signal can be collected through a microphone array and obtained by signal extraction through a voiceprint feature extraction model. Exemplarily, the voiceprint feature extraction model can combine Mel Frequency Cepstral Coefficients (MFCC) with a Gaussian Mixture Model (GMM), and use a high-pass filter to pre-emphasize the collected signal to optimize the frequency of the sound signal, making the spectrum of the sound signal flatter and compensating for the suppressed high-frequency part.

[0077] Furthermore, multi-modal anomaly analysis is performed on the target object according to the physiological sign signal and the physiological behavior signal. The health state of the target object is analyzed in multiple modalities to determine the number and type of modalities in which the target object shows sign anomalies. When the number and type of modalities in which the target object shows sign anomalies meet the preset anomaly conditions, it is determined that the health state of the target object is abnormal. Analyze the specific anomalies of the target object according to the physiological sign signal and the physiological behavior signal, and generate targeted monitoring measures according to the specific anomalies to timely intervene in the target object to ensure that the vital signs of the target object are in a stable state and reduce the impact of anomalies on the target object.

[0078] Exemplarily, abnormal analysis is performed on the physiological sign signals and various physiological behavior signals of the target object to determine whether the modalities corresponding to the above signals are abnormal. When a single-modal abnormality occurs in the target object, an audible and visual alarm can be combined with alarm devices in the environment where the target object is located to remind medical staff to intervene in the target object. When a double-modal abnormality occurs in the target object, on the basis of the audible and visual alarm, the abnormal state of the target object is pushed to the medical staff terminal to quickly notify the medical staff of the abnormal state. When a triple-modal or multi-modal abnormality occurs in the target object, it is necessary to compare the physiological sign signals of the target object at the current moment with the physiological sign signals at the historical moment. If the physiological sign signals at the current moment deviate significantly from the physiological sign signals at the historical moment, it indicates that the target object is in a serious abnormal state, and it is necessary to initiate first aid linkage to quickly intervene in the target object to ensure the life safety of the target object.

[0079] Correspondingly, please refer to Figure 9 , an infant intelligent monitoring device based on multi-modal monitoring provided by an embodiment of the present application includes: A vibration feature acquisition module 910, configured to acquire vibration physiological image data of a target object; wherein, the vibration physiological image data is fluctuating optical information characterizing the physiological vibration of the target object.

[0080] A vibration signal extraction module 920, configured to perform signal component extraction on the vibration physiological image data to obtain vibration sign signals of the target object.

[0081] A combined sign analysis module 930, configured to perform combined sign analysis on the target object according to the vibration sign signals and the visual physiological image data of the target object to obtain physiological sign signals of the target object; wherein, the visual physiological image data is visual imaging information characterizing the sign signals of the target object.

[0082] A modality abnormality analysis module 940, configured to acquire physiological behavior signals of the target object; perform multi-modal abnormality analysis on the target object according to the physiological sign signals and the physiological behavior signals, and generate monitoring measures for the target object.

[0083] In some alternative embodiments, the combined sign analysis module 930 includes: A sign feature analysis unit, configured to perform sign feature analysis on the target object based on the visual physiological image data to obtain physiological sign features of the target object; wherein, the physiological sign features are signal features of the visual sign signals of the target object, and the visual sign signals are sign signals of the target object obtained from the visual physiological image data.

[0084] A sign signal extraction unit, configured to perform feature matching on the vibration sign signals according to the physiological sign features, and extract physiological sign signals from the vibration sign signals.

[0085] In some alternative embodiments, the physical sign feature analysis unit includes: A physical sign signal separation sub-unit, configured to perform signal component analysis on the visual physiological image data and extract visual physical sign signals.

[0086] A signal feature extraction sub-unit, configured to perform signal feature extraction on the visual physical sign signals to obtain physiological physical sign features.

[0087] In some alternative embodiments, the vibration physical sign signals include multiple physical sign signals of the target object; the vibration signal extraction module 920 includes: A phase signal restoration unit, configured to perform phase restoration according to the vibration physiological image data to obtain a composite phase signal of the target object; wherein, the composite phase signal is a fusion signal of the phase signals corresponding to the multiple physical sign signals.

[0088] A phase signal separation unit, configured to perform frequency separation on the composite phase signal to obtain a physical sign phase signal corresponding to any one of the multiple physical sign signals.

[0089] A vibration signal reconstruction unit, configured to perform signal reconstruction respectively according to the physical sign phase signals to obtain corresponding physical sign signals.

[0090] A physical sign signal acquisition unit, configured to obtain vibration physical sign signals according to all the physical sign signals.

[0091] In some alternative embodiments, the phase signal restoration unit includes: A phase extraction sub-unit, configured to perform phase extraction on the vibration physiological image data to obtain an initial phase signal of the target object.

[0092] A phase unwrapping sub-unit, configured to perform phase unwrapping on the initial phase signal to obtain a composite phase signal.

[0093] In some alternative embodiments, the vibration feature acquisition module 910 includes: A vibration acquisition unit, configured to use an optical sensing device to perform physiological vibration acquisition on the target object and obtain physiological vibration signals of the target object.

[0094] A signal imaging unit, configured to perform signal imaging according to the physiological vibration signals to obtain vibration physiological image data.

[0095] In some alternative embodiments, the modal anomaly analysis module 940 includes: An anomaly analysis unit, configured to perform anomaly analysis on the physiological physical sign signals and physiological behavior signals respectively to obtain an anomaly analysis result of the target object; wherein, the anomaly analysis result represents the number of modalities and the modality types in which the target object has physical sign anomalies.

[0096] A measure generation unit, configured to generate monitoring measures according to the abnormal analysis result.

[0097] The further function descriptions of the above-mentioned modules and units are the same as those in the corresponding embodiments above, and will not be repeated here.

[0098] The intelligent infant monitoring device based on multimodal monitoring in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0099] Please refer to Figure 10 , Figure 10 , which is a schematic structural diagram of a computer device provided by an embodiment of the present application. As shown in the figure, the computer device includes: one or more processors 10, a memory 20, and an interface for connecting each component, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (such as an array of servers, a set of blade servers, or a multi-processor system). Figure 10 In

[0100] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device can be a complex programmable logic device, a field programmable gate array, a general array logic, or any combination thereof.

[0101] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0102] The memory 20 may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function. The data storage area may store data created according to the use of the computer device and the like. In addition, the memory 20 may include a high-speed random access memory and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 may optionally include a memory remotely disposed relative to the processor 10, and these remote memories may be connected to the computer device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0103] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state drive; the memory 20 may further include a combination of the above types of memories.

[0104] The computer device further includes a communication interface 30 for communicating the computer device with other devices or communication networks.

[0105] The embodiments of the present application further provide a computer-readable storage medium. The methods according to the embodiments of the present application may be implemented in hardware, firmware, or may be implemented as computer code recorded on a storage medium, or may be implemented as computer code originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and to be stored in a local storage medium, so that the methods described herein may be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium may also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component capable of storing or receiving software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0106] The embodiments of the present application provide a computer program product. The computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods of any embodiment of the present application.

[0107] Although the embodiments of the present application are described with reference to the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations fall within the scope defined by the appended claims.

[0108] The systems, devices, modules or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0109] For the convenience of description, the above devices are described by dividing them into various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0110] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0112] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions in Figure 1 one process or multiple processes and / or blocks Figure 1The functions specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide for implementing the steps of the functions specified in one Figure 1 process or more processes and / or boxes Figure 1 or more boxes.

[0114] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising said element.

[0115] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference can be made to the relevant part of the method embodiment for the related content.

[0116] The above are only the embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

[0117] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. An intelligent infant monitoring method based on multimodal monitoring, characterized in that, The method includes: Obtaining vibration physiological image data of a target object; wherein, the vibration physiological image data is fluctuation optical information characterizing the physiological vibration of the target object; Performing signal component extraction on the vibration physiological image data to obtain a vibration sign signal of the target object; Performing joint sign analysis on the target object according to the vibration sign signal and visual physiological image data of the target object to obtain a physiological sign signal of the target object; wherein, the visual physiological image data is visual imaging information characterizing the sign signal of the target object; Obtaining a physiological behavior signal of the target object; performing multi-modal anomaly analysis on the target object according to the physiological sign signal and the physiological behavior signal, and generating a monitoring measure for the target object.

2. The method according to claim 1, wherein The performing joint sign analysis on the target object according to the vibration sign signal and visual physiological image data of the target object to obtain a physiological sign signal of the target object includes: Performing sign feature analysis on the target object based on the visual physiological image data to obtain physiological sign features of the target object; wherein, the physiological sign features are signal features of the visual sign signal of the target object, and the visual sign signal is the sign signal of the target object obtained from the visual physiological image data; Performing feature matching on the vibration sign signal according to the physiological sign features, and extracting the physiological sign signal from the vibration sign signal.

3. The method according to claim 2, characterized in that, The performing sign feature analysis on the target object based on the visual physiological image data to obtain physiological sign features of the target object includes: Performing signal component analysis on the visual physiological image data to extract the visual sign signal; Performing signal feature extraction on the visual sign signal to obtain the physiological sign features.

4. The method according to claim 1, wherein The vibration sign signal includes multiple sign signals of the target object; the performing signal component extraction on the vibration physiological image data to obtain a vibration sign signal of the target object includes: Performing phase recovery according to the vibration physiological image data to obtain a composite phase signal of the target object; wherein, the composite phase signal is a fusion signal of phase signals corresponding to the multiple sign signals; Performing frequency separation on the composite phase signal to obtain a sign phase signal corresponding to any one of the multiple sign signals; Performing signal reconstruction respectively according to the sign phase signal to obtain corresponding sign signals; Obtaining the vibration sign signal according to all the sign signals.

5. The method according to claim 4, characterized in that, The performing phase recovery according to the vibration physiological image data to obtain a composite phase signal of the target object includes: Performing phase extraction on the vibration physiological image data to obtain an initial phase signal of the target object; Performing phase unwrapping on the initial phase signal to obtain the composite phase signal.

6. The method according to any one of claims 1 to 5, characterized in that, The obtaining vibration physiological image data of a target object includes: Using an optical sensing device to collect physiological vibration of the target object to obtain a physiological vibration signal of the target object; Perform signal imaging based on the physiological vibration signal to obtain the vibration physiological image data.

7. The method according to any one of claims 1 to 5, characterized in that, The physiological behavior signal includes the behavior signals of the target object in multiple modalities; the multi-modal anomaly analysis of the target object according to the physiological sign signal and the physiological behavior signal to generate a monitoring measure for the target object includes: Perform anomaly analysis on the physiological sign signal and the physiological behavior signal respectively to obtain the anomaly analysis result of the target object; wherein, the anomaly analysis result represents the number and type of modalities in which the target object has sign anomalies. Generate the monitoring measure according to the anomaly analysis result.

8. An intelligent infant monitoring device based on multimodal monitoring, characterized in that, The device includes: A vibration feature acquisition module for acquiring vibration physiological image data of a target object; wherein, the vibration physiological image data is the wave optical information characterizing the physiological vibration of the target object. A vibration signal extraction module for extracting signal components from the vibration physiological image data to obtain the vibration sign signal of the target object. A combined sign analysis module for performing combined sign analysis on the target object according to the vibration sign signal and the visual physiological image data of the target object to obtain the physiological sign signal of the target object; wherein, the visual physiological image data is the visual imaging information characterizing the sign signal of the target object. A modality anomaly analysis module for acquiring the physiological behavior signal of the target object; performing multi-modal anomaly analysis on the target object according to the physiological sign signal and the physiological behavior signal to generate a monitoring measure for the target object.

9. A computer device, characterized in that, Includes: A memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Computer instructions are stored on the computer-readable storage medium, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.

Images