Infant nursing method and device based on multi-modal data analysis

Abstract

The invention discloses an infant nursing method and device based on multi-modal data analysis. The method comprises the steps that multi-mode monitoring data and caregiver chief complaint data related to a nursed infant are acquired, an infant nursing model is input, a nursing request is triggered through the infant nursing model, a caregiver nurses a target object, and the nursing request comprises the caregiver and the content and time of nursing operation. According to the method, an artificial intelligence technology is creatively applied to an infant nursing scene, and the evolution process in each physiological cycle of eating, sleeping, waking and the like of an infant is accurately tracked by utilizing environment information such as images and sound dynamically collected in the growth process and physiological information such as pulse, heartbeat and breath; the physiological characteristics and energy operation preference of each infant are known, more accurate nursing is used for assisting the growth of the infants, and scientific nursing guidance and prediction supported by physiological behaviors such as feeding and sleep inducing under the driving of physiological signal characteristic values of the infants are realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an infant care method and apparatus based on multimodal data analysis. Background Technology

[0002] Currently, infant and toddler care technology mainly relies on traditional methods and is heavily dependent on the experience level of caregivers. While some more established infant and toddler care centers have improved the standardization of care through innovative management methods such as establishing care paradigms and standards, these methods still remain largely manual and lack solutions for achieving intelligent infant and toddler care using advanced technologies such as artificial intelligence and big data.

[0003] It should be noted that the statements herein provide only background information relevant to this application and do not necessarily constitute prior art. Summary of the Invention

[0004] This application provides an infant care method and device based on multimodal data analysis, offering a solution for intelligent infant care that rationally utilizes artificial technology. In different embodiments, the combined effects of caregiver differences, care environment response, observer influence, and the use of care tools can be comprehensively considered to achieve a holographic operational system that meets the needs of infants' individualized development and their physiological activities, encompassing historical physiological information evolution.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide an infant care method based on multimodal data analysis, comprising: acquiring multimodal monitoring data and caregiver complaint data related to a target object, wherein the target object is an infant to be cared for; inputting the multimodal monitoring data and caregiver complaint data into an infant care model, triggering a care request through the infant care model to enable a caregiver to provide care to the target object, wherein the care request includes the caregiver, the content of the care operation, and the time; the infant care model is specifically used for: extracting features from the multimodal monitoring data to obtain physiological features characterizing the physiological state of the target object; performing similarity recognition on the physiological features according to a constructed baseline feature space to predict the physiological needs of the target object; each baseline feature in the baseline feature space characterizes various physiological needs of the infant; triggering a care request according to a constructed nursing norm field, the caregiver complaint data, and the physiological need prediction results; the nursing norm field includes multiple nursing norm models, which characterize the mapping relationship between nursing operation features and physiological features when various physiological needs of the infant are reasonably met.

[0007] In some embodiments, the multimodal monitoring data includes: first monitoring data collected by a physiological information acquisition device and second monitoring data collected by an environmental information acquisition device.

[0008] In some embodiments, the step of extracting features from the multimodal monitoring data to obtain physiological features characterizing the physiological state of the target object includes: extracting a first physiological feature and a first behavioral feature of the target object based on the first monitoring data; extracting a second physiological feature and a second behavioral feature of the target object based on the second monitoring data; using the first behavioral feature and the second behavioral feature as behavioral indexing information, fusing the first physiological feature and the second physiological feature based on the behavioral indexing information to obtain the physiological features characterizing the physiological state of the target object.

[0009] In some embodiments, the multimodal monitoring data related to the target object includes: multimodal monitoring data of the target object and multimodal monitoring data of the caregiver; the infant care method based on multimodal data analysis further includes: during the process of the caregiver providing care to the target object, the infant care model determines nursing operation characteristics based on the multimodal monitoring data related to the target object and the caregiver's subjective complaint data, and determines nursing optimization suggestions based on the nursing norm field, the nursing operation characteristics, and the physiological characteristics obtained during the nursing process.

[0010] In some embodiments, the infant care method based on multimodal data analysis further includes: during the process of the caregiver caring for the target object, if the infant care model identifies physiological characteristics that match the target artificial intervention scenario from the multimodal monitoring data, then it generates real-time intervention suggestions corresponding to the target artificial intervention scenario.

[0011] In some embodiments, the infant care model is trained as follows: acquiring infant multimodal monitoring data required for training as training data, wherein the infant multimodal monitoring data includes annotation information; using a base model to perform iterative training on the training data for feature extraction and physiological need identification until an infant care model containing the baseline feature space and the nursing norm field is obtained.

[0012] In some embodiments, the infant care method based on multimodal data analysis further includes: generating training data containing positive feedback information and / or negative feedback information based on multimodal monitoring data collected during the care process and the caregiver's subjective complaints submitted by the caregiver regarding the care process; training the infant care model based on the training data; and updating the baseline feature space and / or care norm field.

[0013] In some embodiments, training the infant care model based on the training data and updating the baseline feature space and / or nursing norm field includes: training a problem feature field based on the training data, wherein the problem feature field represents the mapping relationship between nursing operation features and physiological features when various physiological needs of infants are not properly met; and performing adversarial training based on the problem feature field and the nursing norm field.

[0014] In some embodiments, the nursing norm field includes reasonable rhythmic information of various physiological needs, and the problem characteristic field includes abnormal rhythmic information of various physiological needs; the confrontation training based on the problem characteristic field and the nursing norm field includes: adjusting the reasonable rhythmic information according to the normal growth rhythm of infants and young children and the abnormal rhythmic information, so that the updated nursing norm field adapts to the growth of the target object.

[0015] In some embodiments, the infant care method based on multimodal data analysis further includes: constructing a caregiver feature profile for each caregiver based on the caregiver's characteristics during the care process; the care request corresponds to a first care operation, and triggering the care request includes: calculating a matching degree based on the caregiver feature profile and the care normative field, determining the first caregiver with the highest matching degree to the first care operation, and triggering a care request for the first caregiver.

[0016] In some embodiments, the infant care method based on multimodal data analysis further includes: receiving care request feedback information, the care request feedback information indicating that the first caregiver is unable to provide care as expected; and determining an alternative care request based on the care request feedback information and the urgency of the care operation, the alternative care request being directed to a second caregiver who is different from the first caregiver.

[0017] In some embodiments, the alternative care request corresponds to the first care action; or, the alternative care request corresponds to a second care action, wherein the second caregiver is the caregiver with the highest matching degree to the second care action.

[0018] In some embodiments, the first nursing operation includes multiple third nursing operations at adjacent time points, and the triggering of the nursing request includes: determining the third caregiver with the highest comprehensive matching degree with the third nursing operation based on the caregiver feature profile and the nursing norm field, and arranging the nursing operations, and triggering a nursing request for the third caregiver according to the arrangement result of parallel execution and / or sequential execution.

[0019] Secondly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the infant care methods based on multimodal data analysis in embodiments of this application.

[0020] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of any of the infant care methods based on multimodal data analysis in embodiments of this application.

[0021] Fourthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of any of the infant care methods based on multimodal data analysis in embodiments of this application.

[0022] The above-mentioned technical solutions adopted in this application embodiment can achieve the following beneficial effects: Multimodal monitoring data of the infant to be cared for and the caregiver's subjective complaint data are input into the infant care model, enabling the infant care model to determine the infant's multidimensional physiological characteristics and predict physiological needs based on the received data, triggering a care request so that the caregiver can provide care for the infant. This method innovatively applies artificial intelligence technology to the infant care scenario, utilizing dynamically collected environmental information such as images and sounds during growth, as well as physiological information such as pulse, heartbeat, and respiration, to accurately track the evolutionary process of infants in various physiological cycles such as eating, sleeping, and waking. It understands each infant's own physiological characteristics and energy preferences, using more precise care to support the baby's growth, and achieving scientific nursing guidance and prediction supported by physiological behaviors such as feeding and soothing to sleep, driven by the infant's physiological signal characteristic values. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 This is a flowchart illustrating an infant care method based on multimodal data analysis, as described in an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The technical concept of this application lies in inputting multimodal monitoring data of infants and toddlers to be cared for, along with the caregiver's subjective complaints, into an infant care model. This model then determines the infant's multidimensional physiological characteristics and predicts physiological needs based on the received data, triggering a care request to enable the caregiver to provide care. This method innovatively applies artificial intelligence technology to infant care scenarios. It utilizes dynamically collected environmental information such as images and sounds during the growth process, as well as physiological information such as pulse, heartbeat, and respiration, to accurately track the evolutionary process of infants in various physiological cycles, including eating, sleeping, and waking. This allows for understanding each infant's unique physiological characteristics and energy preferences, enabling more precise care to support the baby's growth. It achieves scientific nursing guidance and prediction driven by the infant's physiological signal characteristics, supporting physiological behaviors such as feeding and soothing to sleep.

[0028] It should be noted that the inventor has been deeply involved in the infant and toddler care industry for many years, and collected the multimodal data required for training by obtaining authorization from the guardians. The inventor strictly followed relevant laws and regulations and there was no violation of laws, social ethics, or harm to public interests.

[0029] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0030] Figure 1 This is a flowchart illustrating an infant care method based on multimodal data analysis, as described in an embodiment of this application. Figure 1 As shown, infant and toddler care methods based on multimodal data analysis include:

[0031] Step S110: Acquire multimodal monitoring data and caregiver complaints related to the target subject. The target subject is the infant or toddler requiring care.

[0032] The multimodal monitoring data here can be monitoring data collected from the environment in which the infant is located using various sensors such as image sensors, sound sensors, temperature sensors, and infrared sensors. For example, image sensors can identify the movements of the infant's mouth and throat, and further identify the infant's swallowing movements in subsequent steps; sound sensors can identify the waveforms of the infant's voice, and further identify the physiological needs expressed by different sounds in subsequent steps by analyzing the frequency, amplitude, and timing characteristics of the sound; temperature sensors, infrared sensors, etc., can monitor room temperature and the infant's body temperature; and measure the infant's heart rate, respiratory rate, pulse, pulse pattern, etc., etc., which will not be listed here.

[0033] The caregiver's subjective data can be care description data actively triggered and generated by natural persons such as parents or postpartum caregivers. For example, it can include what time the baby ate, what time the baby urinated, and whether there were any physiological conditions that were observed by natural persons but not identified by the infant care model.

[0034] It should be noted that multimodal monitoring data related to the target object may include monitoring data of infants and young children, as well as monitoring data of their caregivers.

[0035] Step S120: Input the multimodal monitoring data and caregiver's complaints into the infant care model, and trigger a care request through the infant care model to enable the caregiver to provide care to the target individual. The care request includes the caregiver, the content of the care procedure, and the time.

[0036] An example care request would be: "Mom, please get ready, it's time to feed the baby in about 5 minutes." Here, the caregiver is the mother, the care action is breastfeeding, and the time is 5 minutes later.

[0037] In some embodiments, the infant care method based on multimodal data analysis can be implemented using one or more computer devices. For example, using a smart home system, a server with an infant care model can be deployed in the home, and each caregiver can deploy an application (APP) on a smartwatch or mobile phone that communicates with the server. Caregivers can submit their subjective complaints through the APP, and can also sense care requests through the APP. Of course, care requests can also be notified to caregivers in various forms such as sound, images, etc., and this application does not limit this.

[0038] In situations where local deployment of all infant care models is insufficient, some functions can be implemented on cloud servers. The local server is only responsible for the collection and transmission of multimodal monitoring data, while the cloud server is responsible for subsequent analysis and triggering of care requests. Alternatively, some processing can be performed locally before sending the data to the cloud. For example, images and sounds in the multimodal monitoring data can be identified first to obtain behavioral characteristics that represent infant behavior. Only these behavioral characteristics can be transmitted to the cloud, which can reduce users' concerns about uploading sensitive image information that can be used for identification.

[0039] The infant care model used in this embodiment is specifically used for: extracting features from the multimodal monitoring data to obtain physiological features that characterize the physiological state of the target object; performing similarity recognition on the physiological features based on a constructed baseline feature space to predict the physiological needs of the target object; each baseline feature in the baseline feature space characterizes various physiological needs of the infant; triggering a nursing request based on a constructed nursing norm field, the caregiver's complaints, and the physiological needs prediction results; the nursing norm field includes multiple nursing norm models, which characterize the mapping relationship between nursing operation features and physiological features when various physiological needs of the infant are reasonably met.

[0040] The physiological state of infants and young children exhibits certain regularities. After vectorization and other processing of the collected multimodal monitoring data, the resulting time-series characteristic data can be regarded as physiological signals reflecting the physiological state of infants and young children. In simple terms, the amplitude of the signal waveform can be regarded as the energy potential position of the infant's physiological wave at different spatiotemporal phases, and the frequency of the signal waveform reflects the speed of the physiological wave, which can comprehensively reflect the stability of the infant's current physiological state.

[0041] Therefore, by pre-learning the physiological patterns of infants and toddlers in different physiological needs scenarios, we can pre-construct baseline features representing various physiological needs of infants and toddlers. For example, what are the characteristics of hunger, and what are the characteristics of drowsiness? Based on this, if we identify similar physiological characteristics, we can approximate that the infant or toddler has the corresponding need. In other words, based on the collected multimodal monitoring data, when the infant's physiological signals show a specific trend, we can predict whether the infant or toddler has physiological needs such as needing to defecate or being hungry. Anchoring key nursing nodes with caregiver complaints can also improve the predictive accuracy of infant care models. For example, the weights of multiple potential physiological needs can be adjusted based on caregiver complaints. In one example, if the infant has just been fed, the predictive weight of the physiological need of "hunger" during crying will be lowered.

[0042] However, the example above is merely a simplified illustration of how the infant care model works. In actual care, we find that the physiological needs exhibited by infants do not necessarily represent actual needs. For instance, sometimes a baby may cry and show a desire to nurse even after being fed, and this desire may be satisfied after feeding. However, the baby may simply need sucking for comfort. In this case, feeding is not a proper way to meet the infant's physiological needs and could potentially lead to overfeeding and other problems.

[0043] What's more complex is that as infants and toddlers develop, their needs constantly change. Taking feeding as an example, suppose 100 ml of milk is usually enough to satisfy a baby, but after one feeding, the baby still wants more. How do we determine if this is because the baby is growing and needs more food, or if there are other needs such as comfort or a need for a specific routine, but the baby is simply expressing a need to be fed? If it's the former, not continuing to feed might hinder the baby's growth and development; if it's the latter, continuing to feed could lead to the overfeeding problem mentioned above.

[0044] This problem cannot be completely solved by relying solely on real-time physiological feature recognition, because real-time physiological feature recognition is more biased towards the real-time behavior of infants and young children, and it is difficult to reflect the periodic growth and development status.

[0045] Furthermore, even if the identified physiological needs are accurate, the care procedures that new parents can provide may not be suitable for their baby. This is why many families need to hire experienced postpartum nannies or relatives to care for their newborns. To address this, we introduce the concept of a nursing norm field. A nursing norm field includes multiple nursing norm models, specifically representing the mapping relationship between nursing procedures and physiological characteristics when various physiological needs of infants are reasonably met. Nursing norm models are trained based on multimodal monitoring data from historical data and can be continuously updated based on data collected during actual care, "growing" alongside the cared-for individual. Specifically, since the collected multimodal monitoring data can also include relevant information about the caregiver, by indexing standardized nursing actions, we can learn the nursing procedure characteristics that reasonably meet various physiological needs of infants, such as the appropriate flow rate and angle for feeding, and the frequency of gentle patting to soothe a baby to sleep. In addition, by combining scientific guidelines for infant growth and development, we can also determine the specific needs of babies at different ages, such as how much sleep they need per day, how many times they need to be fed per day, and how many milliliters per feeding. Furthermore, by allocating time reasonably, we can avoid a situation where a large number of needs are concentrated at once (such as being both sleepy and hungry), causing the baby to break down and cry uncontrollably, making it difficult to soothe them.

[0046] As described above, after extracting physiological features, identifying similarities, and predicting physiological needs, a comprehensive judgment can be made based on the nursing standard field, the caregiver's complaints, and the predicted physiological needs, so as to output the nursing request that best matches the baby's actual needs.

[0047] Specifically, nursing requests can also be output in stages. For example, the caregiver can be informed of what needs to be done first, and then specific operation instructions can be output when the caregiver begins the nursing operation.

[0048] In the above embodiment, we briefly introduced a scheme for triggering nursing requests corresponding to the physiological needs of infants and young children based on multimodal data analysis of infant and young child care models. Next, we will further illustrate other achievable beneficial effects with other embodiments.

[0049] In some embodiments, the multimodal monitoring data includes: first monitoring data collected by a physiological information acquisition device, including heart rate, respiratory rate, pulse, pulse pattern, etc. as described above; and second monitoring data collected by an environmental information acquisition device, such as images, sounds, etc.

[0050] Physiological information collection devices can be divided into multiple sets, such as one set for the infant being cared for and one set for each caregiver; environmental information collection devices can usually collect information over a wider range, and one set can be set up, or multiple sets can be set up as needed. These are just a few possible device deployment examples; in practice, you can choose flexibly according to your needs.

[0051] In some embodiments, the step of extracting features from the multimodal monitoring data to obtain physiological features characterizing the physiological state of the target object includes: extracting a first physiological feature and a first behavioral feature of the target object based on the first monitoring data; extracting a second physiological feature and a second behavioral feature of the target object based on the second monitoring data; using the first behavioral feature and the second behavioral feature as behavioral indexing information, fusing the first physiological feature and the second physiological feature based on the behavioral indexing information to obtain the physiological features characterizing the physiological state of the target object.

[0052] Information collected by different acquisition devices can be processed as multiple signals separately. For example, the first monitoring data collected by physiological information acquisition devices is more related to physiological state, and the first physiological features generated from it can have a higher weight in the final fusion of physiological features; however, it should also be noted that the first monitoring data can also reflect certain behavioral states of infants and young children, so behavioral features can also be extracted. Secondary monitoring data, such as images and sounds, can more intuitively reflect infant behavior, so secondary behavioral features can have a higher weight in behavioral indexing.

[0053] In the embodiments of this application, behavioral features may be used only as indexing, with the ultimate goal of obtaining physiological characteristics. As described in the embodiments above, the purpose of behavioral features is solely to identify the behavior of infants or caregivers; therefore, features related to identity expression can be ignored or discarded. For example, behavioral features can be output locally to the cloud to avoid uploading identity-related information to the cloud, thus protecting personal privacy.

[0054] Of course, in some embodiments described later, when it is necessary to create a profile of a caregiver, specific features are required to distinguish each caregiver. The extraction and use of features can be selective based on actual needs.

[0055] In some embodiments, the multimodal monitoring data related to the target object includes: multimodal monitoring data of the target object and multimodal monitoring data of the caregiver; the infant care method based on multimodal data analysis further includes: during the process of the caregiver providing care to the target object, the infant care model determines nursing operation characteristics based on the multimodal monitoring data related to the target object and the caregiver's subjective complaint data, and determines nursing optimization suggestions based on the nursing norm field, the nursing operation characteristics, and the physiological characteristics obtained during the nursing process.

[0056] Experienced caregivers, such as postpartum nannies, are clearly better at caring for infants and toddlers than new parents. However, in reality, new parents often rely on book learning, oral traditions passed down from older generations, and trial and error to improve their childcare skills—a process that is inefficient, and many practices considered gospel may actually be incorrect. Fortunately, an infant's physiological state usually doesn't lie. If we can learn about the physiological state and care procedures of an infant under proper care and continue to imitate them in subsequent care, we can ensure that the infant receives the same standard of appropriate care. Therefore, this embodiment identifies which characteristics in the care process are not optimal and provides corresponding care suggestions to the caregiver. For example, if the father is impatient during care, causing the baby to swallow too quickly and choke, he can be reminded to "slow down."

[0057] At the same time, we also considered the impact of individual differences. While all infants and toddlers generally exhibit similarities in each scenario, each infant and toddler also possesses unique characteristics. Therefore, the nursing care model can be optimized based on the individual characteristics of each infant and toddler. Specifically, if an infant and toddler behaves very well during a care session, the preset nursing care model can be optimized based on this process. This also allows for adaptation to the growth of infants and toddlers, because babies are not static, and the infant care model must evolve along with their development.

[0058] It's also important to note that multimodal monitoring of caregivers is essential, especially since data can be collected from multiple sources, such as the physiological and environmental information acquisition devices mentioned earlier. For instance, a father might move quite vigorously while caring for a baby, which might be perceived as agitation based solely on images; however, physiological signal analysis might reveal no signs of rapid heartbeat, pulse abnormalities, or emotional distress, indicating that the father's vigorous movements are still under his reasonable control and likely will not affect the baby.

[0059] In some embodiments, the infant care method based on multimodal data analysis further includes: during the process of the caregiver caring for the target object, if the infant care model identifies physiological characteristics that match the target artificial intervention scenario from the multimodal monitoring data, then it generates real-time intervention suggestions corresponding to the target artificial intervention scenario.

[0060] For example, in a swallowing scenario, the captured signal waveforms of a complete swallowing sequence will exhibit a statistical pattern: sparse at the beginning → dense and regular in the middle → sparse at the end. This corresponds precisely to the entire process of hunger-fullness. Simultaneously, the wavelength of swallowing is relatively constant, and if there is a noticeable pause midway, it can be determined whether the infant has swallowed air, causing hiccups, prompting caregivers to intervene with burping or other interventions. This is a simplified explanation using the swallowing scenario as an example; it is easy to understand and can be extended to other situations requiring intervention.

[0061] In some embodiments, the infant care model is trained as follows: acquiring infant multimodal monitoring data required for training as training data, wherein the infant multimodal monitoring data includes annotation information; using a base model to perform iterative training on the training data for feature extraction and physiological need identification until an infant care model containing the baseline feature space and the nursing norm field is obtained.

[0062] The infant care model used in the embodiments of this application can be trained based on a base model with an existing network structure. In some embodiments, the base model can be a Large Language Model (LLM) that supports multimodal data. In other embodiments, the base model can be a classic model using a Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), or Transformer architecture. As mentioned above, the data used is training data that has been legally and compliantly collected. Furthermore, for infants and young children, sensitive aspects such as identity recognition are not involved; instead, the model utilizes the commonalities of infants and young children under the same physiological needs to learn common features, and some specific features can be discarded. For example, features unrelated to action recognition can be masked using a mask, making it easier to highlight features of key areas. In some embodiments, the training data can carry scene annotation information and event annotation information such as "feeding scene" or "feeding scene + burping".

[0063] In some embodiments, the infant care method based on multimodal data analysis further includes: generating training data containing positive feedback information and / or negative feedback information based on multimodal monitoring data collected during the care process and the caregiver's subjective complaints submitted by the caregiver regarding the care process; training the infant care model based on the training data; and updating the baseline feature space and / or care norm field.

[0064] Specifically, in some embodiments, training the infant care model based on the training data and updating the baseline feature space and / or nursing norm field includes: training a problem feature field based on the training data, wherein the problem feature field represents the mapping relationship between nursing operation characteristics and physiological characteristics when various physiological needs of infants are not properly met; and conducting adversarial training based on the problem feature field and the nursing norm field. In some embodiments, the nursing norm field includes reasonable rhythm information of various physiological needs, and the problem feature field includes abnormal rhythm information of various physiological needs; the adversarial training based on the problem feature field and the nursing norm field includes: adjusting the reasonable rhythm information according to the normal growth rhythm of infants and the abnormal rhythm information, so that the updated nursing norm field adapts to the growth of the target object.

[0065] The problem feature field can be obtained by combining the feature values ​​of the caregiver's subjective complaints (e.g., labeled as negative feedback information through semantic recognition), such as reflecting overfeeding mentioned earlier. Through adversarial training and mirror learning using the problem feature field and the aforementioned nursing norm field, the infant care model is continuously optimized to adapt to the infant's growth rhythm. For example, as the baby grows, the sleep cycle progresses from 45 minutes to 60 minutes. Experiences such as growth rhythm guidelines can only provide a rough prediction of the progression time (e.g., within a week, but not accurate to a specific day or sleep session). Specific adjustments are needed based on data collected during actual care to suit the baby's growth curve. Some of this data is positive, and some is negative, but both can be appropriately indexed to aid in model training.

[0066] In the above embodiments, the reasonable rhythm information serves as the basis for generating care requests during the actual operation of the infant care model. Therefore, it needs continuous adjustment. Specifically, adjustments can be made based on normal growth rhythms (such as the patterns summarized in textbooks like the growth rhythm guidelines mentioned earlier) and abnormal rhythm information obtained during actual care. It should be noted that abnormal rhythm information is not necessarily abnormal information. For example, if the original feeding amount was 100 ml, but it has been found to be insufficient on multiple occasions, then the feeding amount needs to be reasonably increased. However, this is a slight change from the original 100 ml per feeding, and is referred to here as "abnormal," not "abnormal." Of course, abnormal situations also exist, such as when the adjusted reasonable rhythm information is somewhat overcorrected, in which case it needs to be adjusted back.

[0067] In some embodiments, the infant care method based on multimodal data analysis further includes: constructing a caregiver feature profile for each caregiver based on the caregiver's characteristics during the care process; the care request corresponds to a first care operation, and triggering the care request includes: calculating a matching degree based on the caregiver feature profile and the care normative field, determining the first caregiver with the highest matching degree to the first care operation, and triggering a care request for the first caregiver.

[0068] As mentioned earlier, different caregivers have varying levels of experience. Furthermore, even with training, individual differences mean that some caregivers may excel in one area but not necessarily in others. This results in different caregivers being suitable for different scenarios, leading to varying priorities. Therefore, during the care process, it's possible to profile different caregivers to determine the physiological needs each agent is best suited to address and the specific care procedures they should perform.

[0069] In some embodiments, the infant care method based on multimodal data analysis further includes: receiving care request feedback information, the care request feedback information indicating that the first caregiver is unable to provide care as expected; and determining an alternative care request based on the care request feedback information and the urgency of the care operation, the alternative care request being directed to a second caregiver who is different from the first caregiver.

[0070] While babies are every parent's darling, caregivers can't always prioritize infant care. Therefore, situations may arise where overtime work, cooking, or other reasons prevent them from responding promptly to the infant's needs. In such cases, it's necessary to adjust accordingly, triggering alternative care requests to ensure the infant's needs are met as quickly as possible.

[0071] The nursing request feedback information here can be triggered proactively by the caregiver after receiving the nursing request, such as clicking "Temporarily Unable to Respond" on the APP, or sending a voice feedback "Cannot Come for the Time," or it can be automatically triggered by the infant care model after it determines that the caregiver has failed to respond on time.

[0072] In some embodiments, the alternative care request corresponds to the first care action; or, the alternative care request corresponds to a second care action, wherein the second caregiver is the caregiver with the highest matching degree to the second care action.

[0073] There are several approaches to addressing alternative ways to meet an infant's physiological needs. For example, if the highest priority caregiver is temporarily unable to respond, the second highest priority caregiver can take over. If that still doesn't work, the priority levels can be lowered sequentially until a caregiver responds. Another method is to meet other physiological needs of the infant, creating a substitution effect. For instance, if the mother is better at breastfeeding and the father is better at playing games, and the mother is cooking and needs to wait 5 minutes to satisfy her hunger, one possible solution is for the father to play with the baby for 5 minutes first, and then the mother can breastfeed.

[0074] In some embodiments, the first nursing operation includes multiple third nursing operations at adjacent time points, and the triggering of the nursing request includes: determining the third caregiver with the highest comprehensive matching degree with the third nursing operation based on the caregiver feature profile and the nursing norm field, and arranging the nursing operations, and triggering a nursing request for the third caregiver according to the arrangement result of parallel execution and / or sequential execution.

[0075] In some situations, caregivers possess high levels of expertise and are prioritized for various physiological needs. When an infant has multiple needs in a short period, selecting the highest-priority caregiver for each could lead to a situation where the mother feeds, the father changes diapers, and the grandmother soothes the infant to sleep. While this might seem like the highest score for each individual need, it might not be the optimal experience for either the caregiver or the infant. Therefore, when it's possible to select the optimal caregiver, that caregiver can perform the necessary tasks to meet multiple needs in the short term. Additionally, in some scenarios, multiple caregivers may need to work together to perform certain tasks in parallel. For example, when an infant is both sleepy and hungry, one caregiver can feed the baby while another gently hums a lullaby and pats the baby to soothe their anxiety and help them fall asleep quickly after feeding.

[0076] In some embodiments, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the infant care methods based on multimodal data analysis in the embodiments of this application.

[0077] For example, Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 2 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0078] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 2 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0079] Memory is used to store programs. Specifically, programs can include program code, which includes computer operation instructions. Memory can include main memory and non-volatile memory, and it provides instructions and data to the processor. For example... Figure 2 The computer program shown is represented by a virtual device called an infant care device based on multimodal data analysis.

[0080] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a multimodal data analysis-based infant care device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0081] The process involves acquiring multimodal monitoring data and caregiver complaint data related to a target individual, where the target individual is an infant or toddler requiring care. The multimodal monitoring data and caregiver complaint data are input into an infant care model, which triggers a care request, instructing the caregiver to provide care to the target individual. The care request includes the caregiver, the content of the care procedure, and the timeframe. Specifically, the infant care model is used to: extract features from the multimodal monitoring data to obtain physiological characteristics representing the target individual's physiological state; perform similarity recognition on the physiological characteristics based on a constructed baseline feature space to predict the target individual's physiological needs; each baseline feature in the baseline feature space represents a specific physiological need of the infant or toddler; and trigger a care request based on a constructed nursing norm field, the caregiver complaint data, and the predicted physiological needs. The nursing norm field includes multiple nursing norm models, which represent the mapping relationship between nursing operation characteristics and physiological characteristics when the infant or toddler's various physiological needs are reasonably met.

[0082] The above is as stated in this application. Figure 1The method for an infant care device based on multimodal data analysis disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0083] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the infant care device based on multimodal data analysis in the illustrated embodiment is specifically used to perform:

[0084] The process involves acquiring multimodal monitoring data and caregiver complaint data related to a target individual, where the target individual is an infant or toddler requiring care. The multimodal monitoring data and caregiver complaint data are input into an infant care model, which triggers a care request, instructing the caregiver to provide care to the target individual. The care request includes the caregiver, the content of the care procedure, and the timeframe. Specifically, the infant care model is used to: extract features from the multimodal monitoring data to obtain physiological characteristics representing the target individual's physiological state; perform similarity recognition on the physiological characteristics based on a constructed baseline feature space to predict the target individual's physiological needs; each baseline feature in the baseline feature space represents a specific physiological need of the infant or toddler; and trigger a care request based on a constructed nursing norm field, the caregiver complaint data, and the predicted physiological needs. The nursing norm field includes multiple nursing norm models, which represent the mapping relationship between nursing operation characteristics and physiological characteristics when the infant or toddler's various physiological needs are reasonably met.

[0085] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0087] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0089] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0090] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0091] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0092] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0093] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for infant and toddler care based on multimodal data analysis, characterized in that, The method includes: Acquire multimodal monitoring data and caregiver complaints related to the target object, wherein the target object is an infant or toddler requiring care; The multimodal monitoring data and caregiver complaint data are input into the infant care model, and a care request is triggered through the infant care model to enable the caregiver to provide care to the target object. The care request includes the caregiver, the content of the care operation, and the time. The infant care model is specifically used for: Feature extraction is performed on the multimodal monitoring data to obtain physiological features that characterize the physiological state of the target object; similarity recognition is performed on the physiological features based on the constructed baseline feature space to predict the physiological needs of the target object; each baseline feature in the baseline feature space characterizes various physiological needs of infants and young children. Nursing requests are triggered based on the established nursing norm field, the caregiver's subjective complaint data, and the physiological needs prediction results; the nursing norm field includes multiple nursing norm models, which represent the mapping relationship between nursing operation characteristics and physiological characteristics when various physiological needs of infants and young children are reasonably met.

2. The infant care method based on multimodal data analysis according to claim 1, characterized in that, The multimodal monitoring data includes: first monitoring data collected by physiological information acquisition equipment and second monitoring data collected by environmental information acquisition equipment.

3. The infant care method based on multimodal data analysis according to claim 2, characterized in that, The step of extracting features from the multimodal monitoring data to obtain physiological features characterizing the physiological state of the target object includes: Based on the first monitoring data, extract the first physiological characteristics and first behavioral characteristics of the target object; Extract the second physiological and second behavioral characteristics of the target object based on the second monitoring data; Using the first behavioral feature and the second behavioral feature as behavioral indexing information, the first physiological feature and the second physiological feature are fused according to the behavioral indexing information to obtain the physiological features characterizing the physiological state of the target object.

4. The infant care method based on multimodal data analysis according to claim 1, characterized in that, The multimodal monitoring data related to the target object includes: multimodal monitoring data of the target object and multimodal monitoring data of the caregiver; the method further includes: During the process of the caregiver providing care to the target subject, the infant care model determines the nursing operation characteristics based on multimodal monitoring data related to the target subject and the caregiver's subjective complaint data, and determines nursing optimization suggestions based on the nursing standard field, the nursing operation characteristics, and the physiological characteristics obtained during the nursing process.

5. The infant care method based on multimodal data analysis according to claim 1, characterized in that, The method further includes: During the care of the target subject by the caregiver, if the infant care model identifies physiological characteristics that match the target human intervention scenario from the multimodal monitoring data, it generates real-time intervention suggestions corresponding to the target human intervention scenario.

6. The infant care method based on multimodal data analysis according to claim 1, characterized in that, The infant care model was trained in the following manner: The infant multimodal monitoring data required for training is obtained as training data, and the infant multimodal monitoring data includes annotation information; The training data is iteratively trained using a base model to extract features and identify physiological needs until an infant care model containing the baseline feature space and the nursing norm field is obtained.

7. The infant care method based on multimodal data analysis according to claim 1, characterized in that, The method further includes: Based on the multimodal monitoring data collected during the nursing process and the nursing complaints submitted by the caregivers in response to the nursing process, training data containing positive feedback information and / or negative feedback information is generated. The infant care model is trained based on the training data, and the baseline feature space and / or care norm field are updated.

8. The infant care method based on multimodal data analysis according to claim 7, characterized in that, The step of training the infant care model based on the training data and updating the baseline feature space and / or care norm field includes: A problem feature field is obtained by training based on training data. The problem feature field represents the mapping relationship between nursing operation characteristics and physiological characteristics when various physiological needs of infants and young children are not properly met. Adversarial training is conducted based on the problem characteristic field and the nursing standard field.

9. The infant care method based on multimodal data analysis according to claim 8, characterized in that, The nursing standard field includes reasonable rhythmic information of various physiological needs, and the problem characteristic field includes abnormal rhythmic information of various physiological needs; The method of conducting adversarial training based on the problem characteristic field and the nursing standard field includes: adjusting the reasonable rhythm information according to the normal growth rhythm of infants and young children and the abnormal rhythm information, so that the updated nursing standard field adapts to the growth of the target object.

10. The infant care method based on multimodal data analysis according to claim 1, characterized in that, The method also includes: constructing a nursing characteristic profile for each nursing staff member based on their nursing characteristics during the nursing process; The nursing request corresponds to a first nursing operation, and triggering the nursing request includes: Based on the nurse's feature profile and the nursing standard field, a matching degree is calculated to determine the first nurse with the highest matching degree to the first nursing operation, and a nursing request is triggered for the first nurse.

11. The infant care method based on multimodal data analysis according to claim 10, characterized in that, The method further includes: Receive nursing request feedback information, the nursing request feedback information indicating that the first caregiver is unable to provide care as expected; Based on the nursing request feedback information and the urgency of the nursing operation, an alternative nursing request is determined, which is directed to a second caregiver who is different from the first caregiver.

12. The infant care method based on multimodal data analysis according to claim 11, characterized in that, The alternative care request corresponds to the first care operation; or, the alternative care request corresponds to the second care operation, wherein the second caregiver is the caregiver with the highest matching degree to the second care operation.

13. The infant care method based on multimodal data analysis according to claim 10, characterized in that, The first nursing operation includes multiple third nursing operations at adjacent time points, and the triggering of the nursing request includes: Based on the nurse's characteristic profile and the nursing standard field, the third nurse with the highest comprehensive matching degree with the third nursing operation is determined, and the nursing operation is arranged. The nursing request is triggered for the third nurse according to the arrangement result of parallel execution and / or sequential execution.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 13.

16. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 13.

Images