Postpartum care monitoring method and system
By collecting image and voice information in the maternal environment, using target recognition and emotion analysis technology, the problem of difficulty in timely discovering maternal psychological abnormalities in the existing technology is solved, and the efficiency and accuracy of postpartum care is achieved, ensuring the psychological health of the maternal population Get timely attention.
Patent Information
- Application Number
- CN202411333477.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing technology is difficult to detect psychological abnormalities after delivery of mothers in a timely manner, resulting in the inability to provide necessary psychological support and intervention in a timely manner.
By collecting image and speech information in the target environment, using target recognition and emotion analysis technology, we can identify the emotional state of the mother and her associated objects, realize cross-modal emotional consistency judgment, and timely discover abnormal states.
It improves the accuracy and efficiency of postpartum care, promptly identify the psychological state of the mother, and ensures that medical staff can provide necessary support at critical moments.
Smart Images

Figure CN119055240A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent nursing technology, and in particular to a postpartum nursing monitoring method and system. Background Art
[0002] When a mother gives birth to a child, in addition to facing the challenge of physical recovery, her mental health also faces an important test. Many mothers may experience psychological problems such as mood swings, anxiety, and depression. These emotions not only affect their quality of life, but may also have a profound impact on the mother-child relationship. At present, the common methods for assessing the psychological status of mothers mainly rely on questionnaires and observations by family members. However, these methods have certain limitations. First, although questionnaires can collect some data, they often cannot fully capture the true feelings of the mother. Many mothers may be unwilling to express their emotions frankly due to social pressure or self-concealment, resulting in inaccurate assessment results. Secondly, although family members can care and support mothers to a certain extent, their attention to mothers may also be limited by their own cognition, and they may not be able to keenly identify potential psychological problems. For example, family members may not understand the specific manifestations of postpartum depression, or even mistakenly regard it as normal mood swings, so it is difficult to detect problems in time and provide necessary support. Therefore, there is an urgent need for a postpartum care monitoring method to detect maternal psychological problems in time. Summary of the invention
[0003] The main purpose of the embodiments of the present invention is to provide a postpartum care monitoring method and system, aiming to solve the problem in the related art that maternal psychological abnormalities cannot be discovered in time.
[0004] In a first aspect, an embodiment of the present invention provides a postpartum care monitoring method, comprising:
[0005] Collecting initial image information and initial voice information corresponding to the target environment, and performing target recognition on the initial image information to obtain a target object and an associated object of the target object in the same target environment;
[0006] Obtaining first emotion information corresponding to the target object according to the association relationship between the target object and the associated object in the initial image information;
[0007] Obtaining first voice information corresponding to the target object and second voice information corresponding to the associated object from the initial voice information;
[0008] Determine second emotion information corresponding to the target object according to the first voice information and the second voice information;
[0009] Determining an emotional state corresponding to the target object according to the first emotional information and the second emotional information;
[0010] A target monitoring result corresponding to the target object is determined according to the emotional state.
[0011] In a second aspect, an embodiment of the present invention provides a postpartum care monitoring system, comprising:
[0012] A data acquisition module, used to acquire the initial image information and initial voice information corresponding to the target environment, and to perform target recognition on the initial image information to obtain the target object and the associated objects of the target object in the same target environment;
[0013] An image analysis module, configured to obtain first emotion information corresponding to the target object according to an association relationship between the target object and the associated object in the initial image information;
[0014] A speech segmentation module, used for obtaining the first speech information corresponding to the target object and the second speech information corresponding to the associated object from the initial speech information;
[0015] A speech analysis module, used to determine second emotion information corresponding to the target object according to the first speech information and the second speech information;
[0016] A state recognition module, used to determine the emotional state corresponding to the target object according to the first emotional information and the second emotional information;
[0017] A result determination module is used to determine the target monitoring result corresponding to the target object according to the emotional state.
[0018] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any one of the postpartum care monitoring methods provided in the specification of the present invention are implemented.
[0019] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any postpartum care monitoring method provided in the specification of the present invention.
[0020] The embodiment of the present invention provides a postpartum care monitoring method and system. The method performs an in-depth analysis of a target object and its social environment by collecting initial image information and initial voice information in a target environment. First, by accurately identifying the target object through the initial image information, not only the target object can be accurately identified, but also the associated objects that are related to it can be identified. This correlation analysis provides valuable data support for understanding the social environment and emotional influencing factors of the target object. Further, the method derives the first emotional information of the target object based on the association relationship between the target object and the associated objects in the initial image information. At the same time, the first voice information of the target object and the second voice information of the associated objects are extracted from the initial voice information. By deeply analyzing these voice data, the system determines the second emotional information of the target object. Thus, by comparing the consistency between the first emotional information of the target object under the initial image information and the second emotional information of the target object under the initial voice information, the real emotional state of the target object can be accurately judged. This cross-modal emotion analysis method significantly improves the accuracy and reliability of emotion recognition. The method can judge whether the target object is consistent in words and deeds in terms of emotional expression by comparing the emotions expressed by the target object in the initial image information and the initial voice information. Thus, the abnormal state of the target object can be discovered in time. It solves the problem that related technologies cannot detect maternal psychological abnormalities in a timely manner, and improves the quality and efficiency of postpartum care. In addition, timely emotion recognition enables medical staff to intervene at critical moments, providing good support for subsequent maternal psychological support and professional help. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1 A schematic diagram of a process of a postpartum care monitoring method provided by an embodiment of the present invention;
[0023] Figure 2 A schematic diagram of the module structure of a postpartum care monitoring system provided by an embodiment of the present invention;
[0024] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0027] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0028] The embodiment of the present invention provides a postpartum care monitoring method and system. The postpartum care monitoring method can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.
[0029] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0030] Please refer to Figure 1 , Figure 1 A flowchart of a postpartum care monitoring method provided by an embodiment of the present invention.
[0031] like Figure 1 As shown, the postpartum care monitoring method includes steps S101 to S106.
[0032] Step S101: Collecting initial image information and initial voice information corresponding to a target environment, and performing target recognition on the initial image information to obtain a target object and an associated object of the target object in the same target environment.
[0033] Exemplarily, the target environment may be a hospital, a confinement center, or a home, and a high-definition camera and a high-quality microphone are installed in the target environment to ensure the initial image information and the clarity of the initial image information.
[0034] It should be noted that when installing high-definition cameras and high-quality microphones in the target environment, the target object and the associated objects in the same target environment as the target object need to be informed and their consent must be obtained, and the initial image information and initial voice information collected must be ensured to be used for subsequent nursing monitoring.
[0035] Exemplarily, initial image information corresponding to a high-definition camera and initial voice information corresponding to a high-quality microphone in a target environment are obtained.
[0036] Exemplarily, a target recognition algorithm such as a deep learning model is used to perform target recognition on the initial image information to obtain a target object and associated objects related to the target object, where the target object is a parturient and the associated objects are family members, doctors or friends who communicate with the parturient.
[0037] Step S102: obtaining first emotion information corresponding to the target object according to the association relationship between the target object and the associated object in the initial image information.
[0038] Exemplarily, according to the association relationship between the target object and the associated object in the initial image information, where the association relationship includes but is not limited to position information and interaction mode. First, by analyzing the relative positions of the target object and the associated object in the initial image information, the first facial information of the associated object is identified. The first facial information includes the facial expression, eye contact, etc. of the associated object. The emotional information of the associated object in the first facial information, such as happiness, anxiety, anger, etc., is obtained through the emotion recognition algorithm, which provides an important emotional basis for subsequent analysis.
[0039] Exemplarily, the emotional influence weight of the associated object on the target object is determined by combining the location information and interaction mode between the target object and the associated object. The emotional influence weight reflects how the emotional state of the associated object affects the emotional response of the target object. For example, positive emotions may enhance the target object's sense of happiness, while negative emotions may have a negative impact.
[0040] Exemplarily, the second facial information of the target object is extracted from the initial image information. The emotion recognition model is used to perform emotion recognition on the first facial information of the associated object to obtain a first emotion distribution corresponding to the associated object, and the emotion recognition model is used to perform emotion recognition on the second facial information of the target object to obtain a second emotion distribution corresponding to the target object.
[0041] Exemplarily, the first emotion distribution is adjusted based on the emotion influence weight of the associated object on the target object to obtain the third emotion distribution corresponding to when the associated object is used to perform emotion mapping on the target object, so that the second emotion distribution and the third emotion distribution are summed and averaged to obtain the first emotion information corresponding to the target object.
[0042] Exemplarily, by introducing the emotional influence of associated objects, the emotional influence of associated objects can be considered when calculating the first emotional information of the target object, which helps to capture the emotional changes of the target object in the context and can better understand the emotional response of the target object in a social environment, making the obtained first emotional information more accurate.
[0043] In some embodiments, obtaining the first emotion information corresponding to the target object based on the association relationship between the target object and the associated object in the initial image information includes: identifying features of the target object in the initial image information to obtain first feature information, and identifying features of the associated object in the initial image information to obtain second feature information; performing scene feature recognition on the initial image information to obtain initial scene features corresponding to the initial image information; performing feature fusion of the initial scene features with the first feature information and the second feature information to obtain target feature information; obtaining facial expression information corresponding to the target object in the initial scene information; and performing emotion recognition based on the facial expression information and the target feature information to obtain the first emotion information corresponding to the target object.
[0044] Exemplarily, the feature information of the target object in the initial image information is identified by a deep learning model such as a convolutional neural network to obtain the first feature information of the target object. The first feature information includes but is not limited to position information, body posture, clothing, etc. Similarly, the second feature information is extracted by performing feature identification on the associated object in the initial image by a deep learning model such as a convolutional neural network. The second feature information includes but is not limited to its relative position with the target object, body posture, clothing, etc.
[0045] Exemplarily, scene feature recognition is performed on the initial image information to obtain initial scene features corresponding to the initial image information, and the initial scene features may involve the spatial layout, lighting conditions, background elements, etc. of the target environment. Extraction of initial scene features can help understand the context of the target object and related objects, and provide important context for subsequent analysis.
[0046] For example, the initial scene features, the first feature information of the target object and the second feature information of the associated object are fused by feature weighting, concatenation or other fusion algorithms to obtain the target feature information. This fusion process helps to capture the association relationship between the target object and the associated object, laying the foundation for emotion recognition.
[0047] Exemplarily, facial expression information of the target object is extracted from the initial image information by using facial recognition technology. Facial expression information can reflect the emotional state of the target object and provide a direct basis for emotion recognition. The target feature information is classified by using a relationship classification model to obtain the association relationship between the target object and the associated object, and then the emotional influence weight of the associated object on the target object is obtained according to the association relationship. The emotional influence weight is then used to adjust the features of the facial expression information of the target object to obtain the target facial information of the target object, and then the first emotional information corresponding to the target object is obtained by using the emotional classification model to classify the target facial information.
[0048] Exemplarily, by using a relational classification model to analyze the association relationship between a target object and an associated object, their interaction dynamics can be better understood. This helps to identify the degree of influence of an associated object on the target object's emotional state. By obtaining the emotional influence weight of an associated object on a target object, the intensity of emotional interaction can be quantified. By adjusting the features of the target object's facial expression information through the emotional influence weight, a more personalized emotional analysis can be achieved. Based on the unique influence of each associated object, the adjusted target facial information more realistically reflects the target object's emotional state. Thus, by using an emotional classification model to classify the target facial information, the accuracy of emotion recognition can be improved. After feature adjustment, the model can better capture the emotional expression of the target object in a specific relationship and environment.
[0049] In some embodiments, the step of fusing the initial scene feature with the first feature information and the second feature information to obtain target feature information includes: fusing the initial scene feature with the first feature information and the second feature information into the same feature space to obtain the target scene feature corresponding to the initial scene feature, the third feature information corresponding to the first feature information, and the fourth feature information corresponding to the second feature information; fusing the third feature information and the target scene information to obtain the first weight corresponding to the target object in the target scene; updating the target scene feature according to the first weight and the third feature information to obtain the updated first scene feature; fusing the fourth feature information and the first scene feature to obtain the second weight corresponding to the associated object in the target scene; updating the first scene feature according to the second weight and the fourth feature information to obtain the updated second scene feature; fusing object features according to the first weight, the second weight, the third feature information, and the fourth feature information to obtain the fifth feature information; and fusing the second scene feature with the fifth feature information to obtain the target feature information.
[0050] Exemplarily, the initial scene features, the first feature information, and the second feature information are standardized or normalized by using principal component analysis (PCA) or other dimensionality reduction techniques to ensure that the dimensions of each feature are consistent, so that they can be compared and fused in the same feature space, thereby obtaining the target scene features corresponding to the initial scene features, the third feature information corresponding to the first feature information, and the fourth feature information corresponding to the second feature information.
[0051] Exemplarily, the third feature information corresponding to the target object and the target scene feature are first fused to obtain a first weight corresponding to the target object in the target scene. The first weight reflects the importance or influence of the target object in the target scene. The target scene feature is updated by weighted combination according to the first weight and the third feature information to obtain the first scene feature. Thus, the first feature information of the target object is added to the target environment.
[0052] Exemplarily, the fourth feature information and the updated first scene feature are fused to obtain the second weight corresponding to the associated object in the target scene. Thus, the influence of the associated object in the target environment can be quantified by the second weight. The first scene feature is then updated using the second weight and the fourth feature information to obtain the updated second scene feature. Thus, the second feature information of the associated object is added to the target environment.
[0053] Exemplarily, adding the first feature information and the second feature information one by one to the target scene feature corresponding to the target environment helps capture the complex emotions and behavior patterns between the target object and the associated objects. Then, the first weight, the second weight, the third feature information and the fourth feature information are combined to perform object feature fusion to obtain the fifth feature information. This fusion process combines the features of the target object and the associated objects to form a more comprehensive feature description.
[0054] Exemplarily, the second scene feature and the fifth feature information are feature concatenated to obtain the target feature information, thereby ensuring that the obtained target feature information not only reflects the state of the target object, but also takes into account its interactive impact with the associated objects in the target scene.
[0055] Step S103: Obtain first voice information corresponding to the target object and second voice information corresponding to the associated object from the initial voice information.
[0056] Exemplarily, a first voiceprint feature corresponding to the target object and a second voiceprint feature corresponding to the associated object are obtained, so that the voiceprint recognition model is used to identify the first voice information corresponding to the target object when it is close to the first voiceprint feature from the initial voice information, and the voiceprint recognition model is used to identify the second voice information corresponding to the associated object when it is close to the second voiceprint feature from the initial voice information.
[0057] Step S104: Determine second emotion information corresponding to the target object according to the first voice information and the second voice information.
[0058] Exemplarily, acoustic feature extraction is performed on the first voice information and the second voice information to obtain a first acoustic feature corresponding to the first voice information and a second acoustic feature corresponding to the second voice information, and the first acoustic feature and the second acoustic feature acquire acoustic features including pitch, speaking speed, volume, rhythm, intonation changes, etc.
[0059] For example, a suitable emotion recognition model is selected. The emotion recognition model can be a machine learning-based model (such as a support vector machine, a random forest) or a deep learning model (such as a convolutional neural network, a recurrent neural network). The emotion recognition model is trained using an existing emotion annotation dataset to ensure that the emotion recognition model can recognize different emotion types (such as happiness, anger, sadness, surprise, etc.).
[0060] Exemplarily, the first acoustic feature of the target object and the second acoustic feature of the associated object are fused and input into the trained emotion recognition model to obtain the second emotion information corresponding to the target object. This process not only takes into account the acoustic features of the target object itself, but also fully integrates the acoustic features of the associated object, so that the emotional state of the target object can be captured more comprehensively.
[0061] Exemplarily, the first acoustic feature of the target object provides basic information for emotion recognition, including the pitch, speaking speed, tone, etc. of the voice. At the same time, the second acoustic feature of the associated object also provides important background information for emotion recognition. By fusing the first acoustic feature and the second acoustic feature, the emotion recognition model can more accurately understand the emotional state of the target object in a specific social environment. The emotional state of the associated object may have a direct impact on the emotional expression of the target object, especially in social interactions. For example, when the target object hears the happy tone of the associated object, he or she may feel a stronger happy emotion. By incorporating this influence into the emotion recognition model, the second emotional information of the target object can be captured more accurately.
[0062] In some embodiments, determining the second emotion information corresponding to the target object according to the first voice information and the second voice information includes: performing voice recognition on the first voice information to obtain first text information and performing voice recognition on the second voice information to obtain second text information; performing pragmatic paralanguage extraction on the first voice information to obtain first paralanguage information and performing pragmatic paralanguage extraction on the second voice information to obtain second paralanguage information; calculating the duration of the first paralanguage information in the first voice information to obtain a first time feature corresponding to the first paralanguage information and calculating the frequency of occurrence of the first paralanguage information in the first voice information to obtain a first frequency feature corresponding to the first paralanguage information; calculating the duration of the second paralanguage information in the second voice information to obtain a second time feature corresponding to the second paralanguage information and calculating the frequency of occurrence of the second paralanguage information in the second voice information to obtain a second frequency feature corresponding to the second paralanguage information; fusing the first text information, the first time feature and the first frequency feature to determine the first emotion feature corresponding to the target object; fusing the second text information, the second time feature and the second frequency feature to determine the second emotion feature corresponding to the associated object; and obtaining the second emotion information corresponding to the target object according to the first emotion feature and the second emotion feature.
[0063] Exemplarily, the first voice information and the second voice information are respectively processed by voice recognition technology to obtain first text information corresponding to the first voice information and second text information corresponding to the second voice information.
[0064] Exemplarily, extracting pragmatic paralanguage from the first voice information and the second voice information includes identifying non-language components such as intonation, pauses, stress, and speech rate changes to capture potential emotional information. First paralanguage information corresponding to the first voice information and second paralanguage information corresponding to the second voice information are obtained respectively.
[0065] Exemplarily, by analyzing the first duration in the first voice information corresponding to each paralanguage information in the first paralanguage information, the first time feature is obtained according to the ratio between the first duration and the duration corresponding to the first voice information. Similarly, by analyzing the second duration in the second voice information corresponding to each paralanguage information in the second paralanguage information, the second time feature is obtained according to the ratio between the second duration and the duration corresponding to the second voice information.
[0066] Exemplarily, the occurrence frequency of each paralanguage information in the first voice information is counted to obtain the first frequency feature. Similarly, the occurrence frequency of each paralanguage information in the second voice information is counted to obtain the second frequency feature.
[0067] Exemplarily, the first text information, the first time feature and the first frequency feature are fused through feature weighting and feature fusion algorithm to obtain the first emotion feature corresponding to the target object. Similarly, the second text information, the second time feature and the second frequency feature are fused through feature weighting and feature fusion algorithm to determine the second emotion feature corresponding to the associated object.
[0068] Exemplarily, the first emotion feature of the target object and the second emotion feature of the associated object are fused and input into an emotion classification model that has been trained to recognize emotions based on emotion features, thereby obtaining the second emotion information corresponding to the target object. This process not only takes into account the emotion features of the target object itself, but also fully integrates the emotion features of the associated object, so that the emotional state of the target object can be captured more comprehensively.
[0069] For example, the first emotion feature of the target object provides basic information about the target object's own emotions for emotion recognition, including text, speech speed, tone, etc. At the same time, the second emotion feature of the associated object also provides important background information for emotion recognition. By integrating the first emotion feature and the second emotion feature, the emotion classification model can more accurately understand the emotional state of the target object in a specific social environment.
[0070] In some embodiments, the fusing of the first text information, the first time feature, and the first frequency feature to determine the first emotion feature corresponding to the target object includes: performing emotion classification based on the first text information to obtain a first classification result, and determining a first fusion weight corresponding to the first text information based on the first classification result; performing emotion classification based on the first time feature to obtain a second classification result, and determining a second fusion weight corresponding to the first time feature based on the second classification result; performing emotion classification based on the first frequency feature to obtain a third classification result, and determining a third fusion weight corresponding to the first frequency feature based on the third classification result; performing feature fusion on the first text information, the first time feature, and the first frequency feature based on the first fusion weight, the second fusion weight, and the third fusion weight to obtain the first emotion feature corresponding to the target object.
[0071] Exemplarily, the first text information is firstly sentimentally classified by using a trained first classification model to identify the sentiment type expressed in the first text information, such as positive, negative or neutral, so that the first classification result can be effectively obtained. The first classification result includes the classification probability of each sentiment type, reflecting the possibility that the first text information belongs to each sentiment type. Among them, the first classification model can be a machine learning model or a deep learning model, and this application does not make specific restrictions.
[0072] Exemplarily, for the first classification result, attention should be paid to the difference in classification probabilities between different emotion types. When the difference in classification probabilities between different emotion types is large, it means that the first classification model has a very clear emotional judgment on the first text information. For example, if the first classification model predicts that the first text information belongs to a certain emotion type with a high probability, while the probability of other emotion types is relatively low, this usually indicates that the first classification model has a high accuracy rate in this emotion classification. In this case, the first fusion weight can be set larger to reflect the credibility of the first classification result and enhance its influence on the overall emotion characteristics. On the contrary, when the difference in classification probabilities between emotion types in the first classification result is small, it means that there is uncertainty in the emotional judgment of the first classification model on the first text information, which may be because the text content is vague, complex, or contains multiple emotions. In this case, the classification accuracy of the model is low, and the first fusion weight should be set smaller to reduce the influence of the emotion characteristics on the final emotion analysis results. This can effectively avoid misleading information introduced by uncertainty, thereby improving the accuracy of overall emotion recognition.
[0073] Exemplarily, the first time feature is firstly sentimentally classified by the trained second classification model to obtain a second classification result, and the second classification result includes the classification probability of each emotion type, reflecting the possibility that the first time feature belongs to each emotion type. Among them, the second classification model can be a machine learning model or a deep learning model, and this application does not make specific restrictions.
[0074] Exemplarily, when focusing on the second classification result, attention should be paid to the classification probability differences between different emotion types. When the classification probability differences between different emotion types are large, it means that the second classification model has a very clear emotional judgment on the first time feature, and the second fusion weight can be set larger to reflect the credibility of the second classification result and enhance its influence on the overall emotional feature. On the contrary, when the classification probability differences between emotion types in the second classification result are small, it means that there is uncertainty in the emotional judgment of the second classification model on the first time feature, and the reliability of emotion classification based on the first time feature is low. The second fusion weight should be set smaller to reduce the influence of the first time feature on the final emotion analysis result.
[0075] Exemplarily, the first frequency feature is firstly sentimentally classified by the trained third classification model to obtain a third classification result, and the third classification result includes the classification probability of each emotion type, reflecting the possibility that the first frequency feature belongs to each emotion type. The third classification model can be a machine learning model or a deep learning model, and this application does not make specific restrictions.
[0076] Exemplarily, when focusing on the third classification result, attention should be paid to the classification probability differences between different emotion types. When the classification probability differences between different emotion types are large, it means that the third classification model has a very clear emotional judgment on the first frequency feature, and the third fusion weight can be set larger to reflect the credibility of the third classification result and enhance its influence on the overall emotional characteristics. On the contrary, when the classification probability differences between emotion types in the third classification result are small, it means that there is uncertainty in the third classification model's emotional judgment on the first frequency feature, and the reliability of emotion classification based on the first frequency feature is low. The third fusion weight should be set smaller to reduce the influence of the first frequency feature on the final emotion analysis result.
[0077] Exemplarily, the first text information, the first time feature and the first frequency feature are weightedly fused according to the first fusion weight, the second fusion weight and the third fusion weight to obtain the first emotion feature corresponding to the target object.
[0078] In some embodiments, the fusing of the second text information, the second time feature, and the second frequency feature to determine the second emotional feature corresponding to the associated object includes: performing emotion classification based on the second text information to obtain a fourth classification result, and determining a fourth fusion weight corresponding to the second text information based on the fourth classification result; performing emotion classification based on the second time feature to obtain a fifth classification result, and determining a fifth fusion weight corresponding to the second time feature based on the fourth classification result; performing emotion classification based on the second frequency feature to obtain a sixth classification result, and determining a sixth fusion weight corresponding to the second frequency feature based on the sixth classification result; performing feature fusion on the second text information, the second time feature, and the second frequency feature based on the fourth fusion weight, the fifth fusion weight, and the sixth fusion weight to obtain the second emotional feature corresponding to the associated object.
[0079] Exemplarily, the second text information is firstly sentimentally classified by the trained fourth classification model to identify the sentiment type expressed in the second text information, such as positive, negative or neutral, so that the fourth classification result can be effectively obtained. The fourth classification result includes the classification probability of each emotion type, reflecting the possibility that the second text information belongs to each emotion type. Among them, the fourth classification model can be a machine learning model or a deep learning model, and this application does not make specific restrictions.
[0080] Exemplarily, when targeting the fourth classification result, attention should be paid to the difference in classification probabilities between different emotion types. When the difference in classification probabilities between different emotion types is large, it means that the fourth classification model has a very clear emotional judgment on the second text information. For example, if the fourth classification model predicts that the second text information belongs to a certain emotion type with a high probability, while the probability of other emotion types is relatively low, this generally indicates that the fourth classification model has a high accuracy rate in this emotion classification. In this case, the fourth fusion weight can be set larger to reflect the credibility of the fourth classification result and enhance its influence on the overall emotion characteristics. On the contrary, when the difference in classification probabilities between emotion types in the fourth classification result is small, it means that there is uncertainty in the emotional judgment of the fourth classification model on the second text information, which may be because the text content is vague, complex, or contains multiple emotions. In this case, the classification accuracy of the model is low, and the fourth fusion weight should be set smaller to reduce the influence of the emotion feature on the final emotion analysis result. This can effectively avoid misleading information introduced by uncertainty, thereby improving the accuracy of overall emotion recognition.
[0081] Exemplarily, the emotion classification of the second time feature is first performed through the trained fifth classification model to obtain a fifth classification result, and the fifth classification result includes the classification probability of each emotion type, reflecting the possibility that the second time feature belongs to each emotion type. Among them, the fifth classification model can be a machine learning model or a deep learning model, and this application does not make specific restrictions.
[0082] Exemplarily, when focusing on the fifth classification result, attention should be paid to the classification probability differences between different emotion types. When the classification probability differences between different emotion types are large, it means that the fifth classification model has a very clear emotional judgment on the second time feature, and the fifth fusion weight can be set larger to reflect the credibility of the fifth classification result and enhance its influence on the overall emotional characteristics. On the contrary, when the classification probability differences between emotion types in the fifth classification result are small, it means that there is uncertainty in the fifth classification model's emotional judgment on the second time feature, and the reliability of emotion classification based on the second time feature is low. The fifth fusion weight should be set smaller to reduce the influence of the second time feature on the final emotion analysis result.
[0083] Exemplarily, the emotion classification of the second frequency feature is first performed through the trained sixth classification model to obtain the sixth classification result, and the sixth classification result includes the classification probability of each emotion type, reflecting the possibility that the second frequency feature belongs to each emotion type. Among them, the sixth classification model can be a machine learning model or a deep learning model, and this application does not make specific restrictions.
[0084] Exemplarily, when focusing on the sixth classification result, attention should be paid to the classification probability differences between different emotion types. When the classification probability differences between different emotion types are large, it means that the sixth classification model has a very clear emotional judgment on the second frequency feature. The sixth fusion weight can be set larger to reflect the credibility of the sixth classification result and enhance its influence on the overall emotional characteristics. On the contrary, when the classification probability differences between emotion types in the sixth classification result are small, it means that the sixth classification model has uncertainty in its emotional judgment on the second frequency feature, and the reliability of emotional classification based on the second frequency feature is low. The sixth fusion weight should be set smaller to reduce the influence of the second frequency feature on the final emotional analysis result.
[0085] Exemplarily, the second text information, the second time feature and the second frequency feature are weightedly fused according to the fourth fusion weight, the fifth fusion weight and the sixth fusion weight to obtain the second emotion feature corresponding to the associated object. Thus, the first emotion feature of the target object and the second emotion feature of the associated object are feature fused, and input into the emotion classification model that has been trained to perform emotion recognition based on the emotion feature, so as to obtain the second emotion information corresponding to the target object. This process not only takes into account the emotion features of the target object itself, but also fully integrates the emotion features of the associated object, so that the emotional state of the target object can be captured more comprehensively.
[0086] Step S105: determining an emotional state corresponding to the target object according to the first emotional information and the second emotional information.
[0087] Exemplarily, by analyzing the probabilities of each emotion category in the first emotion information, the facial emotion expressed by the target object in the initial image information can be determined. Specifically, facial emotion refers to the emotional signal transmitted by facial expressions, which may include joy, sadness, anger and other emotions. When the probability information of a certain emotion category reaches the maximum value, it means that the facial emotion of the target object at that moment is clear and obvious, and can effectively reflect its external emotional state. Similarly, by analyzing the second emotion information, the language emotion expressed by the target object in the initial voice information can be obtained. Language emotion involves a variety of non-verbal factors such as voice intonation, speech speed, and stress, which work together to convey the subtle differences in emotion. When the probability information of a certain emotion category is the largest in the language emotion analysis, the language emotion expressed by the target object at that moment can be inferred.
[0088] Exemplarily, facial emotions and verbal emotions are compared. If the emotional states expressed by the two are consistent, such as both pointing to happiness or sadness, the emotional state of the target object can be considered to be "emotionally normal". This consistency shows that the target object has reached a unity in facial expressions and verbal expressions, reflecting the true emotional experience in the heart. However, when facial emotions are inconsistent with verbal emotions, for example, facial expressions show a smile, but verbal emotions convey sadness, the emotional state of the target object can be judged as "emotionally abnormal". This inconsistency may point to inner conflicts, emotional repression, or maladaptation in social situations, suggesting that the target object may be experiencing psychological discomfort or distress.
[0089] In some embodiments, determining the emotional state corresponding to the target object according to the first emotional information and the second emotional information includes: obtaining a moment difference value between the first emotional information and the second emotional information at different moments; determining a target difference value of the target object between the initial image information and the initial voice information according to the moment difference value; determining the emotional state corresponding to the target object according to the target difference value and a preset value; wherein the target difference value is obtained according to the following formula:
[0090] ;
[0091] in, represents the target difference value, sum represents the number of moments, style represents the total number of emotions, represents the emotion probability corresponding to the i-th type of the first emotion information of the target object at the t-th time, represents the emotion probability corresponding to the j-th type of the second emotion information of the target object at the t-th time, and e represents a natural constant.
[0092] Exemplarily, the first emotion information and the second emotion information corresponding to the target object at different moments are analyzed according to the initial image information and the initial voice information at different moments in the target environment. Then, the moment difference value between the first emotion information and the second emotion information at each moment is calculated. The moment difference value is obtained according to the following formula:
[0093] ;
[0094] in, represents the difference between the first emotion information and the second emotion information at the tth moment, style represents the total number of emotions, represents the emotion probability corresponding to the i-th first emotion information of the target object at the t-th time, It represents the emotion probability corresponding to the j-th second emotion information of the target object at the t-th time, and e represents a natural constant.
[0095] Exemplarily, by using the above formula to calculate the moment difference value between the first emotion information and the second emotion information, the emotional difference between the target objects in different modes at the same moment can be more clearly captured. It can effectively evaluate whether the target object's emotions are consistent in different modes. As a result, the emotional state of the target object can be more comprehensively and accurately evaluated and understood, which has a positive impact in multiple application scenarios.
[0096] By using the above formula to calculate the moment difference value between the first emotional information and the second emotional information, the emotional differences conveyed by different modes (such as facial emotions and verbal emotions) at the same moment can be captured more clearly. The above formula can reveal the differences in emotional expression of the target object in different modes, so that the emotional consistency of the target object in different modes can be effectively evaluated through the above formula, and potential conflicts or consistency in emotional expression can be identified. For example, when facial expressions and voice emotions echo each other, it means that the emotional state of the target object is relatively stable and real; when the two are obviously inconsistent, it may imply inner contradictions or emotional repression. This information is particularly important in fields such as mental health intervention and social interaction analysis.
[0097] Exemplarily, the moment difference value between the first emotion information and the second emotion information of the target object at each moment is obtained, and then the mean of the moment difference values at all moments is calculated to determine the target difference value between the initial image information and the initial voice information of the target object, and the target difference value is obtained according to the following formula:
[0098] ;
[0099] in, represents the target difference value, sum represents the number of moments, style represents the total number of emotions, represents the emotion probability corresponding to the i-th first emotion information of the target object at the t-th time, It represents the emotion probability corresponding to the j-th second emotion information of the target object at the t-th time, and e represents a natural constant.
[0100] Exemplarily, a preset value is set with the help of expert experience or historical data for emotional state judgment in sentiment analysis. The calculated target difference value is compared with the preset value. When the target difference value is less than or equal to the preset value, it can be determined that the emotional state of the target object is "normal emotion". When the target difference value is greater than the preset value, the emotional state of the target object is judged as "abnormal emotion". This situation usually suggests that the target object has a significant deviation in emotional expression, which may reflect inner distress, conflict or the influence of the external environment. By comparing the target difference value with the preset value, the emotional state of the target object can be evaluated more accurately.
[0101] Step S106: determining a target monitoring result corresponding to the target object according to the emotional state.
[0102] For example, when the emotional state of the target object is evaluated as "normal emotion", the corresponding target monitoring result is determined to be "normal monitoring". This judgment indicates that the emotional state of the target object is stable and does not show significant emotional fluctuations or abnormal performance. On the contrary, when the emotional state of the target object is evaluated as "abnormal emotion", the corresponding target monitoring result is determined to be "abnormal monitoring". This state indicates that the target object has a significant deviation in emotional expression, which may indicate that he is facing psychological pressure, emotional distress or other external factors. This judgment of abnormal monitoring can provide an important basis for subsequent intervention and support measures.
[0103] The embodiment of the present invention provides a postpartum care monitoring method and system. The method performs an in-depth analysis of a target object and its social environment by collecting initial image information and initial voice information in a target environment. First, by accurately identifying the target object through the initial image information, not only the target object can be accurately identified, but also the associated objects that are related to it can be identified. This correlation analysis provides valuable data support for understanding the social environment and emotional influencing factors of the target object. Further, the method derives the first emotional information of the target object based on the association relationship between the target object and the associated objects in the initial image information. At the same time, the first voice information of the target object and the second voice information of the associated objects are extracted from the initial voice information. By deeply analyzing these voice data, the system determines the second emotional information of the target object. Thus, by comparing the consistency between the first emotional information of the target object under the initial image information and the second emotional information of the target object under the initial voice information, the real emotional state of the target object can be accurately judged. This cross-modal emotion analysis method significantly improves the accuracy and reliability of emotion recognition. The method can judge whether the target object is consistent in words and deeds in terms of emotional expression by comparing the emotions expressed by the target object in the initial image information and the initial voice information. Thus, the abnormal state of the target object can be discovered in time. It solves the problem that related technologies cannot detect maternal psychological abnormalities in a timely manner, and improves the quality and efficiency of postpartum care. In addition, timely emotion recognition enables medical staff to intervene at critical moments, providing good support for subsequent maternal psychological support and professional help.
[0104] See also Figure 2 , Figure 2A postpartum care monitoring system 200 provided in an embodiment of the present application includes a data acquisition module 201, an image analysis module 202, a speech segmentation module 203, a speech analysis module 204, a state recognition module 205, and a result determination module 206, wherein the data acquisition module 201 is used to collect initial image information and initial speech information corresponding to a target environment, and to perform target recognition on the initial image information to obtain a target object and an associated object of the target object in the same target environment; the image analysis module 202 is used to identify the target object and the associated object in the initial image information according to the target object and the associated object in the initial image information. The first emotion information corresponding to the target object is obtained by the association relationship in the image information; the speech segmentation module 203 is used to obtain the first speech information corresponding to the target object and the second speech information corresponding to the associated object from the initial speech information; the speech analysis module 204 is used to determine the second emotion information corresponding to the target object according to the first speech information and the second speech information; the state recognition module 205 is used to determine the emotional state corresponding to the target object according to the first emotion information and the second emotion information; the result determination module 206 is used to determine the target monitoring result corresponding to the target object according to the emotional state.
[0105] In some implementations, the image analysis module 202, in the process of obtaining the first emotion information corresponding to the target object according to the association relationship between the target object and the associated object in the initial image information, performs:
[0106] Identifying features of the target object in the initial image information to obtain first feature information, and identifying features of the associated object in the initial image information to obtain second feature information;
[0107] Performing scene feature recognition on the initial image information to obtain initial scene features corresponding to the initial image information;
[0108] Performing feature fusion on the initial scene feature, the first feature information and the second feature information to obtain target feature information;
[0109] Obtaining facial expression information corresponding to the target object in the initial scene information;
[0110] Emotion recognition is performed based on the facial expression information and the target feature information to obtain the first emotion information corresponding to the target object.
[0111] In some implementations, the image analysis module 202 performs, during the process of fusing the initial scene feature with the first feature information and the second feature information to obtain target feature information:
[0112] The initial scene feature is merged with the first feature information and the second feature information into the same feature space to obtain a target scene feature corresponding to the initial scene feature, third feature information corresponding to the first feature information, and fourth feature information corresponding to the second feature information;
[0113] Obtaining a first weight corresponding to the target object in the target scene by fusing the third feature information and the target scene information;
[0114] Updating the target scene feature according to the first weight and the third feature information to obtain an updated first scene feature;
[0115] Obtaining a second weight corresponding to the associated object in the target scene by fusing the fourth feature information and the first scene feature;
[0116] Updating the first scene feature according to the second weight and the fourth feature information to obtain an updated second scene feature;
[0117] Performing object feature fusion according to the first weight, the second weight, the third feature information, and the fourth feature information to obtain fifth feature information;
[0118] The target feature information is obtained by performing feature fusion based on the second scene feature and the fifth feature information.
[0119] In some implementations, the speech analysis module 204, in the process of determining the second emotion information corresponding to the target object according to the first speech information and the second speech information, performs:
[0120] Performing voice recognition on the first voice information to obtain first text information and performing voice recognition on the second voice information to obtain second text information;
[0121] Extracting pragmatic paralanguage from the first voice information to obtain first paralanguage information and extracting pragmatic paralanguage from the second voice information to obtain second paralanguage information;
[0122] Calculating the duration of the first paralanguage information in the first voice information to obtain a first time feature corresponding to the first paralanguage information and calculating the appearance frequency of the first paralanguage information in the first voice information to obtain a first frequency feature corresponding to the first paralanguage information;
[0123] Calculating the duration of the second paralanguage information in the second voice information to obtain a second time feature corresponding to the second paralanguage information and calculating the appearance frequency of the second paralanguage information in the second voice information to obtain a second frequency feature corresponding to the second paralanguage information;
[0124] Determine a first emotion feature corresponding to the target object by fusing the first text information, the first time feature, and the first frequency feature;
[0125] Determine a second emotion feature corresponding to the associated object by fusing the second text information, the second time feature, and the second frequency feature;
[0126] The second emotion information corresponding to the target object is obtained according to the first emotion feature and the second emotion feature.
[0127] In some implementations, the speech analysis module 204 performs, during the process of fusing the first text information, the first time feature, and the first frequency feature to determine the first emotion feature corresponding to the target object:
[0128] Performing sentiment classification according to the first text information to obtain a first classification result, and determining a first fusion weight corresponding to the first text information according to the first classification result;
[0129] Performing sentiment classification according to the first time feature to obtain a second classification result, and determining a second fusion weight corresponding to the first time feature according to the second classification result;
[0130] Performing sentiment classification according to the first frequency feature to obtain a third classification result, and determining a third fusion weight corresponding to the first frequency feature according to the third classification result;
[0131] The first text information, the first time feature and the first frequency feature are subjected to feature fusion according to the first fusion weight, the second fusion weight and the third fusion weight to obtain the first emotion feature corresponding to the target object.
[0132] In some implementations, the speech analysis module 204 performs, during the process of fusing the second text information, the second time feature, and the second frequency feature to determine the second emotion feature corresponding to the associated object:
[0133] Performing sentiment classification according to the second text information to obtain a fourth classification result, and determining a fourth fusion weight corresponding to the second text information according to the fourth classification result;
[0134] Performing sentiment classification according to the second time feature to obtain a fifth classification result, and determining a fifth fusion weight corresponding to the second time feature according to the fourth classification result;
[0135] Performing emotion classification according to the second frequency feature to obtain a sixth classification result, and determining a sixth fusion weight corresponding to the second frequency feature according to the sixth classification result;
[0136] The second text information, the second time feature and the second frequency feature are subjected to feature fusion according to the fourth fusion weight, the fifth fusion weight and the sixth fusion weight to obtain the second emotion feature corresponding to the associated object.
[0137] In some implementations, during the process of determining the emotional state corresponding to the target object according to the first emotional information and the second emotional information, the state recognition module 205 performs:
[0138] Obtaining a time difference value between the first emotion information and the second emotion information at different times;
[0139] Determining a target difference value of the target object between the initial image information and the initial voice information according to the moment difference value;
[0140] Determining the emotional state corresponding to the target object according to the target difference value and a preset value;
[0141] The target difference value is obtained according to the following formula:
[0142] ;
[0143] in, represents the target difference value, sum represents the number of moments, style represents the total number of emotions, represents the emotion probability corresponding to the i-th type of the first emotion information of the target object at the t-th time, represents the emotion probability corresponding to the j-th type of the second emotion information of the target object at the t-th time, and e represents a natural constant.
[0144] In some embodiments, the postpartum care monitoring system 200 may be applied to a terminal device.
[0145] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the postpartum care monitoring system 200 described above can refer to the corresponding process in the aforementioned postpartum care monitoring method embodiment, and will not be repeated here.
[0146] See also Figure 3 , Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.
[0147] like Figure 3 As shown, the terminal device 300 includes a processor 301 and a memory 302 , and the processor 301 and the memory 302 are connected via a bus 303 , such as an I2C (Inter-integrated Circuit) bus.
[0148] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0149] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.
[0150] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0151] The processor is used to run a computer program stored in a memory, and implement any one of the postpartum care monitoring methods provided by an embodiment of the present invention when executing the computer program.
[0152] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:
[0153] Collecting initial image information and initial voice information corresponding to the target environment, and performing target recognition on the initial image information to obtain a target object and an associated object of the target object in the same target environment;
[0154] Obtaining first emotion information corresponding to the target object according to the association relationship between the target object and the associated object in the initial image information;
[0155] Obtaining first voice information corresponding to the target object and second voice information corresponding to the associated object from the initial voice information;
[0156] Determine second emotion information corresponding to the target object according to the first voice information and the second voice information;
[0157] Determining an emotional state corresponding to the target object according to the first emotional information and the second emotional information;
[0158] A target monitoring result corresponding to the target object is determined according to the emotional state.
[0159] In some implementations, during the process of obtaining the first emotion information corresponding to the target object according to the association relationship between the target object and the associated object in the initial image information, the processor 301 executes:
[0160] Identifying features of the target object in the initial image information to obtain first feature information, and identifying features of the associated object in the initial image information to obtain second feature information;
[0161] Performing scene feature recognition on the initial image information to obtain initial scene features corresponding to the initial image information;
[0162] Performing feature fusion on the initial scene feature, the first feature information and the second feature information to obtain target feature information;
[0163] Obtaining facial expression information corresponding to the target object in the initial scene information;
[0164] Emotion recognition is performed based on the facial expression information and the target feature information to obtain the first emotion information corresponding to the target object.
[0165] In some implementations, the processor 301, during the process of fusing the initial scene feature with the first feature information and the second feature information to obtain target feature information, executes:
[0166] The initial scene feature is merged with the first feature information and the second feature information into the same feature space to obtain a target scene feature corresponding to the initial scene feature, third feature information corresponding to the first feature information, and fourth feature information corresponding to the second feature information;
[0167] Obtaining a first weight corresponding to the target object in the target scene by fusing the third feature information and the target scene information;
[0168] Updating the target scene feature according to the first weight and the third feature information to obtain an updated first scene feature;
[0169] Obtaining a second weight corresponding to the associated object in the target scene by fusing the fourth feature information and the first scene feature;
[0170] Updating the first scene feature according to the second weight and the fourth feature information to obtain an updated second scene feature;
[0171] Performing object feature fusion according to the first weight, the second weight, the third feature information, and the fourth feature information to obtain fifth feature information;
[0172] The target feature information is obtained by performing feature fusion based on the second scene feature and the fifth feature information.
[0173] In some implementations, during the process of determining the second emotion information corresponding to the target object according to the first voice information and the second voice information, the processor 301 executes:
[0174] Performing voice recognition on the first voice information to obtain first text information and performing voice recognition on the second voice information to obtain second text information;
[0175] Extracting pragmatic paralanguage from the first voice information to obtain first paralanguage information and extracting pragmatic paralanguage from the second voice information to obtain second paralanguage information;
[0176] Calculating the duration of the first paralanguage information in the first voice information to obtain a first time feature corresponding to the first paralanguage information and calculating the appearance frequency of the first paralanguage information in the first voice information to obtain a first frequency feature corresponding to the first paralanguage information;
[0177] Calculating the duration of the second paralanguage information in the second voice information to obtain a second time feature corresponding to the second paralanguage information and calculating the appearance frequency of the second paralanguage information in the second voice information to obtain a second frequency feature corresponding to the second paralanguage information;
[0178] Determine a first emotion feature corresponding to the target object by fusing the first text information, the first time feature, and the first frequency feature;
[0179] Determine a second emotion feature corresponding to the associated object by fusing the second text information, the second time feature, and the second frequency feature;
[0180] The second emotion information corresponding to the target object is obtained according to the first emotion feature and the second emotion feature.
[0181] In some implementations, during the process of fusing the first text information, the first time feature, and the first frequency feature to determine the first emotion feature corresponding to the target object, the processor 301 executes:
[0182] Performing sentiment classification according to the first text information to obtain a first classification result, and determining a first fusion weight corresponding to the first text information according to the first classification result;
[0183] Performing sentiment classification according to the first time feature to obtain a second classification result, and determining a second fusion weight corresponding to the first time feature according to the second classification result;
[0184] Performing sentiment classification according to the first frequency feature to obtain a third classification result, and determining a third fusion weight corresponding to the first frequency feature according to the third classification result;
[0185] The first text information, the first time feature and the first frequency feature are subjected to feature fusion according to the first fusion weight, the second fusion weight and the third fusion weight to obtain the first emotion feature corresponding to the target object.
[0186] In some implementations, during the process of fusing the second text information, the second time feature, and the second frequency feature to determine the second emotion feature corresponding to the associated object, the processor 301 executes:
[0187] Performing sentiment classification according to the second text information to obtain a fourth classification result, and determining a fourth fusion weight corresponding to the second text information according to the fourth classification result;
[0188] Performing sentiment classification according to the second time feature to obtain a fifth classification result, and determining a fifth fusion weight corresponding to the second time feature according to the fourth classification result;
[0189] Performing emotion classification according to the second frequency feature to obtain a sixth classification result, and determining a sixth fusion weight corresponding to the second frequency feature according to the sixth classification result;
[0190] The second text information, the second time feature and the second frequency feature are subjected to feature fusion according to the fourth fusion weight, the fifth fusion weight and the sixth fusion weight to obtain the second emotion feature corresponding to the associated object.
[0191] In some implementations, during the process of determining the emotional state corresponding to the target object according to the first emotional information and the second emotional information, the processor 301 executes:
[0192] Obtaining a time difference value between the first emotion information and the second emotion information at different times;
[0193] Determining a target difference value of the target object between the initial image information and the initial voice information according to the moment difference value;
[0194] Determining the emotional state corresponding to the target object according to the target difference value and a preset value;
[0195] The target difference value is obtained according to the following formula:
[0196] ;
[0197] in, represents the target difference value, sum represents the number of moments, style represents the total number of emotions, represents the emotion probability corresponding to the i-th type of the first emotion information of the target object at the t-th time, represents the emotion probability corresponding to the j-th type of the second emotion information of the target object at the t-th time, and e represents a natural constant.
[0198] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned postpartum care monitoring method embodiment, and will not be repeated here.
[0199] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any postpartum care monitoring method provided in the description of the embodiment of the present invention.
[0200] The storage medium may be an internal storage unit of the terminal device described in the foregoing embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.
[0201] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0202] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.
[0203] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A postpartum care monitoring method, It is characterized in that The method comprises: Collecting initial image information and initial voice information corresponding to the target environment, and performing target recognition on the initial image information to obtain a target object and an associated object of the target object in the same target environment; Obtaining first emotion information corresponding to the target object according to the association relationship between the target object and the associated object in the initial image information; Obtaining first voice information corresponding to the target object and second voice information corresponding to the associated object from the initial voice information; Determine second emotion information corresponding to the target object according to the first voice information and the second voice information; Determining an emotional state corresponding to the target object according to the first emotional information and the second emotional information; A target monitoring result corresponding to the target object is determined according to the emotional state.
2. The method according to claim 1, It is characterized in that The obtaining, according to the association relationship between the target object and the associated object in the initial image information, the first emotion information corresponding to the target object comprises: Identifying features of the target object in the initial image information to obtain first feature information, and identifying features of the associated object in the initial image information to obtain second feature information; Performing scene feature recognition on the initial image information to obtain initial scene features corresponding to the initial image information; Performing feature fusion on the initial scene feature, the first feature information and the second feature information to obtain target feature information; Obtaining facial expression information corresponding to the target object in the initial scene information; Emotion recognition is performed based on the facial expression information and the target feature information to obtain the first emotion information corresponding to the target object.
3. The method according to claim 2, It is characterized in that The step of fusing the initial scene feature with the first feature information and the second feature information to obtain target feature information includes: The initial scene feature is merged with the first feature information and the second feature information into the same feature space to obtain a target scene feature corresponding to the initial scene feature, third feature information corresponding to the first feature information, and fourth feature information corresponding to the second feature information; Obtaining a first weight corresponding to the target object in the target scene by fusing the third feature information and the target scene information; Updating the target scene feature according to the first weight and the third feature information to obtain an updated first scene feature; Obtaining a second weight corresponding to the associated object in the target scene by fusing the fourth feature information and the first scene feature; Updating the first scene feature according to the second weight and the fourth feature information to obtain an updated second scene feature; Performing object feature fusion according to the first weight, the second weight, the third feature information, and the fourth feature information to obtain fifth feature information; The target feature information is obtained by performing feature fusion based on the second scene feature and the fifth feature information.
4. The method according to claim 1, It is characterized in that The determining the second emotion information corresponding to the target object according to the first voice information and the second voice information includes: Performing voice recognition on the first voice information to obtain first text information and performing voice recognition on the second voice information to obtain second text information; Extracting pragmatic paralanguage from the first voice information to obtain first paralanguage information and extracting pragmatic paralanguage from the second voice information to obtain second paralanguage information; Calculating the duration of the first paralanguage information in the first voice information to obtain a first time feature corresponding to the first paralanguage information and calculating the appearance frequency of the first paralanguage information in the first voice information to obtain a first frequency feature corresponding to the first paralanguage information; Calculating the duration of the second paralanguage information in the second voice information to obtain a second time feature corresponding to the second paralanguage information and calculating the appearance frequency of the second paralanguage information in the second voice information to obtain a second frequency feature corresponding to the second paralanguage information; Determine a first emotion feature corresponding to the target object by fusing the first text information, the first time feature, and the first frequency feature; Determine a second emotion feature corresponding to the associated object by fusing the second text information, the second time feature, and the second frequency feature; The second emotion information corresponding to the target object is obtained according to the first emotion feature and the second emotion feature.
5. The method according to claim 4, It is characterized in that The fusing the first text information, the first time feature, and the first frequency feature to determine the first emotion feature corresponding to the target object includes: Performing sentiment classification according to the first text information to obtain a first classification result, and determining a first fusion weight corresponding to the first text information according to the first classification result; Performing sentiment classification according to the first time feature to obtain a second classification result, and determining a second fusion weight corresponding to the first time feature according to the second classification result; Performing sentiment classification according to the first frequency feature to obtain a third classification result, and determining a third fusion weight corresponding to the first frequency feature according to the third classification result; The first text information, the first time feature and the first frequency feature are subjected to feature fusion according to the first fusion weight, the second fusion weight and the third fusion weight to obtain the first emotion feature corresponding to the target object.
6. The method according to claim 4, It is characterized in that The fusing the second text information, the second time feature, and the second frequency feature to determine the second emotion feature corresponding to the associated object includes: Performing sentiment classification according to the second text information to obtain a fourth classification result, and determining a fourth fusion weight corresponding to the second text information according to the fourth classification result; Performing sentiment classification according to the second time feature to obtain a fifth classification result, and determining a fifth fusion weight corresponding to the second time feature according to the fourth classification result; Performing emotion classification according to the second frequency feature to obtain a sixth classification result, and determining a sixth fusion weight corresponding to the second frequency feature according to the sixth classification result; The second text information, the second time feature and the second frequency feature are subjected to feature fusion according to the fourth fusion weight, the fifth fusion weight and the sixth fusion weight to obtain the second emotion feature corresponding to the associated object.
7. The method according to claim 1, It is characterized in that The determining the emotional state corresponding to the target object according to the first emotional information and the second emotional information includes: Obtaining a time difference value between the first emotion information and the second emotion information at different times; Determining a target difference value of the target object between the initial image information and the initial voice information according to the moment difference value; Determining the emotional state corresponding to the target object according to the target difference value and a preset value; The target difference value is obtained according to the following formula: ; in, represents the target difference value, sum represents the number of moments, style represents the total number of emotions, represents the emotion probability corresponding to the i-th type of the first emotion information of the target object at the t-th time, represents the emotion probability corresponding to the j-th type of the second emotion information of the target object at the t-th time, and e represents a natural constant.
8. A postpartum care monitoring system, It is characterized in that include: A data acquisition module, used to acquire the initial image information and initial voice information corresponding to the target environment, and to perform target recognition on the initial image information to obtain the target object and the associated objects of the target object in the same target environment; An image analysis module, configured to obtain first emotion information corresponding to the target object according to an association relationship between the target object and the associated object in the initial image information; A speech segmentation module, used for obtaining the first speech information corresponding to the target object and the second speech information corresponding to the associated object from the initial speech information; A speech analysis module, used to determine second emotion information corresponding to the target object according to the first speech information and the second speech information; A state recognition module, used to determine the emotional state corresponding to the target object according to the first emotional information and the second emotional information; A result determination module is used to determine the target monitoring result corresponding to the target object according to the emotional state.
9. A terminal device, It is characterized in that The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program and implement the postpartum care monitoring method according to any one of claims 1 to 7 when executing the computer program.
10. A computer storage medium for computer storage, It is characterized in that The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the postpartum care monitoring method described in any one of claims 1 to 7.
