Multi-source data respiratory state discrimination apparatus and electronic device

Abstract

This invention provides a multi-source data respiratory state discrimination device and electronic device. The device includes: a sample feature extraction module for determining audio sample features and visual sample features of a target user under different respiratory states; a first comparison module for comparing the currently acquired first audio signal and audio sample features of the target user to obtain an audio confidence score; a second comparison module for comparing the currently acquired first visual signal and visual sample features of the target user to obtain a visual discrimination result; a determination module for obtaining a preliminary recognition result based on the visual discrimination result; and using the audio confidence score to assist in verifying the preliminary recognition result to obtain a target recognition result. This device and electronic device can improve the accuracy of respiratory state discrimination in respiratory training application scenarios.

Description

Technical Field

[0001] This invention relates to the field of technology, and in particular to a multi-source data respiratory status discrimination device and electronic device. Background Technology

[0002] There is a close two-way relationship between human breathing and emotions. Emotional state can change breathing patterns, and actively regulating breathing can, in turn, affect emotions. The physiological basis of this relationship lies primarily in the autonomic nervous system, which is divided into the sympathetic and parasympathetic nervous systems. Activation of the sympathetic nervous system accelerates breathing and increases heart rate; activation of the parasympathetic nervous system slows breathing and decreases heart rate. Breathing not only reflects emotional state but also acts as a powerful, actively controllable valve. Breathing training can influence the autonomic nervous system, thereby regulating emotions. For example, people recovering from drug addiction often experience anxiety, depression, and mood disorders. Guiding them to perform slow, deep exhalation training is a non-pharmacological, low-cost, and easily adhered-to emotional support intervention. A necessary prerequisite for effective breathing training is the ability to accurately monitor the user's breathing status. Currently, judging breathing status often relies on a single data source, such as audio signals, making it easy for environmental interference factors to severely impact accuracy, resulting in low accuracy in breathing status assessment. Summary of the Invention

[0003] This invention provides a multi-source data respiratory status discrimination device and electronic device to improve the accuracy of user respiratory status monitoring.

[0004] This invention provides a breathing state discrimination device, comprising: a sample feature extraction module for determining audio sample features and visual sample features of a target user under different breathing states; a first comparison module for comparing a currently acquired first audio signal and audio sample features of the target user to obtain an audio confidence score; a second comparison module for comparing a currently acquired first visual signal and visual sample features of the target user to obtain a visual discrimination result; a determination module for obtaining a preliminary recognition result based on the visual discrimination result; and using the audio confidence score to assist in verifying the preliminary recognition result to obtain a target recognition result.

[0005] In one embodiment, the sample feature extraction module, when determining the audio sample features of the target user under different breathing states, is specifically used to: collect the audio signals generated by the target user under the states of exhalation, inhalation and breath-holding, respectively, as audio signal samples; and determine the audio sample features based on the audio signal samples.

[0006] In one embodiment, the sample feature extraction module includes a first extraction unit and a second extraction unit; the audio sample features include first audio sample features and second audio sample features; when determining audio sample features based on audio signal samples, the sample feature extraction module is specifically used to: call the first extraction unit to calculate the root mean square energy (RMS) of the audio signal sample; call the second extraction unit to calculate the normalized frequency amplitude spectrum of the audio signal sample; use the RMS of the audio signal sample as the first audio sample feature; and use the normalized frequency amplitude spectrum of the audio signal sample as the second audio sample feature.

[0007] In one embodiment, the first comparison module, when comparing the currently acquired first audio signal of the target user with audio sample features to obtain audio confidence, specifically performs the following steps: calling a first extraction unit to calculate the first root-mean-square energy of the first audio signal; calling a second extraction unit to calculate the first normalized frequency amplitude spectrum of the first audio signal; comparing the first audio sample features with the first root-mean-square energy to obtain a first comparison result; comparing the second audio sample features with the first normalized frequency amplitude spectrum to obtain a second comparison result; and obtaining the audio confidence based on the first comparison result and the second comparison result.

[0008] In one embodiment, when comparing the first audio sample features and the first root mean square energy to obtain a first comparison result, the first comparison module is specifically used to: calculate the first function value of the two-sided negative exponential function or the one-sided negative exponential function centered on the first audio sample features, with the first root mean square energy as the independent variable; and obtain the first comparison result based on the difference between the peak value corresponding to the first audio sample features and the first function value.

[0009] In one embodiment, when comparing the features of the second audio sample and the first normalized frequency amplitude spectrum to obtain a second comparison result, the first comparison module is specifically used to: calculate the similarity between the features of the second audio sample and the first normalized frequency amplitude spectrum as the second comparison result.

[0010] In one embodiment, the sample feature extraction module, when determining the visual sample features of the target user under different breathing states, is specifically used to: acquire images generated by the target user under exhalation, inhalation, or breath-holding states respectively; and determine the visual sample features based on the images.

[0011] In one embodiment, the sample feature extraction module, when extracting visual signal features from an image, is specifically configured to: call a face recognition service interface to perform face recognition on the image and obtain at least two boundary points of the target user's mouth; based on the at least two boundary points, obtain the target user's first average mouth opening degree in exhalation, the second average mouth opening degree in inhalation, and the third average mouth opening degree in breath-holding; perform a weighted summation of the first average mouth opening degree, the second average mouth opening degree, and the third average mouth opening degree to obtain an exhalation mouth opening degree threshold, and use the exhalation mouth opening degree threshold as a visual sample feature; the second comparison module, when comparing the currently acquired first visual signal of the target user with the visual sample features to obtain a visual discrimination result, is specifically configured to: determine the first mouth opening degree based on the currently acquired first visual signal of the target user; compare the mouth opening degree with the exhalation mouth opening degree threshold to obtain a visual discrimination result indicating whether the target user is in an exhalation state.

[0012] In one embodiment, the determining module is specifically configured to: output a target recognition result indicating that the breathing state is exhaling when the visual discrimination result indicates that the target user is currently in an exhaling state and the audio confidence score indicates that the target user is currently in an exhaling state; output a target recognition result indicating that the breathing state is inhaling when the visual discrimination result indicates that the target user is not currently in an exhaling state and the audio confidence score indicates that the target user is currently in an inhaling state; and output a target recognition result indicating that the breathing state is in a breath-holding state when the visual discrimination result indicates that the target user is not currently in an exhaling state and the audio confidence score indicates that the target user is currently in a breath-holding state.

[0013] In one embodiment, the device further includes a data acquisition module, configured to: acquire audio sequences and / or image sequences generated by the target user over a preset duration; process the audio sequences and / or image sequences using a moving average algorithm with a preset time window to obtain a smooth sequence; and calculate the duration of multiple historical respiratory cycles based on the smooth sequence, and obtain the average respiratory cycle duration by weighted averaging.

[0014] Secondly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following functions: determining audio sample features and visual sample features of a target user under different breathing states; comparing the currently acquired first audio signal of the target user with the audio sample features to obtain an audio confidence score; comparing the currently acquired first visual signal of the target user with the visual sample features to obtain a visual discrimination result; obtaining a preliminary recognition result based on the visual discrimination result; and using the audio confidence score to assist in verifying the preliminary recognition result to obtain a target recognition result.

[0015] In one embodiment, when the processor executes the computer program, it performs the following specific functions: acquiring audio signals generated by the target user during exhalation, inhalation, and breath-holding, respectively, as audio signal samples; calling a first extraction unit to calculate the root mean square energy of the audio signal samples; calling a second calculation unit to calculate the normalized frequency amplitude spectrum of the audio signal samples; using the root mean square energy of the audio signal samples as a first audio sample feature; and using the normalized frequency amplitude spectrum of the audio signal samples as a second audio sample feature.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described respiratory state discrimination devices.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the breathing state discrimination devices described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the breathing state discrimination devices described above.

[0019] The multi-source data respiratory state discrimination device and electronic device provided by this invention first extracts audio sample features and visual sample features corresponding to different respiratory states of the target user. By comparing the currently acquired audio signal with the audio sample features, an audio-dimensional discrimination result—audio confidence score—is obtained. By comparing the currently acquired visual signal with the visual sample features, a visual-dimensional discrimination result is obtained. A dual discrimination mechanism of visual-based and audio-assisted verification is adopted. First, a preliminary recognition result is obtained based on the visual discrimination result. Then, the audio confidence score is used to verify the preliminary recognition result to obtain the final target recognition result. Thus, this scheme achieves dual verification of audio and visual signals by fusing the discrimination result of the visual signal, which can correct misjudgments that may occur when relying solely on audio and improve the accuracy of respiratory state discrimination. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1This is a schematic diagram of the software module of the respiratory state discrimination device provided in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of the electronic device provided in the embodiment of the present invention.

[0022] Figure label: 101: Sample feature extraction module; 102: First comparison module; 103: Second comparison module; 104: Determination module; 210: Processor; 220: Communication interface; 230: Memory; 240: Communication bus. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Studies have shown that breathing training interventions have a significant effect on mood regulation and alleviating depression and anxiety. Breathing patterns can inversely regulate mood. For example, slow breathing patterns, especially those with prolonged exhalation (such as the 4-7-8 breathing method or long exhalation training), stimulate the vagus nerve (the main trunk of the parasympathetic nervous system), sending signals of safety and relaxation to the locus coeruleus, amygdala, and other emotional centers in the brain. This lowers heart rate, blood pressure, and stress hormones (such as cortisol), effectively reducing anxiety and tension and promoting calmness and relaxation. Rapid breathing (such as the active, forceful breathing in some "release" breathing techniques) slightly activates the sympathetic nervous system, which can temporarily increase alertness, excitement, and energy levels, and so on.

[0025] Correspondingly, different emotions trigger typical changes in breathing patterns, a direct response of the autonomic nervous system. For example, in states of anxiety, fear, and tension, breathing becomes shallow and rapid (increased respiratory rate, but smaller volume of air inhaled each time), primarily using the chest cavity for auxiliary breathing, sometimes accompanied by sighing or breath-holding. In this state, the sympathetic nervous system is activated, and the body prepares to cope with threats. In states of relaxation, calmness, and happiness, breathing becomes deep and slow, usually diaphragmatic breathing dominated by the diaphragm, with a steady rhythm. At this time, the parasympathetic nervous system is dominant, and the body is in a state of rest and recovery. In states of anger and frustration, breathing may be deep and forceful, even with explosive exhalation, while heart rate and blood pressure increase, also a manifestation of highly active sympathetic nervous systems. In states of sadness and depression, breathing may become slow and weak, but accompanied by irregular pauses or frequent sighs, and overall ventilation may be insufficient.

[0026] Respiratory status monitoring is an important prerequisite for implementing breathing training. By accurately identifying the user's breathing status, targeted breathing training can be conducted, thereby achieving emotion regulation.

[0027] Current breathing discrimination solutions often rely on a single data source (such as acoustic signals) to determine breathing. This makes it easy for interference factors in the environment to seriously affect the accuracy of the discrimination. The robustness and anti-interference ability of breathing state recognition are poor, and the accuracy of the recognition results is difficult to guarantee.

[0028] In view of this, embodiments of the present invention provide a multi-source data respiratory state discrimination device and electronic device, which combines acoustic signals and computer vision signals to perform discrimination through multi-source data fusion, effectively improving the system's discrimination accuracy and robustness to environmental interference factors.

[0029] This invention provides a multi-source data respiratory state discrimination device and electronic device, which can be applied to many application scenarios such as drug rehabilitation training, rehabilitation medicine and psychological intervention, human-computer interaction and serious games, driving safety and fatigue monitoring, intelligent fitness and sports training, sleep monitoring and health management, education assessment and concentration training.

[0030] For example, in rehabilitation medicine and psychological intervention settings for anxiety disorders, depression, or substance addiction, it is necessary to objectively assess whether the user's breathing training is effective. This can be achieved by capturing the user's visual signals (e.g., facial images) through a camera and their audio signals (or acoustic signals) through a microphone. The device provided in this invention can accurately identify whether the user is holding their breath, exhaling, or inhaling. If the user's breathing is disordered due to agitation (noisy audio), the visual signals can help obtain accurate breathing status identification results.

[0031] In driving safety and fatigue monitoring scenarios, the device provided in this invention can be used to monitor the driver's state. When driving while fatigued, the breathing pattern becomes deep and irregular; when emotionally agitated, breathing becomes rapid. For example, an existing in-vehicle dashcam or other image acquisition device (acquiring visual signals) and an in-vehicle microphone (acquiring audio signals) can be used. When the confidence level of the audio signal is low (e.g., music is playing in the car, or passengers are talking), the visual signal can accurately determine the breathing state by monitoring the driver's facial features (e.g., mouth opening and closing), thereby providing an early warning of fatigued driving or road rage.

[0032] The multi-source data respiratory state discrimination device and electronic device of the present invention are described below with reference to the accompanying drawings.

[0033] Figure 1 This is one of the schematic diagrams of the module architecture of the respiratory state discrimination device provided by the present invention, such as... Figure 1 As shown, it can specifically include the following modules: The sample feature extraction module 101 is used to determine the audio sample features and visual sample features of the target user under different breathing states.

[0034] Different breathing states, including at least two of exhalation, inhalation, and breath-holding.

[0035] Exhalation specifically refers to the process of the diaphragm and external intercostal muscles relaxing, reducing the volume of the thoracic cavity, increasing internal air pressure, and expelling gas (rich in carbon dioxide) from the lungs. Inhalation specifically refers to the process of the diaphragm and external intercostal muscles contracting, expanding the volume of the thoracic cavity, decreasing internal air pressure, and allowing outside air (rich in oxygen) to enter the lungs through the respiratory tract. Breath-holding refers to the state of actively or passively temporarily ceasing breathing, neither inhaling nor exhaling.

[0036] The audio and visual sample features extracted by the sample feature extraction module 101 are reference features obtained during the detection preparation phase and are used as a reference benchmark. The audio sample features reflect the personalized characteristics generated in the audio dimension when the target user breathes. The visual sample features reflect the personalized characteristics generated in the visual dimension when the target user breathes.

[0037] Specifically, in some embodiments, the sample feature extraction module is configured to perform the following operations when determining the audio sample features of the target user under different breathing states: collecting the audio signals generated by the target user under exhalation, inhalation and breath-holding states respectively as audio signal samples; and determining the audio sample features based on the audio signal samples.

[0038] In some embodiments, the sample feature extraction module is specifically configured to perform the following operations when determining the visual sample features of the target user under different breathing states: acquiring images generated by the target user under exhalation, inhalation, or breath-holding states; and determining visual sample features based on the images.

[0039] The first comparison module 102 is used to compare the first audio signal of the target user and the audio sample features currently collected to obtain the audio confidence level.

[0040] The first audio signal is the audio signal generated by the target user's breathing at the current moment, which is collected in real time by the first comparison module 102 during the detection phase. By comparing the first audio signal with the pre-determined audio sample features of the target user under different breathing states, the target user's current breathing state can be identified from the audio dimension.

[0041] The obtained audio confidence can specifically include the confidence of the target user's current breathing state as exhalation, inhalation, and breath-holding.

[0042] The second comparison module 103 is used to compare the first visual signal and visual sample features of the target user currently collected to obtain the visual discrimination result.

[0043] The first visual signal is a visual signal generated by the target user's breathing at the current moment, which is collected in real time by the first comparison module 102 during the discrimination phase. For example, the first visual signal may be an image that includes at least the target user's mouth, such as a facial image. By comparing the first visual signal with pre-determined visual sample features of the target user under different breathing states, the target user's current breathing state can be identified from a visual dimension.

[0044] The determination module 104 is used to obtain a preliminary identification result of the target user's current breathing state based on the visual discrimination result; and to verify the preliminary identification result using audio confidence to obtain the target identification result.

[0045] The device proposed in this invention combines visual discrimination results as the primary factor with audio confidence levels to comprehensively determine the breathing state of a target user. Related technologies that determine breathing state based on data features from a single audio dimension are easily affected by factors such as environmental noise and microphone placement. Figure 1 The device shown employs a dual discrimination mechanism that combines visual and audio verification. By integrating visual discrimination results with audio confidence levels, it can correct misjudgments that may occur when relying solely on audio, effectively improving the accuracy of discrimination and its robustness to interference factors in the environment.

[0046] Specific examples are listed below.

[0047] In this specific embodiment, the multi-source data breathing state discrimination device may include an interactive interface, which is used to connect to devices such as a display screen, touch screen, or speaker. The user interactive interface is displayed on the display screen or touch screen, and guidance information is played by displaying guidance information and / or playing it through voice. The guidance information may include one or more of text, images, voice, and video to guide the user to configure devices such as microphone and camera.

[0048] Next, the multi-source data breathing state discrimination device triggers the sample feature extraction module to collect sample features in the audio (acoustic signal) dimension and determine the audio sample features of the target user in different breathing states.

[0049] During the audio sample feature acquisition phase, the audio signals of user A in three states—exhalation, inhalation, and breath-holding—were picked up and recorded.

[0050] Then, the sample feature extraction module determines the audio sample features based on the sampled audio signals. In this specific embodiment, determining the audio sample features based on the audio signals can specifically involve extracting features from sampled samples of multiple audio signals to obtain audio signal features, and then obtaining audio sample features based on multiple audio signal features. For example, the audio sample features can be obtained by calculating the mean or median of multiple audio signal features, or by first calculating the standard deviation of multiple audio signal features, removing outlier samples based on the mean and standard deviation, and then calculating the mean of the remaining samples, and so on.

[0051] The sample feature extraction module may include a first extraction unit and a second extraction unit, and the audio sample features may include first audio sample features and second audio sample features. The first extraction unit is used to extract the first audio sample features, and the second extraction unit is used to extract the second audio sample features.

[0052] For example, the extraction of audio signal features from an audio signal can be performed as follows: Calculate the root mean square energy (RMS) of the audio signal: ; An audio signal consists of N sampling points with floating-point values. An array (the first array) is formed. Represents the first element in the first array. i The audio signal has N sampling points, where N represents the number of sampling points.

[0053] The calculated root mean square energy (RMS) of the audio signal is used as the first audio sample feature. Alternatively, for multiple audio signals from user A, the RMS of each signal is calculated, and the average or weighted sum of the multiple RMS values ​​is obtained to get the first audio sample feature.

[0054] The process of calculating the root mean square energy of an audio signal can be achieved by calling the first extraction unit and calculating the root mean square energy of the audio signal sample through the first extraction unit.

[0055] Calculate the normalized frequency amplitude spectrum Spec[k] of the audio signal: ; This represents the complex spectrum of the audio segment; The amplitude spectrum is an array of length N (defined as the second array). n This indicates the index of the element in the first array. k Represents the element number in the spectrum array (second array).n and k The values ​​range from 0 to N-1.

[0056] The normalized frequency amplitude spectrum Spec[k] of the calculated audio signal is used as the second audio sample feature. Alternatively, for multiple audio signals from user A, the normalized frequency amplitude spectrum Spec[k] is calculated for each segment, and the average or weighted sum of the multiple normalized frequency amplitude spectra Spec[k] is obtained to get the second audio sample feature.

[0057] The normalized frequency amplitude spectrum of the audio signal can be calculated by calling the second extraction unit and receiving the result of the normalized frequency amplitude spectrum of the audio signal sample calculated by the second extraction unit.

[0058] The extraction process of visual sample features can be performed synchronously or sequentially with the extraction process of audio sample features. This embodiment of the invention does not limit the execution order of the two.

[0059] The multi-source data breathing state discrimination device triggers the sample feature extraction module to determine the visual sample features of the target user under different breathing states. Specifically, it performs computer vision signal processing on the visual signals (e.g., facial images) generated when the target user breathes. During the sample collection phase, the multi-source data respiratory state discrimination device triggers the sample feature extraction module to acquire facial images of the target user during exhalation, inhalation, and breath-holding. Using facial recognition technology, at least two boundary points from multiple boundary points (upper, lower, left, and right) of the target user's mouth are marked. The facial recognition service interface is then invoked to obtain the coordinate data of these boundary points. Based on the coordinate data of at least two boundary points, the average mouth opening degree of the target user under different respiratory states is obtained. For example, the vertical mouth opening degree can be obtained from the upper and lower boundary points; the horizontal mouth opening degree can be obtained from the left and right boundary points.

[0060] Record the average mouth opening of the target user during each phase of exhalation, inhalation, and breath-holding: ; This represents the three-dimensional spatial coordinates of the left boundary point of the mouth. This represents the three-dimensional spatial coordinates of the right boundary point of the mouth. This represents the three-dimensional spatial coordinates of the upper boundary point of the mouth. This represents the three-dimensional spatial coordinates of the lower boundary point of the mouth. ∥∥ represents the Euclidean distance. The Euclidean distance between the left and right boundary points; This represents the Euclidean distance between the upper and lower boundary points.

[0061] The average mouth opening during exhalation, inhalation, and breath-holding is determined based on visual signal features extracted from the visual signal (the target user's facial image). Further visual sample features are then determined based on these visual signal features. Specifically, the process for further determining visual sample features is as follows: The multi-source data respiratory state discrimination device triggers the sample feature extraction module to obtain the expiratory mouth opening threshold by performing a weighted average calculation based on the average mouth opening during expiration, inhalation, and breath-holding. ; k This indicates three states: exhalation, inhalation, and breath-holding. For example, k =1 indicates exhalation. k =2 indicates inhalation. k =3 indicates holding your breath; The weights of the average mouth opening and closing degree corresponding to the three states are empirical values.

[0062] The obtained threshold for the opening and closing of the exhalation mouth is the feature of the video sample.

[0063] Based on the above exemplary description, the sample feature extraction module, when extracting visual signal features from an image, is specifically configured to perform the following operations: Call the face recognition service interface to perform face recognition on the image, obtaining at least two boundary points of the target user's mouth; based on the at least two boundary points, obtain the first average mouth opening degree of the target user in exhalation, the second average mouth opening degree in inhalation, and the third average mouth opening degree in breath-holding. Then, perform a weighted summation of the first average mouth opening degree, the second average mouth opening degree, and the third average mouth opening degree to obtain the exhalation mouth opening degree threshold, which is the visual sample feature.

[0064] The process of determining the features of audio samples and visual samples described above is the detection preparation stage. Next, the discrimination stage begins. The multi-source data breathing state discrimination device triggers the first comparison module to compare the features in the audio dimension and triggers the second comparison module to compare the features in the visual dimension.

[0065] During the discrimination phase, the first audio signal generated by the target user's breathing at the current moment is acquired. The root-mean-square energy and normalized frequency amplitude spectrum of the measured first audio signal are calculated. For example, the first extraction unit is called to obtain the first root-mean-square energy of the first audio signal, and the second extraction unit is called to calculate the first normalized frequency amplitude spectrum of the first audio signal.

[0066] Then, the first root mean square energy is compared with the features of the first audio sample, and the first normalized frequency amplitude spectrum is compared with the features of the second audio sample to obtain the acoustic confidence (i.e., audio confidence) of the target user's current state of exhalation, inhalation, and breath-holding. The corresponding state is determined based on the magnitude of the audio confidence. For example, if the obtained audio confidence is [0.2, 0.79, 0.01], where 0.2 is the confidence for exhalation, 0.79 is the confidence for inhalation, and 0.01 is the confidence for breath-holding, then the target user's current breathing state is inhalation. As another example, if the obtained audio confidence is [0.65, 0.15, 0.20], where 0.65 is the confidence for exhalation, 0.15 is the confidence for inhalation, and 0.20 is the confidence for breath-holding, then the target user's current breathing state is exhalation.

[0067] The audio confidence score is obtained based on a comparison of two parts. Specifically, in this embodiment, a first comparison result is obtained by comparing the first audio sample features and the first root mean square energy; a second comparison result is obtained by comparing the second audio sample features and the first normalized frequency amplitude spectrum; and the audio confidence score is obtained based on the first and second comparison results. The first comparison result can be obtained based on a two-sided / one-sided negative exponential function centered on the first audio sample features, and the second comparison result can be obtained by calculating the similarity between the second audio sample features and the first normalized frequency amplitude spectrum.

[0068] For example, the first comparison module is specifically configured to perform the following operations to obtain a first comparison result: based on a two-sided negative exponential function or a one-sided negative exponential function centered on the features of the first audio sample, with the first root mean square energy as the independent variable, calculate the first function value of the two-sided negative exponential function or the one-sided negative exponential function, and obtain the first comparison result based on the difference between the peak value corresponding to the features of the first audio sample and the first function value.

[0069] For example, the expression for a two-sided negative exponential function is as follows: ; The expression for a one-sided negative exponential function is as follows: ; in, Audio root mean square energy, The root-mean-square energy baseline (first root-mean-square energy) represents the three states of inhalation, exhalation, and breath-holding.

[0070] It should be noted that, in one embodiment, a bilateral negative exponential function is used to calculate the first function value during breath-holding and inhalation, and a unilateral negative exponential function is used to calculate the first function value during exhalation.

[0071] The first comparison result is obtained based on the difference between the peak value corresponding to the first audio sample feature and the first function value. This can be achieved by normalizing the difference and then subtracting the normalized difference from the constant 1, with the resulting value serving as the first comparison result.

[0072] For example, the second comparison module is specifically configured to perform the following operation to obtain a second comparison result: calculate the similarity between the features of the second audio sample and the first normalized frequency amplitude spectrum as the second comparison result.

[0073] For example, similarity can be one of the following: cosine similarity, Euclidean distance, Manhattan distance, or Pearson correlation coefficient.

[0074] Based on the first and second comparison results, the audio confidence score is obtained. Specifically, this can be achieved by weighted summation of the first and second comparison results, with the weight of the first comparison result set as the first weight (e.g., 0.45-0.65) and the weight of the second comparison result set as the second weight (e.g., 0.55-0.35). For example, if the first weight is set to 0.5 and the second weight is also set to 0.5, and the first comparison result is [0.8, 0.1, 0.1] and the second comparison result is [0.76, 0.01, 0.22], then the final calculated audio confidence score is [0.78, 0.06, 0.16], where 0.78 represents the confidence score for exhalation, 0.06 represents the confidence score for inhalation, and 0.16 represents the confidence score for breath-holding.

[0075] During the discrimination phase, the multi-source data breathing state discrimination device triggers the second comparison module to perform visual dimension feature comparison. Based on the real-time acquired facial image of the target user (first visual signal), the first mouth opening degree is determined. Specifically, based on the currently acquired first visual signal of the target user, the first mouth opening degree is determined. This can be achieved by using facial recognition technology to mark at least two boundary points from multiple boundary points (upper, lower, left, and right) of the target user's mouth, and then calculating the first mouth opening degree of the target user at the current moment based on these at least two boundary points. For example, the vertical mouth opening degree can be obtained based on the upper and lower boundary points; the horizontal mouth opening degree can be obtained based on the left and right boundary points.

[0076] Then, the first mouth opening degree is compared with the exhalation mouth opening degree threshold to obtain a visual discrimination result. For example, if it is greater than or equal to the exhalation mouth opening degree threshold, the target user's current breathing state is judged to be exhalation. If the first mouth opening degree is less than the exhalation mouth opening degree threshold, the target user's current breathing state is judged to be non-exhalation (inhalation or breath-holding).

[0077] In other words, the second comparison module, when comparing the first visual signal of the target user and the visual sample features to obtain the visual discrimination result, is specifically configured to perform the following operations: determine the first mouth opening degree based on the first visual signal of the target user, compare the mouth opening degree with the exhalation mouth opening degree threshold, and obtain the visual discrimination result of whether the target user is in an exhalation state.

[0078] During the discrimination stage, the first audio signal and the first visual signal can be a frame of audio and video data. That is, the first audio signal and the first visual signal can be a frame of audio and a frame of image (face image) with the same timestamp. After comparing the first audio signal and the first visual signal separately, the device will combine the audio confidence and the visual discrimination result for comprehensive discrimination. First, a preliminary situation classification is made based on the visual discrimination result to obtain a preliminary recognition result. If the visual discrimination result is that the person is exhaling, the audio confidence is further combined for auxiliary discrimination to obtain the final discrimination result (target recognition result).

[0079] Specifically, the visual discrimination result is a binary classification result of exhalation and non-exhalation. Based on the visual discrimination result, it can be determined whether the target user is currently in an exhalation state. The audio confidence score specifically provides the confidence score of which specific breathing state the target user is currently in.

[0080] If the visual discrimination result indicates that the target user is currently in an exhalation state, and the audio confidence score also indicates that the target user is currently in an exhalation state, then the target user's breathing state is determined to be exhalation, and the target recognition result indicating that the breathing state is exhalation is output.

[0081] If the visual discrimination result indicates that the target user is not currently in an exhalation state, and the audio confidence score indicates that the target user is currently in an inhalation state, then the target recognition result with the breathing state being inhalation will be output.

[0082] If the visual discrimination result indicates that the target user is not currently in an exhalation state, and the audio confidence indicates that the target user is currently in a breath-holding state, then the target recognition result indicating that the breathing state is a breath-holding state will be output.

[0083] If there is a significant discrepancy between the visual discrimination result and the audio confidence score, for example, if the visual discrimination result indicates that the target user is currently in a non-exhaling state, while the audio confidence score shows the highest confidence score for exhalation, then the audio and visual signals should be collected again and compared once more. If the discrepancy persists after multiple consecutive comparisons, then the visual discrimination result should be taken as the primary basis.

[0084] Optionally, in some embodiments, after determining the breathing state, at least one sensory feedback is provided to the user. For example, the determination result is displayed in real time via a breathing indicator on the screen. During breath-holding, the indicator is displayed as a horizontal line; during inhalation, the indicator is displayed as a slightly undulating water wave shape; during exhalation, the indicator is displayed as a large, undulating wave. In this way, the user can intuitively feel the visual feedback brought about by the changes in their own breathing state.

[0085] It should be noted that, in practice, audio and video data arrive at the device continuously, forming audio and / or image sequences. The device stores audio and / or image sequences within a preset historical duration (e.g., 1 minute to 3 minutes). To prevent data jitter from affecting detection, the device uses a short-window moving average algorithm to process the sequences, creating a smooth sequence. The value of the short window should avoid perceptible delay; for example, every 3 to 10 frames constitutes a short window.

[0086] Using a smooth sequence, the device can calculate the duration of several past respiratory cycles and obtain the current average respiratory cycle duration by weighted averaging.

[0087] Indicates the first respiratory cycle in the past several respiratory cycles. i The duration of a respiratory cycle i This is the numbering of the respiratory cycle; Indicates assigning the first i Weight of each respiratory cycle.

[0088] It should be noted that the smooth sequence and respiratory cycle duration here are based on the discrimination results of the above discrimination scheme, and are used to detect the user's respiratory duration index and avoid frequent changes in the system detection status.

[0089] Specifically, based on the above exemplary description, the multi-source data respiratory state discrimination device may further include an acquisition module. The acquisition module is used to acquire audio sequences and / or image sequences generated by the target user within a preset duration, process the audio sequences and / or image sequences using a moving average algorithm with a preset time window to obtain a smooth sequence, and then calculate the duration of multiple historical respiratory cycles based on the smooth sequence, and obtain the average respiratory cycle duration by weighted averaging.

[0090] In one specific embodiment, the device can store the calculated sound intensity values ​​for each audio frame over the past minute, forming a time-ordered raw data sequence. A sliding window is defined as a window of three frames in length. This window moves one frame at a time along the time sequence. Next, the average value within the window is calculated: for the frame currently being judged, the device takes the frame itself and the two preceding frames (a total of three frames), and calculates the arithmetic mean of the sound intensities of these three frames. As the window slides forward frame by frame, the above average value is calculated for each frame, and the calculated smoothed values ​​constitute a new, smoother sequence. This smoothed sequence eliminates random spikes or troughs (i.e., high-frequency jitter) that may appear in individual frames. When making a judgment, the device no longer uses the original first audio signal, but instead calculates the root-mean-square energy and normalized frequency amplitude spectrum of the smoothed sequence based on the smoothed sequence obtained after processing the first audio signal, and then compares them with the features of the first audio sample and the features of the second audio sample, respectively.

[0091] Optionally, through test feedback, various parameters in the above process can be adjusted to achieve a balance between accuracy, timely feedback, and memory and computational consumption.

[0092] The breathing state discrimination device proposed in this invention has undergone biofeedback application training and development, and a 14-day training intervention has been completed. Studies have shown that breathing training intervention has a significant effect on mood regulation and alleviating depression and anxiety. This approach can be widely used in clinical intervention and medical institution treatment, and can also be extended to other training scenarios, possessing broad application potential and market demand.

[0093] The electronic device provided in the embodiments of the present invention will be described below. The electronic device described below can be referred to in correspondence with the breathing state discrimination device described above, and can achieve the same technical effect as the breathing state discrimination device.

[0094] Figure 2 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 2As shown, the electronic device may include a processor 210, a communications interface 220, a memory 230, and a communication bus 240, wherein the processor 210, the communications interface 220, and the memory 230 communicate with each other via the communication bus 240. In some embodiments, the processor 210 may include a processor 210 as described in any of the above embodiments, for example, the processor 210 may be a processor 210 as described in any of the above embodiments, or the processor 210 may be a processor array composed of processors 210 as described in any of the above embodiments. The processor 210 can execute computer programs in the memory 230, and at least perform the following functions: The audio and visual sample features of the target user under different breathing states are determined; the first audio signal and audio sample features of the target user currently acquired are compared to obtain the audio confidence score; the first visual signal and visual sample features of the target user currently acquired are compared to obtain the visual discrimination result; based on the visual discrimination result, the preliminary recognition result is obtained; the audio confidence score is used to assist in the verification of the preliminary recognition result to obtain the target recognition result.

[0095] In some embodiments, when the processor executes the computer program, it performs the following specific functions: acquiring audio signals generated by the target user during exhalation, inhalation, and breath-holding, respectively, as audio signal samples; calling the first extraction unit to calculate the root mean square energy (RMS) of the audio signal samples; calling the second calculation unit to calculate the normalized frequency amplitude spectrum (Spec[k]) of the audio signal samples; using the RMS of the audio signal samples as a first audio sample feature; and using the normalized frequency amplitude spectrum of the audio signal samples as a second audio sample feature.

[0096] In some embodiments, when the processor executes the computer program, it performs the following specific functions: acquiring audio signals generated by the target user during exhalation, inhalation, and breath-holding, respectively, as audio signal samples; and determining audio sample features based on the audio signal samples.

[0097] In some embodiments, the sample feature extraction module includes a first extraction unit and a second extraction unit; the audio sample features include a first audio sample feature and a second audio sample feature; when the processor executes the computer program, it implements the following specific functions: calling the first extraction unit to calculate the root mean square energy (RMS) of the audio signal sample; calling the second extraction unit to calculate the normalized frequency amplitude spectrum (Spec[k]) of the audio signal sample; using the RMS of the audio signal sample as the first audio sample feature; and using the normalized frequency amplitude spectrum of the audio signal sample as the second audio sample feature.

[0098] In some embodiments, when the processor executes a computer program, it performs the following specific functions: calling a first extraction unit to calculate the first root mean square energy of the first audio signal; calling a second extraction unit to calculate the first normalized frequency amplitude spectrum of the first audio signal; comparing the first audio sample features and the first root mean square energy to obtain a first comparison result; comparing the second audio sample features and the first normalized frequency amplitude spectrum to obtain a second comparison result; and obtaining the audio confidence level based on the first comparison result and the second comparison result.

[0099] In some embodiments, when the processor executes the computer program, it performs the following specific functions: based on a two-sided negative exponential function or a one-sided negative exponential function centered on the features of the first audio sample, and using the first root mean square energy as the independent variable, it calculates the first function value of the two-sided negative exponential function or the one-sided negative exponential function; and based on the difference between the peak value corresponding to the features of the first audio sample and the first function value, it obtains a first comparison result.

[0100] In some embodiments, when the processor executes a computer program, it performs the following specific function: calculating the similarity between the features of the second audio sample and the first normalized frequency amplitude spectrum as a second comparison result.

[0101] In some embodiments, when the processor executes a computer program, it performs the following specific functions: acquiring images generated by the target user during exhalation, inhalation, or breath-holding; and determining visual sample features based on the images.

[0102] In some embodiments, when the processor executes the computer program, it performs the following specific functions: calling a face recognition service interface to perform face recognition on an image and obtain at least two boundary points of the target user's mouth; based on the at least two boundary points, obtaining a first average mouth opening degree of the target user in exhalation, a second average mouth opening degree in inhalation, and a third average mouth opening degree in breath-holding; performing a weighted summation of the first average mouth opening degree, the second average mouth opening degree, and the third average mouth opening degree to obtain an exhalation mouth opening degree threshold, and using the exhalation mouth opening degree threshold as a visual sample feature; determining a first mouth opening degree based on the first visual signal of the target user currently acquired; comparing the mouth opening degree with the exhalation mouth opening degree threshold to obtain a visual discrimination result indicating whether the target user is in an exhalation state.

[0103] In some embodiments, when the processor executes the computer program, it performs the following specific functions: when the visual discrimination result indicates that the target user is currently in an exhalation state and the audio confidence score indicates that the target user is currently in an exhalation state, it outputs a target recognition result indicating that the breathing state is exhalation; when the visual discrimination result indicates that the target user is not currently in an exhalation state and the audio confidence score indicates that the target user is currently in an inhalation state, it outputs a target recognition result indicating that the breathing state is inhalation; when the visual discrimination result indicates that the target user is not currently in an exhalation state and the audio confidence score indicates that the target user is currently in a breath-holding state, it outputs a target recognition result indicating that the breathing state is breath-holding.

[0104] In some embodiments, when the processor executes the computer program, it performs the following specific functions: acquiring audio sequences and / or image sequences generated by the target user over a preset duration; processing the audio sequences and / or image sequences using a moving average algorithm with a preset time window to obtain a smooth sequence; and calculating the duration of multiple historical respiratory cycles based on the smooth sequence, and obtaining the average respiratory cycle duration by weighted averaging.

[0105] Furthermore, the logical instructions in the aforementioned memory 230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to perform the following operations: The audio and visual sample features of the target user under different breathing states are determined; the first audio signal of the target user currently acquired and the audio sample features are compared to obtain an audio confidence score; the first visual signal of the target user currently acquired and the visual sample features are compared to obtain a visual discrimination result; a preliminary recognition result is obtained based on the visual discrimination result; the audio confidence score is used to assist in the verification of the preliminary recognition result to obtain the target recognition result.

[0107] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following operations: The audio and visual sample features of the target user under different breathing states are determined; the first audio signal of the target user currently acquired and the audio sample features are compared to obtain an audio confidence score; the first visual signal of the target user currently acquired and the visual sample features are compared to obtain a visual discrimination result; a preliminary recognition result is obtained based on the visual discrimination result; the audio confidence score is used to assist in the verification of the preliminary recognition result to obtain the target recognition result.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source data respiratory state discrimination device, characterized in that, The device includes: The sample feature extraction module is used to determine the audio and visual sample features of the target user under different breathing states; The first comparison module is used to compare the currently acquired first audio signal of the target user with the audio sample features to obtain the audio confidence level; The second comparison module is used to compare the first visual signal of the target user currently collected with the visual sample features to obtain a visual discrimination result; The determination module is used to obtain a preliminary recognition result based on the visual discrimination result; and to use the audio confidence score to assist in verifying the preliminary recognition result to obtain the target recognition result.

2. The apparatus according to claim 1, characterized in that, The sample feature extraction module, when determining the audio sample features of the target user under different breathing states, is specifically used for: The audio signals generated by the target user during exhalation, inhalation, and breath-holding were collected as audio signal samples. Based on the audio signal samples, determine the characteristics of the audio samples.

3. The apparatus according to claim 2, characterized in that, The sample feature extraction module includes a first extraction unit and a second extraction unit; the audio sample features include first audio sample features and second audio sample features; when determining audio sample features based on the audio signal samples, the sample feature extraction module is specifically used for: The first extraction unit is invoked to calculate the root mean square energy (RMS) of the audio signal sample. The second extraction unit is invoked to calculate the normalized frequency amplitude spectrum of the audio signal sample. The root mean square energy of the audio signal sample is used as the first audio sample feature; The normalized frequency amplitude spectrum of the audio signal sample is used as the second audio sample feature.

4. The apparatus according to claim 3, characterized in that, The first comparison module, when comparing the currently acquired first audio signal of the target user with the audio sample features to obtain the audio confidence level, is specifically used for: The first extraction unit is invoked to calculate the first root mean square energy of the first audio signal; the second extraction unit is invoked to calculate the first normalized frequency amplitude spectrum of the first audio signal. The first audio sample features and the first root mean square energy are compared to obtain the first comparison result; By comparing the features of the second audio sample with the first normalized frequency amplitude spectrum, a second comparison result is obtained; Based on the first comparison result and the second comparison result, the audio confidence level is obtained.

5. The apparatus according to claim 4, characterized in that, The first comparison module, when comparing the features of the first audio sample and the first root mean square energy to obtain the first comparison result, is specifically used for: Based on a two-sided negative exponential function or a one-sided negative exponential function centered on the features of the first audio sample, and with the first root mean square energy as the independent variable, the first function value of the two-sided negative exponential function or the one-sided negative exponential function is calculated. The first comparison result is obtained based on the difference between the peak value corresponding to the first audio sample feature and the first function value.

6. The apparatus according to claim 4, characterized in that, The first comparison module, when comparing the features of the second audio sample and the first normalized frequency amplitude spectrum to obtain the second comparison result, is specifically used for: The similarity between the features of the second audio sample and the first normalized frequency amplitude spectrum is calculated as the second comparison result.

7. The apparatus according to claim 1, characterized in that, The sample feature extraction module, when determining the visual sample features of the target user under different breathing states, is specifically used for: Images of the target user during exhalation, inhalation, or breath-holding are captured. Based on the image, determine the visual sample features.

8. The apparatus according to claim 7, characterized in that, The sample feature extraction module, when extracting visual signal features from the image, is specifically used for: The face recognition service interface is called to perform face recognition on the image to obtain at least two boundary points of the target user's mouth; based on the at least two boundary points, the first average mouth opening degree of the target user in the exhalation state, the second average mouth opening degree in the inhalation state, and the third average mouth opening degree in the breath-holding state are obtained. The first average mouth opening degree, the second average mouth opening degree, and the third average mouth opening degree are weighted and summed to obtain the exhalation mouth opening degree threshold, and the exhalation mouth opening degree threshold is used as a visual sample feature. The second comparison module, when comparing the currently acquired first visual signal of the target user with the visual sample features to obtain a visual discrimination result, is specifically used for: Based on the first visual signal of the target user currently collected, determine the first mouth opening degree; By comparing the mouth opening degree with the exhalation mouth opening degree threshold, a visual judgment result is obtained as to whether the target user is in an exhalation state.

9. The apparatus according to any one of claims 1-8, characterized in that, The determining module is specifically used for: If the visual discrimination result indicates that the target user is currently in an exhalation state, and the audio confidence indicates that the target user is currently in an exhalation state, then the target recognition result with the breathing state being exhalation state is output. If the visual discrimination result indicates that the target user is not currently in an exhalation state, and the audio confidence indicates that the target user is currently in an inhalation state, then the target recognition result with the breathing state being inhalation is output. If the visual discrimination result indicates that the target user is not in an exhalation state at the current moment, and the audio confidence indicates that the target user is in a breath-holding state at the current moment, the target recognition result with the breathing state being a breath-holding state is output.

10. The apparatus according to any one of claims 1-8, characterized in that, The device further includes a data acquisition module for: Collect audio and / or image sequences generated by the target user within a preset duration; The audio sequence and / or image sequence are processed using a moving average algorithm with a preset time window to obtain a smooth sequence; Based on the smoothed sequence, the duration of multiple historical respiratory cycles is calculated, and the average respiratory cycle duration is obtained by weighted averaging.

11. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following functions: Determine the audio and visual sample features of the target user under different breathing states; The audio confidence level is obtained by comparing the first audio signal of the target user currently collected with the features of the audio sample. By comparing the first visual signal of the target user currently acquired with the visual sample features, a visual discrimination result is obtained; Based on the visual discrimination result, a preliminary recognition result is obtained; the audio confidence score is used to assist in the verification of the preliminary recognition result, and a target recognition result is obtained.

12. The electronic device according to claim 11, characterized in that, When the processor executes the computer program, it performs the following specific functions: The audio signals generated by the target user during exhalation, inhalation, and breath-holding were collected as audio signal samples. The first extraction unit is invoked to calculate the root mean square energy of the audio signal sample. The second calculation unit is invoked, and the normalized frequency amplitude spectrum of the audio signal sample is calculated by the second extraction unit. The root mean square energy of the audio signal sample is used as the first audio sample feature; The normalized frequency amplitude spectrum of the audio signal sample is used as the second audio sample feature.

Images