Respiration detection, guidance and health assessment device and method

Abstract

The invention relates to a breath detection and health assessment device and method. The breath detection device comprises an audio evaluation unit which is configured to evaluate the audio confidence degree of breath detection based on an audio signal; a video evaluation unit configured to evaluate the video confidence of the respiration detection based on the video signal; a model matching unit configured to match at least one breath detection model suitable for performing breath detection based on a vehicle condition; the respiration detection unit is configured to call the at least one respiration detection model to process the audio signal and the video signal based on the audio confidence coefficient and the video confidence coefficient so as to generate a first respiration detection sequence corresponding to the audio signal and a second respiration detection sequence corresponding to the video signal; and a detection fusion unit configured to fuse the first respiration detection sequence and the second respiration detection sequence to generate a respiration classification detection sequence.

Description

Technical Field

[0001] The present invention relates to breathing detection, and in particular to breathing detection of passengers in a vehicle and breathing guidance and health assessment for health improvement. Background Art

[0002] In order to relieve the fatigue of the driver or passengers and increase the fun of the journey, the driver or passengers can make full use of the driving time to exercise their body and mind, such as breathing training. At present, vehicle multimedia applications usually set up vehicle breathing applications to provide customers with yoga deep breathing guidance courses, etc. Therefore, it is very beneficial to be able to properly detect the breathing behavior of passengers and evaluate the completion of breathing guidance courses and the health of users. Summary of the invention

[0003] The present invention proposes a solution that can adaptively apply different breathing detection methods to different vehicles, and the present invention does not require additional equipment and can directly use the vehicle microphone and / or camera to achieve such detection. Based on the above breathing detection results, the present invention also provides a solution for health assessment and breathing guidance course recommendation.

[0004] According to one aspect of the present invention, a breathing detection device is provided, comprising an audio evaluation unit, configured to evaluate the audio confidence of breathing detection based on the audio signal using the audio signal collected from the vehicle compartment; a video evaluation unit, configured to evaluate the video confidence of the breathing detection based on the video signal using the video signal stream collected from the vehicle compartment; a model matching unit, configured to match at least one breathing detection model suitable for performing breathing detection based on vehicle conditions; a breathing detection unit, configured to call the at least one breathing detection model to process the audio signal and the video signal based on the audio confidence and the video confidence to generate a first breathing detection sequence corresponding to the audio signal and a second breathing detection sequence corresponding to the video signal; a detection fusion unit, configured to fuse the first breathing detection sequence with the second breathing detection sequence to generate a third breathing detection sequence.

[0005] According to another aspect of the present invention, there is provided a breath evaluation device, comprising a breath receiving module for receiving a sequence of breath segments generated in response to a breathing guidance course, the sequence comprising exhalation, inhalation and breath holding segments; a breath index scoring module, configured to determine a plurality of breath characteristics within a period of time based on the breath segments; and a comprehensive evaluation module, configured to generate a multidimensional index representing breathing quality based on the plurality of breath characteristics.

[0006] According to another aspect of the present invention, there is provided a health assessment device, comprising a deep breathing guidance module, which provides a deep breathing course guidance to a user through at least one output medium, wherein the medium comprises one or more of text, video and audio; a breathing detection module, which is configured to assess the user's breathing ability based on the degree of match between the user's breathing during the deep breathing course guidance and the preset requirements of the course guidance; and a health assessment module, which is configured to determine the user's health status based on the breathing ability.

[0007] According to another aspect of the present invention, there is provided a course recommendation device, comprising a course evaluation module for evaluating a user's training status under a first training guidance course; a personalized recommendation module configured to provide a recommended ranked list of a plurality of breathing guidance courses based on a plurality of matching modes and the training status, wherein the ranking represents a matching degree between a user's characteristics and alternative breathing guidance courses in a breathing guidance course library, the user characteristics including the training status; and a comprehensive recommendation module configured to fuse the plurality of ranked lists to provide a second training guidance course.

[0008] According to another aspect of the present invention, a health assessment device is provided, comprising a real-time detection module for detecting the breathing characteristics and physiological signs of a user within at least one time period; a breathing adjustment assessment module for determining the breathing quality based on the breathing characteristics, and determining the user's breathing adjustment completion by comparing the breathing characteristics with preset guidance requirements; and a health assessment module for assessing the user's breathing adjustment potential within the at least one time period based on the breathing adjustment completion, breathing quality and physiological signs.

[0009] According to one aspect of the present invention, a breathing detection method is provided, comprising evaluating the audio confidence of breathing detection based on the audio signal using the audio signal collected from the vehicle compartment; evaluating the video confidence of the breathing detection based on the video signal using the video signal stream collected from the vehicle compartment; matching at least one breathing detection model suitable for performing breathing detection based on vehicle conditions; processing the audio signal and the video signal using the at least one breathing detection model to generate a first breathing detection sequence corresponding to the audio signal and a second breathing detection sequence corresponding to the video signal; and fusing the first breathing detection sequence with the second breathing detection sequence to generate a third breathing detection sequence.

[0010] In a preferred embodiment, the audio signal is divided into a plurality of audio signal frames according to a sliding time window, and the video signal stream is divided into a plurality of video signal frames with time stamps.

[0011] In a preferred embodiment, the method further comprises performing audio preprocessing on the audio signal frame and extracting signal features of the preprocessed audio signal frame, and invoking an audio confidence assessment model to process the signal features to generate an audio confidence of each audio signal frame. The method further comprises performing video preprocessing on the video signal frame to extract optical features of the preprocessed video signal frame, and invoking a video confidence assessment model to process the optical features to generate a video confidence of each video signal frame.

[0012] In a preferred embodiment, the breathing detection model library is used to store a plurality of different trained audio-breathing detection models and video-breathing detection models, wherein each of the plurality of different audio-breathing detection models and video-breathing detection models is trained to adapt to different constraints, wherein the constraints include at least one of resource requirements or prediction accuracy; wherein the vehicle conditions include vehicle available resources; the method further includes determining, based on the degree of matching between the vehicle available resources and the constraints of each model, at least a portion of the models among the plurality of different audio-breathing detection models and video-breathing detection models that meet the resource matching conditions as dynamic switching candidate models, wherein the dynamic switching candidate models include the at least one breathing detection model.

[0013] In a preferred example, the method also includes, for each current frame among the multiple audio signal frames and the multiple video signal frames, when the confidence of the current frame is greater than or equal to the confidence threshold condition, selecting an audio-respiration detection model or a video-respiration detection model corresponding to the current frame type from the dynamically switching candidate models; processing the current frame using the selected audio-respiration detection model or video-respiration detection model to generate a classification prediction of exhalation, inhalation or pause; wherein the first breathing detection sequence includes classification predictions corresponding to the multiple audio signal frames and the second breathing detection sequence includes classification predictions corresponding to the multiple video signal frames.

[0014] In a preferred example, the fusion operation further includes time-aligning the first breathing detection sequence with the second breathing detection sequence based on the sliding time window and the timestamp; fusing the first breathing detection sequence and the second breathing detection sequence that are time-aligned to generate a fused detection sequence; and performing smoothing on the fused detection sequence based on a predetermined decision mode to generate the third breathing detection sequence.

[0015] In a preferred example, the fusion operation includes fusing the first breathing detection sequence with the second breathing detection sequence based on at least one of the audio confidence and the video confidence and a predicted probability of the at least one breathing detection model.

[0016] In a preferred example, the smoothing process includes: reclassifying a portion of the classification predictions in the fusion detection sequence based on the decision mode.

[0017] According to another aspect of the present invention, a breath evaluation device is provided, comprising generating a breath classification detection sequence for forming a breath segment sequence, wherein the breath classification detection sequence is generated based on collected audio signals and video signals according to the breath detection method of the present invention; receiving a breath segment sequence generated in response to a breathing guidance course, the sequence comprising exhalation, inhalation and breath holding segments; determining a plurality of breath features within the time period based on the respective breath segments; and generating a multidimensional indicator representing the breathing quality based on the plurality of breath features.

[0018] In a preferred example, the multiple breath characteristics include breath compliance, which indicates the ratio of the average duration of breathing segments to the expected average duration; breath duration, which indicates the ratio of the number of times the duration of exhalation and inhalation segments exceeds a threshold to a target number; breath stability, which indicates the variance of the duration of exhalation and inhalation segments; and breath positivity, which indicates the ratio of the number of exhalation segments that show a continuous increase or no decrease in duration to the total number of exhalation segments in the breath detection sequence, and the ratio of the number of inhalation segments that show a continuous increase or no decrease in duration to the total number of inhalation segments in the breath detection sequence.

[0019] In a preferred example, the method further includes determining an attenuation factor based on the duration of breathing pauses in the breath segment sequence within the time period; and using the attenuation factor to attenuate the multiple breath features.

[0020] In a preferred example, the method further includes constructing a pattern diagram based on the breath continuity, breath stability, breath compliance, and breath positivity to indicate breath health; calculating emotional stability based on the ratio of the pattern diagram to a reference pattern diagram; determining breathing stability indicating a changing trend of breath adjustment using a function of the breath continuity, breath stability, and breath positivity; and determining breathing potential indicating a predicted breathing trend using a change in the breathing stability, wherein the multidimensional features include the breath health, emotional stability, breath stability, and breathing potential.

[0021] According to another aspect of the present invention, a vehicle control method is provided, which can select a vehicle control linkage mode based on the characteristics from the breath evaluation to realize the linkage control of multiple functional units of the vehicle, wherein the functional unit dials at least one of the atmosphere lights, lighting effects, air conditioning, and sound effects.

[0022] According to another aspect of the present invention, a health assessment method is provided, comprising providing a deep breathing course guidance to a user through at least one output medium, the medium comprising one or more of text, video and audio; generating a breathing classification detection sequence according to the breathing detection method of the present invention based on audio signals and video signals during the deep breathing course guidance, the breathing classification detection sequence representing the breathing breath of the user; evaluating the user's breathing ability based on the degree of matching between the user's breathing breath during the deep breathing course guidance and preset requirements of the course guidance; and determining the user's health status based on the breathing regulation ability.

[0023] In a preferred example, the method also includes detecting the user's physiological signs in real time during the provision of the deep breathing course guidance; determining the user's health status based on the breathing ability and the physiological signs; calculating the health improvement degree based on comparing the changes in health status before and after the deep breathing course guidance; and instructing the deep breathing guidance module to update the deep breathing course guidance based on the health improvement degree.

[0024] In a preferred example, the health level includes: emotional stability determined based on breathing frequency and heart rate; breath duration, breathing regulation and breathing regulation initiative determined based on the exhalation and inhalation duration characteristics of the user's breath; cardiopulmonary level determined based on breathing frequency, heart rate and blood oxygen saturation; wherein the health detection module determines the health level based on emotional stability, breathing regulation, breath duration and breathing regulation initiative and cardiopulmonary level using a predetermined health pattern diagram.

[0025] According to another aspect of the present invention, a course recommendation method is provided, comprising generating a breathing classification detection sequence according to the breathing detection method of the present invention based on audio signals and video signals during a first training guidance course; evaluating the training status of the user under the first training guidance course; providing a recommended ranked list of multiple breathing guidance courses based on multiple matching modes and the training status, wherein the ranking represents the matching degree between user characteristics and alternative breathing guidance courses in a breathing guidance course library, the user characteristics including the training status; and fusing the multiple ranked lists to provide a second training guidance course.

[0026] In a preferred example, the user characteristics also include the user's preferences and interests in the course; the training status includes at least one of health status, course completion, breathing potential and course acceptance, wherein the course acceptance is related to the changing trend of the user's physiological signs, wherein the health status is determined based on the user's physiological signs; wherein the matching pattern includes a user interest pattern and a user ability pattern.

[0027] In a preferred example, the breathing exercise course library includes a guide word library, a background music library, and a micro-animation library, and the selected alternative courses include one or more of the guide words, background music, and micro-animations; wherein the breathing exercise course library further includes a course template, which defines the course characteristics of each alternative breathing exercise course; the method includes processing the user characteristics and the course characteristics according to the multiple matching modes to generate a recommended ranked list of the multiple breathing exercise courses.

[0028] In a preferred example, the method further includes generating a ranked list of the most popular courses based on big data statistics of online users; generating a similarity recommendation list of similar users based on user portrait similarity; and generating a recommendation list of the day based on the environment library; wherein

[0029] The method further includes fusing the most popular course ranking list, the similarity recommendation list, and the today's recommendation list based on weights determined based on user feedback on historically recommended courses.

[0030] According to another aspect of the present invention, a health assessment method is provided, comprising detecting the breathing characteristics and physiological signs of a user within at least one time period, comprising, according to the breathing detection method of the present invention, processing audio data collected by a microphone and a video stream captured by a camera to generate a breathing classification detection sequence of the user, wherein the breathing classification detection sequence indicates breathing characteristics including exhalation, inhalation and breath holding, and processing the video stream captured by the camera to generate the physiological signs, including at least one of heart rate, blood oxygen saturation and blood pressure; determining the breathing quality based on the breathing characteristics, and determining the user's breathing completion by comparing the breathing characteristics with preset guidance requirements; and evaluating the user's breathing potential within the at least one time period based on the breathing completion, breathing quality and physiological signs.

[0031] In a preferred example, the real-time detection further includes: processing the audio data collected by a microphone to generate the breathing characteristics in the form of a frame sequence, wherein each frame indicates exhalation, inhalation, and breath holding; processing the video stream of the user captured by a camera to generate the physiological signs, including at least one of heart rate, blood oxygen saturation, and blood pressure.

[0032] In a preferred example, the method further includes: based on the frame sequence, calculating the duration of the exhalation segment, the inhalation segment, and the pause segment contained in the sequence to determine the breathing quality; comparing the exhalation segment, the inhalation segment, and the pause segment contained in the sequence with the preset breathing guidance requirements to determine the user's breathing completion degree; based on the breathing completion degree, breathing quality and physiological sign signals such as heart rate and blood oxygen saturation, determining the breathing potential and health indicators.

[0033] In a preferred example, the method further includes providing breathing scenario guidance based on pranayama potential, pranayama completion index and / or health index.

[0034] In a preferred example, the method further includes determining the best matching scenario based on the completion degree and potential of breathing regulation; and determining the timing of inserting the best matching scenario into the preset guide. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram showing a breathing detection device according to an example;

[0036] Figure 2A A health quality assessment device according to an example of the present invention is shown;

[0037] Figure 2B An exemplary linkage control system is shown;

[0038] Figure 2C A health quality assessment device according to another example of the present invention is shown;

[0039] Figure 3A A health assessment device according to an example of the present invention is shown;

[0040] Figure 3B A health assessment device according to another example of the present invention is shown;

[0041] Figure 4 A course recommendation device according to an example of the present invention is shown.

[0042] Figure 5 A flow chart of a breathing detection method according to an example of the present invention is shown;

[0043] Figure 6 A health quality assessment method according to an example of the present invention is shown;

[0044] Figure 7 A flowchart of a health assessment method according to another example of the present invention is shown;

[0045] Figure 8 A flow chart showing a method for recommending a personalized breathing course according to an example of the present invention;

[0046] Fig. 9 A health assessment method according to another example of the present invention is shown. DETAILED DESCRIPTION

[0047] Conventionally, for breathing detection, the most convenient and lowest-cost non-contact detection usually used includes receiving sound through a microphone and detecting breathing behavior in the sound, and detecting the chest and shoulder ups and downs of the user (such as the driver, other passengers, etc.) through a camera to detect the breathing behavior. However, in different scenarios, these two methods have their own advantages and disadvantages. For example, when the light in the car is dim or blocked, the visual detection effect will be affected or not feasible; and when the noise in the car is large, the detection based on the breathing sound will also be affected. Moreover, due to the differences in system configurations of different vehicles, such as differences in computing power, the accuracy and time of implementing breathing detection will be different. Obviously, if the time is too delayed, it will be detrimental to user experience and timeliness. According to one embodiment of the present invention, an improved scheme for comprehensively evaluating breathing detection based on audio and vision is provided.

[0048] The vehicle user can start the in-vehicle deep breathing application APP, and then play the BreathGuide breathing guidance course selected by the user or recommended by the system through the multimedia device, such as playing rhythmic music, light music through the speaker, playing text prompt information and micro animation through the display screen, etc.; and the user can perform breathing training according to the sound, text or animation instructions of the BreathGuide guidance course. According to an example of the present invention, the in-vehicle breathing detection device can be used to detect the user's breathing behavior. Figure 1 A schematic diagram of a breathing detection device according to an example is shown.

[0049] like Figure 1As shown, the breathing detection device 100 includes an audio evaluation unit 101, a video evaluation unit 102, a model matching unit 103, a breathing detection unit 104 and a detection fusion unit 105. Here, the audio evaluation unit 101 can use the microphone in the vehicle to collect the audio signal Sound_Sig in the vehicle compartment for a period of time, such as a T time period. Obviously, the audio Sound_Sig collected here includes many sound source signals produced in the vehicle compartment, such as the breathing sound of the user to be detected, the music sound controlled by the guided course emitted by the speaker, and other scene noises. The audio evaluation unit 101 further evaluates the credibility of the breathing detection based on the audio signal Sound_Sig, which is also referred to as audio confidence here. According to the present invention, since the breathing action of a person is an alternation of exhalation (represented by H), inhalation (represented by X), and breath holding (represented by B) over time, in order to accurately evaluate these breathing behaviors, after collecting the audio signal Sound_Sig, the audio evaluation unit 101 uses a sliding time window w to sample the signal Sound_Sig within a time period T, and divides it into multiple audio frames SS_F, where each frame SS_F occupies a time interval, such as T0, thereby obtaining N audio frames, where T0*N=T. In addition, for each frame SS_F, its acquisition time w is also recorded T , that is, the time when the frame signal occurs. Therefore, for the current audio signal Sound_Sig, N audio frames Sound_Sig = {SS_F1, SS_F2, SS_F3, ... SS_F N}.

[0050] The audio evaluation unit 101 determines the credibility of the breathing behavior prediction based on each audio frame based on the audio frame SS_F. Figure 1 As shown, the audio evaluation unit 101 includes an audio preprocessing unit 1011, an audio feature extraction unit 1012 and an audio evaluation unit 1013. The audio preprocessing unit 1011 is used to preprocess the audio signal Sound_Sig, including resampling, noise reduction, data enhancement, etc. Further, the preprocessing performed by the audio preprocessing unit 1011 also includes using the pre-stored music signal played by the guided course BreathGuide to perform cancellation processing on the collected audio signal Sound_Sig, that is, removing the music signal played by the course from the audio signal Sound_Sig, thereby obtaining an audio signal with only the user's breathing sound and background sound, and then sampling the audio signal with only the user's breathing sound and background sound as described above to generate each audio frame {SS_F1, SS_F2, SS_F3, ... SS_F N Any technology known in the prior art can be used to implement the above preprocessing.

[0051] The audio feature extraction unit 1012 is used to extract the mathematical features ST of the audio signal from each audio frame SS_F, such as the signal amplitude and volume in the time domain space, and the spectrum or frequency band distribution in the frequency domain space. At the same time, the audio feature extraction unit 1012 also extracts the environmental scene audio feature RT. The environmental scene audio is various sounds emitted in the current scene in the car, such as environmental noise or other possible background music in the car (such as practice music played by the user through the mobile phone), etc. Therefore, the environmental scene audio feature RT extracted here includes, for example, the volume of background music, music emotional features (such as intensity and pleasure), etc., and the size of environmental noise (amplitude or decibel number). Here, the audio feature extraction unit 1012 can use various music recognition technologies known in the prior art to determine whether there is music in the environmental sound in the current car, and extract the multidimensional features of music when it exists. In one embodiment, when it is determined that there is music, the environmental scene audio feature RT extracted by the audio feature extraction unit 1012 can only include the multidimensional features of the music; when it cannot be determined or there is no music, the extracted environmental scene audio feature RT includes various features such as environmental noise.

[0052] In addition, further, the audio feature extraction unit 1012 can also determine the matching degree between the extracted environmental scene sound and multiple music templates pre-stored in the database, for example, by determining the matching degree between the feature RT of the extracted environmental scene sound and the predetermined features of multiple music templates. When the matching degree is high (that is, there is at least one music template that matches the environmental scene sound), a matching feature PT is formed, which represents that the background sound in the current car can be used for breathing training (for example, it may include training music played by the user through the mobile phone); and when the matching degree is low, the health assessment is abandoned because there may be no suitable music in the car for breathing training. Thus, the audio feature extraction unit 1012 generates a feature vector SS_F for each audio frame SS_F. i ={ST, RT, PT}. It should be noted here that the present invention is not limited to the extracted features in the above example, and the audio feature extraction unit 1012 can also extract other features and exclude one or more of ST, RT, and PT as needed.

[0053] The audio evaluation unit 1013 processes the feature vector SS_F of each audio frame by calling the audio confidence evaluation model Sound_CEM i (={ST, RT, PT}) to generate the audio prediction confidence A_CON for each audio signal to perform corresponding breathing prediction i. According to the present invention, the audio confidence assessment model Sound_CEM is trained to learn the input audio frame to evaluate the accuracy of determining the user's breathing behavior within the time interval corresponding to the audio frame based on such an audio signal, that is, the reliability. The audio confidence assessment model Sound_CEM here can be determined by machine learning using a variety of neural networks known in the prior art, such as a long-term and short-term neural network architecture. Thus, for the currently collected audio signal Sound_Sig, its N confidence frame sequences {A_CON1, A_CON2, A_CON3, ..., A_CON N}, where each value in the sequence represents the confidence of predicting the breathing behavior based on the corresponding audio frame.

[0054] Synchronously with the audio evaluation unit 101, the video evaluation unit 102 can use the camera in the vehicle to collect the video signal stream Video_Steam in the vehicle compartment within the T time period. Obviously, the collected video signal stream Video_Steam includes the video of the chest and shoulder movements related to breathing in the vehicle compartment, as well as other activity videos. The video evaluation unit 102 evaluates the credibility of breathing detection based on the video signal stream Video_Steam, which is also referred to as the video confidence V_CON. Figure 1 As shown, the video evaluation unit 102 includes a video preprocessing unit 1021, a video feature extraction unit 1022 and a video evaluation unit 1023. Similar to the aforementioned audio signal processing concept, the video preprocessing unit 1021 is used to preprocess the video signal Video_Steam, including sampling the signal Video_Steam according to a certain sampling frequency after collecting the signal stream Video_Steam. The sampling frequency here can be set to T0 / T, for example, wherein the key frame is extracted in each sampling interval T0, and each video frame can be scaled as needed to generate N video frames VS_F. Other sampling frequencies can also be used here to obtain multiple video frames within a period T, and any preprocessing technology known in the prior art can be used to implement the preprocessing of the video signal VS, including the identification and extraction of key frames. In addition, for each frame VS_F, its sampling time t is also recorded as the timestamp of the current video frame. It is assumed here that N video frames VS={VS_F1, VS_F2, VS_F3, ...VS_F N}. Note that for different sampling frequencies, the value of the number of frames N obtained is different, and therefore is also different from the number of audio frames.

[0055] Therefore, the video evaluation unit 102 judges the breathing behavior based on each video frame based on the video frame VS_F. Specifically, the video feature extraction unit 1022 extracts image features, such as color features, texture features, shape features, spatial relationship features, etc., from the preprocessed video frame VS_F, and processes these image features to generate features for video confidence prediction. These video features reflect the characteristics of light brightness IT, user background and clothing color difference CT, user breathing action amplitude MT, such as chest rise and fall amplitude. It is assumed here that the video feature extraction unit 1022 generates a feature vector VS_F for video confidence prediction for each video frame i ={IT,CT,MT}.

[0056] The video evaluation unit 1023 processes the feature vector VS_F of each video frame by calling the video confidence evaluation model Video_CEM i (={IT, CT, MT}) to generate the video confidence V_CON of each video signal frame i . According to the present invention, the video confidence assessment model Video_CEM is trained to learn the input video frames to evaluate the accuracy, i.e., the confidence, of determining the user's breathing behavior within the sampling time interval corresponding to the video frame based on such video signals. The video confidence assessment model Video_CEM here can be determined by machine learning using a variety of neural networks known in the prior art, such as a long-term and short-term neural network architecture. Thus, for the currently acquired video stream Video_Steam, its N confidence frame sequences {V_CON1, V_CON2, V_CON3, ..., V_CON N}, where each value in the sequence represents the confidence of predicting the breathing behavior based on the corresponding video frame.

[0057] According to an embodiment of the present invention, multiple deep learning models for respiratory event detection are trained in advance for audio signals and video signals, for example, including convolutional neural network CNN, recurrent neural network RNN ​​and convolutional recurrent neural network CRNN, etc., or further can include a fusion of multiple neural network models. This model fusion can be implemented by currently known model averaging, voting or stacking to improve the model detection accuracy, stability and robustness, and is used for respiratory event classification detection at each moment. As an example, the deep learning model trained by the present invention includes p audio-based models S_BreathM1~S_BreathM P And q video-based models V_BreathM1~V_BreathM q, these neural network models can be created through knowledge distillation strategies, such as offline distillation, semi-supervised distillation, and self-supervised distillation. As a result, the p + q models obtained are different in terms of accuracy, latency, and complexity (i.e., network parameters), and thus can well adapt to the machine resource requirements for executing model learning capabilities, including the computing power requirements for in-vehicle computing systems. For example, for a vehicle with a high-performance computing power ECU, the models it adapts to can be S_BreathM1 or V_BreathM1, while for an ECU with lower computing power, S_BreathM p or V_BreathM q etc. For an ECU with medium computing power, S_BreathM i or V_BreathM j can be used, where 1 < i < p and 1 < j < q. Of course, in specific model adaptation, not only machine resource limitations are considered, but also requirements such as accuracy can be referred to. Therefore, in the vehicle's model library DB, not only multiple models are stored, but also the computing power limit conditions PowerReq and accuracy ACC requirements adapted to these models are correspondingly stored.

[0058] According to an example of the present invention, the model matching unit 103 identifies the resource conditions of the current vehicle. For example, in this example, by identifying the model of the ECU used to perform predictions in the vehicle, etc., the computing power Power_ID of the ECU can be confirmed, and the breathing detection model suitable for performing breathing detection can be found from the model library DB through the confirmed computing power Power_ID, including not only the audio model S_BreathM, but also the video prediction model V_BreathM. Usually, the model found is a model with higher accuracy under the condition that Power_ID meets the computing power limit condition PowerReq. However, if there is no high requirement for accuracy, in order to save computing power resources, a model with lower accuracy can also be selected. Here, for the convenience of description, it is assumed that the audio model S_BreathM4 and the video model V_BreathM8 are determined to be adapted to the current vehicle from the model library.

[0059] The breathing detection unit 104 then based on N audio frame confidence sequences {A_CON1, A_CON2, A_CON3,..., A_CON N} and N video frame confidence sequences {V_CON1, V_CON2, V_CON3,..., V_CON N}, uses the adapted breathing detection models such as S_BreathM4 and V_BreathM8 in this example to process the audio frame sequence {SS_F1, SS_F2, SS_F3,... SS_F N} and the video frame sequence {VS_F1, VS_F2, VS_F3,... VS_FN} to generate the corresponding breathing classification prediction sequence. Specifically, since {SS_F1, SS_F2, SS_F3, ... SS_F N} are frames under N time windows usually obtained by sliding time window w, and {VS_F1, VS_F2, VS_F3, …VS_F N} is a frame obtained from N time sampling points, so as an example, the breathing detection unit 104 processes the above frames based on the time sequence. For example, first, for the audio frame confidence sequence {A_CON1, A_CON2, A_CON3, ..., A_CON N}, the breathing detection unit 104 first determines whether the value of A_CON1 is greater than a predetermined confidence threshold CON T ; If it is greater than or equal to the threshold CON T , then S_BreathM4 is called to process the audio frame SS_F1 corresponding to A_CON1. ​​According to an example of the present invention, the S_BreathM model is trained to generate prediction scores (for example, P ) for the three actions of exhalation, inhalation and breath holding when processing the audio frame SS_F. A , P B , P C Indicates that P A Predicted score of instructed inspiration, P B Predicted score for instructed exhalation, P C The predicted score of breath holding indicated by S_BreathM4 is P(0) and the one with the highest score is used as the breathing classification of the current audio frame. Assume that in this example, when S_BreathM4 processes the audio frame SS_F1 corresponding to A_CON1, the predicted score of inhalation corresponding to S_BreathM4 is P(0) A1 The highest, so the current frame SS_F1 is classified as inhalation, for example, represented by '1', and the score P corresponding to the frame is recorded at the same time. A1 ; If the predicted score corresponding to exhalation P B1 The highest, so the current frame SS_F1 is classified as exhalation, for example, represented by '0', and the score P corresponding to the frame is recorded at the same time. B1 ; If the predicted score corresponding to breath holding is P C1 The highest, so the current frame SS_F1 is classified as breath holding, for example, represented by 'B', and the score P corresponding to the frame is recorded at the same time. C1 Other symbols or marks may also be used to represent exhalation, inhalation and breath holding. T , since the credibility of this frame is low, the breathing learning based on this frame is abandoned, and the breathing learning of the audio frame of the next time window wt2 is performed. The breathing detection unit 104 can form a breathing classification prediction matrix SS_P based on the audio frame:

[0060]

[0061] Where SS_CLass represents the respiratory classification prediction sequence, for example, SS_CLass = {1, 1, 1, 0, 0, B, 1 ..., 1}; and SS_Score represents the corresponding prediction probability for each classification in the classification prediction sequence, for example, corresponding to the above classification prediction sequence SS_CLass, its corresponding probability SS_Score = {P A1 ,P A2 ,P A3 ,P B4 ,P B5 ,P C6 ,P A7 ,……,P AN}.

[0062] It should be noted here that in this example, the features or vectors used by S_BreathM when processing the audio frame SS_F may be the same as or different from the vectors used by the audio evaluation unit 1013 when performing confidence prediction. For example, in addition to ST, RT, and PT, other vectors may also be included.

[0063] Similarly, for the video confidence frame sequence {V_CON1, V_CON2, V_CON3, ..., V_CON N}, the breathing detection unit 104 also processes the above frames based on the time sequence. For example, for the first frame in the video confidence frame sequence at the first time t1 in the time sequence, the breathing detection unit 104 first determines whether the value of V_CON1 is greater than the confidence threshold CON T ; If it is greater than or equal to the threshold CON T , then V_BreathM8 is called to process the feature vector of the video frame VS_F1 corresponding to V_CON1. ​​Similar to the audio frame prediction model S_BreathM, according to an example of the present invention, the V_BreathM model is trained to generate prediction scores for the three actions of exhalation, inhalation and breath holding when processing the video frame VS_F (for example, P′ A , P′ B , P′ C Indicates that P′ A Predicted score of instructed inspiration, P′ B Predicted score of instructed exhalation, P′ C The predicted score of breath holding indicated by V_BreathM8 is P′, and the one with the highest score is used as the breathing classification of the current video frame. Assume that in this example, when V_BreathM8 processes the audio frame VS_F1 corresponding to V_CON1, the predicted score of breath holding indicated by V_BreathM8 is P′ A1The highest score P' is recorded, and the current frame VS_F1 is classified as inhalation. For example, '1' indicates inhalation classification. A1 ; If the predicted score corresponding to exhalation is P′ B1 The highest, so the current frame VS_F1 is classified as exhalation, for example, '0' represents the exhalation classification, and the score P' corresponding to the frame is recorded at the same time B1 ; If the predicted score corresponding to breath holding is P′ C1 The highest, so the current frame VS_F1 is classified as breath holding, for example, 'B' represents the breath holding classification, and the score P' corresponding to the frame is recorded at the same time C1 .

[0064] However, if V_CON1 is less than the threshold CON T , since the credibility of this frame is low, the breathing learning based on this frame is abandoned, and the breathing learning of the video frame at the next time t2 is performed. Similarly, the feature vector when V_BreathM processes the video frame can be the same as or different from the vector used when predicting the video confidence. Based on the above processing, the breathing detection unit 104 can thus form a breathing prediction matrix VS_P based on N video frames:

[0065]

[0066] Wherein VS_CLass represents the respiratory classification prediction sequence, for example, VS_CLass = {1, 1, 1, 1, 0, B, 0 ..., 1}; and VS_Score represents the corresponding prediction probability for each classification in the classification prediction sequence, for example, corresponding to the above classification prediction sequence VS_CLass, its corresponding probability VS_Score = {P A1 ,P A2 ,P A3 ,P A4 ,P B5 ,P C6 ,P B7 ,……,P AN}.

[0067] In this example, for simplicity, we assume that the sampling timestamp of the video frame is t and the sliding window of the audio frame is w t are aligned, that is, each timestamp t is located in the sliding time window w of the corresponding audio frame t At the same time, it can be seen that the classification predictions based on audio and video under the aligned timestamps / time windows are not necessarily the same, such as the two classification prediction sequences shown in the above example:

[0068] SS_CLass={1, 1, 1, 0, 0, B, 1..., 1}

[0069] VS_CLass={1, 1, 1, 1, 0, B, 0..., 1}

[0070] It can be seen that for the fourth audio frame and video frame at time t4 and wt4, the prediction results are '0' and '1' respectively, which represent the two opposite actions of exhalation and inhalation respectively, and there is obviously a prediction error. Therefore, in order to obtain a unified final classification prediction, the present invention fuses the classification prediction based on audio and video.

[0071] The detection fusion unit 105 is used to fuse the generated breathing classification prediction sequences SS_CLass and VS_CLass including the audio prediction and the video prediction, so as to determine the final breathing classification prediction within the time period T. To this end, Figure 1 As shown, the detection fusion unit 105 includes a time alignment module 1051 and a fusion module 1052. The time alignment module 1051 is based on the sliding time window w when creating the audio frame. t The classification prediction sequence SS_CLass corresponding to the audio signal frame is aligned with the prediction sequence VS_CLass corresponding to the video signal frame in time with the timestamp t when the video frame is generated. t1 Aligned with timestamp t1, time window w t2 Aligned with timestamp t2, etc. However, it should be noted that due to the time window sliding process and the different video sampling rates, there may be other corresponding methods, for example, determining that timestamp t1 is located in time window w t1 With w t2 Therefore, after time alignment, in different embodiments of the present invention, the predicted probability score or the confidence score or a combination of the two can be used to perform call classification fusion based on audio and video.

[0072] For example, according to an example of the present invention, the detection fusion unit 105 is based on the confidence sequence {A_CON1, A_CON2, A_CON3, ..., A_CON N} with {V_CON1, V_CON2, V_CON3,…, V_CON N} to fuse the breathing prediction sequence VS_CLass and SS_CLass, thereby forming the final breathing classification prediction. For example, in the above example, the opposite prediction results '0' and '1' occur at timestamp t4 and time window wt4. The fusion unit 1052 can select the final classification prediction based on the confidence level of the audio frame SS_F4 and the video frame VS_F4. For example, when the confidence level A_CON4 of the audio frame SS_F4 is greater than the confidence level V_CON4 of the video frame VS_F4, "0" or exhalation is selected as the final classification result; otherwise, "1" or exhalation is selected as the final classification result.

[0073] In another example of the present invention, the detection fusion unit 105 fuses the breathing prediction sequences VS_CLass and SS_CLass based on the prediction probabilities of the classification models S_BreathM and V_BreathM when classifying and predicting the audio frame and the video frame respectively as the selection criteria for the final classification, thereby forming the final breathing classification prediction. As the classification sequence exemplified above, the corresponding probability prediction sequence is:

[0074] SS_Score = {P A1 ,P A2 ,P A3 ,P B4 ,P B5 ,P C6 ,P A7 ,……,P AN}

[0075] VS_Score = {P′ A1 ,P′ A2 ,P′ A3 ,P′ A4 ,P′ B5 ,P′ C6 ,P′ B7 ,……,P′ AN}.

[0076] Similarly, in the above example, the opposite prediction results '0' and '1' occur at the time stamp t4 and the time window wt4. The fusion unit 1052 compares P B4 and P′ A4 , if P B4 >P′ A4 Then select "0", i.e. exhalation, as the final classification result; otherwise select "1", i.e. exhalation, as the final classification result.

[0077] In another embodiment of the present invention, the probability score and the confidence level can be comprehensively considered to make the final classification prediction. For example, the sum of the probability score and the confidence level can be selected as the classification standard.P B4 and The sum of A_CON4 is greater than P′ A4 and If the sum of V_CON4 is equal to the sum of V_CON4, "0" (i.e., exhalation) is selected as the classification result; otherwise, "1" is selected as the classification result. Of course, those skilled in the art can also use a method to perform fusion classification prediction based on the above processing information.

[0078] The following table exemplarily shows a more general final classification prediction after time alignment and fusion, including the fusion of the collected audio frames and video frames with alignment and misalignment:

[0079]

[0080] In the table: the first row SS_F represents the time frame of audio data, where 12 frames are shown as an example; the third row SS_F represents the time frame of video data, where 8 frames are shown as an example; R represents the final classification prediction result at each time after fusion. As shown in the table, first of all, because the frame rates used by audio and video are inconsistent, they need to be aligned in time. For example, as shown in the figure, the first frame of the video occurs between the first and second frames of the audio, the second frame of the video and the third frame of the audio occur at the same time, and the eighth video frame is aligned with the twelfth audio frame in time. It can also be seen from the table that the same frame or adjacent frames have mutually exclusive results. For example, the classification class of the third frame of audio is '0', while the classification based on the second frame of video is 'B'. According to the above confidence confirmation rules, the final classification at this time is '0'. In this way, the detection fusion unit 105 outputs a final breathing classification prediction sequence Breath_Predict_Seq within the time period T represented by '0', '1' and 'B', such as R={0,0,0,0,0,0,0,B,B,B,B,B,B,B,B,1,1,1,1,1,1,1} as shown in Table 1.

[0081] Due to various reasons such as calculation and model accuracy, it is not difficult to understand that there may still be classification prediction errors in the breath classification prediction sequence Breath_Predict_Seq. For example, during the time corresponding to the fourth and fifth frames '0', the user may be inhaling, but according to the current prediction result, it is exhaling. Although the possibility of such prediction errors is very low, in order to avoid errors as much as possible, according to an example of the present invention, the detection fusion unit 105 can also perform smoothing processing on the breath prediction sequence Breath_Predict_Seq to avoid errors. In one example, Figure 1As shown, the detection fusion unit 105 may also include a decision module 1053, which is used to perform smoothing processing on the sequence Breath_Predict_Seq according to a pre-established decision tree model DecisionPattern, and to fine-tune or correct the classification prediction error. The decision tree model DecisionPattern is determined by prior learning or based on experience, and is used to determine which of the exhalation, inhalation, or breath holding classifications a group of continuous frames in the breathing prediction sequence Breath_Predict_Seq most likely represents. Here, continuous frames are defined as adjacent frames belonging to the same classification and are classified as continuous frames. The purpose of using the decision tree model DecisionPattern is to identify the true classification represented by the possibly abnormal continuous frames. Take a classification sequence as shown below as an example: {1,1,1,1,0,0,B,B,B,B…,0,0,…B…,1}, which includes two groups of continuous frames from the left, namely {1,1,1,1}, {0,0}, and the decision module 1053 can make the second continuous frame {0,0} as an incorrect prediction according to the decision tree model, and thus correct the second continuous frame, thereby forming a complete first continuous frame {1,1,1,1,1,1,1}, which represents the inhalation action. In this way, with the help of the decision tree model, the decision module 1053 can realize the correction of abnormal continuous frames in the entire breathing prediction sequence Breath_Predict_Seq, so that the final frame prediction sequence Final_Predict_Seq = {1,1,1,1,1,1,1,B,B,B,B…,0,0,…B…,1}.

[0082] According to another example of the present invention, the breathing detection device 100 may further include a breathing segment generation module 106 and a statistical module 107. The breathing segment determination module 106 is used to determine which segment of exhalation, inhalation, and breath holding each continuous frame in the sequence Final_Predict_Seq specifically represents, and the statistical module 107 counts the segment length occupied by the segment, such as the duration. For example, for the first continuous frame {1, 1, 1, 1, 1, 1} in the above Final_Predict_Seq sequence, the breathing segment generation module 106 can determine that it represents a complete inhalation segment X, and the statistical module 107 records the duration of the inhalation segment as w0*6, where w0 represents the sampling time interval. The number of frames in the continuous frame can also be used here to represent the segment length. For example, for the inhalation segment, the segment length is 6 frames. In this way, the detection fusion unit 105 finally determines and stores a complete exhalation, inhalation, and breath-holding segment sequence SegSeq={X, B, ...H...X} within the time period T, where H represents the exhalation action and B represents the breath-holding. In association with the breathing segment sequence, the detection fusion unit 105 further stores the statistical features of each segment, including duration or number of frames.

[0083] According to the example of the present invention, the breathing detection device 100 can display the generated breathing breath sequence in real time on the human-machine interface HMI of the vehicle, for example, a variable graphic symbol such as a flower pattern can be used to vividly represent the breath sequence. In addition, using the HMI, the breathing detection device 100 can also present in real time the confidence of the breathing prediction using the audio signal and the video signal within the time period T, so that the impact of different environmental scene predictions can be observed in real time.

[0084] According to the present invention, breathing behavior can be detected in real time using video data collected by a camera (such as an infrared camera) or audio collected by a microphone. Based on this detection information, it is not only possible to evaluate whether the current guided course is suitable for the user or the completion quality; but also to confirm the user's health quality based on the user's breathing. Moreover, the atmosphere lights and sound effects in the car can be further controlled in linkage, thereby giving the user an immersive experience.

[0085] Figure 2A FIG. 2 shows a health quality assessment device according to an example of the present invention. As shown in the figure, the health quality assessment device 200 includes a breath receiving module 201, a breath index scoring module 202, and a comprehensive assessment module 203. The breath receiving module 201 is used to obtain a frame classification prediction sequence Final_Predict_Seq. For example, in an example of the present invention, the assessment device 200 may include the aforementioned reference Figure 1 The described breathing detection device 100 is used to generate a breathing classification detection sequence Final_Predict_Seq based on the collected audio signal and video signal and provide it to the breath receiving module 201; or directly receive a breathing segment sequence, such as the SegmentSeq generated in response to the aforementioned breathing guidance course, which sequence contains exhalation H, inhalation X, and breath holding B segments, and at the same time receive the duration characteristics corresponding to each segment in the sequence.

[0086] The breath index scoring module 202 calculates multiple breath features within a time period T based on each breathing segment (X, H, B) in the sequence and the corresponding duration features. These breath features can be used to evaluate the user's breath index within the time period T. These breath features include, but are not limited to, breath compliance (BD), breath persistence (BP), breath stability (BS) and breath positivity (BI). The breath compliance BD represents the score of the ratio of the average duration of the breathing segments to the expected average duration. According to an example of the present invention, the following algorithm can be used to calculate:

[0087]

[0088] In the above formula, ∑(X,H) represents the count of the number of frames of all breathing segments in the sequence, while Average represents the average duration of each segment, Et represents the ideal average duration, that is, the target breathing average duration of the course; f represents the audio sampling rate, that is, the number of samples per second from the continuous audio. ； w represents the time window size, that is, the number of samples in one frame of audio. Here we take the classification sequence shown below as an example:

[0089] 【1111111111111111BBBBBBBBBBBBBBBB

[0090] 000000000000000BBBBBBBBBBBBBBBB11111111111111BBBBBBBBBBBBBBBB000000000000000BBBBBBBBBBBBBBBB111111111111111BBBBBBBBBBBBBBBB000000000000000].

[0091] In this sequence, there are 3 segments of "1" that can be counted, and the duration of each segment can be expressed in frames, that is, [6, 4, 5]; there are 5 segments of "B", and the lengths are [16, 16, 14, 14, 16]; there are 3 segments of "0", and the lengths are [15, 15, 15]. Considering that the segments corresponding to actual breathing are only "1" and "0", the average duration of breathing in this sequence is: (16+14+15+15+15+15) / 6=15 (frames). In addition, in the above formula (1), Et represents the ideal average duration. For example, if the average duration of breathing can reach 2 seconds (s) or more, and the ideal sequence of breathing segments should be: exhale 2s, hold breath, inhale 2s, hold breath, alternating, then the score of BD should be close to full marks. For the classification sequence shown above, assuming that the sampling rate of the audio data is 44100 Hz, and the sliding window length set according to the single-channel PCM encoded audio signal is 1024 (the number of single-channel samples), then the frame rate is 44100÷1024≈43, that is, 1 second contains 43 frames, each frame is about 23.25 ms. Therefore, for the sequence in the above example, when Et=2 seconds, the breath reaching scale BD=15÷(2*43)=0.1744.

[0092] Breath duration BP can be defined as the score of the ratio of the number of times the duration of the exhalation and inhalation segments exceeds the threshold to the target number, where the threshold and target number are pre-set by the system.

[0093] Breath stability BS is defined as the score of the variance of the duration of the breathing segment, which can be expressed, for example, by the following formula:

[0094]

[0095] Wherein x represents the variance of the duration of the breathing segments, for example, variance x=Σ(a-μ)2 / n, where a represents the duration of each exhalation segment or inhalation segment, and μ represents the average duration of the breathing segments.

[0096] Usually in the active deep breathing training guided by the course, each exhalation or inhalation is required to be deeper and longer than the previous exhalation or inhalation, or at least not shorter than the previous time. Here, the present invention adopts the breath positivity BI index representing active breathing to measure, that is, the breath positivity BI is defined as the ratio of the number of exhalation segments with continuously increasing or non-decreasing duration in the breath sequence to the total number of exhalation segments, and the ratio of the number of inhalation segments with continuously increasing or non-decreasing duration in the breath sequence to the total number of inhalation segments.

[0097] It should be pointed out here that although the above exemplary method for calculating the breath index score is given, the scoring module 202 can also use other breath assessment algorithms commonly used in medicine to calculate these breath characteristics BD, BP, BS, BI.

[0098] The comprehensive evaluation module 203 generates multi-dimensional indicators representing the breathing quality based on the calculated multiple breath characteristics such as BD, BP, BS, and BI. As an example, these multi-dimensional indicators include but are not limited to breath health BJK, emotional stability EMS, breath regulation stability BRS, and breath regulation potential BRP. Here, the comprehensive evaluation module 203 uses a pattern diagram generated based on medical or expert experience to determine these multi-dimensional indicators. For example, the breath health BJK can be determined based on the breath continuity BP, breath stability BS, breath compliance BD, and breath positivity BI using a predetermined breath health pattern diagram (BH_Pattern). Here, the breath health pattern diagram (BH_Pattern) is a distribution of values ​​of different levels of health in four dimensions of breath continuity, breath stability, breath compliance, and breath positivity pre-calibrated for healthy people, and thus generates different levels of health radar diagrams. As an example, a health radar chart can be generated using the calculated BP, BS, BD, and BI. By displaying the health radar chart on the HMI interface, the impact of each breath indicator on health can be clearly seen by comparing it with a predetermined breath health pattern chart (BH_Pattern).

[0099] In addition, the comprehensive evaluation module 203 determines the score of emotional stability EMS based on BP, BS, BD, and BI, for example, the area of ​​the radar chart formed based on the calculated BP, BS, BD, and BI is compared with the area of ​​the graded health radar chart of different BH_Pattern, and the one with the largest ratio is determined as the current stability EMS score; through the sliding window method, according to the breath continuity BP, breath stability BS, and breath positivity BI indicators in each time window, the change trend of breath adjustment is calculated, and a score of breathing stability BRS is given, where a variety of functions BRS=f(BP, BS, BI) known in the prior art can be used to calculate BRS, such as a weighted average function; and based on the current breathing stability BRS, the breathing trend is predicted and the breathing potential BRP is calculated, that is, these breathing potentials represent the change measure of breathing stability BRS over time, such as whether it is constantly rising or fluctuating, which can be achieved by a variety of methods known in the prior art, such as a moving average method or an exponential smoothing method.

[0100] In addition to the radar chart, in other examples of the present invention, a progress bar can be selected to present the degree of completion of each comprehensive indicator. For example, for the breath control stability BRS, the ratio of the current breath control stability BRS to the target stability can be calculated as an indication of the degree of completion. For the breath control potential BRP, a line chart can be used to represent the change of breath control stability over time.

[0101] From this, it can be seen that the comprehensive evaluation module 203 can calculate comprehensive indicators such as breath health indicators, emotional stability, and breathing stability based on different combination strategies of various breath indicators, so as to comprehensively quantify the breathing ability; and the present invention can further use the quantified breathing ability to link the vehicle control response, for example, through ambient lights, sound effects, air conditioning, etc. to motivate or encourage users to continue to complete the remaining courses, etc., thereby giving users timely feedback and immersive experience. Figure 2B An exemplary linkage control system is shown as an example. According to the linkage control system of the present invention, multiple linkage modes can be predetermined, wherein each mode corresponds to a specific ambient light effect (including color, color change, and frequency, etc.), sound effect (for example, including specific music, voice, etc.), air conditioning (blowing out corresponding cold air or hot air, and corresponding duration, etc.). Figure 2BAs shown, the linkage control system may include the aforementioned breath assessment device 100, health quality assessment device 200 and control module 205, wherein the control module may select one or more appropriate modes from the above-mentioned multiple linkage modes according to the completion of one or more indicators of breath health, emotional stability, breath regulation stability and breath regulation potential provided by the breath assessment device 200, so as to enhance the user's experience and sense of participation. For example, when evaluating the overall completion of breath health, emotional stability, breath regulation stability and breath regulation potential, the area of ​​the radar chart generated by these indicators may be calculated, and a suitable mode may be selected based on the area.

[0102] In the above example of the present invention, the breath receiving module 201 can Figure 1 The breathing detection device 100 of the example shown receives the final classification sequence SegmentSeq or the frame classification sequence Final_Predict_Seq, or receives SegmentSeq or Final_Predict_Seq from other third parties. However, in other examples, the health quality assessment device 200 can also generate the detection sequence by itself only from the audio signal, such as detecting the sound in the car. Figure 2C Another example of the health quality assessment device 200 is shown, which further includes a generation module 204 for generating a breathing segment detection sequence SegmentSeq based on audio signal collection.

[0103] As shown in the figure, the sequence generation module 204 includes an audio collection and processing unit 2041 and a breathing detection unit 2042. The audio collection and processing unit 2041 uses a microphone in the vehicle to collect audio signals in the vehicle compartment for a period of time, such as a T time period, and generates an audio signal Sound_Sig and its N audio frames Sound_Sig = {SS_F1, SS_F2, SS_F3, ... SS_F N In addition, the audio collection and processing unit 2041 also extracts audio features ST, RT and PT in the manner of the feature extraction unit 1022. Thus, the audio feature extraction unit 2041 generates a feature vector for each audio frame SS_F, for example, SS_Fi = {ST, RT, PT}.

[0104] The breathing detection unit 2042 calls the breathing detection model S_BreathM to process the N audio frame sequences SS_F1-SS_F N To generate the corresponding breath / exhale detection classification sequence Breath_Predict_Seq.

[0105] Similar to the above-mentioned embodiment, in order to avoid the classification prediction error that may still exist in the breathing prediction frame sequence Breath_Predict_Seq, according to an example of the present invention, the generation module 204 may also include a decision module 2043, which is used to perform smoothing processing on the breathing prediction sequence Breath_Predict_Seq to avoid errors. Similar to the processing method of the decision module 1053, the decision module 2043 performs smoothing processing on the sequence Breath_Predict_Seq according to the pre-established decision tree model DecisionPattern, fine-tunes or corrects the classification prediction error, generates the final frame prediction sequence Final_Predict_Seq, and further determines which segment of exhalation, inhalation, breath holding and breath holding each continuous frame in the sequence Final_Predict_Seq specifically represents, and counts the segment length occupied by the segment, such as the duration. In this way, the decision module 2043 finally determines and stores the complete exhalation, inhalation, and breath holding segment sequence SegmentSeq within the time period T, and submits the sequence to the receiving module 201.

[0106] In addition, it can be seen that there may be occasional factors that affect signal acquisition or correct evaluation in reality. For example, when the user is normally following the guided course activities, a sudden phone call may affect the customer's mood, breathing, etc. Therefore, the health quality assessment device 200 will suspend the acquisition of signals midway, resulting in data collection interruptions during the entire time period T, and restart the acquisition when, for example, the phone call ends. Obviously, the longer the interruption time, the greater the impact on the evaluation effect of the system. To this end, according to an example of the present invention, the breathing detection model called by the breathing detection unit 104 is also trained to be able to recognize these interruptions, so that it can be identified in the breathing segment sequence SegmentSeq, such as {X, H, S, ... B ... H}, where S represents an interruption.

[0107] In order to take into account the impact of such interruption, according to an example of the present invention, for each of the above-calculated indicator scores BD, BI, BP, BS, etc., an attenuation factor α is applied, for example, BD·α→BD, that is, BD is updated using BD to which the attenuation factor is applied. Here, α can be determined by the following formula:

[0108]

[0109] Wherein Mp represents the pause duration or the corresponding pause frame number, Mo represents the total frame number or duration of exhalation, inhalation and breath holding, and β is a predetermined number greater than or equal to 1. Thus, the health quality assessment device 200 can calculate a multi-dimensional comprehensive indication based on the updated indicator scores BD, BI, BP, and BS.

[0110] In the above embodiments, it is described that the user's breathing is detected under the guidance of the breathing course and the quality assessment is performed based on the breathing detection results. However, in reality, as the user participates in the breathing course, in addition to changes in breathing, changes will also occur in physical signs, such as heart rate, blood oxygen saturation, etc. Therefore, a more accurate assessment of the health level can be achieved by comprehensively considering physical sign changes and breathing detection. Figure 3A A health assessment device according to an example of the present invention is shown.

[0111] like Figure 3A As shown, the health assessment device 300 includes a real-time detection module 301, a breathing assessment module 302 and a health assessment module 303. The real-time detection module 301 is used to detect the user's breathing characteristic signal and physiological sign signal when playing the guided course BreadthGuide1 within a time period T. For example, the breathing characteristic signal can be a breathing segment sequence SegmentSeq, and the physiological sign signal can include heart rate, blood oxygen saturation, etc. To this end, according to an example of the present invention, the real-time detection module 301 may include a breathing detection unit 3011 and a sign detection unit 3012. The breathing detection unit 3011 performs breathing detection based on the signals collected in the car, such as audio or video signals. For example, in one example, the breathing detection unit 3011 can be implemented by the sequence generation module 204, and a trained audio-breathing detection model S_BreathM is used to process the audio signal Sound_Sig collected by the microphone to generate a breathing frame classification prediction sequence, wherein each frame indicates one of exhalation, inhalation and breath holding, and then a breathing segment sequence SegmentSeq is generated and features such as the duration of each segment are determined; or the breathing detection unit 3011 can use a trained video-breathing detection model V_BreathM to process the video signal VideoStream collected by the camera to generate a breathing frame classification prediction sequence, and then a breathing segment sequence SegSeq is generated and features such as the duration of each segment are determined. In another example, the breathing detection unit 3011 can also be implemented by the breathing detection device 100, and a breathing segment sequence SegSeq is generated based on the audio signal and the video signal.

[0112] The physical sign detection unit 3012 is used to process the video signal stream VideoStream of the user in the car obtained by the camera to generate a physiological sign signal PhySig. The camera used here is an RGB camera or an infrared camera. For example, for an image based on an RGB camera, the physical sign detection unit 3012 can detect a region of interest (ROI) for each frame, such as a face or a forehead. The detected ROI area is subjected to feature extraction, such as extracting signal periodic features; or the ROI is spatially divided into several more subdivided areas, thereby forming a spatiotemporal mapping diagram of the subdivided areas of the ROI in each frame as an image feature. Therefore, the physical sign detection unit 3012 can process the image features (such as periodic features or spatiotemporal mapping features) of each extracted video frame by calling a trained image-physical sign prediction model ImageSignModel to generate a physiological sign signal PhySig, including heart rate and blood oxygen saturation. In this example, the physiological sign signal PhySig shows the heart rate and blood oxygen saturation level at the time point represented by each frame in the form of frames. Here, the image-sign prediction model ImageSignModel can be determined by supervised training using a variety of neural network architectures known in the prior art. Such network architectures include but are not limited to ResNET18 and the like.

[0113] In another example of the present invention, the vital sign detection unit 3012 can also realize the relevant detection of breathing behavior. For example, when the camera is an RGB camera, when performing breathing recognition, the vital sign detection unit 3012 uses an edge detection algorithm to identify the shoulders or chest edges of the characters in each RGB image frame, and then makes a direction judgment on the edge detection of adjacent frames, thereby determining the final breathing or inhalation or breath holding behavior. In another example, when using an infrared camera to identify breathing, according to each frame of infrared thermal imaging, the vital sign detection unit 3012 first extracts the region of interest (ROI), such as the nostril area, and calculates the grayscale average of the nostril area to obtain an initial dynamic breathing waveform based on a time series, thereby converting the temperature change of the thermal infrared image into a periodic change of the grayscale value, and then identifying the breathing, inhalation, and breath holding state. Based on the identified breathing, inhalation, and breath holding states, the respiratory frequency can be further calculated.

[0114] According to an embodiment of the present invention, changes in breathing frames or breathing segments over time can be displayed in the form of frames or segments on the human-machine interface HMI, or such changes can be displayed in comparison with the breathing requirements prompted by the guided course BreadthGuide1; at the same time, changes in heart rate and blood oxygen saturation level over time as the guided course BreadthGuide1 progresses are also displayed on the human-machine interface.

[0115] The breathing assessment module 302 determines the user's breathing completion degree and / or breathing quality based on the breathing characteristic signal of the breathing sequence provided by the module 301 and the reference signal based on the preset guidance course language. Figure 3A As shown, the breathing assessment module 302 includes a quality assessment unit 3021 and a guide matching unit 3022. The quality assessment unit 3021 determines the duration of the exhalation segment, the inhalation segment, and the pause segment contained in the sequence based on the sequence SegmentSeq to determine the breathing quality BreathQuality. The specific measurement index of the breathing quality here may include, for example, the breath duration BP, or may be based on a comprehensive assessment of the breath reaching degree BD, the breath duration BP, the breath stability BS, and the breath positivity BI, such as a weighted average.

[0116] Usually, the guided courses have preset requirements and guiding words such as the duration of exhalation and inhalation, breath holding, etc. The guided matching unit 3022 obtains the breathing sequence expected by the course setting as a reference signal based on the preset requirements and guiding words of the guided course BreadthGuide1, such as the specific rhythmic law of breathing, frequency, duration of exhalation, inhalation, breath holding, etc.; then compares the features related to the exhalation segment, inhalation segment, and breath holding segment contained in the SegmentSeq sequence (such as the duration of exhalation, inhalation, breath holding, and the calculated rhythm, frequency, etc.) with the reference signal, for example, calculates the PSI (group stability index) of the two, and the obtained value is used as the completion degree of the user's breathing training during the course BreadthGuide1, BreathMatch.

[0117] The health assessment module 303 evaluates the health status of the user during the playing of the guided course BreadthGuide1 within the time period T based on the breath control completion degree BreathMatch, the breathing quality BreathQuality, the physiological sign signal heart rate, and the blood oxygen saturation. For example, the changes of BreathMatch, the breathing quality BreathQuality, the physiological sign signal heart rate, and the blood oxygen saturation within the time T can be dynamically displayed in the form of a radar chart on the human-machine interface HMI. In another example, the health assessment module 303 can also directly generate multidimensional health indicators based on the breathing segment sequence and the physiological sign signal. Such health indicators include emotional stability EMS, breath duration BP, breathing regulation, regulation initiative, and cardiopulmonary level. For example, the slower the number of exhalations (H) and inhalations (X) per minute and the heart rate, the more stable the emotional EMS; otherwise, the emotional EMS is unstable. For example, the emotional stability EMS at the current moment of the rth second can be calculated as follows:

[0118] Emotional stability EMS = EMS_End-EMS_start, (Formula 4) where

[0119] EMS_Start=(w1*Br_Start_score+w2*Hr_Start_score) / (w1+w2);

[0120] EMS_End=(w1*Br_End_score+w2*Hr_End_score) / (w1+w2);

[0121] Wherein, Br_Start_score is the emotional stability score corresponding to the average respiratory rate of the previous r seconds (denoted as A1); Br_End_score is the emotional stability score corresponding to the average respiratory rate from the rth second to the current moment (denoted as A2). According to one embodiment of the present invention, a respiratory rate-emotional stability segmentation mapping table can be used to calculate the emotional stability score. For example, for a specific respiratory rate x, the respiratory rate segmentation interval [start_r, end_r] in the segmentation mapping table can be determined according to the segmentation mapping table. The emotional stability score x_score corresponding to x will be linearly mapped to the emotional stability score interval [start_e, end_e]. Therefore, the emotional stability score x_score corresponding to the respiratory rate x can be determined by the following formula:

[0122] X_score=start_e+(x-start_r)*(end_e–start_e) / (end_r–start_r) (Formula 5).

[0123] Therefore, using formula (5), the emotional stability score Br_Start_score corresponding to the breathing frequency A1 and the emotional stability score Br_End_score corresponding to the breathing frequency A2 can be calculated respectively.

[0124] In addition, in formula (4), Hr_Start_score is the emotional stability score corresponding to the average heart rate of the previous r seconds; Hr_End_score is the emotional stability score corresponding to the average heart rate from the rth second to the present. In this example, a heart rate-emotional stability segment mapping table is used to determine the emotional stability score, and Hr_Start_score and Hr_End_score can be calculated in a manner similar to the above-mentioned function of converting breathing frequency to emotional stability score, i.e., formula (5).

[0125] In addition, the weights w1 and w2 in formula (5) may be predetermined values; and in another embodiment, they may vary according to the accuracy of the prediction model, for example, they may be set to the corresponding audio confidence A_CON and video confidence V_CON.

[0126] In addition, as mentioned above, the strength of breath duration BP can be represented by the length of breathing time. As for the breathing regulation power, it is determined by calculating the ratio of exhalation time to inhalation time. When the ratio of exhalation time to inhalation time is high, it means that the regulation power is strong; otherwise, it means that the regulation power is weak. The regulation initiative is judged by whether the breathing time is reduced. When the breathing time does not decrease with the progress of time and the guided course, it means that the breathing coordination is strong; otherwise, it means that the breathing coordination is weak. In addition, for the cardiopulmonary level, it can be comprehensively judged based on the breathing sequence, the heart rate decrease index and the blood oxygen saturation increase index.

[0127] According to another example of the present invention, the health assessment module 303 can also evaluate the user's breathing potential BreathPower based on the breathing completion degree BreathMatch, the breathing quality BreathQuality and / or the physiological sign signals heart rate and blood oxygen saturation. This breathing potential indicates the comparison of the user's breathing or breathing ability relative to the breathing training intensity required by the current guided course or the current progress of the course. The ratio result can indicate a large potential, that is, the user has great potential to strengthen training; or a small potential, that is, the current guided course is at least suitable. In one example of the present invention, a conventional efficiency evaluation method can be used to evaluate the breathing potential BreathPower, for example, by constructing a data envelopment analysis (DEA) CCR model, in which behavioral data such as breathing completion and breathing quality are used as input variables, and vital signs such as heart rate, blood oxygen saturation, blood pressure, etc. are used as output data, so as to evaluate the efficiency of the decision-making unit (DMU) including breathing completion BreathMatch, breathing quality BreathQuaility, physiological vital signs heart rate, blood oxygen saturation, blood pressure and other data, and use the estimated comprehensive efficiency as the breathing potential BreathPower.

[0128] It should be noted that the health assessment device 300 continuously calculates and updates the scores of the user's breathing completion, breathing potential and other indicators as the guided course progresses. Therefore, according to another embodiment of the present invention, when the health assessment device 300 finds problems with the user during the course, it can issue timely reminders, or give timely encouragement when the user completes the breathing with high quality. Figure 3BAs shown, in addition to modules 301, 302, and 303, the health assessment device 300 also includes a guide language matching module 304, which is used to provide users with timely breathing scene guidance based on the original course based on the BreathPower potential and BreathMatch index of the breathing completion degree and / or the multi-dimensional health index determined by the health assessment module 303, that is, to match additional scenes as the guided course BreadthGuide1 proceeds, so as to enhance the user experience without changing the course. Figure 3B As shown, the guide language matching module 304 may include a scene matching unit 3041 and a time point detection unit 3042. The scene matching unit 3041 is configured to determine the best matching scene based on the breathing completion degree BreathMatch and BreathPower. For example, during the guided course, if the user's breathing completion degree BreathMatch has been very good, praise words such as "You are awesome" can be generated; and when the breathing completion degree BreathMatch is low, encouraging words such as "Keep it up" can be generated. In addition, when the breathing potential BreathPower indicates that the user has greater potential, a prompt scene such as "You have to exceed the current breathing depth" can be generated. These additional scenes can be guide words, micro-animations or background sound effects in text or voice form.

[0129] After determining the above-mentioned additional scenes, the timing detection unit 3042 can be used to determine the timing of inserting these additional scenes into the guided course BreadthGuide1, referred to as the timing. Assume that the entire guided course BreadthGuide1 is divided into M sections of guidance language according to factors such as the content and intensity of the guidance, wherein the M sections of guidance language contain one or more sections with the same intensity. For example, when the course reaches the m1th section, the scene matching unit 3041 generates encouraging words such as "Very good, keep up the good work!", and when it reaches the m2th section, the scene matching unit 3041 generates a prompt scene of "Please hold your breath for a short time after inhaling and then exhale", where m2>m1>1. According to an example, these prompts can be presented on the human-computer interface:

[0130] “Very good, keep up the good work!”

[0131] "After inhaling, please hold your breath briefly before exhaling."

[0132] Then, the time point detection unit 3042 analyzes the content, intensity and other factors of each paragraph of the guiding language in the guided course BreadthGuide1 except for the first n paragraphs of guiding language. For example, when it is found that the m5th paragraph is similar to the m1th paragraph, it is determined to insert "Pay attention to breathing, please take a deep breath" at the beginning of the m5th paragraph. Then, when approaching the beginning of the m5th paragraph, "Pay attention to breathing, please take a deep breath" is highlighted or the encouraging words are played in voice form, so as to remind the user to follow the m5th paragraph of the guiding language for breathing training. Similarly, when it is found that the m8th paragraph is similar to the m2th paragraph, it is determined to insert "Please hold your breath for a short time after inhaling and then exhale" at the beginning of the m8th paragraph. Then, when approaching the beginning of the m8th paragraph, "Please hold your breath for a short time after inhaling and then exhale" is highlighted or played in voice form, so as to suggest the user to perform breathing training with a greater intensity than the breathing training intensity of the reference guiding language.

[0133] In another embodiment of the present invention, a health assessment device can also be used to directly perform a health assessment as the breathing guidance course proceeds. As an example, the health assessment device may include a deep breathing guidance module, a breathing detection device, a breath adjustment detection module, and a health assessment module. The breathing guidance module presents a deep breathing course guidance to the user through an on-board output medium such as a display screen, a speaker, a light, etc., and the guidance content includes presentation in one or more forms of text, video, and audio. The breathing detection device generates a breathing classification detection sequence based on the audio signal and the video signal collected during the deep breathing course guidance, where the breathing classification detection sequence represents the breathing breath of the user. Here, the breathing detection device can generate the breathing breath in the aforementioned manner, for example, in a manner similar to that of the breathing detection device 100 or module 204. The breath adjustment detection module determines the matching degree of the breathing breath with the preset requirements of the course guidance based on the user's breathing breath detected during the presentation of the deep breathing course guidance so as to evaluate the user's breath adjustment ability. Thus, the health assessment module can determine the user's health based on the breath adjustment ability. In one example, the health assessment module can be implemented by the aforementioned health assessment module 303, and further utilizes the breath control ability assessment and the physiological signs provided by the sign detection unit 3012 to determine the user's health, and calculates the health improvement by comparing the health before and after the deep breathing course guidance, for example, by calculating the emotional stability EMS (=EMS_End-EMS_Start) as described above to characterize the health improvement. Of course, other methods can also be used to determine the health. As an example, first determine the emotional stability based on the breathing rate and heart rate; and determine the breath duration, breathing regulation and breath control initiative based on the exhalation and inhalation time characteristics of the user's breath; and determine the cardiopulmonary level based on the breathing rate, heart rate, and blood oxygen saturation. Thus, the health assessment module constructs a pattern diagram such as a radar diagram based on emotional stability, breathing regulation, breath duration, breath control initiative and cardiopulmonary level, and compares it with a predetermined standard health pattern diagram to determine the health, for example, by calculating the area ratio of the constructed pattern diagram to the standard pattern diagram as the health standard. In another example of the present invention, the health assessment module may instruct the deep breathing guidance module to update the deep breathing course guidance based on the health improvement degree such as EMS, such as making the guidance process stronger or more soothing, or changing the guidance course.

[0134] Practice has shown that correct breathing training is very beneficial for purifying the respiratory tract, repairing the respiratory system, strengthening the respiratory muscles and diaphragm function to improve lung capacity, etc. However, due to individual differences among users and their different levels of mastery of breathing training, the difficulty requirements for the courses are also different. It is necessary to recommend personalized breathing training courses to users in a targeted manner.

[0135] According to the present invention, a plurality of personalized breathing courses BreathGuide are stored in the course library, such as yoga breathing methods, such as basic abdominal breathing, complete breathing, to common throat breathing, holy light breathing methods. These breathing methods vary according to application scenarios, difficulty and expected corresponding effects. In an example of the present invention, according to different breathing methods, the multidimensional course features of each personalized breathing course are extracted, including but not limited to: breathing method effects, suitable population, taboos, difficulty classification, scoring criteria, music style, guide word library, etc. According to an example of the present invention, for each guided course, a specific mark can be used to mark the corresponding course, and the mark can be an identification that highlights a specific course feature of the course, such as using a specific guide + micro animation of the course to identify the course difficulty, or using a specific music background style to mark, etc. These course features are stored in the course library together with the breathing course and its mark. Figure 4 The course recommendation device according to the present invention is shown, which can recommend a suitable next course from multiple pranayama courses stored in the course library according to the user's learning of historical courses. As shown in the figure, the course recommendation device 400 includes a course evaluation module 401, a feature extraction module 402, a personalized recommendation module 403, and a comprehensive recommendation module 404.

[0136] After the user has completed the corresponding breathing training according to the currently recommended or selected breathing course, such as BreadthGuide1, the course evaluation module 401 is used to evaluate the user's training status under the guided course. As described in the previous embodiments, the training status here includes health status, course completion, breathing potential (such as breathing adjustability, breathing sustainability, and breathing initiative, etc.) and course acceptance. The course acceptance is related to the changing trend of the user's physiological signs, wherein the health status can be determined based on the user's physiological signs, such as cardiopulmonary level, and can also be determined based on breathing measurements, such as breath compliance (BD), breath sustainability (BP), breath stability (BS) and breath positivity (BI). In one example of the present invention, the course recommendation device 400 may include the above, for example, in combination with Figure 1 The described breathing detection device 100 generates a breathing classification detection sequence based on the audio signal and the video signal collected during the guidance of the course BreadthGuide1, so that the course evaluation module 401 evaluates the user's training status under the course BreadthGuide1 based on the breathing classification detection sequence from the breathing detection device 100. Here, the course evaluation module 401 can be implemented by means such as health evaluation introduced in the above embodiments, for example, in the same way as the health evaluation device 300.

[0137] The feature extraction module 402 receives the evaluated training status from the evaluation module 401; at the same time, it further optionally obtains other features of the user, such as the user behavior preference (preference) extracted from the user's click behavior on the course browsing during the course BreadthGuide1, and the interest matrix for the course, where the interest matrix represents the user's participation in different stages or points of the course. As an example, in this interest matrix, for example, the matrix row information can represent the course stage, the matrix column information is the course attribute, and the value in the matrix is ​​the user's interest in different attributes of the course extracted at different course stages, which is determined by the user behavior continuously collected during the course. Thus, the feature extraction module 402 generates user features that include training status and user personalized features such as preferences and points of interest in the course.

[0138] Based on the multiple matching modes and the user features provided by the feature extraction module 402, the personalized recommendation module 403 selects multiple recommended ranking lists of pranayama courses from the course library through different algorithms, where each list is the result of algorithm processing corresponding to a matching mode, and the ranking of each course in the list represents the matching degree between the user features and the candidate pranayama courses in the pranayama course library. According to an embodiment of the present invention, multiple matching modes for calculating the matching degree between users and courses are defined, including a user interest-based mode and a user ability-based mode in one example. The user interest-based mode includes:

[0139] The user's recent interest pattern, based on which the courses in the generated ranked list are all related to the courses that the user has actively participated in recently, such as having the same course characteristics or tags;

[0140] The user's long-term interest pattern, based on which the courses in the generated ranked list are related to the courses that the user has actively participated in over a long period of time, such as the courses on the list have the same or similar course features or tags;

[0141] User differentiated interest model, in which the courses on the list have significantly different course characteristics or tags based on the wide range of user interests.

[0142] The user capability-based model may include a user history completion association model, a user's last course association model, a user health status model, etc.

[0143] For the user history completion degree association mode: the personalized recommendation module 403 can select courses with similar course characteristics to the history courses with better completion degrees from the course library to generate a list based on the completion degrees of the user's history courses determined by the course evaluation module 401;

[0144] For the user's last course association mode: for example, the training status of the last course, such as completion, can be directly analyzed. When the completion is not good, a course with similar course characteristics is selected from the course library; and when the completion is good, a course with upgraded characteristics is selected from the course library, such as selecting a course with increased difficulty or intensity as a subsequent course.

[0145] For the user's health status model: for example, based on the user's emotional stability, endurance, and cardiopulmonary level, choose courses that are more helpful in improving these health indicators.

[0146] In another example of the present invention, the matching mode further includes a user online mode and an offline mode, wherein in the online mode, the personalized recommendation module 403 can learn user information, big data statistics, and environmental factors of other users to generate a recommendation list. For example, when the device 400 can be connected to the network, the personalized recommendation module 403 can request big data statistics of the current online user from the remote server based on the course statistics algorithm, including the breathing courses currently being used by the current online user, thereby generating a ranking list of the most popular courses; in addition, based on the portrait statistics algorithm, based on the user level (such as course completion, breathing ability, etc.), usage habits, preferences for course features, etc., other users registered on the server with similar portraits to the current user are determined. The portrait here is used to define multiple characteristic elements of the user. Furthermore, the personalized recommendation module 403 extracts courses that other users with similar portraits are currently studying or have studied in the past from the course library, and generates a similar user recommendation list to provide to the customer. In addition, in online mode, the personalized recommendation module 403 can also recommend courses that other users are currently studying or have studied in the past in similar environments based on the environment at the time, such as the temperature and humidity inside and outside the car, the weather of the day, solar terms, holidays, weeks, current time points, etc., and form a recommended list of today's courses.

[0147] After receiving the most popular courses ranking list, similar user recommendation list, and today's course recommendation list generated by the personalized recommendation module 403, the comprehensive recommendation module 404 evaluates the influence of these courses on the user, for example, by analyzing the user's feedback behavior on historical recommendations. For example, when a user has rejected certain popular courses or other specific courses many times in history, the comprehensive recommendation module will eliminate courses similar to the courses that the user has rejected in history from the above three lists, or even reject such lists; at the same time, for courses that the user has actively participated in history, priority weights will be given to courses in the above three lists that have similar characteristics to the courses that the user has actively participated in. In this way, the comprehensive recommendation module 404 merges a comprehensive ranking list from the most popular courses ranking list, the similar user recommendation list, and the today's course recommendation list for the user to choose from; or automatically selects one or more courses with the highest rankings from them.

[0148] In offline mode, the comprehensive recommendation module 404 can generate a recommendation list based on the user's activity in participating in the course. This activity can be indicated by the user's local historical data statistics, such as each participation time of the course, long-term usage frequency statistics, and short-term usage frequency statistics. When the activity is low, the recommendation list can be determined only based on the user's training status for the currently completed course. Based on the training status, the course list mainly includes courses with similar course features or higher-level courses as the currently completed course. When the activity is low, the course recommendation list can be generated based on the evaluation of the training status of the current course and the training status evaluation of historical courses, usage habits, and environmental factors.

[0149] Figure 5 The flowchart of the breathing detection method according to the example of the present invention is shown. In step 501, the microphone in the vehicle is used to collect the audio signal Sound_Sig in the cabin for a period of time, such as T time period, and the camera in the vehicle is used to collect the video signal stream Video_Steam in the cabin at the same time. Then these signals are processed, for example, the audio signal Sound_Sig is processed by performing back-elimination processing on the collected audio signal Sound_Sig using the music signal played by the pre-stored guided course BreathGuide, and the sliding time window w is used for sampling, and divided into multiple audio frames {SS_F1, SS_F2, SS_F3, ... SS_F N}, and extract the acoustic signal features, such as {ST, RT, PT} discussed above. Similarly, for the video signal stream Video_Steam, after preprocessing, N video frames VS = {VS_F1, VS_F2, VS_F3, ...VS_F N} and its corresponding video signal features such as {IT, CT, MT}.

[0150] Then, in step 503, the audio confidence evaluation model Sound_CEM is called to process the acoustic signal features {ST, RT, PT}) to generate an audio prediction confidence A_CON for performing corresponding breathing prediction for each audio signal. i Thus, for the currently collected audio signal Sound_Sig, its N confidence frame sequences {A_CON1, A_CON2, A_CON3, ..., A_CON N}, where each value in the sequence represents the confidence of predicting breathing behavior based on the corresponding audio frame. Similarly, the video confidence evaluation model Video_CEM is called to process the feature vector {IT, CT, MT} of each video frame to generate the video confidence V_CON of each video signal frame i, thus forming N confidence frame sequences {V_CON1, V_CON2, V_CON3, ..., V_CON N}, where each value in the sequence represents the confidence of predicting the breathing behavior based on the corresponding video frame.

[0151] In step 505, the resource conditions of the current vehicle are identified, such as confirming the computing power Power_ID of the current vehicle, and searching the model library DB for a breathing detection model suitable for performing breathing detection through the confirmed computing power Power_ID, including not only the audio model S_BreathM, but also the video prediction model V_BreathM. For ease of description, it is assumed that the audio model S_BreathM4 and the video model V_BreathM8 are determined to be suitable for the current vehicle from the model library. Based on the audio frame confidence sequence {A_CON1, A_CON2, A_CON3, ..., A_CON N} and the video frame confidence sequence {V_CON1, V_CON2, V_CON3, …, V_CON N}, using the adapted breathing detection model, such as S_BreathM4 and V_BreathM8 in this example, to process the audio frame sequence {SS_F1, SS_F2, SS_F3, ... SS_F N} and the video frame sequence {VS_F1, VS_F2, VS_F3, ...VS_F N Specifically, in step 507, for example, taking the audio frame confidence sequence as an example, the current confidence A_CON is first determined. i Is the value greater than the predetermined confidence threshold CON? T If A_CON is determined in step 507 i Less than the threshold CON T , abandon the current frame SS_F i The classification prediction of , because the prediction at this time is not credible. Otherwise, if A_CON i Greater than or equal to the threshold CON T , then go to step 509 and call S_BreathM4 to process the A_CON i The corresponding audio frame SS_F i To generate a breathing classification prediction for the frame. In this way, a breathing classification prediction sequence SS_CLass based on the audio frame sequence can be formed; optionally, the prediction probability score SS_Score of each frame classification prediction can be recorded at the same time. In a similar manner, a breathing classification prediction sequence VS_CLass based on the video frame sequence is also formed; optionally, the prediction probability score VS_Score of each frame classification prediction can be recorded at the same time.

[0152] In step 511, the generated breathing classification prediction sequences SS_CLass and VS_CLass including audio prediction and video prediction are fused to determine the final breathing classification prediction within the time period T. To this end, according to an example of the present invention, firstly based on the sliding time window w when creating the audio frame t The classification prediction sequence SS_CLass corresponding to the audio signal frame and the prediction sequence VS_CLass corresponding to the video signal frame are aligned in time with the timestamp t when the video frame is generated, and then the prediction probability score SS_Score / VS_Score or the confidence A_CON / V_CON or a combination of the two can be used to perform call classification fusion based on audio and video. For example, for the classification prediction frames in the classification prediction sequence SS_CLass and the classification prediction sequence VS_CLass that do not conflict in time, that is, the corresponding audio frame SS_F and video frame VS_F occur at different times, the non-conflicting classification prediction frames are all recorded in the final breathing classification prediction; and when there are two classification predictions of audio and video at the same time, according to an example, a selection can be made based on the audio confidence and video confidence corresponding to the two classification predictions, for example, the classification prediction with high confidence is selected as the classification prediction result at that time, thereby generating a breathing / respiration classification prediction sequence Breath_Predict_Seq that integrates the audio prediction and the video prediction.

[0153] According to a further embodiment of the present invention, a smoothing process is performed on the breathing prediction sequence Breath_Predict_Seq to avoid errors. As an example, a smoothing process can be performed on the sequence Breath_Predict_Seq according to a pre-established decision tree model DecisionPattern, and the classification prediction error can be fine-tuned or corrected. The decision tree model DecisionPattern is determined by prior learning or based on experience, and is used to determine which of the exhalation, inhalation, or breath holding classifications a group of continuous frames in the breathing prediction sequence Breath_Predict_Seq most likely represents, and the purpose of using the decision tree model DecisionPattern is to identify the true classification represented by the possibly abnormal continuous frames. Thus, the final frame prediction sequence Final_Predict_Seq is formed. Furthermore, a breathing segment sequence SegmentSeq can also be generated based on Final_Predict_Seq, wherein each segment in the sequence defines which segment of exhalation, inhalation, or breath holding each continuous frame specifically represents, and the segment length occupied by the segment is counted.

[0154] Figure 6A health quality assessment method according to an example of the present invention is shown. As shown in the figure, in step 601, a frame classification prediction sequence Final_Predict_Seq is obtained; or a breathing segment sequence is directly received, such as SegmentSeq generated in response to the aforementioned breathing guidance course, the sequence includes exhalation H, inhalation X, and breath holding B segments, and the duration feature corresponding to each segment in the sequence is received at the same time. In one example, the classification prediction sequence Final_Predict_Seq or the breath segment sequence SegmentSeq is obtained by reference Figure 1 The breathing detection device shown is generated based on the collected audio signal and video signal.

[0155] In step 603, based on each breathing segment and the corresponding duration characteristics in the classification prediction sequence Final_Predict_Seq, multiple breath characteristics within the time period T are calculated, including but not limited to breath compliance (BD), breath persistence (BP), breath stability (BS) and breath positivity (BI).

[0156] According to a preferred embodiment, the influence of incidental factors such as interruptions in the breathing guidance process on the evaluation is also considered. For this reason, in step 605, these interruptions are identified from the classification prediction sequence Final_Predict_Seq. It should be pointed out here that these interruptions are also a classification confirmed when performing classification prediction using the trained breathing detection model. Then, the attenuation factor α is calculated using these interruptions, and then the attenuation factor α is applied to the indicator scores BD, BI, BP, and BS determined in step 603 to achieve the update of the indicator scores BD, BI, BP, and BS.

[0157] In step 607, based on the indicator scores BD, BI, BP, BS determined in step 603 or the BD, BP, BS, BI updated in step 605, multi-dimensional indicators representing breathing quality are generated, such as breath health BJK, emotional stability EMS, breathing stability BRS and breathing potential BRP.

[0158] In a further embodiment, the vehicle control response can be linked based on the calculated comprehensive indicators such as breath health index, emotional stability, breathing stability, etc., for example, through ambient lighting, sound effects, air conditioning, etc. to motivate or encourage the user to complete the remaining courses, thereby giving the user timely feedback and immersive experience.

[0159] Figure 7A flowchart of a health assessment method according to another example of the present invention is shown, wherein changes in the user's physical signs, such as heart rate, blood oxygen saturation, etc., are also considered when performing health assessment. In step 701, a deep breathing course guide is presented to the user through an in-vehicle output medium such as a display screen, a speaker, a light, etc., and the guide content includes presentation in one or more forms of text, video, and audio. In step 703, based on the user's breathing breath detected during the presentation of the deep breathing course guide, such as a breathing segment sequence, the matching degree of the breathing breath with the preset requirements of the course guide (such as the duration and frequency of exhalation and inhalation) is determined to evaluate the user's breathing ability. Here, a breathing classification detection sequence Final_Predict_Seq or a breath segment SegmentSeq representing the user's breathing breath can be generated in the aforementioned manner, for example, in a manner similar to that of the breathing detection device 100 or the breathing detection module 204. In addition, in step 705, the user's physiological sign signal is detected, and any of the aforementioned methods can be used to detect, for example, heart rate, blood oxygen saturation, etc. Then, in step 707, the user's health is determined by using the breathing ability determined in step 703 and the physiological signs determined in step 705, including calculating the health improvement by comparing the health before and after providing the deep breathing course guidance, for example, by calculating the emotional stability EMS (=EMS_End-EMS_Start) as described above to characterize the health improvement. Finally, in step 709, the deep breathing course guidance is updated based on the health improvement, such as EMS, for example, to make the guidance process stronger or more relaxed, or to change the guidance course, etc.

[0160] Although breathing training is very beneficial for purifying the respiratory tract, etc., due to the individual differences of users and the different levels of mastery of breathing training, the difficulty requirements of the course are also different. It is necessary to recommend personalized breathing training courses to users in a targeted manner. Figure 8 A flow chart showing a method for recommending a personalized breathing course according to an example of the present invention.

[0161] After the user completes the corresponding breathing training according to the currently recommended or selected breathing course, such as BreadthGuide1, or during the training process, in step 801, based on the audio signal and video signal collected during the training guide course BreadthGuide1, a breathing classification detection sequence is generated, for example, using Figure 1 The breathing detection device 100 is used to generate a breathing classification detection sequence Final_Predict_Seq, and based on Final_Predict_Seq, the user's training status under the guided course is evaluated, where the training status includes health status, course completion, breathing potential, such as breathing adjustability, breath duration, and breathing initiative, as well as course acceptance.

[0162] In a preferred example, step 803 is also included to obtain other characteristics of the user, such as the user behavior preference (preference) and the interest matrix for the course extracted from the user's click behavior on the course browsing during the BreadthGuide1 course. As an example, the interest matrix may include information such as the course stage, course attributes, and the user's interest in different attributes of the course extracted at different course stages.

[0163] In step 805, based on multiple matching modes and user characteristics, multiple recommended ranking lists of pranayama courses are selected from the course library through different algorithms, where each list is the result of algorithm processing corresponding to a matching mode, and the ranking of each course in the list represents the matching degree between the user characteristics and the candidate pranayama courses in the pranayama course library. According to an embodiment of the present invention, multiple matching modes for calculating the matching degree between users and courses are defined, for example, including user interest-based modes and user ability-based modes, wherein the user interest-based modes include user recent interest modes, user long-term interest modes, user differentiated interest modes, user historical completion association modes, user last course association modes, user health status modes, etc. In another embodiment, the recommended ranking list can also be generated accordingly based on whether the user's vehicle is in an online state. For example, in an online mode, the recommendation list can be generated by using user information of other users learned through the network, big data statistics, and environmental factors.

[0164] In step 807, in order to integrate multiple recommendation ranking lists generated by different matching methods, the influence of different courses on the user is evaluated, for example, by analyzing the user's feedback behavior on historical recommendations. For example, when the user has rejected certain popular courses or other specific courses many times in history, or courses in which the user has actively participated in history. Then in step 809, based on historical feedback, such as different feedback impacts caused by different courses, different ranking weights are assigned to different courses in the multiple recommendation ranking lists, such as giving priority weights to courses with similar characteristics to courses in which the user has actively participated. In this way, the above multiple recommendation rankings are integrated into a comprehensive ranking list for the user to choose from; or one or more courses with the highest rankings are automatically selected from them.

[0165] Fig. 9 Another example of a health assessment method according to the present invention is shown. As shown in the figure, in step 901, the user's breathing characteristics and physiological signs are detected during the time period T when playing the guided course BreadthGuide1, for example, using Figure 1The described breathing detection device 100 processes the audio data collected by the microphone and the video stream captured by the camera to generate the user's breathing classification detection sequence Final_Predict_Seq or breathing segment sequence SegmentSeq, wherein the breathing classification detection sequence Final_Predict_Seq or the breathing segment sequence SegmentSeq indicates breathing characteristics including exhalation, inhalation and breath holding; and the physiological sign signal can be generated by processing the video stream captured by the camera, including heart rate, blood oxygen saturation, blood pressure, etc.

[0166] In step 903, the breathing quality BreathQuality is determined based on the duration of the exhalation segment, the inhalation segment, the breath holding segment, etc. contained in the sequence. Here, the breathing quality may include, for example, the breath duration BP, or may be based on a comprehensive evaluation of the breath reaching standard BD, the breath duration BP, the breath stability BS, and the breath positivity BI, such as a weighted average.

[0167] In step 905, the user's breathing completion degree BreathMatch is determined by comparing the exhalation segment, inhalation segment, and breath holding segment contained in the sequence with the preset breathing guidance requirements. Usually, the guidance courses have preset requirements such as the duration of exhalation and inhalation, breath holding, etc. Therefore, the breathing sequence expected by the course setting can be obtained as a reference signal based on the preset requirements of the guidance course BreadthGuide1, such as the specific rhythm and regularity of breathing, frequency, duration of exhalation, inhalation, and breath holding, etc.; then the breathing-related features contained in the SegmentSeq sequence (such as the duration of exhalation, inhalation, breath holding, and the calculated rhythm, frequency, etc.) are compared with the reference signal, for example, the PSI of the two is calculated as the completion degree BreathMatch.

[0168] In step 907, based on the breath control completion degree BreathMatch, the breathing quality BreathQuality, the physiological sign signal heart rate, and the blood oxygen saturation, the health status of the user during the playing of the guided course BreadthGuide1 within the time period T is evaluated, for example, the changes of BreathMatch, the breathing quality BreathQuality, the physiological sign signal heart rate, and the blood oxygen saturation within the time period T can be dynamically displayed in the form of a radar chart on the human-machine interface HMI. In another example, a multi-dimensional health index is directly generated based on the breathing segment sequence and the physiological sign signal, and such health index includes emotional stability EMS, breath duration BP, breathing regulation ability, regulation initiative, and cardiopulmonary level. According to another example of the present invention, the user's breathing potential BreathPower can also be evaluated based on the breathing completion BreathMatch, breathing quality BreathQuality and / or physiological sign signals heart rate, blood oxygen saturation. This breathing potential indicates the comparison of the user's breathing or breathing ability with the breathing training intensity required by the current guided course or the current progress of the course. For example, the efficiency evaluation of the data envelopment analysis (DEA) CCR model can be used as the breathing potential BreathPower. This indicator can also indicate the user's health status. Based on the above-mentioned determined health status information, the present invention can also modify the current guided course to better enhance the user experience. To this end, in step 909, based on the breathing potential BreathPower and the breathing completion BreathMatch indicators, determine the best breathing scenario guidance that can be provided to the user on the basis of the original course, such as the additional scenarios matched with the guided course BreadthGuide1, so as to enhance the user experience without changing the course. For example, during the course, if the user's BreathMatch is always good, praise words such as "You are awesome" can be generated; and when the BreathMatch is low, encouraging words such as "Keep working hard" can be generated. These additional scenes can be guiding words in text or voice form, micro animations or background sound effects.

[0169] In step 911, after the above additional scenes are determined, the time point for inserting these additional scenes into the guide course BreadthGuide1 is further determined, that is, the best introduction time, such as inserting the corresponding additional scenes before or after a certain guide course in the whole course.

[0170] Although the exemplary equipment and methods of the present invention are described above in conjunction with specific examples, it is understood that the units or modules disclosed herein may include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software codes, firmware codes, etc., or any combination thereof. It will also be appreciated by the technician that the various illustrative logic blocks and method steps described in conjunction with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of the two. Moreover, the equipment or apparatus according to the present invention may be implemented as a processor and a memory, wherein the memory stores a computer program, wherein the processor may implement each module, unit, and method step by executing the computer program. In addition, a machine-readable medium provided in another embodiment of the present invention stores a machine-readable instruction, which, when executed by a processor, causes the processor to execute any of the aforementioned methods disclosed herein.

[0171] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A breathing detection device, comprising: an audio evaluation unit configured to evaluate audio confidence of breathing detection based on the audio signal using the audio signal collected from the vehicle cabin; A video evaluation unit configured to evaluate the video confidence of the breathing detection based on the video signal using the video signal stream collected from the vehicle; a model matching unit configured to match at least one breathing detection model suitable for performing breathing detection based on vehicle conditions; a breathing detection unit configured to call the at least one breathing detection model to process the audio signal and the video signal based on the audio confidence and the video confidence to generate a first breathing detection sequence corresponding to the audio signal and a second breathing detection sequence corresponding to the video signal; The detection fusion unit is configured to fuse the first breathing detection sequence and the second breathing detection sequence to generate a breathing classification detection sequence.

2. The apparatus of claim 1, wherein the audio evaluation unit is configured to divide the audio signal into a plurality of audio signal frames according to a sliding time window, and wherein the video evaluation unit divides the video signal stream into a plurality of video signal frames with time stamps.

3. The apparatus of claim 2, wherein the audio evaluation unit comprises: A first preprocessing unit, configured to perform audio preprocessing on the audio signal frame; A first feature extraction unit, configured to extract signal features of the preprocessed audio signal frame; A first evaluation module is configured to call an audio confidence evaluation model to process the signal features to generate an audio confidence of each audio signal frame; The video evaluation unit comprises: A second preprocessing unit, configured to perform video preprocessing on the video signal frame; A first feature extraction unit configured to extract optical features of the preprocessed video signal frame; The first evaluation module is configured to call a video confidence evaluation model to process the optical features to generate a video confidence of each video signal frame.

4. The apparatus of claim 3, further comprising a breathing detection model library, wherein a plurality of different audio-breathing detection models and video-breathing detection models that have been trained are stored, wherein each of the plurality of different audio-breathing detection models and video-breathing detection models is trained to adapt to different constraints, wherein the constraints include at least one of resource requirements or prediction accuracy; The vehicle conditions include available resources of the vehicle; The model matching unit is further configured as: Based on the matching degree between the available resources of the vehicle and the restriction conditions of each model, at least a part of the models among the multiple different audio-breathing detection models and video-breathing detection models that meet the resource matching conditions are determined as dynamic switching candidate models, wherein the dynamic switching candidate models include the at least one breathing detection model.

5. The apparatus of claim 4, wherein the breathing detection unit is further configured to: for each current frame in the plurality of audio signal frames and the plurality of video signal frames, when the confidence of the current frame is greater than or equal to a confidence threshold condition, select an audio-breathing detection model or a video-breathing detection model corresponding to the type of the current frame from the dynamic switching candidate models; Processing the current frame using the selected audio-breathing detection model or video-breathing detection model to generate a classification prediction of one of exhalation, inhalation, or pause; The first breathing detection sequence includes classification predictions corresponding to the plurality of audio signal frames and the second breathing detection sequence includes classification predictions corresponding to the plurality of video signal frames.

6. The device of claim 5, wherein the detection fusion unit further comprises: a time alignment module configured to time-align the first breathing detection sequence with the second breathing detection sequence based on the sliding time window and the timestamp; a fusion module configured to fuse the first breathing detection sequence and the second breathing detection sequence aligned in time to generate a fused detection sequence; The decision module is configured to perform a smoothing process on the fused detection sequence based on a predetermined decision mode to generate the breathing classification detection sequence.

7. The apparatus according to any one of claims 1 to 6, wherein the detection fusion unit fuses the first breathing detection sequence with the second breathing detection sequence based on at least one of the audio confidence and the video confidence and the predicted probability of the at least one breathing detection model.

8. The apparatus of claim 6, wherein the smoothing process performed by the decision module comprises: Based on the decision mode, a portion of the classification prediction in the fusion detection sequence is reclassified.

9. The apparatus of claim 8, further comprising: The breathing segment generation module is used to determine which segment of exhalation, inhalation and breath holding each continuous frame in the breathing classification detection sequence represents to generate a breath segment sequence, The statistics module is used to count the length of each segment.

10. The apparatus according to any one of claims 1 to 6, wherein the audio signal comprises background music excluding the guiding music provided by the pranayama guiding course and the breathing sound of the user in the vehicle.

11. A breath evaluation device, comprising: The breathing detection device according to any one of the preceding claims 1-10, configured to generate a breathing classification detection sequence based on the collected audio signal and video signal, wherein the breathing classification detection sequence is used to form a breath segment sequence; A breath receiving module, used to receive the breath segment sequence generated in response to the breath regulation guidance course, the sequence including exhalation, inhalation and breath holding segments; A breath index scoring module, configured to determine a plurality of breath characteristics within the period of time based on the breath segments; The comprehensive evaluation module is configured to generate a multi-dimensional index representing breathing quality based on the multiple breath characteristics.

12. The device of claim 11, wherein the plurality of breath characteristics comprises: A breath attainment score indicating the ratio of the average duration of the breathing segments to the expected average duration; Breath duration, which indicates the ratio of the number of times the duration of the exhalation and inhalation segments exceeds the threshold to the target number; Breath stability, which indicates the variance in the duration of the exhalation and inhalation segments; and The breath positivity indicates the ratio of the number of exhalation segments whose duration continuously increases or does not decrease in the breath detection sequence to the total number of exhalation segments, and the ratio of the number of inhalation segments whose duration continuously increases or does not decrease in the breath detection sequence to the total number of inhalation segments.

13. The apparatus of claim 12, wherein the breath index scoring module further comprises a decay determination module configured to determine a decay factor based on a duration of a breathing pause in the sequence of breath segments within the time period; The breath index scoring module outputs the plurality of breath features attenuated by the attenuation factor.

14. The device of claim 12 or 13, wherein the comprehensive evaluation module is further configured to: A pattern diagram is constructed based on the breath continuity, breath stability, breath compliance, and breath positivity to indicate breath health; Calculating emotional stability based on a ratio of the pattern graph to a reference pattern graph; Determining the breath regulation stability indicating the changing trend of breath regulation by using the function of breath continuity, breath stability and breath positivity; as well as Using the change in the breathing stability, determining the breathing potential indicating the predicted breathing trend, The multi-dimensional features include breath health, emotional stability, breath stability and breathing potential.

15. A vehicle control linkage device, comprising: The breath evaluation device according to any one of claims 11 to 14; The control module is used to select a vehicle control linkage mode according to the multi-dimensional characteristics from the breath evaluation device to realize linkage control of multiple functional units of the vehicle, wherein the functional unit dials at least one of the atmosphere light, lighting effect, air conditioning, and sound effect.

16. A health assessment device, comprising: A deep breathing guidance module, providing a deep breathing course guidance to the user through at least one output medium, wherein the medium includes one or more of text, video and audio; The breathing detection device according to any one of the preceding claims 1-10, configured to generate a breathing classification detection sequence based on the audio signal and the video signal during the deep breathing course guidance, wherein the breathing classification detection sequence represents the breathing breath of the user; a breathing detection module configured to evaluate the user's breathing ability based on the matching degree between the user's breathing breath and the preset requirements of the course guidance during the provision of the deep breathing course guidance; A health assessment module is configured to determine the user's health based on the breathing ability.

17. The device of claim 16, further comprising a physical sign monitoring module configured to detect the user's physiological signs in real time during the provision of the deep breathing course guidance; The health assessment module is further configured as follows: determining the user's health based on the pranayama ability and the physiological signs; Calculate the health improvement degree based on comparing the changes in health before and after the deep breathing course guidance; The deep breathing course guidance module is instructed to update the deep breathing course guidance based on the health level improvement.

18. The apparatus of claim 17, wherein the health comprises: emotional stability based on breathing rate and heart rate; The breath duration, breathing regulation and breath regulation initiative determined based on the exhalation and inhalation duration characteristics of the user's breath; Cardiopulmonary level based on respiratory rate, heart rate, and blood oxygen saturation; The health detection module determines the health level based on emotional stability, breathing regulation, breath duration, breathing initiative and cardiopulmonary level using a predetermined health pattern diagram.

19. A course recommendation device, comprising: The breathing detection device according to any one of the preceding claims 1-10, configured to generate a breathing classification detection sequence based on the audio signal and the video signal during the first training guidance course; a course evaluation module, configured to evaluate the user's training status under the first training guidance course based on the breathing classification detection sequence; The personalized recommendation module is configured to provide a recommended ranked list of multiple breathing guidance courses based on multiple matching patterns and user characteristics, wherein the ranking represents the matching degree between the user characteristics and the alternative breathing guidance courses in the breathing guidance course library, wherein the user characteristics include the training status. The comprehensive recommendation module is configured to fuse the multiple ranked lists to provide a second training guidance course.

20. The course recommendation device according to claim 19, further comprising: A feature extraction module, configured to receive the training state and obtain the user's preferences and interests in the course to generate the user features; The training status includes at least one of a health status, a course completion, a pranayama potential, and a course acceptance, wherein the course acceptance is related to a change trend of a user's physiological signs, wherein the health status is determined based on the user's physiological signs; The matching mode includes a user interest mode and a user capability mode.

21. The course recommendation device according to claim 20, wherein: The pranayama course library includes a guide word library, a background music library, and a micro-animation library. The selected alternative course includes one or more of the guide words, background music, and micro-animation. Wherein the pranayama course library further includes a course template, which defines the course characteristics of each alternative pranayama course; The personalized recommendation unit processes the user characteristics and the course characteristics according to the multiple matching modes to generate a recommended ranking list of the multiple breathing courses.

22. The course recommendation device according to claim 21, wherein the personalized recommendation module is further configured to: Generate a ranked list of the most popular courses based on big data statistics of online users; Generate a similarity recommendation list of similar users based on user portrait similarity; as well as Generate a recommended list for the day based on the environment library; The comprehensive recommendation module integrates the most popular course ranking list, the similarity recommendation list and the today's recommendation list based on the weights determined by the user's feedback on the historically recommended courses.

23. A health assessment device, comprising: The real-time detection module is used to detect the user's breathing characteristics and physiological signs within at least one time period, including: The breathing detection device according to any one of the preceding claims 1-10 is configured to generate a breathing classification detection sequence of the user by processing audio data collected by a microphone and a video stream captured by a camera, wherein the breathing classification detection sequence indicates breathing features including exhalation, inhalation, and breath holding; and a vital sign detection module, configured to process the video stream collected by the camera to generate the physiological signs, including at least one of heart rate, blood oxygen saturation and blood pressure; A breathing assessment module, which determines the breathing quality based on the breathing characteristics, and determines the user's breathing completion by comparing the breathing characteristics with preset guidance requirements; The health assessment module is used to assess the user's breathing potential within the at least one time period based on the breathing completion degree, breathing quality and physiological signs.

24. The apparatus of claim 23, wherein the pranayama assessment module further comprises: A first evaluation unit is configured to calculate the duration of the exhalation segment, the inhalation segment, and the pause segment contained in the frame sequence based on the frame sequence to determine the breathing quality; A second evaluation unit is configured to compare the exhalation segment, the inhalation segment, and the pause segment included in the sequence with the preset breathing guidance requirements to determine the user's breathing completion degree; The health assessment module further determines the breathing potential and health indicators based on the breathing completion, breathing quality and physiological sign signals, heart rate and blood oxygen saturation.

25. The apparatus of claim 24, further comprising a guide language matching module configured to provide breathing scenario guidance based on pranayama potential, pranayama completion index and / or health index.

26. The device of claim 25, wherein the guide matching module further comprises: The scene matching unit determines the best matching scene based on the completion and potential of pranayama; The timing detection unit determines the timing of inserting the best matching scene into the preset guide.

27. A breathing detection method, comprising: evaluating audio confidence of breathing detection based on the audio signal using the collected audio signal; Using a video signal stream collected from inside the vehicle to evaluate the video confidence of the breathing detection based on the video signal; matching at least one breath detection model adapted to perform breath detection based on vehicle conditions; Based on the audio confidence and the video confidence, calling the at least one breathing detection model to process the audio signal and the video signal to generate a first breathing detection sequence corresponding to the audio signal and a second breathing detection sequence corresponding to the video signal; The first breathing detection sequence and the second breathing detection sequence are fused to generate the breathing classification detection sequence.

28. A breath evaluation method, comprising: Generate a breath classification detection sequence for forming a breath segment sequence, wherein the breath classification detection sequence is The method according to claim 27 is generated based on the collected audio signal and video signal; receiving a sequence of breath segments generated in response to a guided pranayama session, the sequence comprising exhalation, inhalation, and breath-holding segments; Determining a plurality of breath characteristics within the period of time based on the breath segments; A multidimensional index representing breathing quality is generated based on the plurality of breath characteristics.

29. A health assessment method comprising: Providing a deep breathing course guide to the user through at least one output medium, wherein the medium includes one or more of text, video, and audio; A breathing classification detection sequence is generated based on the audio signal and the video signal during the deep breathing course guidance, wherein the breathing classification detection sequence is Generated according to the method of claim 27, the breath classification detection sequence represents the breath of the user; evaluating the user's breathing ability based on the degree of matching between the user's breathing during the deep breathing course guidance and the preset requirements of the course guidance; The user's fitness is determined based on the pranayama ability.

30. A course recommendation method, comprising: generating a breathing classification detection sequence based on the audio signal and the video signal during the first training induction course, wherein the breathing classification detection sequence is generated according to the method of claim 27; evaluating the user's training status under the first training guidance course based on the breathing classification detection sequence; Based on the multiple matching modes and the training status, a recommended ranked list of multiple breathing guidance courses is provided, wherein the ranking represents the matching degree between the user characteristics and the alternative breathing guidance courses in the breathing guidance course library, and the user characteristics include the training status. The plurality of ranked lists are merged to provide a second training guide course.

31. A health assessment method, comprising: Detecting the user's breathing characteristics and physiological signs during at least one time period, including: The audio data collected by the microphone and the video stream collected by the camera are processed to generate a breathing classification detection sequence of the user, wherein the breathing classification detection sequence is Generated according to the method of claim 27, the breathing classification detection sequence indicates breathing characteristics including exhalation, inhalation and breath holding; Processing the video stream captured by the camera to generate the physiological sign, including at least one of heart rate, blood oxygen saturation and blood pressure; Determining the breathing quality based on the breathing characteristics, and determining the user's breathing completion degree by comparing the breathing characteristics with preset guidance requirements; The user's pranayama potential during the at least one time period is evaluated based on the pranayama completion degree, breathing quality and physiological signs.

Images