A control method and control device of an intelligent terminal device

Abstract

The application relates to a control method and a control device of an intelligent terminal device. The device comprises a processor configured to receive sound information collected by the terminal device, extract breathing frequency information from the sound information, and reduce the playing volume of a current function module by a first limited volume difference after judging that a user enters a sleep state based on the breathing frequency information. If the intelligent terminal device does not detect any input instruction information within a first limited time after the playing volume is reduced, the playing volume is gradually reduced in a stepped volume manner. When the playing volume is reduced to a shutdown threshold range, the function module is directly shut down. If a micro change in stimulation information of the function module generated by the user has no feedback, control information is sent to the function module according to preset control instructions, so that the function module is controlled in a preset manner. The application only collects breathing information to judge that a user enters a sleep state, and then closes a specified function.

Description

[0001] The present application is a divisional application of the application with the application number 202110701370.8, the application date of June 25, 2021, the application name of a control system and method based on sleep perception, and the application type of invention. TECHNICAL FIELD

[0002] The present application relates to the technical field of sleep perception, and in particular to a control method and control device of an intelligent terminal device. BACKGROUND

[0003] Before going to sleep, the number of people using mobile phones to listen to music, videos, etc. is increasing, but the situation of falling asleep unknowingly is common. At this time, the music and videos are still in the playing state. This makes the intelligent terminal consume a large amount of electric energy, and even wakes up the user again. Therefore, it is necessary to automatically close the music, video, and other application programs after the user enters the sleep state.

[0004] Currently, the way of closing music, video, etc. is generally controlled by timing. However, some users do not enter sleep at the timing time, and some users enter sleep quickly and are woken up again by the audio and video that have not been closed. How to close the audio and video application functions according to sleep perception is a technical problem that has not been solved.

[0005] The prior art even relies on intelligent sensing mattress, smart watch, and other devices to detect the sleep state of the user, so that the terminal device indirectly obtains the sleep state of the user. However, this requires the user to configure an intelligent sensing mattress, a smart watch, and other supporting devices, which increases the cost of the intelligent device sensing the sleep state of the user, and is not conducive to the user to close the audio and video based on sleep perception in any sleeping place.

[0006] For example, Chinese patent CN111772583A discloses a sleep monitoring and analysis method, device, and electronic equipment for an intelligent sound box, relating to the technical fields of artificial intelligence, computer vision, and voice interaction. The specific implementation scheme is as follows: obtaining multiple frames of images and audio collected by the intelligent sound box when the user is sleeping; generating an image curve of the user's sleep according to the multiple frames of images; generating an audio curve of the user's sleep according to the audio; obtaining the sleep quality information of the user according to the image curve and the audio curve; and outputting the sleep information of the user, which includes at least one of the image curve, the audio curve, the sleep quality information of the user, the image when the user is sleeping, and the audio when the user is sleeping. This invention enables the user to know the detailed change process of the sleep condition and the external factors affecting sleep, thereby helping the user to better find a method to improve sleep and greatly improving the user's experience. However, the problem of when to close the audio and video without affecting the sleep quality of the user has not been solved.

[0007] For example, Chinese patent CN109920532B discloses a control method for a medical wearable device with sleep function. The VR device is worn on the head of the user, and the camera of the device corresponds to the eyes. The camera of the VR device recognizes a series of movements of the eyes and matches the feature data. The matched feature data in S2 is processed and converted into a control signal to send to the processing center. For different scene states in S3, the VR device plays corresponding video materials. When the sleep signal is matched in S2, the processing center controls the VR device to turn off. The application uses the eye capture system to recognize a series of movements of the user's eyes and match the feature data in the feature library. Through the recognition and matching of the excited eye feature, the calm eye feature, the manic eye feature, the sad eye feature and the sleep eye feature, different qualities are adjusted to help the user enter sleep in time, and the application has strong practicality. Although the device can turn off the audio and video when the user's sleep state is detected, the user must wear the VR device, which will undoubtedly affect the user's sleep, and the device does not have the convenience and convenience of automatically turning off the audio and video function based on the sleep state of the user's portable intelligent device.

[0008] For example, Chinese patent CN105407217A discloses a method for playing music on a mobile terminal. The method includes detecting that the mobile terminal is playing music, and starting a sound input device. The user's breathing frequency is obtained through the sound input device. The user's rest state is determined according to the obtained breathing frequency, and the rest state includes a sleep state and a wake state. If the rest state is a sleep state, the music playing mode is switched to a sleep playing mode. The application also discloses a mobile terminal for implementing the above method. According to the different sleep states of the user, the music playing is managed, making the mobile terminal more intelligent and user-friendly, and greatly improving the user's experience of playing music on the mobile terminal. However, in this technical solution, the breathing information is obtained through the sound input device. According to common sense, in the daily use environment, there are many noise disturbances around ordinary users (this is one of the main reasons why some existing mobile terminals have call noise reduction function). There are a large amount of sound signals in the sound collected by the mobile terminal, and the sound input device cannot directly obtain the breathing information related to sleep from the large amount of sound signals. Therefore, there is a significant error between the breathing frequency obtained from the sound signal collected by the mobile terminal and the user's real breathing frequency, and the sleep state result obtained by the frequency threshold method based on the error is also inaccurate.

[0009] For example, Chinese patent CN107743289A discloses a smart home scene in which a smart sound box control method is disclosed. The invention first determines whether the current time is sleep time based on the user's biological clock information, and actively determines whether the user is in a sleep state only during sleep time, and passively determines whether the user is in a sleep state outside of sleep time, thereby reducing the number of determinations, improving the efficiency of voice prompts, and improving user experience. In addition, in the case where the user does not feedback or does not correctly feedback the voice prompt information, it is determined that the user is entering sleep, at which time the volume of the sound box is gradually reduced to continuously adjust to the user's needs for environmental sound during the sleep process, thereby helping the user to enter sleep, which is a humanized improvement. Finally, in order to avoid the inconvenience of repeatedly setting and adjusting the fixed cycle detection time by the user, the invention sets a random time within a specific time range, thereby improving the efficiency of cycle detection and further improving user experience. However, this technical solution first needs to be limited to the time corresponding to the biological clock in order to perform the adjustment and shutdown process of the sound. In addition, the sound will be in normal working condition, and the feedback information of the user cannot affect the control process of the sound. However, the biological clock, also known as the physiological clock, is an invisible "clock" in the body, which is actually the internal rhythmicity of the life activities of the organism, determined by the time structure sequence in the organism. Biological clock usually has certain regularity, usually belongs to default fixed setting, and does not have randomness like sound information. This technical solution first limits the adjustment and shutdown control process of the smart sound box to a fixed time period, and in fact the sleep state of the user within this time period is not determined, and normal people usually cannot guarantee to enter sleep state accurately at the specified time point.

[0010] Therefore, how to enable a smart terminal device such as a mobile phone to directly determine the sleep state by sensing the breathing sound of a human being, and then to shut down the audio and video functions, is still a technical problem that the prior art has not solved.

[0011] In addition, on the one hand, there are differences in understanding of the technical personnel in the art, and on the other hand, the inventors have studied a large number of literatures and patents when making the invention, but due to the limited space, all the details and contents are not listed in detail, which does not mean that the invention does not have these characteristics of the prior art, on the contrary, the invention has all the characteristics of the prior art, and the applicant reserves the right to add relevant prior art in the background art. SUMMARY

[0012] In the prior art, the terminal intelligent device must determine the sleep state of the user by means of the collected blood pressure information, pulse information and respiratory information, so that the intelligent terminal device such as a mobile phone without the functions of collecting pulse information and sleep information must be connected to the corresponding functional device to monitor the sleep state and perform individualized control, which makes the user have to carry multiple functional devices, otherwise the sleep state cannot be monitored. For example, the existing smart watch needs to maintain a continuous Bluetooth signal connection with the smart phone to realize the monitoring of the sleep state. This makes the user who only has a smart phone unable to accurately determine the sleep state by collecting respiratory information and close the played audio and video functions. Moreover, when the user uses the intelligent terminal device to play audio or video, a large amount of noise affecting the respiratory information is generated, which further increases the difficulty of respiratory information recognition and extraction.

[0013] Therefore, how to accurately determine that the user enters the sleep state and closes the played audio and video only by the respiratory sound is a technical problem that has not been solved at present. The current filtering of the respiratory sound to obtain accurate respiratory information is also a technical problem that has not been solved.

[0014] The control system provided by the present application can make the intelligent terminal device only collect respiratory information, process the respiratory information to obtain individualized respiratory sound, determine that the user enters the sleep state according to the respiratory sound, and then close the specified function. The present application makes the user be able to realize the control closing through the intelligent terminal device with only sound collection function, simplifies the use conditions of user control, and makes the user realize the control of audio and video during sleep through the simple intelligent terminal device such as smart phone, desktop computer, notebook computer, tablet computer and other intelligent terminal devices with only microphone.

[0015] In view of the deficiencies of the prior art, the present application provides a control system based on sleep perception, which comprises at least one processor, the processor is connected with at least one terminal intelligent device, and the processor is configured to: receive sound information collected by at least one audio collection module; extract at least one respiratory frequency information from the sound information, identify the sleep state based on the extraction model and the respiratory frequency information; and send preset instruction information to at least one associated functional module in the specified sleep state. The control system of the present application eliminates the defect that the object characteristic information needs to be collected by a third device or itself to accurately determine the sleep state, so that the intelligent terminal device can determine the sleep state only by collecting the respiratory sound, and can determine whether the user is asleep by perceiving the stimulation micro-change in sleep, reduce the error of inaccurate detection, so that the control system can accurately determine the sleep state and close the functional module according to the preset instruction.

[0016] Preferably, the processor is further configured to, in the case of determining that the user enters the sleep state, send a stimulation information micro-change instruction to the running at least one functional module to control the functional module to test the sleep perception of the user in a manner of causing the stimulation information to micro-change.

[0017] Preferably, the processor is further configured to, in the case of no feedback of the user to the micro-change of the stimulation information of the functional module, send control information to the functional module according to the preset control instruction, so as to control the functional module in a preset manner.

[0018] Preferably, the processor is further configured to, in the case of feedback of the user to the micro-change of the stimulation information of the functional module, restore the micro-changed stimulation information of the controlled functional module.

[0019] Preferably, the processor is further configured to control the functional module in a trend of reducing the stimulation intensity of the stimulation information, so as to form a sleep perception test that does not hinder sleep.

[0020] Preferably, the processor is further configured to, in the case of the user entering the sleep state and no feedback to the micro-change of the stimulation information of the currently running functional module, form and update the personalized frequency sample based on the current breathing frequency information.

[0021] Preferably, the processor is further configured to, after the user enters the sleep state, the feedback information after the stimulation information of the currently running module micro-changes at least includes movement information, input instruction information and / or action information of the intelligent terminal device.

[0022] The application also provides a sleep perception-based control method, which at least includes:

[0023] Receiving sound information collected by at least one audio collection module;

[0024] Extracting at least one breathing frequency information from the sound information,

[0025] Identifying the sleep state based on the extraction model and the breathing frequency information;

[0026] Sending preset instruction information to at least one associated functional module in the specified sleep state.

[0027] Preferably, the method at least includes: in the case of determining that the user enters the sleep state, sending a stimulation information micro-change instruction to the running at least one functional module to control the functional module to test the sleep perception of the user in a manner of causing the stimulation information to micro-change.

[0028] Preferably, the method comprises: in the absence of feedback of the micro-change of the stimulation information of the user to the functional module, the processor sends control information to the functional module according to the preset control instruction, so as to control the functional module in a preset manner. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a logic diagram of the sleep-aware control system of the present application.

[0030] LIST OF REFERENCE NUMERALS

[0031] 10: breath acquisition module; 20: processor; 21: database; 30: functional module. DETAILED DESCRIPTION

[0032] The present application will be described in detail below with reference to the accompanying drawings.

[0033] The present application provides a sleep-aware control system and method, and can also provide a breath frequency self-defined control system and a portable intelligent terminal device. The sleep-aware control system of the present application can exclude one or more noise and vibration information while retaining the breath information, so as to extract accurate breath frequency information.

[0034] The intelligent terminal device in the present application refers to a use device with wireless access to the Internet, capable of data processing, display, and execution of corresponding data functions. The intelligent terminal device is, for example, a smart phone, an Ipad, a player, a computer, and the like, which can operate based on data instructions. The intelligent terminal device can be equipped with a sound acquisition device itself, or can be connected to a sound acquisition device to acquire the breath sound of the user. The sound acquisition device is, for example, a microphone.

[0035] Preferably, the microphone of the intelligent terminal device is coupled with a sound amplification device, which can further clearly acquire the breath information of the user.

[0036] The processor of the present application refers to one or more general-purpose devices, such as a microprocessor, a central processing unit (CPU), an application specific integrated chip, and the like.

[0037] The breath information of the present application is breath sound information, such as wheezing sound and snoring sound. The breath frequency information refers to the frequency of a user completing one breath, and the period refers to the time for a user to complete one breath.

[0038] The sleep-aware control system of the present application can be installed and run in the intelligent terminal device or establish a data connection with the intelligent terminal device.

[0039] The difference between a user's breathing and other noises is that the sound of breathing is periodic and regular, and the breathing frequency is within a specific range. For example, the breathing frequency varies between 0.15 and 0.4 times per second.

[0040] The acquisition module includes a sound acquisition module and an acceleration sensor. The acquisition module is part of the smart terminal device and can collect corresponding information based on the instructions of the processor.

[0041] Preferably, the control system of the present invention further includes a database. The database is used to store the collected respiratory information and time, store instructions associated with the neural network, and store process data for extracting respiratory frequency from the respiratory information. The processor reads data from the database, or the processor sends processed data to the database for storage.

[0042] The database is preferably a memory. Memory refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or other computer-readable instructions. Memory includes, but is not limited to, RAM, ROM, flash memory, or any other suitable storage device. The memory can be coupled to the processor or provided in a signal connection with the processor.

[0043] Preferably, the processor may also include a memory control module, an input / output (I / O) controller, and a communication interface, each of which may be interconnected via a communication infrastructure.

[0044] Examples of communication infrastructure include, but are not limited to, communication buses (such as Industry Standard Architecture (ISA), Peripheral Component Interconnect (PCI), PCI Express (PCIe), or similar buses) and networks.

[0045] An I / O controller refers to any type or form of module capable of coordinating and / or controlling the input and output functions of a computing device. An I / O controller can control or facilitate the transfer of data between one or more components of a processor.

[0046] A communication interface represents any type or form of communication device or adapter capable of facilitating communication between a processor and one or more additional devices.

[0047] Figure 1 This is one embodiment of the control system of the present invention. Figure 1 As shown, the sleep sensing-based control system includes at least a processor 20. The processor 20 establishes connections with a plurality of acquisition modules 10, a database 21, and a functional module 30 to enable the transmission of data information and instructions. The database 21 can be driven and stored by a storage drive such as a floppy disk drive, a tape drive, an optical disk drive, a flash drive, or the like.

[0048] Preferably, the intelligent terminal device is provided with a terminal display module. The terminal display module is connected with the processor in a wired or wireless manner, and the terminal display module establishes a data connection with the acquisition module. The terminal display module sends control instructions to the functional module. The terminal display module can display the collected mobile information, respiratory rate information, sleep time, first time, second time, stimulation information micro-change mode, closing or pausing time and the like. The terminal display module can display information in various forms such as charts or curves.

[0049] The functional module of the present application refers to a module provided in the intelligent terminal device which can adjust and start and stop various execution functions based on the instructions of the processor. The functional module is, for example, an audio player, a video player, a start and stop module, a lamp adjustment and start and stop module, start and stop and adjustment of various application modules and the like.

[0050] As shown in Figure 1 The control system based on sleep perception of the present application at least includes a processor, which is configured to:

[0051] receive sound information collected by at least one audio acquisition module;

[0052] extract at least one respiratory rate information from the sound information,

[0053] identify a sleep state based on the extraction model and the respiratory rate information;

[0054] send preset instruction information to at least one associated functional module in a specified sleep state.

[0055] The respiratory rate information extraction model in the present application, which can be simply referred to as an extraction model, is established based on a deep learning model and is used to extract respiratory rate information. The deep learning model is, for example, a convolutional neural network model.

[0056] Preferably, an analog filter is further provided in the processor, which is used to amplify the received respiratory information to avoid missing the breathing sound with small sound.

[0057] Preferably, the acceleration sensor in the intelligent terminal device is connected with the processor and collects the mobile signal of the intelligent terminal device. The mobile signal includes the moving speed of audio and video in three-dimensional space.

[0058] The processor extracts the respiratory rate information from the amplified sound information. The processor eliminates noise and audio and video sound without periodicity from the sound information to obtain breathing information with periodicity.

[0059] Preferably, the processor is further provided with an analog gain module for analog gain between 10 and 100 times to cover the range of respiration. Preferably, the processor is further capable of connecting hardware for realizing gain, such as a filter to limit the frequency response of the signal within the range of respiration frequency.

[0060] Preferably, the analog gain module sends the filtered respiration information to the extraction module.

[0061] The linear filtering of the sound information by the analog filter in the processor is in the form of:

[0062]

[0063] The signal quality y[n] of the linear filtering is determined by the error sequence e[n] = d[n] - y[n]. Wherein, n represents the frequency of the mixed sound, and m represents the sound beyond the specified frequency range. ω m represents the weight, and x represents the collected data value.

[0064] Preferably, the weight is selected in the form of obtaining the minimum mean square error:

[0065] E{e 2 [n]} = E{d[n] - y[n 2}

[0066] The selected weight of the linear filtering is calculated in the form of:

[0067]

[0068] According to the orthogonality principle, when the error e[n] and the data value are zero, we get:

[0069] E{x[n-k](d[n] - y[n])} = 0, k = 0, 1, …, M-1

[0070] At this time, the effect of the linear filtering is best.

[0071] Preferably, the weight module in the processor can also obtain the correct weight through training based on the training algorithm.

[0072] After the processor filters the obtained original sound information to eliminate the noise caused by audio and video or other sounds, the respiration information formed after filtering is extracted for respiration frequency. At this time, the respiration information is not completely noise-free, but also contains a small part of noise, so the extraction module needs to further extract based on the deep learning algorithm.

[0073] Preferably, the processor extracts the respiration frequency information in the form of:

[0074] establishing a respiration frequency information extraction model,

[0075] comparing a first respiratory rate extracted by the respiratory rate information extraction model with a verifiable respiratory rate of the first test subject to calculate an error;

[0076] The front-end parameters of one or more layers of the extraction model are reversely adjusted based on the calculation error to improve the prediction accuracy of the extraction model.

[0077] After acquiring the respiratory frequency information, the processor determines whether the user has entered a sleep state based on the respiratory frequency information, and even determines which sleep stage the user has entered.

[0078] Preferably, the respiratory information signal has the following relationship:

[0079]

[0080] After the transmission delay information is introduced, the breathing information signal is expressed as:

[0081]

[0082] Finally, the relationship between the mixed collected respiratory information received by the acquisition module is:

[0083]

[0084] Wherein, the frequency f b It is directly related to the distance between the object and the acquisition module. The distance b is related to the speed of the object. Frequency f b and distance b are calculated using the Fourier transform algorithm.

[0085] For the user's vital signs, the frequency f b It can be used to measure the user's distance interval, and the distance b can reflect the displacement change of the user's sound source.

[0086] Preferably, before extracting the respiratory frequency, the processor can remove noise from the respiratory information data, for example, by normalizing it using the equation f(x)=(x-μ) / σ, where μ represents the mean value of the waveform and σ represents the standard deviation.

[0087] By removing noise from the received respiratory information, other components and unwanted frequencies in the sound information signal can also be removed. For example, the respiratory rate varies between 0.15 and 0.4 per second. Other frequencies outside the respiratory rate range can be eliminated.

[0088] In the extraction model of the present invention, the convolutional neural network model can be a two-layer convolution model, and can also be extended to more than two layers.

[0089] Each volume layer includes a one-dimensional convolution layer and a pooling layer. The one-dimensional convolution layer effectively derives notable features from a short segment of the entire data set, and the position of the features in the segment has no high correlation. The one-dimensional convolution layer can derive any type of signal data within a fixed length period, thus improving the efficiency of data screening and derivation. Therefore, the convolutional neural network uses one-dimensional or two-dimensional convolution layers at each layer for training. After the convolution layer data processing, the pooling layer is used for pooling processing, which can reduce the complexity of the output and prevent data overfitting.

[0090] Preferably, the size of the pooling layer is set to 3, indicating that the size of the output matrix is only one third of the input matrix. The pooling layer is used to reduce the input size by mapping the size of the defined window to a single result by taking the maximum value of the elements in the window.

[0091] The processor is configured to:

[0092] The input three-dimensional waveform information used to train the extraction model is discretized, thereby improving the signal-to-noise ratio.

[0093] Data fitting of the respiratory information can reduce the influence of small fluctuations in the data on the extraction model, which are generally noise.

[0094] Preferably, the processor is further configured to:

[0095] The data output by the convolution layer is directed to an average pooling layer. The average pooling layer is another pooling layer used to avoid overfitting. The average pooling layer transforms the matrix output by the convolutional network into a single vector.

[0096] In the present application, the processor adjusts and controls the function modules according to the preset instructions after determining that the user enters the sleep state.

[0097] Preferably, the extraction model in the processor compares the extracted respiratory rate information with the respiratory rate samples, and on the basis that the respiratory rate samples and the sleep state have a corresponding correlation, the processor obtains the corresponding sleep state according to the respiratory rate information.

[0098] The respiratory rate samples in the present application include initial frequency samples and personalized frequency samples. The initial frequency samples are set based on the sample set data of the correlation between the respiratory rate characteristics of the general public and the sleep state.

[0099] The personalized frequency samples in the present application are formed by personalized adjustment of the initial frequency samples based on the personalized respiratory characteristics of the user, thereby forming information on the correlation between the personalized frequency samples and the sleep state.

[0100] Preferably, the initial frequency sample in the extraction model and the sleep state association information are preset, which can be provided by a third-party data platform or learned based on a deep learning model according to sleep experiments of a plurality of sample populations.

[0101] The initial frequency sample and the sleep state association relationship are as follows:

[0102] In different sleep stages, the breathing frequency of the user's sleep state is significantly different. In the rapid eye movement sleep stage (REM), the brain wave changes rapidly, the sleeper often appears the phenomenon of rapid eye jumping and dreams in this stage, so the breathing frequency of the sleeper is usually unstable in this stage. In the light sleep stage (Light Sleep), the eye jumping phenomenon stops, and the breathing frequency of the sleeper tends to be stable. In the deep sleep stage (Deep Sleep), the breathing frequency of the sleeper is further slowed down. Therefore, the size of the breathing frequency is different with the sleep stage of the sleeper.

[0103] The sound collection module collects the breathing sound data at a sampling frequency of 8KHz. The breathing sound data is divided into N groups of data according to every 400 sample points. N represents the number of groups of sample points, and the sampling frequency is 8kHz. The time interval of 400 sample points is 0.05s.

[0104] After the audio and video sounds are filtered by linear filtering to form the breathing information, the breathing frequency information is extracted according to the extraction model.

[0105] The present application does not need to monitor the sleep state of the user throughout the process, but only needs to determine that the user enters the initial stage of the sleep state to close the specified function module. However, how to accurately determine that the user enters the sleep state only by analyzing the breathing sound, and in a way that does not affect the user's sleep feeling to close the playing function module is important, and is difficult to achieve by the prior art.

[0106] When entering the sleep state and closing the function module, especially closing the audio and video that the user is watching or listening to, if the closing is too early, the user is just about to enter the sleep state, and closing the function module will cause the user to wake up due to environmental changes, which will have the opposite effect. If the closing is too late, the function of controlling the function module based on sleep perception is lost.

[0107] Based on this defect, the processor tests the perception level of the user in the sleep state in a test mode. The function module is controlled when the perception level of the user is reduced to a preset level. The present application tests the perception level of the user in the test mode to regulate and control the function module in a way that does not affect the user's sleep feeling, or even closes the function module.

[0108] The test mode in the present application tests the perception level of the user in the form of slight changes in the stimulation information. The stimulation in the present application refers to the sound stimulation, visual stimulation, play progress stimulation, play content change and other forms of stimulation presented by the intelligent terminal device when the user uses the function module. In view of the fact that the user of the present application is in a sleep state or close to a sleep state, the slight changes in the stimulation information in the present application refer to sensory stimulation that will not stimulate the user to wake up. The slight changes in the stimulation information of the present application are slight adjustments and changes to the stimulation already applied to the function module, and no new stimulation is added to avoid waking up the user.

[0109] Preferably, the implementation of the slight changes in the stimulation information of the processor at least includes the following.

[0110] After determining that the user enters the sleep state based on the breathing rate information, the processor first reduces the play volume of the current function module by a limited volume difference. In the case that the intelligent terminal device does not detect any input instruction information within the first limited time after the play volume is reduced, the processor gradually reduces the play volume in the form of stepped volume, and directly closes the function module when the play volume is reduced to the shutdown threshold range. The stepped reduction of the play volume will not bring new stimulation information to the user.

[0111] When the user has not yet completely entered the sleep state, the user's perception ability is strong, and the reduction of the play volume will inevitably bring a poor experience, and the user will move the intelligent terminal device or input related instructions or action signals to check. When the processor detects the movement signal collected by the acceleration sensor in the intelligent terminal device, or the action signal of the related input device, or the input instruction, the processor controls the function module to restore the play volume.

[0112] After the user enters the sleep state, the user's perception ability becomes weak, and the processor first reduces the play volume of the current function module by a limited volume difference. In the case that the user does not perceive, the intelligent terminal device will not be moved, or the input device will not be used to input the action signal to check the change of the function module, or the instruction will not be input, so the processor will not receive the movement signal collected by the acceleration sensor, or the action signal of the related input device, or the input instruction. At this time, the processor can secondly reduce the play volume. In the case that no interference information is still received within the second limited time, the processor closes or pauses the operation of the current specified function module.

[0113] In the present application, the action signal of the input device is, for example, a touch screen action signal, a mouse movement signal, a keyboard action signal, etc. The input instruction refers to any input instruction information.

[0114] After determining that the user enters the sleep state based on the breathing frequency information, the processor detects the user's perception by pausing the function module. For example, the playing content of the audio module and the video module is temporarily suspended. If no input instruction information is detected within a third limited time, the processor sends a preset control instruction to at least one function module. The control instruction includes a user preset instruction such as closing a specified function module, pausing, shutting down, etc.

[0115] If the movement information of the intelligent terminal device, any input instruction information, and the action signal of the input device are detected within the fourth limited time, the processor sends a running resuming instruction to the function module to be controlled.

[0116] Preferably, the processor detects the user's perception by pausing the function module at least once. Preferably, the processor detects the user's perception by pausing the function module twice or three times.

[0117] After detecting the user's feedback by pausing the function module once, the processor determines whether the user enters the sleep state based on the current breathing frequency information. The perception detection is repeated when the user enters the sleep state.

[0118] Preferably, after determining that the user enters the sleep state based on the breathing frequency information, the processor can also detect the user's perception by changing the playing content. Preferably, the processor changes the playing content in a way that is conducive to sleep. For example, the playing music content is changed to light music with no lyrics that is conducive to sleep, and the playing video content is changed to video content with a soothing atmosphere that is conducive to sleep, such as news videos, interview videos, and other videos that are not easy to cause sensory stimulation.

[0119] After changing the playing content, if no input instruction information or movement signal is detected within a fourth limited time, the processor sends a preset control instruction to at least one function module. The control instruction includes a user preset instruction such as closing a specified function module, pausing, shutting down, etc.

[0120] If the movement information of the intelligent terminal device, any input instruction information, and the action signal of the input device are detected within the fourth limited time, the processor sends a running resuming instruction to the function module to be controlled.

[0121] Preferably, when the processor controls the function module to pause, the processor can control the function module to display a static picture or a question. For example, the text content of the display picture is: the playing function will be closed soon, or a picture is used to cover the display picture. If the user does not feed back any signal within a fifth limited time, the processor determines that the user's perception is weak and enters the sleep state, and thus executes a preset control instruction.

[0122] The stimulation information micro-variation of the present application is not limited to the above-mentioned several ways, and the stimulation information micro-variation capable of achieving the same effect can be implemented.

[0123] The test mode of the present application can also be used to adaptively adjust the initial frequency sample of the breathing frequency to form a personalized individual frequency sample.

[0124] During the period of initial use of the control system, the processor can take the breathing frequency when no user feedback signal is received as the personalized frequency sample of the user after the control of the stimulation information micro-variation of the functional module.

[0125] With the extension of the use time of the user and the increase of the detection times, the processor can extract and update the breathing frequency information of the user when entering the sleep state when using each kind of functional module as the personalized frequency sample and store it, which makes the personalized frequency sample in the database more and more abundant. When the processor compares the extracted breathing frequency information with the personalized frequency sample, the accuracy of the judgment of whether to enter the sleep state will also be higher and higher, forming a positive cycle.

[0126] After the control system is turned on, the processor sends an instruction to the sound collection module to collect sound information and sends an instruction to the acceleration sensor to collect movement information. The processor receives the collected sound information and the movement information of the intelligent terminal device itself.

[0127] The processor of the present application can also be a cloud server connected with the intelligent terminal device through a network. The control system of the present application collects sound information and movement information by connecting with the intelligent terminal device without adding new hardware, and detects the perception of the user based on the stimulation information micro-variation of the functional module, so that the preset control of the functional module can be performed at the appropriate time to save energy consumption and improve the sleep quality of the user.

[0128] In the present application, the breathing information includes exhalation sound, inhalation sound and snoring sound. Snoring and breathing sound have some common characteristics, and the breathing sound and the snoring sound have similarity in waveform, both have periodicity, and the difference is that the average amplitude of the snoring sound is larger than that of the breathing sound, which is due to the fact that the snoring sound is usually larger than the breathing sound. The frequency distribution of the two is significantly different on the frequency spectrum, and the frequency distribution of the snoring sound is concentrated in the low frequency band of 0 to 1000 Hz, while the breathing sound is distributed in both low and high frequency bands, so the ratio of the low frequency band to the high frequency band can be used to judge whether the collected sound sample contains snoring sound in the breathing sound.

[0129] The current breathing sound occupies the dominant frequency range to judge the existence of snoring sound, and f represents the sound frame to be tested containing n sample points, si Let f represent the i-th data value after Fourier transformation, and let the ratio of the low frequency band to the high frequency band be represented as:

[0130]

[0131] When the user changes from normal breathing to snoring, the sound frequency band gradually tilts towards the low frequency, causing the value of A(f) to gradually increase. By analyzing snoring data of more than 100 different users, it is agreed that when the ratio of the low frequency to the high frequency gradually increases and A(f)>1.5, it means that the current user is in a snoring state, otherwise, it is in a normal breathing state. That is, when A(f)>1.5, it is determined that the user is snoring.

[0132] Preferably, the processor of the present application can distinguish snoring sound based on the above distinction. The processor of the present application can also be configured to:

[0133] In the case where the processor determines that the user enters the sleep state and detects snoring sound, the processor directly controls the function module according to the preset instruction without sleep perception detection of the user. For example, some users snore as soon as they enter the sleep state, at which time there is no need to detect the user's senses, and the processor can control the function module to be directly turned off or paused, avoiding the function module to continue to release the stimulating information to disturb the user's sleep.

[0134] In the case where the processor determines that the user enters the sleep state and does not detect snoring sound, the processor detects the user's perception by detecting the micro-change of the stimulating information, and controls the function module in the case where the user's perception is weak.

[0135] Preferably, the sleep perception level of the present application can be divided based on the user's feedback to the micro-change of the stimulating information.

[0136] For example, the processor determines that the user enters the sleep state based on the breathing frequency information. When the stimulating information is sound change, in the first ladder range of the sound intensity decrease, if the user has no feedback information or feedback action, the user's sensory level is level one. When the user is detected to enter the sleep state for the second time, in the second ladder range of the sound intensity decrease, if the user has no feedback information or feedback action, the user's sensory level is level two, i.e. the senses are further weakened. The second ladder range is larger than the first ladder range. The first ladder range is 0-5 degrees, and the second ladder range is 5-10 degrees.

[0137] By analogy, the user's perception level can be divided into multiple levels, and the specific division range and level setting can be flexibly adjusted as needed.

[0138] Preferably, the processor judges the user's perception level by using the stimulus information to mix the micro changes. For example, for the playing video module, the processor first detects the user's perception by using the way of covering the playing picture and not changing the sound. If the user has no feedback to the first micro change of the stimulus information, the perception level belongs to the first level.

[0139] After the first micro change of the stimulus, the processor detects the user's perception level for the second time by controlling the sound of the video module, for example, changing the intensity of the sound. If the user has no feedback to the second micro change of the stimulus information, the perception level belongs to the second level.

[0140] After the second micro change of the stimulus, the processor detects the user's perception level for the third time by changing the sound content of the playing content. For example, the sound content of the video is changed to the song content. If the user has no feedback to the third micro change of the stimulus information, the perception level belongs to the third level.

[0141] Obviously, in the sensory intensity, the first level of the sensory level corresponds to the first level of the sensory intensity, which is greater than the second level of the sensory intensity, and the second level of the sensory intensity is greater than the third level of the sensory intensity. When the user's sleep perception level enters the second or third level of the sensory intensity, the processor sends a pause or close instruction to the video module.

[0142] As shown in the examples, the present application can also control the micro changes of the stimulus information from multiple aspects to form the detection of the user's perception. The perception detection of the micro changes of the stimulus information of the present application aims to not disturb the user's sleep.

[0143] Preferably, the processor determines which level of the sensory level to perform the close or pause operation of the function module based on the user's sensory level and the corresponding feedback information.

[0144] For example, for the perception level, the user always has feedback to the first level of the sensory detection in the first time range, and has no feedback to the second level of the sensory detection in the second time. The processor is adjusted based on the user's use case personalization. The timing starts after entering the sleep state, and the first level of the sensory detection is performed when the first time is met. If there is no feedback after the preset feedback time, the processor performs the second level of the sensory detection when the second time is met. If there is no feedback after the preset feedback time, the processor confirms that the user's sleep perception is sufficient, and sends a close or pause instruction to the function module. The second time length is greater than the first time length.

[0145] If the user has no feedback after the first time is met for a long time and the execution times exceed the preset number threshold, the processor can send a close or pause instruction to the function module after the first level of the sensory detection is completed and the preset feedback time is not met.

[0146] If the user turns off the function module after the second time is met for a long time and the ratio of the number of times of turning off the function module compared to the number of times of turning off the first sensory organ exceeds a preset ratio, the processor can skip the first sensory organ detection, directly perform the second sensory organ detection, and control the function module to be turned off.

[0147] Thus, the present application can reduce the influence on the user's sleep in the case of reducing sensory stimulation, and determine the degree of weakening of the user's perception by judging the breathing frequency information and the user's sleep perception, so that the suspension or turning off of the function module has no influence on the user's sleep.

[0148] Preferably, when the breathing frequency information is completely unfamiliar breathing frequency information, the processor can store the new breathing frequency information and the judgment result of the sleep state into a new account by establishing a new account, so as to distinguish the user, and perform the control function of the preset function module.

[0149] It should be noted that the above specific embodiments are exemplary, and those skilled in the art can think of various solutions under the inspiration of the disclosure of the present application, and these solutions also belong to the disclosed range of the present application and fall within the protection scope of the present application. Those skilled in the art should understand that the specification and drawings of the present application are illustrative and do not constitute a limitation on the claims. The protection scope of the present application is defined by the claims and their equivalents.

[0150] The specification of the present application contains multiple inventive concepts, and the applicant reserves the right to file a divisional application according to each inventive concept. The specification of the present application contains multiple inventive concepts, such as "preferably", "according to a preferred embodiment" or "optionally", which all indicate that the corresponding paragraph discloses an independent concept, and the applicant reserves the right to file a divisional application according to each inventive concept.

Claims

1. A control device for an intelligent terminal device, characterized by comprising: The processor is configured to: receive sound information collected by the audio collection module, extract at least one breathing frequency information from the sound information, identify a sleep state based on an extraction model and the breathing frequency information, send preset instruction information to at least one associated function module in a specified sleep state, and send a stimulation information micro-change instruction to at least one function module in operation to test the user's sleep perception in a manner of causing the function module to produce a micro-change in stimulation information, The manner in which the processor extracts the breathing frequency information includes: establishing a breathing frequency information extraction model, comparing a first breathing rate extracted by the breathing frequency information extraction model with a verifiable breathing rate of the first test subject to calculate an error, and inversely adjusting front-end parameters of one or more layers of the extraction model based on the calculated error to improve the prediction accuracy of the extraction model, After determining that the user enters the sleep state based on the breathing frequency information, the playback volume of the current function module is first reduced by a limited volume difference, and in a case where the intelligent terminal device does not detect any input instruction information within a first limited time after the playback volume is reduced, the playback volume is gradually reduced in a stepped volume manner, and the function module is directly turned off when the playback volume is reduced to a shutdown threshold range; In a case where the user does not provide feedback to the micro-change in the stimulation information of the function module, control information is sent to the function module according to a preset control instruction, so as to control the function module in a preset manner, the micro-change in the stimulation information refers to a sensory stimulation that does not stimulate the user to wake up, the extraction model compares the extracted breathing frequency information with a breathing frequency sample, and the processor obtains a corresponding sleep state based on the corresponding association between the breathing frequency sample and the sleep state.

2. The control apparatus of an intelligent terminal device according to claim 1, wherein The processor is further configured to: extract the breathing frequency information from the amplified sound information, eliminate noise, audio and video sound that do not have a periodic rule from the sound information to obtain breathing information that has a periodic rule.

3. The control apparatus of an intelligent terminal device according to claim 1, wherein The processor is configured to: discretize the input three-dimensional waveform information used to train the extraction model, perform data fitting on the breathing information, and guide the data output by the convolution layer to the average pooling layer.

4. The control apparatus of an intelligent terminal device according to claim 1, wherein The processor is configured to: test the user's perception level in the sleep state in a test mode, and control the function module when the user's perception level is reduced to a preset level, the test mode tests the user's perception level in a manner of micro-change in the stimulation information, the micro-change in the stimulation information is a slight adjustment and change of the stimulation already applied to the function module, and no new stimulation is added to avoid waking up the user.

5. The control apparatus of an intelligent terminal device according to claim 1, wherein The processor is configured to: When the user has not completely entered the sleep state, the user's perception ability is strong, the user can move the intelligent terminal device or input related instructions or action signals for viewing, and when the processor detects a movement signal collected by an acceleration sensor in the intelligent terminal device or an action signal of a related input device or an input instruction, the processor controls the function module to restore the playback volume. After the user enters the sleep state, the user's perception ability becomes weak, the processor reduces the current function module's playing volume by a first limited volume difference, and the user does not move the smart terminal device or input an action signal to view the change of the function module or input an instruction without perception, at which time the processor can reduce the playing volume for a second time, and in the case that no intervention information is received within a second limited time, the processor closes or suspends the current specified function module.

6. The control apparatus of an intelligent terminal device according to claim 1, wherein The processor is configured to: After determining that the user enters the sleep state based on the breathing rate information, the processor detects the user's perception by pausing the function module, After detecting the user's feedback by testing the perception by pausing the function module once, the processor determines whether the user enters the sleep state based on the current breathing rate information, and repeats the perception detection after the user enters the sleep state, After determining that the user enters the sleep state based on the breathing rate information, the processor can also detect the user's perception by changing the playing content, and the processor changes the playing content in a way that is conducive to sleep.

7. The control apparatus of an intelligent terminal device according to claim 1, wherein The processor is configured to: As the user's use time is prolonged and the detection times are increased, the processor can extract and update the breathing rate information of the user entering the sleep state when using each type of function module as a personalized frequency sample, and store it at the same time. When the processor compares the extracted breathing rate information with the personalized frequency sample, the accuracy of determining whether to enter the sleep state will also be higher and higher, forming a positive cycle.

8. A control method of an intelligent terminal device, characterized by, It includes: Receiving sound information collected by an audio collection module, extracting at least one breathing rate information from the sound information, identifying a sleep state based on an extraction model and the breathing rate information, sending preset instruction information to at least one associated function module in a specified sleep state, and in the case that the user enters the sleep state, sending a stimulation information micro-change instruction to at least one running function module to test the user's sleep perception in a way that the stimulation information of the function module produces a micro-change, The way the processor extracts the breathing rate information includes: establishing a breathing rate information extraction model, comparing the first breathing rate extracted by the breathing rate information extraction model with the verifiable breathing rate of the first test object to calculate the error, and inversely adjusting the front-end parameters of one or more layers of the extraction model based on the calculation error to improve the prediction accuracy of the extraction model, After determining that the user enters the sleep state based on the breathing rate information, the processor reduces the current function module's playing volume by a first limited volume difference, and in the case that the smart terminal device does not detect any input instruction information within a first limited time after the playing volume is reduced, the processor gradually reduces the playing volume in a stepped volume manner, and directly closes the function module when the playing volume is lowered to a closing threshold range. In the case that there is no feedback for the micro change of the stimulation information of the user to the function module, the processor sends control information to the function module according to the preset control instruction, so as to control the function module in a preset mode, the micro change of the stimulation information refers to the sensory stimulation which does not stimulate the user to be awake, the extracted model compares the extracted respiratory frequency information with the respiratory frequency sample, and on the basis that the respiratory frequency sample and the sleep state exist corresponding association, the processor obtains the corresponding sleep state according to the respiratory frequency information.

Images