Electronic equipment and sleep monitoring method

Abstract

The embodiment of the invention discloses electronic equipment and a sleep monitoring method, relates to the technical field of intelligent household appliances, and is used for solving the problem that existing electronic equipment cannot accurately monitor the sleep state of a monitored human body. A thermal imaging thermodetector; the audio acquisition equipment is used for acquiring audio information of a monitored human body; the controller is configured to acquire a sleep image of a monitored human body through the image acquisition device, acquire a face temperature of the monitored human body through the thermal imaging thermodetector and acquire a sleep audio of the monitored human body through the audio acquisition device; determining the sleep state of the monitored human body according to the sleep image, the face temperature, the sleep audio and a sleep detection model; determining a target monitoring strategy of the electronic equipment according to the sleep state of the monitored human body; the target monitoring strategy is used for processing the sleep state of the monitored human body; and executing the target monitoring strategy.

Description

Technical Field

[0001] The present application relates to the technical field of smart home appliances, and in particular to an electronic device and a sleep monitoring method. Background Art

[0002] With the continuous advancement of smart home device technology, electronic devices with sleep monitoring functions are becoming more and more widely used in daily life, reducing people's work intensity and reducing people's monitoring pressure.

[0003] Existing electronic devices can be bound to monitoring solutions to achieve full monitoring of the sleeping person in a specific environment. Users can watch monitoring videos on mobile devices to determine whether the current sleeping state of the monitored person is abnormal.

[0004] However, this monitoring method requires the guardian to judge the sleep state of the monitored person based on the video information, and some functions still require the guardian to control remotely, and it is impossible to provide timely feedback on the abnormal sleep state of the monitored person. Summary of the invention

[0005] The present application provides an electronic device and a sleep monitoring method, which are used to solve the problem that existing electronic devices cannot accurately monitor the sleep state of a monitored person.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions.

[0007] In a first aspect, an embodiment of the present application provides an electronic device, comprising: an image acquisition device, used to acquire an image of a target area, in which there is a monitored person; a thermal imaging thermometer, used to detect the facial temperature of the monitored person; an audio acquisition device, used to acquire audio information of the monitored person; a controller, configured to: acquire a sleep image of the monitored person through the image acquisition device, acquire the facial temperature of the monitored person through the thermal imaging thermometer, and acquire the sleep audio of the monitored person through the audio acquisition device; determine the sleep state of the monitored person based on the sleep image, facial temperature, sleep audio, and sleep detection model; determine a target monitoring strategy of the electronic device based on the sleep state of the monitored person; the target monitoring strategy is used to process the sleep state of the monitored person; and execute the target monitoring strategy.

[0008] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects: the technical solution collects the sleep image, facial temperature and sleep audio of the monitored person, and then determines the sleep state of the monitored person based on the collected sleep image, facial temperature, sleep audio and sleep detection model, and controls the electronic device to execute different monitoring strategies according to the different sleep states of the monitored person. It can not only accurately identify the abnormal sleep state of the monitored person, but also respond to various abnormal sleep states in a timely manner, thereby reducing the user's monitoring pressure and ensuring the sleep safety of the monitored person.

[0009] In some embodiments, the sleep detection model includes a face covering detection model, a kicking quilt detection model and a crying detection model, and the sleep state includes whether the face of the monitored person is covered, whether the monitored person kicks the quilt and whether the monitored person cries; the controller is configured to determine the sleep state of the monitored person based on the sleep image, facial temperature, sleep audio and the sleep detection model, and is specifically configured to: input the sleep image into the face covering detection model to obtain a face covering detection result of the monitored person; the face covering detection result includes that the face of the monitored person is covered or the face of the monitored person is not covered; or, input the sleep image into the kicking quilt detection model to obtain a kicking quilt detection result of the monitored person; the kicking quilt detection result includes that the monitored person has kicked the quilt or the monitored person has not kicked the quilt; or, input the sleep audio into the crying detection model to obtain a crying detection result of the monitored person; the crying detection result includes that the monitored person has cried or the monitored person has not cried.

[0010] In some embodiments, the electronic device also includes: an environmental parameter regulator, used to adjust the environmental parameters of the target area; a controller, configured to determine the target monitoring strategy according to the sleeping state of the monitored person, specifically configured to: when the face coverage detection result of the monitored person is that the face of the monitored person is covered or the facial temperature is not within a preset temperature range, determine that the target monitoring strategy of the electronic device is to issue a reminder to the user; or, when the kicking detection result of the monitored person is that the monitored person has kicked the quilt, determine that the target monitoring strategy of the electronic device is to control the environmental parameter regulator to increase the temperature of the target area and issue a reminder to the user; or, when the crying result of the monitored person is that the monitored person has cried, determine that the target monitoring strategy of the electronic device is to issue a reminder to the user and turn on the coaxing function.

[0011] In some embodiments, the controller is further configured to: obtain the position of the monitored person and the bed in the sleep image; determine whether the monitored person is at risk of falling out of bed based on the positional relationship between the position of the monitored person and the bed; and when the monitored person is at risk of falling out of bed, determine the target monitoring strategy of the electronic device to issue a reminder to the user.

[0012] In some embodiments, the electronic device further includes: a temperature sensor for detecting the ambient temperature of the target area; a humidity sensor for detecting the ambient humidity of the target area; a light intensity sensor for detecting the light intensity of the target area; and an air quality sensor for detecting the air quality index of the target area; the controller is further configured to: obtain the ambient temperature of the target area through the temperature sensor, obtain the ambient humidity of the target area through the humidity sensor, obtain the light intensity of the target area through the light intensity sensor, and obtain the air quality index of the target area through the air quality sensor; when the ambient temperature, ambient humidity, light intensity or air quality index is not within the preset environmental parameter range, control the environmental parameter regulator to adjust the ambient temperature, ambient humidity, light intensity or air quality index of the target area until the ambient temperature, ambient humidity, light intensity or air quality index is within the preset environmental parameter range.

[0013] In a second aspect, an embodiment of the present application provides a sleep monitoring method, the method comprising: obtaining a sleep image, facial temperature, and sleep audio of a monitored person; determining the sleep state of the monitored person based on the sleep image, facial temperature, sleep audio, and sleep detection model; determining a target monitoring strategy of an electronic device based on the sleep state of the monitored person; the target monitoring strategy is used to process the sleep state of the monitored person; and executing the target monitoring strategy.

[0014] In a third aspect, an embodiment of the present application provides a controller comprising: one or more processors; one or more memories; wherein the one or more memories are used to store computer program codes, the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the controller executes any one of the sleep monitoring methods provided in the second aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a computer, the computer executes the method provided in the second aspect and possible implementation methods.

[0016] In a fifth aspect, an embodiment of the present invention provides a computer program product, which can be directly loaded into a memory and contains software code. After being loaded and executed by a computer, the computer program product can implement the method provided in the second aspect and possible implementation methods.

[0017] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the controller, or may be packaged separately from the processor of the controller, which is not limited in this application.

[0018] The beneficial effects described in the second to fifth aspects of the present application can be referred to the beneficial effect analysis of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0020] Figure 1 A hardware configuration diagram of an electronic device provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of the structure of a service robot provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the structure of another service robot provided in an embodiment of the present application;

[0023] Figure 4 A hardware configuration block diagram of a service robot provided in an embodiment of the present application;

[0024] Figure 5 A flowchart of a sleep monitoring method provided in an embodiment of the present application;

[0025] Figure 6 A flowchart of another sleep monitoring method provided in an embodiment of the present application;

[0026] Figure 7 A flowchart of another sleep monitoring method provided in an embodiment of the present application;

[0027] Figure 8 A flowchart of another sleep monitoring method provided in an embodiment of the present application;

[0028] Fig. 9 A flowchart of another sleep monitoring method provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the position relationship between a human body and a bed provided in an embodiment of the present application;

[0030] Fig.11 A flowchart of another sleep monitoring method provided in an embodiment of the present application;

[0031] Fig.12 A flowchart of another sleep monitoring method provided in an embodiment of the present application;

[0032] Fig.13A flowchart of another sleep monitoring method provided in an embodiment of the present application;

[0033] Fig.14 A flowchart of another sleep monitoring method provided in an embodiment of the present application;

[0034] Fig.15 A flowchart of another sleep monitoring method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0037] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0038] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, when describing a pipeline, the "connected" and "connection" used in this application have the meaning of conduction. The specific meaning needs to be understood in conjunction with the context.

[0039] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0040] In the embodiments of the present application, "and / or" is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0041] In the prior art, the electronic device can be a service robot. A service robot is a machine device that automatically performs work. It can accept human commands, run pre-programmed programs, and act according to principles and guidelines formulated with artificial intelligence technology. Its task is to assist or replace human work, such as manufacturing, construction, and dangerous work. With the continuous development of science and technology, people are gradually using service robots to monitor and replace human work, which is conducive to reducing human work intensity and ensuring normal sleep and rest in production and life.

[0042] For example, in the application scenario of kindergarten, the kindergarten will gather the children in a nap room during the lunch break, but some children like to move around during the nap, covering their faces with the quilt, kicking the quilt or falling off the bed; because there are many children in the kindergarten and relatively few kindergarten teachers, the teachers may not be able to find the situation during the lunch break in time. Therefore, more kindergartens use service robots to assist kindergarten teachers in taking care of children during their naps.

[0043] As described in the above technical background, existing electronic devices have limited functions and are unable to make judgments on various sleep states of the monitored human body, thereby making it impossible for the electronic devices to provide effective services to users in a timely manner.

[0044] Based on this, an embodiment of the present application provides an electronic device, including: an image acquisition device, used to acquire an image of a target area, in which there is a monitored person; a thermal imaging thermometer, used to detect the facial temperature of the monitored person; an audio acquisition device, used to acquire audio information of the monitored person; a controller, configured to: acquire a sleep image of the monitored person through the image acquisition device, acquire the facial temperature of the monitored person through the thermal imaging thermometer, and acquire the sleep audio of the monitored person through the audio acquisition device; determine the sleep state of the monitored person based on the sleep image, facial temperature, sleep audio and sleep detection model; determine a target monitoring strategy of the electronic device based on the sleep state of the monitored person; the target monitoring strategy is used to process the sleep state of the monitored person; and execute the target monitoring strategy.

[0045] In this way, the sleep state of the monitored person is monitored in real time, and when the sleep state of the monitored person is abnormal, a processing strategy is selected in time to achieve the purpose of effective monitoring.

[0046] The embodiments provided in this application are described in detail below in conjunction with the accompanying drawings.

[0047] Figure 1 A hardware configuration diagram of an electronic device provided in an embodiment of the present application, such as Figure 1 As shown, taking the electronic device as a service robot as an example, the service robot 10 includes a robot body 1, a display screen 2, an image acquisition device 3, an audio acquisition device 4 ( Figure 1 ) and the controller 1000 ( Figure 1 not shown).

[0048] It should be noted that Figure 1 The electronic device is taken as a service robot as an example for explanation, and the sleep monitoring function of the service robot is specifically explained. However, in a specific implementation, the electronic device may also be other possible devices, and the electronic device may also have other functions (such as boiling detection function, etc.), and this application does not limit this.

[0049] In some embodiments, the display screen 2 is used to display the environmental parameters of the target area, the sleep report of the monitored person, and the operation log, etc.

[0050] In some embodiments, the image acquisition device 3 is used to acquire images of the monitored human body in the target area. The image acquisition device 3 is communicatively connected with the robot body 1 , and the robot body 1 controls the operation of the image acquisition device 3 .

[0051] In some embodiments, the image acquisition device 3 is also used to acquire room map information of the user's residence.

[0052] In some embodiments, the image acquisition device 3 may be composed of a color (red green blue, RGB) camera, a depth camera memory laser radar, and can realize all-round and multi-angle monitoring of the monitored human body. Among them, this RGB camera has the function of infrared fill light. This application does not limit the type of image acquisition device.

[0053] It should be understood that the RGB camera with infrared fill light function can support shooting in both visible light and infrared light bands. When the ambient light is sufficient, the RGB camera performs visible light shooting; when the ambient light is insufficient, the RGB camera can provide fill light function to achieve shooting in a lightless or dim environment.

[0054] In some embodiments, the audio collection device 4 may be a microphone for collecting audio information of the monitored body.

[0055] In some embodiments, Figure 1As shown, the service robot 10 also includes a plurality of pulleys 5, which are connected to the robot body 1 and are located at the bottom of the robot body 1. The pulleys 5 are in contact with the ground and can support the service robot 10. The pulleys 5 realize the functions of moving, walking, turning, etc. of the service robot 10, expand the monitoring range of the service robot 10, reduce manual handling, and are more convenient.

[0056] It should be noted that Figure 1 The schematic diagram of the composition of a service robot is shown only for exemplary purposes. In a specific implementation, the service robot may be composed of more or fewer hardware parts, and this application does not limit this.

[0057] Figure 2 FIG. 1 is a schematic diagram of the structure of a service robot shown in an embodiment of the present application. Figure 2 As shown, the service robot 10 has a mapping and navigation module 11, which is responsible for building a point cloud map of the user's residence and navigating to the specified coordinates.

[0058] It should be noted that mapping is based on simultaneous localization and mapping (SLAM) technology. To turn on the mapping function mode, the user needs to push the robot to move around the house to cover the entire house as much as possible. Residence space identification points can also be set during the mapping process.

[0059] For example, after being pushed to the children's room, manually or by voice setting that this is the children's room, the robot will record the current coordinate information name as the children's room.

[0060] Furthermore, the navigation function automatically plans the motion path based on the established spatial map information and controls the robot to move to the designated location.

[0061] In some embodiments, Figure 2 As shown, the service robot 10 also has a voice announcement module 12 for issuing voice reminders to the user.

[0062] In some embodiments, Figure 2 As shown, the service robot 10 also has an environmental parameter control module 13. Figure 3 A schematic diagram of the structure of another service robot provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the environmental parameter control module 13 includes a temperature sensor 131 , a humidity sensor 132 , an air quality sensor 133 , a light intensity sensor 134 and an environmental parameter regulator 135 .

[0063] In some embodiments, the temperature sensor 131 is used to detect the ambient temperature of the target area.

[0064] In some embodiments, the humidity sensor 132 is used to detect the ambient humidity of the target area.

[0065] In some embodiments, the air quality sensor 133 is used to detect the air quality index of the target area.

[0066] In some embodiments, the light intensity sensor 134 is used to detect the light intensity of the target area.

[0067] In some embodiments, the environmental parameter adjuster 135 is used to adjust the environmental parameters of the target area.

[0068] In the embodiment shown in the present application, the controller 1000 refers to a device that can generate an operation control signal according to the instruction operation code and the timing signal to instruct the service robot 10 to execute the control instruction. Exemplarily, the controller 1000 can be a central processing unit (CPU), a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD) or any combination thereof. The controller 1000 can also be other devices with processing functions, such as circuits, devices or software modules, and the embodiments of the present application do not impose any restrictions on this.

[0069] In addition, the controller 1000 can be used to control various components inside the service robot 10 so that each component operates to achieve various predetermined functions of the service robot 10.

[0070] Figure 4 FIG. 1 is a hardware configuration block diagram of a service robot 10 provided by the present application according to an exemplary embodiment. Figure 4 As shown, the service robot 10 may also include the following three items: a thermal imaging thermometer 6 , a memory 1002 and a communication interface 1003 .

[0071] In some embodiments, the thermal imaging thermometer 6 is used to detect the facial temperature of the monitored person.

[0072] In some embodiments, the memory 1002 may be used to store software programs and data. The controller 1000 executes various functions of the service robot 10 and performs data processing by running the software programs or data stored in the memory 1002 .

[0073] Optionally, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1002 stores an operating system that enables the service robot 10 to run. In the present application, the memory 1002 may store an operating system and various application programs, and may also store code for executing the sleep monitoring method provided in the embodiment of the present application.

[0074] In some embodiments, the communication interface 1003 is used to establish a communication connection with other network entities, for example, to establish a communication connection with a terminal device. The communication interface 1003 may include a radio frequency (RF) module, a cellular module, a wireless fidelity (WIFI) module, and a GPS module. Taking the RF module as an example, the RF module can be used to receive and send signals, in particular, to send the received information to the controller 1000 for processing; in addition, the signal generated by the controller 1000 is sent out. Typically, the RF circuit may include but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like.

[0075] Those skilled in the art will understand that Figure 4 The hardware structure shown in the figure does not constitute a limitation of the service robot. The service robot may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0076] Figure 5 A flowchart of a sleep monitoring method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method comprises the following steps:

[0077] S101. A controller obtains a sleeping image, facial temperature, and sleeping audio of a monitored person.

[0078] Specifically, the controller can obtain the sleeping image of the monitored person through the image acquisition device, obtain the facial temperature of the monitored person through the thermal imaging thermometer, and obtain the sleeping audio of the monitored person through the audio acquisition device.

[0079] Optionally, the image acquisition device may be an RGB camera with infrared fill light.

[0080] In some embodiments, taking the electronic device as a service robot as an example, when the user issues a monitoring instruction by voice or by clicking on the display screen of the service robot, the mapping and navigation module of the service robot navigates the service robot to the designated room.

[0081] Exemplarily, if the supervised object is a baby, the service robot is navigated to the child's room.

[0082] Furthermore, when the service robot arrives at the children's room, the environmental parameter control module of the service robot can turn on the lights in the room, find the baby's location through image acquisition equipment and human recognition algorithms, and after finding the baby, automatically calculate the guardianship location 1 meter to 1.5 meters away from the baby, convert it into a map coordinate point, and navigate the service robot to the location through the mapping and navigation modules.

[0083] Alternatively, if the baby cannot be found or the destination coordinates cannot be navigated to due to being blocked by obstacles that cannot be circumvented, the service robot is controlled to navigate to a default preset location in the child's room.

[0084] In some embodiments, since the service robot body is relatively low in height, the RGB camera built into the service robot body cannot meet the camera field of view requirements for watching the baby's sleep, so a USB camera is added to the service robot. This USB camera uses the universal serial bus video class (UVC) protocol, and this USB camera supports infrared fill light in insufficient light, so barrier-free monitoring in home rooms when there is insufficient light is achieved, and the camera is developed through the UVC driver to obtain real-time captured sleep images.

[0085] Furthermore, after arriving at a set location in the target area, the controller acquires a sleeping image of the baby through an image acquisition device.

[0086] In some embodiments, the image acquisition technology of the image acquisition device can also be developed using the universal serial bus video class (UVC) protocol. The UVC protocol is a protocol standard specifically defined for USB video capture devices. An image acquisition device that supports the UVC protocol supports infrared fill light in insufficient light, thereby achieving barrier-free monitoring in home rooms when there is insufficient light.

[0087] Figure 6 A flowchart of another sleep monitoring method provided in an embodiment of the present application is used to drive the opening of an image acquisition device, such as Figure 6 As shown, when the USB device is initialized (unit), the relevant information of the USB device is obtained, and after obtaining the permission to drive the image acquisition device in the permission algorithm (onAttach) in the USB callback event, the image acquisition device is opened through the opening algorithm (onConnect).

[0088] Furthermore, the frame preview subfunction (onFrame) in the image acquisition device callback class (IFrameCallback) obtains the real-time captured image information in the ByteBuffer format, and draws the image information in this format to the preview function (SurfaceView) to realize the screen preview of the image acquisition device.

[0089] In some embodiments, when sleep detection needs to obtain sleep images, the image information in ByteBuffer format is first converted into image information in NV21 format, and then the image information in NV21 format is converted into image information in Bitmap format, and then the image information in Bitmap format is converted into image information in Mat format during inference in the neural network forward computing framework.

[0090] Optionally, the audio acquisition device may be an array microphone. This application does not limit the type of the audio acquisition device.

[0091] In some embodiments, the sleeping audio signal collected by the service robot array is subjected to Fourier transform, and the transformed frequency domain signal is then sent to a machine learning algorithm for processing, thereby realizing the judgment of the crying of the baby.

[0092] It should be noted that the Fourier transform can represent a function that meets certain conditions as a linear combination of trigonometric functions (sine and / or cosine functions) or their integrals. The embodiment of the present application applies the Fourier transform to analyze the components of the sleep audio signal, and these components can also be used to synthesize the sleep audio signal.

[0093] In some embodiments, the thermal imaging thermometer may be an infrared thermal imaging thermometer. The present application does not limit the type of thermometer.

[0094] It should be noted that infrared thermal imaging technology is based on the level of radiation energy of the detected object, which is converted into a thermal image of the target object through system processing and displayed in grayscale or pseudo-color, that is, the temperature distribution of the measured target is obtained to determine the state of the object. All objects in nature will emit infrared radiation, and their radiation energy is proportional to the fourth power of their own temperature, and the wavelength of radiation is inversely proportional to their temperature. Thermal imaging detects infrared energy non-contactly and converts it into electrical signals, thereby generating thermal images and temperature values ​​on the display, and the temperature values ​​can be calculated.

[0095] In some embodiments, the service robot is connected to an infrared thermal imaging thermometer, which communicates with the service robot via a USB interface. The infrared thermal imaging thermometer uses infrared light to perform thermal imaging and can detect the temperature value of each point in the thermal image. While the robot's camera monitors the baby's face, the infrared thermal imaging thermometer also monitors the temperature of the baby's face in real time.

[0096] S102: The controller determines the sleep state of the monitored person according to the sleep image, facial temperature, sleep audio, and sleep detection model.

[0097] Among them, the sleep detection model includes a face covering detection model, a kicking quilt detection model and a crying detection model. The sleeping state includes whether the face of the monitored person is covered, whether the monitored person kicks the quilt and whether the monitored person cries.

[0098] Figure 7 A flowchart of another sleep monitoring method provided in an embodiment of the present application is used to determine the sleep state of a monitored person, such as Figure 7 As shown, the method comprises the following steps:

[0099] S201. The controller inputs a sleep image into a face cover detection model to obtain a face cover detection result of the monitored person.

[0100] The face coverage detection result includes that the face of the monitored person is covered or that the face of the monitored person is not covered.

[0101] In some embodiments, the facial coverage detection model adopts the target detection method in the field of deep learning, and extracts a specific algorithm by observing the scene where the face is covered. It can be attributed to the face detection and facial key point detection to provide the underlying core information, which is realized through peripheral information integration.

[0102] Optionally, the algorithm selects the feature extraction module in the target detection framework YoloV8

[0103] (Backbone), use the (Common Objects in Context, COCO) dataset and the (Visual Object Classes, VOC) dataset to extract the box coordinates of the human body and the information of the five key points of the human face as the original input information of the network. The classification idea is used for face category training, the regression idea is used for position positioning box, and the regression idea is used for key point training in the detected face area.

[0104] The batch_size, epoch and learning rate are set to 16, 500 and yolo, respectively. The other hyperparameters use the initial values ​​recommended by yolo. The yolov8s model is used as the pre-training parameter for fine-tinning. Finally, the information of the face and facial key points is obtained.

[0105] Furthermore, according to the acquired face and facial key point information, after passing through the submodules involved in the face coverage rule, a final artificial intelligence (AI) detection result of whether the baby's face is covered is output.

[0106] Specifically, based on the face box and five key points output by the face coverage detection model, the face coverage detection result of the monitored person is output in combination with comprehensive information such as the shape ratio, score, and nose key points of the face box.

[0107] S202: The controller inputs the sleep image into the quilt kicking detection model to obtain the quilt kicking detection result of the monitored person.

[0108] The kicking-the-quilt detection result includes whether the monitored person has kicked the quilt or whether the monitored person has not kicked the quilt.

[0109] In some embodiments, the algorithm of the kicking quilt detection model is consistent with that of the face detection model. Specifically, through scene observation of kicking quilt detection, a specific algorithm is extracted that can be attributed to providing underlying core information for human body detection and human skeleton key point detection, and is realized through peripheral information integration.

[0110] It should be noted that, for details about the training method of the kicking detection model, please refer to the training method of the face detection model in the above step S201, which will not be repeated here.

[0111] Furthermore, after obtaining the key points of the human body and human skeleton, based on the comprehensive information such as the shape ratio, score, belly and leg key points of the human body box, the kicking detection result of the monitored person can be correctly output.

[0112] For example, the shape ratio threshold can be set to person_thr1 based on the shape ratio, area, score, abdominal key points and other information of the human body box. The shape ratio calculation formula is: person_ratio = abs(max(w / h,h / w)). When person_ratio is greater than person_thr1, it means that the current human body area is a very narrow box that does not conform to the shape of the human body. At this time, the detected object is not considered to be a correct human body and is a false detection.

[0113] Or, set the human body area threshold to person_thr2, person_thr3, area=

[0114] w*h, when area is smaller than person_thr2 or larger than person_thr3, the object is considered to be a particularly small object or a particularly large object, which does not conform to the human body area characteristics of a sleeping baby and is a false detection.

[0115] Alternatively, in cases other than the above two cases, the human body result output by the current model is considered to be the correct human body result. After the correct human body judgment, the value of v in the human abdominal key point information is combined to set the threshold person_thr5 of the abdominal key point. When the v value is greater than person_thr5, it is considered that the baby's abdomen is exposed in the frame image, and the kicking quilt state is determined to be yes (True), and the kicking quilt is output. Otherwise, the quilt is not kicked.

[0116] In some embodiments, the facial coverage detection model and the kicking detection model can also use the target detection framework YoloV7 to screen human category images in the COCO and VOC international open source data sets. Since these data sets already have prior knowledge of human boxes and key points, but no prior knowledge of faces, it is necessary to annotate the face box information through the label me marking tool.

[0117] Furthermore, the acquired training sample set is used as a label, and the collected pictures of the face of the supervised person being covered or the supervised person kicking the quilt are used as input data.

[0118] Optionally, the collected pictures of the face of the supervised person being covered or the supervised person kicking off the quilt can be stored in a preset folder (such as a Train folder), and the label data format of each picture can be set to a text file (.txt).

[0119] Among them, the .txt file contains the input (int) type and floating (float) type values ​​of the specific area and point of the label (label), and the folder path of the picture is specified in the file of file type yaml in the data (data) in the YoloV7 detection framework.

[0120] In some embodiments, when the sleep detection model starts training, when executing the Train.py part of the code in the Train folder, the corresponding image data and label data can be loaded into the project through the dataset loading interface (create_dataloader) of the YoloV7 detection framework.

[0121] Furthermore, the image features are extracted through the feature extraction module (backbone). When calculating based on the loss function (loss), the faces and bodies are classified and the position is located using the regression idea for training and learning.

[0122] In some embodiments, taking a 3060ti graphics card with a batch_size of 32 images loaded at a time, an epoch (epochs) of 500, and a learning rate (lr) of an initial value as an example, the parameters of the YoloV7s model are used as pre-training parameters, fine-tinning training is performed, and training is performed using an open source deep learning framework (Pytorch). When the epochs training is completed, a sleep detection model after training is finally obtained.

[0123] In some embodiments, after the trained sleep detection model is obtained, in order to protect user privacy, an AI end-side deployment method is adopted. However, the Android side of the robot system cannot directly infer the model trained by the Pytorch framework, so a neural network forward computing framework is adopted.

[0124] In some embodiments, the model format of the sleep detection model is changed from

[0125] The face_person_keypoints.pt format is converted to the TorchScript format through export.py in YoloV7, and then the boiling detection model in the TorchScript format is converted into a neural network forward computing framework model through the pytorch model deployment method (pytorch neuralnetwork exchange, PNNX).

[0126] Among them, the files of the neural network forward calculation framework model are divided into

[0127] There are two files: face_person_keypoints.param and face_person_keypoints.bin.

[0128] It should be noted that when used in the terminal, the fundamental purpose of model loading is to load information in the .param format and the .bin file format into the target neural network (an ncnn::Net structure).

[0129] Among them, the .param format file describes the structure of the neural network, including the layer name, layer input and output information, and layer parameter information (such as the kernel size of the convolutional layer, etc.).

[0130] Furthermore, the .bin format file records the data information required for neural network operations (such as the weights and bias information of the convolutional layer). When deployed, the model can be loaded and inference prediction can be achieved by configuring the include and .a / .so of the neural network forward computing framework in the robot, and then calling the relevant APIs opened by the neural network forward computing framework.

[0131] In some embodiments, after the sleep detection model is loaded, the detection result of each frame of the sleep image is obtained.

[0132] The specific form of the detection result is [x, y, w, h, label, score, key points], where x and y represent the coordinates of the upper left corner of the area rectangle of the detected object, w and h represent the width and height of the area rectangle, label represents the type of object (0 represents face, 1 represents person), score represents the score predicted by the model (the value is between 0 and 1), keypoints represents the key point information (if a face is detected, this item is not empty, if a person is detected, key points is an array representing the information of 17 key points of the skeleton), a key point is (x, y, v), x represents the horizontal axis coordinate, y represents the vertical axis coordinate, v represents the visibility (the value is 0 to 1), and it contains a total of 51 values.

[0133] Furthermore, based on the above detection results, the baby's face covering detection result and the quilt kicking detection result are output.

[0134] It should be noted that the concept of deep learning is to mine the universal characteristics of an object based on the prior knowledge of a large number of samples provided. It is a statistical field, so it is difficult to achieve 100% accuracy. In terms of application, based on the idea of ​​video stream, in order to improve the accuracy of events, the final detection event result of the frame is output after combining the detection results of multiple frames. Set the threshold frame_thr1 of multiple consecutive frames. If the sum of the consecutive detections of the face being covered and the kicking of the quilt as True is greater than frame_thr1, the detection result of the corresponding event in the frame is judged to be True.

[0135] S203: The controller inputs the sleep audio into the crying detection model to obtain a crying detection result of the monitored person.

[0136] The crying detection result includes whether the monitored person has cried or whether the monitored person has not cried.

[0137] In some embodiments, the crying detection model includes a crying detection algorithm. The training parameters of the crying detection model are pre-recorded audios of multiple infant cries. The crying audios are converted into frequency domain signals, and the frequency domain eigenvalues ​​are extracted and sent to the machine learning algorithm. The model is trained using the idea of ​​a classifier.

[0138] Furthermore, the trained model is imported into the service robot algorithm board to complete the code writing related to model deployment. In actual use, the sleep audio signal collected by the service robot array is Fourier transformed, and the transformed frequency domain signal is sent to the machine learning algorithm for processing, so as to realize the judgment of the baby's crying.

[0139] S103: The controller determines a target monitoring strategy of the electronic device according to the sleep state of the monitored person.

[0140] In some embodiments, the controller determines a target monitoring strategy of the electronic device according to different sleep states of the monitored human body output by different models. Figure 8 A flowchart of another sleep monitoring method provided in an embodiment of the present application is used to determine a target monitoring strategy for an electronic device, such as Figure 8 As shown, the method comprises the following steps:

[0141] S301. When the face cover detection result of the monitored person is that the face of the monitored person is covered or the face temperature is not within a preset temperature range, the controller determines that the target monitoring strategy of the electronic device is to issue a reminder to the user.

[0142] In some embodiments, if the baby's face is not covered, the temperature value measured by the infrared thermal imaging thermometer is approximately 36°C; if the baby's face is covered, the temperature value measured by the infrared thermal imaging thermometer is the temperature of the quilt surface, which is equivalent to the ambient temperature and lower than the human body temperature.

[0143] Optionally, the service robot can set a preset temperature range, also known as a temperature threshold. If the temperature data received by the service robot from the infrared thermal imaging thermometer is not within the preset temperature range, the baby's face may be covered or the baby's body temperature is too high, and a text message reminder is sent and a voice alarm is issued to search the whole house.

[0144] Optionally, the preset temperature range may be (35°C-37°C).

[0145] Exemplarily, when the detection result of the face cover detection model is that the face of the monitored person is covered or the face temperature is not within the range of (35° C.-37° C.), the controller determines that the target monitoring strategy of the electronic device is to issue a reminder to the user.

[0146] Specifically, when the output result of the face cover detection model is that the face of the monitored person is covered or the face temperature of the baby is T b When the temperature is not within the preset range, for example, T b When the temperature is less than 35°C, the baby's face may be blocked, and the controller determines that the target monitoring strategy of the electronic device is to issue a reminder to the user.

[0147] Or, when T b >37℃, the baby's body temperature is too high and a fever may occur. The controller determines the target monitoring strategy of the electronic device to issue a reminder to the user.

[0148] S302: When the result of the detection of the monitored person kicking the quilt is that the monitored person has kicked the quilt, the controller determines that the target monitoring strategy of the electronic device is to control the environmental parameter regulator to increase the temperature of the target area, and sends a reminder to the user.

[0149] Exemplarily, when the output result of the kicking detection model is that the monitored person has kicked the quilt, the controller determines that the target monitoring strategy of the electronic device is to control the environmental parameter regulator to increase the temperature of the target area, and sends a reminder to the user.

[0150] S303: When the crying detection result indicates that the monitored person has cried, the controller determines that the target monitoring strategy of the electronic device is to send a reminder to the user and activate a sleep-coaxing function.

[0151] In some embodiments, if the face coverage detection result of the monitored person is that the face of the monitored person is not covered, or the kicking of the quilt detection result of the monitored person is that the monitored person does not kick the quilt, or the crying detection result of the monitored person is that the monitored person is not crying or the facial temperature of the monitored person is within a preset temperature range, the controller controls the electronic device to continue detecting.

[0152] S104: The controller executes the target monitoring strategy.

[0153] Exemplarily, when the controller determines that the target monitoring strategy of the electronic device is to issue a reminder to the user, the service robot will actively leave the child's room to find the parents to issue a voice alarm, and the controller controls the electronic device to issue a reminder to the user.

[0154] Optionally, the electronic device may remind the user via a voice broadcast module, or may remind the user by sending a text message to the user's mobile phone.

[0155] Exemplarily, when the controller determines that the target monitoring strategy of the electronic device is to control the environmental parameter regulator to increase the temperature of the target area and issue a reminder to the user, the controller increases the temperature of the target area through the environmental parameter control module of the service robot to prevent the baby from catching a cold and issues a reminder to the user.

[0156] For example, when the controller determines that the target monitoring strategy of the electronic device is to send a reminder to the user and turn on the soothing function, the controller controls the service robot to send a reminder to the user's mobile phone and play the user's preset soothing video. If the crying continues, the preset nursery rhyme video will be played. When the crying stops, the service robot stops playing the nursery rhyme and the soothing video.

[0157] In some embodiments, the service robot can also determine whether the monitored person is at risk of falling out of bed by measuring the positional relationship between the monitored person and the bed. Fig. 9 A flowchart of another sleep monitoring method provided in an embodiment of the present application is used to determine whether the monitored person is at risk of falling out of bed, such as Fig. 9 As shown, the method comprises the following steps:

[0158] S401: The controller obtains the position of the monitored person and the position of the bed in the sleeping image.

[0159] Optionally, the bed can be a common bed, a crib or an electronic fence. Fig.10 A schematic diagram of the positional relationship between a human body and a bed provided in an embodiment of the present application.

[0160] S402: The controller determines whether the monitored person is at risk of falling out of bed based on the positional relationship between the monitored person and the bed.

[0161] In some embodiments, when the coordinates of the upper left corner and the lower right corner of the baby's body area reach the edge area of ​​the bed, it is determined that the baby is at a dangerous boundary and has a risk of falling out of bed.

[0162] S403: When the monitored person is at risk of falling out of bed, the controller determines the target monitoring strategy of the electronic device to issue a reminder to the user.

[0163] In some embodiments, when it is determined that the baby is at risk of falling out of bed, music can be played to attract the baby's attention, and the electronic device can be controlled to send a text message reminder and search the whole house for people to make a voice alarm.

[0164] In some embodiments, if it is determined that the infant is not at risk of falling out of bed, the controller controls the electronic device to continue monitoring.

[0165] In some embodiments, if the service robot receives a request for video surveillance from a user's mobile phone, the controller controls the image acquisition device to encode the currently acquired image and video data, and pushes them to a remote server via the real-time messaging protocol (RTMP). The user can pull and view the image and video data on the corresponding application on the mobile phone as needed.

[0166] In some embodiments, the service robot will store the data of each monitoring record, and the user can also view it on the service robot to evaluate the sleep quality of the monitored person over a period of time.

[0167] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects: the technical solution acquires the sleep images of the monitored person, and then inputs the acquired sleep images into a sleep detection model for detection, thereby obtaining the sleep state of the monitored person, and controls the electronic device to execute different monitoring strategies according to the different sleep states of the monitored person. It can not only accurately identify the abnormal sleep state of the monitored person, but also respond to various abnormal sleep states in a timely manner, thereby reducing the monitoring pressure of the user and ensuring the sleep safety of the monitored person.

[0168] In some embodiments, when the service robot is monitoring the monitored body, it can monitor the environmental parameters in the target area in real time, and adjust the environmental parameters according to the current environmental parameters to keep the environment in the target area suitable. Fig.11 A flowchart of another sleep monitoring method provided in an embodiment of the present application is used to adjust the environmental parameters of the target area, such as Fig.11 As shown, the method comprises the following steps:

[0169] S501. The controller obtains the ambient temperature, ambient humidity, light intensity and air quality index of the target area.

[0170] Optionally, the controller may obtain the temperature of the target area through a temperature sensor, obtain the humidity of the target area through a temperature sensor, obtain the light intensity of the target area through a light intensity sensor, and obtain the air quality index of the target area through an air quality sensor.

[0171] S502: When the ambient temperature, ambient humidity, light intensity or air quality index is not within the preset ambient parameter range, control the ambient parameter regulator to adjust the ambient temperature, ambient humidity, light intensity or the air quality index of the target area.

[0172] The preset environmental parameter range includes a preset temperature range, a preset humidity range, a preset light intensity range, and a preset air quality index range.

[0173] It should be noted that the preset environmental parameter range is preset by the electronic device manufacturer and stored in the memory. Different electronic device manufacturers set different preset environmental parameter ranges, and this application does not limit this.

[0174] Optionally, the temperature range may be [22° C.-24° C.], the humidity range may be 50%, the light intensity range may be [40 lx, 60 lx], and the air quality index range may be [0, 35 ug / m3].

[0175] In some embodiments, when the ambient temperature, ambient humidity, light intensity or air quality index is not within the preset environmental parameter range, the environmental parameter regulator is controlled to adjust the ambient temperature, ambient humidity, light intensity or the air quality index of the target area until the ambient temperature, ambient humidity, light intensity or the air quality index is within the preset environmental parameter range.

[0176] For example, when the temperature T of the target area t When it is greater than 24℃, that is, T t When the temperature is higher than 24°C, the controller controls the environmental parameter regulator to lower the air outlet temperature of the air conditioner in the target area to reduce the temperature of the target area until the temperature of the target area is within the preset temperature range.

[0177] Or, when the temperature T of the target area t When it is less than 22℃, that is, T t When the temperature is less than 22°C, the controller controls the environmental parameter regulator to increase the air outlet temperature of the air conditioner in the target area to increase the temperature of the target area until the temperature of the target area is within the preset temperature range.

[0178] For example, when the humidity H of the target area t When it is greater than 50%, that is, H t When the humidity is greater than 50%, the controller turns on the dehumidification mode of the air conditioner in the target area to reduce the humidity in the target area until the humidity in the target area is within the preset humidity range.

[0179] Or, when the humidity H in the target area t When it is less than 50%, that is, H t When the humidity is less than 50%, the controller turns on the humidifier to increase the humidity in the target area until the humidity in the target area is within the preset humidity range.

[0180] For example, when the light intensity L in the target area t When it is greater than 60lx, that is, L t When the temperature is higher than 60lx, the controller reduces the opening of the electric curtains in the target area or lowers the light intensity in the target area to reduce the light intensity in the target area until the light intensity in the target area is within the preset light intensity range.

[0181] Or, when the temperature L of the target area t When it is less than 40lx, that is, L t When the light intensity is less than 40lx, the controller increases the opening of the electric curtains in the target area or increases the light intensity in the target area to increase the light intensity in the target area until the light intensity in the target area is within the preset light intensity range.

[0182] Exemplarily, when the air quality index AQI of the target area is greater than 35ug / m3, that is, AQI>35ug / m3, the controller turns on the air purifier in the target area to lower the air quality index of the target area until the air quality index of the target area is within the preset air quality index range.

[0183] In some embodiments, when the user issues a command to stop monitoring by voice or by clicking on the display screen of the service robot, the controller controls the service robot to turn off the air conditioner, humidifier, air purifier and open the curtains. At the same time, the service robot displays the sleep monitoring report of the monitored person on the display screen.

[0184] Optionally, the sleep monitoring report may include information such as changes in room temperature, humidity, total sleep duration, time and frequency of crying, face covered, kicking off the quilt, falling out of bed, etc., for the user to review. This application does not limit the content of the sleep monitoring report.

[0185] In some embodiments, the electronic device can also determine whether the monitored person has woken up through the sleep image of the monitored person. This technology is based on the OpenCV framework. After converting the format of the original sleep image into the mat format, the multi-frame difference method of the video stream is used. The sleep image is a two-dimensional rectangular data. N frames of data are set, and the image rectangles between frames are subtracted from each other.

[0186] Get the difference image, set the threshold thr, compare the size of the pixels in the difference image with thr, if it is greater than thr, the pixel value is set to 255 (255 is the area where the monitored human body is active), otherwise it is set to 0.

[0187] Furthermore, the number of pixels sum with a value of 255 is extracted, and the activity threshold thr2 is set. If sum>thr2, it is determined that the monitored person is in an awake state, and if sum≤thr2, it is determined that the monitored person is in a deep sleep state.

[0188] In some embodiments, the above step S201 may also be performed as follows: Fig.12 The method shown is implemented as Fig.12 As shown, when the service robot starts the monitoring task, the service robot navigates to the child's room and looks for a suitable shooting angle. After that, the image acquisition device starts monitoring and collects the facial image of the monitored person. If the face can be recognized and detected, it is determined whether the face of the monitored person is covered based on the facial image. If the face cannot be recognized or is covered, the service robot sends a reminder to the user and searches for people throughout the house for voice alarm.

[0189] Alternatively, if the face is not covered, the image acquisition device of the service robot continues to monitor.

[0190] In some embodiments, the facial temperature of the monitored person can be detected by a thermal imaging thermometer. When the face can be recognized and the facial temperature can be detected, if the facial temperature is ≥37°C or ≤35°C, the service robot sends a reminder to the user and searches the whole house for people to issue a voice alarm. Otherwise, the thermal imaging thermometer continues to monitor the facial temperature of the monitored person.

[0191] In some embodiments, the above steps S201-S203 and determining whether the monitored person has woken up can also be performed as follows: Fig.13 The method shown is implemented. Fig.13 As shown, when the monitoring task conditions of the service robot are triggered, the image acquisition device is driven and the detection model is loaded, and then each frame of image data is called back to identify whether the person being monitored is asleep or awake, and output the recognition result.

[0192] On the other hand, after calling back each frame of image data, model inference begins, and the model results are parsed to obtain information about the face and body. According to the peripheral rule settings and combined with video stream processing, the event results of the face, kicking the quilt, and falling out of bed are output.

[0193] Furthermore, the business layer functions are performed according to the results of the algorithm output to determine whether the user needs to stop monitoring. If so, the model and image acquisition device are destroyed (that is, the resources occupied by the model and image acquisition device are released, and the image acquisition device is turned off to avoid invalid occupation of resources and infringement of user privacy), and the monitoring is ended; if not, each frame of image data is called back again.

[0194] In some embodiments, the above steps S202-S203 can also be performed as follows: Fig.14 The method shown is implemented. Fig.14 As shown, after the service robot starts the monitoring task, it navigates to the children's room to find the baby and to the best monitoring position, then starts the environmental control program, adjusts the light intensity by adjusting the opening of the curtains, plays lullaby music to lull the baby to sleep, and after the baby falls asleep, turns off the lullaby music and monitors whether the baby kicks the quilt and whether the baby cries. If the baby kicks the quilt, the user is notified and the temperature is increased. If the baby cries, the user is notified and lullaby audio is played.

[0195] Alternatively, if the service robot cannot find the baby or cannot navigate to the best monitoring position, it will navigate to the default position for monitoring, and if the baby is not asleep, it will continue to play music, and if the baby does not kick the quilt or cry, it will continue to monitor.

[0196] Furthermore, if the user requests streaming, the video stream is pushed to the server for the user to view. If the user closes the monitoring task, the sleep monitoring report is displayed and the monitoring task is ended.

[0197] In some embodiments, the above steps S201-S202 can also be performed as follows: Fig.15 The method shown is implemented. Fig.15 As shown, the service robot starts the monitoring task, the model loads and drives the image acquisition device, and after calling back to obtain each frame of image data, it combines human body detection and key point detection algorithms to integrate peripheral information to determine the baby's kicking status.

[0198] Alternatively, by combining face detection with key point detection algorithms, the facial coverage status of the infant can be determined based on the integration of peripheral information.

[0199] Further, the state of the baby kicking the quilt and the state of the face covering are output, which is true if yes and false if no. Then it is determined whether the user has issued an instruction to stop monitoring. If yes, the model and the image acquisition device are destroyed and the monitoring task is terminated. If no, the image acquisition device is continued to be driven.

[0200] An embodiment of the present invention further provides a computer-readable storage medium, which includes computer-executable instructions. When the computer-executable instructions are executed on a computer, the computer executes the method provided in the above embodiment.

[0201] An embodiment of the present invention further provides a computer program product, which can be directly loaded into a memory and contains software code. After being loaded and executed by a computer, the computer program product can implement the method provided in the above embodiment.

[0202] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented using software, the functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium may be any available medium that a general or special-purpose computer can access.

[0203] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0204] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely exemplary, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed in multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0205] In addition, each functional unit in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a ROM, a RAM, a disk, or an optical disk.

[0206] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An electronic device, characterized in that: include: An image acquisition device, used to acquire an image of a target area, where a monitored human body is located; A thermal imaging thermometer, used to detect the facial temperature of the monitored person; An audio collection device, used to collect audio information of the monitored person; The controller is configured as: Acquire the sleeping image of the monitored person through the image acquisition device, acquire the facial temperature of the monitored person through the thermal imaging thermometer, and acquire the sleeping audio of the monitored person through the audio acquisition device; Determining the sleep state of the monitored person according to the sleep image, the facial temperature, the sleep audio, and the sleep detection model; Determining a target monitoring strategy of the electronic device according to the sleep state of the monitored person; the target monitoring strategy is used to process the sleep state of the monitored person; The target monitoring strategy is executed.

2. The electronic device according to claim 1, characterized in that: The sleep detection model includes a face covering detection model, a quilt kicking detection model and a crying detection model, and the sleep state includes whether the face of the monitored person is covered, whether the monitored person kicks the quilt and whether the monitored person cries; The controller is configured to determine the sleep state of the monitored person according to the sleep image, the facial temperature, the sleep audio, and the sleep detection model, and is specifically configured to: Inputting the sleep image into the face coverage detection model to obtain a face coverage detection result of the monitored person; the face coverage detection result includes that the face of the monitored person is covered or that the face of the monitored person is not covered; or, The sleep image is input into the quilt kicking detection model to obtain a quilt kicking detection result of the monitored person; the quilt kicking detection result includes whether the monitored person has kicked the quilt or whether the monitored person has not kicked the quilt; or, The sleep audio is input into a crying detection model to obtain a crying detection result of the monitored person; the crying detection result includes whether the monitored person has cried or whether the monitored person has not cried.

3. The electronic device according to claim 2, characterized in that: The electronic device further includes: An environmental parameter regulator, used to adjust the environmental parameters of the target area; The controller is configured to determine a target monitoring strategy according to the sleep state of the monitored person, and is specifically configured to: When the face cover detection result of the monitored person is that the face of the monitored person is covered or the face temperature is not within a preset temperature range, determining the target monitoring strategy of the electronic device to issue a reminder to the user; or, In the case where the result of the detection of the monitored person kicking the quilt is that the monitored person has kicked the quilt, determining the target monitoring strategy of the electronic device to control the environmental parameter regulator to increase the temperature of the target area and send a reminder to the user; or When the crying result of the monitored person is that the monitored person has cried, the target monitoring strategy of the electronic device is determined to send a reminder to the user and start a sleep-coaxing function.

4. The electronic device according to claim 1, characterized in that: The controller is further configured to: Acquire the position of the monitored body and the bed in the sleep image; Determining whether the monitored person is at risk of falling out of bed according to the positional relationship between the position of the monitored person and the position of the bed; In the case that the monitored person is at risk of falling out of bed, the target monitoring strategy of the electronic device is determined to issue a reminder to the user.

5. The electronic device according to claim 1, characterized in that: The electronic device further includes: A temperature sensor, used to detect the ambient temperature of the target area; A humidity sensor, used to detect the ambient humidity of the target area; A light intensity sensor, used to detect the light intensity of the target area; An air quality sensor, used to detect the air quality index of the target area; The controller is further configured to: Acquire the ambient temperature of the target area through the temperature sensor, acquire the ambient humidity of the target area through the humidity sensor, acquire the light intensity of the target area through the light intensity sensor, and acquire the air quality index of the target area through the air quality sensor; When the ambient temperature, ambient humidity, light intensity or air quality index is not within the preset environmental parameter range, the environmental parameter regulator is controlled to adjust the ambient temperature, ambient humidity, light intensity or air quality index of the target area until the ambient temperature, ambient humidity, light intensity or air quality index is within the preset environmental parameter range.

6. A sleep monitoring method, characterized in that: The method comprises: Obtain the sleep image, facial temperature and sleep audio of the monitored person; Determining the sleep state of the monitored person according to the sleep image, the facial temperature, the sleep audio, and the sleep detection model; Determining a target monitoring strategy of the electronic device according to the sleep state of the monitored person; the target monitoring strategy is used to process the sleep state of the monitored person; The target monitoring strategy is executed.

7. The method according to claim 6, characterized in that The sleep detection model includes a face covering detection model, a quilt kicking detection model and a crying detection model, and the sleep state includes whether the face of the monitored person is covered, whether the monitored person kicks the quilt and whether the monitored person cries; The step of determining the sleep state of the monitored person according to the sleep image, the facial temperature, the sleep audio, and the sleep detection model includes: Inputting the sleep image into the face coverage detection model to obtain a face coverage detection result of the monitored person; the face coverage detection result includes that the face of the monitored person is covered or that the face of the monitored person is not covered; or, The sleep image is input into the quilt kicking detection model to obtain a quilt kicking detection result of the monitored person; the quilt kicking detection result includes that the monitored person has kicked the quilt or that the monitored person has not kicked the quilt; or, The sleep audio is input into a crying detection model to obtain a crying detection result of the monitored person; the crying detection result includes whether the monitored person has cried or whether the monitored person has not cried.

8. The method according to claim 7, characterized in that The step of determining a target monitoring strategy of the electronic device according to the sleep state of the monitored person includes: When the face cover detection result of the monitored person is that the face of the monitored person is covered or the face temperature is not within a preset temperature range, determining the target monitoring strategy of the electronic device to issue a reminder to the user; or, In the case where the result of the detection of the monitored person kicking the quilt is that the monitored person has kicked the quilt, determining the target monitoring strategy of the electronic device to control the environmental parameter regulator to increase the temperature of the target area, and sending a reminder to the user; or When the crying result of the monitored person is that the monitored person has cried, the target monitoring strategy of the electronic device is determined to send a reminder to the user and start a sleep-coaxing function.

9. The method according to claim 6, characterized in that The method further comprises: Acquire the position of the monitored body and the bed in the sleep image; Determining whether the monitored person is at risk of falling out of bed according to the positional relationship between the position of the monitored person and the position of the bed; In the case that the monitored person is at risk of falling out of bed, the target monitoring strategy of the electronic device is determined to issue a reminder to the user.

10. The method according to claim 6, characterized in that The method further comprises: Obtain the ambient temperature, ambient humidity, light intensity and air quality index of the target area; When the ambient temperature, ambient humidity, light intensity or air quality index is not within the preset environmental parameter range, the environmental parameter regulator is controlled to adjust the ambient temperature, ambient humidity, light intensity or air quality index of the target area until the ambient temperature, ambient humidity, light intensity or air quality index is within the preset environmental parameter range.

Images