Sleep monitoring method, large model fine tuning method and related equipment

Abstract

The embodiment of the invention discloses a sleep monitoring method, a large model fine tuning method and related equipment, and relates to the technical field of smart home. The method comprises the following steps: in response to a sleep monitoring instruction, periodically obtaining multiple pieces of multi-modal data corresponding to a monitored user based on first time; according to each piece of multi-modal data, generating first prompt information corresponding to the multi-modal data; wherein the first prompt information is used for indicating the large model to judge whether the monitored user enters a sleep state or not based on the multi-modal data; the multiple pieces of first prompt information are sequentially input into the large model, and sleep judgment results corresponding to the first prompt information output by the large model are obtained; and when a sleep judgment result representing that the monitored user enters the sleep state is detected, recording a sleep event, and stopping judging whether the monitored user enters the sleep state or not through the large model. According to the method, whether the monitored user falls asleep or not is judged through the large model, the judgment accuracy can be effectively improved, and important nodes of the sleep cycle are monitored in time.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a sleep monitoring method, a large model fine-tuning method, and related equipment. Background Art

[0002] It is very important to monitor the sleep of groups that require care, such as infants, toddlers, and the elderly. For example, for infants, healthy sleep patterns contribute to brain development, emotional regulation, learning ability, and physical growth. Good sleep is essential for infant growth and development. Sleep monitoring for these groups can ensure their safety and identify health issues that may affect sleep, such as apnea and sudden infant death syndrome (SIDS), early on. Sleep monitoring for infants can also help parents understand their babies' natural sleep cycles and adjust their sleep schedules to promote better sleep quality.

[0003] However, the current method of sleep monitoring for groups that need care, such as infants or the elderly, is very simple. It only involves setting up monitoring equipment including cameras and microphones in the sleeping area so that the monitoring user can view the sleep status of the monitored user through the monitoring equipment. Summary of the Invention

[0004] The embodiments of the present application provide a sleep monitoring method, a large model fine-tuning method, and related equipment to solve the above problems. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a sleep monitoring method, the method comprising:

[0006] In response to the sleep monitoring instruction, periodically acquiring a plurality of multimodal data corresponding to the monitored user based on a first time period;

[0007] Generate first prompt information corresponding to each multimodal data according to each multimodal data; wherein the first prompt information is used to instruct the large model to determine whether the monitored user has entered a sleep state based on the multimodal data;

[0008] sequentially inputting a plurality of the first prompt information into the large model, and obtaining a sleep determination result corresponding to each of the first prompt information output by the large model;

[0009] When a sleep judgment result indicating that the monitored user has entered a sleep state is detected, a sleep event is recorded, and the judgment of whether the monitored user has entered a sleep state by using the large model is stopped.

[0010] In a second aspect, an embodiment of the present application provides a large model fine-tuning method, wherein the large model is used to determine whether the monitored user has entered a sleep state in the sleep monitoring method of the first aspect, the method comprising:

[0011] Obtain the large model to be fine-tuned and multiple multimodal sample data;

[0012] Generate sample prompt information corresponding to the multimodal sample data based on the prompt information template to be fine-tuned and the multimodal sample data; wherein the sample prompt information is used to instruct the large model to be fine-tuned to determine whether the target user corresponding to the multimodal sample data has entered a sleep state based on the multimodal sample data;

[0013] Inputting the sample prompt information into the large model to be fine-tuned, and obtaining a sleep judgment result corresponding to the sample prompt information output by the large model to be fine-tuned;

[0014] The large model and the prompt information template to be fine-tuned are fine-tuned according to the difference information between the sleep judgment result corresponding to the sample prompt information and the actual sleep state of the target user until the fine-tuning conditions are met, thereby obtaining the large model and prompt information template; wherein the prompt information template is used to generate the first prompt information corresponding to the multimodal data according to the multimodal data.

[0015] In a third aspect, an embodiment of the present application provides a sleep monitoring device, comprising:

[0016] a first sleep monitoring module, configured to periodically acquire a plurality of multimodal data corresponding to a monitored user based on a first time period in response to a sleep monitoring instruction;

[0017] a second sleep monitoring module, configured to generate, based on each of the multimodal data, first prompt information corresponding to the multimodal data; wherein the first prompt information is used to instruct the large model to determine whether the monitored user has entered a sleep state based on the multimodal data;

[0018] a third sleep monitoring module, configured to sequentially input a plurality of the first prompt information into the large model, and obtain a sleep determination result corresponding to each of the first prompt information output by the large model;

[0019] The fourth sleep monitoring module is configured to record a sleep event and stop determining whether the monitored user has entered a sleep state by using the large model when a sleep judgment result indicating that the monitored user has entered a sleep state is detected.

[0020] In a fourth aspect, an embodiment of the present application provides a large model fine-tuning device, the device comprising:

[0021] A first model fine-tuning module is used to obtain a large model to be fine-tuned and a plurality of multimodal sample data;

[0022] A second model fine-tuning module is configured to generate sample prompt information corresponding to the multimodal sample data based on the prompt information template to be fine-tuned and the multimodal sample data; wherein the sample prompt information is used to instruct the large model to be fine-tuned to determine whether the target user corresponding to the multimodal sample data has entered a sleep state based on the multimodal sample data;

[0023] a third model fine-tuning module, configured to input the sample prompt information into the large model to be fine-tuned, and obtain a sleep judgment result corresponding to the sample prompt information output by the large model to be fine-tuned;

[0024] A fourth model fine-tuning module is used to fine-tune the large model and the prompt information template to be fine-tuned according to the difference information between the sleep judgment result corresponding to the sample prompt information and the actual sleep state of the target user, until the fine-tuning conditions are met, thereby obtaining the large model and the prompt information template; wherein, the prompt information template is used to generate the first prompt information corresponding to the multimodal data based on the multimodal data, and the large model is used to determine whether the monitored user has entered a sleep state in the sleep monitoring method described in the first aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0026] In a sixth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0027] The beneficial effects of the technical solutions provided by some embodiments of the present application include at least:

[0028] In this application, in response to a sleep monitoring instruction, multiple multimodal data corresponding to a monitored user are periodically acquired based on a first time. The multimodal data includes multiple types of data specific to the monitored user, such as image data recording the monitored user's body posture and facial expressions. Furthermore, a first prompt message corresponding to each multimodal data item is generated, the first prompt message being used to instruct a large model to determine whether the monitored user has entered a sleep state based on the multimodal data. The multiple first prompt messages are sequentially input into the large model to obtain a sleep determination result output by the large model. Furthermore, when the monitored user is detected to have entered a sleep state based on the sleep determination result, a sleep event is recorded, and the large model is stopped from determining whether the monitored user has entered a sleep state. A sleep event may include information such as the time, posture, and location of the monitored user entering a sleep state. In this application, determining whether the monitored user is asleep using the large model can effectively improve the accuracy of the determination, timely monitor important nodes in the sleep cycle, and, by recording sleep events, enable the monitoring user to understand the monitored user's sleep time, posture, and location, understand the monitored user's natural sleep patterns, and improve the monitored user's sleep quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 This is a schematic diagram of the architecture of a sleep monitoring method provided in an embodiment of the present application;

[0031] Figure 2 This is a flowchart of a sleep monitoring method provided in an embodiment of the present application;

[0032] Figure 3 This is a schematic diagram of a sleep monitoring method provided in an embodiment of the present application;

[0033] Figure 4 This is a flowchart of a sleep monitoring method provided in an embodiment of the present application;

[0034] Figure 5 This is a page diagram of a sleeping companion function provided by an embodiment of the present application;

[0035] Figure 6 This is a flowchart of a sleep monitoring method provided in an embodiment of the present application;

[0036] Figure 7This is a schematic diagram of a page for recording sleep events provided by an embodiment of the present application;

[0037] Figure 8 This is a schematic diagram of a multi-system interaction provided by an embodiment of the present application;

[0038] Figure 9 This is a flow chart of a large model fine-tuning method provided in an embodiment of the present application;

[0039] Figure 10 This is a schematic structural diagram of a large model fine-tuning device provided in an embodiment of the present application;

[0040] Figure 11 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0042] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance. In the description of this application, it should be noted that, unless otherwise expressly specified and limited, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. "and / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0043] The present application is described in detail below with reference to specific embodiments.

[0044] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the features, information, and data involved in this application are all obtained with full authorization.

[0045] It is very important to monitor the sleep of groups that require care, such as infants, toddlers, or the elderly. For example, for infants, healthy sleep patterns contribute to brain development, emotional regulation, learning ability, and physical growth. Good sleep is essential for the growth and development of infants. Sleep monitoring for these groups that require care can ensure their safety and identify health issues that may affect sleep, such as apnea and sudden infant death syndrome (SIDS), at an early stage. Sleep monitoring for infants can also help parents understand their infants' natural sleep cycles and adjust their sleep schedules to promote better sleep quality.

[0046] However, the current method of sleep monitoring for groups that need care, such as infants or the elderly, is very simple. It only involves setting up monitoring equipment including cameras and microphones in the sleeping area so that the monitoring user can view the sleep status of the monitored user through the monitoring equipment.

[0047] In view of the above problems, this application proposes a sleep monitoring method to solve them. Figure 1 As shown, Figure 1 This is a scenario diagram of a sleep monitoring method provided in an embodiment of the present application. Figure 1 At least includes a server 101 and multiple electronic devices. The multiple electronic devices include at least electronic device 1021, electronic device 1022 and electronic device 1023. It is understandable that, Figure 1 The number of servers and electronic devices shown is for illustration only and is not limited in this embodiment of the present application.

[0048] In one embodiment, the electronic device can be used to receive a sleep monitoring instruction and send the sleep companion instruction to the server 101, so that the server 101 executes the sleep monitoring method in response to the sleep monitoring instruction. In another embodiment, a large model is deployed on the server 101, and the electronic device receives the sleep monitoring instruction and periodically obtains multiple multimodal data corresponding to the monitored user based on a first time, generates first prompt information corresponding to each multimodal data according to the multimodal data, and sends multiple first prompt information to the server 101, so that the server 101 sequentially inputs the multiple first prompt information into the large model to obtain a sleep judgment result corresponding to each first prompt information output by the large model. The electronic device receives the multiple sleep judgment results and, upon detecting a sleep judgment result indicating that the monitored user has entered a sleep state, records a sleep event and stops using the large model to determine whether the monitored user has entered a sleep state.

[0049] The above-mentioned server 101 can be a separate server device, such as: a rack-mounted, blade, tower, or cabinet-mounted server device, or a hardware device with strong computing power such as a workstation or a mainframe computer; it can also be a server cluster composed of multiple servers. The servers in the service cluster can be composed in a symmetrical manner, wherein each server has equivalent functions and status in the transaction link, and each server can provide services to the outside world independently. Providing services independently can be understood as not requiring the assistance of other servers.

[0050] For example, the server may be multiple physical servers that are independent in hardware. Alternatively, the server may be multiple virtual servers that are deployed in the same hardware resource pool. Virtual server deployment methods include, but are not limited to, VMware, Virtual Box, and Virtual PC.

[0051] It is understood that the server 101 also has other service capabilities and functions to complete the tasks in the following embodiments. For example, the server 101 also provides portal services, resource management services, and CI / CD services.

[0052] Electronic devices include, but are not limited to, wearable devices, handheld devices, personal computers, tablets, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. Electronic devices may be referred to by different names in different networks, such as user equipment, access terminals, subscriber units, subscriber stations, mobile stations, mobile stations, remote stations, remote terminals, mobile devices, user terminals, terminals, wireless communication devices, user agents or user devices, cellular phones, cordless phones, personal digital assistants (PDAs), and electronic devices in 5G networks or future evolution networks.

[0053] In the embodiment of the present application, electronic devices such as electronic device 1021, electronic device 1022, and electronic device 1023 may also be equipped with a display device. The display device may be any device capable of realizing a display function, for example, a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), a plasma display panel (PDP), etc. For example, a user may use the display device on electronic device 1021 to send a call request for a certain application to server 101. The call request corresponds to a certain call behavior. In other words, the user requests to call the application through the call method corresponding to the call behavior.

[0054] Multiple electronic devices and multiple servers can communicate with each other through communication links established by a communication protocol, for example: wherein the network can be a wireless network or a wired network, the wireless network includes but is not limited to a cellular network, a wireless local area network, an infrared network or a Bluetooth network, and the wired network includes but is not limited to an Ethernet, a universal serial bus (USB) or a controller area network. In one or more embodiments of the specification, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. are used to represent data (such as a target compressed package) exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0055] like Figure 2FIG. 1 is a schematic diagram of a sleep monitoring method according to an embodiment of the present application. An electronic device 1021 is configured to receive a sleep monitoring instruction and execute the sleep monitoring method, thereby accompanying a child, a monitored user 103, from a sleep-ready state to a sleep state. The sleep-ready state indicates that the monitored user 103 is preparing to fall asleep, while the sleep state indicates that the monitored user 103 has fallen asleep.

[0056] In one embodiment, Figure 3 The figure shows a flowchart of a sleep monitoring method provided by an embodiment of the present application. The method can be implemented by a computer program and can be run on a sleep monitoring device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone tool application.

[0057] Specifically, the sleep monitoring method includes:

[0058] S101 : In response to a sleep monitoring instruction, periodically obtain a plurality of multimodal data corresponding to a monitored user based on a first time period.

[0059] In this application, sleep monitoring instructions can be received through user input and preset conditions. The user can directly input instructions through voice, virtual interaction, touch interaction, and other interactive methods. For example, the monitoring user can send a sleep monitoring instruction to the electronic device by saying "Start sleep monitoring mode" or clicking a control on the target page of the target application through the electronic device.

[0060] Alternatively, the receipt of a sleep monitoring instruction may be triggered based on preset conditions. For example, if the electronic device detects changes in the monitored user's behavior or environment through sensors (such as motion sensors or light sensors), such as the monitored user lying down in the sleeping area or the ambient light dimming, the electronic device may automatically infer that the monitored user has entered a sleep-ready state. If the preset conditions for triggering a sleep monitoring instruction are met, the sleep monitoring instruction will be triggered.

[0061] The term "sleep area" can be understood as the space or location where the monitored user sleeps, such as a bed or sofa. In this application, it specifically refers to a specific area monitored by an electronic device. Within this area, the electronic device uses a camera module to collect information such as the monitored user's facial expressions and body movements to determine whether the monitored user is in a sleep state or in a sleep state. The sleep area can be preset or specified by a monitoring user with designated permissions, and this is not limited in this application.

[0062] It is understandable that the terms "guardian user" and "guarded user" are relative terms. For example, a guarded user is a younger child and a guardian is a parent. A method for determining a user as a guarded object may be to designate a guardian user with designated authority, which is not limited in this application.

[0063] In response to the sleep monitoring instruction, a plurality of multimodal data corresponding to the monitored user is periodically acquired based on a first time period. The first time period can be any time period as a period, for example, the first time period is a five-minute period or a three-minute period.

[0064] Multimodal data includes multiple types of data. For example, multimodal data includes at least one of the following types: image data, sound data, temperature data, and infrared data. Specifically, multimodal data includes sound data such as snoring and sleep talking, image data including facial expressions and body movements, heart rate data collected by smart bracelets, mattress sensors, or dedicated heart rate monitoring devices, respiratory rate measured by pressure pads, chest straps, or other biosensors, or infrared data collected by infrared cameras.

[0065] S102: Generate first prompt information corresponding to each multimodal data according to the multimodal data.

[0066] The first prompt information is used to instruct the large model to determine whether the monitored user has entered a sleep state based on the multimodal data. The large model can be any machine learning model that can process multimodal data. For example, the large model is a ResNet network model, a YOLO (You Only Look Once) model, a ViT (Vision Transformer) model, a CLIP (Contrastive Language–Image Pretraining), a GCN (Graph Convolutional Networks) model, etc. The first prompt information is generated using a preset prompt template and the specific content of the multimodal data.

[0067] In one embodiment, prompt instruction templates are respectively corresponding to multiple data types in the multimodal data to obtain prompt instructions corresponding to each type of data; wherein the prompt instruction is used to prompt the large model to determine whether the monitored user has entered a sleep state based on the data of the type corresponding to the prompt instruction; and a first prompt message is generated based on multiple prompt instructions and multimodal data.

[0068] In other words, the first prompt information includes multiple prompt instructions, each of which is used to instruct the large model to determine whether the monitored user has entered a sleep state based on different types of data. For example, if the multimodal data includes an image type, the prompt instruction generated based on the image type data is used to instruct the large model to determine whether the monitored user has entered a sleep state based on the image.

[0069] For example, the specific content of the prompt instruction is: extracting facial features and body posture features of the monitored user in the image, and judging whether the monitored user has entered a sleeping state based on the facial features.

[0070] Based on the prompt, the large model extracts facial features from the facial information in the image using a keypoint algorithm or a deep learning-based facial keypoint detection algorithm (such as MTCNN and Dlib). Keypoint algorithms accurately describe facial features by locating a series of key points on the face (such as the corners of the eyes, corners of the mouth, and the tip of the nose). Furthermore, the large model determines the monitored user's facial expression based on the extracted facial features. Facial expressions can be understood as observable changes on the face that can indicate whether the monitored user is asleep. The large model pre-defines a series of facial feature combinations that indicate the monitored user may be asleep, including but not limited to the state of the eyes (open or closed), the position of the eyebrows, and the shape of the mouth. The state of the eyes, in particular, is a key factor in determining whether a person is asleep. Furthermore, the large model acquires body posture features from the image. These body posture features represent specific information about the monitored user's current body posture, including, for example, whether the user is lying flat, lying on their side, lying on their back, or standing up. Combining facial and body posture features, the large model can determine whether the monitored user is asleep.

[0071] In this embodiment, different prompt instructions are determined by different data types, so that a large model based on multiple prompt instructions can be used to judge whether the monitored user has entered a sleep state from multiple different types of data, and the judgment results corresponding to multiple prompt instructions are analyzed to obtain the final sleep judgment result of whether the monitored user has entered a sleep state, which can effectively utilize multimodal data.

[0072] S103 , inputting a plurality of first prompt information into the large model in sequence, and obtaining a sleep determination result corresponding to each first prompt information output by the large model.

[0073] Multiple first prompt information is sequentially input into the large model, and a sleep determination result corresponding to each first prompt information output by the large model is obtained. A sleep determination result obtained by the large model by analyzing multiple consecutive sleep determination results can also be obtained. For example, the large model determines that the monitored user's eyes have been closed for a period of time (e.g., several minutes) based on multiple consecutive sleep determination results, tracks changes in the monitored user's head position through infrared data, and infers that the monitored user's head is in a stationary state and that the length of time in this stationary state exceeds a preset time (e.g., more than ten minutes). The large model then outputs a sleep determination result indicating that the monitored user has entered a sleep state.

[0074] S104: When a sleep judgment result indicating that the monitored user has entered a sleep state is detected, the sleep event is recorded, and the judgment of whether the monitored user has entered a sleep state by using the large model is stopped.

[0075] A sleep event includes at least information such as the time, location, and posture of the monitored user when they fall asleep. Upon detecting a sleep judgment result indicating that the monitored user has fallen asleep, the system continuously collects multiple multimodal data corresponding to the monitored user to continuously monitor their sleep status. However, the system stops sending the first prompt text corresponding to the multimodal data to the large model, thereby conserving computing resources.

[0076] In this application, in response to a sleep monitoring instruction, multiple multimodal data corresponding to a monitored user are periodically acquired based on a first time. The multimodal data includes multiple types of data specific to the monitored user, such as image data recording the monitored user's body posture and facial expressions. Furthermore, a first prompt message corresponding to each multimodal data item is generated, the first prompt message being used to instruct a large model to determine whether the monitored user has entered a sleep state based on the multimodal data. The multiple first prompt messages are sequentially input into the large model to obtain a sleep determination result output by the large model. Furthermore, when the monitored user is detected to have entered a sleep state based on the sleep determination result, a sleep event is recorded, and the large model is stopped from determining whether the monitored user has entered a sleep state. A sleep event may include information such as the time, posture, and location of the monitored user entering a sleep state. In this application, determining whether the monitored user is asleep using the large model can effectively improve the accuracy of the determination, timely monitor important nodes in the sleep cycle, and, by recording sleep events, enable the monitoring user to understand the monitored user's sleep time, posture, and location, understand the monitored user's natural sleep patterns, and improve the monitored user's sleep quality.

[0077] In one embodiment, Figure 4The figure shows a flowchart of a sleep monitoring method provided by an embodiment of the present application. The method can be implemented by a computer program and can be run on a sleep monitoring device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone tool application.

[0078] Specifically, the sleep monitoring method includes:

[0079] S201. In response to a sleep companion instruction, start a sleep companion function of an electronic device to accompany a monitored user into a sleep state.

[0080] Wherein, starting the sleep companion function of the electronic device at least includes controlling the electronic device to emit a sleep companion light or play a sleep companion music. In response to the sleep companion instruction, the light module of the electronic device is started to emit a sleep companion light of a preset color and / or preset brightness, and the sound module of the electronic device is started to play a sleep companion music of a preset volume and / or preset content. Figure 2 As shown, the electronic device 1021 emits sleep-inducing light through the light module to create a soft and warm environment, and activates the sound module to play sleep-inducing music such as light music, white noise, fairy tales, etc. to guide the monitored user 103 to fall asleep.

[0081] Obtain light selection information in the sleep companion instruction, start the light module to emit sleep companion light that matches the light selection information; wherein the light selection information represents the brightness and / or color of the sleep companion light; obtain music selection information in the sleep companion instruction, start the sound module to play sleep companion music related to the music selection information; the content of the sleep companion music can be white noise, ambient sound, pure music or fairy tales, etc., or other content.

[0082] Sleep companion instructions carry light selection information and music selection information. This information can be provided by pre-setting multiple color and brightness combinations, obtaining a target combination from the multiple combinations based on the user's input, and determining specific color and brightness values ​​based on this target combination. Alternatively, the information can be provided by separately setting brightness and color, with the specific brightness value and color type determined based on the user's input.

[0083] The sleep companion instruction can include music selection information by presetting multiple combinations of content and volume, obtaining a target combination from the multiple combinations based on the monitoring user's input data, and determining specific values ​​for the content and volume based on the target combination. The sleep companion instruction can also include volume selection information by separately setting the content and volume, and determining the specific brightness value and content content based on the monitoring user's input data.

[0084] like Figure 5As shown, Figure 5 This is a schematic diagram of a page for a sleep companion function provided by an embodiment of the present application. Figure 5 The triggering instruction of the target control on the target page shown determines the light selection information, which represents the sleeping light brightness of 80% and the color of warm yellow. Figure 5 The trigger instruction of the target control on the target page shown determines the music selection information, which indicates that the content of the sleep music is a fairy tale and the volume is 80%.

[0085] S202: When it is detected that the duration of activating the sleep companion function exceeds a preset activation duration, a sleep monitoring instruction is obtained.

[0086] The preset activation duration can be understood as a pre-set time threshold. The preset activation duration can be any set duration, for example, 5 minutes, 10 minutes, or 30 minutes. When it is detected that the duration of activating the sleep companion function to accompany the monitored user to sleep exceeds the preset activation duration, the monitored user gradually falls asleep, thus triggering the sleep monitoring instruction and starting to obtain the monitored user's multimedia data to determine whether the monitored user has completely fallen asleep.

[0087] In one embodiment, activating the sleep-accompanying function of an electronic device includes controlling the electronic device to emit sleep-accompanying lights and play sleep-accompanying music; in response to a sleep-accompanying instruction, after activating the sleep-accompanying function of the electronic device to accompany the monitored user into a sleep state, it also includes: adjusting the brightness of the sleep-accompanying lights and switching the content and / or volume of the sleep-accompanying music during the sleep-accompanying time period.

[0088] During the sleep-to-sleep time period, the sound module is controlled to switch the content of the sleep-to-sleep music, for example, switching the content from a fairy tale to white noise. Alternatively, during the sleep-to-sleep time period, the sound module is controlled to adjust the volume of the sleep-to-sleep music, for example, adjusting the volume from 80% to 30%.

[0089] Adjust the brightness of the sleep-inspired light during sleep time, for example, from 80% to 30%. Alternatively, adjust the color of the sleep-inspired light during sleep time, for example, from warm white to warm yellow. Alternatively, adjust both the brightness and color of the sleep-inspired light during sleep time, for example, from 80% to 30% while also changing the color from warm white to warm yellow.

[0090] In this embodiment, the monitored user is accompanied by lights and music from the beginning of falling asleep to the complete sleep state. The lights and music can provide a warm atmosphere for the monitored user, and the content and form of the lights and music are automatically adjusted over time to better meet the needs of the monitored user to gradually fall asleep, thereby improving the monitored user's sleeping experience.

[0091] S203 : In response to the sleep monitoring instruction, periodically obtain a plurality of multimodal data corresponding to the monitored user based on the first time period.

[0092] See the above S101, which will not be repeated here.

[0093] S204: Generate first prompt information corresponding to each multimodal data according to each multimodal data.

[0094] See above S102, which will not be described again here.

[0095] S205 , inputting a plurality of first prompt information into the large model in sequence, and obtaining a sleep determination result corresponding to each first prompt information output by the large model.

[0096] See the above S103, which will not be repeated here.

[0097] S206: When a sleep judgment result indicating that the monitored user has entered a sleep state is detected, the sleep event is recorded, and the judgment of whether the monitored user has entered a sleep state by using the large model is stopped.

[0098] See the above S104, which will not be repeated here.

[0099] In one embodiment, when a sleep judgment result indicating that the monitored user has entered a sleep state is detected, a sleep event is recorded, and after stopping judging whether the monitored user has entered a sleep state through the large model, the method includes: turning off the sleep companion function.

[0100] When the sleep monitoring method detects that the monitored user has fallen asleep, the sleep-accompanying function is automatically turned off, thereby stopping the electronic device from emitting sleep-accompanying lights and music. In this embodiment, a condition is set for turning off the sleep-accompanying function, which is when the monitored user falls asleep. This condition turns off the sleep-accompanying function of the electronic device when the monitored user falls asleep, thereby saving energy consumption of the electronic device and preventing the sleep-accompanying lights and music from disturbing the monitored user's sleep.

[0101] In this application, in response to a sleep monitoring instruction, multiple multimodal data corresponding to a monitored user are periodically acquired based on a first time. The multimodal data includes multiple types of data specific to the monitored user, such as image data recording the monitored user's body posture and facial expressions. Furthermore, a first prompt message corresponding to each multimodal data item is generated, the first prompt message being used to instruct a large model to determine whether the monitored user has entered a sleep state based on the multimodal data. The multiple first prompt messages are sequentially input into the large model to obtain a sleep determination result output by the large model. Furthermore, when the monitored user is detected to have entered a sleep state based on the sleep determination result, a sleep event is recorded, and the large model is stopped from determining whether the monitored user has entered a sleep state. A sleep event may include information such as the time, posture, and location of the monitored user entering a sleep state. In this application, determining whether the monitored user is asleep using the large model can effectively improve the accuracy of the determination, timely monitor important nodes in the sleep cycle, and, by recording sleep events, enable the monitoring user to understand the monitored user's sleep time, posture, and location, understand the monitored user's natural sleep patterns, and improve the monitored user's sleep quality.

[0102] In one embodiment, Figure 6 The figure shows a flowchart of a sleep monitoring method provided by an embodiment of the present application. The method can be implemented by a computer program and can be run on a sleep monitoring device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone tool application.

[0103] Specifically, the sleep monitoring method includes:

[0104] S301 : In response to a sleep monitoring instruction, periodically obtain a plurality of multimodal data corresponding to a monitored user based on a first time period.

[0105] See the above S101, which will not be repeated here.

[0106] S302: Generate first prompt information corresponding to each multimodal data according to each multimodal data.

[0107] The first prompt information is used to instruct the large model to determine whether the monitored user has entered a sleeping state based on the multimodal data.

[0108] S303: Input a plurality of first prompt information into the large model in sequence, and obtain a sleep determination result corresponding to each first prompt information output by the large model.

[0109] See the above S103, which will not be repeated here.

[0110] S304: When a sleep judgment result indicating that the monitored user has entered a sleep state is detected, the sleep event is recorded, and the judgment of whether the monitored user has entered a sleep state by using the large model is stopped.

[0111] See the above S104, which will not be repeated here.

[0112] S305: Periodically obtain a plurality of multimodal data corresponding to the monitored user based on the second time period, and compare whether a difference between two multimodal data obtained at adjacent times satisfies a preset difference condition.

[0113] The specific values ​​corresponding to the first time period and the second time period can be the same or different. For example, the specific duration of the second time period is greater than the first time period, where the specific duration of the first time period is 1 minute and the specific duration of the second time period is 5 minutes. In other words, before the monitored user enters a sleep state, multimodal data of the monitored user is frequently acquired to determine whether the monitored user has entered a sleep state. After the monitored user enters a sleep state, multimodal data of the monitored user is acquired less frequently, and sleep events of the monitored user are determined based on this multimodal data.

[0114] S306 : When it is determined that the difference between two multimodal data acquired at adjacent times satisfies a preset difference condition, the multimodal data acquired at a later time is used as the target multimodal data.

[0115] Determine whether two multimodal data sets acquired adjacently at the same time have changed, and whether the resulting difference satisfies a preset difference condition. In one embodiment, the multimodal data further includes at least one of the following types: image type, sound type, temperature type, and infrared type. The preset difference condition includes: the degree of difference between data of the same type included in the two adjacent multimodal data sets is greater than a difference threshold.

[0116] For example, multimodal data includes image data that represents the body posture of the monitored user. Based on the body postures represented by two adjacent multimodal data sets, the system can determine whether the monitored user's body posture has changed and whether the degree of change meets a preset difference threshold. For example, based on two adjacent multimodal data sets, the system can determine whether the monitored user has switched from lying down to standing.

[0117] For another example, multimodal data includes sound type data, and the sound data represents the ambient sound of the monitored user and the sound made by the monitored user. The sound data represented by two multimodal data acquired adjacently in time are used to determine whether the monitored user is crying, for example, to determine the type of sound data and whether the volume difference is greater than the predicted difference.

[0118] Multimodal data can also include temperature data. Whether the difference between two adjacent temperature data points exceeds a difference threshold can be used to determine whether the monitored user has left the monitoring area. Multimodal data can also include infrared data. Whether the difference between two adjacent infrared data points exceeds a difference threshold can be used to determine whether the monitored user's body posture has significantly changed.

[0119] S307: Generate second prompt information corresponding to the target multimodal data according to the target multimodal data.

[0120] The second prompt information is used to instruct the large model to determine whether the monitored user is in a non-sleep state based on the target multimodal data. A second prompt instruction corresponding to each type of data is obtained based on the second prompt instruction template corresponding to each data type in the multimodal data. The second prompt instruction is used to prompt the large model to determine whether the monitored user is in a non-sleep state based on the data of the type corresponding to the second prompt instruction; the second prompt information is generated based on the multiple second prompt instructions and the multimodal data.

[0121] S308: Input the second prompt information into the large model, and obtain the sleep judgment result corresponding to the second prompt information output by the large model.

[0122] Multiple second prompt messages are sequentially input into the large model to obtain a sleep determination result corresponding to each second prompt message output by the large model. The large model can also obtain a sleep determination result obtained by analyzing multiple consecutive sleep determination results. For example, the large model determines that the monitored user's eyes have been open for more than a certain period of time (e.g., several minutes) based on multiple consecutive sleep determination results. The model tracks changes in the position of the monitored user's head through infrared data and infers that the monitored user's head is in motion and that the duration of the motion exceeds a preset duration (e.g., more than ten minutes). The large model then outputs a sleep determination result indicating that the monitored user is not asleep.

[0123] S309: When it is determined that the monitored user is in a non-sleeping state according to the sleep determination result corresponding to the second prompt information, a wake-up event is recorded.

[0124] The awakening event includes at least one of the following types: turning over, getting up, crying, and opening eyes. Figure 7 As shown, Figure 7 1 is a schematic diagram of a page for recording sleep events provided by an embodiment of the present application. After recording sleep event 3011, based on the multimodal data and the sleep judgment results output by the large model, waking events 3012, waking events 3013, waking events 3014, and waking events 3015 are recorded in sequence.

[0125] Multiple sleep events are summarized into a sleep report, which includes graphic and text types and video types. The content of the graphic and text type sleep report includes basic information such as total sleep time, time to fall asleep, time to wake up, sleep efficiency, and the time when multiple wake-up events occurred. It mainly includes bar charts, line charts and other charts that show data trends, images of key image frames when the user is sleeping (such as closed eyes, turning over, etc.), interpretation of charts and images, and text descriptions that provide summaries and suggestions. The content of the video type sleep report includes dynamic charts that use animation effects to show the changing trends of data, playing the user's sleep video clips, highlighting the key frame playback of important wake-up events, voice commentary with charts and videos, and soft background music and other visual elements.

[0126] In this application, in response to a sleep monitoring instruction, multiple multimodal data corresponding to a monitored user are periodically acquired based on a first time. The multimodal data includes multiple types of data specific to the monitored user, such as image data recording the monitored user's body posture and facial expressions. Furthermore, a first prompt message corresponding to each multimodal data item is generated, the first prompt message being used to instruct a large model to determine whether the monitored user has entered a sleep state based on the multimodal data. The multiple first prompt messages are sequentially input into the large model to obtain a sleep determination result output by the large model. Furthermore, when the monitored user is detected to have entered a sleep state based on the sleep determination result, a sleep event is recorded, and the large model is stopped from determining whether the monitored user has entered a sleep state. A sleep event may include information such as the time, posture, and location of the monitored user entering a sleep state. In this application, determining whether the monitored user is asleep using the large model can effectively improve the accuracy of the determination, timely monitor important nodes in the sleep cycle, and, by recording sleep events, enable the monitoring user to understand the monitored user's sleep time, posture, and location, understand the monitored user's natural sleep patterns, and improve the monitored user's sleep quality.

[0127] In one embodiment, Figure 8 The figure shows a flow chart of a large-scale model fine-tuning method provided in an embodiment of the present application. This large-scale model is used in the sleep monitoring method proposed in this application to determine whether the monitored user has entered a sleep state. This method can be implemented using a computer program and run on a large-scale model fine-tuning device based on the von Neumann architecture. This computer program can be integrated into an application or run as a standalone tool application.

[0128] Specifically, the large model fine-tuning method includes:

[0129] S401: Obtain a large model to be fine-tuned and multiple multimodal sample data.

[0130] The large model to be fine-tuned can be any machine learning model that can process multimodal data. Examples include ResNet, YOLO (You Only Look Once), ViT (Vision Transformer), CLIP (Contrastive Language–Image Pretraining), and GCN (Graph Convolutional Networks). The multimodal sample data is annotated with labels indicating whether the target user corresponding to the multimodal sample data is asleep.

[0131] S402: Generate sample prompt information corresponding to the multimodal sample data according to the prompt information template to be fine-tuned and the multimodal sample data.

[0132] The sample prompt information is used to instruct the large model to be fine-tuned to determine whether the target user corresponding to the multimodal sample data has entered a sleep state based on the multimodal sample data. The various types of data included in the multimodal sample data are respectively filled into the prompt information template to be fine-tuned to form the sample prompt information corresponding to the multimodal sample data.

[0133] S403: Input the sample prompt information into the large model to be fine-tuned, and obtain the sleep judgment result corresponding to the sample prompt information output by the large model.

[0134] The sample prompt information is input into the large model to be fine-tuned, instructing the large model to be fine-tuned to extract features from multiple types of data in the multimodal data and determine whether the monitored user corresponding to the multimodal sample data has entered a sleep state, and obtain the sleep judgment results corresponding to multiple sample prompt information respectively.

[0135] S404 , fine-tuning the large model and the prompt information template to be fine-tuned according to the difference information between the sleep judgment result corresponding to the sample prompt information and the actual sleep state of the target user, until the fine-tuning conditions are met, thereby obtaining the large model and the prompt information template.

[0136] In the process of fine-tuning the large model to be fine-tuned, the present application compares the case clue labels corresponding to the structured text samples with the training case clues, calculates the loss function value based on the difference information between the labels corresponding to the multimodal sample data and the judgment results, calculates the gradient using the backpropagation algorithm, adjusts the model parameters, and reduces the error between the labels corresponding to the multimodal sample data and the judgment results. The fine-tuning method is repeated until the large model to be fine-tuned and the prompt information template to be fine-tuned meet the preset conditions, thereby obtaining the large model and the prompt information template. The prompt information template is used to generate the first prompt information corresponding to the multimodal data based on the multimodal data.

[0137] Among them, the preset conditions can be that the accuracy of the large model in judging whether the monitored user has entered a sleep state reaches a set percentage or the cross entropy loss is reduced to a certain predetermined range, or performance indicators such as precision, recall rate, and F1 value reach preset standards.

[0138] In this application, in response to a sleep monitoring instruction, multiple multimodal data corresponding to a monitored user are periodically acquired based on a first time. The multimodal data includes multiple types of data specific to the monitored user, such as image data recording the monitored user's body posture and facial expressions. Furthermore, a first prompt message corresponding to each multimodal data item is generated, the first prompt message being used to instruct a large model to determine whether the monitored user has entered a sleep state based on the multimodal data. The multiple first prompt messages are sequentially input into the large model to obtain a sleep determination result output by the large model. Furthermore, when the monitored user is detected to have entered a sleep state based on the sleep determination result, a sleep event is recorded, and the large model is stopped from determining whether the monitored user has entered a sleep state. A sleep event may include information such as the time, posture, and location of the monitored user entering a sleep state. In this application, determining whether the monitored user is asleep using the large model can effectively improve the accuracy of the determination, timely monitor important nodes in the sleep cycle, and, by recording sleep events, enable the monitoring user to understand the monitored user's sleep time, posture, and location, understand the monitored user's natural sleep patterns, and improve the monitored user's sleep quality.

[0139] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0140] See Figure 9 , which shows a schematic diagram of the structure of a sleep monitoring device provided by an exemplary embodiment of the present application. The sleep monitoring device can be implemented as all or part of the device through software, hardware, or a combination of both. The sleep monitoring device includes:

[0141] A first sleep monitoring module 501 is configured to periodically obtain a plurality of multimodal data corresponding to a monitored user based on a first time period in response to a sleep monitoring instruction;

[0142] The second sleep monitoring module 502 is configured to generate, based on each multimodal data, a first prompt message corresponding to the multimodal data; wherein the first prompt message is used to instruct the large model to determine whether the monitored user has entered a sleep state based on the multimodal data;

[0143] The third sleep monitoring module 503 is configured to sequentially input the plurality of first prompt information into the large model, and obtain a sleep determination result corresponding to each of the first prompt information output by the large model;

[0144] The fourth sleep monitoring module 504 is configured to record a sleep event and stop determining whether the monitored user has entered a sleep state using the large model when a sleep determination result indicating that the monitored user has entered a sleep state is detected.

[0145] In one embodiment, the sleep monitoring device further comprises:

[0146] A first sleep companion activation module is configured to activate a sleep companion function of an electronic device in response to a sleep companion instruction, so as to accompany the monitored user into a sleep state; wherein activating the sleep companion function of the electronic device at least includes controlling the electronic device to emit a sleep companion light or play sleep companion music;

[0147] The second sleep companion activation module is used to obtain a sleep monitoring instruction when it is detected that the duration of activating the sleep companion function exceeds a preset activation duration.

[0148] In one embodiment, the sleep monitoring device further comprises:

[0149] The sleeping companion closing module is used to close the sleeping companion function.

[0150] In one embodiment, activating the sleep-accompanying function of the electronic device includes controlling the electronic device to emit a sleep-accompanying light and play sleep-accompanying music, and the sleep monitoring device further includes:

[0151] The third sleep companion start module is used to adjust the brightness of the sleep companion light and switch the content and / or volume of the sleep companion music during the sleep companion time period.

[0152] In one embodiment, the sleep monitoring device further comprises:

[0153] a fifth sleep monitoring module, configured to periodically acquire a plurality of multimodal data corresponding to the monitored user based on a second time period, and compare whether a difference between two multimodal data acquired at adjacent times satisfies a preset difference condition;

[0154] a sixth sleep monitoring module, configured to, when it is determined that a difference between two multimodal data acquired at adjacent times satisfies the preset difference condition, use the multimodal data acquired later as the target multimodal data;

[0155] a seventh sleep monitoring module, configured to generate, based on the target multimodal data, second prompt information corresponding to the target multimodal data; wherein the second prompt information is configured to instruct the large model to determine, based on the target multimodal data, whether the monitored user is in a non-sleeping state;

[0156] an eighth sleep monitoring module, configured to input the second prompt information into the large model, and obtain a sleep determination result corresponding to the second prompt information output by the large model;

[0157] The ninth sleep monitoring module is configured to record a waking event when it is determined that the monitored user is in a non-sleeping state according to the sleep judgment result corresponding to the second prompt information; wherein the waking event includes at least one of the following types: turning over, getting up, crying, and opening eyes.

[0158] In one embodiment, the multimodal data includes at least one of the following types: image type, sound type, temperature type, infrared type;

[0159] The preset difference condition includes: a difference between data of the same type respectively included in two adjacent multimodal data is greater than a difference threshold.

[0160] In one embodiment, the second sleep monitoring module 502 includes:

[0161] a prompt instruction unit, configured to obtain a prompt instruction corresponding to each type of data according to prompt instruction templates corresponding to the multiple data types in the multimodal data; wherein the prompt instruction is used to prompt the large model to determine whether the monitored user has entered a sleep state based on the data of the type corresponding to the prompt instruction;

[0162] The prompt information unit is used to generate first prompt information according to the plurality of prompt instructions and the multimodal data.

[0163] In this application, in response to a sleep monitoring instruction, multiple multimodal data corresponding to a monitored user are periodically acquired based on a first time. The multimodal data includes multiple types of data specific to the monitored user, such as image data recording the monitored user's body posture and facial expressions. Furthermore, a first prompt message corresponding to each multimodal data item is generated, the first prompt message being used to instruct a large model to determine whether the monitored user has entered a sleep state based on the multimodal data. The multiple first prompt messages are sequentially input into the large model to obtain a sleep determination result output by the large model. Furthermore, when the monitored user is detected to have entered a sleep state based on the sleep determination result, a sleep event is recorded, and the large model is stopped from determining whether the monitored user has entered a sleep state. A sleep event may include information such as the time, posture, and location of the monitored user entering a sleep state. In this application, determining whether the monitored user is asleep using the large model can effectively improve the accuracy of the determination, timely monitor important nodes in the sleep cycle, and, by recording sleep events, enable the monitoring user to understand the monitored user's sleep time, posture, and location, understand the monitored user's natural sleep patterns, and improve the monitored user's sleep quality.

[0164] It should be noted that the sleep monitoring device provided in the above embodiments, when executing the sleep monitoring method, is merely illustrated by the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, i.e., the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the sleep monitoring device provided in the above embodiments and the sleep monitoring method embodiments are based on the same concept. The implementation process is detailed in the method embodiments and will not be further described here.

[0165] See Figure 10 , which shows a schematic diagram of the structure of a large model fine-tuning device provided by an exemplary embodiment of the present application. The large model fine-tuning device can be implemented as all or part of the device through software, hardware, or a combination of both. The large model fine-tuning device includes a first model fine-tuning module 601, a second model fine-tuning module 602, a third model fine-tuning module 603, and a fourth model fine-tuning module 604.

[0166] A first model fine-tuning module 601 is configured to obtain a large model to be fine-tuned and a plurality of multimodal sample data;

[0167] The second model fine-tuning module 602 is configured to generate sample prompt information corresponding to the multimodal sample data based on the prompt information template to be fine-tuned and the multimodal sample data; wherein the sample prompt information is used to instruct the large model to be fine-tuned to determine whether the target user corresponding to the multimodal sample data has entered a sleep state based on the multimodal sample data;

[0168] The third model fine-tuning module 603 is used to input the sample prompt information into the large model and obtain the sleep judgment result corresponding to the sample prompt information output by the large model;

[0169] The fourth model fine-tuning module 604 is used to fine-tune the large model and the prompt information template to be fine-tuned based on the difference information between the sleep judgment result corresponding to the sample prompt information and the actual sleep state of the target user, until the fine-tuning conditions are met, thereby obtaining the large model and the prompt information template; wherein, the prompt information template is used to generate the first prompt information corresponding to the multimodal data based on the multimodal data, and the large model is used to determine whether the monitored user has entered a sleep state in the sleep monitoring method proposed in this application.

[0170] In this application, in response to a sleep monitoring instruction, multiple multimodal data corresponding to a monitored user are periodically acquired based on a first time. The multimodal data includes multiple types of data specific to the monitored user, such as image data recording the monitored user's body posture and facial expressions. Furthermore, a first prompt message corresponding to each multimodal data item is generated, the first prompt message being used to instruct a large model to determine whether the monitored user has entered a sleep state based on the multimodal data. The multiple first prompt messages are sequentially input into the large model to obtain a sleep determination result output by the large model. Furthermore, when the monitored user is detected to have entered a sleep state based on the sleep determination result, a sleep event is recorded, and the large model is stopped from determining whether the monitored user has entered a sleep state. A sleep event may include information such as the time, posture, and location of the monitored user entering a sleep state. In this application, determining whether the monitored user is asleep using the large model can effectively improve the accuracy of the determination, timely monitor important nodes in the sleep cycle, and, by recording sleep events, enable the monitoring user to understand the monitored user's sleep time, posture, and location, understand the monitored user's natural sleep patterns, and improve the monitored user's sleep quality.

[0171] It should be noted that the large model fine-tuning device provided in the above embodiment, when executing the large model fine-tuning method, only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the large model fine-tuning device provided in the above embodiment and the large model fine-tuning method embodiment are based on the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0172] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0173] The present application also provides a computer storage medium that can store multiple instructions, which are suitable for being loaded and executed by a processor as described above. Figure 1 - Figure 8 The sleep monitoring method of the embodiment shown, the specific execution process can be found in Figure 1 - Figure 8 The detailed description of the illustrated embodiment will not be repeated here.

[0174] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by a processor as described above. Figure 1 - Figure 8 The sleep monitoring method of the embodiment shown, the specific execution process can be found in Figure 1 - Figure 8The detailed description of the illustrated embodiment will not be repeated here.

[0175] See Figure 11 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 11 As shown, the electronic device 700 may include: at least one processor 701 , at least one network interface 704 , a user interface 703 , a memory 705 , and at least one communication bus 702 .

[0176] The communication bus 702 is used to implement the connection and communication between these components.

[0177] The user interface 703 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 703 may also include a standard wired interface and a wireless interface.

[0178] The network interface 704 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0179] The processor 701 may include one or more processing cores. The processor 701 utilizes various interfaces and lines to connect the various components within the server 700. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 705, and calling data stored in the memory 705, the processor 701 performs various functions of the server 700 and processes data. Optionally, the processor 701 may be implemented in at least one hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 701 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display; and the modem is used to handle wireless communications. It is understood that the modem may not be integrated into the processor 701 and may be implemented separately on a single chip.

[0180] Among them, the memory 705 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 705 includes a non-transitory computer-readable storage medium. The memory 705 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 705 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 705 may also be optionally at least one storage device located away from the aforementioned processor 701. As Figure 11 As shown, the memory 705 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a sleep monitoring and / or large model fine-tuning application.

[0181] exist Figure 11 In the electronic device 700 shown, the user interface 703 is mainly used to provide an input interface for the user and obtain user input data; and the processor 701 can be used to call the sleep monitoring application stored in the memory 705 and specifically perform the following operations:

[0182] In response to the sleep monitoring instruction, periodically acquiring a plurality of multimodal data corresponding to the monitored user based on a first time period;

[0183] Generate first prompt information corresponding to each multimodal data according to each multimodal data; wherein the first prompt information is used to instruct the large model to determine whether the monitored user has entered a sleep state based on the multimodal data;

[0184] sequentially inputting a plurality of the first prompt information into the large model, and obtaining a sleep determination result corresponding to each of the first prompt information output by the large model;

[0185] When a sleep judgment result indicating that the monitored user has entered a sleep state is detected, a sleep event is recorded, and the judgment of whether the monitored user has entered a sleep state by using the large model is stopped.

[0186] In one embodiment, before the processor 701 executes the step of periodically acquiring a plurality of multimodal data corresponding to the monitored user based on the first time in response to the sleep monitoring instruction, the step further includes:

[0187] In response to the sleep companion instruction, activating a sleep companion function of the electronic device to accompany the monitored user into a sleep state; wherein activating the sleep companion function of the electronic device at least includes controlling the electronic device to emit a sleep companion light or play sleep companion music;

[0188] When it is detected that the duration of starting the sleep companion function exceeds the preset starting duration, a sleep monitoring instruction is obtained.

[0189] In one embodiment, after the processor 701 executes the steps of recording a sleep event upon detecting a sleep determination result indicating that the monitored user has entered a sleep state and stopping determining whether the monitored user has entered a sleep state using the large model, the steps include:

[0190] Turn off the sleep-with feature.

[0191] In one embodiment, the sleep-accompanying function of the electronic device includes controlling the electronic device to emit sleep-accompanying lights and play sleep-accompanying music;

[0192] After the processor 701 executes the sleep companion instruction in response to start the sleep companion function of the electronic device to accompany the monitored user into a sleep state, the method further includes:

[0193] During the sleeping time period, the brightness of the sleeping light is adjusted, and the content and / or volume of the sleeping music is switched.

[0194] In one embodiment, after the processor 701 executes the steps of recording a sleep event and stopping determining whether the monitored user has entered a sleep state using the large model when detecting a sleep determination result indicating that the monitored user has entered a sleep state, the steps further include:

[0195] Periodically acquiring a plurality of multimodal data corresponding to the monitored user based on a second time period, and comparing whether a difference between two multimodal data acquired at adjacent times satisfies a preset difference condition;

[0196] When it is determined that the difference between the two multimodal data acquired at adjacent times satisfies the preset difference condition, the multimodal data acquired at a later time is used as the target multimodal data;

[0197] Generate, based on the target multimodal data, second prompt information corresponding to the target multimodal data; wherein the second prompt information is used to instruct the large model to determine whether the monitored user is in a non-sleeping state based on the target multimodal data;

[0198] Inputting the second prompt information into the large model, and obtaining a sleep determination result corresponding to the second prompt information output by the large model;

[0199] When it is determined that the monitored user is in a non-sleeping state according to the sleep judgment result corresponding to the second prompt information, a waking event is recorded; wherein the waking event includes at least one of the following types: turning over, getting up, crying, and opening eyes.

[0200] In one embodiment, the multimodal data includes at least one of the following types: image type, sound type, temperature type, infrared type;

[0201] The preset difference condition includes: a difference between data of the same type respectively included in two adjacent multimodal data is greater than a difference threshold.

[0202] In one embodiment, the processor 701 generates, according to each of the multimodal data, first prompt information corresponding to the multimodal data, including:

[0203] Obtain prompt instructions corresponding to each type of data according to prompt instruction templates corresponding to the multiple data types in the multimodal data; wherein the prompt instructions are used to prompt the large model to determine whether the monitored user has entered a sleep state based on the data of the type corresponding to the prompt instructions;

[0204] First prompt information is generated according to the plurality of prompt instructions and the multimodal data.

[0205] In one embodiment, the processor 701 may be configured to call a large model fine-tuning application stored in the memory 705 and perform the following operations:

[0206] Obtain the large model to be fine-tuned and multiple multimodal sample data;

[0207] Generate sample prompt information corresponding to the multimodal sample data based on the prompt information template to be fine-tuned and the multimodal sample data; wherein the sample prompt information is used to instruct the large model to be fine-tuned to determine whether the target user corresponding to the multimodal sample data has entered a sleep state based on the multimodal sample data;

[0208] Inputting the sample prompt information into the large model to be fine-tuned, and obtaining a sleep judgment result corresponding to the sample prompt information output by the large model to be fine-tuned;

[0209] The large model and the prompt information template to be fine-tuned are fine-tuned according to the difference information between the sleep judgment result corresponding to the sample prompt information and the actual sleep state of the target user until the fine-tuning conditions are met, thereby obtaining the large model and prompt information template; wherein the prompt information template is used to generate the first prompt information corresponding to the multimodal data according to the multimodal data.

[0210] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by computer programs instructing related hardware. The corresponding programs can be stored in a computer-readable storage medium. When executed, the programs can include the processes in the above-described method embodiments. The storage medium of electronic device 700 can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0211] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0212] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A sleep monitoring method, characterized in that: The method comprises: In response to the sleep monitoring instruction, periodically acquiring a plurality of multimodal data corresponding to the monitored user based on a first time period; Generate first prompt information corresponding to each multimodal data according to each multimodal data; wherein the first prompt information is used to instruct the large model to determine whether the monitored user has entered a sleep state based on the multimodal data; sequentially inputting a plurality of the first prompt information into the large model, and obtaining a sleep determination result corresponding to each of the first prompt information output by the large model; When a sleep judgment result indicating that the monitored user has entered a sleep state is detected, a sleep event is recorded, and the judgment of whether the monitored user has entered a sleep state by using the large model is stopped.

2. The sleep monitoring method according to claim 1, wherein: Before periodically acquiring a plurality of multimodal data corresponding to the monitored user based on the first time in response to the sleep monitoring instruction, the method further includes: In response to the sleep companion instruction, activating a sleep companion function of the electronic device to accompany the monitored user into a sleep state; wherein activating the sleep companion function of the electronic device at least includes controlling the electronic device to emit a sleep companion light or play sleep companion music; When it is detected that the duration of starting the sleep companion function exceeds the preset starting duration, a sleep monitoring instruction is obtained.

3. The sleep monitoring method according to claim 2, characterized in that: When a sleep judgment result indicating that the monitored user has entered a sleep state is detected, the sleep event is recorded, and after the large model is stopped for judging whether the monitored user has entered a sleep state, the method includes: Turn off the sleep-with feature.

4. The sleep monitoring method according to claim 2, characterized in that: Activating the sleep-accompanying function of the electronic device includes controlling the electronic device to emit a sleep-accompanying light and play sleep-accompanying music; After the sleep companion function of the electronic device is activated in response to the sleep companion instruction to accompany the monitored user into a sleep state, the method further includes: During the sleeping time period, the brightness of the sleeping light is adjusted, and the content and / or volume of the sleeping music is switched.

5. The sleep monitoring method according to claim 1, wherein: When a sleep judgment result indicating that the monitored user has entered a sleep state is detected, the sleep event is recorded, and after the large model is stopped for judging whether the monitored user has entered a sleep state, the method further includes: Periodically acquiring a plurality of multimodal data corresponding to the monitored user based on a second time period, and comparing whether a difference between two multimodal data acquired at adjacent times satisfies a preset difference condition; When it is determined that the difference between the two multimodal data acquired at adjacent times satisfies the preset difference condition, the multimodal data acquired at a later time is used as the target multimodal data; Generate, based on the target multimodal data, second prompt information corresponding to the target multimodal data; wherein the second prompt information is used to instruct the large model to determine whether the monitored user is in a non-sleeping state based on the target multimodal data; Inputting the second prompt information into the large model, and obtaining a sleep determination result corresponding to the second prompt information output by the large model; When it is determined that the monitored user is in a non-sleeping state according to the sleep judgment result corresponding to the second prompt information, a waking event is recorded; wherein the waking event includes at least one of the following types: turning over, getting up, crying, and opening eyes.

6. A large model fine-tuning method, characterized in that: The large model is used to determine whether the monitored user has entered a sleep state in the sleep monitoring method according to any one of claims 1 to 5, the method comprising: Obtain the large model to be fine-tuned and multiple multimodal sample data; Generate sample prompt information corresponding to the multimodal sample data based on the prompt information template to be fine-tuned and the multimodal sample data; wherein the sample prompt information is used to instruct the large model to be fine-tuned to determine whether the target user corresponding to the multimodal sample data has entered a sleep state based on the multimodal sample data; Inputting the sample prompt information into the large model to be fine-tuned, and obtaining a sleep judgment result corresponding to the sample prompt information output by the large model to be fine-tuned; The large model and the prompt information template to be fine-tuned are fine-tuned according to the difference information between the sleep judgment result corresponding to the sample prompt information and the actual sleep state of the target user until the fine-tuning conditions are met, thereby obtaining the large model and prompt information template; wherein the prompt information template is used to generate the first prompt information corresponding to the multimodal data according to the multimodal data.

7. A sleep monitoring device, characterized in that: The device comprises: a first sleep monitoring module, configured to periodically acquire a plurality of multimodal data corresponding to a monitored user based on a first time period in response to a sleep monitoring instruction; a second sleep monitoring module, configured to generate, based on each of the multimodal data, first prompt information corresponding to the multimodal data; wherein the first prompt information is used to instruct the large model to determine whether the monitored user has entered a sleep state based on the multimodal data; a third sleep monitoring module, configured to sequentially input a plurality of the first prompt information into the large model, and obtain a sleep determination result corresponding to each of the first prompt information output by the large model; The fourth sleep monitoring module is configured to record a sleep event and stop determining whether the monitored user has entered a sleep state by using the large model when a sleep judgment result indicating that the monitored user has entered a sleep state is detected.

8. A large model fine-tuning device, characterized in that: The device comprises: A first model fine-tuning module is used to obtain a large model to be fine-tuned and a plurality of multimodal sample data; A second model fine-tuning module is configured to generate sample prompt information corresponding to the multimodal sample data based on the prompt information template to be fine-tuned and the multimodal sample data; wherein the sample prompt information is used to instruct the large model to be fine-tuned to determine whether the target user corresponding to the multimodal sample data has entered a sleep state based on the multimodal sample data; a third model fine-tuning module, configured to input the sample prompt information into the large model to be fine-tuned, and obtain a sleep judgment result corresponding to the sample prompt information output by the large model to be fine-tuned; A fourth model fine-tuning module is used to fine-tune the large model and the prompt information template to be fine-tuned according to the difference information between the sleep judgment result corresponding to the sample prompt information and the actual sleep state of the target user, until the fine-tuning conditions are met, thereby obtaining the large model and the prompt information template; wherein, the prompt information template is used to generate the first prompt information corresponding to the multimodal data based on the multimodal data, and the large model is used to determine whether the monitored user has entered a sleep state in the sleep monitoring method according to any one of claims 1 to 5.

9. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, which are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps according to any one of claims 1 to 6.

Images