A sleep quality assessment method and system based on video and physiological data fusion

Abstract

The present invention provides a sleep quality assessment method and system based on the fusion of video and physiological data, the method comprising obtaining monitoring video data and physiological data of a monitored person; obtaining a video data image mask by manually or automatically selecting the monitoring video data; reading an image of a frame in the monitoring video data, and obtaining a region of interest by performing a logical AND operation with the video data image mask; drawing a sleep state curve after image processing the region of interest; fusing the sleep state curve with the physiological data to generate a comprehensive sleep state curve; calculating the difference between the comprehensive sleep state curve and the average sleep curve by curve similarity to obtain the sleep quality assessment result of the monitored person. The present invention can select the mask coordinates manually and automatically, and can select according to the different identities of the monitored person, so that the target area is more accurate. At the same time, the use of the region of interest reduces the workload of digital image processing and improves the detection efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a sleep quality assessment method and system based on the fusion of video and physiological data. Background Art

[0002] There is currently a serious problem in the management of supervised persons, that is, some people with poor psychological stress tolerance will have extreme thoughts, resulting in extreme behaviors such as self-mutilation or revengeful harm to others, which poses a huge challenge to the supervisory work of regulatory agencies.

[0003] Such extreme behaviors are difficult to eliminate. At the same time, the number of supervisors in the regulatory agency is limited, and it is impossible to supervise the recent psychological changes of every supervised person. The best way to solve such problems is to use technical means to promptly discover supervised persons whose activities are abnormal from normal within a certain period of time. Such supervised persons have the potential to engage in extreme behaviors. The staff should be informed in time to pay special attention to such persons, confirm and verify them, and teach and solve problems in time when they are discovered, so as to prevent them from engaging in extreme behaviors that harm themselves or others.

[0004] At the same time, the control agency is a special place, and all the supervised persons are under surveillance 24 hours a day. Considering that the surveillance video data during the day is monitored in real time by a certain number of staff, and when the supervised persons decide to perform certain dangerous behaviors or have extreme thoughts, their sleep state will also be abnormal with a high probability, this method focuses on processing the monitoring data and physiological data of the supervised persons during their sleep, so as to narrow the scope of investigation and reduce the workload of the supervisory personnel by analyzing whether the sleep state of the supervised persons is abnormal.

[0005] However, relying solely on physiological data to assess the sleep quality of supervised persons cannot achieve a high accuracy rate, and the image target detection technology has not been integrated into various video surveillance scenarios, so that the detection efficiency cannot be improved.

[0006] Therefore, there is an urgent need for a sleep quality assessment method and system based on the fusion of video and physiological data. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a sleep quality assessment method and system based on the fusion of video and physiological data. The present invention can greatly improve the efficiency of sleep monitoring, conduct effective sleep quality monitoring of personnel, and thus predict the potential extreme behaviors of the monitored personnel to a certain extent.

[0008] Terminology explanation:

[0009] YOLO3: It is a target detection algorithm based on the darknet deep learning framework, which can achieve fast and accurate real-time object detection. Its core is a deep convolutional neural network that implements regression function.

[0010] Sleep state curve: A curve drawn by preliminarily processing the monitoring video of the supervised person with time as the axis. The data of each sampling point is essentially the ratio of the area of ​​the changing image area in the video image to the area of ​​the selected area of ​​interest.

[0011] Sleep status indicators: indicators that indicate the quality of sleep, including: sleep duration, number of tossing and turning, and number of getting up at night.

[0012] Region of interest: When performing preliminary processing of monitoring data, in order to reduce the amount of calculation, the part of the image to be processed is manually or automatically selected, usually a quadrilateral area, and subsequent calculations are only for the image data inside the area.

[0013] The technical solution of the present invention is:

[0014] A sleep quality assessment method based on video and physiological data fusion includes the following steps:

[0015] S1. Obtaining surveillance video data and physiological data of the monitored person;

[0016] S2. Obtain a video data image mask by manually selecting or automatically selecting the surveillance video data;

[0017] S3. Read an image of a frame in the surveillance video data, and obtain the region of interest by performing a logical AND operation with the video data image mask;

[0018] S4. Draw a sleep state curve by performing image processing on the region of interest;

[0019] S5. integrating the sleep state curve with the physiological data to generate a comprehensive sleep state curve;

[0020] S6. Calculate the difference between the comprehensive sleep state curve and the average sleep curve through curve similarity to obtain the sleep quality assessment result of the monitored person.

[0021] Furthermore, the acquisition of monitoring video data and physiological data of the monitored person includes collecting monitoring video data of the monitored person through a camera, and collecting physiological data of the monitored person through a bracelet, including heart rate data and motion data combined with 3-axis acceleration.

[0022] Furthermore, in S2, the video data image mask is obtained by manual selection or automatic selection of the surveillance video data, including automatic selection, that is, identifying the person in the surveillance video data through a deep learning target detection model, and outputting the bounding box coordinates of the identified person; manual selection, that is, according to the identity of the monitored person, manually clicking to frame the target area in the surveillance video, and feeding back the area coordinates, wherein the manually and automatically selected target area is the area with a value of 255 in the mask, and the mask and the video image are logically ANDed to obtain the area of ​​interest.

[0023] Further, in S3, an image of a frame in the surveillance video data is read, and a logical AND operation is performed with the video data image mask to obtain the region of interest, including the size of the image frame obtained by reading the image of the first frame in the surveillance video data, recreating a zero array matrix of the same size, and setting all pixel points of the target area in the zero array matrix to 255, and then performing a logical bitwise AND operation with the read image of a frame of surveillance video data, that is, performing a logical bitwise AND operation with the array matrix in which all pixels in the target area are set to 255 and the video data, thereby obtaining the region of interest.

[0024] Furthermore, in S4, obtaining a sleep state curve by performing image processing on the region of interest includes:

[0025] (1) Setting up a background frame;

[0026] (2) Convert the image grayscale of the region of interest;

[0027] (3) performing noise reduction on the converted image by Gaussian filtering;

[0028] (4) Obtaining a differential image by performing a differential operation on the background frame and the current frame;

[0029] (5) Obtaining the image changes within the region of interest by performing a threshold binarization operation on the difference image;

[0030] (6) Performing morphological dilation on the thresholded binarized difference image to eliminate noise, connect the active area images, and obtain the dilated image data; wherein the function of the dilation is to connect the active area images and erode them into segmented active area images.

[0031] Specifically, a 3x3 square structure data (i.e., a 3x3 array matrix with all elements set to 1) is used to operate on the binary image. Assuming the image data is A and the structure data is B, the calculation is defined as:

[0032]

[0033] (7) Based on the expanded image data, including the image data after expansion calculation in step (6), the ratio of the number of pixels with a value of 255 in the region of interest to the number of all pixels in the region of interest is used as data for drawing a sleep state curve, and the sleep state curve is drawn with time as the axis.

[0034] Furthermore, in S5, the sleep state curve is integrated with the physiological data, including:

[0035] By setting the distribution range, the physiological data of the monitored personnel are subjected to the removal of extreme data;

[0036] The physiological data after removing the extreme data are normalized and weighted averaged with the sleep state curve to draw a comprehensive sleep state curve.

[0037] Furthermore, in S6, the difference between the comprehensive sleep state curve and the average sleep curve is calculated by curve similarity, including: calculating three indicators of curve similarity comparison of the comprehensive sleep state curve and the average sleep curve, namely, the Pearson correlation coefficient, the Euclidean distance and the Manhattan distance, wherein:

[0038] The Pearson correlation coefficient calculation formula is as follows:

[0039]

[0040] The Euclidean distance calculation formula is as follows:

[0041]

[0042] The Manhattan distance calculation formula is as follows:

[0043]

[0044] Results of three distance indicators

[0045] By setting thresholds for the three distance indicators, a sleep quality warning is issued for data that exceeds or falls below the threshold.

[0046] A sleep quality assessment system based on video and physiological data fusion, including

[0047] A data acquisition module is configured to acquire monitoring video data and physiological data of a monitored person;

[0048] The image mask module is configured to obtain a video data image mask by manually selecting or automatically selecting the monitoring video data;

[0049] The logic operation module is configured to read an image of a frame in the monitoring video data, and obtain an area of ​​interest by performing a logic AND operation with the video data image mask;

[0050] The curve module is configured to obtain a sleep state curve by performing image processing on the region of interest;

[0051] A fusion module is configured to fuse the sleep state curve with the physiological data to generate a comprehensive sleep state curve;

[0052] The evaluation module is configured to calculate the difference between the comprehensive sleep state curve and the average sleep curve through curve similarity to obtain the sleep quality evaluation result of the monitored person.

[0053] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a sleep quality assessment method based on the fusion of video and physiological data.

[0054] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing a sleep quality assessment method based on the fusion of video and physiological data.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) In the present invention, the mask coordinates can be selected manually or automatically, and can be selected according to the identity of the monitored person, making the target area more accurate. At the same time, the use of the region of interest reduces the workload of digital image processing and improves detection efficiency.

[0057] (2) Combine the monitored video data and physiological data of the supervised person to generate the corresponding person's sleep status curve, further improving the detection accuracy.

[0058] (3) It can timely detect and warn supervised persons with abnormal sleep status, help guards narrow the scope of monitoring and investigation, and timely discover supervised persons with potential extreme behaviors so that they can pay special attention to them, verify and confirm them, and provide them with guidance and assistance, so as to prevent supervised persons from causing harm to themselves or others due to extreme behaviors.

[0059] (4) This solution is based on actual needs. By analyzing the characteristics of the scenario, a sleep quality assessment method based on the fusion of video and physiological data is designed. This can narrow the scope of supervised persons with psychological excess tendencies. This solution is low-cost and has extremely low requirements on the computing performance of the hardware platform, which can greatly reduce manpower investment. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0061] Figure 1 is a flow chart of this embodiment;

[0062] Figure 2 is a diagram of the YOLO3 network architecture of this embodiment;

[0063] Figure 3 This is a flow chart of determining sleep quality based on a comprehensive sleep state curve in the present invention;

[0064] Figure 4 It is the normal state sleep state curve and sleep index diagram of the present invention;

[0065] Figure 5 It is the abnormal sleep state curve and sleep index diagram of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0067] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0068] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0069] Example 1.

[0070] like Figure 1 As shown, this example provides a method for converting monitoring video data into a sleep state curve, which specifically includes:

[0071] S1. Obtain surveillance video data:

[0072] The supervised person is under 24-hour surveillance. Considering that when a person is planning and thinking about certain illegal behaviors or having depression and radical thoughts for a period of time, his sleep state will also be abnormal with a high probability, so the present invention starts with the monitoring video data of the supervised person when he is sleeping. The present invention introduces the respiratory rate and 3-axis acceleration motion data collected by the supervised person's wristband. The respiratory rate data is collected once a minute, and the 3-axis acceleration motion data of the wristband uses the average value per minute, and the data collection frequency is changed to once a minute.

[0073] S2. Get the mask area:

[0074] In digital image processing, masks are mainly implemented through two-dimensional array matrices or multi-valued images. Their main function is to extract the region of interest in digital image processing, with the aim of reducing the amount of data required for video image data processing.

[0075] In general, the corresponding pruned YOLO3 target recognition network is used to identify people in the area and output the bounding box coordinates of the identified people. YOLO3 network architecture Figure 2 As shown. Yolov3 is a network based on the Darknet-53 architecture, with a total of 103 layers. The entire network will output feature maps of three sizes, namely, the 79th layer plus two convolutional layers output a feature map of size 13*13, the 91st layer plus two convolutional layers output a feature map of size 26*26, and the last output feature map of size 52*52. The high-level network of the Yolov3 network model uses the feature maps of the low-level network, so the high-level network can use fine-grained features and more semantic information. The feature map output of the last size uses the feature maps calculated by the first two sizes, so that the final output feature map can also use fine-grained features.

[0076] There are currently three main strategies for Yolo3 channel pruning: normal pruning, regular pruning, and extreme pruning. The first one, normal pruning, is used in this invention. This is a relatively conservative pruning strategy. Yolov3 has five groups of 23 direct connections, corresponding to which is the addition operation of feature vectors. In order to ensure that the dimensions of the two directly connected input feature vectors are consistent, normal pruning does not prune the directly connected layers, but it can still greatly reduce the model parameters.

[0077] According to the actual situation and the identity of the monitored person, when the target recognition network cannot accurately identify the target person, you can use manual selection to select the corresponding area on the image with the mouse.

[0078] Among them, the overall structural process of the yolo3 network includes:

[0079] YOLO3 recognition process:

[0080] (1) The input image data will be automatically resized to 416*416. After calculation by the darknet feature extraction network, three feature maps of different sizes are obtained: 13*13*255, 26*26*255, and 52*52*255.

[0081] (2) Use upsampling to fuse the feature maps into a 52*52*255 feature map, specifically 52*52*3*(5+80), where 5 represents the information of the four coordinates and the predicted confidence value, and 80 represents the prediction of 80 categories. Both data are normalized.

[0082] (3). Use the Logistic classifier to classify the results and output the detection results.

[0083] S3. Image grayscale processing:

[0084] In order to reduce the amount of data to be processed, the image is converted into a grayscale image after obtaining the video data. The original three-channel image can be converted into a single-channel image, which can effectively improve the computing speed.

[0085] S4. Get the region of interest:

[0086] By performing a logical AND operation on the image after grayscale processing and Gaussian filtering noise reduction and the obtained mask area, the video image data corresponding to the area of ​​interest can be obtained. In the post-processing, only this part of the data is used for sleep state detection. Among them, the logical AND operation is to perform a logical AND operation on the value of each pixel of the image and the value of the corresponding position of the mask, such as: 1·1=1, 1·0=0, 0·0=0.

[0087] The grayscale value range of each pixel in the processed image is 0-255, corresponding to 8-bit binary data. The pixel data is logically ANDed with the mask (the same-size array matrix established). The data in the area selected by the mask area is all 255, corresponding to the binary value 11111111, and the image pixel data remains unchanged after the operation. The data in the non-selected area of ​​the mask area is all 0, corresponding to the binary value 00000000, and the image pixel data is all 00000000 after the operation, which is displayed as black on the image.

[0088] S5. Output sleep status curve:

[0089] First, a background frame is set up, and the image frame that enters the sleep state is read as the background frame for differential operation with subsequent images. After the subsequent image frame is read, the image is converted into a grayscale image through grayscale conversion. Grayscale conversion converts the original color RGB three-channel image into a single-channel grayscale image, which can reduce the amount of calculated data and greatly speed up the image processing. Gaussian filtering is used to eliminate the noise in the image to a certain extent.

[0090] The basic principle of Gaussian filtering is: by weighted averaging each pixel in the image and the domain pixel, just like the convolution operation in the neural network, a convolution kernel scans each pixel of the image and performs a weighted average operation, so as to achieve the effect of smoothing the image through Gaussian filtering operation. Perform image difference operation between the background frame and the current frame, and then perform threshold binarization operation on the difference image, so as to obtain the image changes in the region of interest, and finally perform morphological expansion on the image after threshold binarization to further eliminate noise, segment and link the active area image, and finally obtain the sleep state curve of the monitored person through the image changes.

[0091] Example 2.

[0092] like Figure 3 As shown, this example provides a method for determining sleep quality based on a sleep state curve integrating physiological data, which specifically includes:

[0093] Unified data frequency: The sleep state data obtained by preliminary calculation is consistent with the frame rate of the monitoring video, which is 25hz. When fused with the physiological data collected by the bracelet, it cannot correspond. First, it is necessary to unify the data frequency, the frequency of the heart rate data and the 3-axis acceleration motion data. V and acceleration motion data D A Downsampling, changing the data frequency to once per second, heart rate data D H The original frequency is once per minute, so interpolation calculation is needed to increase the data frequency to once per second. All three data need to be normalized before fusion. In actual use, the inventor found that video data plays a greater role in the judgment of sleep quality. Finally, the weighted calculation formula for the comprehensive sleep state data is adjusted as follows:

[0094] D=0.5*D V +0.3*D A +0.2*D H

[0095] Finally, the fused sleep state data with a frequency of once per second is obtained, and the comprehensive sleep state curve is derived from it.

[0096] The normality of the sleep state curve is represented by three indicators: Pearson correlation coefficient, Euclidean distance, and Manhattan distance, so as to realize the abnormal judgment of sleep state, mainly through the real-time sleep state curve and its statistical regular sleep state curve fit and sleep state indicators under normal state. The larger the Pearson correlation coefficient, the greater the similarity of the two curves, and the smaller the Pearson correlation coefficient, the smaller the similarity of the two curves. The Pearson correlation coefficient takes values ​​between -1 and +1, that is, it can be normalized by the Pearson coefficient formula: ρ N =0.5+0.5*ρ X，Y To adjust the value to normalize between 0 and 1. Based on the Pearson correlation coefficient result, we can see the similarity between the sleep curve of a certain day and the average regular sleep curve, and set the threshold parameter of the automatic alarm. When the curve similarity is less than the threshold, the alarm is automatically triggered. Among them, according to the experiment, the threshold parameter of the Pearson correlation coefficient is set to 0.52 and 0.85, and the alarm is triggered when the calculated result is less than 0.52 or greater than 0.85.

[0097] Similarly, Euclidean distance and Manhattan distance can also be used as a measure of curve similarity, and the values ​​of both are inversely proportional to the curve similarity. Based on Euclidean distance and Manhattan distance, we can see the similarity between a certain day's sleep curve and the average regular sleep curve, and set the threshold parameters for automatic alarms. When the curve similarity is greater than the threshold, an automatic alarm is triggered.

[0098] For the convenience of calculation, the Euclidean distance and Manhattan distance are first normalized. Since the value range of the comprehensive sleep curve is 0-1, it can be directly calculated by:

[0099]

[0100]

[0101] According to the experiment, the threshold parameter of the Euclidean distance is set to 0.31, and an alarm is triggered when it is greater than 0.31; the Manhattan distance is set to 0.22, and an alarm is triggered when it is greater than 0.22

[0102] Before judging the sleep state, we must first count the regular sleep state curve, obtain the sleep fluctuation curve of 100 days under normal conditions through the program, remove 20% of the maximum value and 20% of the minimum value, and average the remaining values ​​as the value at this moment, so as to obtain the statistical daily average sleep fluctuation curve. The principle is to obtain the regularity of the sleep state of the designated person by counting the sleep state of the normal state for many days. Therefore, by calculating the degree of fit between the sleep state curve obtained daily and the statistical regular sleep state curve, it is judged whether the sleep state is abnormal. The three important indicators reflecting the quality of a person's sleep state are sleep duration, number of turning over and number of getting up at night. The judgment threshold is set by the degree of change of the image of the region of interest to calculate the number of turning over and getting up at night of the monitored person, and the average sleep duration, number of turning over and number of getting up at night of this person under normal conditions are statistically calculated, and the sleep quality is calculated by the changes of these three sleep indicators. By comparing with the statistical average indicators, if the difference in sleeping time is greater than 45 minutes, or the difference in the number of turning over is greater than four times, or the difference in the number of getting up is greater than three times, the sleep quality is judged to be poor and the sleep state of this person is abnormal. The system will inform the staff to check and confirm the status of the person. If the difference in sleeping time is less than 15 minutes, the difference in the number of turning over is less than two times, and the difference in the number of getting up is less than two, it is judged to be excellent, and other situations are judged to be good.

[0103] Example 3.

[0104] A sleep quality assessment system based on video and physiological data fusion, including

[0105] A data acquisition module is configured to acquire monitoring video data and physiological data of a monitored person;

[0106] The image mask module is configured to obtain a video data image mask by manually selecting or automatically selecting the monitoring video data;

[0107] The logic operation module is configured to read an image of a frame in the monitoring video data, and obtain an area of ​​interest by performing a logic AND operation with the video data image mask;

[0108] The curve module is configured to obtain a sleep state curve by performing image processing on the region of interest;

[0109] A fusion module is configured to fuse the sleep state curve with the physiological data to generate a comprehensive sleep state curve;

[0110] The evaluation module is configured to calculate the difference between the comprehensive sleep state curve and the average sleep curve through curve similarity to obtain the sleep quality evaluation result of the monitored person.

[0111] Example 4.

[0112] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device. A sleep quality assessment method based on the fusion of video and physiological data provided in this embodiment.

[0113] Example 5.

[0114] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded and executed by the processor. A sleep quality assessment method based on video and physiological data fusion provided in this embodiment.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0119] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0120] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A sleep quality assessment method based on video and physiological data fusion, It is characterized in that The steps include: S1. Obtaining surveillance video data and physiological data of the monitored person; physiological data includes heart rate data and motion data combined with 3-axis acceleration; S2. Obtain a video data image mask by manually selecting or automatically selecting the surveillance video data; S3. Read an image of a frame in the surveillance video data, and obtain the region of interest by performing a logical AND operation with the video data image mask; S4. Draw a sleep state curve by performing image processing on the region of interest; S5. integrating the sleep state curve with the physiological data to generate a comprehensive sleep state curve; S6. Calculate the difference between the comprehensive sleep state curve and the average sleep curve by curve similarity to obtain the sleep quality assessment result of the monitored person; including: calculating the Pearson correlation coefficient of the comprehensive sleep state curve and the average sleep curve to compare the curve similarity, the Euclidean distance to compare the curve similarity and the Manhattan distance to compare the curve similarity, and setting a threshold to issue a sleep quality warning for data that exceeds or falls below the threshold; In S4, obtaining a sleep state curve by performing image processing on the region of interest includes: (1) Setting up a background frame; (2) Convert the image grayscale of the region of interest; (3) De-noising the converted image by Gaussian filtering; (4) Obtain a differential image by performing a differential operation on the background frame and the current frame; (5) Obtain the image changes in the region of interest by performing a threshold binarization operation on the difference image; (6) Perform morphological dilation on the thresholded binary difference image to eliminate noise, segment the connected active area image, and obtain the dilated image data; (7) Draw a sleep state curve based on the expanded image data with time as the axis; In S5, the sleep state curve is integrated with the physiological data, including: By setting the distribution range, the physiological data of the monitored personnel are subjected to the removal of extreme data; The physiological data after removing the extreme data are normalized and weighted averaged with the sleep state curve to draw a comprehensive sleep state curve.

2. A sleep quality assessment method based on video and physiological data fusion as claimed in claim 1, It is characterized in that The obtaining of the monitoring video data and physiological data of the monitored person includes collecting the monitoring video data of the monitored person through a camera and collecting the physiological data of the monitored person through a wristband.

3. The sleep quality assessment method based on video and physiological data fusion as claimed in claim 1, It is characterized in that In S2, the video data image mask is obtained by manual selection or automatic selection of the monitoring video data, including automatic selection, that is, identifying the person in the monitoring video data through a deep learning target detection model, and outputting the bounding box coordinates of the identified person; manual selection, that is, according to the identity of the monitored person, manually clicking to frame the area of ​​interest in the monitoring video, and feeding back the area coordinates.

4. The sleep quality assessment method based on video and physiological data fusion as claimed in claim 1, It is characterized in that In S3, the first frame image in the monitoring video data is read, and a logical AND operation is performed with the video data image mask to obtain the region of interest, including the size of the image frame obtained by reading an image frame in the monitoring video data, recreating a zero array matrix of the same size, and setting all the pixels of the target area in the array matrix to 255, and then performing a logical bitwise AND operation with the monitoring video data to obtain the region of interest.

5. A sleep quality assessment system based on the fusion of video and physiological data, It is characterized in that include: A data acquisition module is configured to acquire monitoring video data and physiological data of a monitored person; Physiological data includes heart rate data and motion data combined with 3-axis acceleration; The image mask module is configured to obtain a video data image mask by manually selecting or automatically selecting the monitoring video data; The logic operation module is configured to read an image of a frame in the monitoring video data, and obtain an area of ​​interest by performing a logic AND operation with the video data image mask; The curve module is configured to obtain a sleep state curve by performing image processing on the region of interest; A fusion module is configured to fuse the sleep state curve with the physiological data to generate a comprehensive sleep state curve; The evaluation module is configured to calculate the difference between the comprehensive sleep state curve and the average sleep curve by curve similarity to obtain the sleep quality evaluation result of the monitored person; including: calculating the Pearson correlation coefficient comparison curve similarity, the Euclidean distance comparison curve similarity and the Manhattan distance comparison curve similarity of the comprehensive sleep state curve and the average sleep curve, and setting a threshold to issue a sleep quality warning for data exceeding or below the threshold; The sleep state curve is obtained by image processing of the region of interest, including: (1) Setting up a background frame; (2) Convert the image grayscale of the region of interest; (3) De-noising the converted image by Gaussian filtering; (4) Obtain a differential image by performing a differential operation on the background frame and the current frame; (5) Obtain the image changes in the region of interest by performing a threshold binarization operation on the difference image; (6) Perform morphological dilation on the thresholded binary difference image to eliminate noise, segment the connected active area image, and obtain the dilated image data; (7) Draw a sleep state curve based on the expanded image data with time as the axis; Fusion of sleep state curves with physiological data, including: By setting the distribution range, the physiological data of the monitored personnel are subjected to the removal of extreme data; The physiological data after removing the extreme data are normalized and weighted averaged with the sleep state curve to draw a comprehensive sleep state curve.

6. A computer-readable storage medium, It is characterized in that A plurality of instructions are stored therein, and the instructions are suitable for being loaded by a processor of a terminal device and executing a sleep quality assessment method based on the fusion of video and physiological data according to any one of claims 1 to 4.

7. A terminal device, It is characterized in that The invention comprises a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executed by a sleep quality assessment method based on the fusion of video and physiological data according to any one of claims 1 to 4.

Images