Non-contact respiratory rate monitoring method, device and equipment and readable storage medium

Abstract

The invention provides a non-contact respiratory rate monitoring method, device and equipment and a readable storage medium. The non-contact respiratory rate monitoring method comprises the following steps: acquiring multiple frames of video images to be monitored; based on the ROI detection network, determining a target area of the chest and abdomen corresponding to each frame of video image in sequence; extracting spatial features corresponding to the target region in the spatial stream convolutional network; inputting the spatial features into a time sequence feature generation network, and generating time sequence features corresponding to the spatial features in the time sequence feature generation network; according to the non-contact breathing frequency monitoring method, the time sequence characteristics are input into the breathing monitoring and early warning system, the breathing frequency corresponding to the time sequence characteristics is output in the breathing monitoring and early warning system, the output breathing frequency is monitored, the non-contact breathing frequency monitoring method avoids direct contact with the to-be-monitored object, and the experience feeling and comfort of the to-be-monitored object are improved; the respiratory rate is determined based on the accurately positioned target area of the thoracic and abdominal part, and the accuracy of respiratory rate monitoring is improved.

Description

Technical Field

[0001] The present application belongs to the field of health monitoring technology and relates to a non-contact respiratory rate monitoring method, and in particular to a non-contact respiratory rate monitoring method, device, equipment and readable storage medium. Background Art

[0002] Respiratory rate is one of the important vital signs of human health and is of great significance in clinical and daily health monitoring. Traditional respiratory monitoring methods mostly rely on contact devices (such as respiratory belts, surface electrodes) or complex non-contact devices (such as radar sensors, infrared imaging devices) to obtain respiratory data. However, these methods have many limitations in practical applications: contact devices affect wearing comfort, are not suitable for long-term use, and have low penetration rates in daily environments; and current non-contact devices are difficult to use for daily home monitoring due to their high cost and complex equipment. Therefore, how to reduce the complexity and cost of respiratory rate monitoring based on non-contact devices and improve the accuracy of respiratory rate monitoring has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a non-contact respiratory rate monitoring method, device, equipment and readable storage medium, which are used to reduce the complexity and cost of respiratory rate monitoring and improve the accuracy of respiratory rate monitoring.

[0004] In a first aspect, an embodiment of the present application provides a non-contact respiratory rate monitoring method, the method comprising:

[0005] Acquire multiple frames of video images to be monitored; based on the ROI monitoring network, determine the target area of ​​the chest and abdomen corresponding to each frame of the video image in turn; input the target area into the spatial stream convolution network, and extract the spatial features corresponding to the target area in the spatial stream convolution network; input the spatial features into the temporal feature generation network, and generate the temporal features corresponding to the spatial features in the temporal feature generation network; input the temporal features into the respiratory monitoring and early warning system, output the respiratory frequency corresponding to the temporal features in the respiratory monitoring and early warning system, and monitor the output respiratory frequency.

[0006] In an implementation of the first aspect, the ROI-based monitoring network sequentially determines the target area of ​​the chest and abdomen corresponding to each frame of the video image, including: performing a feature extraction operation on each frame of the video image in turn to obtain a feature image corresponding to the video image, the feature image including spatial feature information corresponding to the video image; marking multiple candidate areas of the chest and abdomen on the feature image; and screening the target area of ​​the chest and abdomen from the multiple candidate areas.

[0007] In an implementation of the first aspect, screening the target area of ​​the chest and abdomen from the multiple candidate areas includes: determining the position offset and the confidence score of each candidate area in turn; if the position offset is not within a preset offset error range, and / or the confidence score is not within a preset confidence score error range, discarding the candidate area corresponding to the position offset and the confidence score; if the position offset is within the preset offset error range, and the confidence score is within the preset confidence score error range, determining the candidate area corresponding to the position offset and the confidence score as the target area.

[0008] In an implementation of the first aspect, the target area includes multiple frames of ROI images, and after screening the target area of ​​the chest and abdomen from the multiple candidate areas, the non-contact respiratory rate monitoring method also includes: based on RoI pooling, dividing the target area into a fixed number of grid areas; performing a maximum pooling operation on each of the grid areas to obtain the ROI image corresponding to the grid area.

[0009] In an implementation of the first aspect, after the target area is input into a spatial stream convolutional network and the spatial features corresponding to the target area are extracted in the spatial stream convolutional network, the method further includes: splicing the spatial features corresponding to each frame of the video image according to a preset splicing order to obtain the spliced ​​spatial features.

[0010] In an implementation of the first aspect, the inputting the timing feature into the respiratory monitoring and early warning system, and outputting the respiratory frequency corresponding to the timing feature in the respiratory monitoring and early warning system, includes: performing timing smoothing processing on the timing feature to obtain a first timing sequence after the timing smoothing processing; performing denoising processing on the first timing sequence to obtain a second timing sequence after the denoising processing of the first timing sequence; performing periodic peak monitoring on the second timing sequence to obtain a peak position corresponding to the second timing sequence; based on the peak position, determining the average period of the video image corresponding to the timing feature; based on the average period, obtaining the respiratory frequency of the video image.

[0011] In an implementation of the first aspect, the monitoring of the output respiratory frequency includes: comparing the respiratory frequency with a preset respiratory frequency; if the respiratory frequency is not within the range of the preset respiratory frequency, the respiratory monitoring and early warning system will issue an early warning signal.

[0012] The non-contact respiratory rate monitoring method provided in the embodiment of the present application includes: obtaining multiple frames of video images to be monitored, and determining the target area of ​​the chest and abdomen corresponding to the video image, extracting the spatial features corresponding to the target area based on the spatial flow convolution network, and inputting the spatial features into the timing feature generation network, generating the timing features corresponding to the spatial features in the timing feature generation network; outputting the respiratory frequency corresponding to the timing features in the respiratory monitoring and early warning system, and monitoring the output respiratory frequency. By obtaining the video image to be monitored of the object to be monitored, and monitoring the respiratory frequency of the object to be monitored based on the video image, direct contact with the object to be monitored is avoided, and the experience and comfort of the object to be monitored are improved; by determining the target area of ​​the chest and abdomen corresponding to the video image, and extracting the spatial features and timing features of the target area of ​​the chest and abdomen, the timing features are input into the respiratory monitoring and early warning system, and the output respiratory frequency is monitored in the respiratory monitoring and early warning system, which improves the accuracy of respiratory frequency monitoring.

[0013] In a second aspect, an embodiment of the present application provides a non-contact respiratory rate monitoring device, which includes: an image acquisition module for acquiring multiple frames of video images to be monitored; a target area determination module for determining, in sequence, the target area of ​​the chest and abdomen corresponding to each frame of the video image based on the ROI monitoring network; a spatial feature extraction module for inputting the target area into a spatial stream convolutional network, and extracting the spatial features corresponding to the target area in the spatial stream convolutional network; a temporal feature generation module for inputting the spatial features into a temporal feature generation network, and generating temporal features corresponding to the spatial features in the temporal feature generation network; a respiratory rate monitoring module for inputting the temporal features into the respiratory monitoring and early warning system, outputting the respiratory frequency corresponding to the temporal features in the respiratory monitoring and early warning system, and monitoring the output respiratory frequency.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement a non-contact respiratory rate monitoring method as described in any one of the first aspects of the embodiments of the present application when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the non-contact respiratory rate monitoring method described in any one of the first aspects of the embodiments of the present application is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1AShown is a schematic diagram of an application scenario corresponding to the non-contact respiratory rate monitoring method provided in one embodiment of the present application.

[0017] Figure 1B Shown is a flow chart of a non-contact respiratory rate monitoring method provided by an embodiment of the present application.

[0018] Figure 2 Shown is a flow chart for determining target areas of the chest and abdomen in one embodiment of the present application.

[0019] Figure 3 Shown is a flow chart of determining the respiratory frequency corresponding to the timing characteristics in one embodiment of the present application.

[0020] Figure 4 Shown is a non-contact respiratory rate monitoring network corresponding to the non-contact respiratory rate monitoring method provided by an embodiment of the present application.

[0021] Figure 5 Shown is a structural diagram corresponding to the spatial stream convolutional network provided in one embodiment of the present application.

[0022] Figure 6 Shown is a schematic diagram of a non-contact respiratory rate monitoring device in an embodiment of the present application.

[0023] Figure 7 Shown is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0024] Component number description

[0025] Steps S11 to S15

[0026] Steps S21 to S23

[0027] Steps S31 to S35

[0028] 40 Non-contact respiratory rate monitoring network

[0029] 41 Input layer

[0030] 42 ROI Monitoring Network

[0031] 43 Spatial Stream Convolutional Network

[0032] 431 Conv1

[0033] 432 Conv2

[0034] 433 Conv3

[0035] 434 Conv4

[0036] 435 Conv5

[0037] 436 Fully Connected Layer

[0038] 44 Time Series Feature Generation Network

[0039] 45 Respiratory Monitoring Warning System

[0040] 60 Non-contact respiratory rate monitoring device

[0041] 61 Image acquisition module

[0042] 62 Target area determination module

[0043] 63 Spatial feature extraction module

[0044] 64 Time series feature generation module

[0045] 65 Respiratory rate monitoring module

[0046] 70 Electronic equipment

[0047] 71 Processor

[0048] 72 Non-volatile storage media

[0049] 73 System Bus

[0050] 74 Internal memory

[0051] 75 Network Interface DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0053] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0054] The prior art lacks an accurate non-contact respiratory rate monitoring method in the process of monitoring the respiratory rate of video images.

[0055] In response to the above problems, an embodiment of the present application provides a non-contact respiratory rate monitoring method. The non-contact respiratory rate monitoring method obtains multiple frames of video images to be monitored; based on the ROI monitoring network, sequentially determines the target area of ​​the chest and abdomen corresponding to each frame of the video image; inputs the target area into the spatial stream convolution network, extracts the spatial features corresponding to the target area in the spatial stream convolution network; inputs the spatial features into the temporal feature generation network, generates the temporal features corresponding to the spatial features in the temporal feature generation network; inputs the temporal features into the respiratory monitoring and early warning system, outputs the respiratory frequency corresponding to the temporal features in the respiratory monitoring and early warning system, and monitors the output respiratory frequency, which can solve the technical problem of the lack of accurate non-contact respiratory rate monitoring methods in the prior art.

[0056] Figure 1A The following is a schematic diagram of an application scenario corresponding to the non-contact respiratory rate monitoring method provided in an embodiment of the present application. Figure 1A As shown, the application scenario includes an object to be detected, a close-up camera and an electronic device. The close-up camera is used to capture relevant video images of the chest and abdomen of the object to be detected, and send the captured video images to the electronic device. The ROI detection network, the spatial stream convolution network, and the temporal feature generation network in the electronic device analyze the video images respectively to obtain the respiratory frequency of the chest and abdomen corresponding to the video image, so as to monitor the respiratory frequency in the respiratory monitoring and early warning system of the electronic device.

[0057] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0058] Figure 1B The flowchart of the non-contact respiratory rate monitoring method provided by one embodiment of the present application is shown. Figure 1B As shown, the non-contact respiratory rate monitoring method provided in the embodiment of the present application includes the following steps S11 to S15.

[0059] S11, obtaining multiple frames of video images to be monitored.

[0060] Exemplarily, multiple frames of video images to be monitored may be acquired based on a close-up camera or a wide-angle camera.

[0061] Specifically, the multiple frames of video images include video images in different scenes and states.

[0062] It should be noted that the above-mentioned methods of obtaining multiple frames of video images to be monitored based on a close-up camera or a wide-angle camera are only for illustrative purposes. In actual applications, any other suitable methods can be selected, and this application does not impose any restrictions on this.

[0063] S12, based on the ROI monitoring network, sequentially determining the target area of ​​the chest and abdomen corresponding to each frame of the video image.

[0064] Specifically, before determining the target area of ​​the chest and abdomen corresponding to each frame of the video image in sequence based on the ROI (region of interest) monitoring network, it also includes training the ROI monitoring network by annotating the data set corresponding to the video image and the data set corresponding to the video image. The process of training the ROI monitoring network includes the following steps: Collection of the data set corresponding to the video image: by collecting the monitoring data corresponding to the video images of the relevant cooperative hospitals, the respiratory monitoring data of different individuals in various scenes and states are collected to build a diverse respiratory frequency database. Among them, the specifically collected video images include video images corresponding to the spontaneous breathing of individuals in a natural state, video images under various breathing modes such as deep breathing and fast and slow breathing with controlled rhythm. In addition, it also includes collecting individuals of different ages, genders and body shapes to adapt to a wider range of people. At the same time, the collection process is carried out under a variety of environmental lighting conditions, covering scenes such as daytime and nighttime, so as to enhance the adaptability of the ROI monitoring network to lighting changes in practical applications. Annotation of the dataset corresponding to the video image: (1) Segmented annotation: Divide the video image into multiple segments according to the time period, assign a corresponding respiratory frequency label (such as the number of breaths per minute) to each segment, and record the respiratory state of the segment (respiratory states include: normal, fast, slow, deep breathing, etc.) so as to identify the respiratory characteristics of different states during training. (2) Chest and abdomen annotation: Annotate the ROI of the chest and abdomen region in each frame to ensure that the ROI monitoring network can accurately locate the respiratory-related motion area in the video image during training. Specifically, this annotation can be assisted by a semi-automatic annotation tool. The algorithm first automatically monitors and marks the chest and abdomen region, and then manually proofreads and fine-tunes it to improve the annotation efficiency and accuracy. Accurate positioning of the chest and abdomen helps the ROI monitoring network focus on the extraction of respiratory features and avoid interference from background areas.

[0065] Specifically, the video image is input into the trained ROI monitoring network, in which a feature extraction operation is performed on the video image, a candidate area of ​​the chest and abdomen is determined on the video image, and a target area of ​​the chest and abdomen is obtained based on the candidate area.

[0066] S13, inputting the target area into a spatial stream convolutional network, and extracting spatial features corresponding to the target area in the spatial stream convolutional network.

[0067] Specifically, before the target area is input into the spatial stream convolutional network, the respiratory rate monitoring method provided in the embodiment of the present application may also include: training the spatial stream convolutional network. The specific training steps are as follows: input the target area of ​​the chest and abdomen into the spatial stream convolutional network, and determine the respiratory state label corresponding to the target area of ​​the chest and abdomen of the frame. The spatial stream convolutional network is optimized by classification loss and regression loss, so that it can accurately predict the respiratory state label based on the input target area of ​​the chest and abdomen. Among them, the classification loss includes: cross entropy loss, exponential loss, logarithmic loss, etc. The regression loss includes: mean square error loss, mean absolute error loss, quantile loss, etc. The spatial features of the chest and abdomen are captured by the convolutional layer of the spatial stream convolutional network, and the target area of ​​the chest and abdomen is supervised using the labeled data with the respiratory state label to ensure that the spatial stream convolutional network can separate different respiratory states.

[0068] It should be noted that the specific loss functions of the classification loss and regression loss listed above are only for illustrative purposes. In practical applications, any other suitable classification loss and regression loss can be selected according to the specific application scenario, and this application does not impose any restrictions on this.

[0069] For example, the structure diagram corresponding to the spatial stream convolutional network is as follows Figure 5 As shown. Figure 5 It can be seen that the spatial stream convolutional network includes 5 convolutional layers and fully connected layers, among which the 5 convolutional layers are composed of convolutional layers 431 to 435 (Conv1 to Conv5 in the figure), among which Conv1 is a non-residual convolutional layer, Conv2 to Conv5 are convolutional layers containing residual blocks, and the residual blocks are connected by activation function layers, and the activation function layers include activation functions. Among them, Conv2 contains n residual blocks and n-1 activation function layers, respectively: the first residual block, the second residual block, the third residual block, ....., the nth residual block, the first activation function layer, the second activation function layer, the third activation function layer, ..., the n-1th activation function layer. The specific number of residual blocks and activation function layers can be reasonably selected according to actual conditions, and this application does not limit this.

[0070] It should be noted that the five convolutional layers listed above are only for illustrative purposes. In actual applications, any other appropriate number of convolutional layers can be selected according to the specific application scenario, and this application does not impose any limitation on this.

[0071] For example, the expression of the non-residual convolution layer Conv1 is:

[0072] Y 1 =Conv t (LR(BN(Conv s (X))))

[0073] Among them, X represents the ROI feature corresponding to the target area, Conv t (·) represents temporal convolution, LR represents activation function, BN represents BN layer normalization, Conv s (·) represents spatial convolution, Y 1 Represents the spatial features of Conv1 output.

[0074] The expression corresponding to the residual block is:

[0075] Y n =X n +Convn(BN(LR(Convn(X n ))))

[0076] Among them, X n represents the input of the nth residual block, Y n Represents the output of the nth residual block, Conv n represents the nth convolutional layer.

[0077] If the above residual block formula is simply described as: Block(), then the corresponding expression of Conv2 is:

[0078] Y 2 =Block(LR(Block(LR(Block(Y 1 )))))

[0079] Among them, Y 1 Represents the spatial features of Conv1 output, Y 2 Represents the spatial features of Conv2 output.

[0080] It should be noted that the structures of Conv2 to Conv5 are similar, and the specific formulas of Conv3 to Conv5 are not described in detail in this application.

[0081] After the ROI features corresponding to the target area pass through Conv1, Conv2, ..., Conv5 respectively, Conv5 outputs the spatial features corresponding to the target area, and the spatial features corresponding to the target area are sent to the fully connected layer 436, and the spatial features corresponding to the target area are output through the fully connected layer 436.

[0082] In some embodiments, after the target area is input into a spatial stream convolutional network and the spatial features corresponding to the target area are extracted in the spatial stream convolutional network, the method further includes: splicing the spatial features corresponding to each frame of the video image according to a preset splicing order to obtain the spliced ​​spatial features.

[0083] Specifically, the spliced ​​spatial features are a temporal feature sequence.

[0084] The preset splicing order includes: the time order of spatial feature output fully connected layers or the time order of video image acquisition, etc.

[0085] It should be noted that the preset splicing order listed above is only for illustrative purposes. In practical applications, other suitable splicing orders may be selected according to actual conditions, and this application does not impose any limitation on this.

[0086] S14, inputting the spatial feature into a temporal feature generation network, and generating a temporal feature corresponding to the spatial feature in the temporal feature generation network.

[0087] Specifically, before inputting the spatial features into the temporal feature generation network, the temporal feature generation network needs to be trained. The specific process of training the temporal feature generation network is as follows: obtain multiple spatial features and the temporal features corresponding to the spatial features, input the existing spatial features into the temporal feature generation network, and the label corresponding to the spatial features is the breathing frequency of the temporal features; optimize the temporal feature generation network based on the mean square error loss, so that the temporal features output by the temporal feature generation network are compared with the temporal features corresponding to the existing spatial features. If the error between the temporal features output by the temporal feature generation network and the temporal features corresponding to the existing spatial features is not within the preset temporal error range, continue to use the mean square error loss to optimize the temporal feature generation network until the error between the temporal features output by the temporal feature generation network and the temporal features corresponding to the existing spatial features is within the preset temporal error range, and stop training the temporal feature generation network. At this time, the optimization training of the temporal feature generation network is completed.

[0088] Specifically, the time series feature generation network can be any one of a long short-term memory network (Long Short-Term Memory, LSTM), a time convolutional network, a time-aware neural network, etc.

[0089] For example, before the spatial features are input into the temporal feature generation network, the spatial feature extraction network outputs T-frame spatial features, and the T-frame spatial features can be expressed as F={F 1 ,F 2 ,......F T}, for any one of F i Flattened into a one-dimensional vector f = {f 1 , f 2 ,......,f T} is used as the input of the time series feature generation network to generate time series features in the time series feature generation network. The formula for generating time series features in the time series feature generation network is expressed as:

[0090] h=LSTM(f)

[0091] Where h = {h 1 ,h 2 ,......h T} represents the time series characteristics, h t Contains the time series characteristics from the initial moment to moment t.

[0092] It should be noted that, in addition to the several timing feature generation networks listed above, in practical applications, any other suitable timing feature generation network can be selected according to actual needs, and this application does not impose any limitation on this.

[0093] S15, inputting the time series feature into the respiratory monitoring and early warning system, outputting the respiratory frequency corresponding to the time series feature in the respiratory monitoring and early warning system, and monitoring the output respiratory frequency.

[0094] Specifically, in the respiratory monitoring and early warning system, preprocessing operations such as time series smoothing and noise removal are performed on the time series features to improve the accuracy of the corresponding respiratory frequency obtained based on the time series features.

[0095] The embodiment of the present application provides a non-contact respiratory rate monitoring method, which captures the subtle fluctuations of the chest and abdomen through a camera to obtain multiple frames of video images to be monitored. Compared with expensive infrared, radar and other equipment, the method of obtaining multiple frames of video images to be monitored through a camera in the present invention is simpler, low-cost and easy to deploy; it avoids the monitored object directly wearing contact equipment, realizes non-contact monitoring of the monitored object, avoids the interference and discomfort of traditional contact equipment to the wearer, and improves the experience and comfort of the monitored object; the application scenarios of the non-contact respiratory rate monitoring are more diverse, and can be widely used in daily health monitoring in places such as homes, nursing homes and hospitals, and is suitable for long-term monitoring and family health monitoring; based on the ROI monitoring network, the target area of ​​the chest and abdomen corresponding to the video image is accurately obtained, and the spatial characteristics and time series characteristics are obtained based on the target area of ​​the chest and abdomen, and the time series characteristics are input into the respiratory monitoring and early warning system, and the respiratory frequency output by the time series characteristics corresponding to the video image is monitored in the respiratory monitoring and early warning system, thereby improving the accuracy of respiratory rate monitoring.

[0096] Figure 2 The flowchart of determining the target area of ​​the chest and abdomen in one embodiment of the present application is shown. Figure 2 As shown, the process of determining characteristic elements in the embodiment of the present application includes the following steps S21 to S23.

[0097] S21, performing a feature extraction operation on each frame of the video image in turn to obtain a feature image corresponding to the video image, wherein the feature image includes spatial feature information corresponding to the video image.

[0098] Specifically, the video frame sequence corresponding to the video image is X∈R T×H×W×C , where T represents the time dimension (number of frames), H and W represent the spatial dimensions (height and width), and C represents the number of channels (for RGB images, C = 3).

[0099] Specifically, the ROI monitoring network includes multiple convolutional layers and multiple pooling layers. After the convolutional layers and pooling layers of the ROI monitoring network perform a series of feature extraction operations on the video image, a feature image is obtained. The video frame sequence corresponding to the feature image is represented as F∈R T×H′×W′×C′ The feature image retains the spatial information of the input video image, which facilitates the subsequent monitoring of the respiratory monitoring and early warning system.

[0100] S22, marking a plurality of candidate regions of the chest and abdomen on the feature image.

[0101] Specifically, multiple candidate regions of the chest and abdomen are possible locations of the chest and abdomen.

[0102] For example, multiple candidate regions of the chest and abdomen may be calibrated on the feature image based on a Region Proposal Network (RPN). The bounding box of each candidate region is an anchor box, and the sizes and aspect ratios of the anchor boxes of the multiple candidate regions are the same or different.

[0103] S23, selecting the target area of ​​the chest and abdomen from the plurality of candidate areas.

[0104] Specifically, the target area of ​​the chest and abdomen can be screened from multiple candidate areas based on the position offset and confidence score of the candidate area. The position offset can be expressed as Δx, Δy, Δw, Δh, where Δx represents the offset of the anchor frame of the candidate area relative to the real frame on the x-axis; Δy represents the offset of the anchor frame of the candidate area relative to the real frame on the y-axis; Δw represents the scaling factor of the anchor frame of the candidate area relative to the real frame in width; Δh represents the scaling factor of the anchor frame of the candidate area relative to the real frame in height. The confidence score is used to evaluate the probability that the anchor frame of each candidate area contains the chest and abdomen area.

[0105] In some embodiments, screening the target area of ​​the chest and abdomen from the multiple candidate areas includes: determining the position offset and confidence score of each candidate area in turn; if the position offset is not within a preset offset error range, and / or the confidence score is not within a preset confidence score error range, discarding the candidate area corresponding to the position offset and the confidence score; if the position offset is within the preset offset error range, and the confidence score is within the preset confidence score error range, determining the candidate area corresponding to the position offset and the confidence score as the target area.

[0106] Exemplarily, in the RPN network, the position offset and confidence score of each candidate region are determined in turn, and the position offset of the candidate region is output through the regression layer. The confidence score of the candidate region is represented as the score output by the classification layer.

[0107] By comparing the position offset of each candidate area with a preset offset error, and comparing the confidence score of each candidate area with a preset confidence score error, a candidate area whose position offset satisfies the preset offset error and whose confidence score satisfies the preset confidence score error is obtained, and the candidate area is determined as the target area.

[0108] The number of target regions may be one or more. The target region of the chest and abdomen may be obtained directly from one candidate region or from a combination of multiple candidate regions.

[0109] It should be noted that this application does not limit the specific data range corresponding to the preset offset error and the preset confidence score error. The specific data range corresponding to the preset offset error and the preset confidence score error can be determined according to actual conditions, and this application will not go into details.

[0110] In some embodiments, the target area includes multiple frames of ROI images. After screening the target area of ​​the chest and abdomen from the multiple candidate areas, the non-contact respiratory rate monitoring method also includes: based on RoI pooling, dividing the target area into a fixed number of grid areas; performing a maximum pooling operation on each of the grid areas to obtain the ROI image corresponding to the grid area.

[0111] Exemplarily, based on RoI pooling, the target area is divided into a fixed number of 7x7 grid areas, and a maximum pooling operation is performed on each grid: determine the pooling area: for each grid, calculate its corresponding area on the feature image; apply maximum pooling: apply maximum pooling on the feature image area corresponding to each grid, and select the largest eigenvalue in each small area as output; generate a feature map of a fixed size: repeat the above steps until each grid area is subjected to maximum pooling, and the grid area corresponds to the ROI image.

[0112] An embodiment of the present application provides a method for determining a target area of ​​the chest and abdomen. In the method, a feature extraction operation is performed on a video image to obtain a feature image corresponding to the video image. Based on the RPN, multiple candidate areas of the chest and abdomen are calibrated on the feature image, thereby reducing the workload and errors of manual labeling and achieving efficient positioning of the candidate areas of the chest and abdomen. The RPN can filter out overlapping candidate areas and reduce the possibility of false detection. The labeled multiple candidate areas provide a large number of regional bases for the subsequent determination of the target area of ​​the chest and abdomen. The target area of ​​the chest and abdomen is further extracted from the multiple candidate areas based on the position offset and confidence score of the candidate area, thereby improving the accuracy of extracting the target area of ​​the chest and abdomen.

[0113] Figure 3 The flowchart of determining the respiratory frequency corresponding to the timing characteristics in one embodiment of the present application is shown. Figure 3 As shown, the process of determining the respiratory frequency corresponding to the timing feature in the embodiment of the present application includes the following steps S31 to S35.

[0114] S31, performing time series smoothing processing on the time series features to obtain a first time series sequence after the time series smoothing processing.

[0115] Specifically, time series smoothing processing may be performed on the time series features based on moving average, exponential smoothing, autoregressive model, Kalman filtering, and the like.

[0116] For example, the time series feature is a multidimensional feature, which is represented by h={h 1 ,h 2 ,......h T}, to simplify the calculation, the first dimension can be selected to represent the changing trend of the respiratory cycle. The smoothed feature sequence is:

[0117] s={s 1 ,s 2 ,......s T}

[0118] in, s t represents the time series feature after the tth smoothing; Represents the time series features of the first dimension before the t-th smoothing.

[0119] It should be noted that the above-mentioned methods of performing timing smoothing on the timing characteristics are only for illustrative purposes. In actual applications, any other suitable method can be selected to perform smoothing on the timing characteristics according to specific application requirements and / or application scenarios, and this application does not impose any restrictions on this.

[0120] S32, performing denoising processing on the first time series sequence to obtain a second time series sequence after the denoising processing of the first time series sequence.

[0121] Specifically, the first time series sequence can be denoised based on smoothing techniques (such as avitzky-Golay filter, moving average), statistical methods (such as robust regression, median filter), filtering methods (low-pass filter, band-pass filter, band-stop filter), etc., to obtain a second time series sequence after the first time series sequence is denoised.

[0122] It should be noted that the above-mentioned methods of performing denoising on the timing features are only for illustrative purposes. In actual applications, any other suitable method can be selected to perform denoising on the timing features according to specific application requirements and / or application scenarios, and this application does not impose any restrictions on this.

[0123] S33, performing periodic peak monitoring on the second time series to obtain a peak position corresponding to the second time series.

[0124] Specifically, the periodicity of the breathing corresponding to the second time series is a series of peaks and troughs. The peaks can be identified by the peak detection algorithm. Assume that peak(s) is the sequence of peak positions detected:

[0125] peak(s)={t 1 ,t 2 ,......,t T},

[0126] Among them, s represents the second time series feature after smoothing, t T Indicates the position of the Tth peak in the second time series.

[0127] S34: Determine an average period of the video image corresponding to the timing feature based on the peak position.

[0128] Specifically, the expression of the average period of the video image corresponding to the time series feature is:

[0129]

[0130] Where Δt represents the time series characteristic of each two adjacent peaks t i and t i+1 The time difference is the duration of a complete breathing cycle. n represents the number of peaks.

[0131] S35, obtaining the respiratory frequency of the video image based on the average period.

[0132] Specifically, the reciprocal of the above average period can be used to obtain the breathing frequency of the video image.

[0133] After determining the respiratory rate of the video image, the respiratory monitoring and early warning system can also automatically monitor the respiratory rate, automatically identify and distinguish different respiratory states (such as fast and slow breathing, deep breathing, etc.), and provide doctors or caregivers with more comprehensive respiratory information.

[0134] In some embodiments, the monitoring of the output respiratory frequency includes: comparing the respiratory frequency with a preset respiratory frequency; if the respiratory frequency is not within the range of the preset respiratory frequency, the respiratory monitoring warning system will issue a warning signal.

[0135] For example, if the respiratory rate is not within the range of the preset respiratory rate, a graded response can be made to the abnormal condition corresponding to the respiratory rate based on a multi-level warning mechanism, so that relevant medical staff can intervene and respond to the abnormal condition corresponding to the respiratory rate in a graded manner. Among them, the multi-level warning mechanism includes: multi-level warning modes such as slight abnormality and severe abnormality. The graded response methods include: prompt sound, remote notification, etc.

[0136] Specifically, before monitoring the respiratory frequency corresponding to the time series characteristics in the respiratory monitoring and early warning system, the respiratory monitoring and early warning system can be trained based on a threshold classifier, using the labeled abnormal state data to ensure the high sensitivity and low false alarm rate of the respiratory monitoring and early warning system.

[0137] The respiratory frequency not being within the range of the preset respiratory frequency includes: when breathing too fast, the respiratory frequency is higher than the range of the preset respiratory frequency, and when breathing too slow, the respiratory frequency is lower than the range of the preset respiratory frequency.

[0138] Specifically, the preset respiratory frequency is a respiratory frequency range under normal conditions. For example, the preset respiratory frequency may be 12 to 20 times per minute.

[0139] It should be noted that the preset respiratory rates of 12 to 20 times per minute listed above are only for illustrative purposes. For example, in corresponding monitoring scenarios or other application scenarios when monitoring the respiratory rates of infants or the elderly, any other suitable respiratory rate range corresponding to the preset respiratory rate can be selected according to the specific circumstances. This application does not impose any restrictions on this.

[0140] For example, in the process of monitoring the respiratory rate, the respiratory rate is also monitored based on the multimodal feature fusion strategy, specifically: by analyzing the linkage between the respiratory rate and the human body posture and chest and abdominal displacement, the complex respiratory pattern can be accurately captured. The method of monitoring the mechanical energy of the respiratory rate based on the multimodal feature fusion strategy greatly improves the detection reliability in complex environments and can adapt to different individual characteristics and multiple behavioral states.

[0141] In a method for determining the respiratory frequency corresponding to a timing feature provided in an embodiment of the present application, by performing smoothing processing on the timing feature, the smoothed timing feature is made clearer, which is convenient for subsequent observation and analysis; by denoising processing, the accuracy and reliability of the timing feature are improved; based on the second timing feature after smoothing and denoising processing, the respiratory frequency of the video image output by the respiratory monitoring and early warning system is determined, thereby improving the accuracy of determining the respiratory frequency. After determining the respiratory frequency, the respiratory state of the individual to be monitored is monitored in real time. Based on the respiratory frequency, it can be automatically determined whether the respiratory frequency is within the range of the preset respiratory frequency. If the respiratory frequency is not within the range of the preset respiratory frequency, the respiratory monitoring and early warning system sends a warning signal to remind medical staff or caregivers to intervene.

[0142] Please refer to Figure 4 , Figure 4 The non-contact respiratory rate monitoring network corresponding to the non-contact respiratory rate monitoring method provided in one embodiment of the present application is shown. The non-contact respiratory rate monitoring network 40 includes an input layer 41, a ROI monitoring network 42, a spatial stream convolution network 43, a temporal feature generation network 44, and a respiratory monitoring early warning system 45. The input layer 41 is used to input multiple frames of video images to be monitored; the ROI monitoring network 42 is used to determine the target area of ​​the chest and abdomen corresponding to each frame of the video image;

[0143] The spatial stream convolution network 43 is used to extract the spatial features corresponding to the target area; the temporal feature generation network 44 is used to generate the temporal features corresponding to the spatial features; the respiratory monitoring and early warning system 45 is used to output the respiratory frequency corresponding to the temporal features and monitor the output respiratory frequency. The specific execution steps are similar to the steps in the above-mentioned methods, and this application will not repeat them.

[0144] The protection scope of the non-contact respiratory rate monitoring method of the embodiment of the present application is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0145] The embodiment of the present application also provides a non-contact respiratory rate monitoring device, which can implement the non-contact respiratory rate monitoring method of the present application. However, the implementation device of the non-contact respiratory rate monitoring method of the present application includes but is not limited to the structure of the non-contact respiratory rate monitoring device listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0146] like Figure 6 As shown, in one embodiment, the non-contact respiratory rate monitoring device 60 of the present application includes an image acquisition module 61, a target area determination module 62, a spatial feature extraction module 63, a time series feature generation module 64, and a respiratory rate monitoring module 65.

[0147] The image acquisition module 61 is used to acquire multiple frames of video images to be monitored.

[0148] The target region determination module 62 is used to sequentially determine the target region of the chest and abdomen corresponding to each frame of the video image based on the ROI monitoring network.

[0149] The spatial feature extraction module 63 is used to input the target area into the spatial stream convolution network, and extract the spatial features corresponding to the target area in the spatial stream convolution network.

[0150] The time series feature generation module 64 is used to input the spatial feature into a time series feature generation network, and generate a time series feature corresponding to the spatial feature in the time series feature generation network.

[0151] The respiratory frequency monitoring module 65 is used to input the time series feature into the respiratory monitoring and early warning system, output the respiratory frequency corresponding to the time series feature in the respiratory monitoring and early warning system, and monitor the output respiratory frequency.

[0152] Among them, the structures and principles of the image acquisition module 61, the target area determination module 62, the spatial feature extraction module 63, the timing feature generation module 64, and the respiratory rate monitoring module 65 correspond one by one to the steps in the above-mentioned non-contact respiratory rate monitoring method, so they will not be repeated here.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules or units, which can be electrical, mechanical or other forms.

[0154] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0155] Those of ordinary skill in the art should further appreciate that the units and steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0156] The present application embodiment also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0157] An embodiment of the present application also provides an electronic device. Figure 7 The structure diagram of the electronic device 70 in one embodiment of the present application is shown. The non-contact respiratory rate monitoring method provided in the embodiment of the present application can be applied to Figure 7 The electronic device 70 shown is, but not limited to, Figure 7 As shown, the electronic device 70 includes a processor 71 , a memory, a system bus 73 , and a network interface 75 , wherein the memory may include a non-volatile storage medium 72 and an internal memory 74 .

[0158] The non-volatile storage medium 72 can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the non-contact respiratory rate monitoring methods provided in the embodiments of the present application.

[0159] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0160] The internal memory 74 provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any one of the non-contact respiratory rate monitoring methods provided in the embodiments of the present application.

[0161] The network interface 75 is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 1AThe structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] It should be understood that the processor 71 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0163] The electronic device 70 in the embodiment of the present application may include terminal devices such as tablet computers, laptop computers, mobile phones, supercomputers, smart wearable devices, etc., and can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. The embodiment of the present application does not impose any restrictions on the specific type of electronic device.

[0164] For example, the electronic device can be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on wireless systems and next-generation communication systems, such as mobile terminals in 5G networks, mobile terminals in future evolved public land mobile networks (Public Land Mobile Network, PLMN) or mobile terminals in future evolved non-terrestrial networks (Non-terrestrial Network, NTN), etc.

[0165] As an example but not limitation, when the electronic device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as gloves and watches equipped with near-field communication modules. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. It is attached to the user and performs payment, authentication and other operations through a pre-bound electronic card. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart watches and smart bracelets with display screens.

[0166] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0167] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A non-contact respiratory rate monitoring method, characterized in that: The method comprises: Acquire multiple frames of video images to be monitored; Based on the ROI monitoring network, sequentially determine the target area of ​​the chest and abdomen corresponding to each frame of the video image; Inputting the target area into a spatial stream convolutional network, and extracting spatial features corresponding to the target area in the spatial stream convolutional network; Inputting the spatial feature into a temporal feature generation network, and generating a temporal feature corresponding to the spatial feature in the temporal feature generation network; The time series feature is input into the respiratory monitoring and early warning system, the respiratory frequency corresponding to the time series feature is output in the respiratory monitoring and early warning system, and the output respiratory frequency is monitored.

2. The non-contact respiratory rate monitoring method according to claim 1, characterized in that: The ROI monitoring network is based on sequentially determining the target area of ​​the chest and abdomen corresponding to each frame of the video image, including: Performing a feature extraction operation on each frame of the video image in turn to obtain a feature image corresponding to the video image, wherein the feature image includes spatial feature information corresponding to the video image; Marking a plurality of candidate regions of the chest and abdomen on the feature image; The target area of ​​the chest and abdomen is selected from the plurality of candidate areas.

3. The non-contact respiratory rate monitoring method according to claim 2, characterized in that: The step of selecting the target area of ​​the chest and abdomen from the plurality of candidate areas comprises: Determining the position offset and confidence score of each candidate region in turn; If the position offset is not within a preset offset error range, and / or the confidence score is not within a preset confidence score error range, discarding the candidate region corresponding to the position offset and the confidence score; If the position offset is within the range of the preset offset error, and the confidence score is within the range of the preset confidence score error, the candidate area corresponding to the position offset and the confidence score is determined as the target area.

4. The non-contact respiratory rate monitoring method according to claim 2, characterized in that: The target region includes multiple frames of ROI images. After the target region of the chest and abdomen is screened from the multiple candidate regions, the non-contact respiratory rate monitoring method further includes: Based on RoI pooling, the target area is divided into a fixed number of grid areas; A maximum pooling operation is performed on each of the grid areas to obtain a ROI image corresponding to the grid area.

5. The non-contact respiratory rate monitoring method according to claim 1, characterized in that: After inputting the target area into a spatial stream convolutional network and extracting spatial features corresponding to the target area in the spatial stream convolutional network, the method further includes: According to a preset splicing order, the spatial features corresponding to each frame of the video image are spliced ​​to obtain the spliced ​​spatial features.

6. The non-contact respiratory rate monitoring method according to claim 1, characterized in that: The step of inputting the time series feature into the respiratory monitoring and early warning system, and outputting the respiratory frequency corresponding to the time series feature in the respiratory monitoring and early warning system, comprises: Performing time series smoothing processing on the time series features to obtain a first time series sequence after the time series smoothing processing; Performing denoising processing on the first time series sequence to obtain a second time series sequence after the denoising processing of the first time series sequence; Performing periodic peak monitoring on the second time series to obtain a peak position corresponding to the second time series; Based on the peak position, determining an average period of the video image corresponding to the timing feature; Based on the average period, the respiratory frequency of the video image is obtained.

7. The non-contact respiratory rate monitoring method according to claim 1, characterized in that: The step of monitoring the outputted respiratory rate comprises: comparing the breathing rate with a preset breathing rate; If the respiratory rate is not within the range of the preset respiratory rate, the respiratory monitoring and early warning system will issue an early warning signal.

8. A non-contact respiratory rate monitoring device, characterized in that: include: An image acquisition module, used to acquire multiple frames of video images to be monitored; A target region determination module, used to sequentially determine the target region of the chest and abdomen corresponding to each frame of the video image based on the ROI monitoring network; A spatial feature extraction module, used for inputting the target area into a spatial stream convolutional network, and extracting spatial features corresponding to the target area in the spatial stream convolutional network; A temporal feature generation module, used for inputting the spatial feature into a temporal feature generation network, and generating a temporal feature corresponding to the spatial feature in the temporal feature generation network; The respiratory frequency monitoring module is used to input the time series characteristics into the respiratory monitoring and early warning system, output the respiratory frequency corresponding to the time series characteristics in the respiratory monitoring and early warning system, and monitor the output respiratory frequency.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the non-contact respiratory rate monitoring method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The electronic device comprises: A memory storing a computer program; A processor is communicatively connected to the memory, and executes the non-contact respiratory rate monitoring method according to any one of claims 1 to 7 when calling the computer program.

Images