Respiration risk detection method and device based on visible light and infrared thermal image

Abstract

The invention provides a breathing risk detection method and device based on visible light and infrared thermal images. The method comprises the following steps: acquiring a visible light image and an infrared image to be processed; inputting the visible light image into a trained first network to obtain a first recognition result; obtaining a first detection result according to the first identification result; inputting the infrared image into a trained second network to obtain a second recognition result; obtaining a second detection result according to the second identification result; and according to the first detection result and the second detection result, obtaining a baby breathing risk judgment result. Two networks are arranged, the first network is used for recognizing a specified area of a baby and providing a judgment basis for whether the face of the baby is blocked subsequently, the second network is used for recognizing a forehead area of the baby and providing a judgment basis for the forehead temperature of the baby subsequently, and finally whether the baby has a breathing risk or not is comprehensively judged by combining the judgment of the first network and the second network. The detection result is very comprehensive and accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of safety monitoring, and in particular to a method and device for detecting respiratory risks based on visible light and infrared thermography. Background Art

[0002] With the progress of technology and the increasing attention of new parents to the health of infants, traditional infant monitoring methods are gradually developing towards intelligence and automation. To ensure the safety of infants during sleep, it is particularly important to timely detect and prevent the suffocation risk caused by the obstruction of the mouth and nose by foreign objects, which has become one of the important research tasks of the infant intelligent monitoring system.

[0003] Currently, the monitoring means on the market mainly include contact sensors (such as heart rate monitoring belts and respiratory belts) and visible light image monitoring. However, these traditional methods face many limitations in practical applications. For contact sensors, although they can provide relatively accurate vital sign data, the contact design may cause discomfort to infants, affect their normal activities, and there are problems of falling off or false triggering; for visible light image monitoring, ordinary visible light cameras are greatly affected by environmental lighting conditions, and it is difficult to obtain clear images in low-light and dim environments, resulting in limited accuracy in judging whether the mouth and nose are blocked, and thus inaccurate monitoring results. Summary of the Invention

[0004] In view of the above defects of the prior art, the present invention provides a method and device for detecting respiratory risks based on visible light and infrared thermography to solve the technical problem of inaccurate monitoring results.

[0005] To achieve the above object and other related objects, the present invention provides a method for detecting respiratory risks based on visible light and infrared thermography, including: acquiring a visible light image and an infrared image to be processed; inputting the visible light image into a trained first network to obtain a first recognition result for recognizing a designated area of the infant in the visible light image; obtaining a first detection result according to the first recognition result, where the first detection result is used to represent whether the infant's face is blocked; inputting the infrared image into a trained second network to obtain a second recognition result for recognizing the forehead area of the infant in the infrared image; obtaining a second detection result according to the second recognition result, where the second detection result is used to represent the infant's body temperature; obtaining an infant respiratory risk judgment result according to the first detection result and the second detection result.

[0006] In an embodiment of the present invention, the designated area of the infant includes the head, nose, mouth, and hand area of the infant.

[0007] In an embodiment of the present invention, the first network includes a third network applicable to low-illumination scenarios and a fourth network applicable to normal-illumination scenarios; inputting the visible light image into the trained first network to obtain a first recognition result, including: obtaining the actual illuminance of the current scenario; determining whether the actual illuminance is less than or equal to a preset illuminance value: if so, performing brightness enhancement processing on the visible light image and then inputting it into the trained third network to obtain the first recognition result; if not, inputting the visible light image into the trained fourth network to obtain the first recognition result.

[0008] In an embodiment of the present invention, the preset illuminance value is 10 to 20 lux.

[0009] In an embodiment of the present invention, performing brightness enhancement processing on the visible light image includes: performing brightness enhancement processing on the visible light image using a zero-reference depth curve estimation algorithm.

[0010] In an embodiment of the present invention, the first network includes a backbone feature extraction unit, a lightweight feature pyramid structure, and a detection head; inputting the visible light image into the trained first network to obtain a first recognition result, including: using the backbone feature extraction unit to perform multi-level feature extraction on the visible light image to obtain a feature map with rich semantic information; using the lightweight feature pyramid structure to fuse the feature maps output by the backbone feature extraction unit to obtain a multi-scale feature map; using the detection head to process the multi-scale feature map to obtain the first recognition result.

[0011] In an embodiment of the present invention, the backbone feature extraction unit includes five stages, and the expressions of the feature maps Y1 to Y5 output by each stage are as follows: Y1 = ConvHead(X in ); Y2 = SE(DWBlock(DWBlock(MaxPool(Y1)))); Y i+1 = MaxPool(SE(DWBlock(Y i )), i ∈ {2, 3, 4}; where X in is the input feature of the backbone feature extraction unit, ConvHead is composed of a cascaded convolutional layer, batch normalization and rectified linear unit, and a depthwise separable convolutional unit, SE is an attention mechanism module, DWBlock is composed of two cascaded depthwise separable convolutional units, MaxPool is a max pooling layer, and the depthwise separable convolutional unit is composed of a cascaded first convolutional layer, second convolutional layer, and batch normalization and rectified linear unit. The convolutional kernel sizes of the first convolutional layer and the second convolutional layer are 1×1 and 3×3 respectively.

[0012] In one embodiment of the present invention, the lightweight feature pyramid structure is divided into three branches, which respectively process the feature maps output by the last three stages of the backbone feature extraction unit, and its expression is as follows: Y5’ = DWUnit(DWUnit(Y5)); Y4’ = DWUnit(DWUnit(Y4 + Upsample(DWUnit(Y5)))); Y3’ = DWUnit(DWUnit(Y3 + Upsample(DWUnit(Y4)))); where Y3’, Y4’, and Y5’ are the multi-scale feature maps, and DWUnit is the depthwise separable convolution unit.

[0013] In one embodiment of the present invention, three detection heads are provided, which respectively process the features output by the three branches of the lightweight feature pyramid structure to obtain object detection results at different scales.

[0014] In one embodiment of the present invention, according to the first recognition result, a first detection result is obtained, including: using the non-maximum suppression algorithm with an adaptive threshold to process the first recognition result to obtain the detection result of the specified area of the infant; according to the detection conditions of the head, nose, mouth, and hands of the infant in the detection result of the specified area of the infant, the first detection result is obtained.

[0015] In one embodiment of the present invention, according to the detection conditions of the head, nose, mouth, and hands of the infant in the detection result of the specified area of the infant, the first detection result is obtained, including: determining whether the head of the infant is detected in the detection result of the specified area of the infant: if so, then determining whether the nose and mouth of the infant are simultaneously detected in the detection result of the specified area of the infant: if so, the first detection result is normal; if not, then determining whether the hands of the infant are detected in the detection result of the specified area of the infant and the area of the hands of the infant is included in the area of the head of the infant: if so, the first detection result is normal; if not, the first detection result is abnormal; if not, the first detection result is abnormal.

[0016] In one embodiment of the present invention, the structure of the second network is the same as that of the first network. The training data of the first network is visible light images labeled with the specified area of the infant, and the training data of the second network is infrared images labeled with the forehead area of the infant.

[0017] In an embodiment of the present invention, obtaining a second detection result according to the second recognition result includes: using a non-maximum suppression algorithm with an adaptive threshold to process the second recognition result to obtain a detection result of the baby's forehead area; obtaining the baby's forehead body temperature according to the position of the detection frame in the detection result of the baby's forehead area and the mapping relationship between the pixel value and the temperature of the infrared image; determining whether the baby's forehead body temperature is within a preset normal temperature range: if so, the second detection result is normal, and if not, the second detection result is abnormal.

[0018] In an embodiment of the present invention, obtaining a baby's breathing risk judgment result according to the first detection result and the second detection result includes: determining whether the first detection results of the first preset number of consecutive frames of visible light images are all abnormal: if so, the baby has a breathing risk, otherwise the baby does not have a breathing risk; determining whether the second detection results of the second preset number of consecutive frames of infrared images are all abnormal: if so, the baby has a breathing risk, otherwise the baby does not have a breathing risk.

[0019] To achieve the above and other related purposes, the present invention also provides a breathing risk detection device based on visible light and infrared thermography, including: a data acquisition unit for acquiring the visible light image and the infrared image to be processed; a first recognition unit for inputting the visible light image into a trained first network to obtain a first recognition result for recognizing a specified area of the baby in the visible light image; a first detection unit for obtaining a first detection result according to the first recognition result, where the first detection result is used to represent whether the baby's face is blocked; a second recognition unit for inputting the infrared image into a trained second network to obtain a second recognition result for recognizing the baby's forehead area in the infrared image; a second detection unit for obtaining a second detection result according to the second recognition result, where the second detection result is used to represent the baby's body temperature; and a determination unit for obtaining a baby's breathing risk judgment result according to the first detection result and the second detection result.

[0020] Advantages of the present invention: The present invention provides a breathing risk detection method and device based on visible light and infrared thermography. By setting two networks, the first network is used to recognize the specified area of the baby, providing a basis for judging whether the baby's face is blocked subsequently, and the second network is used to recognize the baby's forehead area, providing a basis for judging the baby's forehead temperature subsequently. Finally, combining these two judgments to comprehensively judge whether the baby has a breathing risk, such a detection result is very comprehensive and accurate. Description of the Drawings

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 The first flowchart of the breathing risk detection method provided by an embodiment of the present invention; Figure 2 The first detailed flowchart of step S200 provided by an embodiment of the present invention; Figure 3 The second detailed flowchart of step S200 provided by an embodiment of the present invention; Figure 4 The architecture diagram of the first network provided by an embodiment of the present invention; Figure 5 The detailed flowchart of step S300 provided by an embodiment of the present invention; Figure 6 The judgment logic diagram of step S302 provided by an embodiment of the present invention; Figure 7 The detailed flowchart of step S500 provided by an embodiment of the present invention; Figure 8 The detailed flowchart of step S600 provided by an embodiment of the present invention; Figure 9 The second flowchart of the breathing risk detection method provided by an embodiment of the present invention. Detailed implementation manners

[0023] The following illustrates the embodiments of the present invention through specific specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that, without conflict, the following examples and the features in the examples can be combined with each other. In addition to the specific methods, devices, and materials used in the examples, according to the knowledge of those skilled in the art in the technical field and the records of the present invention, any methods, devices, and materials similar or equivalent to those described in the embodiments of the present invention can also be used to implement the present invention.

[0024] It should be understood that the terms used in the embodiments of the present invention are for the purpose of describing specific specific implementation manners, rather than limiting the protection scope of the present invention. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the technical field of the present invention.

[0025] In the following description, numerous specific details are set forth in order to provide a more thorough explanation of embodiments of the present invention. However, it will be apparent to one of ordinary skill in the art that embodiments of the present invention may be practiced without these specific details. In some of these embodiments, well-known structures and devices are shown in block diagram form rather than in detail, in order to avoid obscuring the embodiments of the present invention.

[0026] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0027] Please refer to Figure 1 , Figure 1 A method for detecting respiratory risk based on visible light and infrared thermal images provided for an embodiment of the present invention includes steps S100 to S600.

[0028] Step S100: Obtain a visible light image and an infrared image to be processed. The visible light image can be obtained, for example, using an ordinary camera, and its image format can be RGB, for example. The infrared image, also known as an infrared thermal image, is obtained through an infrared camera and is mainly used for monitoring the temperature of an infant.

[0029] Step S200: Input the visible light image into a trained first network to obtain a first recognition result for identifying a specified area of the infant in the visible light image. Object recognition networks are relatively common and mainly perform recognition processing on images to obtain recognition results. It can be understood that generally, the first network has requirements for the size of the input image. Therefore, it is necessary to preprocess the visible light image to make it suitable for the input of the first network.

[0030] In a specific embodiment of the present invention, the specified area of the infant includes the infant's head, nose, mouth, and hand areas. The reason for choosing these areas is that these areas can best reflect whether the infant's face is blocked, resulting in a respiratory risk. Of course, other areas can also be added or other areas can be selected according to needs.

[0031] Please refer to Figure 2 In a specific embodiment of the present invention, the first network includes a third network applicable to low-illumination scenarios and a fourth network applicable to normal-illumination scenarios. Step S200 includes: S201, obtaining the actual illumination of the current scene. For example, a photosensitive sensor can be used to obtain the actual illumination of the current scene; S202, determining whether the actual illumination is less than or equal to a preset illumination value: If so, after performing brightness enhancement processing on the visible light image, input it into the trained third network to obtain a first recognition result; if not, input the visible light image into the trained fourth network to obtain a first recognition result.

[0032] To ensure that the first network can accurately identify the specified area of the baby under different illumination conditions, the visible light data is specifically divided into two types of scenarios, low illumination and normal illumination, for data partitioning and annotation, and the models for their respective scenarios are trained correspondingly, thus obtaining two trained network models, namely the third network and the fourth network. This can balance the distribution of training samples in low-illumination and normal-illumination scenarios, prevent the detection accuracy of the model from deviating in the learning process for a certain specific illumination condition, and thus ensure that the model can maintain a high-precision detection ability both in low-illumination and normal-illumination environments.

[0033] In a specific embodiment of the present invention, the above-mentioned preset illumination value is 10 to 20 lux, and preferably, 15 lux can be selected.

[0034] In a specific embodiment of the present invention, the step of performing brightness enhancement processing on the visible light image includes: performing brightness enhancement processing on the visible light image using a zero-reference depth curve estimation algorithm. The zero-reference depth curve estimation algorithm is a calculation method that directly estimates depth information from a single image without relying on labeled data or reference information. Its core idea is to infer the scene depth through self-supervised or unsupervised learning, using low-level features of the image itself (such as texture, perspective, occlusion, etc.) or physical priors (such as shadows, blurs, etc.). Its processing process is roughly as follows: First, extract the relative depth curve from the visible light image through a zero-reference algorithm (such as texture gradient, shadow analysis) to distinguish the foreground (subject) from the background; second, dynamically adjust the brightness according to the depth level, enhancing the details (brightening) of the nearby subject and moderately suppressing the distant background (to avoid overexposure); finally, combine the original image with the adjusted brightness layer, retain the natural transition, and output the enhanced image. This method does not require labeled data and realizes light and dark optimization in a lightweight manner.

[0035] Please refer to Figure 3 and Figure 4, in a specific embodiment of the present invention, the first network includes a backbone feature extraction unit, a lightweight feature pyramid structure, and a detection head; step S200 includes: S210, using the backbone feature extraction unit to perform multi-level feature extraction on the visible light image to obtain a feature map with rich semantic information; S220, using the lightweight feature pyramid structure to fuse the feature maps output by the backbone feature extraction unit to obtain a multi-scale feature map; S230, using the detection head to process the multi-scale feature map to obtain a first recognition result. In this embodiment, in order to enable the first network to be applied to edge computing devices, the present invention uses depthwise separable convolutions instead of ordinary convolutions to implement feature extraction. At the same time, in order to achieve accurate recognition of multi-scale targets, the present invention also introduces a feature pyramid structure, thereby obtaining a more accurate recognition result.

[0036] In a specific embodiment of the present invention, the backbone feature extraction unit includes five stages, and the expressions of the feature maps Y1~Y5 output by each stage are as follows: Y1 = ConvHead(X in ); Y2 = SE(DWBlock(DWBlock(MaxPool(Y1)))); Y i+1 = MaxPool(SE(DWBlock(Y i ))), i ∈ {2, 3, 4}, this formula corresponds to three formulas, and a simplified representation is adopted here.

[0037] In the above formulas, X in is the input feature of the backbone feature extraction unit. ConvHead is composed of cascaded convolutional layers (with a convolutional kernel size of 3×3), batch normalization, and a rectified linear unit (corresponding to Figure 4 BN&Relu in Figure 4 ), and a depthwise separable convolution unit (DWUnit). SE is an attention mechanism module, which is an attention mechanism module used to enhance the feature expression ability of convolutional neural networks. Its core idea is to adaptively recalibrate the channel weights of the feature map by dynamically learning the importance relationship between channels, thereby improving the sensitivity of the model to key features. DWBlock is composed of two cascaded depthwise separable convolution units. MaxPool is a max pooling layer,

[0038] The depthwise separable convolution unit (DWUnit) consists of a cascaded first convolutional layer, a second convolutional layer, batch normalization, and a rectified linear unit. The kernel sizes of the first and second convolutional layers are 1×1 and 3×3 respectively. In the depthwise separable convolution unit, pointwise convolution is first performed using the first convolutional layer with a kernel size of 1×1, then depthwise convolution is performed using the second convolutional layer with a kernel size of 3×3. Finally, batch normalization (Batch Normalization, BN) and the Relu activation function are used to stabilize the training process and introduce non-linearity, thereby enhancing the expressive power and generalization performance of the model.

[0039] Please refer to Figure 4 , in a specific embodiment of the present invention, the lightweight feature pyramid structure is divided into three branches, which respectively process the feature maps output by the last three stages of the backbone feature extraction unit. The expression is as follows: Y5’ = DWUnit(DWUnit(Y5)); Y4’ = DWUnit(DWUnit(Y4 + Upsample(DWUnit(Y5)))); Y3’ = DWUnit(DWUnit(Y3 + Upsample(DWUnit(Y4)))); Among them, Y3’, Y4’, and Y5’ are multi-scale feature maps, and DWUnit is the depthwise separable convolution unit. In this embodiment, the lightweight feature pyramid structure is constructed using the last 3 stages of the backbone feature extraction unit. A series of new feature maps with the same resolution but different semantic intensities are formed through the top-down path, and the features of the backbone network at the same level are combined with the features generated by the top-down path using lateral connections to ensure that each layer can obtain sufficient spatial details and semantic information. At the same time, a smoothing layer is used to eliminate the upsampling aliasing effect and improve the feature smoothness.

[0040] In a specific embodiment of the present invention, three detection heads are provided, which respectively process the features output by the three branches of the lightweight feature pyramid structure to obtain object detection results at different scales. In this embodiment, based on the multi-scale feature maps output by the feature pyramid, a task decoupling mechanism is used to construct 3 detection heads at different scales, thereby obtaining object detection results at different scales. The "task decoupling mechanism" means that in the object detection task, different subtasks (such as classification and localization) are separated, so that each task can be optimized independently, thereby improving the detection performance. The outputs of different detection heads are only the object detection results at different scales. For each detection head, its output generally includes the object category, confidence, detection box position parameters, and IoU (Intersection over Union). With these recognition results, the next step of processing can be carried out.

[0041] Understandably, the above steps S201 - S202 and steps S210 - S230 can coexist. That is, first, steps S201 and S202 are used to determine whether to specifically select the third network or the fourth network for processing. After determination, whether it is the third network or the fourth network, its processing process is steps S210 - S230.

[0042] Step S300: Obtain a first detection result according to the first recognition result. The first detection result is used to characterize whether the baby's face is occluded. What the first network outputs generally only includes category, confidence, detection box position parameters, and IoU information, and it needs to be processed to obtain the first detection result.

[0043] Please refer to Figure 5 , in a specific embodiment of the present invention, step S300 includes step S301 and step S302.

[0044] Step S301: Use the non - maximum suppression algorithm with an adaptive threshold to process the first recognition result to obtain the detection result of the baby's specified area. The non - maximum suppression algorithm (Non - Maximum Suppression, NMS) is a commonly used post - processing step in object detection for removing redundant detection boxes. Specifically, NMS will screen the detection boxes according to the confidence and IoU of the detection boxes, retain the detection box with the highest confidence, and at the same time remove other detection boxes whose IoU exceeds a certain threshold.

[0045] The traditional NMS algorithm uses a fixed IoU threshold (such as 0.5 or 0.7) to filter the detection boxes. However, the fixed threshold may not be able to meet the requirements of different scenarios or different target scales. For example, for dense targets, the fixed IoU threshold may cause too many detection boxes to be retained, while for sparse targets, it may over - remove detection boxes. The adaptive threshold NMS optimizes the detection result by dynamically adjusting the IoU threshold. In the present invention, the IoU threshold is set to a range (0.5 - 0.7), which means that the NMS algorithm will dynamically select the most suitable IoU threshold according to the specific situation. For example: for dense targets, a higher IoU threshold (such as 0.7) can be selected to retain more detection boxes. For sparse targets, a lower IoU threshold (such as 0.5) can be selected to avoid over - removing detection boxes.

[0046] Step S302: Obtain the first detection result according to the detection situation of the baby's head, nose, mouth, and hands in the detection result of the baby's specified area.

[0047] Please refer to Figure 6In a specific embodiment of the present invention, step S302 includes: determining whether a baby's head is detected in the baby designated area detection result: If yes, then determine whether the baby's nose and mouth are detected at the same time in the baby's designated area detection results: If yes, the first test result is normal; If not, determine whether the baby's hand is detected in the baby's designated area detection result and whether the baby's hand area is included in the baby's head area: If yes, the first test result is normal; If not, the first test result is abnormal; If not, the first detection result is abnormal.

[0048] This is a multi-layer nested logical relationship, which can also be expressed as the following logical relationship: (1) First determine whether the baby's head is detected. If not, it is an abnormal situation. If the baby's head is detected, further judgment is required; (2) On the premise that the baby's head is detected, determine whether the baby's nose and mouth are detected at the same time. If both are detected, it is normal. If not, further judgment is required; (3) On the premise that the baby's nose and mouth are not detected at the same time, determine whether the baby's hand is detected and the baby's hand area is included in the baby's head area. Here are also two conditions that need to be met at the same time, that is, the baby's hand needs to be detected and the baby's hand area needs to be included in the baby's head area. When both are met, it is normal. Otherwise, it is an abnormal situation. The reason for adding this step of judgment is to avoid the problem of false detection caused by infants covering their noses or mouths with their hands.

[0049] Step S400: input the infrared image into the trained second network to obtain a second recognition result to identify the baby's forehead area in the infrared image.

[0050] In a specific embodiment of the present invention, the structure of the second network is the same as that of the first network. The training data of the first network is a visible light image with a designated area of the baby labeled, and the training data of the second network is an infrared image with a forehead area of the baby labeled. The structures of the first network and the second network are exactly the same. Figure 4 As shown in the figure, the difference is that the training process of the two is different. This is because both network models are used for target detection, so they can be used interchangeably. When annotating data, for example, you can use the LabelImg tool to annotate data, and when selecting training data, try to include overhead image data of babies lying on their left side, right side, and lying flat, so that the trained network model has better recognition effect.

[0051] Step S500: Obtain a second detection result according to the second recognition result, where the second detection result is used to characterize the baby's body temperature.

[0052] Please refer to Figure 7 , in a specific embodiment of the present invention, step S500 includes steps S501 to S503.

[0053] Step S501: Use the non-maximum suppression algorithm with an adaptive threshold to process the second recognition result to obtain the detection result of the baby's forehead area. The processing of this step is similar to the post-processing of the first network. By using the non-maximum suppression algorithm with an adaptive threshold, it can ensure that the finally obtained baby's forehead area is accurate enough.

[0054] Step S502: Obtain the baby's forehead body temperature according to the position of the detection frame in the detection result of the baby's forehead area and the mapping relationship between the pixel values and the temperature of the infrared image. After obtaining the position of the detection frame of the baby's forehead area, the temperature corresponding to the pixel values of the infrared image within the detection frame area can be calculated. Considering that the detection frame has a certain size, there will be a relatively large number of pixel points, and the average value of the temperatures of these pixel points can be calculated as the final baby's forehead body temperature.

[0055] Step S503: Determine whether the baby's forehead body temperature is within a preset normal temperature range: If so, the second detection result is normal; if not, the second detection result is abnormal. The preset normal temperature range can be, for example, 36.2 - 37.2 °C.

[0056] Step S600: Obtain a baby's breathing risk judgment result according to the first detection result and the second detection result. In the present invention, the first detection result and the second detection result are used to comprehensively judge whether the baby has a breathing risk, and the judgment result will be more accurate.

[0057] Please refer to Figure 8 , in a specific embodiment of the present invention, step S600 includes step S601 and step S602.

[0058] Step S601: Determine whether the first detection results of the first preset number of consecutive multi-frame visible light images are all abnormal: If so, the baby has a breathing risk; otherwise, the baby does not have a breathing risk. To avoid the detection error of a single-frame image, in this step, the first detection results of multiple consecutive multi-frame visible light images are used for judgment, and such a judgment result will be more accurate and reliable.

[0059] Step S602: Determine whether the second detection results of the second preset number of consecutive infrared images are all abnormal. If so, the baby is at risk of breathing; otherwise, the baby is not at risk of breathing. For the same reason as above, there is a breathing risk only when the second detection results of multiple infrared images are all abnormal. In this step, the second preset number can be the same as or different from the first preset number in the previous step, which can be set as needed.

[0060] In a specific embodiment of the present invention, it is also possible to only acquire infrared images. In step S200, the infrared images are input into the trained first network to obtain the first recognition result. However, in this case, it is not necessary to judge the actual illuminance of the current scene.

[0061] It should be noted that the step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split into multiple steps. As long as the same logical relationship is included, they are all within the protection scope of this application. Adding insignificant modifications to the algorithm or process or introducing insignificant designs, but not changing the core design of the algorithm and process, are all within the protection scope of this patent. For example Figure 9 shows another step division and expression method. Although the process is different, the overall idea is the same as that of the Figure 1 scheme.

[0062] A breathing risk detection device based on visible light and infrared thermal imaging provided by an embodiment of the present invention includes a data acquisition unit, a first recognition unit, a first detection unit, a second recognition unit, a second detection unit, and a determination unit. Among them, the data acquisition unit is used to acquire the visible light image and the infrared image to be processed; the first recognition unit is used to input the visible light image into the trained first network to obtain the first recognition result to recognize the specified area of the baby in the visible light image; the first detection unit is used to obtain the first detection result according to the first recognition result, and the first detection result is used to characterize whether the baby's face is blocked; the second recognition unit is used to input the infrared image into the trained second network to obtain the second recognition result to recognize the forehead area of the baby in the infrared image; the second detection unit is used to obtain the second detection result according to the second recognition result, and the second detection result is used to characterize the baby's body temperature; the determination unit is used to obtain the baby breathing risk judgment result according to the first detection result and the second detection result.

[0063] It should be noted that the breathing risk detection device in this embodiment corresponds to the above-mentioned breathing risk detection method, and the functional modules in the breathing risk detection device respectively correspond to the corresponding steps in the breathing risk detection method. The breathing risk detection device in this embodiment can be implemented in cooperation with the breathing risk detection method. That is, without conflict, the relevant technical details mentioned in the breathing risk detection method in the above-mentioned embodiment can also be applied to the breathing risk detection device in this embodiment.

[0064] Generally speaking, the present invention realizes the effective detection of whether the baby's face (mouth and nose) is blocked under different illumination conditions by integrating a photosensitive sensor to evaluate the environmental light conditions, selecting an appropriate image preprocessing strategy, and using a lightweight deep learning model and a specific post-processing method. At the same time, infrared thermal imaging technology is used to monitor the baby's forehead temperature in real time to judge whether there is a fever or low temperature phenomenon, and then comprehensively evaluate the baby's respiratory health status.

[0065] During the implementation process, first trigger the photosensitive sensor to determine the illumination level of the current environment, and intelligently switch different image preprocessing modes according to the set threshold (such as 15 lux) to ensure clear image quality even in low-light environments. For the preprocessed images, input them into a specially optimized lightweight deep learning algorithm, which can efficiently identify the characteristics of the baby-related parts and combine post-processing to judge whether there is foreign object occlusion. In addition, the forehead temperature measurement function based on thermal infrared images can supplement visual information to help understand the baby's physical condition more comprehensively. If abnormal situations are detected, such as abnormal occlusion for multiple consecutive frames or body temperature deviating from the normal range, an alarm notification can be automatically sent to the parents and a real-time picture can be provided for viewing, thus ensuring the safety of the baby.

[0066] The present invention not only improves the safety guarantee level during the baby's sleep, but also provides new ideas and technical support for the research and development of intelligent monitoring systems, and has important practical value and social significance. By integrating multiple sensing technologies and advanced computing models, the present invention effectively solves the problems existing in the prior art and brings an innovative solution to the field of baby monitoring.

[0067] The above embodiments are only illustrative of the principles and effects of the present invention, and are not used to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A breathing risk detection method based on visible light and infrared thermal imaging, characterized in that, Including: Obtain a visible light image and an infrared image to be processed; Input the visible light image into a trained first network to obtain a first recognition result for identifying a specified area of the baby in the visible light image; Obtain a first detection result according to the first recognition result, where the first detection result is used to characterize whether the baby's face is blocked; Input the infrared image into a trained second network to obtain a second recognition result for identifying the forehead area of the baby in the infrared image; Obtain a second detection result according to the second recognition result, where the second detection result is used to characterize the baby's body temperature; Obtain a baby breathing risk judgment result according to the first detection result and the second detection result.

2. The breathing risk detection method based on visible light and infrared thermal imaging according to claim 1, wherein The first network includes a third network applicable to low-light scenes and a fourth network applicable to normal illumination scenes; Inputting the visible light image into the trained first network to obtain the first recognition result includes: Obtain the actual illumination of the current scene; Judge whether the actual illumination is less than or equal to a preset illumination value: If so, after performing brightness enhancement processing on the visible light image, input it into the trained third network to obtain the first recognition result; If not, input the visible light image into the trained fourth network to obtain the first recognition result.

3. The breathing risk detection method based on visible light and infrared thermal imaging according to claim 1, characterized in that The first network includes a backbone feature extraction unit, a lightweight feature pyramid structure, and a detection head; Inputting the visible light image into the trained first network to obtain the first recognition result includes: Use the backbone feature extraction unit to perform multi-level feature extraction on the visible light image to obtain a feature map with rich semantic information; Use the lightweight feature pyramid structure to fuse the feature maps output by the backbone feature extraction unit to obtain a multi-scale feature map; Use the detection head to process the multi-scale feature map to obtain the first recognition result.

4. The breathing risk detection method based on visible light and infrared thermal imaging according to claim 3, wherein The backbone feature extraction unit includes five stages, and the expressions of the feature maps Y1~Y5 output by each stage are as follows: Y1 = ConvHead(X in ); Y2 = SE(DWBlock(DWBlock(MaxPool(Y1)))); Y i+1 =MaxPool(SE(DWBlock(Y i ))), i ∈ {2, 3, 4}; Among them, X in is the input feature of the backbone feature extraction unit. ConvHead is composed of cascaded convolutional layers, batch normalization and rectified linear unit, and depthwise separable convolution units. SE is an attention mechanism module. DWBlock is formed by cascading two depthwise separable convolution units. MaxPool is a max pooling layer. The depthwise separable convolution unit is composed of cascaded first convolutional layer, second convolutional layer, batch normalization and rectified linear unit. The convolutional kernel sizes of the first convolutional layer and the second convolutional layer are 1×1 and 3×3 respectively.

5. The breathing risk detection method based on visible light and infrared thermal imaging according to claim 4, wherein The lightweight feature pyramid structure is divided into three branches, which respectively process the feature maps output by the last three stages of the backbone feature extraction unit, and its expression is as follows: Y5’ = DWUnit(DWUnit(Y5)); Y4’ = DWUnit(DWUnit(Y4 + Upsample(DWUnit(Y5)))); Y3’ = DWUnit(DWUnit(Y3 + Upsample(DWUnit(Y4)))); Among them, Y3’, Y4’, and Y5’ are the multi-scale feature maps, and DWUnit is the depthwise separable convolution unit.

6. The breathing risk detection method based on visible light and infrared thermal imaging according to claim 1, wherein Obtaining the first detection result according to the first recognition result includes: Use the non-maximum suppression algorithm with an adaptive threshold to process the first recognition result to obtain the detection result of the specified area of the baby; The first detection result is obtained according to the detection situations of the baby's head, nose, mouth and hands in the baby designated area detection result.

7. The breathing risk detection method based on visible light and infrared thermal imaging according to claim 6, wherein Obtaining the first detection result according to the detection situations of the baby's head, nose, mouth and hands in the baby designated area detection result includes: Judging whether the baby's head is detected in the baby designated area detection result: If so, then judging whether the baby's nose and mouth are simultaneously detected in the baby designated area detection result: If so, the first detection result is normal; If not, then judging whether the baby's hand is detected in the baby designated area detection result and whether the baby's hand area is included in the baby's head area: If so, the first detection result is normal; If not, the first detection result is abnormal; If not, the first detection result is abnormal.

8. The breathing risk detection method based on visible light and infrared thermal imaging according to claim 1, characterized in that Obtaining the second detection result according to the second recognition result includes: Using the non-maximum suppression algorithm with an adaptive threshold to process the second recognition result to obtain the baby forehead area detection result; Obtaining the baby forehead body temperature according to the detection frame position in the baby forehead area detection result and the mapping relationship between the pixel value and the temperature of the infrared image; Judging whether the baby forehead body temperature is within a preset normal temperature range: if so, the second detection result is normal, if not, the second detection result is abnormal.

9. The breathing risk detection method based on visible light and infrared thermal imaging according to claim 1, wherein Obtaining the baby breathing risk judgment result according to the first detection result and the second detection result includes: Judging whether the first detection results of the first preset number of consecutive frames of visible light images are all abnormal: if so, the baby has a breathing risk, otherwise the baby does not have a breathing risk; Judging whether the second detection results of the second preset number of consecutive frames of infrared images are all abnormal: if so, the baby has a breathing risk, otherwise the baby does not have a breathing risk.

10. A breathing risk detection device based on visible light and infrared thermal imaging, characterized in that, Including: A data acquisition unit, configured to acquire the visible light image and the infrared image to be processed; A first recognition unit, configured to input the visible light image into a trained first network to obtain a first recognition result for recognizing the baby designated area in the visible light image; A first detection unit, configured to obtain a first detection result according to the first recognition result, where the first detection result is used to characterize whether the baby's face is blocked; A second recognition unit, configured to input the infrared image into a trained second network to obtain a second recognition result for recognizing the baby forehead area in the infrared image; A second detection unit, configured to obtain a second detection result according to the second recognition result, where the second detection result is used to characterize the baby's body temperature; And A determination unit, configured to obtain the baby breathing risk judgment result according to the first detection result and the second detection result.

Images