Infant behavior recognition method, device and equipment based on visible light-thermal infrared imaging

Abstract

The present invention relates to the technical field of infant behavior recognition, solves the problem in the prior art that it is impossible to accurately analyze and identify real-time and dynamic infant behaviors, and provides a method, device and equipment for infant behavior recognition using visible light-thermal infrared imaging. The method includes: obtaining real-time thermal infrared imaging images and visible light images in an infant care scene; performing target detection on the visible light image, and outputting detection frame position information and category information of a preset target related to infant care; obtaining a preset thermal infrared imaging projection area, and when the detection frame of the preset target appears in the thermal infrared projection area, outputting thermal imaging morphological feature information and thermal imaging temperature change feature information according to preset rules; and classifying infant behaviors and outputting classification results. The present invention realizes real-time, accurate and non-contact infant behavior recognition, providing important support and assistance for the care and safety of infants.

Description

Technical Field

[0001] The present invention relates to the technical field of infant behavior recognition, and in particular to a method, device and equipment for infant behavior recognition using visible light-thermal infrared imaging. Background Art

[0002] Infants and young children have limited cognition and coping abilities for the environment, and are prone to various potential dangers, such as rolling, crawling, and touching dangerous objects. Infant and young children care can detect and avoid these potential dangers in a timely manner, ensuring the safety and health of infants and young children. Since infant and young children care plays a vital role in the lives of infants and young children, it is very important to classify and understand infant and young children's behaviors. Through the classification of infant and young children's behaviors, the behavior patterns, development stages, and possible abnormalities or delays of infants and young children can be monitored and evaluated, which helps to detect and intervene in potential developmental problems early and promote the healthy and all-round development of infants and young children. Since different infants and young children may have different behavioral characteristics and needs, through behavioral classification, we can understand the individual differences of infants and young children and provide them with personalized care and education, which helps to meet the development needs of infants and young children and promote their maximum potential.

[0003] Traditional infant behavior classification technology usually only uses single visual information or sensor data for analysis. However, infant behavior is diverse and complex, and single visual information or sensor data may not be able to fully capture the characteristics and details of the behavior. At the same time, relying solely on single visual information or sensor data for analysis will ignore important information in other perceptual modalities, thereby reducing the accuracy and completeness of behavior classification. Therefore, traditional infant behavior classification technology only uses single visual information or sensor data for analysis, which has limitations in accuracy, completeness, and robustness, and cannot fully capture and understand the diversity and complexity of infant behavior.

[0004] The existing Chinese patent CN114120176A discloses a behavior analysis method for the fusion of far-infrared and visible light video images, the method comprising: extracting frame images from pre-acquired infrared video files and visible light video files respectively to obtain thermal infrared imaging images and visible light images; performing image enhancement on the thermal infrared imaging images and visible light images respectively; registering the enhanced thermal infrared imaging images and visible light images respectively; using a preset generative adversarial network to fuse the registered thermal infrared imaging images and visible light images to obtain a target fused image; using a preset target detection model to perform target detection on the target fused image to obtain user behaviors in the infrared video files and visible light video files. However, the key step in this scheme is to align the thermal infrared imaging images and the visible light images. In infant behavior recognition, there may be large temporal and spatial differences between the thermal infrared imaging images and the visible light images. Since the behavior of infants and young children is highly dynamic and unpredictable, it is difficult to ensure that the two images are completely consistent in time and space, resulting in the accuracy of the alignment results being limited. At the same time, in infant behavior recognition, the information reflected by the thermal infrared imaging images and the visible light images may be different. The two images have different features, resolution, and sensitivity. Using a generative adversarial network for fusion may lead to information loss or confusion, which may have a negative impact on subsequent target detection and behavior recognition.

[0005] Therefore, how to use visible light and infrared imaging to accurately analyze and identify real-time and dynamic infant behavior is an urgent problem to be solved. Summary of the invention

[0006] In view of this, the present invention provides a method, device and equipment for identifying infant behavior using visible light and thermal infrared imaging, so as to solve the problem in the prior art that it is impossible to use visible light and infrared imaging to accurately analyze and identify real-time and dynamic infant behavior.

[0007] The technical solution adopted by the present invention is:

[0008] In a first aspect, the present invention provides a method for identifying infant behavior using visible light-thermal infrared imaging, the method comprising:

[0009] S1: Acquire real-time thermal infrared imaging images and visible light images in infant care scenarios;

[0010] S2: performing target detection on the visible light image, and outputting detection frame position information and category information of preset targets related to infant care;

[0011] S3: obtaining a preset thermal infrared imaging projection area, and when a detection frame of a preset target appears in the thermal infrared projection area, outputting thermal imaging morphological feature information and thermal imaging temperature change feature information corresponding to the target projection area where the detection frame is located according to a preset rule;

[0012] S4: Classify the infant's behavior based on the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information, and output the classification result.

[0013] Preferably, the preset targets include at least the following targets: infant head, infant head and shoulders, adult head, adult head and shoulders, infant face, and adult face.

[0014] Preferably, S3 includes:

[0015] S31: Acquire the target projection area where the detection frame is located, and divide the target projection area into head, face and head-shoulder projection areas;

[0016] S32: performing normalization processing on the pixel points in the head, face and head-shoulder projection areas, and outputting a thermal infrared imaging pixel normalization matrix;

[0017] S33: Processing the thermal infrared imaging pixel normalization matrix, and outputting the thermal imaging morphological feature information according to the image connected domain.

[0018] Preferably, the S3 further includes:

[0019] S34: sampling the pixel points in the head, face and head-shoulder projection areas according to the temperature of the pixel points to obtain a first average temperature of the sampling points;

[0020] S35: Acquire the second average temperature of the previous sampling point in the same projection area, and output the temperature difference as the thermal imaging temperature change characteristic information based on the first average temperature and the second average temperature.

[0021] Preferably, the S31 includes:

[0022] S311: Acquire the head height, where the head height includes the head of an infant and a head of an adult;

[0023] S312: according to the head height, taking a preset ratio of the head height as a target height for extending the shoulders;

[0024] S313: Acquire the starting position of the shoulder, extend the shoulder downward by the target height according to the target height, and output the head-shoulder projection area.

[0025] Preferably, the S32 includes:

[0026] S321: Acquire pixel points in the head, face and head-shoulder projection areas;

[0027] S322: Compare the real-time temperatures of the pixels to obtain the maximum and minimum temperatures in each projection area;

[0028] S323: Output the normalized matrix according to the real-time temperature, the maximum temperature and the minimum temperature.

[0029] Preferably, the S4 includes:

[0030] S41: taking the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information as an observation sequence;

[0031] S42: inputting the observation sequence into a pre-trained hidden Markov model, classifying and identifying the observation sequence, and outputting the matching degree between the observation sequence and a plurality of infant behaviors;

[0032] S43: Classify the infant's behavior according to the matching degree and output the classification result.

[0033] In a second aspect, the present invention provides a device for identifying infant behavior using visible light-thermal infrared imaging, the device comprising:

[0034] An image acquisition module is used to acquire real-time thermal infrared imaging images and visible light images in infant care scenarios;

[0035] A target detection module, used to perform target detection on the visible light image, and output detection frame position information and category information of preset targets related to infant care;

[0036] A feature extraction module is used to obtain a preset thermal infrared imaging projection area, and when a detection frame of a preset target appears in the thermal infrared projection area, output the thermal imaging morphological feature information and thermal imaging temperature change feature information corresponding to the target projection area where the detection frame is located according to a preset rule;

[0037] The behavior classification module is used to classify infant behaviors based on the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information, and output classification results.

[0038] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect in the above-mentioned embodiment.

[0039] In a fourth aspect, an embodiment of the present invention further provides a storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method of the first aspect in the above-mentioned embodiment is implemented.

[0040] In summary, the beneficial effects of the present invention are as follows:

[0041] The present invention provides a method, device and equipment for identifying infant behavior by visible light-thermal infrared imaging, the method comprising: obtaining real-time thermal infrared imaging images and visible light images in an infant care scene; performing target detection on the visible light image, and outputting detection frame position information and category information of a preset target related to infant care; obtaining a preset thermal infrared imaging projection area, and when the detection frame of the preset target appears in the thermal infrared projection area, outputting thermal imaging morphological feature information and thermal imaging temperature change feature information corresponding to the target projection area where the detection frame is located according to a preset rule; classifying infant behavior according to the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information, and outputting classification results. The present invention combines thermal infrared imaging and visible light images, the visible light image provides shape and appearance information, and the thermal infrared imaging provides temperature and heat distribution information. By fusing these two types of information, a more comprehensive and accurate infant behavior recognition result can be obtained; at the same time, thermal infrared imaging technology can monitor without contacting infants and will not cause interference or discomfort to them. This is very important for maintaining the natural state of infants and children, and also reduces the risk of cross infection caused by direct contact with infants and children. In summary, the present invention combines thermal infrared imaging and visible light imaging, and combines target detection and multiple feature analysis methods to achieve real-time, accurate, and non-contact infant behavior recognition, providing important support and assistance for the care and safety of infants and young children. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0043] Figure 1 It is a schematic diagram of the overall working process of the method for identifying infant behavior by visible light-thermal infrared imaging in Example 1 of the present invention;

[0044] Figure 2 This is a schematic diagram of the process of extracting thermal imaging morphological feature information in Example 1 of the present invention;

[0045] Figure 3 A schematic diagram of a process for obtaining a head and shoulder projection area in Embodiment 1 of the present invention;

[0046] Figure 4 This is a schematic diagram of a process for obtaining a normalized matrix of thermal infrared imaging pixels in Embodiment 1 of the present invention;

[0047] Figure 5 This is a schematic diagram of the process of extracting temperature change characteristic information of thermal imaging in Example 1 of the present invention;

[0048] Figure 6 This is a schematic diagram of a process for classifying infant behaviors in Embodiment 1 of the present invention;

[0049] Figure 7 This is a structural block diagram of an infant behavior recognition device using visible light-thermal infrared imaging in Example 2 of the present invention;

[0050] Figure 8 It is a schematic diagram of the structure of the electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments can be combined with each other, all within the protection scope of the present invention.

[0052] Example 1

[0053] See also Figure 1Embodiment 1 of the present invention discloses a method for identifying infant behavior using visible light-thermal infrared imaging, the method comprising:

[0054] S1: Acquire real-time thermal infrared imaging images and visible light images in infant care scenarios;

[0055] Specifically, the image planes of optical imaging and thermal infrared imaging are corrected, and each point on the thermal infrared image plane is projected onto the optical image plane through the affine transformation matrix M. The optical image plane is larger than the thermal infrared image plane, and the thermal infrared image plane is located in the middle of the optical image plane; a thermal infrared camera or thermal infrared sensor is used to capture the thermal radiation information of infants and young children in real time. These devices can detect and record infrared radiation from the surface of target objects or human bodies to generate thermal infrared imaging images; ordinary cameras or camera equipment are used to obtain visible light images of infants and young children in real time. These devices can capture visible light in the surrounding environment and convert it into digital images. By simultaneously acquiring thermal infrared imaging and visible light images, multimodal information can be obtained, providing a more comprehensive data basis for infant behavior recognition. These image data can be used for subsequent image processing, target detection, behavior classification and other tasks to achieve real-time monitoring and recognition of infant behavior.

[0056] S2: performing target detection on the visible light image, and outputting detection frame position information and category information of preset targets related to infant care;

[0057] Specifically, a visible light image dataset containing infant heads, infant heads and shoulders, adult heads, adult heads and shoulders, infant faces, and adult faces is collected. The dataset is annotated to mark the location and category information of the target in each image; the YOLOv6 target detection algorithm is used to train the model using the prepared dataset as a training sample. During the training process, the model will learn the feature representation and position regression of the target to accurately detect the location and category of the target; during the training process, the performance of the model can be optimized by adjusting the model's hyperparameters, network structure, and training strategy. This includes adjusting the learning rate, adding data augmentation operations, and performing model fusion to improve the accuracy and robustness of target detection; the trained target detection model is used to perform target detection on the collected visible light image sequence. The model processes each image and outputs the location information and category information of the detected target. Through this information, the location and category of the head, head and shoulders, and face of infants and adults in the image can be understood.

[0058] S3: obtaining a preset thermal infrared imaging projection area, and when a detection frame of a preset target appears in the thermal infrared projection area, outputting thermal imaging morphological feature information and thermal imaging temperature change feature information corresponding to the target projection area where the detection frame is located according to a preset rule;

[0059] Specifically, a preset thermal infrared imaging projection area is obtained, and when the detection frame of the preset target appears in the thermal infrared projection area, the thermal imaging morphological features and temperature change features of the target are extracted according to preset rules; wherein the preset rules include: normalization processing rules: for the thermal infrared imaging projection area where the face detection frame, the head detection frame and the head and shoulder detection frame are located, the pixel values ​​are normalized to extract the thermal imaging morphological features; morphological feature description rules: by dilating and corroding the normalized thermal infrared image and calculating the image connected domain, information describing the morphological features of the area is obtained; temperature change feature description rules: for the thermal infrared imaging projection area where the face detection frame, the head detection frame and the head and shoulder detection frame are located, the top 10% sampling points with the highest temperature are selected, and the average temperature and the difference with the previous temperature in the same area are calculated to describe the temperature change characteristics of the area. These features can be further used for subsequent behavior classification, recognition and monitoring, providing more detailed information and data for infant care scenarios.

[0060] In one embodiment, see Figure 2 , said S3 comprises:

[0061] S31: Acquire the target projection area where the detection frame is located, and divide the target projection area into head, face and head-shoulder projection areas;

[0062] In one embodiment, see Figure 3 , the S31 comprises:

[0063] S311: Acquire the head height, where the head height includes the head of an infant and a head of an adult;

[0064] Specifically, in the target detection stage, the height of the head is obtained by measuring the height of the target detection frame. This can be achieved by analyzing the position information of the target detection frame to determine the height of the head in the target projection area.

[0065] S312: according to the head height, taking a preset ratio of the head height as a target height for extending the shoulders;

[0066] Specifically, according to a preset proportional relationship, the head height of the preset proportion is used as the target height of the shoulder extension. Specifically, according to a preset proportional relationship, the head height is multiplied by the proportional coefficient to obtain the target height of the shoulder extension. For example, if the proportional coefficient is 4, the target height is 4 times the head height.

[0067] S313: Acquire the starting position of the shoulder, extend the shoulder downward by the target height according to the target height, and output the head-shoulder projection area.

[0068] Specifically, the starting position of the shoulder is determined according to the position and height of the head; the statistical area is expanded by 4 head heights below the shoulder, while keeping the width unchanged; the expanded area is used as the head-shoulder projection area for subsequent normalization processing. By expanding the statistical area by 4 head heights below the shoulder, more thermal infrared information related to the head and shoulders can be included, which improves the richness and accuracy of the features; keeping the width of the head-shoulder area unchanged ensures that the same horizontal information as the original head area is maintained in the expanded area, thereby better preserving the head features.

[0069] S32: performing normalization processing on the pixel points in the head, face and head-shoulder projection areas, and outputting a thermal infrared imaging pixel normalization matrix;

[0070] Specifically, determine the thermal infrared imaging projection area where the face detection frame, the head detection frame, and the head and shoulder detection frame are located; find the maximum value Tmax and the minimum value Tmin in the area where each detection frame is located; for any pixel point T in the area, calculate its normalized value Tnorm according to the following formula: Tnorm = (T-Tmin) / (Tmax-Tmin), and map the temperature range of the thermal infrared imaging projection area to a standardized value between 0 and 1 by normalizing the pixel points, so that the temperature characteristics of different areas are comparable and consistent. By normalizing the pixel points in the thermal infrared imaging projection area, the temperature difference between different areas can be eliminated, so that the temperature characteristics of different areas can be effectively compared and analyzed; the normalized value range is 0 to 1, which can provide a unified standard, so that the temperature characteristics of different areas have a consistent reference benchmark, which is convenient for subsequent morphological characteristics and temperature change characteristics analysis; after normalization, the morphological characteristics and temperature change characteristics of the thermal imaging projection area can be more accurately extracted, thereby providing more reliable feature information for subsequent behavior classification.

[0071] In one embodiment, see Figure 4 , the S32 comprises:

[0072] S321: Acquire pixel points in the head, face and head-shoulder projection areas;

[0073] S322: Compare the real-time temperatures of the pixels to obtain the maximum and minimum temperatures in each projection area;

[0074] S323: Output the normalized matrix according to the real-time temperature, the maximum temperature and the minimum temperature.

[0075] S33: Processing the thermal infrared imaging pixel normalization matrix, and outputting the thermal imaging morphological feature information according to the image connected domain.

[0076] Specifically, the normalized matrix obtained is prepared before image processing, and the normalized matrix is ​​converted into a binary image. By setting a threshold, the pixels greater than the threshold are set to 1, and the pixels less than or equal to the threshold are set to 0; the binary image is dilated. The dilation operation can expand the pixels of the target area, making the target area more continuous and complete. Basic dilation algorithms can be used, such as convolution operations or dilation operations in mathematical morphology; the dilated image is corroded. The corrosion operation can eliminate small noise points in the image, making the target area clearer and more accurate. Convolution operations or corrosion operations in mathematical morphology can also be used; the image connected domain algorithm is used to perform connected domain analysis on the processed image. Connected domain analysis can identify the area composed of continuous pixels in the image, and then calculate the morphological features of the area, such as area, perimeter, center position, etc. The morphological feature description of the thermal imaging projection area is obtained, including the connectivity, geometric shape and other information of the target area. These feature descriptions can be used for further behavior classification and analysis to help identify specific behavior patterns or actions of infants and young children.

[0077] In one embodiment, see Figure 5 , the S3 further includes:

[0078] S34: sampling the pixel points in the head, face and head-shoulder projection areas according to the temperature of the pixel points to obtain a first average temperature of the sampling points;

[0079] S35: Acquire the second average temperature of the previous sampling point in the same projection area, and output the temperature difference as the thermal imaging temperature change characteristic information based on the first average temperature and the second average temperature.

[0080] Specifically, when the target detection frame appears in the thermal infrared imaging projection area, the corresponding face detection frame, head detection frame and head and shoulder detection frame are first determined; for each detection frame, the top 10% sampling points with the highest temperature are extracted. The top 10% sampling points with the highest temperature can be selected by sorting the temperature values; the average temperature Ti of the selected sampling points is calculated, that is, the temperature values ​​of the selected sampling points are averaged; and the difference TD i from the temperature of the same area last time is calculated. If a similar process has been performed before and the last average temperature value Ti-1 is obtained, the difference TD i=TiTi-1 can be calculated. The temperature change characteristics of the thermal imaging of the corresponding area can be described by the average temperature and temperature difference of the sampling points. These characteristics can be used to analyze and compare the temperature changes in different time periods, and further understand the heat distribution and heat energy transfer in the behavior of infants and young children. This helps to identify the thermal characteristic patterns of specific behaviors, such as fever, sweating, etc., and provides additional information to assist in behavior classification and analysis.

[0081] S4: Classify the infant's behavior based on the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information, and output the classification result.

[0082] Specifically, the position information and category information of the detection frame, the thermal imaging morphological feature information and the thermal imaging temperature change feature information are obtained as an observation sequence; the observation sequence is input into a pre-trained classification model, such as a hidden Markov model (HMM) or other machine learning model, for classifying and identifying the observation sequence; the classification model classifies the infant's behavior according to the position information, category information, thermal imaging morphological features and thermal imaging temperature change features in the observation sequence, and outputs the classification result. By using the position information, category information, thermal imaging morphological features and thermal imaging temperature change features for classification, the behavior of infants and young children can be accurately judged and classified, helping guardians to understand the status of infants and young children in a timely manner and provide effective care and management.

[0083] In one embodiment, see Figure 6 , said S4 comprises:

[0084] S41: taking the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information as an observation sequence;

[0085] Specifically, the location information and category information of the detection frame are obtained from the preset target detection results, and the morphological feature information and temperature change feature information corresponding to the target detection frame in the thermal imaging image are extracted. Combining multi-source information (location, category, morphological features, temperature change features) enriches the observation sequence of infant behavior, provides a more comprehensive and accurate data foundation, and helps to more accurately describe and identify infant behavior.

[0086] S42: inputting the observation sequence into a pre-trained hidden Markov model, classifying and identifying the observation sequence, and outputting the matching degree between the observation sequence and a plurality of infant behaviors;

[0087] Specifically, the pre-trained hidden Markov model is used to analyze and pattern match the observation sequence, and the matching degree between the observation sequence and various infant behaviors is calculated. Through the classification and recognition of the hidden Markov model, the matching degree between the observation sequence and various infant behaviors can be more accurately judged based on the various feature information in the observation sequence, thereby improving the accuracy and reliability of behavior classification.

[0088] S43: Classify the infant's behavior according to the matching degree and output the classification result.

[0089] Specifically, according to the matching results, the infant behaviors corresponding to the observation sequence are classified and the final classification results are output. Behavior classification based on matching can determine the most likely behavior category based on the matching degree between the observation sequence and various infant behaviors, provide specific classification results for infant behaviors, and provide useful information and guidance for infant caregivers or system users.

[0090] Example 2

[0091] See also Figure 7 Embodiment 2 of the present invention further provides a device for identifying infant behavior using visible light-thermal infrared imaging, the device comprising:

[0092] An image acquisition module is used to acquire real-time thermal infrared imaging images and visible light images in infant care scenarios;

[0093] A target detection module, used to perform target detection on the visible light image, and output detection frame position information and category information of preset targets related to infant care;

[0094] A feature extraction module is used to obtain a preset thermal infrared imaging projection area, and when a detection frame of a preset target appears in the thermal infrared projection area, output the thermal imaging morphological feature information and thermal imaging temperature change feature information corresponding to the target projection area where the detection frame is located according to a preset rule;

[0095] The behavior classification module is used to classify infant behaviors based on the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information, and output classification results.

[0096] Specifically, the device for identifying infant behavior using visible light-thermal infrared imaging provided by the embodiment of the present invention comprises: an image acquisition module for acquiring real-time thermal infrared imaging images and visible light images in infant care scenes; a target detection module for performing target detection on the visible light image and outputting detection frame position information and category information of preset targets related to infant care; a feature extraction module for acquiring a preset thermal infrared imaging projection area, and when the detection frame of the preset target appears in the thermal infrared projection area, outputting thermal imaging morphological feature information and thermal imaging temperature change feature information corresponding to the target projection area where the detection frame is located according to preset rules; a behavior classification module for classifying infant behavior according to the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information, and outputting classification results. The device combines thermal infrared imaging and visible light images, the visible light images provide shape and appearance information, and the thermal infrared imaging provides temperature and heat distribution information. By fusing these two types of information, a more comprehensive and accurate infant behavior recognition result can be obtained; at the same time, thermal infrared imaging technology can monitor infants without contacting them, and will not cause interference or discomfort to them. This is very important for maintaining the natural state of infants and young children, and also reduces the risk of cross-infection caused by direct contact with infants and young children. In summary, this device combines thermal infrared imaging and visible light imaging, and combines target detection and multiple feature analysis methods to achieve real-time, accurate, and non-contact infant behavior recognition, providing important support and assistance for the care and safety of infants and young children.

[0097] Example 3

[0098] In addition, combined Figure 1 The method for identifying infant behavior using visible light-thermal infrared imaging described in Example 1 of the present invention can be implemented by an electronic device. Figure 8 A schematic diagram of the hardware structure of an electronic device provided in Embodiment 3 of the present invention is shown.

[0099] The electronic device may include a processor and a memory storing computer program instructions.

[0100] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.

[0101] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside a data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0102] The processor implements any one of the infant behavior recognition methods using visible light-thermal infrared imaging in the above embodiments by reading and executing computer program instructions stored in the memory.

[0103] In one example, the electronic device may further include a communication interface and a bus. Figure 8 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.

[0104] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0105] The bus includes hardware, software or both, and the parts of the device are coupled to each other. For example, but not limitation, the bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In suitable cases, the bus may include one or more buses. Although the embodiment of the present invention describes and shows a specific bus, the present invention considers any suitable bus or interconnection.

[0106] Example 4

[0107] In addition, in combination with the method for identifying infant behavior by visible light-thermal infrared imaging in the above embodiment 1, embodiment 4 of the present invention can also provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any of the methods for identifying infant behavior by visible light-thermal infrared imaging in the above embodiments is implemented.

[0108] In summary, the embodiments of the present invention provide a method, device and apparatus for infant behavior recognition using visible light-thermal infrared imaging.

[0109] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0110] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0111] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0112] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A method for identifying infant behavior using visible light-thermal infrared imaging, characterized in that: The method comprises: S1: Acquire real-time thermal infrared imaging images and visible light images in infant care scenarios; S2: performing target detection on the visible light image, and outputting detection frame position information and category information of preset targets related to infant care; S3: obtaining a preset thermal infrared imaging projection area, and when a detection frame of a preset target appears in the thermal infrared projection area, outputting thermal imaging morphological feature information and thermal imaging temperature change feature information corresponding to the target projection area where the detection frame is located according to a preset rule; S4: classifying the infant's behavior according to the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information, and outputting the classification result; The preset targets include at least the following targets: infant head, infant head and shoulders, adult head, adult head and shoulders, infant face, and adult face; The S3 includes: S31: Acquire the target projection area where the detection frame is located, and divide the target projection area into head, face and head-shoulder projection areas; S32: performing normalization processing on the pixel points in the head, face and head-shoulder projection areas, and outputting a thermal infrared imaging pixel normalization matrix; S33: Processing the thermal infrared imaging pixel normalization matrix, and outputting the thermal imaging morphological feature information according to the image connected domain; The S31 includes: S311: Acquire the head height, where the head height includes the head of an infant and a head of an adult; S312: according to the head height, taking a preset ratio of the head height as a target height for extending the shoulders; S313: Obtain the starting position of the shoulder, extend the shoulder downward by the target height according to the target height, and output the head-shoulder projection area; The S4 includes: S41: taking the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information as an observation sequence; S42: inputting the observation sequence into a pre-trained hidden Markov model, classifying and identifying the observation sequence, and outputting the matching degree between the observation sequence and a plurality of infant behaviors; S43: Classify the infant's behavior according to the matching degree and output the classification result.

2. The method for identifying infant behavior by visible light-thermal infrared imaging according to claim 1, characterized in that: The S3 further includes: S34: sampling the pixel points in the head, face and head-shoulder projection areas according to the temperature of the pixel points to obtain a first average temperature of the sampling points; S35: Acquire the second average temperature of the previous sampling point in the same projection area, and output the temperature difference as the thermal imaging temperature change characteristic information based on the first average temperature and the second average temperature.

3. The method for identifying infant behavior by visible light-thermal infrared imaging according to claim 1, characterized in that: The S32 includes: S321: Acquire pixel points in the head, face and head-shoulder projection areas; S322: Compare the real-time temperatures of the pixels to obtain the maximum and minimum temperatures in each projection area; S323: Output the normalized matrix according to the real-time temperature, the maximum temperature and the minimum temperature.

4. A visible light-thermal infrared imaging infant behavior recognition device, characterized in that: The device comprises: An image acquisition module is used to acquire real-time thermal infrared imaging images and visible light images in infant care scenarios; A target detection module, used to perform target detection on the visible light image, and output detection frame position information and category information of preset targets related to infant care; A feature extraction module is used to obtain a preset thermal infrared imaging projection area, and when a detection frame of a preset target appears in the thermal infrared projection area, output the thermal imaging morphological feature information and thermal imaging temperature change feature information corresponding to the target projection area where the detection frame is located according to a preset rule; A behavior classification module, used to classify infant behaviors according to the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information, and output classification results; The preset targets include at least the following targets: infant head, infant head and shoulders, adult head, adult head and shoulders, infant face, and adult face; The preset thermal infrared imaging projection area is obtained, and when the detection frame of the preset target appears in the thermal infrared projection area, the thermal imaging morphological feature information and the thermal imaging temperature change feature information corresponding to the target projection area where the detection frame is located are output according to the preset rules, including: Obtaining a target projection area where the detection frame is located, and dividing the target projection area into a head, a face, and a head-shoulder projection area; Normalizing the pixels in the head, face and head-shoulder projection areas, and outputting a thermal infrared imaging pixel normalization matrix; Processing the thermal infrared imaging pixel normalization matrix, and outputting the thermal imaging morphological feature information according to the image connected domain; Get the target projection area where the detection frame is located, and divide the target projection area into head, face and head-shoulder projection areas including: Acquire the head height, wherein the head height includes the head of an infant and a head of an adult; According to the head height, a preset proportion of the head height is used as a target height for extending the shoulders; Obtaining the starting position of the shoulder, extending the shoulder downward by the target height according to the target height, and outputting the head-shoulder projection area; The infant behavior is classified according to the detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information, and the classification result is outputted, including: The detection frame position information and category information, thermal imaging morphological feature information and thermal imaging temperature change feature information are used as an observation sequence; Inputting the observation sequence into a pre-trained hidden Markov model, classifying and identifying the observation sequence, and outputting the matching degree between the observation sequence and a plurality of infant behaviors; According to the matching degree, the infant's behavior is classified and the classification result is output.

5. An electronic device, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 3.

6. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images