Baby monitoring demand identification method and device based on sound and image, equipment and medium

Abstract

The invention relates to an infant monitoring demand identification method and device based on sound and images, equipment and a medium. The method comprises the following steps: when a camera detects an infant and a sound signal acquired by a microphone exceeds a threshold value, entering a continuous detection stage; in the continuous detection stage, the multiple cameras and the microphones collect video data and audio data within a preset time length at intervals; extracting image features in the video data by using a lightweight convolutional network, and extracting sound features in the audio data; fusing the image features and the sound features acquired in each path, and performing classification according to a pre-trained classifier to obtain state information of the baby; and monitoring the baby through the continuously output state information of the baby. According to the method, the pressure of the calculated amount can be reduced, and the monitoring accuracy can be ensured.

Description

Technical Field

[0001] The present application relates to the technical field of infant monitoring, and in particular to a method, device, equipment and medium for identifying infant monitoring needs based on sound and image. Background Art

[0002] Infant monitoring is one of the most important tasks in infant care. At present, infant monitoring is mainly carried out manually by personnel. During the monitoring, it is necessary not only to pay attention to the physical condition and sleep status of the infant at all times, but also for inexperienced caregivers (such as new parents), for example, when the infant cries, it is difficult to quickly judge the infant's needs at this time, which makes new parents anxious and competitive, greatly increasing the risk of postpartum depression in mothers. At the same time, since it is impossible to watch the infant in real time 24 hours a day, and the infant cannot convey his current feelings and thoughts to the caregiver through effective means, it is very easy to get into danger. For example, when a baby swallows accidentally, it is difficult for the caregiver to find it in the first time because he cannot make a sound due to choking.

[0003] In addition, Dr. Dunstan, a child psychology researcher, studied a large amount of information on infant cries and found that infant cries from 0 to 3 months old correspond to physiological needs. However, it is difficult for new parents to understand in the early stages and they cannot scientifically judge the meaning of infant cries and how to comfort them, which brings physical and mental pain to infants and young children, increases psychological trauma in early years, and also increases family friction in parenting and increases the incidence of postpartum depression.

[0004] In addition, in maternity and infant hospitals, nurses often face a one-to-many task when caring for infants. When the nurse is not in the nursing ward, the only way to monitor the infants is to use the nursing camera in the nursing ward. However, when multiple infants cry or scream at the same time, the nurse cannot immediately determine which infant is the one in question.

[0005] From this we can see that infant monitoring requires multi-faceted judgment, that is, it requires fixed-point analysis based on the infant's own facial expressions, movements, and voices.

[0006] There are two drawbacks in the existing algorithm technology. The first is that it cannot effectively filter out background noise, such as interference from TV and adult talking in the baby care room, which can easily cause false detection. The current mainstream network design only considers the structural design based on the speech of ordinary people, but lacks the optimization design for extracting the pronunciation features of infants. The second is the video actions and sounds of infants crying and shouting. Usually, the traditional neural network design is more complicated. The high amount of calculation forces the algorithm model to be deployed on the server side. Since there are not many types of infant expressions, it causes a lot of waste of computing resources and bandwidth pressure for transmitting massive data to the background server.

[0007] From a cost perspective, existing behavioral action recognition algorithms often require the use of complex neural networks and large-scale server-level computing equipment to perform calculations and analysis. Not only is the hardware cost high, but it also requires extremely high video image transmission bandwidth requirements and high energy consumption. Summary of the invention

[0008] Based on this, it is necessary to provide a method, device, equipment and medium for identifying infant monitoring needs based on sound and image to address the above technical problems.

[0009] A method for identifying infant monitoring needs based on sound and image, applied to an edge end, wherein the edge end includes at least one camera, multiple microphones, and an AI computing power card, and the method includes:

[0010] When the camera detects a baby and the sound signal collected by the microphone exceeds the threshold, it enters the continuous detection phase;

[0011] In the continuous detection stage, multiple cameras and microphones collect video data and audio data of a preset time length at intervals; the video data and audio data have the same time length;

[0012] Using a lightweight convolutional network to extract image features from the video data and extract sound features from the audio data;

[0013] The image features and sound features collected from each channel are fused and classified according to the pre-trained classifier to obtain the baby's status information;

[0014] The baby is monitored by continuously outputting the baby's status information.

[0015] In one of the embodiments, the method further includes: obtaining multiple audio data segments from sparse to dense in the time domain, performing short-time Fourier transform on the multiple audio data segments to convert them into frequency domain signals;

[0016] Filter the sparsely sampled data through a high-pass filter with predetermined hyperparameters;

[0017] The frequency domain signal is reduced in dimension through the encoder and pyramid U-Net network, and decoded to the time domain to obtain the sound characteristics of noise-free baby sounds.

[0018] In one of the embodiments, it also includes: extracting multiple frames of image data from the video data, inputting the multiple frames of image data into a lightweight convolutional network to obtain image features.

[0019] In one of the embodiments, the method further includes: normalizing the image features and the sound features to obtain fusion features.

[0020] In one of the embodiments, the loss function of the pre-trained classifier includes: classification loss, waveform loss of the sound feature and denoising loss of the sound signal.

[0021] In one of the embodiments, it also includes: in the continuous detection stage, multiple cameras and microphones use a pipeline method to collect video data and audio data of a preset time length at intervals, and continuously output the video data and audio data.

[0022] In one of the embodiments, the method further includes: when the continuously outputted status information of the infant is in a critical state, outputting warning information, so as to monitor the infant.

[0023] A device for identifying infant monitoring needs based on sound and image, applied to an edge end, wherein the edge end includes at least one camera, multiple microphones, and an AI computing power card, and the device includes:

[0024] A detection module, used to enter a continuous detection phase when the camera detects a baby and the sound signal collected by the microphone exceeds a threshold;

[0025] A collection module, used for collecting video data and audio data of a preset time length at intervals by multiple cameras and microphones during the continuous detection stage; the video data and audio data have the same time length;

[0026] A feature extraction module, used to extract image features from the video data and sound features from the audio data using a lightweight convolutional network;

[0027] A fusion module is used to fuse the image features and sound features collected from each channel, classify them according to the pre-trained classifier, and obtain the status information of the baby;

[0028] The monitoring module is used to monitor the baby by continuously outputting the baby's status information.

[0029] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0030] When the camera detects a baby and the sound signal collected by the microphone exceeds the threshold, it enters the continuous detection phase;

[0031] In the continuous detection stage, multiple cameras and microphones collect video data and audio data of a preset time length at intervals; the video data and audio data have the same time length;

[0032] Using a lightweight convolutional network to extract image features from the video data and extract sound features from the audio data;

[0033] The image features and sound features collected from each channel are fused and classified according to the pre-trained classifier to obtain the baby's status information;

[0034] The baby is monitored by continuously outputting the baby's status information.

[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0036] When the camera detects a baby and the sound signal collected by the microphone exceeds the threshold, it enters the continuous detection phase;

[0037] In the continuous detection stage, multiple cameras and microphones collect video data and audio data of a preset time length at intervals; the video data and audio data have the same time length;

[0038] Using a lightweight convolutional network to extract image features from the video data and extract sound features from the audio data;

[0039] The image features and sound features collected from each channel are fused and classified according to the pre-trained classifier to obtain the baby's status information;

[0040] The baby is monitored by continuously outputting the baby's status information.

[0041] The above-mentioned sound and image-based infant monitoring demand identification method, device, equipment and medium, in order to solve the current problem of large amount of infant monitoring data and complex model resulting in poor real-time monitoring, the present invention deploys calculation and analysis on the edge through edge deployment. In order to solve the problem of small data volume, on the one hand, a lightweight neural network model is adopted to reduce the calculation overhead. On the other hand, in order to improve the accuracy of monitoring and early warning, multiple cameras and microphones are deployed on the edge, and video data and audio data of a preset time length are collected at intervals to ensure that a status information can be output at the interval. Through continuous status information, the accuracy of monitoring is guaranteed while reducing the calculation overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a method for identifying infant monitoring needs based on sound and image in one embodiment;

[0043] Figure 2 A schematic diagram of a DAVF-Net network in one embodiment;

[0044] Figure 3 is a schematic diagram of a Denoise block module in one embodiment;

[0045] Figure 4 is a schematic diagram of a downsampling and upsampling structure in one embodiment;

[0046] Figure 5 is a schematic diagram of a self-attention structure in one embodiment;

[0047] Figure 6 is a schematic diagram of a lightweight convolutional network in one embodiment;

[0048] Figure 7 is a schematic diagram of a Connect block module in one embodiment;

[0049] Figure 8 A schematic diagram of a classification part in one embodiment;

[0050] Fig. 9 The figure is a structural block diagram of a device for identifying infant monitoring needs based on sound and image in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] In one embodiment, Figure 1 As shown, a method for identifying infant monitoring needs based on sound and image is provided, comprising the following steps:

[0053] Step 102, when the camera detects a baby and the sound signal collected by the microphone exceeds a threshold, the continuous detection phase is entered.

[0054] In this step, the edge end is deployed at a position for monitoring a baby, for example, monitoring a baby on a bed. The camera and the microphone can be aimed at the position of the baby's bed. When the camera does not detect the baby, it indicates that no monitoring is required, that is, no calculation is required. When the camera detects the baby and the sound signal collected by the microphone exceeds the threshold, it indicates that the baby has a need. Whether the need is positive or critical needs to be judged by the method of the present invention.

[0055] The continuous detection stage refers to the detection during a period of time, through non-alarm, early warning, alarm and other actions to detect and give feedback, so as to achieve infant monitoring.

[0056] It is worth noting that the status information of the baby in the present invention can generally be:

[0057] A. Sleep state

[0058] B. Daze state

[0059] C. Happy state

[0060] D. Excited state

[0061] E. Severe discomfort

[0062] F. Suspected dangerous condition (suffocation)

[0063] G. Crying state Physiological needs (hunger, sleepiness, need to burp, flatulence / colic, feeling unwell)

[0064] The status information can be used as a classification label. For monitoring, sleeping, dazed, happy, and excited states are in a positive state and do not require an alarm, while severe discomfort, suspected dangerous, and crying states are physiological needs that require an alarm.

[0065] Step 104, in the continuous detection stage, multiple cameras and microphones collect video data and audio data of a preset time length at intervals.

[0066] The interval time can be 1s. Taking three cameras and microphones as an example, at 0s, camera 1 and microphone 1 collect video data and audio data, and the preset time length of collection can be 3s, that is, 3s video data and 3s audio data are collected with the same time length, and their time is also completely aligned. At 1s, camera 2 and microphone 2 collect video data and audio data, at 2s, camera 3 and microphone 3 collect video data and audio data, and at 3s, camera 1 and microphone 1 collect video data and audio data.

[0067] Step 106, using a lightweight convolutional network to extract image features from the video data and extract sound features from the audio data.

[0068] Step 108, the image features and sound features collected from each channel are fused, and classified according to the pre-trained classifier to obtain the status information of the baby.

[0069] Step 110, monitoring the baby by continuously outputting the baby's status information.

[0070] In this step, taking the above-mentioned continuous collection as an example, it can be known that the baby's status information can be obtained within 3 seconds. If the status information is critical, a warning can be issued. If the status information is still critical in the 4th second, an alarm will be issued. If the status information is still positive after the 4th second, the warning will be canceled, thereby ensuring the timeliness and accuracy of monitoring.

[0071] In the above-mentioned sound and image-based infant monitoring demand identification method, in order to solve the problem that the current infant monitoring data volume is large and the model is complex, resulting in poor real-time monitoring, the present invention deploys calculation and analysis on the edge through edge deployment. In order to solve the problem of small data volume, on the one hand, a lightweight neural network model is adopted to reduce the calculation overhead. On the other hand, in order to improve the accuracy of monitoring and early warning, multiple cameras and microphones are deployed on the edge. The method of collecting video data and audio data of a preset time length at intervals is adopted to ensure that a status information can be output at the interval time. Through continuous status information, the accuracy of monitoring is guaranteed, and the calculation overhead is reduced.

[0072] In one embodiment, multiple audio data segments from sparse to dense in the time domain are obtained, and short-time Fourier transform is performed on the multiple audio data segments to convert them into frequency domain signals; the sparsely sampled data is filtered through a high-pass filter with predetermined hyperparameters; the frequency domain signal is reduced in dimension through an encoder and a pyramid U-Net network, and decoded to the time domain to obtain the sound characteristics of noise-free infant sounds.

[0073] Specifically, in order to realize the recognition method of infant monitoring needs based on sound and image, a DAVF-Net (Denoising audio and video fusion classification network) network is proposed. Figure 2 As shown in the figure, DAVF-Net first inputs two data sources, audio and video. Denoiseblock is a denoising module for purifying high-frequency baby crying sounds, and 3DCNN block is a module for normalizing multi-frame video features. The denoised audio data and video data are merged through connect, and finally output after passing through a lightweight backbone and classification layer.

[0074] First, in audio denoising, the Denoise block is composed of a network structure that uses self-attention to perform global receptive field feature fusion, such as Figure 3As shown in the figure, since the sound of a baby's cry is mainly concentrated in the high frequency band, and the baby's cry has a more continuous feature, we first select a certain length (3s-6s is selected to control the amount of calculation) of the baby's sound audio collected by the microphone. In the case of external noise interference, we first perform three-segment sampling from dense to sparse in the time domain, as shown in the interval extraction part in the figure. The three-segment sampling can reduce the impact of short noise interference on network fitting. In the network structure, the interval extraction layer is the audio signal sampling and acquisition layer to obtain audio data with more sparse time domain. The audio signal is processed in the time domain as STFT, which is a short-time Fourier transform module, which is used to convert the time domain signal into the frequency domain signal. The down sampling layer is an encoder structure, and the corresponding Upsampling layer is a decoder structure. The overall network structure is similar to U-Net, which is used to extract the features of audio data. The data is converted into the frequency domain through the short-time Fourier transform, and the downsampling-self-attention-upsampling mechanism is used to achieve feature learning, and the audio data of different scales are established through a dense connection mode. Residual channels. The two high pass filtering matrices are high pass filters based on a priori theory. Since the frequency of infant pronunciation is higher than most external noise and adult voices, we set two hyperparameter matrices to reduce the interference of external noise, which helps the network converge quickly. Matrix2 uses a stricter high pass filtering threshold than matrix1.

[0075] Figure 4 The structure of downsampling and upsampling forms the structure of encoder and decoder. The encoder consists of 1x1 convolution (Conv1×1) and gated linear unit (GLU). The 1×1 convolution is responsible for downsampling and increasing the number of channels, and the gated linear unit halves the number of channels. The decoder consists of a gated linear unit and a transposed 1×1 convolution to complete upsampling. After all 1×1 convolution layers, the SELU activation function is selected instead of RELU to help the network converge quickly.

[0076] Figure 5It is a self-attention structure. In the self-attention part, based on the traditional structure, RMSnorm is used to optimize the regularization function to make the network calculation more efficient. Mask multi-head self-attention module is the same as the general multi-head attention mechanism. Multi-Head Attention is actually multiple parallel Self-Attention layers, and each "head" learns different attention weights independently. The output of the head is finally merged. In this way, it can focus on information from different subspaces of the input data. In the task of infant sound purification, it is manifested as focusing on the information correlation of different frequency bands in the sound spectrum data. Mask is a mask, which reduces the fusion of irrelevant information on the basis of self-attention and accelerates network convergence.

[0077] This network replaces the regularization part of norm with RMSnorm based on the traditional add and norm. RMSNorm removes the mean term in LayerNorm (since the mean is not calculated, the variance calculation also does not have the operation of subtracting the mean). It has higher computational efficiency and is convenient for lightweight deployment and network training. The Feed Forward layer is a feedforward mechanism layer. It is the output of Multi-Head Attention with residual connection and Norm to obtain data. Then FeedForward performs two linear transformations. Through linear transformation, the data is first mapped to a high-dimensional space and then to a low-dimensional space, in order to extract features more deeply.

[0078] In terms of data flow, the audio data is first sparsely sampled once or twice in the time domain. The data that has not been sparsely sampled, the data that has been sparsely sampled once, and the data that has been sparsely sampled twice are uniformly converted to the frequency domain by short-time Fourier transform, and the sparsely sampled data is filtered by a high-pass filter with predetermined hyperparameters.

[0079] After that, several sets of data enter the denoising frequency network part of DAVF-Net, and after the encoder and pyramid connection, the data features are reduced in dimension, and then sent to the self-attention module for global receptive field feature extraction. Next, the noise-free part of the baby's voice is restored to the frequency domain signal through the decoder and pyramid connection for subsequent multimodal fusion.

[0080] In one of the embodiments, multiple frames of image data are extracted from the video data, and the multiple frames of image data are input into a lightweight convolutional network to obtain image features.

[0081] Specifically, infants’ expressions are much simpler than those of adults, so a compromise is made when integrating infants’ expression data. While collecting infants’ voices, infants’ expressions are sampled frame by frame, with a principle of one frame per second. This not only avoids the risk of misjudging infants’ expressions in a single frame, but also effectively controls the amount of calculation. Figure 6 This is a lightweight 3D baby expression CNN network. By extracting the features of baby expressions in 3 frames (or more frames) in 3 seconds, effective information about baby expressions can be obtained. The basic structure of the network uses 3DCNN convolution, convolves three frames (or more frames) with an interval of 1 second, and fuses the convolution features of multiple images. Res connect is a residual structure that can prevent overfitting. After every two 3x3 convolutions of the image matrix data, a convolution downsampling with a step of 2 is performed to reduce the size and increase the number of convolution kernels. At the same time, a depth-wise structure is used to increase the receptive field and reduce the amount of calculation.

[0082] In one of the embodiments, the image features and the sound features are normalized to obtain fused features.

[0083] Specifically, the main purpose of the Connect block is to normalize the audio features and visual features, unify the number of channels, and reshape the audio vector into a matrix of the same width as the visual features, such as Figure 7 As shown in the figure, the network concatenates the image feature matrix with the reshaped dimension and size of the audio information after 1x1 convolution to form a new feature. The width w and the number of channels c of the time domain sound feature are matched with the image feature, so that the connection can be made.

[0084] In one of the embodiments, the loss function of the pre-trained classifier includes: classification loss, waveform loss of the sound feature, and denoising loss of the sound signal.

[0085] Specifically, in the final classification part, a convolution is performed to reduce the number of channels, and all features are transformed into one-dimensional ones through a fully connected layer to achieve classification output, such as Figure 8 shown.

[0086] Since the network integrates audio and visual features, we designed Contrastive Loss here. The loss here has three items, including classification loss l classification , l on the waveform l1 The loss is the STFT loss l between the clean speech x and the denoised speech STFT .

[0087] l loss = l classification +l STFT +ll1

[0088] Loss consists of three parts, which uniformly constrain the sound denoising, sound waveform fitting, and the final classification result. Different losses are turned on and off in different pre-training stages. For example, when training the audio denoising part, the classification loss is turned off.

[0089] In terms of data annotation, the training part of the data is divided into video data and audio data. The video data and audio data are matched in the time dimension. The video data extracts three frames per second, and with the final classification category label, it is used as training data, which contains balanced positive and negative sample data for each category. The audio data is extracted every three seconds to form a set of audio data. The audio data contains data of baby sounds, audio data without baby sounds, and audio data containing audio sounds and noise. The original data is used as the training input sample, and the pure baby sound domain data obtained after short-time Fourier transform (STFT) is used as the ground truth of the training output of the intermediate layer of the audio denoising network. Among them, the baby sound data containing noise should account for the largest proportion.

[0090] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0091] In one embodiment, Fig. 9 As shown, a device for identifying infant monitoring needs based on sound and image is provided, which is applied to an edge end, and the edge end includes at least multiple cameras, multiple microphones, and an AI computing power card, including: a detection module 902, a collection module 904, a feature extraction module 906, a fusion module 908, and a monitoring module 910, wherein:

[0092] The detection module 902 is used to enter the continuous detection phase when the camera detects a baby and the sound signal collected by the microphone exceeds a threshold;

[0093] The acquisition module 904 is used for acquiring video data and audio data of a preset time length at intervals by multiple cameras and microphones during the continuous detection stage; the video data and audio data have the same time length;

[0094] A feature extraction module 906, configured to extract image features from the video data and sound features from the audio data using a lightweight convolutional network;

[0095] A fusion module 908 is used to fuse the image features and sound features collected from each channel, classify them according to the pre-trained classifier, and obtain the status information of the baby;

[0096] The monitoring module 910 is used to monitor the baby by continuously outputting the baby's status information.

[0097] In one embodiment, the feature extraction module 906 is also used to filter the sparsely sampled data through a high-pass filter with predetermined hyperparameters; reduce the dimension of the frequency domain signal through an encoder and a pyramid U-Net network, and decode it to the time domain to obtain the sound characteristics of noise-free infant sounds.

[0098] In one embodiment, the feature extraction module 906 is further used to extract multiple frames of image data from the video data, and input the multiple frames of image data into a lightweight convolutional network to obtain image features.

[0099] In one embodiment, the feature extraction module 906 is further used to normalize the image features and the sound features to obtain fused features.

[0100] In one of the embodiments, the loss function of the pre-trained classifier includes: classification loss, waveform loss of the sound feature, and denoising loss of the sound signal.

[0101] In one of the embodiments, the detection module 902 is also used for, during the continuous detection phase, multiple cameras and microphones use a pipeline method to collect video data and audio data of a preset time length at intervals, and continuously output the video data and audio data.

[0102] In one embodiment, the monitoring module 910 is also used to output warning information when the continuously output status information of the infant is in a critical state, so as to monitor the infant.

[0103] For the specific definition of the sound and image-based infant monitoring needs identification device, please refer to the definition of the sound and image-based infant monitoring needs identification method above, which will not be repeated here. Each module in the above-mentioned sound and image-based infant monitoring needs identification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0104] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0107] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for identifying infant monitoring needs based on sound and image, It is characterized in that Applied to an edge end, the edge end includes at least multiple cameras, multiple microphones, and an AI computing power card, and the method includes: When the camera detects a baby and the sound signal collected by the microphone exceeds the threshold, it enters the continuous detection phase; In the continuous detection stage, multiple cameras and microphones collect video data and audio data of a preset time length at intervals; the video data and audio data have the same time length; Using a lightweight convolutional network to extract image features from the video data and extract sound features from the audio data; The image features and sound features collected from each channel are fused and classified according to the pre-trained classifier to obtain the baby's status information; The baby is monitored by continuously outputting the baby's status information.

2. The method according to claim 1, It is characterized in that Extracting sound features from the audio data includes: Acquire multiple audio data segments from sparse to dense in the time domain, perform short-time Fourier transform on the multiple audio data segments, and convert them into frequency domain signals; Filter the sparsely sampled data through a high-pass filter with predetermined hyperparameters; The frequency domain signal is reduced in dimension through the encoder and pyramid U-Net network, and decoded to the time domain to obtain the sound characteristics of noise-free baby sounds.

3. The method according to claim 2, It is characterized in that A lightweight convolutional network is used to extract image features in the video data, including: Extract multiple frames of image data from the video data, input the multiple frames of image data into a lightweight convolutional network to obtain image features.

4. The method according to claim 3, It is characterized in that The image features and sound features collected from each channel are integrated, including: The image features and sound features are normalized to obtain fusion features.

5. The method according to any one of claims 1 to 4, It is characterized in that The loss function of the pre-trained classifier includes: classification loss, waveform loss of the sound feature, and denoising loss of the sound signal.

6. The method according to claim 5, It is characterized in that In the continuous detection stage, multiple cameras and microphones collect video data and audio data of a preset time length at intervals, including: In the continuous detection stage, multiple cameras and microphones collect video data and audio data of a preset time length at intervals in a pipeline manner, and continuously output the video data and audio data.

7. The method according to claim 6, It is characterized in that Monitor the baby through the continuous output of the baby's status information, including: When the continuously outputted status information of the baby is in a critical state, an early warning message is outputted so as to monitor the baby.

8. A device for identifying infant monitoring needs based on sound and image, It is characterized in that Applied to the edge end, the edge end includes at least multiple cameras, multiple microphones and AI computing power cards, and the device includes: A detection module, used to enter a continuous detection phase when the camera detects a baby and the sound signal collected by the microphone exceeds a threshold; A collection module, used for collecting video data and audio data of a preset time length at intervals by multiple cameras and microphones during the continuous detection stage; the video data and audio data have the same time length; A feature extraction module, used to extract image features from the video data and sound features from the audio data using a lightweight convolutional network; A fusion module is used to fuse the image features and sound features collected from each channel, classify them according to the pre-trained classifier, and obtain the status information of the baby; The monitoring module is used to monitor the baby by continuously outputting the baby's status information.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images