Multi-modal input baby intention recognition method and device

Abstract

The invention discloses a multi-modal input baby intention identification method and device. The method comprises the following steps: acquiring video data containing the whole body of a baby and corresponding baby crying data; inputting the video data containing the whole body of the baby and the corresponding baby crying data into a baby intention recognition model, and recognizing the intention of the baby; according to the invention, the video and audio information is collected synchronously, the facial expressions, limb actions and crying sound of the baby are used as input, and the deep neural network is used for end-to-end prediction of the intention of the baby, so that the accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a method and device for multi-modal input infant intention recognition. Background Art

[0002] Infants' language functions are immature and they cannot express their opinions independently, so others cannot understand the meaning of their movements, expressions, and cries. The present invention is based on a deep learning algorithm to automatically identify the intentions behind infants' behaviors, solving the problem that adults cannot understand infants' intentions.

[0003] The existing technology only uses images or audio data of babies crying as input data, and mainly judges the baby's intention based on the baby's facial expression or crying audio signal. This type of method, because it only uses image information or audio, inevitably omits important information sources such as sound, body movements or images, body movements, etc., which affects the accuracy of algorithm recognition. Moreover, because the image only contains one frame of information, it is impossible to analyze the continuity of the baby's movements. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for multi-modal input infant intention recognition in view of the deficiencies in the prior art.

[0005] The objective of the present invention is achieved through the following technical solution: A method for identifying infant intentions with multimodal input, comprising:

[0006] Obtaining video data containing the whole body of the baby and corresponding baby crying data;

[0007] Input the video data containing the whole body of the baby and the corresponding baby crying data into the baby intention recognition model to recognize the baby's intention;

[0008] The infant intention recognition model includes the following four modules:

[0009] The first module: Its input is video data containing the whole body of the baby. The face is detected using the face detection model, and the attention operator is superimposed on it to output the face feature vector;

[0010] The second module: Its input is the video data containing the whole body of the baby. The limb detection model is used to detect the limb movements, and the attention operator is superimposed on it to obtain the limb movement feature vector;

[0011] The third module: its input is the baby crying data, which is processed to obtain the frequency spectrum feature vector corresponding to the baby crying data;

[0012] The fourth module: Its input is the face feature vector, body movement feature vector and spectrum feature vector. The three vectors are concatenated and connected to the fully connected layer, and then activated by the softmax function. The result after the softmax function is the final prediction result.

[0013] Furthermore, the training of the face detection model includes:

[0014] The baby face dataset is labeled to mark the location of the face in each sample image, and the labeled dataset is used to train the face detection model. The loss function used during training is a multi-task function, which is: 1) calculating the MSE loss of the candidate area, 2) calculating the cross entropy loss of multiple classifications, and 3) calculating the cross entropy loss of binary classification. The weighted sum of the three losses is the final loss function.

[0015] Furthermore, the training of the limb detection model includes:

[0016] Using the baby full-body photo dataset, each baby picture is annotated with the key points of the human body, which are: [{0, "Nose"}, {1, "Neck"} {2, "RShoulder"}, {3, "RElbow"}, {4, "RWrist"}, {5, "LShoulder"}, {6, "LElbow"}, {7, "LWrist"}, {8, "RHip"}, {9, "RKnee"}, {10, "RAnkle"}, {11, "LHip"}, {12, "LKnee"}, {13, "LAnkle"}, {14, "REye"}, {15, "LEye"}, {16, "REar"}, {17, "LEar"}] to obtain the baby key point dataset; the baby key point dataset is used to train the limb detection model; the loss functions used in training are multi-task functions, which are: 1) Part Affinity Field for training joint connections Loss, 2) Heatmap Loss for training joint heat maps, and 3) Object Detection Loss for regressing human body frames; the weighted sum of the three losses is the final loss function.

[0017] Furthermore, the training of the infant intention recognition model includes:

[0018] The four-tuple consisting of [face detection data, limb key point data, baby crying audio data, label] is the training data, wherein the label is the identifier of the final baby intention; the training data is used to train the baby intention recognition model, wherein the face detection model and the limb detection model have been trained. When training the baby intention recognition model, the parameters of the face detection model and the limb detection model need to be frozen.

[0019] Furthermore, image data preprocessing in face detection data includes: using flipping, translation, cropping, and contrast adjustment data enhancement techniques to expand the diversity of training data;

[0020] Image data preprocessing in limb key point data includes: using flipping, translation, cropping, and contrast adjustment data enhancement techniques to expand the diversity of training data;

[0021] The preprocessing of label data includes: discretization first, then one-hot encoding;

[0022] The preprocessing of the baby crying audio data includes: dividing the baby crying audio data into several data frames to obtain the first matrix, which is named frames; performing FFT on each row of the frames matrix to obtain the second matrix, which is named pow_frames; finally, applying the Mel filter to pos_frames to obtain the third matrix, which is named mel-spectrogram, which is the final processed audio data.

[0023] Furthermore, the obtaining of the frequency spectrum feature vector corresponding to the baby crying data includes:

[0024] The waveform signal of the baby crying data is converted into a Mel-spectrogram, and then the attention operator is superimposed on it to obtain the spectrum feature vector.

[0025] Furthermore, it also includes building an online service for infant intention recognition, specifically:

[0026] Build a public network springboard server and host the domain name resolution service through cloudflare;

[0027] Deploy the baby intention recognition model inference service on the intranet server. The server is connected to a camera, which records the baby in real time. The server is connected to a microphone, which records the baby's voice in real time.

[0028] The intranet server predicts the baby's intention every n seconds. The prediction process is as follows: receive the latest n seconds of video captured by the camera and the corresponding n seconds of audio captured by the microphone as input, execute the forward propagation of the neural network, and finally give the prediction result through the network output layer, which represents the recognition result of the baby's intention by the baby's intention recognition model.

[0029] The present invention also provides a multi-modal input baby intention recognition device, comprising:

[0030] A data acquisition module, used to acquire video data containing the whole body of the baby and corresponding baby crying data;

[0031] The intention recognition module is used to input the video data containing the whole body of the baby and the corresponding baby crying data into the baby intention recognition model to recognize the baby's intention;

[0032] The infant intention recognition model includes the following four modules:

[0033] The first module: Its input is video data containing the whole body of the baby. The face is detected using the face detection model, and the attention operator is superimposed on it to output the face feature vector;

[0034] The second module: Its input is the video data containing the whole body of the baby. The limb detection model is used to detect the limb movements, and the attention operator is superimposed on it to obtain the limb movement feature vector;

[0035] The third module: its input is the baby crying data, which is processed to obtain the frequency spectrum feature vector corresponding to the baby crying data;

[0036] The fourth module: Its input is the face feature vector, body movement feature vector and spectrum feature vector. The three vectors are concatenated and connected to the fully connected layer, and then activated by the softmax function. The result after the softmax function is the final prediction result.

[0037] The present invention also provides a multi-modal input baby intention recognition device, comprising one or more processors, for implementing the above-mentioned multi-modal input baby intention recognition method.

[0038] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-mentioned method for identifying infant intentions with multimodal input.

[0039] Compared with the prior art, the beneficial effects of the embodiments of the present invention are: the present invention synchronously collects video and audio information, uses the baby's facial expressions, body movements, and crying sounds as input, and uses a deep neural network to perform end-to-end prediction of the baby's intentions, and the accuracy is improved (the improvement comes from two parts: 1. Compared with image input, video input can not only obtain more frames of image information, but also combine the coherence in the time dimension. 2. Combining information in video, baby crying audio and body movements). BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0041] Figure 1 A schematic diagram of an infant intention recognition model architecture provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a hardware structure provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0044] A method for identifying infant intentions using multimodal input according to the present invention comprises:

[0045] Obtaining video data containing the whole body of the baby and corresponding baby crying data;

[0046] Input the video data containing the whole body of the baby and the corresponding baby crying data into the baby intention recognition model to recognize the baby's intention;

[0047] The infant intention recognition model includes the following four modules:

[0048] The first module: Its input is video data containing the whole body of the baby. The face is detected using the face detection model, and the attention operator is superimposed on it to output the face feature vector;

[0049] The second module: Its input is the video data containing the whole body of the baby. The limb detection model is used to detect the limb movements, and the attention operator is superimposed on it to obtain the limb movement feature vector;

[0050] The third module: its input is the baby crying data, which is processed to obtain the frequency spectrum feature vector corresponding to the baby crying data;

[0051] The fourth module: Its input is the face feature vector, body movement feature vector and spectrum feature vector. The three vectors are concatenated and connected to the fully connected layer, and then activated by the softmax function. The result after the softmax function is the final prediction result.

[0052] In one embodiment, the method specifically includes the following steps:

[0053] Step 1: Build a dataset

[0054] 1.1 Build a face detection dataset.

[0055] We labeled our own baby face dataset, marked the position of the face in each sample image (x, y, w, h), and merged the labeled dataset with the open source face dataset "Glint360K" to form the final face count detection dataset. Among them, x represents the horizontal coordinate of the upper left corner of the rectangular box, y represents the vertical coordinate of the upper left corner of the rectangular box, w represents the width of the rectangular box, and h represents the height of the rectangular box.

[0056] 1.2 Build a limb key point dataset.

[0057] We use our own full-body baby photo dataset to label the baby’s body key points. For each baby picture, we annotate 18 body key points, namely [{0,“Nose”},{1,“Neck”}{2,“RShoulder”},{3,“RElbow”},{4,“RWrist”},{5,“LShoulder”},{6,“LElbow”},{7,“LWrist”},{8,“RHip”},{9,“RKnee”},{10,“RAnkle”},{11,“LHip”},{12,“LKnee”},{13,“LAnkle”},{14,“REye”},{15,“LEye”},{16,“REar”},{17,“LEar”}], forming the baby key point dataset. The above infant key point dataset is fused with the open source COCO dataset (with manually supplemented Neck key points) to form the final limb key point dataset. Among them, the first letter "R" in "RShoulder", "RElbow", "RWrist", "RHip", "RKnee", "RAnkle", "REye" and "REar" is the abbreviation of right; the first letter "L" in "LShoulder", "LElbow", "LWrist", "LHip", "LKnee", "LAnkle", "LEye" and "LEar" is the abbreviation of left.

[0058] 1.3 Building a baby crying audio dataset

[0059] The baby crying data is collected and cut into short audios of 3 seconds in length to form a baby crying dataset.

[0060] 1.4 Building a Labeled Dataset

[0061] Labels are identifiers of the baby's ultimate intentions, such as happy, hungry, peeing, etc.

[0062] 1.5 Processing Dataset

[0063] For the above data, the following data preprocessing work needs to be done:

[0064] 1.5.1 For the image data in 1.1, use data enhancement techniques such as flipping, translation, cropping, contrast adjustment, etc. to expand the diversity of training data and obtain data.

[0065] 1.5.2 For the image data in 1.2, use data enhancement techniques such as flipping, translation, cropping, and contrast adjustment to expand the diversity of training data.

[0066] 1.5.3 For the label data, first discretize it and then perform one-hot encoding.

[0067] 1.5.4 For the baby crying audio data in 1.3, each audio with a time length of 3s (sampling rate 22050Hz) is divided into 240 data frames of 25ms (overlap degree 50%), and a matrix of dimension 240x552 is obtained, which is named frames. Perform FFT on each row of the above frames matrix, and a matrix of dimension 240x512 is obtained, which is named pow_frames. Finally, the Mel filter is applied to the above pos_frames to obtain a matrix of dimension 240x40, which is named mel-spectrogram, which is the final processed audio data.

[0068] The data in 1.4.1, 1.4.2, 1.4.3, and 1.4.4 are concatenated to form a four-tuple of [face detection data, limb key point data, baby crying audio data, label], which is the training data. After completing the above process, the four-tuple data set is randomly sampled and shuffled, and can be used for model training.

[0069] Step 2: Build and train the model

[0070] 2.1 Training the face detection model

[0071] Build the YOLO algorithm model and use the data set in 1.1 for training. The loss function used in training is a multi-task function, which is: 1) calculate the MSE loss of the candidate area, 2) calculate the cross entropy loss of multiple classifications, and 3) calculate the cross entropy loss of two classifications. The weighted sum of the above three is the final loss function.

[0072] 2.2 Training the Limb Detection Model

[0073] Build the openPose algorithm model and use the data set in 1.2 for training. The loss function used in training is a multi-task function, which is: 1) Part Affinity Field Loss for training joint connections, 2) Heatmap Loss for training joint heatmaps, and 3) Object Detection Loss for regressing human body frames. The weighted sum of the above three is the final loss function.

[0074] 2.3 Assembly Model

[0075] The model is divided into four parts:

[0076] The first module receives continuous image input (3s 12fps video, 3x12=36 frames of images), detects faces using the face detection model trained in 2.1, and superimposes the attention operator on it to obtain the face feature vector.

[0077] The second module uses the limb detection model trained in 2.2 to detect limb movements on the image input in step 1 (3s 12fps video, 3x12=36 frames of images), and superimposes the attention operator on it to obtain the limb movement feature vector.

[0078] In the third module, for the audio input, the waveform is first converted into a Mel spectrum, and then the attention operator is superimposed on it to obtain the spectrum feature vector.

[0079] The fourth module concatenates the feature vectors of the above three modules and then connects them to the fully connected layer. The last layer is activated by the softmax function. The result after the softmax function is the final prediction result. Figure 1 shown.

[0080] 2.4 End-to-end training model

[0081] Using the dataset in 1.4, train the deep model assembled in 2.3 end-to-end, where the parameters of the first module and the second module need to be frozen during training. The trained model can be used to recognize infant intentions.

[0082] In one embodiment, step 3 is also included: building an online service for infant intention recognition, which specifically includes the following steps:

[0083] 3.1 Build a public network springboard server and host the domain name resolution service through cloudflare.

[0084] 3.2 Deploy the model inference service on the intranet server. The server is connected to a camera, which records the baby in real time (for example, when the baby is lying in the crib, the camera is aimed at the location of the crib); the server is connected to a microphone to record the baby's voice in real time.

[0085] 3.3 The intranet server predicts the baby's intention every 3 seconds. The prediction process is: receive the latest 3s of video captured by the camera and the corresponding 3s of audio captured by the microphone as input, execute the forward propagation of the neural network, and finally give the prediction result through the network output layer, which represents the algorithm model's recognition result of the baby's intention.

[0086] The present invention also provides a multi-modal input baby intention recognition device, comprising:

[0087] A data acquisition module, used to acquire video data containing the whole body of the baby and corresponding baby crying data;

[0088] The intention recognition module is used to input the video data containing the whole body of the baby and the corresponding baby crying data into the baby intention recognition model to recognize the baby's intention;

[0089] The infant intention recognition model includes the following four modules:

[0090] The first module: Its input is video data containing the whole body of the baby. The face is detected using the face detection model, and the attention operator is superimposed on it to output the face feature vector;

[0091] The second module: Its input is the video data containing the whole body of the baby. The limb detection model is used to detect the limb movements, and the attention operator is superimposed on it to obtain the limb movement feature vector;

[0092] The third module: its input is the baby crying data, which is processed to obtain the frequency spectrum feature vector corresponding to the baby crying data;

[0093] The fourth module: Its input is the face feature vector, body movement feature vector and spectrum feature vector. The three vectors are concatenated and connected to the fully connected layer, and then activated by the softmax function. The result after the softmax function is the final prediction result.

[0094] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0095] Corresponding to the aforementioned embodiment of a method for recognizing an infant's intention with multi-modal input, the present invention further provides an embodiment of a device for recognizing an infant's intention with multi-modal input.

[0096] See also Figure 2 An embodiment of the present invention provides a multi-modal input baby intention recognition device, which includes one or more processors for implementing a multi-modal input baby intention recognition method in the above embodiment.

[0097] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0098] An embodiment of a multimodal input infant intention recognition device of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory and running them. From a hardware perspective, if Figure 2 As shown, it is a hardware structure diagram of a multi-modal input baby intention recognition device of the present invention, in which any device with data processing capability is located, except Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0099] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0100] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0101] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for identifying infant intentions with multimodal input in the above embodiment is implemented.

[0102] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0103] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for identifying infant intentions using multimodal input, characterized in that: include: Obtaining video data containing the baby's entire body and corresponding baby crying data; Input the video data of the baby's whole body and the corresponding baby crying data into the baby intention recognition model to identify the baby's intention; The infant intention recognition model includes the following four modules: The first module takes as input a video of the baby's entire body, detects the face using a face detection model, and applies an attention operator to it to output a facial feature vector. The second module takes as input video data containing the baby's entire body, uses a limb detection model to detect limb movements, and superimposes an attention operator on it to obtain a limb movement feature vector; The third module: Its input is the baby crying data, which is processed to obtain the spectral feature vector corresponding to the baby crying data; The fourth module: Its input is the facial feature vector, body movement feature vector and spectrum feature vector. The three vectors are concatenated and connected to the fully connected layer, and then activated by the softmax function. The result after the softmax function is the final prediction result.

2. The method for identifying infant intentions using multimodal input according to claim 1, wherein: The training of the face detection model includes: The baby face dataset was labeled to mark the location of the face in each sample image. The labeled dataset was then used to train the face detection model. The loss function used during training was a multi-task function, which consisted of: 1) calculating the MSE loss of the candidate region, 2) calculating the cross-entropy loss of multiple classifications, and 3) calculating the cross-entropy loss of the binary classification. The weighted sum of these three losses was the final loss function.

3. The method for identifying infant intentions using multimodal input according to claim 1, wherein: Training the limb detection model includes: Using the baby full-body photo dataset, each baby picture is annotated with the human body key points, which are: [{0,"Nose"},{1,"Neck"}{2,"RShoulder"},{3,"RElbow"},{4,"RWrist"},{5,"LShoulder"},{6,"LElbow"},{7,"LWrist"},{8,"RHip"},{9,"RKnee"},{10,"RAnkle"},{11,"LHip"},{12,"LKnee"},{13,"LAnkle"},{14,"REye"},{15,"LEye"},{16,"REar"},{17,"LEar"}] to obtain the baby key point dataset; the limb detection model is trained using the baby key point dataset; the loss function used in training is a multi-task function, which is: 1) Part Affinity Field Loss for training joint connections, 2) Heatmap for training joint heatmaps Loss, 3) Object Detection Loss for regressing the human body frame; the weighted sum of the three losses is the final loss function.

4. The method for identifying infant intentions using multimodal input according to claim 1, wherein: The training of the infant intention recognition model includes: The four-tuple consisting of [face detection data, limb key point data, baby crying audio data, label] is the training data, where the label is the identifier of the final baby intention; the baby intention recognition model is trained using this training data, where the face detection model and the limb detection model have both been trained. When training the baby intention recognition model, the parameters of the face detection model and the limb detection model need to be frozen.

5. The method for identifying infant intentions using multimodal input according to claim 4, wherein: Image data preprocessing in face detection data includes: using flipping, translation, cropping, contrast adjustment data enhancement techniques to expand the diversity of training data; Image data preprocessing in limb keypoint data includes: using flipping, translation, cropping, and contrast adjustment data enhancement techniques to expand the diversity of training data; The preprocessing of label data includes: discretization and then one-hot encoding; The preprocessing of baby crying audio data includes: dividing the baby crying audio data into several data frames to obtain the first matrix, which is named frames; performing FFT on each row of the frames matrix to obtain the second matrix, which is named pow_frames; finally, applying the Mel filter to pos_frames to obtain the third matrix, which is named mel-spectrogram, which is the final processed audio data.

6. The method for identifying infant intentions using multimodal input according to claim 1, wherein: The obtaining of the frequency spectrum feature vector corresponding to the baby crying data includes: The waveform signal of the baby crying data is converted into a mel-spectrogram, and then the attention operator is superimposed on it to obtain the spectrum feature vector.

7. The method for identifying infant intentions using multimodal input according to claim 1, wherein: It also includes building an online service for identifying infant intentions, specifically: Build a public network springboard server and host the domain name resolution service through Cloudflare; Deploy the infant intent recognition model inference service on the intranet server, connect the server to the camera, and the camera records the baby in real time; The server is connected to a microphone to record the baby's voice in real time; The intranet server predicts the baby's intention every n seconds. The prediction process is as follows: receiving the latest n seconds of video captured by the camera and the corresponding n seconds of audio captured by the microphone as input, performing forward propagation of the neural network, and finally giving a prediction result through the network output layer, which represents the recognition result of the baby's intention recognition model.

8. A multimodal input infant intention recognition device, characterized in that: include: A data acquisition module is used to acquire video data containing the entire body of the baby and corresponding baby crying data; The intention recognition module is used to input the video data containing the whole body of the baby and the corresponding baby crying data into the baby intention recognition model to identify the baby's intention; The infant intention recognition model includes the following four modules: The first module takes as input a video of the baby's entire body, detects the face using a face detection model, and applies an attention operator to it to output a facial feature vector. The second module takes as input video data containing the baby's entire body, uses a limb detection model to detect limb movements, and superimposes an attention operator on it to obtain a limb movement feature vector; The third module: Its input is the baby crying data, which is processed to obtain the spectral feature vector corresponding to the baby crying data; The fourth module: Its input is the facial feature vector, body movement feature vector and spectrum feature vector. The three vectors are concatenated and connected to the fully connected layer, and then activated by the softmax function. The result after the softmax function is the final prediction result.

9. A multimodal input infant intention recognition device, characterized in that: The invention comprises one or more processors for implementing a method for recognizing infant intentions based on multimodal input according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement a multimodal input infant intention recognition method according to any one of claims 1 to 7.

Images