Autistic child intervention treatment auxiliary system and method based on intelligent interaction

Abstract

The invention provides an autistic child intervention treatment auxiliary system and method based on intelligent interaction, relates to the field of data intelligent analysis, and adopts an AI-based data processing technology to extract audio features from audio data of an autistic child object and extract facial features from a facial expression image of the autistic child object, so that the autistic child intervention treatment auxiliary system is obtained. And intelligently judging whether the behavior state of the autistic child object is trying to communicate or not based on the multi-modal fine-grained joint representation between the audio feature coding feature and the face image feature coding feature. Thus, through continuous data acquisition, extremely fine changes of autistic children can be captured more accurately, and it is ensured that no instant behavior is missed. Moreover, by combining facial expression analysis and audio feature extraction, the judgment on the behavior state of the autistic child can be more accurate, and then more targeted personalized intervention feedback is generated.

Description

Technical Field

[0001] The present application relates to the field of intelligent data analysis, and more specifically, to an intelligent interaction-based intervention and treatment assistance system and method for children with autism. Background Art

[0002] Autism, also known as autism spectrum disorder, is a neurodevelopmental disorder that usually appears in early childhood and has a wide-ranging impact on an individual's social interaction, communication skills, interests, and behavior patterns. Therefore, timely intervention is essential to support the development of children with autism, and can significantly promote their ability to improve social interaction, language expression, and behavior management.

[0003] Traditional intervention methods for children with autism mainly rely on manual evaluation by human observers and the experience guidance of therapists. However, due to manpower limitations, it is difficult for professionals to be present at all times for real-time observation, which makes some behaviors that occur in an instant but are extremely important easily overlooked. In particular, some subtle expressions or brief vocal expressions of children with autism, which often only appear in specific situations, are crucial to judging their behavioral state. In addition, traditional methods are highly dependent on the intuition and personal experience of the observer, and the experience level of the therapist directly affects the understanding of behavioral changes and the quality of response. Even in the case of face-to-face observation, human observation may miss key details due to distraction or other reasons, such as slight eye movements, subtle contractions of facial muscles, etc., which in turn affects the accuracy of the assessment and the comprehensiveness of the intervention treatment.

[0004] Therefore, an intervention therapy auxiliary program for children with autism based on intelligent interaction is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The present application provides an intervention therapy auxiliary system and method for children with autism based on intelligent interaction.

[0006] According to one aspect of the present application, there is provided an autism children intervention treatment auxiliary system based on intelligent interaction, which comprises: A child object facial audio acquisition module, used to acquire the facial expression image of the autistic child object acquired by the camera and the audio data of the autistic child object acquired by the microphone; An audio feature extraction module, used for extracting audio features from the audio data to obtain an audio feature encoding vector; A facial feature extraction module, used for extracting facial features from the facial expression image to obtain a facial image feature encoding feature map; An audio-image multimodal joint module is used to perform an audio-image multimodal data fine-grained joint of the audio feature coding vector and the facial image feature coding feature map to obtain a multimodal joint coding vector of the object state of an autistic child, wherein the audio-image multimodal joint module includes: a facial image decomposition unit, used to perform feature decoupling and feature flattening on the facial image feature coding feature map to obtain a set of facial image local feature coding vectors; a clustering fusion unit, used to perform kernel feature calculation and fine-grained clustering fusion on the set of the audio feature coding vector and the facial image local feature coding vector to obtain the multimodal joint coding vector of the object state of an autistic child; The behavior state judgment module is used to determine the behavior state of the autistic child object based on the multimodal joint coding vector of the autistic child object state, and generate intervention feedback based on the behavior state.

[0007] Furthermore, the audio feature extraction module includes: An audio data wavelet transform unit, used for performing discrete wavelet transform on the audio data to obtain a time-frequency diagram of the audio data; An audio feature generating unit is used to extract audio features from the audio data time-frequency diagram using a feature extractor based on a CNN-GRU hybrid model to obtain the audio feature encoding vector.

[0008] Furthermore, the facial feature extraction module is used to: extract facial features from the facial expression image using a facial feature extractor based on a dilated pyramid model to obtain the facial image feature encoding feature map.

[0009] Furthermore, the facial image decomposition unit is used to: Performing feature decoupling on the facial image feature coding feature map along the channel dimension of the facial image feature coding feature map to obtain a set of facial image feature coding local feature matrices; Each facial image feature encoding local feature matrix in the set of facial image feature encoding local feature matrices is feature flattened to obtain the set of facial image local feature encoding vectors.

[0010] Furthermore, the clustering fusion unit includes: A facial image kernel feature extraction subunit, configured to perform facial image kernel feature extraction on the set of facial image local feature coding vectors to obtain a facial image kernel feature coding vector; An audio-facial feature prior clustering center coding vector determining subunit, used to determine an audio-facial feature prior clustering center coding vector based on the audio feature coding vector and the facial image kernel feature coding vector; The multimodal fine-grained joint subunit is used to perform multimodal fine-grained joint encoding on the set of the audio-facial feature prior clustering center encoding vector and the facial image local feature encoding vector to obtain the multimodal joint encoding vector of the autistic child object state.

[0011] Furthermore, the multimodal fine-grained joint subunit is used to: respectively calculating the Poincare distances between the audio-facial feature prior cluster center coding vector and each facial image local feature coding vector in the set of facial image local feature coding vectors to obtain a set of audio-facial feature semantic metric values; Using a binary function to perform clustering judgment on the set of audio-facial feature semantic measurement values ​​to obtain a set of audio-facial feature clustering coefficients; Based on the set of audio-facial feature clustering coefficients, the set of audio-facial feature prior clustering center encoding vectors and the set of facial image local feature encoding vectors are fine-grained cross-domain fused to obtain the multimodal joint encoding vector of the autistic child object state.

[0012] Furthermore, the behavior state judgment module includes: A behavior state result generating unit, used for inputting the multimodal joint coding vector of the state of the autistic child object into a behavior state judgement unit to obtain the behavior state of the autistic child object, wherein the behavior state of the autistic child object is whether to attempt to communicate; The intervention feedback generating unit is used to generate the intervention feedback in response to the behavior state of the autistic child subject being an attempt to communicate.

[0013] Furthermore, the behavior state determiner is a behavior state determiner based on a classifier.

[0014] According to another aspect of the present application, a method for assisting intervention therapy for children with autism based on intelligent interaction is provided, comprising: Acquire a facial expression image of an autistic child object captured by a camera and audio data of the autistic child object captured by a microphone; Extracting audio features from the audio data to obtain an audio feature encoding vector; Extracting facial features from the facial expression image to obtain a facial image feature encoding feature map; The audio feature coding vector and the facial image feature coding feature map are subjected to an audio-image multimodal data fine-grained union to obtain a multimodal joint coding vector of the object state of an autistic child, including: performing feature decoupling and feature flattening on the facial image feature coding feature map to obtain a set of facial image local feature coding vectors; performing kernel feature calculation and fine-grained clustering fusion on the set of the audio feature coding vector and the facial image local feature coding vector to obtain the multimodal joint coding vector of the object state of an autistic child; Based on the multimodal joint encoding vector of the state of the autistic child object, the behavioral state of the autistic child object is determined, and based on the behavioral state, intervention feedback is generated.

[0015] 10. The intelligent interaction-based intervention treatment auxiliary method for children with autism according to claim 9, extracting audio features from the audio data to obtain an audio feature encoding vector, comprising: Performing discrete wavelet transform on the audio data to obtain a time-frequency graph of the audio data; A feature extractor based on a CNN-GRU hybrid model is used to extract audio features from the time-frequency graph of the audio data to obtain the audio feature encoding vector.

[0016] Compared with the prior art, the autistic child intervention treatment auxiliary system and method based on intelligent interaction provided by the present application, which uses AI-based data processing technology to extract audio features from the audio data of the autistic child object, and extracts facial features from the facial expression image of the autistic child object, thereby intelligently judging the behavior state of the autistic child object as whether to try to communicate based on the multimodal fine-grained joint representation between the audio feature coding features and the facial image feature coding features. In this way, through continuous data collection, it is possible to more accurately capture extremely subtle changes in autistic children, ensuring that no instantaneous behavior is missed. In addition, by combining facial expression analysis with audio feature extraction, the judgment of the behavior state of autistic children can be made more accurate, thereby generating more targeted personalized intervention feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 The system block diagram of the intelligent interaction-based intervention therapy auxiliary system for children with autism according to an embodiment of the present application.

[0018] Figure 2 The present invention is a block diagram of an audio feature extraction module in an intelligent interaction-based intervention and treatment auxiliary system for children with autism according to an embodiment of the present application.

[0019] Figure 3 The present invention is a block diagram of an audio-image multimodal joint module in an intelligent interaction-based intervention and treatment auxiliary system for children with autism according to an embodiment of the present application.

[0020] Figure 4 The present invention is a block diagram of a behavior state judgment module in an intelligent interaction-based intervention and treatment auxiliary system for children with autism according to an embodiment of the present application.

[0021] Figure 5 The present invention is a flowchart of an auxiliary method for intervention and treatment of autistic children based on intelligent interaction according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] Autism, scientifically known as autism spectrum disorder, is a neurodevelopmental disorder. The disease usually appears in early childhood and has a wide and profound impact on the individual's social interaction, communication skills, interests and hobbies, and behavior patterns. Therefore, timely intervention is crucial to help the growth and development of autistic children, and can significantly promote their ability in social interaction, language expression, and behavior management.

[0024] Traditional intervention methods for children with autism mainly rely on manual assessments by human observers and guidance from therapists based on experience. However, due to human factors, professionals cannot always be present to conduct real-time observations, which leads to some behaviors that occur in an instant but are extremely important being easily overlooked. In particular, some subtle expressions or brief vocal expressions of children with autism often only appear in specific situations, but they play a key role in judging their behavioral status.

[0025] In addition, traditional methods rely heavily on the observer's intuition and personal experience. The therapist's experience level is directly related to the depth of understanding and response quality of behavioral changes. Even in face-to-face observation, human observation may miss key details such as slight eye movements and subtle contractions of facial muscles due to distraction or other factors, thus affecting the accuracy of the assessment and the comprehensiveness of the intervention treatment.

[0026] In response to the above technical problems, the technical concept of the present application is to obtain the facial expression image of the autistic child object collected by the camera and the audio data of the autistic child object collected by the microphone, and use AI-based image analysis technology and data processing algorithms to extract audio features from the audio data, and extract facial features from the facial expression image, so as to intelligently judge the behavioral state of the autistic child object as whether to try to communicate based on the multimodal fine-grained joint representation between the audio feature coding features and the facial image feature coding features. With the help of camera and microphone equipment, the present application realizes continuous data collection, can accurately capture extremely subtle changes, breaks through the limitations of manpower, and ensures that no instantaneous behavior is missed. In addition, by combining facial expression analysis with audio feature extraction, the behavioral performance of autistic children is fully interpreted from multiple angles, making the judgment of behavioral status more accurate, thereby generating more targeted personalized intervention feedback.

[0027] Figure 1 : is a system block diagram of an autism children intervention treatment auxiliary system based on intelligent interaction according to an embodiment of the present application. Figure 1 As shown, in the autism child intervention treatment auxiliary system 100 based on intelligent interaction, it includes: a child object facial audio acquisition module 110, which is used to obtain the facial expression image of the autism child object collected by the camera and the audio data of the autism child object collected by the microphone; an audio feature extraction module 120, which is used to extract audio features from the audio data to obtain an audio feature coding vector; a facial feature extraction module 130, which is used to extract facial features from the facial expression image to obtain a facial image feature coding feature map; an audio-image multimodal joint module 140, which is used to perform audio-image multimodal data fine-grained joint of the audio feature coding vector and the facial image feature coding feature map to obtain an autism child object state multimodal joint coding vector; a behavior state judgment module 150, which is used to determine the behavior state of the autism child object based on the autism child object state multimodal joint coding vector, and generate intervention feedback based on the behavior state.

[0028] In an embodiment of the present application, the child object facial audio acquisition module 110 is used to obtain the facial expression image of the autistic child object collected by the camera and the audio data of the autistic child object collected by the microphone. It should be understood that the facial expression image of the autistic child object specifically includes the basic facial features, expression dynamic information, eye-related information, etc. of the object; the audio data of the autistic child object specifically includes the language content information, voice pitch information, voice volume information, voice factor information and some non-verbal information of the child object. In detail, positive facial expressions in facial images, such as smiling, bright eyes and looking at the communication object, are usually signals of trying to communicate. These expressions show that the child is interested in the communication object and is willing to start or maintain communication interaction. On the contrary, expressionless, frowning or avoiding eye contact may imply a lack of willingness to communicate or the existence of communication barriers; the voice content in the audio data clearly expresses the child's needs, thoughts or emotions, and is direct evidence for judging communication attempts. For example, actively initiating a conversation, such as "what are you doing", indicates that the child has a strong willingness to communicate. Even if the language expression is incomplete or contains errors, as long as there is an intention to express, it shows the behavior of trying to communicate; and the appropriate pitch, volume and speaking speed are also key information to determine whether the child is trying to communicate. In general, facial expression images and audio data complement each other and provide more comprehensive information together. Through multimodal fusion analysis of the two types of data, it is possible to more accurately determine whether autistic children are trying to communicate, avoid possible misjudgments based on a single data source, and improve the accuracy and reliability of judgments.

[0029] The following is a detailed description of a specific implementation process of "obtaining facial expression images of autistic child objects collected by a camera and audio data of the autistic child objects collected by a microphone": First of all, the preparation of the environment and equipment is crucial. A dedicated intervention therapy space should be created. This space should be quiet, softly lit and well lit. At the same time, the layout should create a warm, comfortable and attractive atmosphere to minimize the interference of external factors on data collection. In terms of the layout of the room, it should be planned reasonably to ensure that children have enough space to move around and that the camera and microphone can be in the best working position. The installation position of the camera needs to be carefully debugged. It is usually installed in a suitable position above or near the screen. The angle and focal length must be adjusted accurately to ensure that every subtle detail of the child's facial expression during the interaction with the virtual character can be fully and clearly captured, such as the blinking of the eyes, the opening and closing of the mouth, the raising or frowning of the eyebrows and other key parts of the dynamic changes, without any omission. The placement of the microphone is also very particular. It should be placed close to the child, such as on a table or near a child seat, to ensure that the various sounds of the child are accurately recorded, whether it is clear voice expression, cheerful laughter, aggrieved crying, or some subtle breathing sounds, mumbling sounds, etc., can be accurately captured and stably connected to the data processing system. Before formal use, the equipment needs to be fully calibrated and rigorously tested to ensure the stability and accuracy of data collection, laying a solid foundation for subsequent analysis.

[0030] Secondly, the design of virtual characters and the setting of interactive programs are one of the core links of the entire implementation process. The virtual characters created should be extremely friendly and attractive, with vivid and distinctive images, such as cute cartoon animals or friendly children. These virtual characters are presented on the screen in the form of exquisite animations, with rich and diverse expressions and natural and smooth movements, which can quickly attract the attention of children with autism and stimulate their interest. In terms of interactive programs, a series of creative and targeted interactive scripts need to be carefully compiled, covering simple and interesting games, fascinating storytelling, daily communication topics close to life, and other forms. For example, a game called "Animal Park Adventure" is designed, in which the virtual character acts as a guide to lead children to explore the virtual animal park, encounter various animals along the way and introduce their characteristics and habits, while asking children questions and guiding them to participate in the interaction. In this process, the virtual character's voice and tone should be natural and gentle, with a moderate speaking speed, and can be flexibly adjusted according to the child's response and ability. If the child shows signs of difficulty in understanding, the voice can be slowed down and the words used are simpler; if the child responds positively, the voice can be more lively and lively to enhance the fun of the interaction. At the same time, the interactive program should cleverly set the trigger points for data collection to ensure that the children’s facial expression images and audio data can be accurately obtained during key interactive links, without missing any valuable information.

[0031] Furthermore, the data collection process is a key operational link in the entire implementation process. When the child enters the intervention treatment area and is ready to interact with the virtual character, the system is officially started. At this time, the virtual character will first give a short and attractive opening introduction and guidance. At the same time, the camera and microphone will start to collect data continuously. The system will record the child’s first reaction expression when he first sees the virtual character, as well as any sounds that may be made, and use it as important basic data. During the entire interaction process, the camera and microphone always keep working in real time, continuously capturing the child’s behavior and sound information. For example, in the interactive session of the "Animal Sound Imitation Show", the virtual character plays the sound of an animal and invites the child to imitate. At this time, the system will focus on collecting the child’s facial expression changes and voice answers when listening to the sound, preparing to imitate, and imitating. Children may show expressions such as focused expressions, curious eyes, puzzled frowns, or excited smiles, and the audio characteristics such as voice intonation, speech speed, and volume when imitating the sound will be recorded in detail. For some non-verbal behaviors of children, such as nodding, shaking head, waving hands and other gestures, the camera will also keenly capture relevant image information, and use advanced image recognition technology to conduct preliminary analysis and marking, providing comprehensive data support for subsequent in-depth research.

[0032] The collected facial expression images and audio data will be instantly transferred to a dedicated storage device and carefully classified and stored according to the time sequence and interaction links. At the same time, the system will perform preliminary processing on the data, use professional algorithms and technologies to remove some obvious noise interference, perform simple clarity enhancement and optimization processing on the image, and ensure the quality of the data for subsequent more in-depth and accurate analysis.

[0033] In the embodiment of the present application, the audio feature extraction module 120 is used to extract audio features from the audio data to obtain an audio feature encoding vector. Specifically, Figure 2 FIG. 1 is a block diagram of an audio feature extraction module in an intelligent interaction-based autism children intervention and treatment auxiliary system according to an embodiment of the present application. Figure 2 As shown, the audio feature extraction module 120 includes: an audio data wavelet transform unit 121, used to perform discrete wavelet transform on the audio data to obtain an audio data time-frequency diagram; an audio feature generation unit 122, used to use a feature extractor based on a CNN-GRU hybrid model to perform audio feature extraction on the audio data time-frequency diagram to obtain the audio feature encoding vector.

[0034] In an embodiment of the present application, the audio data wavelet transform unit 121 is used to perform discrete wavelet transform on the audio data to obtain an audio data time-frequency diagram. Accordingly, considering that not all information in the audio data is of equal importance to judging the behavior state of autistic children, the voice, intonation, speech speed and some background sounds of the child's speech are included. Therefore, in order to focus on key information related to the behavior state, such as the fundamental frequency and formant of the voice, the present application extracts audio features from the audio data to extract features closely related to emotion and intention expression, and obtains an audio feature encoding vector, so that it is helpful to more accurately analyze the behavior of autistic children. In particular, in a specific embodiment of the present application, the audio data is subjected to discrete wavelet transform to decompose the audio signal into sub-signals of different frequency bands and time scales, and an audio data time-frequency diagram is obtained, which presents the frequency change of the audio signal over time in an intuitive visual manner. It should be understood that an important feature of discrete wavelet transform is that it can provide multi-resolution analysis. Audio signals usually contain multiple frequency components, and the performance of these components at different time scales is crucial to understanding the audio behavior of children with autism.

[0035] In an embodiment of the present application, the audio feature generating unit 122 is used to extract audio features from the audio data time-frequency graph using a feature extractor based on a CNN-GRU hybrid model to obtain the audio feature encoding vector. It should be understood that in order to effectively capture and mine the audio feature information in the time-frequency graph, further, a feature extractor based on a CNN-GRU hybrid model is used to extract audio features from the audio data time-frequency graph to obtain the audio feature encoding vector. In other words, the audio data time-frequency graph is essentially a two-dimensional image representation, with the horizontal axis being time and the vertical axis being frequency, and the values ​​at different positions representing the signal strength at the time-frequency point. CNN (Convolutional Neural Networks) has a natural advantage in processing image data, and the convolution kernel in its convolution layer can slide on the time-frequency graph, and capture the local feature pattern in the time-frequency graph through the local receptive field, such as representing the key audio features such as speech onset and intonation changes. The audio signal has a time series characteristic, and there is a correlation between the previous and next audio information. GRU (Gated Recurrent Unit), as a variant of recurrent neural network, can effectively process time series data and capture the dynamic changes of audio signals in the time dimension, such as the changing patterns of speech rhythm and speech speed in time series.

[0036] In an embodiment of the present application, the facial feature extraction module 130 is used to extract facial features from the facial expression image to obtain a facial image feature encoding feature map. Specifically, in an embodiment of the present application, the facial feature extraction module 130 is used to: use a facial feature extractor based on a hollow pyramid model to extract facial features from the facial expression image to obtain the facial image feature encoding feature map. Accordingly, considering that the subtle expression changes in the facial expression image contain rich psychological and behavioral information, such as the corners of the mouth, eye contact, etc. Based on this, in order to dig out the characteristic information hidden in the facial expression, so as to help judge the child's communication willingness and emotional state, the present application extracts facial features from the facial expression image to dig out hidden information such as micro-expressions, and obtains a facial image feature encoding feature map. In particular, in a specific embodiment of the present application, a facial feature extractor based on a hollow pyramid model is used to extract facial features from the facial expression image to obtain the facial image feature encoding feature map. It should be understood that the rich information of facial expressions is reflected at different scales. For example, subtle muscle movements around the eyes (such as blinking and pupil changes) are small-scale features that can convey emotions such as surprise and concentration; while muscle stretching of the entire face, upturned or downturned corners of the mouth, etc. are larger-scale features that reflect more obvious emotional states, such as happiness or sadness. The dilated pyramid model can extract features from images at multiple scales through convolution operations with different dilation rates to capture important information at different scales in facial expression images, that is, it can effectively perceive changes from local details to overall facial structure, thereby more comprehensively portraying facial expression features, so as to more accurately reflect the information conveyed by the facial expressions of children with autism. For example, accurately captured facial muscle movement features can more accurately judge the emotional state of children with autism, and thus provide a more reliable basis for judging whether they are trying to communicate.

[0037] In the embodiment of the present application, the audio-image multimodal joint module 140 is used to perform a fine-grained audio-image multimodal data joint on the audio feature coding vector and the facial image feature coding feature map to obtain a multimodal joint coding vector of the autistic child object state. Specifically, Figure 3 FIG. 1 is a block diagram of an audio-image multimodal joint module in an intelligent interaction-based autism children intervention and treatment auxiliary system according to an embodiment of the present application. Figure 3As shown, the audio-image multimodal joint module 140 includes: a facial image decomposition unit 141, used to perform feature decoupling and feature flattening on the facial image feature coding feature map to obtain a set of facial image local feature coding vectors; a clustering fusion unit 142, used to perform kernel feature calculation and fine-grained clustering fusion on the set of the audio feature coding vector and the facial image local feature coding vector to obtain the autistic child object state multimodal joint coding vector.

[0038] It should be understood that the audio feature coding vector reflects the key information in the audio signal of autistic children, such as voice intonation, sound frequency, etc., while the facial image feature coding feature map captures the information conveyed by facial expressions, such as emotional expressions such as joy, anger, sorrow, and happiness. The data of these two modalities describe the behavioral state of autistic children from different angles. Therefore, in order to integrate these multi-source information, give full play to their respective advantages, overcome the limitations of single modality data, and understand their behavioral state more comprehensively, in the technical solution of the present application, the audio feature coding vector and the facial image feature coding feature map are subjected to audio-image multimodal data fine-grained joint to obtain autistic children's object state multimodal joint coding vector. In particular, audio and facial images belong to different information domains, and there is a potential fine-grained relationship between them. For example, a specific voice intonation may be accompanied by a specific facial expression, such as an angry tone may be accompanied by facial expressions such as frowning and staring. This method can dig out these cross-domain fine-grained relationships and deeply analyze the correlation between different modal data to more accurately interpret their behavioral states.

[0039] Specifically, in the embodiment of the present application, the facial image decomposition unit 141 is used to: perform feature decoupling on the facial image feature coding feature map along the channel dimension of the facial image feature coding feature map to obtain a set of facial image feature coding local feature matrices; and perform feature flattening on each facial image feature coding local feature matrix in the set of facial image feature coding local feature matrices to obtain a set of facial image local feature coding vectors. The above process can be expressed as: ; in, is the facial image feature encoding feature map, It is a feature decoupling and feature flattening operation. is the set of local feature encoding vectors of facial images, , , , and are respectively the first, second, and third in the set of local feature encoding vectors of the facial image , and A facial image local feature encoding vector.

[0040] It should be understood that the facial image feature coding feature map simultaneously contains information on expressions, facial muscle movements, facial features, and other aspects. In order to separate this information and provide more detailed information for subsequent analysis, it is necessary to perform feature decoupling operations on the facial image feature coding feature map along the channel dimension of the facial image feature coding feature map in this application. Each facial image feature coding local feature matrix in the set of facial image feature coding local feature matrices obtained by decoupling only focuses on and carries a single type of semantic information, which can avoid mutual interference between different semantic information, thereby making subsequent processing more targeted and accurate. Accordingly, considering that there may be some redundant information in the multidimensional feature matrix, this redundant information may be caused by the spatial structure of the feature or the correlation between the dimensions. In order to remove some redundancy caused by the spatial structure, so that the model can more effectively capture key information during the learning process without being disturbed by redundant spatial structure information, it is necessary to flatten the features of each facial image feature coding local feature matrix in the set of facial image feature coding local feature matrices in this application to obtain a set of facial image local feature coding vectors. Moreover, through the feature flattening operation, it can be ensured that local features can be processed uniformly while maintaining their respective characteristics.

[0041] Specifically, in the embodiment of the present application, the clustering fusion unit 142 includes: a facial image kernel feature extraction subunit, which is used to extract facial image kernel features from the set of facial image local feature coding vectors to obtain a facial image kernel feature coding vector; an audio-facial feature prior clustering center coding vector determination subunit, which is used to determine the audio-facial feature prior clustering center coding vector based on the audio feature coding vector and the facial image kernel feature coding vector; a multimodal fine-grained joint subunit, which is used to perform multimodal fine-grained joint encoding on the set of the audio-facial feature prior clustering center coding vector and the facial image local feature coding vector to obtain the multimodal joint coding vector of the autistic child object state.

[0042] Next, facial image kernel feature extraction is performed on the set of facial image local feature coding vectors to obtain a facial image kernel feature coding vector. The above process can be expressed as: ; in, is the set of local feature encoding vectors of facial images, , are respectively the first , facial image local feature encoding vector, For Perform kernel feature extraction, represents the one-norm of the eigenvector, represents the number of vectors in the set of local feature encoding vectors of the facial image minus one, yes The corresponding facial image semantic difference value, Expressed as a natural constant The exponential function with base , is the number of vectors in the set of local feature encoding vectors of the facial image, is the facial image kernel feature encoding vector.

[0043] It should be understood that when dealing with the problem of determining the behavioral state of autistic children, the audio feature encoding vector and the facial image local feature encoding vector set represent data of different modalities. In order to effectively fuse the data of these two modalities, it is necessary to find the connection point between them, that is, the cross-modal data interaction anchoring bridge. By performing kernel analysis on the facial image local feature encoding vector set, the most core and representative features in the facial image data can be mined. These features can serve as a key bridge for interaction with audio modality data, helping to establish an effective connection between the two modal data.

[0044] Then, based on the audio feature coding vector and the facial image kernel feature coding vector, the audio-facial feature prior clustering center coding vector is determined. The above process can be expressed as: ; in, is the facial image kernel feature encoding vector, is the audio feature encoding vector, For cascade operation, is the weight transformation matrix, is the bias vector, is the audio-facial feature prior cluster center encoding vector.

[0045] It should be understood that the audio feature encoding vector and the facial image kernel feature encoding vector belong to different modalities of data, each containing unique information about the behavioral state of autistic children. In order to achieve effective fusion and interaction of the two modal data, a clear connection point is required. The audio-facial feature prior clustering center encoding vector plays such a role. It serves as an anchoring bridge for cross-modal data interaction, linking the two modal data of audio and facial image, so that cross-modal fine-grained interaction can be carried out in an orderly manner. Moreover, by determining the audio-facial feature prior clustering center encoding vector, the advantages of these two modalities can be combined to more fully understand the children's behavioral performance during the communication process.

[0046] More specifically, in the embodiment of the present application, the multimodal fine-grained joint subunit is used to respectively calculate the Poincare distance between the audio-facial feature prior cluster center encoding vector and each facial image local feature encoding vector in the set of facial image local feature encoding vectors to obtain a set of audio-facial feature semantic metric values. The process can be expressed as: ; in, is the first in the set of local feature encoding vectors of the facial image facial image local feature encoding vector, is the audio-facial feature prior cluster center encoding vector, is the square of the Euclidean norm of the vector, is the inverse hyperbolic cosine function, for and The audio-facial feature semantic metric between them; The set of audio-facial feature semantic metric values ​​is clustered using a binary function to obtain a set of audio-facial feature clustering coefficients. The process can be expressed as: ; in, for and The audio-facial feature semantic metric between yes The corresponding audio-facial feature clustering coefficient, is the preset threshold; Based on the set of audio-facial feature clustering coefficients, the set of audio-facial feature prior clustering center encoding vectors and the set of facial image local feature encoding vectors are fine-grained cross-domain fused to obtain the multimodal joint encoding vector of the autistic child object state. The process can be expressed as: ; in, is the number of vectors in the set of local feature encoding vectors of the facial image, is the first in the set of local feature encoding vectors of the facial image facial image local feature encoding vector, is the audio-facial feature prior cluster center encoding vector, yes The corresponding audio-facial feature clustering coefficient, It is the multimodal joint encoding vector of the autistic child object state.

[0047] It should be understood that the audio-facial feature prior cluster center encoding vector integrates the core correlation information of the two modal data of audio and facial image, but the facial image local feature encoding vector set also contains rich facial detail information. Through multimodal fine-grained joint encoding, with the audio-facial feature prior cluster center encoding vector as a bridge, these facial detail information can be integrated with the existing core correlation information, and a new feature representation that more comprehensively and comprehensively reflects multiple modal information can be generated, that is, the multimodal joint encoding vector of the object state of autistic children, which can more completely capture all clues related to the autistic children's attempts to communicate. It is also worth mentioning that when actually collecting audio and facial image data of autistic children, due to environmental factors, equipment errors or the abnormal behavior of the children themselves, the feature information encoded in the original data still has noise or outliers. Through multimodal fine-grained joint encoding, feature information that may be noise or outliers can be filtered out, focusing on the part that best reflects the essential characteristics of children's behavior, thereby improving the reliability of the data and the accuracy of the analysis results.

[0048] In the embodiment of the present application, the behavior state determination module 150 is used to determine the behavior state of the autistic child object based on the multimodal joint encoding vector of the autistic child object state, and generate intervention feedback based on the behavior state. Specifically, Figure 4 FIG. 1 is a block diagram of a behavior state judgment module in an intelligent interaction-based autism child intervention and treatment auxiliary system according to an embodiment of the present application. Figure 4 As shown, the behavior state judgment module 150 includes: a behavior state result generating unit 151, which is used to input the multimodal joint encoding vector of the autistic child object state into a behavior state judger to obtain the behavior state of the autistic child object, and the behavior state of the autistic child object is whether to try to communicate; an intervention feedback generating unit 152, which is used to generate the intervention feedback in response to the behavior state of the autistic child object being trying to communicate.

[0049] In the embodiment of the present application, the behavior state result generating unit 151 is used for inputting the multimodal joint coding vector of the autistic child object state into the behavior state judger to obtain the behavior state of the autistic child object, and the behavior state of the autistic child object is whether to try to communicate. Specifically, the behavior state judger described in the present application is a behavior state judger based on a classifier. That is, the multimodal joint coding vector of the autistic child object state obtained by fine-grained combination of the audio feature coding vector and the facial image feature coding feature map is classified and processed, so as to utilize the ability of the classifier to perform well in the category task, to learn the feature pattern in the multimodal joint coding vector, and accurately classify it into the corresponding behavior state category, so as to judge that it belongs to the category of trying to communicate or not trying to communicate. In particular, in a specific embodiment of the present application, the multimodal joint encoding vector of the autistic child object state is input into a behavioral state judge to obtain the behavioral state of the autistic child object, and the behavioral state of the autistic child object is whether to attempt to communicate, including: using the fully connected layer of the classifier to fully connect the multimodal joint encoding vector of the autistic child object state to obtain a multimodal joint fully connected encoding feature vector of the autistic child object state; inputting the multimodal joint fully connected encoding feature vector of the autistic child object state into the Softmax classification function of the classifier to obtain the probability value of the multimodal joint encoding vector of the autistic child object state belonging to each classification label, and the classification label includes a label for indicating that the behavioral state of the autistic child object is attempting to communicate and a label for indicating that the behavioral state of the autistic child object is not attempting to communicate; and the classification label corresponding to the largest of the probability values ​​is determined as the behavioral state of the autistic child object.

[0050] In a preferred example, since the audio feature encoding vector and the facial image feature encoding feature map respectively represent the audio features of the audio data of the autistic child object and the facial image semantic features of its facial expression image, when performing cross-domain fine-grained aggregation analysis based on modality prior, the feature population attributes of different modalities will have differences in cross-domain fine-grained aggregation fairness based on different modality prior levels, thereby affecting the inclusiveness of the aggregated feature distribution of the multimodal joint encoding vector of the autistic child object state, and reducing the accuracy of the determined behavioral state of the autistic child object.

[0051] Therefore, the present application considers that when determining the behavioral state of the autistic child object based on the multimodal joint coding vector of the autistic child object state, the multimodal joint coding vector of the autistic child object state is first optimized, including the steps of: Determine the eigenvalue mean corresponding to the multimodal joint encoding vector of the autistic child object state and the standard deviation of the eigenvalues ; The multimodal joint encoding vector of the autistic child object state and the point subtraction vector of the eigenvalue mean and the eigenvalue standard deviation are multiplied to obtain a first autistic child object state multimodal joint encoding intermediate vector, and the process can be expressed as: ; in, represents the multimodal joint encoding vector of the autistic child object state, It means point multiplication by position. It means to subtract by position point. express The corresponding eigenvalue standard deviation is, express The corresponding feature mean is An intermediate vector representing a multimodal joint encoding of the state of the first autistic child object; The point-wise multiplication of the multimodal joint coding vector of the autistic child object state and the point-wise subtraction vector of the eigenvalue standard deviation and the eigenvalue mean is performed to obtain a second autistic child object state multimodal joint coding intermediate vector. The process can be expressed as: ; in, represents the multimodal joint encoding vector of the autistic child object state, It means point multiplication by position. It means to subtract by position point. express The corresponding eigenvalue standard deviation is, express The corresponding feature mean is A multimodal joint encoding intermediate vector representing the state of the second autistic child object; After the bit-by-bit reciprocal of the second autistic child object state multimodal joint coding intermediate vector is dot-multiplied with the first autistic child object state multimodal joint coding intermediate vector, the bit-by-bit logarithm with base 2 is taken to obtain the autistic child object state multimodal joint coding correction vector. The process can be expressed as: ; in, represents the second autistic child object state multimodal joint encoding intermediate vector, Representation calculation The inverse of each eigenvalue in , It means point multiplication by position. represents the intermediate vector of the multimodal joint encoding of the state of the first autistic child object, A multimodal joint encoding correction vector representing the state of the autistic child object; The eigenvalue mean Divide by the standard deviation of the eigenvalue The square root of the quotient is multiplied by the weight hyperparameter, and then added to the multimodal joint coding correction vector of the autistic child object state to obtain the optimized multimodal joint coding vector of the autistic child object state: ; in, represents the weight hyperparameter, express The corresponding eigenvalue standard deviation is, express The corresponding feature mean is Indicates adding by position point. represents the multimodal joint coding correction vector of the autistic child object state, A multimodal joint encoding vector representing the optimized autistic child object state; The behavioral state of the autistic child object is determined based on the optimized multimodal joint encoding vector of the autistic child object state.

[0052] That is, based on the attribute level fairness differences of the data population corresponding to the sequence fusion features of the multimodal joint encoding vector of the autistic child object state, in order to improve the aggregation inclusiveness under the feature distribution diversity of the multimodal joint encoding vector of the autistic child object state, the group feature information interactive propagation of the multimodal joint encoding vector of the autistic child object state is corrected by taking the crossover probability value constraint based on the multimodal joint encoding vector of the autistic child object state as the interactive fairness target representation, and taking the unified statistical feature response interaction based on the multimodal joint encoding vector of the autistic child object state as the feature distribution multi-level fairness target bias to achieve a robust distribution fairness unified representation of the multimodal joint encoding vector of the autistic child object state, forming a fair collaboration paradigm under the feature distribution framework of the multimodal joint encoding vector of the autistic child object state, and improving the accuracy of the determined behavioral state of the autistic child object.

[0053] In the present application embodiment, the intervention feedback generating unit 152 is used for attempting to communicate in response to the behavior state of the autistic child object, generating the intervention feedback. It should be understood that when the autistic child attempts to communicate, this is a positive behavior performance. Generating intervention feedback can strengthen this positive behavior in time, so that the children realize that their communication attempts are recognized and encouraged, so that the autistic child can gradually establish the connection between the communication behavior and the positive experience, so as to be more willing to actively communicate. The intervention feedback here can be simple voice prompts, virtual character emotions on the screen matching responses, etc.

[0054] The following is a detailed description of a specific implementation process of "generating the intervention feedback in response to the behavioral state of the autistic child being an attempt to communicate": First of all, from the perspective of voice feedback, the system will be designed specifically according to the child's language ability and expression characteristics. If the child's language expression is relatively simple and basic, the voice feedback will use concise, clear and easy-to-understand vocabulary and sentence structures. For example, when a child utters some vague syllables or simple words, the system may respond with "You said it very well, can you tell me more about it?" The voice tone is gentle and encouraging, and the speed will be slowed down to ensure that the child can clearly receive the information and feel positive feedback. For children with slightly stronger language skills, the voice feedback will be richer and more guiding, such as "Your ideas are very interesting, we can discuss them in depth together, what do you want to do next?" In this way, children are encouraged to further express their ideas and feelings and expand the depth of communication.

[0055] Secondly, visual feedback on the screen is also an important part. The animation effects of the virtual characters will be adjusted dynamically according to the children's behavior. If the child's eyes are focused and the expression is positive when trying to communicate, the virtual character will respond with more lively movements, such as jumping happily, waving arms, etc. At the same time, the facial expression will show a happy and encouraging look, with eyes wide open and the corners of the mouth raised, sending positive emotional signals to the children. If the child shows some tension or uneasiness, the virtual character will make soothing movements, such as slowly approaching, nodding gently, accompanied by a gentle expression, and the eyes are soft and full of concern, so that the child can feel safe and supported visually, thereby relieving tension and being more willing to continue communicating.

[0056] Furthermore, the adjustment of interactive content is a key part of intervention feedback. Once it is determined that the child is trying to communicate, the system will quickly analyze the topic or interest point that the child is concerned about. For example, if a child frequently mentions a toy or animal during communication, the system will immediately call up pictures, videos or stories related to the toy or animal from its rich material library and display them on the screen in a vivid form, accompanied by voice explanations, such as "Look, this is the little rabbit you mentioned. It has many interesting stories. Do you want to listen?" This content adjustment based on children's interests can greatly enhance children's participation and willingness to communicate, making them more engaged in the interactive process.

[0057] In addition, intervention feedback also takes into account factors such as time and frequency. The system will not provide too much information or feedback at one time, but will arrange the rhythm of feedback reasonably according to the child's concentration and reaction speed. If the child can respond positively in a short period of time, the system will appropriately speed up the frequency of feedback, but always keep it within the range that the child can accept; if the child needs more time to digest and respond, the system will wait patiently and give gentle reminders at the appropriate time, such as "Don't worry, think slowly."

[0058] During the actual implementation process, professionals will continue to observe and evaluate the effects of intervention feedback. They will record children's reactions to each feedback, including facial expressions, movements, and language responses. By analyzing these data, the feedback strategy of the system is continuously optimized. If it is found that a certain feedback method fails to achieve the expected effect, professionals will promptly adjust and improve the voice, visual or interactive content to ensure that the intervention feedback can always effectively promote children's communication and development.

[0059] In summary, the autistic child intervention treatment auxiliary system 100 based on intelligent interaction according to the embodiment of the present application is illustrated, and it adopts data processing technology based on AI to extract audio features from the audio data of the autistic child object, and extracts facial features from the facial expression image of the autistic child object at the same time, and intelligently judges the behavior state of the autistic child object as whether to try to communicate based on the multimodal fine-grained joint representation between the audio feature coding feature and the facial image feature coding feature. In this way, through continuous data collection, it is possible to more accurately capture the extremely subtle changes of the autistic child, ensuring that no instantaneous behavior is missed. And, by combining facial expression analysis with audio feature extraction, it is possible to make the judgment of the behavior state of the autistic child more accurate, and then generate more targeted personalized intervention feedback.

[0060] Figure 5 FIG. 1 is a flow chart of an auxiliary method for intervention and treatment of autistic children based on intelligent interaction according to an embodiment of the present application. Figure 5As shown, in the auxiliary method for intervention and treatment of autistic children based on intelligent interaction, the method includes: S110, acquiring a facial expression image of an autistic child object collected by a camera and audio data of the autistic child object collected by a microphone; S120, extracting audio features from the audio data to obtain an audio feature coding vector; S130, extracting facial features from the facial expression image to obtain a facial image feature coding feature map; S140, performing audio-image multimodal data fine-grained union of the audio feature coding vector and the facial image feature coding feature map to obtain a multimodal joint coding vector of the autistic child object state; S150, determining the behavioral state of the autistic child object based on the multimodal joint coding vector of the autistic child object state, and generating intervention feedback based on the behavioral state.

[0061] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned intelligent interaction-based autism children intervention treatment auxiliary method have been referred to above. Figures 1 to 4 The invention has been introduced in detail in the description of the intelligent interaction-based intervention treatment auxiliary system for children with autism, and therefore, its repeated description will be omitted.

[0062] In summary, the autistic child intervention treatment auxiliary method based on intelligent interaction based on the embodiment of the present application is illustrated, which uses AI-based data processing technology to extract audio features from the audio data of the autistic child object, and extracts facial features from the facial expression image of the autistic child object, thereby intelligently judging the behavior state of the autistic child object based on the multimodal fine-grained joint representation between the audio feature coding feature and the facial image feature coding feature as to whether to try to communicate. In this way, through continuous data collection, it is possible to more accurately capture the extremely subtle changes of autistic children, ensuring that no instantaneous behavior is missed. Moreover, by combining facial expression analysis with audio feature extraction, it is possible to make the judgment of the behavior state of autistic children more accurate, thereby generating more targeted personalized intervention feedback.

Claims

1. An intelligent interactive intervention and treatment auxiliary system for children with autism, characterized by: include: A child object facial audio acquisition module, used to acquire the facial expression image of the autistic child object acquired by the camera and the audio data of the autistic child object acquired by the microphone; An audio feature extraction module, used for extracting audio features from the audio data to obtain an audio feature encoding vector; A facial feature extraction module, used for extracting facial features from the facial expression image to obtain a facial image feature encoding feature map; An audio-image multimodal joint module is used to perform an audio-image multimodal data fine-grained joint of the audio feature coding vector and the facial image feature coding feature map to obtain a multimodal joint coding vector of the object state of an autistic child, wherein the audio-image multimodal joint module includes: a facial image decomposition unit, used to perform feature decoupling and feature flattening on the facial image feature coding feature map to obtain a set of facial image local feature coding vectors; a clustering fusion unit, used to perform kernel feature calculation and fine-grained clustering fusion on the set of the audio feature coding vector and the facial image local feature coding vector to obtain the multimodal joint coding vector of the object state of an autistic child; The behavior state judgment module is used to determine the behavior state of the autistic child object based on the multimodal joint coding vector of the autistic child object state, and generate intervention feedback based on the behavior state.

2. The intelligent interaction-based autism children intervention and treatment auxiliary system according to claim 1, characterized in that: The audio feature extraction module comprises: An audio data wavelet transform unit, used for performing discrete wavelet transform on the audio data to obtain a time-frequency diagram of the audio data; An audio feature generating unit is used to extract audio features from the audio data time-frequency diagram using a feature extractor based on a CNN-GRU hybrid model to obtain the audio feature encoding vector.

3. The intelligent interaction-based autism children intervention and treatment auxiliary system according to claim 2 is characterized in that: The facial feature extraction module is used to extract facial features from the facial expression image using a facial feature extractor based on a dilated pyramid model to obtain the facial image feature encoding feature map.

4. The intelligent interaction-based autism children intervention and treatment auxiliary system according to claim 3 is characterized in that: The facial image decomposition unit is used to: Performing feature decoupling on the facial image feature coding feature map along the channel dimension of the facial image feature coding feature map to obtain a set of facial image feature coding local feature matrices; Each facial image feature encoding local feature matrix in the set of facial image feature encoding local feature matrices is feature flattened to obtain the set of facial image local feature encoding vectors.

5. The intelligent interaction-based autism children intervention and treatment auxiliary system according to claim 4, characterized in that: The clustering fusion unit comprises: A facial image kernel feature extraction subunit, configured to perform facial image kernel feature extraction on the set of facial image local feature coding vectors to obtain a facial image kernel feature coding vector; An audio-facial feature prior clustering center coding vector determining subunit, used to determine an audio-facial feature prior clustering center coding vector based on the audio feature coding vector and the facial image kernel feature coding vector; The multimodal fine-grained joint subunit is used to perform multimodal fine-grained joint encoding on the set of the audio-facial feature prior clustering center encoding vector and the facial image local feature encoding vector to obtain the multimodal joint encoding vector of the autistic child object state.

6. The intelligent interaction-based autism children intervention and treatment auxiliary system according to claim 5, characterized in that: The multimodal fine-grained joint subunit is used to: respectively calculating the Poincare distances between the audio-facial feature prior cluster center coding vector and each facial image local feature coding vector in the set of facial image local feature coding vectors to obtain a set of audio-facial feature semantic metric values; Using a binary function to perform clustering judgment on the set of audio-facial feature semantic measurement values ​​to obtain a set of audio-facial feature clustering coefficients; Based on the set of audio-facial feature clustering coefficients, the set of audio-facial feature prior clustering center encoding vectors and the set of facial image local feature encoding vectors are fine-grained cross-domain fused to obtain the multimodal joint encoding vector of the autistic child object state.

7. The intelligent interaction-based autism children intervention and treatment auxiliary system according to claim 6, characterized in that: The behavior state judgment module includes: A behavior state result generating unit, used for inputting the multimodal joint coding vector of the state of the autistic child object into a behavior state judgement unit to obtain the behavior state of the autistic child object, wherein the behavior state of the autistic child object is whether to attempt to communicate; The intervention feedback generating unit is used to generate the intervention feedback in response to the behavior state of the autistic child subject being an attempt to communicate.

8. According to the intelligent interaction-based autism children intervention treatment auxiliary system according to claim 7, the behavior state determiner is a classifier-based behavior state determiner.

9. An intelligent interactive intervention and treatment aid method for children with autism, characterized in that: include: Acquire a facial expression image of an autistic child object captured by a camera and audio data of the autistic child object captured by a microphone; Extracting audio features from the audio data to obtain an audio feature encoding vector; Extracting facial features from the facial expression image to obtain a facial image feature encoding feature map; The audio feature coding vector and the facial image feature coding feature map are subjected to an audio-image multimodal data fine-grained union to obtain a multimodal joint coding vector of the object state of an autistic child, including: performing feature decoupling and feature flattening on the facial image feature coding feature map to obtain a set of facial image local feature coding vectors; performing kernel feature calculation and fine-grained clustering fusion on the set of the audio feature coding vector and the facial image local feature coding vector to obtain the multimodal joint coding vector of the object state of an autistic child; Based on the multimodal joint encoding vector of the state of the autistic child object, the behavioral state of the autistic child object is determined, and based on the behavioral state, intervention feedback is generated.

10. The intelligent interaction-based intervention and treatment auxiliary method for autistic children according to claim 9, characterized in that: Extracting audio features from the audio data to obtain an audio feature encoding vector includes: Performing discrete wavelet transform on the audio data to obtain a time-frequency graph of the audio data; A feature extractor based on a CNN-GRU hybrid model is used to extract audio features from the time-frequency graph of the audio data to obtain the audio feature encoding vector.

Images