Child social companion robot interaction system based on emotion recognition

Abstract

The application relates to the technical field of artificial intelligence, and relates to a child social accompanying robot interaction system based on emotion recognition, which comprises a multi-source perception and feature extraction module, a cross-modal correlation analysis module, a multi-dimensional emotion fusion module, an emotion strategy mapping and generation module, a double-channel instruction compiling module and a collaborative execution and interaction response module. Multi-scale feature extraction is performed on a face image and a voice signal to obtain a face feature vector and a voice emotion vector. A correlation weight matrix of a target child is constructed. The face feature vector and the voice feature vector are coupled in multiple dimensions to obtain a composite emotion feature. The emotion state is mapped in a strategy mode to obtain an emotion interaction strategy. The composite emotion feature is compiled in an instruction mode to obtain a voice content instruction and an expression action instruction. The voice content instruction and the expression action instruction are input into an execution terminal to obtain an emotion accompanying interaction response of the robot. The application can improve the efficiency of child emotion interaction response.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a child social companion robot interaction system based on emotion recognition. Background Technology

[0002] Existing interactive systems for children's social companion robots rely heavily on single-modal feature data for emotion recognition, making it difficult to fully capture the complex characteristics of children's emotional expressions. This leads to biased and incomplete judgments of children's emotional states. At the same time, the correlation between data from different modalities has not been effectively explored and integrated, resulting in insufficient accuracy and reliability of emotion recognition and failing to provide precise support for the formulation of subsequent interaction strategies.

[0003] In the interactive response phase, existing systems lack a dynamic adaptation mechanism for children's emotional states. The generated interactive strategies are often fixed and simplistic, making it difficult to effectively match children's real-time emotional needs. This affects the naturalness and effectiveness of the interaction, and fails to fully leverage the emotional companionship function of social companion robots. Therefore, improving the accuracy of emotion recognition and the adaptability of interactive strategies in children's social companion robots has become an urgent problem to be solved. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides a child social companion robot interaction system based on emotion recognition, characterized in that the system includes a multi-source perception and feature extraction module, a cross-modal association analysis module, a multi-dimensional emotion fusion module, an emotion strategy mapping and generation module, a dual-channel instruction compilation module, and a collaborative execution and interactive response module, wherein: The multi-source perception and feature extraction module is used to collect facial images and voice signals of the target child through the robot, and to perform multi-scale feature extraction on the facial images and voice signals to obtain the facial feature vector and voice emotion vector of the target child. The cross-modal association analysis module is used to construct an association weight matrix for the target child, with facial feature vectors as the vertical dimension and voice emotion vectors as the horizontal dimension. The multi-dimensional emotion fusion module is used to couple facial feature vectors and voice feature vectors in multiple dimensions based on the correlation weight matrix to obtain the composite emotional features of the target child. The Emotional Strategy Mapping and Generation Module is used to strategically map the emotional state of the target child based on complex emotional features, thereby obtaining the robot's emotional interaction strategy. The dual-channel instruction compilation module is used to compile complex emotional features into instructions based on the emotional interaction strategy, so as to obtain the robot's voice content instructions and facial expression instructions. The collaborative execution and interactive response module is used to input voice content commands and facial expression commands into the robot's execution terminal to obtain the robot's emotionally-accompanied interactive response.

[0005] In a preferred embodiment, when the multi-source perception and feature extraction module performs multi-scale feature extraction on the facial images and speech signals of the target child via a robot to obtain the facial feature vector and speech emotion vector of the target child, it is specifically used for: The robot collects facial images and speech signals from the target children. Multi-granular semantic segmentation is performed on facial images to obtain facial feature maps of the target child; The facial feature map is vectorized to obtain the facial feature vector of the target child; The speech signal is filtered in the emotional frequency band to obtain the emotional sub-band signal of the target child; By fusing cross-band information from the emotional subband signals, the acoustic fingerprint sequence of the target child is obtained; The acoustic fingerprint sequence is subjected to feature aggregation encoding to obtain the speech emotion vector of the target child.

[0006] In a preferred embodiment, when the multi-source perception and feature extraction module performs multi-granular semantic segmentation on the facial images of the target child collected by the robot to obtain the facial feature map of the target child, it is specifically used for: Multi-scale edge perception is performed on facial images to obtain their edge features; Based on edge features, edge-constrained segmentation is performed on the facial image to obtain the initial semantic region of the facial image; Semantic consistency clustering is performed on the initial semantic regions to obtain optimized semantic regions of the facial images; The topological structure of the optimized semantic region is constructed to obtain the facial feature map of the target child.

[0007] In a preferred embodiment, when the cross-modal association analysis module constructs an association weight matrix for the target child using facial feature vectors as the vertical dimension and voice emotion vectors as the horizontal dimension, it is specifically used for: Interactive attention alignment is performed on facial feature vectors and voice emotion vectors to obtain the initial association strength distribution of the target child; Cross-modal synergy assessment was performed on the initial association strength distribution to obtain the synergy correction coefficient for the target children; Based on the synergy correction coefficient, the initial association strength distribution is iteratively updated to obtain the optimized association strength distribution of the target children; In a preferred embodiment, when the cross-modal association analysis module performs iterative updates to the initial association strength distribution based on the synergy correction coefficient to obtain the optimized association strength distribution of the target child, it is specifically used for: The initial association strength distribution is standardized and analyzed to obtain the distribution association strength values ​​of the target children; Based on the synergy correction coefficient, the distributed correlation strength value is nonlinearly modulated to obtain the corrected correlation strength value of the target child; The distribution of the corrected association strength values ​​is reconstructed to obtain the normalized association strength distribution of the target children.

[0008] In a preferred embodiment, when the multi-dimensional emotion fusion module performs multi-dimensional coupling of facial feature vectors and voice feature vectors based on a correlation weight matrix to obtain the composite emotional features of the target child, it is specifically used for: Based on the correlation weight matrix, cross-modal projection fusion of facial feature vectors and speech feature vectors is performed to obtain the fusion basis vector of the target child; Tensor synthesis is performed on the fused basis vectors to obtain the structured emotional feature tensor of the target child; Based on the correlation weight matrix, the structured emotion feature tensor is correlated and modulated to obtain the enhanced correlation feature tensor of the target child; By resolving conflicts in the enhanced correlation feature tensor, the internally consistent feature representation of the target child is obtained. By refining the internally consistent feature representations, the complex emotional features of the target children are obtained.

[0009] In a preferred embodiment, the multi-dimensional emotion fusion module, when performing conflict resolution on the enhanced correlation feature tensor to obtain the internally consistent feature representation of the target child, is specifically used for: Conflict detection is performed on the enhanced correlation feature tensor to obtain contradictory feature components of the target child; Confidence calibration is performed on the contradictory feature components to obtain the recalibrated contradictory feature subset of the target child; Arbitrarily fusing the recalibrated contradictory feature subset with the enhanced correlation feature tensor yields a preliminary reconciled feature representation of the target child; The initial harmonic feature representation is validated for consistency, and the validated results are smoothed and optimized to obtain the internally consistent feature representation of the target child.

[0010] In a preferred embodiment, when the emotion strategy mapping and generation module performs strategic mapping of the target child's emotional state based on composite emotion features to obtain the robot's emotional interaction strategy, it is specifically used for: By performing strategy retrieval on complex emotional features, candidate interaction strategies for the robot can be obtained. Based on complex emotional features, candidate interaction strategies are filtered in context to obtain the robot's appropriate interaction strategy. The parameters of the adaptive interaction strategy are fine-tuned to obtain a draft of the robot's interaction strategy; The draft strategy was pre-tested and evaluated to obtain the robot's emotional interaction strategy.

[0011] In a preferred embodiment, when the dual-channel instruction compilation module executes instruction-based compilation of composite emotional features according to an emotion-interaction strategy to obtain the robot's voice content instructions and facial expression action instructions, it is specifically used for: Based on the emotional interaction strategy, feature decoupling of complex emotional features is performed to obtain the verbal intention descriptor and performance behavior descriptor of the target child; Based on the speech intent descriptor, the speech parameters of the complex emotional features are encapsulated to obtain the initial speech content instructions of the target child. Based on behavioral descriptors, feature encoding is performed on complex emotional features to obtain the basic motor instruction sequence of the target child; Spatiotemporal collaborative optimization is performed on the initial voice content command and basic action command sequence to obtain synchronized interaction commands for the robot; The synchronized interaction commands are rendered with high expressive detail to obtain the robot's voice content commands and facial expression commands.

[0012] In a preferred embodiment, when the collaborative execution and interactive response module inputs voice content commands and facial expression commands to the robot's execution terminal and obtains the robot's emotionally-accompanied interactive response, it is specifically used for: Dual-mode feature demodulation is performed on voice content commands and facial expression commands to obtain the robot's underlying drive signals; Based on the underlying drive signal set, servo cooperative control is performed on the robot to obtain the robot's initial physical response; The initial physical response is analyzed and judged to obtain the robot's real-time interaction status; Based on the real-time interaction status, the robot is adjusted to adapt its emotions, resulting in an emotionally-accompanied interactive response.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. This system achieves accurate multi-scale feature extraction of facial images and speech signals through a multi-source perception and feature extraction module. Combined with a cross-modal association analysis module, it constructs an association weight matrix to efficiently mine the intrinsic correlation between different modal data. Then, through a multi-dimensional emotion fusion module, it completes multi-dimensional coupling and conflict resolution, which significantly improves the comprehensiveness, accuracy and reliability of children's emotion recognition and can accurately capture children's complex emotional states.

[0014] 2. This system achieves precise adaptation and optimization of emotional interaction strategies by relying on the emotional strategy mapping and generation module. It completes the collaborative compilation of voice content and facial expression commands through the dual-channel instruction compilation module. Combined with the dynamic adjustment mechanism of the collaborative execution and interaction response module, the robot's interactive response is more in line with children's real-time emotional needs, enhancing the naturalness and adaptability of the interaction and giving full play to the emotional companionship value of the social companion robot. Attached Figure Description

[0015] Figure 1 This is a system architecture diagram of a child social companion robot interaction system based on emotion recognition, provided in an embodiment of the present invention.

[0016] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments belong to some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “said” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0019] Depending on the context, the word "if" or "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0020] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0021] In practice, the server-side equipment deployed in an emotion-recognition-based children's social companion robot interaction system may consist of one or more devices. This emotion-recognition-based children's social companion robot interaction system can be implemented as: a business instance, a virtual machine, or hardware devices. For example, this emotion-recognition-based children's social companion robot interaction system can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, this emotion-recognition-based children's social companion robot interaction system can be understood as software deployed on a cloud node, used to provide emotion-recognition-based children's social companion robot interaction system to various user terminals. Alternatively, this emotion-recognition-based children's social companion robot interaction system can also be implemented as a virtual machine deployed on one or more devices in a cloud node. This virtual machine contains application software for managing various user terminals. Alternatively, this emotion-recognition-based children's social companion robot interaction system can also be implemented as a server composed of numerous identical or different types of hardware devices, with one or more hardware devices configured to provide emotion-recognition-based children's social companion robot interaction system to various user terminals.

[0022] In terms of implementation, the emotion-recognition-based children's social companion robot interaction system and the user terminal are mutually adaptable. That is, if the emotion-recognition-based children's social companion robot interaction system is implemented as an application installed on a cloud service platform, then the user terminal is implemented as a client that establishes a communication connection with the application; or if the emotion-recognition-based children's social companion robot interaction system is implemented as a website, then the user terminal is implemented as a webpage; or if the emotion-recognition-based children's social companion robot interaction system is implemented as a cloud service platform, then the user terminal is implemented as a mini-program in an instant messaging application.

[0023] like Figure 1 The diagram shown is a system architecture diagram of a child social companion robot interaction system based on emotion recognition provided in an embodiment of the present invention.

[0024] The emotion-recognition-based children's social companion robot interaction system 100 described in this invention can be set up in a cloud server. In terms of implementation, it can be used as one or more service devices, or as an application installed in the cloud (e.g., a mobile service operator's server, server cluster, etc.), or it can be developed as a website. Depending on the functions implemented, the emotion-recognition-based children's social companion robot interaction system 100 may include a multi-source perception and feature extraction module 101, a cross-modal association analysis module 102, a multi-dimensional emotion fusion module 103, an emotion strategy mapping and generation module 104, a dual-channel instruction compilation module 105, and a collaborative execution and interactive response module 106. The modules described in this invention can also be called units, referring to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0025] In this embodiment of the invention, in the emotion-recognition-based children's social companion robot interaction system, each of the above modules can be implemented independently and can call other modules. Here, "calling" can be understood as one module connecting to multiple modules of another type and providing corresponding services to those connected modules. In the emotion-recognition-based children's social companion robot interaction system provided by this embodiment of the invention, the applicability of the system architecture can be adjusted by adding modules and directly calling them without modifying the program code, achieving cluster-based horizontal expansion to quickly and flexibly expand the emotion-recognition-based children's social companion robot interaction system. In practical applications, the above modules can be set in the same device or different devices, or they can be set in virtual devices, such as service instances in a cloud server.

[0026] The following describes the components and workflow of an emotion-recognition-based children's social companion robot interaction system, using specific embodiments as examples: The multi-source perception and feature extraction module 101 is used to collect facial images and speech signals of the target child through the robot, and to perform multi-scale feature extraction on the facial images and speech signals to obtain the facial feature vector and speech emotion vector of the target child. In this embodiment of the invention, when the multi-source perception and feature extraction module performs multi-scale feature extraction on the facial images and speech signals of the target child via a robot to obtain the facial feature vector and speech emotion vector of the target child, it is specifically used for: The robot collects facial images and speech signals from the target children. Multi-granular semantic segmentation is performed on facial images to obtain facial feature maps of the target child; The facial feature map is vectorized to obtain the facial feature vector of the target child; The speech signal is filtered in the emotional frequency band to obtain the emotional sub-band signal of the target child; By fusing cross-band information from the emotional subband signals, the acoustic fingerprint sequence of the target child is obtained; The acoustic fingerprint sequence is subjected to feature aggregation encoding to obtain the speech emotion vector of the target child.

[0027] When the multi-source perception and feature extraction module performs multi-granular semantic segmentation on the facial images of the target child collected by the robot to obtain the facial feature map of the target child, it is specifically used for: Multi-scale edge perception is performed on facial images to obtain their edge features; Based on edge features, edge-constrained segmentation is performed on the facial image to obtain the initial semantic region of the facial image; Semantic consistency clustering is performed on the initial semantic regions to obtain optimized semantic regions of the facial images; The topological structure of the optimized semantic region is constructed to obtain the facial feature map of the target child.

[0028] The robot's image acquisition device continuously captures facial images of the target child, while the audio acquisition device simultaneously records the target child's voice signal, ensuring that the facial images clearly present the child's facial contours, facial features, and other visual information, and that the voice signal fully preserves the child's tone of voice and other audio information.

[0029] The acquired facial images are processed using a multi-scale edge perception method. By gradually refining the perception scale, the edge contour information of different parts of the facial image is fully captured, including the facial features and the overall facial contour, ultimately forming complete facial image edge features.

[0030] Based on the obtained edge features, the facial image is segmented by edge constraints. According to the boundaries defined by the edge features, the facial image is divided into multiple independent initial semantic regions, each of which corresponds to a specific part or region of the face.

[0031] Semantic consistency clustering is performed on the initial semantic regions obtained by segmentation. Initial semantic regions with the same or similar semantic attributes are grouped into one category, and semantically conflicting or irrelevant regions are removed to obtain optimized semantic regions with clear boundaries and explicit semantics.

[0032] The optimized semantic regions are constructed according to the inherent relationship of facial physiological structure, and the positional relationship and connection mode of each optimized semantic region in the whole face are clarified, so as to form a facial feature map that can completely reflect the distribution of facial features of children.

[0033] By processing the facial feature map through vectorization mapping, the features and positional relationships of each region in the facial feature map are transformed into a standardized vector form. This vector contains key information about the facial features, which is the facial feature vector of the target child.

[0034] The collected speech signals are processed by emotional frequency band filtering to select specific frequency band signals related to emotional expression, and to filter out interference components such as environmental noise and irrelevant audio, so as to obtain emotional sub-band signals that can reflect the emotional state of children.

[0035] By fusing cross-band information from multiple emotional sub-band signals, integrating emotionally relevant information contained in signals from different frequency bands, eliminating information redundancy in each sub-band signal, and forming an acoustic fingerprint sequence that can comprehensively reflect the emotional characteristics of children's speech.

[0036] The acoustic fingerprint sequence is subjected to feature aggregation encoding. The core emotional features in the acoustic fingerprint sequence are extracted and integrated and encoded. The scattered feature information is transformed into a unified vector form, and finally a speech emotion vector that can accurately represent the emotional state of children's speech is obtained.

[0037] The cross-modal association analysis module 102 is used to construct an association weight matrix for the target child, with facial feature vectors as the vertical dimension and voice emotion vectors as the horizontal dimension. In this embodiment of the invention, when the cross-modal association analysis module constructs an association weight matrix for the target child using facial feature vectors as the vertical dimension and voice emotion vectors as the horizontal dimension, it is specifically used for: Interactive attention alignment is performed on facial feature vectors and voice emotion vectors to obtain the initial association strength distribution of the target child; Cross-modal synergy assessment was performed on the initial association strength distribution to obtain the synergy correction coefficient for the target children; Based on the synergy correction coefficient, the initial association strength distribution is iteratively updated to obtain the optimized association strength distribution of the target children; The optimized association strength distribution is restructured into a matrix to obtain the association weight matrix of the target child.

[0038] The cross-modal association analysis module, when iteratively updating the initial association strength distribution based on the synergy correction coefficient to obtain the optimized association strength distribution of the target child, is specifically used for: The initial association strength distribution is standardized and analyzed to obtain the distribution association strength values ​​of the target children; Based on the synergy correction coefficient, the distributed correlation strength value is nonlinearly modulated to obtain the corrected correlation strength value of the target child; The distribution of the corrected association strength values ​​is reconstructed to obtain the normalized association strength distribution of the target children.

[0039] Using facial feature vectors as the vertical dimension and voice emotion vectors as the horizontal dimension, this study establishes a corresponding relationship between the vectors by focusing on key information related to emotion expression in both types of vectors. It clarifies the degree of correlation between each facial feature dimension and each voice emotion dimension, forming an initial correlation strength distribution that reflects the initial correlation between the two types of vectors.

[0040] In accordance with the standard requirements for cross-modal data collaborative expression of emotion, a comprehensive evaluation of the initial association strength distribution is conducted. The analysis is performed to determine whether the association strength of different modal dimensions conforms to the internal logic of emotion expression, whether there are any deviations or irrationalities in the association strength distribution, and the collaborative correction coefficient used to correct the association strength is determined based on the evaluation results.

[0041] Standardize all association strength data in the initial association strength distribution to eliminate the dimensional differences between different dimensions, so that the association strength of each dimension is within a unified reference range, thus obtaining the standardized distribution association strength value.

[0042] By incorporating a synergy correction coefficient, the correlation strength value for each distribution is adjusted accordingly. Based on the magnitude and direction of the correction coefficient, a non-linear adjustment is made to strengthen or weaken the correlation strength value, making the adjusted correlation strength value more closely reflect the actual correlation between the two types of vectors, thus obtaining the corrected correlation strength value. The formula for calculating the corrected correlation strength value is as follows: ; in, Indicates the first Corrected correlation strength values ​​in each dimension Indicates the first The distribution of correlation strength values ​​in each dimension Represents the synergy correction coefficient. The preset adjustment factor, It is a natural constant; The distribution association strength value comes from the result of standardized analysis of the initial association strength distribution. The synergy correction coefficient comes from the result of cross-modal synergy evaluation of the initial association strength distribution. The preset adjustment factor is a fixed value set in advance based on the actual application scenarios and needs of children's emotion recognition before system deployment. The natural constant is a fixed constant in the field of mathematics.

[0043] The significance of this formula is that by combining the synergy correction coefficient, the preset adjustment factor, and the natural constant, the distribution correlation strength value is adjusted, thereby obtaining a corrected correlation strength value that better reflects the actual correlation between facial feature vectors and speech emotion vectors, making the representation of correlation strength more accurate.

[0044] As the distributed correlation strength value increases, the corrected correlation strength value will gradually increase, but the growth rate will gradually slow down, eventually approaching a stable value. When the synergy correction coefficient increases, the corrected correlation strength value will approach the distributed correlation strength value. When the preset adjustment factor increases, the growth rate of the corrected correlation strength value will further slow down, and the overall value will always remain within a reasonable range, ensuring the rationality and effectiveness of the correlation strength adjustment.

[0045] All corrected correlation strength values ​​are rearranged and combined according to the dimensional structure of the initial correlation strength distribution, while keeping the dimensional correspondence unchanged. This results in a new, regular distribution of the adjusted correlation strength values, which is the normalized correlation strength distribution. This distribution is the optimized correlation strength distribution.

[0046] According to the structural requirements of the matrix, all the values ​​in the optimized association strength distribution are filled into the corresponding positions of the matrix in an orderly manner. The dimension of the facial feature vector is used as the row of the matrix and the dimension of the voice emotion vector is used as the column of the matrix, so that each element in the matrix accurately corresponds to the association strength of the corresponding dimension, and finally the association weight matrix of the target child is formed.

[0047] The multi-dimensional emotion fusion module 103 is used to couple facial feature vectors and voice feature vectors in multiple dimensions based on the correlation weight matrix to obtain the composite emotional features of the target child. In this embodiment of the invention, when the multi-dimensional emotion fusion module performs multi-dimensional coupling of facial feature vectors and voice feature vectors based on a correlation weight matrix to obtain the composite emotional features of the target child, it is specifically used for: Based on the correlation weight matrix, cross-modal projection fusion of facial feature vectors and speech feature vectors is performed to obtain the fusion basis vector of the target child; Tensor synthesis is performed on the fused basis vectors to obtain the structured emotional feature tensor of the target child; Based on the correlation weight matrix, the structured emotion feature tensor is correlated and modulated to obtain the enhanced correlation feature tensor of the target child; By resolving conflicts in the enhanced correlation feature tensor, the internally consistent feature representation of the target child is obtained. By refining the internally consistent feature representations, the complex emotional features of the target children are obtained.

[0048] The multi-dimensional emotion fusion module, when performing conflict resolution on the enhanced correlation feature tensor to obtain the internally consistent feature representation of the target child, is specifically used for: Conflict detection is performed on the enhanced correlation feature tensor to obtain contradictory feature components of the target child; Confidence calibration is performed on the contradictory feature components to obtain the recalibrated contradictory feature subset of the target child; Arbitrarily fusing the recalibrated contradictory feature subset with the enhanced correlation feature tensor yields a preliminary reconciled feature representation of the target child; The initial harmonic feature representation is validated for consistency, and the validated results are smoothed and optimized to obtain the internally consistent feature representation of the target child.

[0049] Based on the correlation strength between different dimensions of facial feature vectors and speech feature vectors represented by each element in the correlation weight matrix, facial feature vectors and speech feature vectors are projected into the same feature space, so that the two types of vectors can be accurately superimposed on the corresponding correlation dimensions, forming a fusion basis vector that can initially fuse the two types of modal features.

[0050] The fusion basis vectors are reorganized according to the preset structural rules, and the feature elements in the vectors are arranged and combined according to the multi-dimensional hierarchical relationship to construct a multi-dimensional data structure with a clear structural hierarchy. This multi-dimensional data structure is the structured sentiment feature tensor.

[0051] Based on the correlation weight matrix, the weight ratio of each feature component in the structured sentiment feature tensor is adjusted to strengthen the expression of feature components with high correlation strength and weaken the influence of feature components with low correlation strength, so that the features in the tensor are more in line with the real correlation relationship between the two modalities, thus obtaining the enhanced correlation feature tensor.

[0052] By comparing the meanings of each feature component in the enhanced correlation feature tensor one by one, we can identify those feature components that are contradictory and have conflicting meanings. These feature components are then extracted separately to form contradictory feature components.

[0053] Based on the original emotional expression of facial and voice feature vectors, the credibility of contradictory feature components is adjusted to increase the credibility of contradictory features that conform to children's true emotional tendencies and decrease the credibility of contradictory features that do not conform to them, thus obtaining a recalibrated subset of contradictory features.

[0054] The recalibrated contradictory feature subset is integrated with the non-contradictory feature components in the enhanced correlation feature tensor. Based on the emotional information reflected by the two types of feature components, the core consistent emotional features are retained, the differences between different features are coordinated, and a preliminary harmonized feature representation is formed.

[0055] By comparing the inherent logic and patterns of children's emotional expression, we can check whether there are logical conflicts among the various feature components in the preliminary harmonized feature representation, ensure that all feature components can uniformly point to the same emotional tendency, correct the minor deviations found in the verification process, make the feature representation smoother and more coherent, and finally obtain an internally consistent feature representation.

[0056] The core emotional information is extracted from the internally consistent feature representation, redundant feature details are removed, and the scattered core features are integrated and condensed to form a composite emotional feature that can accurately and concisely reflect the emotional state of the target child.

[0057] The emotion strategy mapping and generation module 104 is used to perform strategic mapping of the emotional state of the target child based on composite emotional features to obtain the robot's emotional interaction strategy. In this embodiment of the invention, when the emotion strategy mapping and generation module performs strategic mapping of the target child's emotional state based on composite emotion features to obtain the robot's emotional interaction strategy, it is specifically used for: By performing strategy retrieval on complex emotional features, candidate interaction strategies for the robot can be obtained. Based on complex emotional features, candidate interaction strategies are filtered in context to obtain the robot's appropriate interaction strategy. The parameters of the adaptive interaction strategy are fine-tuned to obtain a draft of the robot's interaction strategy; The draft strategy was pre-tested and evaluated to obtain the robot's emotional interaction strategy.

[0058] Based on the target child's emotional state as represented by complex emotional features, a comprehensive match is performed in the preset interaction strategy library to find all interaction strategies associated with the emotional state, and all of these strategies are extracted as candidate interaction strategies for the robot.

[0059] By combining the children's emotional types, intensities, and possible emotional triggering scenarios corresponding to complex emotional characteristics, a context judgment standard is constructed. Each candidate interaction strategy is checked to see if it meets the current emotional context needs of the children. Strategies that do not match the context are eliminated, and strategies that are fully adapted are retained as the robot's adaptive interaction strategies.

[0060] The interaction strategy was refined by making detailed adjustments to each element, clarifying key elements such as the language style, tone and rhythm, action range and execution order, so that each part of the strategy could accurately match the emotional characteristics of the target children. After the adjustments were completed, a draft of the robot's interaction strategy was formed.

[0061] A simulated interactive environment was built, and the draft interaction strategy was incorporated into it. The simulated robot interacted with children in the corresponding emotional state according to the draft. The execution of each step of the interaction and possible feedback were recorded. A comprehensive evaluation was conducted against the preset interaction effect standards. After confirming that there were no problems, the draft was determined as the robot's emotional interaction strategy.

[0062] The dual-channel instruction compilation module 105 is used to compile compound emotional features into instructions based on the emotional interaction strategy, so as to obtain the robot's voice content instructions and facial expression action instructions. In this embodiment of the invention, when the dual-channel instruction compilation module executes instruction-based compilation of composite emotional features according to an emotion interaction strategy to obtain the robot's voice content instructions and facial expression action instructions, it is specifically used for: Based on the emotional interaction strategy, feature decoupling of complex emotional features is performed to obtain the verbal intention descriptor and performance behavior descriptor of the target child; Based on the speech intent descriptor, the speech parameters of the complex emotional features are encapsulated to obtain the initial speech content instructions of the target child. Based on behavioral descriptors, feature encoding is performed on complex emotional features to obtain the basic motor instruction sequence of the target child; Spatiotemporal collaborative optimization is performed on the initial voice content command and basic action command sequence to obtain synchronized interaction commands for the robot; The synchronized interaction commands are rendered with high expressive detail to obtain the robot's voice content commands and facial expression commands.

[0063] Based on the clear interaction direction and goal of the emotional interaction strategy, the different types of information contained in the composite emotional features are broken down, and the relevant features used to guide the robot's language expression and the relevant features used to guide the robot's action display are separated. The former is transformed into a speech intention descriptor that clearly describes the speech purpose and core content that the robot needs to convey, and the latter is transformed into a performance behavior descriptor that clearly describes the external behavior and expression tendency that the robot needs to present.

[0064] Based on speech intent descriptors, key information related to speech expression is extracted from complex emotional features, including the tone characteristics, rhythm, and core semantics of the emotion. This information is then transformed into standardized speech parameters that can be recognized and executed by the robot, and encapsulated into initial speech content instructions containing complete information such as speech content, intonation, and speech rate.

[0065] Referring to the requirements of the behavioral descriptor, we analyze the detailed information related to facial expressions and movements in the complex emotional features, clarify the key points of the robot's facial expression changes and the execution steps of the limb movements, and transform this information into an ordered encoding form, which is then arranged in chronological order to form a basic action instruction sequence containing a series of continuous action instructions.

[0066] By analyzing the execution timeline of the initial voice content command and the completion timeline of the basic action command sequence, and comparing the time points of the two, the start and end times of the action commands are adjusted to ensure that the output of the voice content and the presentation of the corresponding facial expressions are accurately synchronized. At the same time, the connection logic between commands is optimized to avoid stuttering or misalignment, thus forming synchronized interactive commands.

[0067] For the voice portion of the synchronized interaction commands, the intonation, clarity of pronunciation, and emotional intensity are refined to make the voice expression more infectious. For the facial expressions and movements, the subtle changes in facial muscles and the smoothness of body movements are optimized to make the movements more in line with emotional needs. After the above refined processing, the robot's voice content commands and facial expression commands are finally obtained.

[0068] The collaborative execution and interactive response module 106 is used to input voice content commands and facial expression commands to the robot's execution terminal to obtain the robot's emotionally-accompanied interactive response.

[0069] In this embodiment of the invention, when the collaborative execution and interactive response module inputs voice content commands and facial expression commands to the robot's execution terminal to obtain the robot's emotionally-accompanied interactive response, it is specifically used for: Dual-mode feature demodulation is performed on voice content commands and facial expression commands to obtain the robot's underlying drive signals; Based on the underlying drive signal set, servo cooperative control is performed on the robot to obtain the robot's initial physical response; The initial physical response is analyzed and judged to obtain the robot's real-time interaction status; Based on the real-time interaction status, the robot is adjusted to adapt its emotions, resulting in an emotionally-accompanied interactive response.

[0070] The voice content commands and facial expression commands are analyzed separately to extract the core control information used to control the execution terminal. The voice content commands are converted into audio drive signals that drive the audio output device, and the facial expression commands are converted into motion drive signals that drive the limb and facial movement execution components. These audio drive signals and motion drive signals together constitute the robot's underlying drive signals.

[0071] All underlying drive signals are integrated to form a complete set of underlying drive signals. According to the function of the execution component corresponding to each drive signal, the audio output device, limb joint drive component and facial expression execution component of the robot are coordinated and controlled so that each component starts synchronously and performs corresponding actions according to the requirements of the signal instructions. The audio output device plays the corresponding voice content, and the limb and facial components make corresponding facial expressions, forming the robot's initial physical response.

[0072] The robot monitors the execution process of the initial physical response in real time, records key indicators such as the clarity of voice playback, accuracy of speech rate, and the degree and fluency of facial expressions and actions. It compares the voice content instructions and facial expression and action instructions with preset standards, analyzes whether each execution step fully meets the instruction requirements, and determines the robot's current real-time interaction status after comprehensive judgment.

[0073] Based on real-time interactive feedback, if there are deviations between the voice playback or action execution and the command standard, the corresponding underlying drive signals are adjusted accordingly. Audio output parameters are optimized to improve voice expression, and the force and rhythm of the action drive are corrected to ensure that facial expressions and movements accurately match the commands. This ensures that the robot's interactive behavior always matches the emotional needs of the target child, ultimately resulting in an emotionally-accompanied interactive response. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0074] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A child social companion robot interaction system based on emotion recognition, characterized in that, The system includes a multi-source perception and feature extraction module, a cross-modal association analysis module, a multi-dimensional emotion fusion module, an emotion strategy mapping and generation module, a dual-channel instruction compilation module, and a collaborative execution and interactive response module, wherein: The multi-source perception and feature extraction module is used to collect facial images and voice signals of the target child through the robot, and to perform multi-scale feature extraction on the facial images and voice signals to obtain the facial feature vector and voice emotion vector of the target child. The cross-modal association analysis module is used to construct an association weight matrix for the target child, with facial feature vectors as the vertical dimension and voice emotion vectors as the horizontal dimension. The multi-dimensional emotion fusion module is used to couple facial feature vectors and voice feature vectors in multiple dimensions based on the correlation weight matrix to obtain the composite emotional features of the target child. The Emotional Strategy Mapping and Generation Module is used to strategically map the emotional state of the target child based on complex emotional features, thereby obtaining the robot's emotional interaction strategy. The dual-channel instruction compilation module is used to compile complex emotional features into instructions based on the emotional interaction strategy, so as to obtain the robot's voice content instructions and facial expression instructions. The collaborative execution and interactive response module is used to input voice content commands and facial expression commands into the robot's execution terminal to obtain the robot's emotionally-accompanied interactive response.

2. The child social companion robot interaction system based on emotion recognition as described in claim 1, characterized in that, The multi-source perception and feature extraction module, when executing the process of acquiring facial images and speech signals of the target child through a robot, and performing multi-scale feature extraction on the facial images and speech signals to obtain the target child's facial feature vector and speech emotion vector, is specifically used for: The robot collects facial images and voice signals from the target children. Multi-granular semantic segmentation is performed on facial images to obtain facial feature maps of the target child; The facial feature map is vectorized to obtain the facial feature vector of the target child; The speech signal is filtered in the emotional frequency band to obtain the emotional sub-band signal of the target child; By fusing cross-band information from the emotional subband signals, the acoustic fingerprint sequence of the target child is obtained; The acoustic fingerprint sequence is subjected to feature aggregation encoding to obtain the speech emotion vector of the target child.

3. The child social companion robot interaction system based on emotion recognition as described in claim 2, characterized in that, When the multi-source perception and feature extraction module performs multi-granular semantic segmentation on the facial images of the target child collected by the robot to obtain the facial feature map of the target child, it is specifically used for: Multi-scale edge perception is performed on facial images to obtain their edge features; Based on edge features, edge-constrained segmentation is performed on the facial image to obtain the initial semantic region of the facial image; Semantic consistency clustering is performed on the initial semantic regions to obtain optimized semantic regions of the facial images; The topological structure of the optimized semantic region is constructed to obtain the facial feature map of the target child.

4. The child social companion robot interaction system based on emotion recognition as described in claim 1, characterized in that, The cross-modal association analysis module, when constructing a correlation weight matrix for the target child using facial feature vectors as the vertical dimension and voice emotion vectors as the horizontal dimension, is specifically used for: Interactive attention alignment is performed on facial feature vectors and voice emotion vectors to obtain the initial association strength distribution of the target child; Cross-modal synergy assessment was performed on the initial association strength distribution to obtain the synergy correction coefficient for the target children; Based on the synergy correction coefficient, the initial association strength distribution is iteratively updated to obtain the optimized association strength distribution of the target children; The optimized association strength distribution is restructured into a matrix to obtain the association weight matrix of the target child.

5. The child social companion robot interaction system based on emotion recognition as described in claim 4, characterized in that, The cross-modal association analysis module, when iteratively updating the initial association strength distribution based on the synergy correction coefficient to obtain the optimized association strength distribution of the target child, is specifically used for: The initial association strength distribution is standardized and analyzed to obtain the distribution association strength values ​​of the target children; Based on the synergy correction coefficient, the distributed correlation strength value is nonlinearly modulated to obtain the corrected correlation strength value of the target child; The distribution of the corrected association strength values ​​is reconstructed to obtain the normalized association strength distribution of the target children.

6. The child social companion robot interaction system based on emotion recognition as described in claim 1, characterized in that, The multi-dimensional emotion fusion module, when performing multi-dimensional coupling of facial feature vectors and voice feature vectors based on a correlation weight matrix to obtain the composite emotional features of the target child, is specifically used for: Based on the correlation weight matrix, cross-modal projection fusion of facial feature vectors and speech feature vectors is performed to obtain the fusion basis vector of the target child; Tensor synthesis is performed on the fused basis vectors to obtain the structured emotional feature tensor of the target child; Based on the correlation weight matrix, the structured emotion feature tensor is correlated and modulated to obtain the enhanced correlation feature tensor of the target child; By resolving conflicts in the enhanced correlation feature tensor, the internally consistent feature representation of the target child is obtained. By refining the internally consistent feature representations, the complex emotional features of the target children are obtained.

7. The child social companion robot interaction system based on emotion recognition as described in claim 6, characterized in that, The multi-dimensional emotion fusion module, when performing conflict resolution on the enhanced correlation feature tensor to obtain the internally consistent feature representation of the target child, is specifically used for: Conflict detection is performed on the enhanced correlation feature tensor to obtain contradictory feature components of the target child; Confidence calibration is performed on the contradictory feature components to obtain the recalibrated contradictory feature subset of the target child; Arbitrarily fusing the recalibrated contradictory feature subset with the enhanced correlation feature tensor yields a preliminary reconciled feature representation of the target child; The initial harmonic feature representation is validated for consistency, and the validated results are smoothed and optimized to obtain the internally consistent feature representation of the target child.

8. The child social companion robot interaction system based on emotion recognition as described in claim 1, characterized in that, The emotion strategy mapping and generation module, when performing strategic mapping of the target child's emotional state based on composite emotion features to obtain the robot's emotion interaction strategy, is specifically used for: By performing strategy retrieval on complex emotional features, candidate interaction strategies for the robot can be obtained. Based on complex emotional features, candidate interaction strategies are filtered in context to obtain the robot's appropriate interaction strategy. The parameters of the adaptive interaction strategy are fine-tuned to obtain a draft of the robot's interaction strategy; The draft strategy was pre-tested and evaluated to obtain the robot's emotional interaction strategy.

9. The child social companion robot interaction system based on emotion recognition as described in claim 1, characterized in that, The dual-channel instruction compilation module, when executing the instruction-based compilation of composite emotional features based on the emotion interaction strategy to obtain the robot's voice content instructions and facial expression action instructions, is specifically used for: Based on the emotional interaction strategy, feature decoupling of complex emotional features is performed to obtain the verbal intention descriptor and performance behavior descriptor of the target child; Based on the speech intent descriptor, the speech parameters of the complex emotional features are encapsulated to obtain the initial speech content instructions of the target child. Based on behavioral descriptors, feature encoding is performed on complex emotional features to obtain the basic motor instruction sequence of the target child; Spatiotemporal collaborative optimization is performed on the initial voice content command and basic action command sequence to obtain synchronized interaction commands for the robot; The synchronized interaction commands are rendered with high expressive detail to obtain the robot's voice content commands and facial expression commands.

10. The child social companion robot interaction system based on emotion recognition as described in claim 1, characterized in that, The collaborative execution and interactive response module, when inputting voice content commands and facial expression commands to the robot's execution terminal and obtaining the robot's emotionally-accompanied interactive response, is specifically used for: Dual-mode feature demodulation is performed on voice content commands and facial expression commands to obtain the robot's underlying drive signals; Based on the underlying drive signal set, servo cooperative control is performed on the robot to obtain the robot's initial physical response; The initial physical response is analyzed and judged to obtain the robot's real-time interaction status; Based on the real-time interaction status, the robot is adjusted to adapt its emotions, resulting in an emotionally-accompanied interactive response.

Images