Generative AI-based emotion connection interaction method, system and equipment

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses a generative AI-based emotion connection interaction method, system and equipment, and the method comprises the steps: converting obtained voice information into a voice text, carrying out word segmentation processing, mapping lexical elements to a continuous vector space, obtaining a text embedding vector, and carrying out text embedding on the text embedding vector; extracting an acoustic embedded vector from the voice information by using an acoustic feature extraction technology, and determining sentiment classification of the voice information by using the text embedded vector and the acoustic embedded vector through a convolutional neural network based on an attention mechanism; performing natural language processing on the data processed by the word segmentation algorithm based on the dictionary, and determining a topic field of the voice information; according to sentiment classification, topic fields, user information and historical interaction data, user sentiment feedback information is generated, effective fusion of texts and acoustics at a feature level is realized, and performance reduction caused by possible non-complementarity of classification results obtained by different processing frameworks in a later fusion method is avoided.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to an emotional connection interaction method, system and device based on generative AI. Background Art

[0002] In the current digital era, generative AI, as a cutting-edge technology, has demonstrated its ability to learn from large-scale data sets and create novel and original content. However, although traditional generative AI technology has made significant progress, its interaction mode still has limitations. Most existing systems follow the "question-and-answer" paradigm, requiring users to clearly raise questions or requirements before they can give corresponding feedback. This passive response mechanism limits the initiative and interactivity of AI in conversations.

[0003] In addition, current generative AI mainly relies on text analysis technology, which ignores the voice features that are indispensable for understanding and perceiving users' emotional states and psychological needs when processing language content.

[0004] Therefore, there is an urgent need for a generative AI emotional connection interaction method and device that can provide convenient emotional comfort to adults who experience loneliness, stress or seek emotional recognition in daily life, listen to and respond to users' voices, and fill users' emotional gaps. Summary of the invention

[0005] In view of the technical problems existing in the above-mentioned prior art, the present invention provides an emotional connection interaction method, system and device based on generative AI.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides an emotional connection interaction method based on generative AI, the method comprising the following steps:

[0008] Step S1, collecting user information, setting the human-computer interaction mode, and receiving the user's voice information through the emotional interaction device;

[0009] Step S2, constructing a sentiment analysis model for human-computer interaction, taking speech information as input of the sentiment analysis model for human-computer interaction, using a speech recognition engine to convert speech information into speech text, using a dictionary-based word segmentation algorithm to perform word segmentation processing on the speech text, mapping the data processed by the dictionary-based word segmentation algorithm to a continuous vector space, capturing the semantic relationship between speech text, obtaining a text embedding vector, and using acoustic feature extraction technology to extract an acoustic embedding vector from the speech information; using the text embedding vector and the acoustic embedding vector through a convolutional neural network based on an attention mechanism to determine the sentiment classification of the speech information;

[0010] Step S3, performing natural language processing on the data processed by the word segmentation algorithm based on the dictionary to determine the topic area of ​​the voice information;

[0011] Step S4, using the large language model, obtaining the sentiment feedback embedding vector according to the text embedding vector, sentiment classification, and topic field;

[0012] Step S5, using natural language generation technology, generating user emotional feedback information according to the emotional feedback embedding vector, user information and historical interaction data.

[0013] Preferably, the word segmentation process of the speech text is implemented by using a dictionary-based word segmentation algorithm, specifically: integrating a two-way maximum match and a word segmentation strategy based on deep understanding to dynamically and accurately determine the optimal word segmentation result.

[0014] Preferably, the specific steps of extracting the acoustic embedding vector are as follows: first, using the acoustic feature extraction technology to extract the speech fundamental frequency, zero-crossing rate rhythm features, Mel-frequency cepstral coefficients, spectrum centroid-based features, frequency perturbation, amplitude perturbation, spectrum bandwidth, spectrum roll-off point, root mean square energy sound quality features, and perform feature splicing on the original form of the root mean square energy mean, standard deviation, maximum value, spectrum roll-off point and spectrum bandwidth flattened into a one-dimensional array, and other features to obtain an acoustic feature vector, and then use a deep neural network to convert the acoustic feature vector into an acoustic embedding vector, wherein the deep neural network includes three fully connected layers and two random dropout layers.

[0015] Preferably, before extracting the acoustic embedding vector, first, the short-time energy and zero-crossing rate of the speech information are extracted, and then a double-threshold endpoint detection method is used to combine the short-time energy and the zero-crossing rate to confirm the valid speech segment.

[0016] Preferably, in step S2, the text embedding vector and the acoustic embedding vector are input into a convolutional neural network based on an attention mechanism to determine the sentiment classification of the speech information, specifically:

[0017] The text embedding vector and the acoustic embedding vector are input into the attention fusion to obtain the fused embedding representation :

[0018] (1)

[0019] in, represents the acoustic embedding vector, represents the text embedding vector, represents element-wise multiplication, Representation and feature weight matrix, represents the product feature weight matrix;

[0020] Embedding the fusion into Enter the convolutional neural network. The convolutional neural network consists of three convolutional layers, three rectified linear unit ReLU layers and one fully connected layer FC:

[0021] (2)

[0022] (3)

[0023] F i = R e s h a p e ( R i , [ − 1 , ( N − f i + 1 ) × n u m f i l t e r s ] ) (4)

[0024] H = C o n c a t ( [ F 1 , F 2 , … , F i ] , a x i s = 1 ) (5)

[0025] (6)

[0026] in, Represents the output of each convolutional layer, represents the number of filters, represents the dimension of the filter, represents the filter dimension corresponding to each convolutional layer, and D represents the fused embedding representation The dimension of They are 1, 2, and 3 respectively. Represents the output of each rectified linear unit ReLU layer, Represents the result of flattening the output of each rectified linear unit ReLU layer, Indicates changing the shape of the activation layer output tensor, Indicates the range of word units considered when the convolution kernel slides, Indicates concatenating multiple tensors along the specified dimension. Indicates the number of dimensions, H indicates the result of concatenating the flattened output. represents the fully connected layer, Indicates the number of tags.

[0027] Preferably, the specific steps of step S3 are as follows: construct a bag-of-words model using word units, and input the bag-of-words model into a latent Dirichlet distribution topic model to obtain the topic domain of the speech information.

[0028] In a second aspect, an emotional connection interaction system based on generative AI is provided, wherein the system includes a speech input module, a speech recognition and processing module, an emotion analysis module, an adaptive emotion module, a human-computer interaction interface module, and a communication module.

[0029] The voice input module is used to obtain the user's voice information.

[0030] The speech recognition and processing module extracts the acoustic features of speech information to obtain the acoustic embedding vector.

[0031] The sentiment analysis module uses the speech recognition engine to convert speech information into speech text, and uses a dictionary-based word segmentation algorithm to perform word segmentation on the speech text. Then, it maps word units to a continuous vector space to capture the semantic relationship between speech and text, obtains text embedding vectors, extracts text features of speech information to obtain text embedding vectors, and fuses acoustic embedding vectors and text embedding vectors through a convolutional neural network based on an attention mechanism to determine the sentiment classification of speech information.

[0032] The adaptive emotion module is responsible for collecting, organizing and storing user interaction information, performing natural language processing on words, and determining the topic area. It uses a large language model to obtain the emotion embedding vector based on the text embedding vector, emotion classification, and topic area. It uses natural language generation technology to generate user emotion feedback information based on the emotion feedback embedding vector, user information, and historical interaction data to identify the potential rules and trends of user emotion changes in different situations.

[0033] The human-computer interaction interface module obtains user registration information.

[0034] The communication module is responsible for data transmission and communication with the user's device and feedback of user emotional feedback information.

[0035] In a third aspect, the present invention provides an emotional connection interaction device based on generative AI, including a processor module, an audio receiving and sending module, a communication module, a power module and a memory module.

[0036] The processor module includes CPU and NPU. The CPU is responsible for processing various computing tasks including voice information; the NPU is responsible for optimizing and accelerating computing tasks related to neural networks.

[0037] The audio receiving and sending module is responsible for receiving the user's voice information and sending the user's emotional feedback information.

[0038] Data transmission and communication between the communication module and the user equipment,

[0039] The power module is responsible for providing the power required for the operation of the equipment.

[0040] The memory module is responsible for storing user data and implementing data interaction with applications and operating systems.

[0041] The present invention has the following beneficial effects: the present invention extracts three types of acoustic features including rhythmic features, spectral features and sound quality features, significantly enhancing the recognition of the features, thereby being able to more accurately determine the current emotional state of the user. In addition, the present invention adopts a method based on early fusion, giving full play to the advantages of speech in expressing the intensity of emotions and the accuracy of text in judging the positive and negative emotions, achieving effective fusion of the two at the feature level, and avoiding the performance degradation caused by the possible non-complementarity of the classification results obtained by different processing frameworks in the late fusion method.

[0042] The present invention has highly intelligent interactive capabilities, integrating information such as emotional tendencies, topic areas and texts, and guiding the large model to generate more targeted responses based on the user's speech. It does not rely on the raising of preset questions, and can provide instant feedback channels at critical moments of emotional communication, thereby improving the fluency and naturalness of user interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of the emotional connection interaction method based on generative AI;

[0044] Figure 2 A schematic diagram of the sentiment analysis model for human-computer interaction;

[0045] Figure 3 This is the overall flow chart of the emotional connection interaction system based on generative AI;

[0046] Figure 4 This is the internal schematic diagram of the emotional connection interaction device based on generative AI;

[0047] Figure 5 A morphological diagram of an emotionally connected interactive device based on generative AI;

[0048] Figure 6 This is a diagram of the operation results of the model of the present invention;

[0049] Figure 7 This is the sentiment classification result diagram of the present invention on the CASIA dataset;

[0050] Figure 8 A comparison chart of the model accuracy of the present invention, the Time Distributed Convolutional Neural Network (TDCNN) model, and the 2D-CNN model under the training set;

[0051] Fig. 9 This is a comparison chart of the model accuracy of the present invention, the Time Distributed Convolutional Neural Network (TDCNN) model, and the 2D-CNN model under the test set. DETAILED DESCRIPTION

[0052] The present invention is further described and defined by the following examples but is not limited thereto. The experimental methods in the following examples are conventional methods unless otherwise specified. The present invention is further described below in conjunction with the accompanying drawings.

[0053] like Figure 1 As shown, the emotional connection interaction method based on generative AI includes the following steps:

[0054] Step S1, obtain user registration information, collect user registration information through the user interface, including but not limited to name, gender, age, occupation, hobbies and social identity and other basic personal information, this information is not only used for user identity authentication, but also for subsequent personalized sentiment analysis, setting the human-computer interaction mode, and determining the interaction paradigm and interaction medium. Specifically, the interaction paradigm covers two options: one is to adopt the traditional question-and-answer mode, that is, the generative AI responds to questions clearly raised by the user; the other is to implement an automated feedback mechanism for voice input, allowing users to interact directly with the system through unstructured voice. Regarding the selection of interaction media, two output methods, text and voice, are provided to adapt to different user interaction preferences and needs; the user's voice information is received through the emotional interaction device;

[0055] Step S2, construct a sentiment analysis model for human-computer interaction, such as Figure 2 As shown, the speech information is used as the input of the model. This embodiment uses the public data set CASIA to obtain Chinese speech information. First, the text embedding vector and the acoustic embedding vector are extracted respectively. Then, the convolutional neural network based on the attention mechanism fuses the text embedding vector and the acoustic embedding vector to determine the emotional classification of the speech information.

[0056] S21, extract text embedding vector;

[0057] S211, using a speech recognition engine, in this embodiment, using the SpeechRecognition library, converting the speech information into text;

[0058] S212, using a dictionary-based word segmentation algorithm to perform word segmentation processing on the text, specifically, integrating a bidirectional maximum match and a word segmentation strategy based on deep understanding to dynamically and accurately determine the optimal word segmentation result;

[0059] S213, extracting the data processed by the word segmentation algorithm based on the dictionary using text feature extraction to obtain a text embedding vector. This embodiment uses the Word2Vec model to map the data processed by the word segmentation algorithm based on the dictionary into a continuous vector space, capture the semantic relationship between texts, and realize text feature extraction;

[0060] S22, extract acoustic embedding vector;

[0061] S221, first, extract the short-time energy and zero-crossing rate of the speech information, then use the double-threshold endpoint detection method to traverse each frame first, if the energy value of a frame is higher than the high threshold value, preliminarily judge that the frame is part of the speech signal, and mark it as a possible starting point, then traverse the frames backward, when the energy value of the frame is continuously higher than the low threshold value, and the zero-crossing rate of the unvoiced segment is high, the zero-crossing rate of the voiced segment is low but the short-time energy is high, it is confirmed as a valid speech segment;

[0062] S222, using librosa acoustic feature extraction technology, extracting rhythmic features such as speech fundamental frequency and zero-crossing rate, spectrum-based features such as Mel-frequency cepstral coefficients and spectrum centroid, and sound quality features such as frequency perturbation, amplitude perturbation, spectrum bandwidth, spectrum roll-off point, and root mean square energy in turn, and performing feature concatenation on the mean, standard deviation, and maximum value of the root mean square energy, the spectrum roll-off point and spectrum bandwidth flattened into a one-dimensional array, and the original forms of other features to obtain an acoustic feature vector;

[0063] S223, designed a deep neural network consisting of three fully connected (Dense) layers and two random dropout layers. Among them, the fully connected layer is responsible for the nonlinear combination of features and learns the complex relationships and patterns of different types of acoustic features; the Dropout layer randomly sets the output of several neurons in the fully connected layer to zero to improve the generalization performance of the model in unknown speech, and uses this deep neural network to convert the acoustic feature vector into an acoustic embedding vector.

[0064] Step S23, using a convolutional neural network based on an attention mechanism to determine the emotional classification of the speech information, and using the attention mechanism to perform feature fusion on the acoustic embedding vector and the text embedding vector to obtain a fused embedding representation , as shown in formula (1):

[0065] (1)

[0066] in, represents the acoustic embedding vector, represents the text embedding vector, represents element-wise multiplication, represents the trainable and feature weight matrix, represents the trainable product feature weight matrix;

[0067] Embedding the fusion into The input is sent to the convolutional neural network to realize the recognition of user emotion classification. The convolutional neural network contains three convolutional layers, three rectified linear unit ReLU layers and one fully connected layer (FC).

[0068] (2)

[0069] As shown in formula (2), the convolutional layer effectively captures the correlation between local and global features, as well as the interaction between different features, and extracts high-level features from user emotion signals by applying different numbers of filters, where: represents the number of filters, that is, the number of features to be learned. is 64; Represents the dimension of the filter. In this embodiment, there are three convolutional layers to learn different types of features in the input data. The values ​​of are 1, 2, and 3 respectively, and the filter dimension of each convolutional layer is They are 2, 3, and 4 respectively. is 256;

[0070] (3)

[0071] As shown in formula (3), the rectified linear unit ReLU layer enhances the network's expressiveness by introducing nonlinear transformations, learning and simulating the complex nonlinear relationship between input data and target output, promoting rapid convergence of the network during training, and further improving the efficiency and accuracy of emotion recognition.

[0072] F i = R e s h a p e ( R i , [ − 1 , ( N − f i + 1 ) × n u m f i l t e r s ] ) (4)

[0073] H = C o n c a t ( [ F 1 , F 2 , … , F i ] , a x i s = 1 ) (5)

[0074] Formulas (4) and (5) respectively represent flattening the output of the rectified linear unit ReLU layer and concatenating the flattened outputs, where: Indicates the range of word units considered when the convolution kernel slides, which is set to 300 in the simulation experiment;

[0075] (6)

[0076] As shown in formula (6), the fully connected layer linearly transforms the input feature vector to obtain a new vector ,in, Refers to the number of labels. In this embodiment, the CASIA dataset used has 6 emotion categories, so That is, the category is 6, Each element of corresponds to the score of a category, indicating the possibility that the input sample belongs to this category;

[0077] Furthermore, based on user emotion data, the deep neural network in S223 and the convolutional neural network based on the attention mechanism in S123 are continuously trained and optimized, and various parameters with the highest emotion recognition accuracy are obtained through iteration of training times to improve the accuracy of emotion recognition.

[0078] Step S3, performing natural language processing on each word unit obtained by the word segmentation algorithm based on the dictionary, determining the topic area of ​​the speech information, and constructing the word unit into a bag-of-words model, wherein each word unit is used as a feature of the bag-of-words model, and its corresponding value is the frequency of occurrence of the word unit in the text; the bag-of-words model is used as input data of the LDA (latent Dirichlet allocation) topic model for model training, so as to enhance the ability to capture complex and abstract topics, and use the trained LDA (latent Dirichlet allocation) topic model to achieve accurate topic area division;

[0079] Step S4, using a large language model, converting sentiment classification and topic domain into embedding vectors in space by feature extraction, and integrating the text embedding vector of speech, sentiment classification embedding vector, and topic domain embedding vector into a sentiment embedding vector by using data fusion technology and splicing technology; in this embodiment, the large language model uses Wenxinyiyan;

[0080] Step S5, using natural language generation (NLG) technology, according to the input sentiment embedding vector and user information and historical interaction data, adopting decoding strategies such as beam search and random search to ensure the diversity and quality of the generated text. At the same time, adjusting the temperature parameter in the generation strategy to control the randomness and innovation of the generated text, adjusting the length penalty to balance the simplicity and completeness of the text, and generating personalized user sentiment feedback information.

[0081] Step S6, displaying the emotional feedback information. When the voice interaction medium is selected, the emotional interaction device is used to directly realize the voice output and emotional interaction with the user; when the text interaction medium is selected, the emotional interaction device establishes a communication connection with the user device through a communication method such as Bluetooth, and emotionally interacts with the user through the user device application.

[0082] like Figure 3 As shown, an emotional connection interaction system based on generative AI is provided, including: a voice input module, a voice recognition and processing module, an emotional analysis module, an adaptive emotional module, a human-computer interaction interface module, and a communication module, wherein:

[0083] Voice input module: This module is the entrance to the entire generative AI-based emotional connection interaction system. It has built-in noise suppression and echo cancellation mechanisms to effectively filter out environmental noise, ensuring high-quality voice signals even in complex environments and accurately capturing user voices.

[0084] Speech recognition and processing module: This module extracts and analyzes the acoustic features of speech information, such as rhythm, spectrum, and sound quality features, and uses deep neural networks to convert acoustic features into acoustic embedding vectors that can be processed by the model to assist in subsequent sentiment analysis.

[0085] Sentiment analysis module: This module uses a speech recognition engine to convert speech information into speech text, and uses a dictionary-based word segmentation algorithm to perform word segmentation on the speech text. Then, it maps word units to a continuous vector space to capture the semantic relationship between speech and text, obtains text embedding vectors, extracts text features of speech information to obtain text embedding vectors, and determines the sentiment classification of speech information by fusing acoustic embedding vectors and text embedding vectors through a convolutional neural network based on an attention mechanism.

[0086] Adaptive emotion module: This module is responsible for collecting, organizing and storing all interaction data and emotion information between users and the system. Perform natural language processing on words to determine the topic area; use the large language model to obtain the emotion embedding vector based on the text embedding vector, emotion classification, and topic area; use natural language generation technology to identify the potential rules and trends of user emotion changes in different situations based on the emotion feedback embedding vector, user information, and historical interaction data, and generate user emotion feedback information.

[0087] Human-computer interaction interface module: As a direct interaction window between the system and the user, this module is designed with an intuitive and friendly user interface for registering user information, supporting users to set interaction paradigms and interaction modes; at the same time, it displays user information, interactive information, etc. to enhance user satisfaction.

[0088] Communication module: This module is responsible for data transmission and communication with user devices to ensure the real-time and reliability of information transmission. Bluetooth, Wi-Fi, 4 / 5G, etc. are used as communication modes to ensure the real-time and reliability of information transmission. It can quickly transmit key data such as user's emotional data and system feedback in both directions between the mobile phone and the emotional interaction device, bringing users a smooth and unimpeded interactive experience.

[0089] Furthermore, it also includes a privacy protection module: as a key link in ensuring user privacy and security, this module uses encryption, decryption, verification and other methods to ensure data privacy and security when the processor processes data. At the same time, user identity authentication and permission management are set in the human-computer interaction interface module to prevent unauthorized users or applications from accessing sensitive data.

[0090] The present invention provides an electronic device, the internal structure of which is shown in FIG. Figure 4 As shown, the device form is as Figure 5As shown. The device includes a processor module, an audio receiving and transmitting module, a communication module, a power module and a memory module. Among them, the communication module of the device includes three communication modes: Bluetooth, Wi-Fi, and 4 / 5G; the memory module includes user data, operating system and application.

[0091] The processor module is the "brain" of the device, mainly including the CPU and NPU. Among them, the CPU is mainly responsible for general computing tasks such as preprocessing, encoding and decoding of voice and text data, and performs security operations such as encryption, decryption, and data verification to ensure the privacy of users; while the NPU is mainly responsible for the calculation of neural networks, efficiently performing matrix operations and convolution operations, and optimizing neural network training and reasoning capabilities. Quickly respond to user operations and system requirements to ensure smooth and stable operation of the device.

[0092] The audio receiving and sending module is responsible for capturing the user's voice information and sending the user's emotional feedback information. It uses an embedded DSP chip with noise suppression, echo cancellation and other functions to ensure that the collected audio signal is clear and pure, providing a high-quality input source for speech recognition and speech feature extraction.

[0093] The communication module is responsible for data transmission and communication between user devices. It integrates three communication methods: Bluetooth, Wi-Fi, and 4 / 5G, realizing wide-coverage, high-speed wireless connection between devices and access to the Internet.

[0094] The power module is mainly responsible for providing the power required for the device to operate. According to the working status and power consumption requirements of the device, it intelligently adjusts the power output to extend the battery life of the device and improve the overall energy efficiency.

[0095] The memory module contains three parts: user data, operating system, and application. Among them, user data includes user-generated voice, text content, personal settings, historical interaction data and other information. The device uses encryption technology and permission management strategies to ensure the security and privacy of user data.

[0096] The operating system processes data flows from audio receiving and transmitting modules, communication modules, applications, etc., and coordinates the interactions between them.

[0097] The application mainly includes an emotional interaction method and a large language model, which is responsible for processing audio data and calling a large language model for text generation in the form of "embedded software". Specifically, the emotional interaction method is mainly responsible for processing audio data, emotion recognition, and topic domain generation; the present invention mainly adopts the Wenxinyiyan large model, which has powerful language understanding capabilities through training of massive text data, and supports the generation of personalized user emotional feedback information.

[0098] The emotion recognition performance of the generative AI-based emotional connection interaction method was simulated and tested based on the public CASIA dataset, which covers six common emotion categories: neutral, angry, scared, happy, sad, and surprised.

[0099] like Figure 6 As shown in the figure, it can be seen that on the training set, the accuracy of the model is as high as over 98%; on the test set, the accuracy of the model also reaches over 80%, and the model converges quickly and has high stability.

[0100] Figure 7 The results of speech recognition using speech data representing the emotion of "surprise" in the CASIA dataset are shown. The results are shown in the form of a bar chart showing the confidence probability of each emotion category, where the confidence probability of the emotion of "surprise" is as high as 0.99.

[0101] Figure 8 The accuracy comparison chart of the present invention, the Time Distributed Convolutional Neural Network (TDCNN) model and the 2D-CNN model within 50 iterations under the training set is shown. It can be found that the accuracy of the TDCNN model and the 2D-CNN model during training cannot reach 95%, and the convergence speed is slow; the accuracy of the model of the present invention during training can be as high as over 98%, and the convergence speed is fast.

[0102] Fig. 9 The accuracy comparison chart of the present invention, the Time Distributed Convolutional Neural Network (TDCNN) model and the 2D-CNN model within 50 iterations under the test set is shown. It can be found that the accuracy of the TDCNN model and the 2D-CNN model during the test did not reach 80%, and the performance was very unstable; the accuracy of the model of the present invention during the test can reach more than 80%, and the convergence speed is fast.

[0103] The above description is only a preferred embodiment of the present invention, but is not intended to limit the present invention. Those skilled in the art may make possible changes and modifications to the present invention using the technical contents disclosed above without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. An emotional connection interaction method based on generative AI, characterized in that: The method comprises the following steps: Step S1, collecting user information, setting the human-computer interaction mode, and receiving the user's voice information through the emotional interaction device; Step S2, constructing a sentiment analysis model for human-computer interaction, taking speech information as input of the sentiment analysis model for human-computer interaction, using a speech recognition engine to convert speech information into speech text, using a dictionary-based word segmentation algorithm to perform word segmentation processing on the speech text, mapping the data processed by the dictionary-based word segmentation algorithm to a continuous vector space, capturing the semantic relationship between speech text, obtaining a text embedding vector, using acoustic feature extraction technology to extract an acoustic embedding vector from speech information, and determining the sentiment classification of the speech information by passing the text embedding vector and the acoustic embedding vector through a convolutional neural network based on an attention mechanism; Step S3, performing natural language processing on the data processed by the word segmentation algorithm based on the dictionary to determine the topic area of ​​the voice information; Step S4, using the large language model, obtaining the sentiment feedback embedding vector according to the text embedding vector, sentiment classification, and topic field; Step S5, using natural language generation technology, generating user emotional feedback information according to the emotional feedback embedding vector, user information and historical interaction data.

2. The generative AI-based emotional connection interaction method according to claim 1, characterized in that: The word segmentation process of speech text is implemented by using a dictionary-based word segmentation algorithm. Specifically, it integrates the two-way maximum matching and the word segmentation strategy based on deep understanding to dynamically and accurately determine the optimal word segmentation result.

3. The generative AI-based emotional connection interaction method according to claim 1, characterized in that: The specific steps of extracting the acoustic embedding vector are as follows: First, the acoustic feature extraction technology is used to extract the speech fundamental frequency, zero-crossing rate rhythm features, Mel-frequency cepstral coefficients, spectrum centroid-based features, frequency perturbation, amplitude perturbation, spectrum bandwidth, spectrum roll-off point, root mean square energy sound quality features, and the mean, standard deviation, maximum value, spectrum roll-off point and spectrum bandwidth of the root mean square energy flattened into a one-dimensional array and the original form of other features are feature concatenated to obtain the acoustic feature vector. Then, the deep neural network is used to convert the acoustic feature vector into an acoustic embedding vector, where the deep neural network includes three fully connected layers and two random dropout layers.

4. The generative AI-based emotional connection interaction method according to claim 1, characterized in that: Before extracting the acoustic embedding vector, first, the short-time energy and zero-crossing rate of the speech information are extracted, and then the double-threshold endpoint detection method is used to combine the short-time energy and zero-crossing rate to confirm the valid speech segment.

5. The generative AI-based emotional connection interaction method according to claim 1, characterized in that: In step S2, the text embedding vector and the acoustic embedding vector are used to determine the sentiment classification of the speech information through a convolutional neural network based on an attention mechanism, specifically: The text embedding vector and the acoustic embedding vector are input into the attention fusion to obtain the fused embedding representation : （1） in, represents the acoustic embedding vector, represents the text embedding vector, represents element-wise multiplication, Representation and feature weight matrix, represents the product feature weight matrix; Embedding the fusion into Input convolutional neural network, which includes three convolutional layers, three rectified linear unit ReLU layers and one fully connected layer FC: （2） （3） （4） （5） （6） in, Represents the output of each convolutional layer, represents the number of filters, represents the dimension of the filter, represents the filter dimension corresponding to each convolutional layer, and D represents the fused embedding representation The dimension of They are 1, 2, and 3 respectively. Represents the output of each rectified linear unit ReLU layer, Represents the result of flattening the output of the rectified linear unit ReLU layer. Indicates changing the shape of the activation layer output tensor, Indicates the range of word units considered when the convolution kernel slides, Indicates concatenating multiple tensors along the specified dimension. Indicates the number of dimensions, H indicates the result of concatenating the flattened output. represents the fully connected layer, Indicates the number of tags.

6. The generative AI-based emotional connection interaction method according to claim 1, characterized in that: The specific steps of step S3 are as follows: construct a bag-of-words model using word units, and input the bag-of-words model into a latent Dirichlet distribution topic model to obtain the topic domain of the speech information.

7. The emotional connection interaction system based on generative AI is characterized by: For implementing the method described in any one of claims 1 to 6, the system comprises a speech input module, a speech recognition and processing module, a sentiment analysis module, an adaptive sentiment module, a human-computer interaction interface module, and a communication module. The voice input module is used to obtain the user's voice information. The speech recognition and processing module extracts the acoustic features of speech information to obtain the acoustic embedding vector. The sentiment analysis module uses the speech recognition engine to convert speech information into speech text, and uses a dictionary-based word segmentation algorithm to perform word segmentation on the speech text. Then, it maps word units to a continuous vector space to capture the semantic relationship between speech and text, obtains text embedding vectors, extracts text features of speech information to obtain text embedding vectors, and fuses acoustic embedding vectors and text embedding vectors through a convolutional neural network based on an attention mechanism to determine the sentiment classification of speech information. The adaptive emotion module is responsible for collecting, organizing and storing user interaction information, performing natural language processing on words, and determining the topic area. It uses a large language model to obtain the emotion embedding vector based on the text embedding vector, emotion classification, and topic area. It uses natural language generation technology to generate user emotion feedback information based on the emotion feedback embedding vector, user information, and historical interaction data to identify the potential rules and trends of user emotion changes in different situations. The human-computer interaction interface module obtains user registration information. The communication module is responsible for data transmission and communication with the user's device and feedback of user emotional feedback information.

8. Emotional connection interactive device based on generative AI, characterized by: Used to implement the method according to any one of claims 1 to 6, comprising a processor module, an audio receiving and sending module, a communication module, a power module and a memory module, The processor module includes CPU and NPU. The CPU is responsible for processing various computing tasks including voice information; the NPU is responsible for optimizing and accelerating computing tasks related to neural networks. The audio receiving and sending module is responsible for receiving the user's voice information and sending the user's emotional feedback information. The communication module is responsible for data transmission and communication with user equipment. The power module is responsible for providing the power required for the operation of the equipment. The memory module is responsible for storing user data and implementing data interaction with applications and operating systems.

Images