Intelligent interaction method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an intelligent interaction method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring current dialogue information input by a user, and acquiring image information; performing emotion recognition according to the current dialogue information and the image information to obtain an emotion label of the user, and performing information matching from a preset corpus by using the current dialogue information to obtain initial response information; determining portrait features of the user according to the image information, and screening the initial response information according to the portrait features and the emotion tag to obtain target response information; and obtaining reply information of the current dialogue information according to the target response information, and performing intelligent interaction according to the reply information. According to the invention, the flexibility and intelligence of the intelligent interaction process can be improved, so that the interaction content better meets the personalized requirements of the user, and the interaction experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an intelligent interaction method, device, electronic device and readable storage medium. Background Art

[0002] With the rapid progress of artificial intelligence technology, products in the field of intelligent interaction have shown a significant growth trend, greatly improving the user's convenient experience. With advanced voice and text recognition technology, users can interact with intelligent interactive devices. Although these products have made significant breakthroughs in technology, such as being able to customize personalized responses based on the user's emotional state, in actual applications, single-modal information often has problems with inaccurate emotion recognition when identifying user emotions, and intelligent interactive devices generally use a relatively fixed voice mode, and their interactive language in most cases fails to fully consider the actual needs of users. This mechanized communication method often lacks the natural flow of emotions, making the entire interaction process appear relatively rigid and lacking in flexibility, making it difficult to meet the user's dialogue needs and unable to provide users with a more vivid and personalized interactive experience. Summary of the invention

[0003] In view of this, the embodiments of the present application provide an intelligent interaction method, device, electronic device and readable storage medium to solve the problem that the interaction process in the prior art cannot provide users with a more vivid interaction experience that meets the user's personalized needs.

[0004] A first aspect of an embodiment of the present application provides an intelligent interaction method, the method comprising:

[0005] The method includes obtaining current conversation information input by a user and image information containing the user; performing emotion recognition based on the current conversation information and the image information to obtain an emotion tag of the user, and using the current conversation information to perform information matching from a pre-set corpus to obtain initial response information; determining a portrait feature of the user based on the image information, and filtering the initial response information based on the portrait feature and the emotion tag to obtain target response information; obtaining reply information of the current conversation information based on the target response information, and performing intelligent interaction based on the reply information.

[0006] A second aspect of the embodiments of the present application provides an intelligent interaction device, the method comprising:

[0007] The acquisition module is configured to acquire the current dialogue information input by the user, and acquire the image information containing the user; the recognition module is configured to perform emotion recognition based on the current dialogue information and the image information to obtain the emotion label of the user, and use the current dialogue information to match information from a pre-set corpus to obtain initial response information; the screening module is configured to determine the portrait features of the user based on the image information, and screen the initial response information based on the portrait features and the emotion label to obtain target response information; the interaction module is configured to obtain reply information of the current dialogue information based on the target response information, and perform intelligent interaction based on the reply information.

[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0009] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0010] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0011] The current dialogue information input by the user is obtained, and the image information containing the user is obtained; emotion recognition is performed according to the current dialogue information and the image information to obtain the emotion label of the user, and the current dialogue information is used to match information from a pre-set corpus to obtain initial response information; according to the image information, the portrait features of the user are determined, and the initial response information is screened according to the portrait features and the emotion label to obtain target response information; according to the target response information, the reply information of the current dialogue information is obtained, and intelligent interaction is performed according to the reply information. Through the combination of text dialogue information and image information, the present application can more accurately identify the emotional state of the user, thereby avoiding the limitation of emotion recognition only through text dialogue information or image information. At the same time, after obtaining the initial response information, it is not limited to directly giving the reply information, but further analyzing the applicability of the initial response information through the portrait features and the emotion label, ensuring that the reply content meets the actual needs and emotional state of the user, and improving the flexibility and intelligence of the intelligent interaction process, so that the interactive content is more in line with the personalized needs of the user, thereby improving the user's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 It is a flowchart of an intelligent interaction method provided in an embodiment of the present application;

[0014] Figure 2 is a structural diagram of an intelligent interactive device provided in an embodiment of the present application;

[0015] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0017] An intelligent interaction method, device, electronic device, and readable storage medium according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0018] Figure 1 is a flow chart of an intelligent interaction method provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0019] S101, obtaining current conversation information input by the user, and obtaining image information containing the user;

[0020] S102, performing emotion recognition based on the current dialogue information and image information to obtain the user's emotion label, and using the current dialogue information to perform information matching from a pre-set corpus to obtain initial response information;

[0021] S103, determining the user's portrait features according to the image information, and filtering the initial response information according to the portrait features and the emotion tags to obtain target response information;

[0022] S104, obtaining reply information of the current dialogue information according to the target response information, and performing intelligent interaction according to the reply information.

[0023] It should be noted that the execution subject of this embodiment can be a cloud-based or artificial intelligence-based intelligent interactive device. In this embodiment and the following embodiments, the execution subject is an intelligent interactive device for illustrative description. It can be understood that the intelligent interaction method of this embodiment can be implemented through various intelligent interactive devices, such as smart speakers, smart phones, smart robots, personal computers or servers and other electronic devices, and can also be other controllers and devices that can achieve the same or similar functions. This embodiment does not limit this.

[0024] It can be understood that the current conversation information is the information output by the user to interact with the intelligent interactive device. The information can be text information, voice information, image information, etc. For example, when the intelligent robot uses the sound pickup device to receive the user's current conversation information, it simultaneously calls the camera device to obtain the portrait information of the user. The intelligent robot uses the current conversation information and image information to perform emotion recognition and obtain the user's emotion label. The emotion label includes but is not limited to happiness, sadness, anxiety, anger, calmness, etc.

[0025] Furthermore, the current dialogue information is used to match information from a pre-set corpus to obtain initial response information. The corpus stores multiple initial response information corresponding to the current dialogue information. As an example, when the current dialogue information input by the user is "The faucet in my home is broken", the intelligent robot can match information from a pre-set corpus based on "The faucet in my home is broken" to obtain initial response information, such as "Dear sir, I'm very sorry to hear that your faucet is broken", "It is recommended that you turn off the water source first", "The maintenance team will provide you with door-to-door service in the shortest time", "Dear ma'am, please don't worry, we understand the inconvenience caused to you by the faucet failure in your home", and other initial response information for different users about broken faucets.

[0026] Furthermore, based on the image information, the user's portrait features are determined, including but not limited to the user's age, gender, and clothing, and then the initial response information is screened based on the portrait features and emotional tags to obtain target response information.

[0027] Continuing with the previous example, User 1: portrait features (male, 30 years old, in a suit), emotion label: calm; User 2: portrait features (female, 15 years old, in casual clothes), emotion label: anxious. Both of them input to the smart interactive device at the same time: "The faucet in my house is broken". When receiving the same input "The faucet in my house is broken" from two users (User 1 and User 2), due to their different portrait features and emotion labels, the smart interactive device will filter out different target response information from the initial response information. For the portrait features and emotional labels of user 1 (male, 30 years old, in a suit, emotional label: calm), it can be shown that user 1 is a relatively mature and professional individual who may have certain life skills. At this time, the target response information screened out is "Dear sir, you can turn off the sluice first", "The maintenance team will provide you with door-to-door service in the shortest time, please wait patiently", "I am very sorry to hear that your faucet is out of order"; while for the portrait features and emotional labels of user 2 (female, 15 years old, casual clothes, emotional label: anxious), it can be shown that user 2 may be relatively young and does not have much experience in dealing with similar problems. At this time, her emotions are also relatively anxious. Therefore, the target response information screened out will focus more on soothing her emotions and providing detailed solutions. At this time, the target response information screened out is "Dear lady, please don't worry", "We understand the inconvenience caused by the failure of your faucet", "We have arranged for maintenance personnel to go to your home for repair as soon as possible", "It is recommended that you turn off the water source first to avoid further losses".

[0028] It can be understood that after the intelligent robot selects the target response information from the initial response information, it can perform semantic processing on the target response information to obtain reply information, so as to make the reply information more fluent and logical, so as to meet the personalized needs of the user. Finally, the intelligent robot outputs the reply information to interact intelligently with the user.

[0029] According to the technical solution provided by the embodiment of the present application, the current dialogue information input by the user is obtained, and the image information containing the user is obtained; emotion recognition is performed based on the current dialogue information and image information to obtain the user's emotion label, and the current dialogue information is used to match information from a pre-set corpus to obtain initial response information; based on the image information, the user's portrait features are determined, and the initial response information is screened based on the portrait features and emotion labels to obtain target response information; based on the target response information, the reply information of the current dialogue information is obtained, and intelligent interaction is performed based on the reply information. Through the introduction of image information, the present application can more accurately identify the user's emotional state, thereby avoiding the limitation of emotion recognition only through text dialogue information. At the same time, after obtaining the initial response information, it is not limited to directly giving the reply information, but further analyzing the applicability of the initial response information through the portrait features and emotion labels, ensuring that the reply content meets the user's actual needs and emotional state, and improving the flexibility and intelligence of the intelligent interaction process, so that the interactive content is more in line with the user's personalized needs, thereby improving the user's interactive experience.

[0030] In some embodiments, emotion recognition is performed based on current conversation information and image information to obtain the user's emotion label, including:

[0031] The emotional features in the current conversation information and the image information are extracted respectively to obtain a first emotional feature corresponding to the current conversation information and a second emotional feature corresponding to the image information; the first emotional feature and the second emotional feature are fused to obtain a fused feature vector; the fused feature vector is input into a pre-trained emotion classification model to obtain the user's emotional label.

[0032] Specifically, in the process of emotion recognition, the extraction of the first emotion feature mainly relies on natural language processing technology. By analyzing the vocabulary, syntax and semantic structure of the current dialogue information, the emotional tendency contained in the text can be identified. For example, using the emotional dictionary matching method, the emotional words in the text can be quickly located, and the overall emotional tendency of the text can be judged accordingly.

[0033] It should be noted that for more complex emotion recognition tasks, deep learning models such as convolutional neural networks (CNN) or recurrent neural networks (RNN) can be used to capture the deep semantic information in the text, thereby more accurately identifying the user's emotional state.

[0034] It is understandable that the extraction of the second emotion feature relies on computer vision technology. The image information obtained by the camera of the intelligent robot can be processed by face detection, expression recognition, etc., so as to obtain image features related to the user's emotions. The image features include but are not limited to the movement of facial muscles, the size and position of the eyes, the shape of the mouth, etc., which can reflect the user's emotional state to a certain extent.

[0035] Furthermore, the first emotion feature and the second emotion feature are fused. Specifically, the fusion method can be through feature splicing, weighted summation, etc., which is not specifically limited in this embodiment. It can be understood that the fused feature vector contains both the first emotion feature of the current dialogue information and the second emotion feature of the image information, so as to more comprehensively reflect the emotional state of the user. Finally, the fused feature vector is input into a pre-trained emotion classification model to obtain the user's emotion label. The emotion classification model can be a support vector machine (SVM), a random forest (RandomForest) or a deep learning model, etc. During the training process, the model will learn how to map the input feature vector to the corresponding emotion label, so as to achieve accurate recognition of the user's emotions.

[0036] According to the technical solution provided by the embodiment of the present application, the emotional features in the current conversation information and the image information are extracted respectively to obtain the first emotional feature corresponding to the current conversation information and the second emotional feature corresponding to the image information; the first emotional feature and the second emotional feature are fused to obtain a fused feature vector; the fused feature vector is input into a pre-trained emotion classification model to obtain the user's emotional label. By combining text conversation information and image information, the present application can more accurately identify the user's emotional state, thereby avoiding the limitation of emotion recognition only through text conversation information or image information.

[0037] In addition, in some embodiments, the current dialogue information is used to match information from a preset corpus to obtain initial response information, including:

[0038] The current dialogue information is vectorized to obtain a target query vector; the similarity between the target query vector and the corresponding vector of each response information in the corpus is calculated, and the response information with a similarity greater than a first preset threshold is taken as a candidate response set; the vector distance between the corresponding vector of each response information in the candidate response set and the target query vector is calculated, and the response information corresponding to the vector with a vector distance less than a second preset threshold is taken as the initial response information.

[0039] Specifically, the current dialogue information input by the user is vectorized, that is, the word embedding technology in natural language processing, such as Word2Vec and GloVe, can be used to convert the text into a numerical vector that can be understood and processed by the computer. The generation of the target query vector is based on the overall semantics of the current dialogue information and can accurately reflect the user's dialogue needs.

[0040] Furthermore, the similarity between the target query vector and the corresponding vector of each response information in the corpus is calculated. The similarity calculation method can be cosine similarity, Euclidean distance, etc. Through the similarity calculation, when the similarity between the target query vector and the corresponding vector of each response information in the corpus is greater than the first preset threshold, the response information closest to the current dialogue information can be screened out to form a candidate response set. The vector distance between the corresponding vector of each response information in the candidate response set and the target query vector is further calculated. The calculation of the vector distance is introduced here to further ensure the accuracy of the initial response information. When the vector distance is less than the second preset threshold, it can be considered that the response information has a high degree of match with the current dialogue information and can be used as the initial response information.

[0041] It should be noted that the specific values ​​of the first preset threshold and the second preset threshold can be set according to requirements, and are not specifically limited in this embodiment.

[0042] According to the technical solution provided in the embodiment of the present application, the current dialogue information is vectorized to obtain the target query vector; the similarity between the target query vector and the vector corresponding to each response information in the corpus is calculated, and the response information corresponding to the vector whose similarity is greater than the first preset threshold is used as the candidate response set; the vector distance between the vector corresponding to each response information in the candidate response set and the target query vector is calculated, and the response information corresponding to the vector whose vector distance is less than the second preset threshold is used as the initial response information. Not only can the response information related to the current dialogue information be quickly found from the massive corpus, but also the response information can be ensured to have high accuracy and applicability. It provides users with a more efficient and accurate intelligent interaction experience.

[0043] In addition, in some embodiments, the initial response information is screened according to the portrait features and the emotion tags to obtain the target response information, including:

[0044] Match the portrait features with the pre-set response requirement classification to obtain the user's response label; use the preset weight allocation table to determine the weight information of the response label and the emotional label, and use the response label and the emotional label after the weight information is allocated to filter out the target response information from the initial response information.

[0045] Specifically, the user's portrait features are matched with the pre-set response requirement classification. The response requirement classification is based on the user's portrait features (age, gender, occupation) to ensure that personalized response information can be provided to different user groups during the intelligent interaction process. For example, user 1: portrait features (male, 30 years old, in a suit), emotional label: calm; user 2: portrait features (female, 15 years old, casual clothes), emotional label: anxious. For user 1, the response label is more inclined to a formal and professional style, while for user 2, it is more inclined to a friendly style. By matching the portrait features with the response requirement classification, the user's response label can be obtained, which can guide the screening of subsequent response information.

[0046] Further, the weight information of the response label and the emotion label is determined by using a preset weight distribution table. The weight distribution table can be set according to the actual application scenario and requirements. For example, for user 2, the user's emotional state may be more important, so the weight of the emotion label will be relatively high, so the target response information screened out will more likely solve the user's emotional problems, followed by the solution to the problem; while for user 1, more attention is paid to solving the problem, so the weight of the response label will be higher, so more practical solutions will be provided, and the user's emotions will not be focused on. Through the weight distribution table, it can be ensured that when screening the response information, the user's portrait characteristics and emotion labels can be comprehensively considered to provide users with more personalized intelligent responses. Finally, the target response information is screened from the initial response information using the response label and emotion label after the weight information is assigned. The screening process can be achieved by calculating the matching degree of each initial response information with the response label and emotion label, and the response information with a high matching degree will be preferentially selected as the target response information.

[0047] In this way, based on the user's portrait features and emotional labels, users are provided with intelligent responses that are both in line with their personal characteristics and can accurately respond to their emotional state, thereby further enhancing the user's interactive experience.

[0048] In some embodiments, obtaining reply information of the current conversation information according to the target response information includes:

[0049] The target response information is vectorized; the target query vector corresponding to the current dialogue information, the vector corresponding to the target response information, and the preset instruction text are input into the large language model to obtain the reply information output by the large language model.

[0050] Specifically, the target response information is converted into a numerical vector so that the large language model can understand and process this information. The target query vector corresponding to the current dialogue information, the vector corresponding to the target response information, and the preset instruction text are input into the large language model together so that the large language model can generate reply information based on these input information. The preset instruction text is an instruction or some prompt words that guide the large language model on how to generate a reply. For example, it can be to require the large language model to generate a concise and clear reply, or to require the large language model to quote the relevant content of the target response information in the reply.

[0051] After receiving these input messages, the large language model will use its training data and complex neural network structure to deeply understand and analyze these messages. It will comprehensively consider the context information of the current conversation, the user's portrait characteristics and emotional state, and the content of the target response information, and generate a reply message that is consistent with the conversation logic and can accurately respond to user needs.

[0052] Continuing with the above example, after inputting the current dialogue information of user 2, "My faucet is broken", the target response information obtained, "Dear Madam, please don't worry", "We understand the inconvenience caused by the faucet failure in your home", "We have arranged for maintenance personnel to go to your home for repair as soon as possible", "It is recommended that you turn off the water source first to avoid further losses", and the preset instruction text "Currently know the following content, please modify and polish it to obtain a response information that is both in line with the dialogue logic and can accurately respond to user needs" into the large language model. The large language model outputs the following response information: "Hello, Madam, I am very sorry to hear that your faucet is broken. We fully understand the inconvenience this has caused you. Please don't worry, in order to solve your problem as soon as possible, we have arranged for professional maintenance personnel to go to your home for repair. Before waiting for the maintenance personnel to arrive, please turn off the water source to avoid further losses. If you have any other questions or need further help, please feel free to contact us. Thank you for your understanding and support."

[0053] In this way, the large language model can give full play to its powerful language understanding and generation capabilities, providing users with a more intelligent and personalized interactive experience.

[0054] In addition, in some embodiments, intelligent interaction is performed according to the reply information, including:

[0055] Determine the device interaction parameters of the intelligent interactive device based on the portrait features and emotion tags; use the device interaction parameters to control the intelligent interactive device to output reply information.

[0056] Specifically, after receiving the reply information from the large language model, the intelligent robot needs to present it to the user in an appropriate way. In order to ensure the personalization and efficiency of the interaction, the intelligent interactive device will determine its device interaction parameters based on the user's portrait characteristics and emotional tags. These parameters include sound effect parameters, such as pitch, speech speed, volume, and display parameters, such as the font size and color of the screen display, to adapt to the personal preferences and current emotional state of different users.

[0057] For example, for user 1, since his portrait features show that he is a middle-aged man, his occupation may be a businessman, and his emotion label is calm, the smart interactive device may choose a more formal, moderate-speed voice to output the reply information, and display the relevant information on the screen in a concise and clear manner. For user 2, a young, anxious woman, the smart interactive device may choose a gentler, slower voice to comfort her, and use larger fonts and softer colors on the screen to reduce her anxiety.

[0058] According to the technical solution provided in the embodiment of the present application, the device interaction parameters of the intelligent interactive device are determined according to the portrait features and the emotion tags; the device interaction parameters are used to control the intelligent interactive device to output the reply information. Using these device interaction parameters, the intelligent interactive device can control its output mode to ensure that the reply information can be presented to the user in a way that best meets the user's needs and emotional state. In this way, not only can the user's interactive experience be improved, but also the user's trust and satisfaction with the intelligent interactive device can be enhanced.

[0059] In addition, in some embodiments, after performing intelligent interaction according to the reply information, the method further includes:

[0060] The current conversation information and reply information are vectorized to obtain an updated vector, in which the reply information carries the user's portrait features and emotional labels; the updated vector is stored in the corpus to update the corpus.

[0061] Specifically, after each intelligent interaction is completed, the current conversation information and reply information are vectorized, and the user's portrait features and emotional labels are added to form an update vector. These update vectors not only contain the context information of the conversation, but also integrate the user's personalized features and emotional state, so they can enrich the corpus and improve the accuracy of future intelligent interactions.

[0062] Furthermore, the updated vectors are stored in the corpus, that is, the actual conversation and response information is used for subsequent model training and data analysis. Over time, the corpus will continue to accumulate a large amount of real and diverse conversation data, which can not only enhance the model's ability to understand complex conversation scenarios, but also make it more adaptable to the personalized needs of different users.

[0063] According to the technical solution provided in the embodiment of the present application, by storing the current conversation information and reply information in the form of update vectors in the corpus and updating them regularly, the system can continuously accumulate and optimize data resources, thereby improving the accuracy and personalization of intelligent interaction.

[0064] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of the present application.

[0065] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0066] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0067] Figure 2 Schematic diagram of a smart interactive device provided in an embodiment of the present application. Figure 2 As shown, the device comprises:

[0068] The acquisition module 201 is configured to acquire the current conversation information input by the user and acquire the image information containing the user;

[0069] The recognition module 202 is configured to perform emotion recognition based on the current dialogue information and image information to obtain the user's emotion label, and use the current dialogue information to match information from a pre-set corpus to obtain initial response information;

[0070] The screening module 203 is configured to determine the user's portrait features according to the image information, and screen the initial response information according to the portrait features and the emotion label to obtain the target response information;

[0071] The interaction module 204 is configured to obtain reply information of the current dialogue information according to the target response information, and perform intelligent interaction according to the reply information.

[0072] In some embodiments, the recognition module 202 is also used to extract emotional features from the current conversation information and the image information respectively, to obtain a first emotional feature corresponding to the current conversation information and a second emotional feature corresponding to the image information; to fuse the first emotional feature and the second emotional feature to obtain a fused feature vector; and to input the fused feature vector into a pre-trained emotion classification model to obtain an emotional label of the user.

[0073] In some embodiments, the recognition module 202 is also used to vectorize the current conversation information to obtain a target query vector; calculate the similarity between the target query vector and the vector corresponding to each response information in the corpus, and use the response information corresponding to the vector whose similarity is greater than a first preset threshold as a candidate response set; calculate the vector distance between the vector corresponding to each response information in the candidate response set and the target query vector, and use the response information corresponding to the vector whose vector distance is less than a second preset threshold as the initial response information.

[0074] In some embodiments, the screening module 203 is also used to match the portrait features with pre-set response requirement classifications to obtain the user's response labels; use a preset weight allocation table to determine the weight information of the response labels and emotional labels, and use the response labels and emotional labels after the weight information is allocated to filter out the target response information from the initial response information.

[0075] In some embodiments, the interaction module 204 is also used to vectorize the target response information; the target query vector corresponding to the current dialogue information, the vector corresponding to the target response information, and the preset instruction text are input into the large language model to obtain the reply information output by the large language model.

[0076] In some embodiments, the interaction module 204 is further used to determine the device interaction parameters of the intelligent interactive device according to the portrait features and the emotion tags; and use the device interaction parameters to control the intelligent interactive device to output reply information.

[0077] In some embodiments, the interaction module 204 is also used to vectorize the current conversation information and reply information to obtain an updated vector, wherein the reply information carries the user's portrait features and emotion labels; and store the updated vector in the corpus to update the corpus.

[0078] According to the technical solution provided by the embodiment of the present application, the intelligent interaction device provided by the present embodiment responds to obtaining the current dialogue information input by the user, and obtaining the image information containing the user; performing emotion recognition according to the current dialogue information and the image information, obtaining the emotion label of the user, and using the current dialogue information to match the information from the pre-set corpus to obtain the initial response information; determining the portrait features of the user according to the image information, and filtering the initial response information according to the portrait features and the emotion label to obtain the target response information; obtaining the reply information of the current dialogue information according to the target response information, and performing intelligent interaction according to the reply information. Through the combination of text dialogue information and image information, the present application can more accurately identify the emotional state of the user, thereby avoiding the limitation of emotion recognition only through text dialogue information or image information. At the same time, after obtaining the initial response information, it is not limited to directly giving the reply information, but further analyzing the applicability of the initial response information through the portrait features and the emotion label, ensuring that the reply content meets the actual needs and emotional state of the user, and improving the flexibility and intelligence of the intelligent interaction process, so that the interactive content is more in line with the personalized needs of the user, thereby improving the user's interactive experience.

[0079] Figure 3 Schematic diagram of an electronic device 3 provided in an embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0080] The electronic device 3 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 The electronic device 3 is merely an example and does not limit the electronic device 3 , and may include more or less components than those shown in the figure, or different components.

[0081] The processor 301 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0082] The memory 302 may be an internal storage unit of the electronic device 3, for example, a hard disk or memory of the electronic device 3. The memory 302 may also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 3. The memory 302 may also include both an internal storage unit and an external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0083] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0084] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0085] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An intelligent interaction method, characterized in that: include: Obtaining current conversation information input by the user, and obtaining image information containing the user; Performing emotion recognition based on the current conversation information and the image information to obtain the user's emotion label, and using the current conversation information to perform information matching from a pre-set corpus to obtain initial response information; Determine the user's portrait features according to the image information, and filter the initial response information according to the portrait features and the emotion label to obtain target response information; According to the target response information, reply information of the current dialogue information is obtained, and intelligent interaction is performed according to the reply information.

2. The method according to claim 1, characterized in that: The performing emotion recognition according to the current conversation information and the image information to obtain the emotion label of the user includes: Extracting emotion features from the current conversation information and the image information respectively, to obtain a first emotion feature corresponding to the current conversation information and a second emotion feature corresponding to the image information; Fusing the first emotion feature and the second emotion feature to obtain a fused feature vector; The fused feature vector is input into a pre-trained emotion classification model to obtain the emotion label of the user.

3. The method according to claim 1, characterized in that The using the current dialogue information to match information from a preset corpus to obtain initial response information includes: Vectorizing the current conversation information to obtain a target query vector; Calculating the similarity between the target query vector and each corresponding vector of the response information in the corpus, and taking the response information corresponding to the vector whose similarity is greater than a first preset threshold as a candidate response set; The vector distance between each vector corresponding to the response information in the candidate response set and the target query vector is calculated, and the response information corresponding to the vector whose vector distance is less than a second preset threshold is used as the initial response information.

4. The method according to claim 1, characterized in that: The screening of the initial response information according to the portrait features and the emotion label to obtain target response information includes: Matching the portrait features with a preset response requirement classification to obtain a response label of the user; The weight information of the response tag and the emotion tag is determined by using a preset weight allocation table, and the target response information is screened from the initial response information by using the response tag and the emotion tag after the weight information is allocated.

5. The method according to claim 3, characterized in that: The step of obtaining reply information of the current dialogue information according to the target response information includes: Vectorizing the target response information; The target query vector corresponding to the current dialogue information, the vector corresponding to the target response information, and a preset instruction text are input into a large language model to obtain reply information output by the large language model.

6. The method according to claim 1, characterized in that The performing intelligent interaction according to the reply information includes: Determining device interaction parameters of the intelligent interactive device according to the portrait features and the emotion tags; The device interaction parameter is used to control the intelligent interactive device to output the reply information.

7. The method according to claim 1, characterized in that After performing intelligent interaction according to the reply information, the method further includes: Vectorizing the current conversation information and the reply information to obtain an updated vector, wherein the reply information carries the user's portrait features and emotion labels; The update vector is stored in the corpus to update the corpus.

8. An intelligent interactive device, characterized in that: include: An acquisition module, configured to acquire current conversation information input by a user, and acquire image information containing the user; a recognition module configured to perform emotion recognition based on the current dialogue information and the image information to obtain the user's emotion label, and use the current dialogue information to perform information matching from a preset corpus to obtain initial response information; A screening module is configured to determine the portrait features of the user according to the image information, and screen the initial response information according to the portrait features and the emotion label to obtain target response information; The interaction module is configured to obtain reply information of the current dialogue information according to the target response information, and perform intelligent interaction according to the reply information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images