User interaction method, storage medium and user interaction system

Abstract

The invention provides a user interaction method, a storage medium and a user interaction system. The method comprises the steps of obtaining interaction information of a current user; wherein the interaction information of the user comprises user image information, user voice information and environment information of the user; determining a current emotional state category of the user according to the user image information, the user voice information and the information of the environment where the user is located; searching a target interactive content set associated with the current emotional state category from a plurality of interactive content sets according to the current emotional state category; wherein each emotional state category is associated with an interactive content set; obtaining preference characteristics of the user; according to the preference characteristics of the user and the target interaction content set, the target interaction content is determined, deep emotional interaction with the user is achieved, more accurate response content with higher emotional value is provided for the user, and then the interaction experience of the user is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a user interaction method, a storage medium, and a user interaction system. Background Art

[0002] In the current smart home market, indoor smart cameras with screens have become common devices. Users can perform various operations through smart cameras with screens to achieve interaction between the device and the user, such as video monitoring and voice channels.

[0003] However, current existing products can only complete basic video monitoring and simple voice call functions. The interaction between users and cameras is mechanical and lacks emotional communication. It is impossible to provide targeted services based on the user's emotional state, behavioral habits, etc., which leads to poor user experience. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a user interaction method, a storage medium, and a user interaction system to solve the problem that current products such as smart cameras with screens can only complete basic video monitoring and simple voice call functions, the interaction between users and cameras is mechanical, lacks emotional communication, and cannot provide targeted services based on the user's emotional state, behavioral habits, etc., which leads to poor user experience.

[0005] In a first aspect, the present application provides a user interaction method, which includes: obtaining the interaction information of the current user; wherein the interaction information of the user includes user image information, user voice information and information about the environment in which the user is located; determining the current emotional state category of the user based on the user image information, user voice information and information about the environment in which the user is located; searching for a target interactive content set associated with the current emotional state category in multiple interactive content sets based on the current emotional state category; wherein each emotional state category is associated with an interactive content set; obtaining the user's preference characteristics; determining the target interactive content based on the user's preference characteristics and the target interactive content set; wherein the target interactive content is used to respond to the user's interaction information.

[0006] The user interaction method designed in this solution accurately identifies the user's emotional state through various interactive information during the user interaction process (such as user image information, user voice information, and user's environmental information), and determines the current interactive response content with the user based on the user's emotional state and user preference characteristics, thereby achieving deep emotional communication and interaction with the user, providing the user with more accurate and emotionally valuable response content, and significantly improving the user's interactive experience.

[0007] In an optional implementation of the first aspect, the user's current emotional state is determined based on the user image information, the user voice information and the user's environmental information, including: inputting the user image information, the user voice information and the user's environmental information into a pre-trained target emotion recognition model; wherein the target emotion recognition model is used to obtain the user's current emotional state.

[0008] In the above implementation mode, this solution uses a pre-trained target emotion recognition model to quickly identify the user's current emotional state category, thereby improving the timeliness and accuracy of interactive content feedback during user interaction, and further improving the user's interactive experience.

[0009] In an optional implementation of the first aspect, the training process of the target emotion recognition model includes: obtaining a target data set; wherein the target data set includes interaction information of multiple different users, and the interaction information of each user includes user image information, user voice information, and the user's environment information and corresponding emotion category labels; dividing the target data set into multiple training groups with a preset number of interaction information as a group; selecting a target group from the multiple training groups as the current training group; performing feature extraction on each interaction information in the current training group to obtain a current training sample group, wherein the current training sample group includes multiple target training samples, and each target training sample includes a fused feature vector and a corresponding emotion category label, wherein the fused feature vector is based on the user image sample features, user voice sample features and environment sample features in the corresponding training sample. Feature generation; calculate the probability distribution of each type of emotional state corresponding to the target training sample according to the fusion features of each target training sample in the current training sample group; calculate the training loss value corresponding to the current training sample group through the loss function according to the probability distribution of each type of emotional state corresponding to each target training sample in the current training sample group and the corresponding emotional category label; update the various training parameters of the model according to the training loss value; judge whether the current cumulative number of iterations exceeds the preset number of iterations, or whether the various training parameters of the model converge; if it is determined that the current cumulative number of iterations exceeds the preset number of iterations or the various training parameters of the model converge, then obtain the trained target emotion recognition model; if it is determined that the current cumulative number of iterations does not exceed the preset number of iterations and the various training parameters of the model do not converge, then return to execute and select a target group from multiple training groups as the current training group.

[0010] In an optional implementation of the first aspect, the probability distribution of each type of emotional state corresponding to the target training sample is calculated based on the fusion features of each target training sample in the current training sample group, including: performing fusion feature learning on the fusion feature vector of each target training sample through the ReLU activation function of the first fully connected layer and the second fully connected layer to obtain the output fusion feature learning vector corresponding to each target training sample; and calculating the probability distribution of each type of emotional state corresponding to each target training sample based on the fusion feature learning vector corresponding to each target training sample through the softmax activation function of the output layer.

[0011] In an optional implementation of the first aspect, feature extraction is performed on each interactive information in the current training group to obtain the current training sample group, including: preprocessing the user image information of the current training group, extracting the facial local features and global features in the preprocessed user image information through a convolutional neural network to obtain user image sample features; preprocessing the user voice information in the current training group by frame splitting and windowing, extracting the temporal features of the preprocessed user voice information through a long short-term memory network to obtain user voice sample features; normalizing the environmental information of the users in the current training group to obtain environmental sample features; and merging the user image sample features, user voice sample features, and environmental sample features into a fused feature vector.

[0012] Among the various implementation methods mentioned above, this solution directly inputs the user image information, user voice information and the environmental feature information of the user as independent features into the model, and then the model automatically fuses the user image information, user voice information and the environmental feature information of the user to obtain the fusion feature, and then allows the model to autonomously learn the dependency between features based on the fusion feature, so as to adaptively adjust the proportion of user image information, user voice information and environmental feature information in the fusion feature of the emotion recognition model based on massive user interaction information samples, thereby significantly improving the accuracy and reliability of the trained target emotion recognition model in identifying user interaction information in different states, and at the same time, it is more flexible and accurate than the manually preset fusion method.

[0013] In an optional implementation of the first aspect, determining the target interactive content based on the user's preference characteristics and the target interactive content set includes: calculating the similarity between each interactive content in the target interactive content set and the user's preference characteristics; and determining the interactive content with the highest similarity as the target interactive content.

[0014] In the above implementation mode, this solution determines the target interactive content based on the similarity between each interactive content and the user's preference characteristics, so that the target interactive content that interacts with the user's feedback is more in line with the user's preferences, and thus it is easier to meet the user's current interactive emotional requirements and improve the user's interactive experience.

[0015] In an optional implementation of the first aspect, after determining the target interactive content based on the user's preference characteristics and the target interactive content set, the method further includes: collecting the user's interactive feedback information on the target interactive content; calculating a user feedback index value based on the user's interactive feedback information on the target interactive content; wherein the user feedback value is used to characterize the user's positive feedback on the target interactive content; when the user feedback index value is less than a preset feedback index threshold, updating the user's current emotional state and the target interactive content.

[0016] In an optional implementation manner of the first aspect, collecting interactive feedback information of users on the target interactive content includes: counting the number of operations of each type of feedback operation generated by the user within a preset time window to obtain interactive feedback information; wherein, multiple feedback operation types are pre-set, weight information belonging to positive feedback operation types is a positive value, and weight information belonging to negative feedback operation types is a negative value, and the preset time window represents a preset time range after outputting the target interactive content; calculating a user feedback index value based on the interactive feedback information of the user on the target interactive content, including: obtaining weight information corresponding to each type of feedback operation generated by the user; calculating the product of the number of each type of operation and the corresponding weight information to obtain a feedback index value for each type of operation; and calculating the sum of the feedback index values ​​of all operations to obtain a user feedback index value.

[0017] In the above implementation mode, the scheme calculates the user feedback index value based on the user's feedback information on the target interactive content, and determines whether to update the interactive content based on the user feedback index value, thereby realizing dynamic adjustment of the user interactive content, further improving the accuracy of the user interactive content, and improving the user interactive experience.

[0018] In an optional implementation of the first aspect, obtaining the user's preference characteristics includes: constructing a user preference model through a multi-layer perceptron, wherein the user preference model is trained based on user operation characteristics, user basic information, and preference characteristics actively set by the user, wherein the user operation characteristics include click location, click frequency, browsed content type, and browse time, and the user basic information includes user age, gender, and occupation; the training process of the user preference model includes: using a stochastic gradient descent algorithm and a mean square error loss function to iteratively optimize the mapping relationship between user operation characteristics, user basic information, and preference characteristics actively set by the user to generate the user preference model.

[0019] In an optional implementation of the first aspect, the method further includes: if the user is a new user, matching a preset general preference module based on the basic user information entered by the new user, wherein the general preference templates corresponding to different user ages, occupations, and genders are different; and based on subsequent operation data of the new user, dynamically adjusting the preference weight of the general preference template corresponding to the new user by a gradient descent method.

[0020] In an optional implementation of the first aspect, the user's environmental information includes the ambient temperature and the ambient light brightness, and a target interactive content set associated with the current emotional state category is searched for in multiple interactive content sets according to the current emotional state category, including: when the user's ambient temperature is lower than the preset ambient temperature and the current emotional state category is calm, a recommendation set formed by warm-toned images and / or hot drinks is selected as the target interactive content set; when the user's ambient light brightness is higher than the preset light brightness and the current emotional state category is irritability, a recommendation set formed by pushing sunshade product information and / or soothing voice is selected as the target interactive content set.

[0021] In a second aspect, the present application provides a user interaction device, which includes an acquisition module, a determination module and a search module. The acquisition module is used to obtain the interaction information of the current user; wherein the user's interaction information includes user image information, user voice information and information about the environment where the user is located; the determination module is used to determine the user's current emotional state category based on the user image information, user voice information and information about the environment where the user is located; the search module is used to search for a target interactive content set associated with the current emotional state category in multiple interactive content sets based on the current emotional state category; wherein each emotional state category is associated with an interactive content set; the acquisition module is also used to obtain the user's preference characteristics; the determination module is also used to determine the target interactive content based on the user's preference characteristics and the target interactive content set; wherein the target interactive content is used to respond to the user's interactive information.

[0022] The user interaction device designed above uses a variety of interactive information during the user interaction process (such as user image information, user voice information, and user environment information) to accurately identify the user's emotional state, and determine the current interactive response content with the user based on the user's emotional state and user preference characteristics, thereby achieving deep emotional communication and interaction with the user, providing the user with more accurate and emotionally valuable response content, and significantly improving the user's interactive experience.

[0023] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the third aspect or any optional embodiment of the third aspect is executed.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the third aspect or any optional embodiment of the third aspect is performed.

[0025] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, performs the method described in the third aspect or any optional embodiment of the third aspect.

[0026] In a sixth aspect, the present application provides a user interaction system, which includes an image acquisition device, a voice acquisition device, an environment acquisition device and a control device, wherein the control device is electrically connected to the image acquisition device, the voice acquisition device and the environment acquisition device respectively; a visual acquisition device is used to acquire user image information and transmit the user image information to the control device; a voice acquisition device is used to acquire user voice information and transmit the user voice information to the control device; an environment acquisition device is used to acquire user environment information and transmit the environment information to the control device; the control device is used to determine the user's current emotional state category based on the user's visual information, the user's voice information and the user's environment information; according to the current emotional state category, search for a target interactive content set associated with the current emotional state category in multiple interactive content sets; wherein each emotional state category is associated with an interactive content set; obtain the user's preference characteristics; determine the target interactive content based on the user's preference characteristics and the target interactive content set; wherein the target interactive content is used to respond to the user's interactive information.

[0027] The user interaction system designed above, this solution collects a variety of interactive information (such as user image information, user voice information and user's environmental information) during the user interaction process through image acquisition equipment, voice acquisition equipment and environment acquisition equipment. The control device uses the various interactive information during the user interaction process to accurately identify the user's emotional state, and determines the response content based on the user's current emotional state and usual preference characteristics, thereby realizing emotional interaction with the user, providing the user with more accurate and emotionally valuable response content, and significantly improving the user's interactive experience.

[0028] In an optional implementation of the sixth aspect, the control device includes a user preference model storage module and a preference feature generation module. The user preference model storage module is used to store the user preference model obtained based on multi-layer perceptron training; the preference feature generation module generates user preference features through the user preference model based on user operation data and basic information.

[0029] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 A first structural diagram of a user interaction system provided in an embodiment of the present application; Figure 2 A second structural diagram of the user interaction system provided in an embodiment of the present application; Figure 3 A schematic diagram of a first flow chart of a user interaction method provided in an embodiment of the present application; Figure 4 A second flow chart of the user interaction method provided in an embodiment of the present application; Figure 5 A third flow chart of the user interaction method provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a user interaction device provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0032] Icons: 10 - image acquisition device; 20 - voice acquisition device; 30 - environment acquisition device; 40 - control device; 50 - display touch device; 600 - acquisition module; 610 - determination module; 620 - search module; 7 - electronic device; 701 - processor; 702 - memory; 703 - communication bus. DETAILED DESCRIPTION

[0033] The following embodiments of the technical solution of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application and are therefore only examples and are not intended to limit the scope of protection of the present application.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0035] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0036] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0038] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0039] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0040] In the description of the embodiments of the present application, unless otherwise expressly specified or limited, technical terms such as "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components or interactions between two components. Those skilled in the art can understand the specific meanings of the above terms in the embodiments of the present application based on specific circumstances.

[0041] In the current smart home market, indoor smart cameras with screens have become common devices. Users can perform various operations through smart cameras with screens to achieve interaction between the device and the user, such as video monitoring and voice channels.

[0042] However, current existing products can only complete basic video monitoring and simple voice call functions. The interaction between users and cameras is mechanical and lacks emotional communication. It is impossible to provide targeted services based on the user's emotional state, behavioral habits, etc., which leads to poor user experience.

[0043] Based on the above problems, this application designs a user interaction method, storage medium and user interaction system. This solution accurately identifies the user's emotional state through various interactive information in the user interaction process (such as user image information, user voice information and user's environmental information), and determines the current interactive response content with the user based on the user's emotional state and user preference characteristics at this time, thereby achieving deep emotional communication and interaction with the user, providing the user with more accurate and emotionally valuable response content, and thus significantly improving the user's interactive experience.

[0044] Based on the above ideas, this application first provides a user interaction system, such as Figure 1 As shown, the user interaction system may include an image acquisition device 10, a voice acquisition device 20, an environment acquisition device 30 and a control device 40, and the control device 40 is electrically connected to the image acquisition device 10, the voice acquisition device 20 and the environment acquisition device 30 respectively.

[0045] The image acquisition device 10 can capture images of the user, specifically facial expressions, body movements, and other image information. Specifically, the image acquisition device designed in this solution can be a camera using a high-resolution CMOS image sensor with a resolution of no less than 4K (3840×2160) and a frame rate of at least 60fps, ensuring smooth video and accurately capturing subtle changes in expression and movement details. The camera can use a 120°-150° viewing angle lens to cover a wide range of indoor activity areas.

[0046] The voice acquisition device 20 can collect the user's voice information, for example, the user's voice content and intonation. Specifically, the voice acquisition device 20 designed in this solution can use a microphone array, for example, a microphone array consisting of four high-sensitivity microphones, to achieve 360° voice acquisition. The microphone sensitivity can be selected to be -42dBV / Pa ±3dB, and the frequency response can be 20Hz-20kHz. This allows accurate acquisition of the user's voice content and intonation in typical indoor noise environments (40dB-60dB).

[0047] The environmental collection device 30 is used to collect information about the user's environment, including temperature, humidity, light brightness, and air quality. Specifically, the environmental collection device 30 designed in this solution can include multiple types of sensors, such as an integrated temperature sensor, humidity sensor, light sensor, and air quality sensor, to collect real-time environmental data.

[0048] The control device 40 is used to determine the user's current emotional state category based on the user's visual information, the user's voice information and the user's environmental information; search for a target interactive content set associated with the current emotional state category in multiple interactive content sets based on the current emotional state category; wherein each emotional state category is associated with an interactive content set; obtain the user's preference characteristics; determine the target interactive content based on the user's preference characteristics and the target interactive content set; wherein the target interactive content is used to respond to the user's interactive information.

[0049] Specifically, the control device 40 can utilize a high-performance AI processor with strong computing capabilities to meet the requirements of data processing and AI model operations. For example, an NVIDIA Jetson Xavier NX processor can be used, which has multiple cores, a main frequency of no less than 1.5 GHz, and is equipped with at least 8 GB of RAM and 64 GB of internal storage (ROM).

[0050] The user interaction system designed above, this solution collects a variety of interactive information (such as user image information, user voice information and user's environmental information) during the user interaction process through image acquisition equipment, voice acquisition equipment and environment acquisition equipment. The control device uses the various interactive information during the user interaction process to accurately identify the user's emotional state, and determines the response content based on the user's current emotional state and usual preference characteristics, thereby realizing emotional interaction with the user, providing the user with more accurate and emotionally valuable response content, and significantly improving the user's interactive experience.

[0051] In an optional implementation manner of this embodiment, as a possible implementation manner, Figure 2 As shown, the user interaction system designed in this solution can also include a display touch device 50, which can be used to display video images, interactive interfaces and push information. The display screen touch function can be single-point touch or multi-point touch, and the touch response time can be set.

[0052] In an optional implementation of this embodiment, as a possible implementation, the control device 40 designed in this scheme may include a user preference model storage module and a preference feature generation module, wherein the user preference model storage module is used to store the user preference model obtained based on multi-layer perceptron training; the preference feature generation module is used to generate user preference features through the user preference model based on user operation data and basic information.

[0053] The present application also provides a user interaction method, which can be applied to the control device described above to achieve the determination and response of the response content of the user interaction, such as Figure 3 As shown, the user interaction method designed in this solution can be implemented in the following ways, including: Step S300: Obtain the interaction information of the current user.

[0054] Step S310: Determine the user's current emotional state category based on the user image information, the user voice information, and the user's environment information.

[0055] Step S320: searching for a target interactive content set associated with the current emotional state category in multiple interactive content sets according to the current emotional state category.

[0056] Step S330: Obtain the user's preference characteristics.

[0057] Step S340: Determine target interactive content based on the user's preference characteristics and the target interactive content set.

[0058] In the above-described embodiment, user interaction information may include user image information, user voice information, and information about the user's environment. User image information may include a user's facial image and / or a user's body movement image; user voice information includes the user's current speech and intonation information; and user environment information may include information such as the temperature, humidity, light brightness, and air quality of the user's environment. Specifically, user interaction information may be collected using the image acquisition device, voice acquisition device, and environment acquisition device described above and transmitted to the control device.

[0059] When the current user image information, user voice information and user environment information are obtained in the above manner, this solution determines the user's current emotional state category based on the user image information, user voice information and user environment information.

[0060] As a possible implementation, this solution can use a neural network model to identify the user's current emotional state category. Specifically, this solution can input user image information, user voice information, and user environment information into a pre-trained target emotion recognition model, thereby obtaining the user's current emotional state category output by the target emotion recognition model.

[0061] As a possible implementation, Figure 4 As shown in Figure 2, the target emotion recognition model can be trained by the following methods: Step S400: Acquire a target data set.

[0062] Step S410: Divide the target data set into multiple training groups, with a preset number of interaction information as a group.

[0063] Step S420: Select a target group from the multiple training groups as the current training group.

[0064] Step S430: extracting features from each interactive information in the current training group to obtain a current training sample group.

[0065] Step S440: Calculate the probability distribution of each type of emotional state corresponding to the target training sample based on the fusion features of each target training sample in the current training sample group.

[0066] Step S450: Calculate the training loss value corresponding to the current training sample group through the loss function according to the probability distribution of each type of emotional state and the corresponding emotional category label corresponding to each target training sample in the current training sample group.

[0067] Step S460: Update various training parameters of the model according to the training loss value.

[0068] Step S470: Determine whether the current cumulative number of iterations exceeds the preset number of iterations, or whether the various training parameters of the model converge. If so, go to step S480; if not, go to step S420.

[0069] Step S480: Obtain the trained target emotion recognition model.

[0070] In the above implementation, this solution first acquires a target dataset, which includes interaction information from multiple different users. Each user's interaction information includes user image information, user voice information, and information about the user's environment. For example, a large number of facial expression images, voice samples, and corresponding environmental data, representing different emotional states (such as happiness, sadness, anger, surprise, and calmness) can be collected to construct the target dataset. The facial expression image data is collected and annotated using professional image acquisition equipment and annotation tools to ensure accurate annotation. The voice sample data is recorded by professional voice actors and manually labeled with emotion categories. Environmental data is collected using actual environmental sensors.

[0071] Then, this scheme divides the target dataset into multiple training groups with a preset number of interaction information as a group. For example, assuming that the target dataset has 100,000 users' interaction information samples, and this scheme presets the number to be 64, then this scheme can be divided into 100,000 / 64≈1563 training groups, which can achieve 1563 iterations.

[0072] During each iterative training process, this scheme can select a target group from multiple training groups as the current training group, and then perform feature extraction on each interactive information in the current training group of this scheme to obtain the current training sample group, wherein the current training sample group includes multiple target training samples, and each target training sample includes a fused feature vector and a corresponding emotion category label, wherein the fused feature vector is generated based on the user image sample features, user voice sample features, and environment sample features in the corresponding training sample.

[0073] As a specific implementation method, the feature extraction process of this solution can be as follows: For user image information, this solution can first preprocess the user image information. The preprocessing process is as follows: Gaussian filtering is used to remove image noise and retain facial details. Histogram equalization is then used to enhance image contrast and make facial expressions more prominent. Bilinear interpolation is used to scale the image to the input size required by the convolutional neural network, resulting in the preprocessed user image information. This solution then uses a convolutional neural network such as ResNet-101 to extract local facial features (such as eyes and mouth) and global features (such as facial contour) from the preprocessed user image information, thereby obtaining user image sample features.

[0074] For user voice information, this solution can first preprocess the user voice information. The preprocessing process is as follows: the user voice information is framed and windowed. For example, the voice is divided into 25ms short frames, the frame shift is 10ms, and a Hamming window is added to reduce spectral leakage. Feature extraction is then performed. For example, Mel-frequency cepstral coefficients (MFCCs) are calculated, 13-dimensional MFCC features are extracted, and first-order and second-order differences are added to obtain a 39-dimensional feature vector. The voice frame feature sequence is then converted into a T*39 matrix (T is the number of frames) to obtain the preprocessed user voice information. Finally, the long short-term memory network layer is used to capture the temporal dependencies in the preprocessed user voice information (voice feature sequence), thereby obtaining the user voice sample features.

[0075] For the user's environmental information, this solution uses minimum-maximum normalization to normalize data such as temperature and humidity to the range of [0,1], thereby obtaining the sample characteristics of the user's environment.

[0076] Based on the above method, this solution can obtain the user image sample features, user voice sample features, and environment sample features corresponding to each target training sample. This solution can determine the fused feature vector corresponding to each target training sample based on the user image sample features, user voice sample features, and environment sample features in each target training sample. For example, the fused feature vector C of the user image sample feature V, the user voice sample feature A, and the environment sample feature E is combined. The fused feature vector C can be obtained by weighted summation, for example, A E, where , The parameters for weight model training are determined through a large number of experiments and optimizations. For example, The value is 0.5, The value is 0.4, The value is 0.1.

[0077] When the fused feature vector is obtained, this scheme can calculate the probability distribution of each type of emotional state corresponding to the target training sample based on the fused feature vector of each target training sample. Specifically, this scheme performs fused feature learning on the fused feature vector through the first fully connected layer and the second fully connected layer to obtain the fused feature learning vector corresponding to each target training sample. Specifically, the fused feature vector C can be first input into the first fully connected layer (such as 512 dimensions), and nonlinearity is introduced through the ReLU activation function. The formula is: h1=ReLU(W1⋅C+b1), where W1 is the weight matrix and b1 is the bias; then it will pass through the second fully connected layer (such as 256 dimensions) to further fuse cross-modal features: h2=ReLU(W2⋅h1+b2), thereby obtaining the fused feature learning vector corresponding to each target training sample.

[0078] Finally, through the activation function of the output layer, the probability distribution of each target training sample corresponding to each type of emotional state is calculated based on the fusion feature learning vector corresponding to each target training sample. For example, the softmax activation function can be used to calculate the probability distribution of each target training sample corresponding to each type of emotional state based on the fusion feature learning vector h2 corresponding to each target training sample. ,in, is the probability that the i-th sample belongs to the k-th emotion.

[0079] When the probability distribution is obtained by the above method, this solution calculates the training loss value corresponding to the current training sample group through the loss function based on the probability distribution of each type of emotional state and the corresponding emotional category label corresponding to each target training sample in the current training sample group.

[0080] Specifically, this solution can use the cross entropy loss function to calculate the training loss value corresponding to the current training sample group, and its formula is: ; Where, N is the number of target training samples in the current training sample, is the true emotional label of the i-th sample (one-hot encoding, such as sadness corresponds to [0,1,0,0,0]), is the probability that the i-th sample belongs to the k-th emotion.

[0081] When the training loss value corresponding to the current training sample group is calculated by the above method, this solution updates the various training parameters of the model according to the training loss value. The training parameter update method can be updated according to the update iteration gradient set in advance. The various training parameters of the model include the convolutional layer weights of ResNet-101, the weights of LSTM, the weights of the fully connected layer, etc.

[0082] After updating each training parameter, this solution can determine whether the current cumulative number of iterations exceeds the preset number of iterations, or whether the various training parameters of the model converge; if it is determined that the current cumulative number of iterations exceeds the preset number of iterations or the various training parameters of the model converge, then a trained target emotion recognition model is obtained; if it is determined that the current cumulative number of iterations does not exceed the preset number of iterations and the various training parameters of the model do not converge, then the step of selecting a target group from multiple training groups as the current training group is returned to, thereby repeatedly updating the various parameters of the model until the number of iterations exceeds the preset number of iterations or the various training parameters of the model converge.

[0083] In the above implementation method, this solution directly inputs user image information, user voice information, and environmental feature information of the user as independent features into the model. The model then fuses the user image information, user voice information, and environmental feature information of the user to obtain fused features. Based on the fused features, the model then autonomously learns the dependencies between features. This allows the model to adaptively adjust the proportion of user image information, user voice information, and environmental feature information in the fused features of the emotion recognition model based on a large number of user interaction information samples, thereby significantly improving the accuracy and reliability of the trained target emotion recognition model in recognizing user interaction information in different states. At the same time, it is more flexible and accurate than manually preset fusion methods. For example, visual features and voice features may have different correlations in different emotional states. When happy, facial expressions are highly correlated with voice intonation, while when sad, only facial expressions may be more prominent. The model can dynamically adjust the fusion logic through training to further improve the accuracy of the emotion recognition model in recognizing the user's current emotional state category.

[0084] This solution can train the target emotion recognition model through the above method. On this basis, this solution inputs the currently obtained user image information, user voice information and user environment information into the target emotion recognition model to obtain the user's current emotional state category output by the target emotion recognition model.

[0085] Based on the above-mentioned acquisition of the current emotional state category, this solution can search for a target interactive content set associated with the current emotional state category in multiple interactive content sets based on the current emotional state category. This solution pre-sets an emotional interaction mapping table, in which each emotional state category corresponds to a set of possible interactive content sets. In this case, this solution can search for the target interactive content set mapped to the current emotional state category in the emotional interaction mapping table based on the current emotional state category output by the target emotion recognition model. For example, when the current emotional state category is sadness, the interactive content set found may include a soothing music playlist, comforting speech text, etc.

[0086] When this scheme obtains the target interactive content set through the above method, this scheme obtains the preference characteristics of the user, wherein the user's preference characteristics can be obtained based on the user preference model, wherein the user's preference model can be obtained by training a multi-layer perceptron (MLP) based on user operation characteristics, preference characteristics and user basic characteristics. The user operation characteristics include user operation characteristic information such as click position, click frequency, browsed content type, and browsing time. The user preference characteristics represent the preferences actively set by the user. For example, the favorite music types are divided into categories such as rock and classical, and the user's selected music preference is rock; the user's basic information includes the user's age, gender, occupation, etc.

[0087] When training the user preference model based on the multi-layer perceptron (MLP) architecture, the stochastic gradient descent (SGD) algorithm is used in combination with the mean square error (MSE) loss function to train the user preference model until the model converges to obtain the user preference model.

[0088] In user preference models, the cold start problem—when new users have no historical data—impacts their initial experience. To effectively address this issue, this solution introduces an initialization mechanism based on a universal preference template. This mechanism consists of three key steps: template construction, matching initialization, and dynamic optimization.

[0089] First, we build a universal preference template and collect the operation data, personalized settings, and basic information of users with different characteristics. These users cover different ages, genders, occupations, regions, etc., thus forming a rich and diverse data set. We use data mining techniques such as cluster analysis to conduct in-depth analysis of these data, and then discover the common preferences of different user groups. Based on the common preferences of different user groups, we build universal preference templates for different user groups.

[0090] When a new user uses the camera for the first time, they enter the matching initialization phase. After the new user completes the entry of basic information such as age, gender, and occupation in the initialization settings interface, this solution can match this information with the previously constructed universal preference template. For example, if the new user is a young office worker, the preference data for this group in the universal preference template is referenced to initialize the preference model for this new user. Specific settings include default music preferences for pop music, news preferences for technology and workplace news, and a preference for work reminders in terms of interface functions.

[0091] This solution will also collect new users' operational data in real time, such as click location, click frequency, browsing time, etc., and will also pay attention to changes in personalized settings. As the collected data continues to increase, the system will use machine learning algorithms, such as gradient descent, to dynamically adjust the initial preference model. For example, if a new user frequently browses health information, the system will increase the weight of health-related content in the preference model, so that the preference model can better fit the user's true preferences. This continuous optimization mechanism allows the system to continuously improve services as users use it after a cold start, thereby enhancing the user experience.

[0092] When this solution constructs a user preference model in the above manner, the user's preference characteristics can be determined based on the user information and the user preference model. Then, based on the user's preference characteristics and the target interaction content set associated with the current emotional state category, the target interaction content to respond to the user can be determined.

[0093] As a possible implementation method, this solution can calculate the similarity between each interactive content in the target interactive content set and the user's preference characteristics, and then determine the interactive content with the highest similarity as the target interactive content. Specifically, this solution can use the cosine similarity calculation method to calculate the similarity between each interactive content in the target interactive content set and the user's preference characteristics. For example, assuming that the current emotional state category is , and its corresponding target interactive content set is ;User preference characteristics , indicating the preference for different types of content (such as music, pictures, and text), this solution can calculate the preference for each response content Similarity score with preference vector P : ; Ultimately, this solution determines the interactive content with the highest similarity as the target interactive content. For example, if the user's current emotional state is sadness, the found interactive content might include a list of soothing music and comforting text. Given that the user's preference is music, the target interactive content determined by this solution might be soothing music.

[0094] The user interaction method designed above uses a variety of interactive information during the user interaction process (such as user image information, user voice information, and user environment information) to accurately identify the user's emotional state, and determine the current interactive response content with the user based on the user's emotional state at this time and the user's preference characteristics, thereby achieving deep emotional communication and interaction with the user, providing the user with more accurate and emotionally valuable response content, thereby significantly improving the user's interactive experience; in addition, this solution uses a pre-trained emotion recognition model to accurately and quickly identify the user's current emotional state category and uses a user preference model to accurately and quickly determine the user's preference characteristics, thereby improving the timeliness and accuracy of user interactive responses, and further improving the user's interactive experience.

[0095] In an optional implementation of this embodiment, after determining the target interactive content, this solution can also timely adjust and update the interactive content through the following implementation methods. Specifically, Figure 5 As shown, including: Step S500: Collecting user interaction feedback information on target interactive content.

[0096] Step S510: Calculate a user feedback index value based on the user's interactive feedback information on the target interactive content.

[0097] Step S520: When the user feedback index value is less than a preset feedback index threshold, the user's current emotional state and target interactive content are updated.

[0098] In the above-mentioned implementation mode, during the emotional interaction process, this solution can continuously monitor the user's feedback to obtain the user's interactive feedback information on the target interactive content, wherein the interactive feedback information may include the user's facial expression image information, feedback voice information, and feedback environmental information, etc.

[0099] Then, this solution can calculate a user feedback index value based on the user's interactive feedback information on the target interactive content, wherein the user feedback index value is used to represent the user's positive feedback level on the target interactive content.

[0100] As a possible implementation method, the user feedback index value can be calculated based on the feedback operations generated by the user on the target interactive content. Specifically, this solution can count the number of operations of each type of feedback operation generated by the user within a preset time window to obtain interactive feedback information, wherein a plurality of feedback operation types are pre-set, and the weight information belonging to the positive feedback operation type is a positive value, and the weight information belonging to the negative feedback operation type is a negative value. The preset time window represents a preset time range after the target interactive content is output.

[0101] Then, the weight information corresponding to each type of feedback operation generated by the user is obtained; the product of the number of each type of operation and the corresponding weight information is calculated to obtain the feedback index value of each type of operation; the sum of the feedback index values ​​of all operations is calculated to obtain the user feedback index value. For example, this solution can count the number of each type of operation generated by the user on the target interactive content within a specified time window (such as 1 hour). j , and then calculate the user feedback index value using the following formula: ; Where m is the total number of operation types, is the weight information of each type of operation. A positive user feedback index value indicates positive feedback, a negative user feedback index value indicates negative feedback, and a user feedback index value of 0 indicates neutral.

[0102] As another possible implementation, this solution can also analyze the sentiment polarity of user speech content using a natural language processing (NLP) model, quantify it into numerical values, and then calculate the average value as the user feedback indicator. Specifically, this solution can use a sentiment analysis model (such as an LSTM or pre-trained BERT model) to classify the text converted from user speech, output the sentiment polarity value of each speech, and collect the sentiment polarity values ​​{y_1, y_2,..., y_n} of n speech messages within a specified time window (such as the last 10 minutes). The average sentiment polarity value of these n speech messages is calculated to obtain the user feedback indicator value.

[0103] When this solution obtains the user feedback index value through the above method, this solution compares the user feedback index value with the preset feedback index threshold. If the user feedback index value is less than the preset feedback index threshold, it is necessary to re-evaluate the current emotional state and re-output the target interactive content.

[0104] Specifically, the steps for updating the user's current emotional state and target interactive content are: re-collecting user image information, user voice information and user's environmental information, outputting the user's current emotional state category again based on the target emotion recognition model described above, obtaining an updated emotional state category, and judging whether the updated emotional state category has changed from the previous current emotional state category. If not, excluding the used interactive content from the original target interactive content set, and selecting the interactive content with the second highest similarity to the user's preference among the remaining content as the updated interactive content.

[0105] If the updated emotional state category is different from the previous current emotional state category, this solution searches for a new target interactive content set based on the updated emotional state category, and then determines the updated interactive content based on the similarity between the interactive content in the new target interactive content set and the user's preferences.

[0106] In the above implementation mode, this solution calculates the user feedback index value based on the user's feedback information on the target interactive content, and determines whether to update the interactive content based on the user feedback index value, thereby realizing dynamic adjustment of the user interactive content, further improving the accuracy of the user interactive content, and improving the user interactive experience.

[0107] As a possible implementation, this solution can also fine-tune the emotion recognition model and user preference model based on user feedback. For example, new user behavior data can be added to the training dataset and the model retrained to improve the model's accuracy in recognizing user emotions and preferences, thereby optimizing subsequent emotional interaction effects.

[0108] In an alternative implementation of this embodiment, this solution can also combine the user's environmental data with the user's current emotional state to determine interactive content. As one possible implementation, when the temperature sensor detects an indoor temperature below 18°C, the system determines it to be a low-temperature environment. In this case, if the emotion recognition model identifies the user as calm, the emotional response strategy will favor content related to warmth and comfort. For example, a display might show a cozy image of a warm-toned interior and recommend videos with hot drink recipes. If the user's emotional state is sad, in addition to the aforementioned content, soothing, slow-tempo warm-up music will be played to provide warmth and comfort. When the temperature is above 28°C, the system identifies it as a high-temperature environment. If the user is happy, the system will push cooling information, such as summer fruit recommendations and homemade cold drink tutorials, and display a refreshing, cool-toned theme on the interface. If the user is irritable, in addition to providing cooling advice, the voice response will be slowed down and spoken more calmly to help the user relieve their irritability.

[0109] In terms of light data, when light intensity falls below 100 Lux, the interface automatically dims to prevent eye irritation from strong light. If the user is watching a video or browsing messages, a prompt will pop up asking if they want to enable night mode. In night mode, the text on the interface is enlarged and the contrast is reduced to improve visibility. If the user interacts with the voice in this environment, the emotional response will be appropriately lowered in volume to create a quiet atmosphere. If the user is detected to be emotionally fatigued, the system may recommend audio content that can help with sleep. When light intensity exceeds 1000 Lux, the system considers the light too bright. At this time, if the user is on a video call, the system automatically adjusts the camera exposure parameters to ensure a clear and non-glaring image. In terms of emotional response, if the user expresses surprise or irritation, the system will recommend links to purchase some light-blocking products and display some lighting adjustment tips on the interface to help the user improve the environment.

[0110] In the above implementation mode, this solution combines the user's environmental data with the user's current emotional state category to realize the shortcomings of the interactive content and the adjustment of the interactive interface, making the user interaction more intelligent and further improving the user interaction experience.

[0111] Finally, this solution can also generate a personalized intelligent interface based on the user preference model and the current scenario requirements. For example, if the user preference model indicates that the user frequently pays attention to health information and it is morning, the camera will highlight the health information section on the main interface, including today's exercise suggestions, dietary recommendations, and other content. Furthermore, based on data from the ambient light sensor, the interface's brightness and contrast are automatically adjusted to provide a comfortable visual experience. The layout of interface elements utilizes reinforcement learning algorithms, such as the Deep Q Network (DQN). The DQN state is defined as the user's action history (the interface layout optimization frequency is set to trigger DQN inference once every five user actions, environmental data is updated every 10 seconds, and the brightness adjustment response time is ≤ 200ms). The action space is the pixel offset (±10px step size) for adjusting button positions, and the reward function is the click efficiency improvement rate (number of successful clicks divided by total number of clicks). The optimal layout is explored through the ε-greedy strategy. During the training process, the reward function weights α=0.6 (click efficiency) and β=0.4 (operation duration) are continuously optimized based on user operation feedback to improve user operation convenience. For example, if the user frequently clicks a function button, the algorithm adjusts the button to a position that is easier to click.

[0112] Figure 6 The present application provides a schematic structural block diagram of a user interaction device. It should be understood that the device is applied to the above description. Figures 3 to 5The device corresponds to the method embodiment executed in the embodiment and is capable of executing the steps involved in the aforementioned method. The specific functions of the device can be found in the description above, and a detailed description is omitted here to avoid repetition. The device includes at least one software functional module that can be stored in a memory in the form of software or firmware or embedded in the device's operating system (OS). Specifically, the device includes: an acquisition module 600, a determination module 610 and a search module 620, the acquisition module 600 is used to obtain the interaction information of the current user; wherein the user's interaction information includes user image information, user voice information and user's environmental information; the determination module 610 is used to determine the user's current emotional state category based on the user image information, user voice information and user's environmental information; the search module 620 is used to search for a target interactive content set associated with the current emotional state category in multiple interactive content sets based on the current emotional state category; wherein each emotional state category is associated with an interactive content set; the acquisition module 600 is also used to obtain the user's preference characteristics; the determination module 610 is also used to determine the target interactive content based on the user's preference characteristics and the target interactive content set; wherein the target interactive content is used to respond to the user's interactive information.

[0113] The user interaction device designed above uses a variety of interactive information during the user interaction process (such as user image information, user voice information, and user environment information) to accurately identify the user's emotional state, and determine the current interactive response content with the user based on the user's emotional state and user preference characteristics, thereby achieving deep emotional communication and interaction with the user, providing the user with more accurate and emotionally valuable response content, and significantly improving the user's interactive experience.

[0114] According to some embodiments of the present application, Figure 7 As shown, the present application provides an electronic device 7, including: a processor 701 and a memory 702, the processor 701 and the memory 702 are interconnected and communicate with each other through a communication bus 703 and / or other forms of connection mechanisms (not shown), the memory 702 stores a computer program executable by the processor 701, and when the computing device is running, the processor 701 executes the computer program to execute any optional implementation method, such as steps S300 to S340: obtaining the current user's interaction information; determining the user's current emotional state category based on the user's image information, the user's voice information and the user's environmental information; searching for a target interactive content set associated with the current emotional state category in multiple interactive content sets based on the current emotional state category; obtaining the user's preference characteristics; and determining the target interactive content based on the user's preference characteristics and the target interactive content set.

[0115] The present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method in any of the aforementioned optional implementations is executed.

[0116] The storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0117] The present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method in any optional implementation manner.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A user interaction method, characterized in that: The method comprises: Obtaining the current user's interactive information; wherein the user's interactive information includes user image information, user voice information, and user's environment information; Determining the user's current emotional state category based on user image information, user voice information, and user environment information; Searching for a target interactive content set associated with the current emotional state category in multiple interactive content sets according to the current emotional state category; wherein each emotional state category is associated with an interactive content set; Obtaining preference characteristics of the user; Target interactive content is determined based on the user's preference characteristics and the target interactive content set; wherein the target interactive content is used to respond to the user's interactive information.

2. The method according to claim 1, characterized in that Determining the current emotional state of the user based on the user image information, the user voice information, and the user's environment information includes: The user image information, user voice information and user environment information are input into a pre-trained target emotion recognition model; wherein the target emotion recognition model is used to obtain the current emotional state of the user.

3. The method according to claim 2, characterized in that The training process of the target emotion recognition model includes: Obtain a target data set; wherein the target data set includes interaction information of multiple different users, and the interaction information of each user includes user image information, user voice information, user environment information, and corresponding emotion category label; Dividing the target data set into multiple training groups, with a preset number of interaction information as a group; Selecting a target group from multiple training groups as a current training group; Performing feature extraction on each interactive information in the current training group to obtain a current training sample group, wherein the current training sample group includes a plurality of target training samples, each target training sample includes a fused feature vector and a corresponding emotion category label, wherein the fused feature vector is generated based on user image sample features, user voice sample features, and environment sample features in the corresponding training sample; Calculating the probability distribution of each type of emotional state corresponding to each target training sample in the current training sample group according to the fusion features of the target training sample; Calculating the training loss value corresponding to the current training sample group through a loss function according to the probability distribution of each type of emotional state and the corresponding emotional category label corresponding to each target training sample in the current training sample group; Update various training parameters of the model according to the training loss value; Determine whether the current cumulative number of iterations exceeds a preset number of iterations, or whether the various training parameters of the model have converged; If it is determined that the current cumulative number of iterations exceeds the preset number of iterations or the various training parameters of the model converge, the trained target emotion recognition model is obtained; If it is determined that the current cumulative number of iterations does not exceed the preset number of iterations and the various training parameters of the model do not converge, the process returns to executing the step of selecting a target group from the multiple training groups as the current training group.

4. The method according to claim 3, characterized in that The step of calculating the probability distribution of each type of emotional state corresponding to the target training sample according to the fused feature vector of each target training sample in the current training sample group includes: Perform fusion feature learning on the fusion feature vector of each target training sample through the ReLU activation function of the first fully connected layer and the second fully connected layer to obtain the output fusion feature learning vector corresponding to each target training sample; Through the softmax activation function of the output layer, the probability distribution of each target training sample corresponding to each type of emotional state is calculated according to the fusion feature learning vector corresponding to each target training sample.

5. The method according to claim 3, characterized in that The step of extracting features from each interactive information in the current training group to obtain a current training sample group includes: After preprocessing the user image information of the current training group, extracting local facial features and global features from the preprocessed user image information through a convolutional neural network to obtain user image sample features; After performing frame-by-frame windowing preprocessing on the user voice information in the current training group, extracting the temporal features of the preprocessed user voice information through a long short-term memory network to obtain user voice sample features; Normalizing the environment information of the users in the current training group to obtain environment sample features; The user image sample features, user voice sample features and environment sample features are combined into a fusion feature vector.

6. The method according to claim 1, wherein The determining of target interactive content according to the user's preference characteristics and the target interactive content set includes: Calculating the similarity between each interactive content in the target interactive content set and the user's preference characteristics; The interactive content with the highest similarity is determined as the target interactive content.

7. The method according to claim 1, characterized in that After determining the target interactive content according to the user's preference characteristics and the target interactive content set, the method further includes: Collecting user interaction feedback information on the target interactive content; Calculating a user feedback index value based on the user's interactive feedback information on the target interactive content; wherein the user feedback index value is used to represent the user's positive feedback on the target interactive content; When the user feedback index value is less than a preset feedback index threshold, the user's current emotional state and target interaction content are updated.

8. The method according to claim 7, characterized in that The collecting of user interaction feedback information on the target interactive content includes: Counting the number of each type of feedback operation generated by the user within a preset time window to obtain interactive feedback information; wherein a plurality of feedback operation types are preset, weight information belonging to positive feedback operation types is positive, and weight information belonging to negative feedback operation types is negative; the preset time window represents a preset time range after the target interactive content is output; The calculating of the user feedback index value according to the user's interactive feedback information on the target interactive content includes: Obtain the weight information corresponding to each type of feedback operation generated by the user; Calculate the product of the number of operations of each type and the corresponding weight information to obtain the feedback index value of each type of operation; The sum of the feedback index values ​​of all operations is calculated to obtain the user feedback index value.

9. The method according to claim 1, characterized in that in, Obtaining the user's preference characteristics, including: A user preference model is constructed using a multi-layer perceptron. The user preference model is trained based on user operation characteristics, basic user information, and preference characteristics set by the user. The user operation characteristics include click location, click frequency, browsed content type, and browsing time. The basic user information includes user age, gender, and occupation. The training process of the user preference model includes: using a stochastic gradient descent algorithm and a mean square error loss function to iteratively optimize the mapping relationship between user operation characteristics, user basic information and preference characteristics actively set by the user to generate the user preference model.

10. The method according to claim 9, characterized in that The method further comprises: If the user is a new user, the preset general preference module is matched according to the basic user information entered by the new user. The general preference templates corresponding to different users' ages, occupations and genders are different; Based on the subsequent operation data of the new user, the preference weight of the general preference template corresponding to the new user is dynamically adjusted through the gradient descent method.

11. The method according to claim 1, wherein The user's environment information includes ambient temperature and ambient light brightness. The searching, according to the current emotional state category, for a target interactive content set associated with the current emotional state category in multiple interactive content sets includes: When the ambient temperature of the user is lower than a preset ambient temperature and the current emotional state category is calm, selecting a recommendation set formed by warm-toned images and / or hot drinks as the target interactive content set; When the ambient light brightness of the user is greater than the preset light brightness and the current emotional state category is irritability, a recommendation set formed by pushing sunshade product information and / or soothing voice is selected as the target interactive content set.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

13. A user interaction system, characterized in that: The user interaction system includes an image acquisition device, a voice acquisition device, an environment acquisition device and a control device, wherein the control device is electrically connected to the image acquisition device, the voice acquisition device and the environment acquisition device respectively; The visual acquisition device is used to acquire user image information and transmit the user image information to the control device; The voice collection device is used to collect user voice information and transmit the user voice information to the control device; The environment collection device is used to collect the user's environment information and transmit the environment information to the control device; The control device is used to determine the current emotional state category of the user based on the user's visual information, the user's voice information and the user's environmental information; search for a target interactive content set associated with the current emotional state category in multiple interactive content sets based on the current emotional state category; wherein each emotional state category is associated with an interactive content set; obtain the user's preference characteristics; determine the target interactive content based on the user's preference characteristics and the target interactive content set; wherein the target interactive content is used to respond to the user's interactive information.

14. The user interaction system according to claim 13, characterized in that: The control device includes a user preference model storage module and a preference feature generation module. The user preference model storage module is used to store the user preference model obtained based on multi-layer perceptron training; the preference feature generation module is used to generate user preference features through the user preference model based on user operation data and basic information.

Images