Dish information recommendation method and device and household appliance

Abstract

The invention relates to the technical field of natural language understanding, and discloses a recommendation method for dish information. The recommendation method comprises the steps of obtaining state information and action information of multi-modal interaction data; wherein the state information is used for representing description of the current environment state, and the action information is used for representing actions or decisions which can be taken in the corresponding state; obtaining an attention feature vector representing an association relationship between the state information and the action information; obtaining a dish interaction fusion feature vector based on user feedback information of the historical dish information; and determining optimized dish information corresponding to the multi-modal interaction data according to the attention feature vector and the dish interaction fusion feature vector. According to the recommendation method, the optimized dish information is determined based on the attention feature vector and the dish interaction fusion feature vector, the intent target of user interaction is better matched, and the accuracy of dish information recommendation is improved. The invention further discloses a recommendation device for the dish information and the household appliance.

Description

Technical Field

[0001] The present application relates to the technical field of natural language understanding, for example, to a method, device and household appliance for recommending dish information. Background Art

[0002] At present, the application of information recommendation technology in smart home scenarios is becoming increasingly important and widespread, but in some smart scenarios, such as food information recommendation scenarios, the accuracy of food information recommendation is still low.

[0003] In order to improve the accuracy of dish information recommendation, related technologies usually use single interaction data to build traditional deep learning models, or multimodal interaction data to build traditional deep neural network models to generate dish information to solve semantic and grammatical problems. However, this method is difficult to fully capture the semantic association features, grammar and context information of single interaction data or multimodal interaction data. Therefore, the accuracy of dish information recommendation achieved by related technologies is still low.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] The embodiments of the present disclosure provide a method, device and household appliance for recommending dish information, so as to improve the accuracy of dish information recommendation.

[0007] In some embodiments, a method for recommending dish information includes: obtaining state information and action information of multimodal interaction data; wherein the state information is used to represent a description of the current environmental state, and the action information is used to represent actions or decisions that can be taken under the corresponding state; obtaining an attention feature vector that characterizes the association between the state information and the action information; obtaining a dish interaction fusion feature vector based on user feedback information of historical dish information; and determining the optimized dish information corresponding to the multimodal interaction data based on the attention feature vector and the dish interaction fusion feature vector.

[0008] Optionally, obtaining an attention feature vector characterizing the association relationship between state information and action information includes: constructing a first association vector having a first association relationship between state information; constructing a second association vector having a second association relationship between action information; constructing a third association vector having a third association relationship between state information and action information; and determining the attention feature vector based on the first association vector, the second association vector and the third association vector.

[0009] Optionally, optimized dish information corresponding to the multimodal interaction data is determined based on the attention feature vector and the dish interaction fusion feature vector, including: when the attention feature vector and the dish interaction fusion feature vector successfully match the features, generating multiple dish information corresponding to the attention feature vector and the dish interaction fusion feature vector; sorting the multiple dish information to obtain sorting results of the multiple dish information; and using the dish information located at a preset sorting position in the sorting result as the optimized dish information corresponding to the multimodal interaction data.

[0010] Optionally, based on user feedback information of historical dish information, a dish interaction fusion feature vector is obtained, including: vectorizing the user feedback information to obtain a user feedback embedding vector; end-to-end encoding and decoding the user feedback embedding vector to obtain the encoded and decoded user feedback embedding vector; performing multi-dimensional reward analysis on the encoded and decoded user feedback embedding vector to obtain a user dish feedback information feature vector with a dish reward value; and aggregating the user dish feedback information feature vector into a dish interaction fusion feature vector.

[0011] Optionally, obtaining state information and action information of multimodal interaction data includes: obtaining total text interaction data of the multimodal interaction data; extracting state features and action features in the total text interaction data; vectorizing the state features and action features respectively to obtain state text feature vectors corresponding to the state features and action text feature vectors corresponding to the action features; binarizing the state text feature vectors and action text feature vectors respectively to obtain state information corresponding to the state text feature vectors and action information corresponding to the action text feature vectors.

[0012] Optionally, the total text interaction data of the multimodal interaction data is obtained in the following manner: the multimodal interaction data is parsed to obtain text summary interaction data; wherein the text summary interaction data includes one or more of the text interaction data, voice-text interaction data and image-text interaction data; the text summary interaction data is processed in a unified format to obtain the total text interaction data.

[0013] Optionally, the multimodal interaction data includes one or more of text interaction data, image interaction data, voice interaction data, video interaction data, and audio-video interaction data; parsing the multimodal interaction data to obtain text summary interaction data includes: parsing the text interaction data to obtain text interaction data; parsing the voice interaction data and / or audio-video interaction data to obtain voice-text interaction data; parsing one or more of the image interaction data, video interaction data, and audio-video interaction data to obtain image-text interaction data.

[0014] In some embodiments, a device for recommending dish information includes: a first acquisition module, configured to obtain state information and action information of multimodal interaction data; wherein the state information is used to represent the description of the current environmental state, and the action information is used to represent the actions or decisions that can be taken under the corresponding state; a second acquisition module, configured to obtain an attention feature vector that characterizes the association between the state information and the action information; a third acquisition module, configured to obtain a dish interaction fusion feature vector based on user feedback information of historical dish information; and a determination module, configured to determine the optimized dish information corresponding to the multimodal interaction data based on the attention feature vector and the dish interaction fusion feature vector.

[0015] In some embodiments, the device for recommending dish information includes a processor and a memory storing program instructions, and the processor is configured to execute the above-mentioned method for recommending dish information when running the program instructions.

[0016] In some embodiments, the home appliance includes: a home appliance body; and, the device for recommending dish information as described above, installed in the home appliance body.

[0017] The method, device and home appliance for recommending dish information provided by the embodiments of the present disclosure can achieve the following technical effects:

[0018] In this embodiment, by obtaining the attention feature vector that represents the state information and action information, the semantic association features, grammar and context information of the multimodal interaction data can be more comprehensively captured, so that the multimodal interaction data has a stronger information expression capability. At the same time, because the dish interaction fusion feature vector contains user feedback information, the dish interaction fusion feature vector has a richer user information expression capability. Therefore, the optimized dish information determined based on the attention feature vector and the dish interaction fusion feature vector is more consistent with the intended goal of the user interaction, and the accuracy of the dish information recommendation is improved.

[0019] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0021] Figure 1 is a schematic diagram of the structure of a household appliance provided by an embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of a method for recommending dish information provided by an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of another method for recommending dish information provided by an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of a device for recommending dish information provided by an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of another device for recommending dish information provided by an embodiment of the present disclosure;

[0026] Figure 6 It is a schematic diagram of a household appliance product provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0028] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0029] Unless otherwise stated, the term "plurality" means two or more.

[0030] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0031] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0032] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0033] The following is an explanation of the technical terms used in the embodiments of the present disclosure:

[0034] Deep reinforcement learning: It is a learning framework that combines deep learning and reinforcement learning. It can automatically learn high-level abstract representations of complex tasks from input data, and make autonomous decisions and actions through reinforcement learning algorithms, thereby achieving end-to-end learning and control.

[0035] Deep learning: A machine learning method based on deep neural networks. It learns the feature representation of input data through the connection of multiple layers of neurons. Deep learning can use a large amount of data to train the model, thereby realizing the automatic recognition and classification of complex data such as images, speech, and natural language. Deep learning has a wide range of applications in computer vision, speech recognition, natural language processing and other fields.

[0036] Reinforcement learning: is an algorithm that learns the optimal action strategy through the interaction between the agent and the environment. In reinforcement learning, the agent interacts with the environment continuously, observes the state of the environment and takes different actions to obtain rewards or penalties. The agent adjusts its strategy based on these rewards or penalties and eventually learns the optimal behavior strategy.

[0037] Agent: refers to the entity that performs the learning task, and it learns the optimal strategy by interacting with the environment. The agent observes the current state at each time step and selects an action to execute based on the current state. The environment refers to the external environment in which the agent is located, which contains all the information that the agent can perceive. The environment can be discrete or continuous, and can be deterministic or random.

[0038] Attention mechanism: It imitates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external sensation to increase the fineness of observation of some areas, and can use limited attention resources to quickly filter out high-value information from a large amount of information. The attention mechanism can quickly extract important features of sparse data.

[0039] Reward modeling: is a reinforcement learning-based approach that fine-tunes the model by optimizing the expected reward.

[0040] GloVe: The full name is Global Vectors for Word Representation. It is a word representation tool based on count-based & overall statistics.

[0041] like Figure 1 As shown, the household appliance 1 can be a device with multiple interaction modes such as audio acquisition, video acquisition, and user interaction interface, and the household appliance includes an intelligent body (for example, a refrigerator including an intelligent body or a range hood including an intelligent body, etc.). Taking the refrigerator as an example, the refrigerator is provided with a microphone 10 or a pickup that can collect audio, a camera 20 that can shoot video, a speaker 30 that can interact with the user's voice, and an interactive screen 40 that can interact with the user's text or graphical interface. The interactive screen 40 can be set on the door of the refrigerator; the user's operation is recorded by the camera 20 to form a video; the speaker 30 and the microphone 10 are combined to interact with the user in the form of question and answer. In this way, through one of the above-mentioned acquisition methods or multiple combined acquisition methods, the refrigerator can collect multimodal interaction data including voice interaction data, text interaction data, image interaction data, video interaction data, and audio and video interaction data.

[0042] In addition, the refrigerator can also interact with the user through the corresponding client 2. The client 2 can be a mobile phone or software on the mobile phone; the software includes one or more of an APP (Application), an information collection tool, and a touch screen. Taking the client 2 as an example, the user can interact with the refrigerator through text, voice, video, audio and video, or images through the mobile phone. Figure 1 As shown, the home appliance 1 and the client 2 can be connected via a wireless signal.

[0043] Multimodal data can be transmitted via the Internet, WWW, Web, WIFI, Bluetooth, etc. Various types of interactive data such as voice, text, image, video, audio and video constitute multimodal interactive data, which can be real-time online or offline interactive data, or stored historical interactive data.

[0044] The home appliance 1 scenario includes the interaction between the user and the home appliance 1, and the interaction between the user and the corresponding client 2 of the home appliance. The disclosed embodiment obtains the multimodal interaction data generated in the home appliance 1 scenario, fully mines the state information and action information of the multimodal interaction data, and at the same time, constructs a dish interaction fusion feature vector including user feedback information, thereby obtaining the correlation and complementarity between the multimodal interaction data and the dish interaction fusion feature vector at a deeper level, strengthening the semantic feature representation, and effectively improving the accuracy of the recommendation of dish information of the home appliance.

[0045] Combination Figure 1 The home appliance shown in the figure, the embodiment of the present disclosure provides a method for recommending dish information, and the execution subject of the recommendation method can be a processor. Figure 2 As shown, the recommended method includes:

[0046] S201: The processor obtains state information and action information of multimodal interaction data.

[0047] In this step, the multimodal interaction data includes one or more of text interaction data, image interaction data, voice interaction data, video interaction data, and audio and video interaction data. The processor may collect the multimodal interaction data in the following manner: the processor obtains the multimodal interaction data through the interaction mode. The interaction mode includes but is not limited to Bluetooth, WIFI, Internet, Web, microphone, pickup, and software.

[0048] Exemplarily, text interaction data is collected through the interactive screen of the client and / or the home appliance device. The client may include mobile phones, pads, PCs and other devices. In addition to the text data generated on the application, it also includes text from customer service, web, and text data on mini-programs or official accounts.

[0049] Exemplarily, the image interaction data may be collected by one or more methods including a mobile phone, a camera of a home appliance, or a user interaction interface of the home appliance.

[0050] Exemplarily, voice interaction data can be collected by microphones of mobile phones and / or household appliances, or can be wirelessly connected to other sound collection units via WiFi to obtain voice from sound collection units on other devices. The microphone can be a single microphone or a microphone array.

[0051] Exemplarily, the video interaction data may be collected by a camera of a client and / or a home appliance, for example, through a mobile phone App, a camera, Bluetooth, etc.

[0052] Exemplarily, the audio and video interaction data may be collected by a combination of a microphone and a camera of the home appliance.

[0053] Through the above-mentioned interactive method, multimodal interactive data collection tasks such as multiple channels and clients are completed to ensure the integrity and multimodal cognitive characteristics of multimodal interactive data.

[0054] Exemplarily, the multimodal interaction data may be historical multimodal interaction data or real-time multimodal interaction data. Specifically, the multimodal interaction data is real-time interaction data. By collecting real-time interaction data, the home appliance can recommend dish information according to the user's current needs.

[0055] Among them, the state information is used to represent the description of the current environment state, and the current environment refers to the environment where the agent is located, that is, the environment where the agent is located can be represented by the state information (such as the ambient temperature, ambient brightness, region and user status of the agent). The state information can be discrete or continuous, depending on the specific application scenario. The action information is used to represent the actions or decisions that can be taken in the corresponding state, and may include the motion information of the agent (such as but not limited to the speed, motion direction and motion state, etc.), the instruction information of the specific selection object, or the probability distribution of specific options (such as but not limited to the probability of option a is 0.15, the probability of option b is 0.25, the probability of option c is 0.35, and the probability of option d is 0.45) and other different forms, which are not specifically limited. The action information can also be discrete or continuous, depending on the specific application scenario. For example: in the embodiment of the present disclosure, the action information can represent the interaction between the user and the agent, the control instructions uploaded by the user for the corresponding state, and the multimodal interaction data uploaded by the user. The state information can represent the data displayed by the home appliance, etc. If it corresponds to the field of game AI, action information may refer to control instructions uploaded by users for game characters (such as but not limited to up, down, left, and right movement control), and state information may be image display data of each frame corresponding to the game screen, etc. State information and action information can generally be represented by vectors, arrays, matrices, or higher-order tensors, etc. In the embodiment of the present disclosure, state information and action information are represented by vectors.

[0056] It should be noted that, in the embodiments of the present disclosure, the words such as “for example”, “illustrate” and “take… as an example” are merely for explanation and not for specific limitation.

[0057] There is a correspondence between the state information and the action information. Different actions can be selected according to different states, and different actions will lead to different states. Therefore, the correspondence between the state information and the action information can be expressed in various forms, and there is no specific limitation. For example, the state information and the action information can be given the same number to indicate the correspondence: the state information is numbered as state information 1, state information 2, state information 3, etc., and the action information is numbered as action information 1, action information 2, and action information 3, etc., then state information 1 and action information 1 can be corresponding information, state information 2 and action information 2 can be corresponding information, and state information 3 and action information 3 can be corresponding information. In this way, the semantic expression of multimodal interaction data can be enhanced through the correspondence between the state information and the action information.

[0058] Take the multimodal interaction data as real-time voice data collected in Qingdao in summer as an example: the real-time voice data corresponds to the meaning of "I want to eat Kung Pao Chicken". Since the state information and action information of the multimodal interaction data are obtained in this embodiment, the dish recommended according to the state information and action information can be Kung Pao Chicken that satisfies the taste of Qingdao people and has a lighter taste. Because the state information and action information are obtained, the recommended dish information is more consistent with the current interaction needs.

[0059] S202: The processor obtains an attention feature vector representing the association between state information and action information.

[0060] Exemplarily, the attention feature vector that characterizes the correlation between state information and action information is obtained through a pre-trained deep reinforcement learning model with an attention mechanism. The disclosed embodiment comprehensively considers the state information and action information of multimodal interaction data, so as to provide richer state representation and action representation. Through the deep reinforcement learning model, the deep correlation between these state representations and action representations can be learned, and the potential effects of different dynamic information and action information can be more accurately evaluated and predicted, which improves the understanding and abstraction ability of state information and action information, and then more accurately outputs the recommended dish information.

[0061] In this step, the state information and action information are linked through the deep neural network in deep reinforcement learning, and the attention mechanism is used to build the association relationship between the state information and the action information to further capture the important feature information of the global information of the state information and the action information, thereby enhancing the semantic feature expression ability of the state information and the action information.

[0062] S203: The processor obtains a dish interaction fusion feature vector based on user feedback information of historical dish information.

[0063] Among them, historical dish information refers to pre-set dish information, which is stored in the dish information library of the home appliance. For example, dish information can be Kung Pao Chicken, Steamed Sea Bass, Braised Spare Ribs and Scrambled Eggs with Tomatoes, etc. Historical dish information can be dish information that has been recommended to the user, or dish information that has not been recommended to the user. It should be noted that dish information includes not only the name of the dish, but also the cuisine, the health status of the ingredients corresponding to the dish, the management status of the ingredients corresponding to the dish, the production map of the dish and the nutrition report of the dish, etc.

[0064] User feedback information is used to indicate the multi-dimensional feedback of users on dish information. Among them, multi-dimensional feedback can include single evaluation and comprehensive evaluation among preference evaluation, price evaluation, taste evaluation and appearance evaluation. Exemplarily, user feedback information can be represented by the score given by the user to the dish, for example, "Kung Pao Chicken, 70 points", "Tomato Scrambled Eggs, 90 points" and "Steamed Sea Bass, 80 points", and the above scores can show that the user's comprehensive evaluation of "Tomato Scrambled Eggs, 90 points" is higher. It can also be represented by text descriptions, such as "Kung Pao Chicken, satisfied", "Tomato Scrambled Eggs, dissatisfied", and the above text descriptions can more intuitively express the user's evaluation of the dish. Specifically, the user feedback information is represented by the score given by the user to the dish.

[0065] The dish interaction fusion feature vector is used to indicate the feature vector corresponding to the fusion information of the dish information and the user feedback information. The dish interaction fusion feature vector includes not only the dish information features, but also the feedback information features after interaction with the user, which is more semantically rich and has stronger expression ability. For example, the meaning corresponding to the dish interaction fusion feature vector can be "Kung Pao Chicken, 70 points", "Tomato Scrambled Eggs, 90 points" and "Steamed Sea Bass, 80 points".

[0066] S204, the processor determines the optimized dish information corresponding to the multimodal interaction data according to the attention feature vector and the dish interaction fusion feature vector.

[0067] In this embodiment, by obtaining the attention feature vector that represents the state information and action information, the semantic association features, grammar and context information of the multimodal interaction data can be more comprehensively captured, so that the multimodal interaction data has a stronger information expression capability. At the same time, because the dish interaction fusion feature vector contains user feedback information, the dish interaction fusion feature vector has a richer user information expression capability. Therefore, the optimized dish information determined based on the attention feature vector and the dish interaction fusion feature vector is more consistent with the intended goal of the user interaction, and the accuracy of the dish information recommendation is improved.

[0068] Optionally, the processor determines the optimized dish information corresponding to the multimodal interaction data based on the attention feature vector and the dish interaction fusion feature vector, including: when the attention feature vector and the dish interaction fusion feature vector feature match successfully, the processor generates multiple dish information corresponding to the attention feature vector and the dish interaction fusion feature vector. Sort the multiple dish information to obtain the sorting results of the multiple dish information. The dish information at the preset sorting position in the sorting result is used as the optimized dish information corresponding to the multimodal interaction data.

[0069] Exemplarily, the feature matching calculation task of the attention feature vector and the dish interaction fusion feature vector is realized by the Q value function in the pre-trained deep reinforcement learning model. Specifically, the attention feature vector and the dish interaction fusion feature vector are input into the pre-trained deep reinforcement learning model, and the feature matching state of the attention feature vector and the dish interaction fusion feature vector is calculated by the Q value function in the pre-trained deep reinforcement learning model.

[0070] Among them, the Q value represents the feature matching value of the attention feature vector and the dish interactive fusion feature vector. When the Q value is greater than or equal to the preset value, it is determined that the feature matching of the attention feature vector and the dish interactive fusion feature vector is successful. When the Q value is less than the preset value, it is determined that the feature matching of the attention feature vector and the dish interactive fusion feature vector fails. The preset value can be set according to the actual needs of the user. The following is an explanation of the meaning of the dish interactive fusion feature vector including "Kung Pao Chicken (Sichuan cuisine), 90 points", "Kung Pao Chicken (Shandong cuisine), 92 points", "Steamed Sea Bass (Hunan cuisine), 90 points", "Braised Spare Ribs (home cooking), 92 points" and "Tomato Scrambled Eggs (home cooking), 80 points". If the meaning of the attention feature vector is "Qingdao", "eat" and "Kung Pao Chicken", then the feature matching values ​​of "Kung Pao Chicken (Sichuan cuisine), 90 points" and "Kung Pao Chicken (Shandong cuisine), 92 points" in the dish interactive fusion feature vector are relatively high, among which the feature matching value with "Kung Pao Chicken (Shandong cuisine), 92 points" is the highest. However, "steamed sea bass (Hunan cuisine), 90 points", "braised spare ribs (home cooking), 92 points", and "scrambled eggs with tomatoes (home cooking), 80 points" may fail to match the feature of the attention feature vector meaning "Qingdao", "eat", and "Kung Pao chicken". If the meaning of the attention feature vector is "Qingdao", "eat", and "home cooking", then the feature matching values ​​of "braised spare ribs (home cooking), 92 points" and "scrambled eggs with tomatoes (home cooking), 80 points" in the interactive fusion feature vector of the dish are relatively high. Among them, since "braised spare ribs (home cooking), 92 points" has the highest score, the feature matching value with "braised spare ribs (home cooking), 92 points" is the highest. However, "Kung Pao chicken (Sichuan cuisine), 90 points" and "steamed sea bass (Hunan cuisine), 90 points" may fail to match the feature of the attention feature vector meaning "Qingdao, eat, home cooking". If the meaning of the attention feature vector is "Qingdao", "eat" and "generally loved by everyone", then the feature matching values ​​of "Kung Pao Chicken (Sichuan cuisine), 90 points", "Kung Pao Chicken (Shandong cuisine), 92 points", "Steamed Sea Bass (Hunan cuisine), 90 points" and "Braised Spare Ribs (home cooking), 92 points" in the interactive fusion feature vector of the dishes are relatively high, among which the feature matching value with "Kung Pao Chicken (Shandong cuisine), 92 points" is the highest.

[0071] It should be noted that, in addition to the dish name and cuisine attributes, the dish information can also include the health status of the ingredients corresponding to the dish, the management status of the ingredients corresponding to the dish, the production map of the dish, and the nutrition report of the dish. Therefore, the Q value will also reflect the feature matching status of the above content. Take the dish information including the dish name, cuisine and nutrition report as an example for explanation. If the interactive fusion feature vector of the dish includes "roast duck with day lily (Hangzhou cuisine, rich in protein, calcium and vitamin C), 90 points", "fried pork slices with broccoli (home cooking, rich in protein and vitamin C), 80 points", "fried fungus with green pepper (home cooking, rich in iron and vitamin K), 85 points" and "fried hairtail (Northeast cuisine, rich in unsaturated fatty acids), 91 points". If the meaning of the attention feature vector is "protein", "eat" and "Hangzhou cuisine", then the feature matching value of "roast duck with day lily (Hangzhou cuisine, rich in protein, calcium and vitamin C), 90 points" in the interactive fusion feature vector of the dish is the highest. If the meaning of the attention feature vector is "protein", "eat" and "home-cooked food", then the feature matching value of "Fried broccoli with pork slices (home-cooked food, rich in protein and vitamin C)" in the interactive fusion feature vector with the dishes is the highest. If the meaning of the attention feature vector is "protein" and "eat", then the feature matching values ​​of "Roast duck with day lily (Hangzhou cuisine, rich in protein, calcium and vitamin C), 90 points" and "Roast duck with broccoli with pork slices (home-cooked food, rich in protein and vitamin C), 80 points" in the interactive fusion feature vector with the dishes are relatively high. Among them, since the score of roast duck with day lily is higher, the feature matching value with "Roast duck with day lily (Hangzhou cuisine, rich in protein, calcium and vitamin C), 90 points" is the highest.

[0072] Exemplarily, when the pre-trained deep reinforcement learning model outputs a Q value, the Q value is normalized so that the Q value is a value between 0 and 1.

[0073] Exemplarily, when the processor successfully matches the attention feature vector and the dish interactive fusion feature vector, the dish information corresponding to the Q value is determined according to the preset corresponding relationship. For example, when the Q value is equal to 0.2, the corresponding dish information is scrambled eggs with tomatoes. When the Q value is equal to 0.3, the corresponding dish information is Kung Pao Chicken. When the Q value is equal to 0.4, the corresponding dish information is steamed sea bass.

[0074] In the case where the dish information includes multiple attributes, if the Q value is equal to 0.2, the corresponding dish information includes "scrambled eggs with tomatoes; healthy; tomatoes are stored in the first storage space of the refrigerator, and eggs are stored in the second storage space of the refrigerator; the recipe for scrambled eggs with tomatoes is XXX, and the nutrition report is XXX".

[0075] Exemplarily, the plurality of dish information may be sorted by sorting the Q values ​​corresponding to the plurality of dish information. Specifically, the plurality of dish information may be sorted from large to small according to the Q values ​​corresponding to the plurality of dish information.

[0076] The preset sorting position is set according to the actual needs of the user. For example, the preset sorting position can be set to the first 50, the first 20, the first 10, the first 8 or the first 6 to provide more choices for the user. The process of selecting the dish information at the preset sorting position in the sorting result can be processed by the Top N algorithm.

[0077] In this embodiment, by calculating the feature matching value of the attention feature vector and the dish interaction fusion feature vector, the correlation and interactivity between the attention feature vector and the dish interaction fusion feature vector are comprehensively considered, so that the correlation and interactivity between the attention feature vector and the dish interaction fusion feature vector are reflected through the feature matching value, making the recommended dish information more accurate. At the same time, by calculating the feature matching value of the intention feature vector and the dish interaction fusion feature vector, redundant vectors in the dish interaction fusion feature vector that are irrelevant to the attention feature vector can also be screened out, which is more conducive to the calculation of the model, thereby improving the efficiency of the recommendation of dish information for home appliances.

[0078] Optionally, when the feature matching between the attention feature vector and the dish interaction fusion feature vector fails, the processor outputs "0" or "failure", which is not limited here.

[0079] Optionally, the processor obtains a dish interaction fusion feature vector based on user feedback information of historical dish information, including: the processor vectorizes the user feedback information to obtain a user feedback embedding vector. The processor performs end-to-end encoding and decoding on the user feedback embedding vector to obtain the encoded and decoded user feedback embedding vector. The processor performs multi-dimensional reward analysis on the encoded and decoded user feedback embedding vector to obtain a user dish feedback information feature vector with a dish reward value. The processor aggregates the user dish feedback information feature vector into a dish interaction fusion feature vector.

[0080] The user feedback embedding vector is the user feedback information after vector processing. The user feedback embedding vector is the user feedback embedding vector after end-to-end encoding and decoding. The user dish feedback information feature vector is the user feedback embedding vector after multi-dimensional reward analysis, that is, the user feedback embedding vector with dish reward value.

[0081] Exemplarily, a dish interaction fusion feature vector is obtained through a pre-trained reward model. The pre-trained reward model can be a pre-trained loss calculation model or a pre-trained Reward reward model. Specifically, the pre-trained reward model is a pre-trained Reward reward model, so that a multi-dimensional evaluation and analysis of user feedback information is performed (for example, the multi-dimensional evaluation and analysis includes preference evaluation, price evaluation, taste evaluation, and appearance evaluation) to obtain a dish reward value corresponding to the user feedback information. The dish reward value is used to indicate the degree of difference between the user feedback embedding vector and the user dish feedback information feature vector. The larger the reward value, the smaller the difference value.

[0082] Exemplarily, the pre-trained reward model can adopt neural network structures such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks), LSTM (Long Short Term Memory), and Transformer (a neural network). Specifically, the pre-trained reward model adopts an end-to-end encoding and decoding Transformer neural network structure to achieve end-to-end encoding and decoding of the user feedback embedding vector to obtain the encoded and decoded user feedback embedding vector. At the same time, the Transformer neural network structure includes a loss function, which can further represent the degree of difference between the user feedback embedding vector and the user dish feedback information feature vector.

[0083] Exemplarily, the pre-trained reward model is used to continuously provide feedback on the degree of difference between the user feedback embedding vector and the user dish feedback information feature vector to reduce the difference value between the user feedback embedding vector and the user dish feedback information feature vector, and continuously optimize the output user dish feedback information feature vector, making the expression of user feedback information more accurate.

[0084] The pre-trained reward model may refer to a reward model obtained by training the initial reward model, and the initial reward model may refer to an untrained reward model. The target reinforcement model may be trained based on historical dish information and user feedback information corresponding to the historical dish information.

[0085] Exemplarily, the processor aggregates the user dish feedback information feature vector into a dish interaction fusion feature vector. The dish interaction fusion feature vector is historical data, that is, the dish interaction fusion feature vector corresponding to all historical dish information, which is stored in the dish interaction fusion feature library. It should be noted that for each additional user feedback information, the pre-trained reward model is used to perform the above processing on the user feedback information, so as to ensure that the dish interaction fusion feature vector is the latest.

[0086] In this embodiment, by updating the information in the dish interaction fusion feature library, the dish interaction fusion feature vector includes the latest data, so that the calculation result of the Q value is more accurate.

[0087] Optionally, the processor obtains the attention feature vector representing the association relationship between the state information and the action information, including: constructing a first association vector having a first association relationship between the state information. Constructing a second association vector having a second association relationship between the action information. Constructing a third association vector having a third association relationship between the state information and the action information. Determining the attention feature vector according to the first association vector, the second association vector, and the third association vector.

[0088] Exemplarily, the attention feature vector is obtained through an attention mechanism, wherein corresponding semantic weights are assigned to the state information and the action information through the attention mechanism, and each semantic weight represents an association relationship.

[0089] Exemplarily, the attention feature vector may be one or more of the first association vector, the second association vector, and the third association vector. In this way, it is possible to improve computational efficiency while obtaining a stronger attention feature vector.

[0090] Optionally, the processor determines the attention feature vector according to the first association vector, the second association vector, and the third association vector, including: the processor constructs a first fusion vector having a fourth association relationship between the first association vectors. The processor constructs a second fusion vector having a fifth association relationship between the second association vectors. The processor constructs a third fusion vector having a sixth association relationship between the first association vector and the second association vector. The processor uses one or more of the first fusion vector, the second fusion vector, and the third fusion vector as the attention feature vector.

[0091] Optionally, the processor uses one or more of the first association vector, the second association vector, the third association vector, the first fusion vector, the second fusion vector and the third fusion vector as the attention feature vector.

[0092] In this embodiment, the attention mechanism is used to realize multiple construction methods of the attention feature vector, which reflects the self-correlation and mutual correlation of the state information and action information, thereby reflecting the semantic feature information of the self-correlation and mutual correlation. This is conducive to establishing the global dependency relationship of the semantic feature information of the state information and action information, thereby more accurately reflecting the semantic information associated with the user interaction and the decision made by the intelligent agent, which is conducive to improving the accuracy and intention of the dish information generation and recommendation.

[0093] Optionally, the pre-trained deep reinforcement learning model is obtained by training the initial deep reinforcement learning model. Among them, the initial deep reinforcement learning model is an untrained deep reinforcement learning model. The training of the pre-trained deep reinforcement learning model is completed by the following steps: obtaining the state information and action information of the historical multimodal interaction data. Obtaining the attention feature vector that represents the association relationship between the state information and the action information. In the case where the feature matching of the attention feature vector and the dish interaction fusion feature vector is successful, generating multiple dish information corresponding to the attention feature vector and the dish interaction fusion feature vector. Sort the multiple dish information to obtain the sorting results of the multiple dish information. The dish information located at the preset sorting position in the sorting result is used as the historical dish information corresponding to the historical multimodal interaction data. Based on the user feedback information of the historical dish information, the dish interaction fusion feature vector is obtained. Then repeatedly perform the feature matching state step of the attention feature vector and the dish interaction fusion feature vector. After the above-mentioned repeated training of the pre-trained deep reinforcement model, the intelligent agent adaptively learns to calculate the relationship between the multimodal interaction data and the dish information, thereby outputting the optimized dish information, which greatly improves the recommendation accuracy of the dish information.

[0094] Exemplarily, based on the data accumulation of the dish interaction fusion feature vector and the multimodal interaction data, the pre-trained deep reinforcement learning model is updated to improve the model effect and accuracy. For example, when the dish interaction fusion feature vector and / or the multimodal interaction data are updated, the updated data is input into the pre-trained deep reinforcement learning model for retraining.

[0095] Optionally, the processor obtains the state information and action information of the multimodal interaction data, including: the processor obtains the total text interaction data of the multimodal interaction data. The processor extracts the state features and action features in the total text interaction data. The processor performs vectorization processing on the state features and the action features, respectively, to obtain a state text feature vector corresponding to the state features and an action text feature vector corresponding to the action features. The processor performs binarization processing on the state text feature vector and the action text feature vector, respectively, to obtain state information corresponding to the state text feature vector and action information corresponding to the action text feature vector.

[0096] Among them, the state feature is used to represent the descriptive features of the current environment state, and the action feature is used to represent the action features or decision features that can be taken in the corresponding state.

[0097] Exemplarily, the processor extracts state features and action features from the total text interaction data by using a target reinforcement model or a deep learning model. Specifically, the target reinforcement model is used to extract state features and action features from the total text interaction data. The target reinforcement model can accurately extract state features and action features from the total text interaction data.

[0098] The target reinforcement model may refer to a reinforcement model obtained by training the initial reinforcement model, and the initial reinforcement model may refer to an untrained reinforcement model. Among them, different types of reinforcement learning algorithms can construct reinforcement models of different structures, and reinforcement learning algorithms can be classified from multiple perspectives, such as model-based and model-free reinforcement learning methods; value-based and policy-based reinforcement learning methods, or the combination of the two actor-critic (AC) reinforcement learning method; Monte Carlo (MC) and temporal-difference (TD) reinforcement learning methods; online policy (OnPolicy) and offline policy (Off Policy) and other reinforcement learning methods. In the disclosed embodiment, the learning method of the target reinforcement model is not limited here.

[0099] The target reinforcement model can be obtained by training based on multimodal interaction data. For example, feature extraction can be performed on multimodal interaction data to obtain state features and action features; state features are used to represent the descriptive features of the current environmental state (such as the ambient temperature features, ambient brightness features, regional features, and user state features of the agent), and all state features are combined to obtain a state space, and the state features in the state space have temporal sequence; action features are used to represent the action features or decision features that can be taken in the corresponding state, and there can be multiple action features (such as the interaction between the user and the home appliance, the control instructions uploaded by the user for the corresponding state, and the multimodal interaction data uploaded by the user), and multiple action features are combined to obtain an action space, and the action features in the action space have temporal sequence.

[0100] Specifically, constructing a target reinforcement model includes: obtaining an untrained reinforcement model, and determining the untrained reinforcement model as an initial reinforcement model; the initial reinforcement model includes preset parameters such as update step size, reward discount factor, labeled rewards, and preset functions such as strategy function, reward function, and value function; inputting the action space and state space into the initial reinforcement model for model training to obtain the value function corresponding to the initial reinforcement model, and when the maximum value can be obtained based on the value function, adjusting parameters such as the weight of the initial reinforcement model to obtain the target reinforcement model.

[0101] Exemplarily, the processor can use One-Hot encoding or embedding technology to respectively vectorize the state features and the action features. Specifically, the embedding technology is used to vectorize the state features and the action features. More specifically, the embedding technology includes Word2Vec (a language model represented in the form of a neural network) and GloVe.

[0102] Exemplarily, the processor can use a deep hash function to perform binarization processing on the state text feature vector and the action text feature vector respectively. In implementation, the state text feature vector and the action text feature vector are binarized by a deep hash function to obtain the state information corresponding to the state text feature vector and the action information corresponding to the action text feature vector. The state information and the action information have lower dimensions than the state text feature vector and the action text feature vector, which is convenient for computer reading.

[0103] Optionally, the processor obtains the total text interaction data of the multimodal interaction data in the following manner: the processor parses the multimodal interaction data to obtain the text summary interaction data. The processor processes the text summary interaction data in a unified format to obtain the total text interaction data.

[0104] The text summary interaction data includes one or more of text interaction data, voice-text interaction data, and image-text interaction data.

[0105] In this embodiment, since most related technologies only use interaction data of a single modality, interaction data of other modalities are ignored in the subsequent interaction data learning, resulting in low recognition accuracy when users interact with home appliances in other forms of interaction data, and further resulting in low accuracy of dish information recommendation during interaction. The total text interaction data provided by the disclosed embodiment includes the collected interaction data of various modalities, and further association and recommendation of them have a higher accuracy.

[0106] Exemplarily, the multimodal interaction data includes one or more of text interaction data, image interaction data, voice interaction data, video interaction data, and audio and video interaction data.

[0107] Optionally, the processor parses the multimodal interaction data to obtain text summary interaction data, including: the processor parses the text interaction data to obtain the text interaction data. The processor parses the voice interaction data and / or the audio and video interaction data to obtain the voice and text interaction data. The processor parses one or more of the image interaction data, the video interaction data, and the audio and video interaction data to obtain the image and text interaction data.

[0108] In this embodiment, by parsing the multimodal interaction data into corresponding text data, the multimodal interaction data are all in the form of text interaction data, so that the interaction data form is unified, ensuring the comprehensiveness and specificity of the interaction data characteristics.

[0109] Optionally, the processor parsing the text interaction data to obtain the text interaction data includes: the processor effectively processes the text interaction data to obtain the effective text interaction data. The processor cleans, converts the format and stores the effective text interaction data to obtain the text interaction data.

[0110] Among them, tools or scripts can be used to achieve the cleaning, format conversion and storage of valid text interaction data. Specifically, the valid text interaction data can be imported into the data cleaning model in the form of a file for processing. In order to prevent data import failure, the data that does not meet the file import format is parsed and converted, and then the irrelevant data and duplicate data in the data set are deleted, and the abnormal values ​​and missing value data are processed to complete the cleaning of the valid text interaction data. At the same time, the cleaned data is output and saved in a specified format, thereby obtaining the text interaction data.

[0111] Optionally, the processor parsing the voice interaction data to obtain the voice-text interaction data includes: the processor effectively processes the voice interaction data to obtain effective voice interaction data. The processor transcribes the effective voice interaction data into voice-text interaction data.

[0112] Exemplarily, the processor processes the effective duration of the voice interaction data to remove invalid speech, punctuation and sentence breaks. The effective duration refers to the actual effective duration of the voice interaction data. Different from the total duration of the voice interaction data, this indicator calculates the voice interaction data after removing silence, background sound, invalid speech and punctuation. By processing the effective duration of the voice interaction data, invalid speech can be effectively processed or eliminated, thereby more effectively reflecting the effectiveness of the voice interaction data.

[0113] Exemplarily, ASR (Automatic Speech Recognition) or deep cyclic convolutional network technology can be used to transcribe effective voice interaction data into voice-text interaction data. More specifically, when using ASR (Automatic Speech Recognition) to transcribe effective voice interaction data into voice-text interaction data, the STT (Speech-to-Text) algorithm is mainly used. More specifically, when using deep cyclic convolutional network technology, the CNN (Convolutional Neural Networks) model, RNN (Recurrent Neural Network) model, and CTC (Connectionist Temporal Classification) model can be used to construct a fusion neural network, which has an attention mechanism for transcribe effective voice interaction data into voice-text interaction data. Among them, the CNN model is used to process the voice input spectrogram of effective voice interaction data and output a voice feature map. The RNN model is a recurrent network composed of multiple bidirectional LSTMs (Long Short-Term Memory Networks), which is used to further process the speech feature map into character sequences of different times or frames. The CTC model is an end-to-end learning model that facilitates the relationship between the input and output sequences to derive the correct character sequence. At the same time, the spatiotemporal characteristics and contextual information relationship characteristics of the effective speech interaction data are further extracted through the attention mechanism. Finally, the fusion neural network outputs the speech-text interaction data corresponding to the effective speech interaction data.

[0114] In this embodiment, the voice interaction data in the multimodal interaction data is transcribed into voice-text interaction. Due to the different usage scenarios of household appliances, variable factors must be fully considered, including the distance factor of data collection, the voice sentence structure, the spatiotemporal characteristics, and the contextual relationship contained in the voice interaction data. Therefore, in the process of transcribing voice interaction data into voice-text interaction data, a unified speech recognition architecture that integrates neural networks is used to transcribe voice interaction data into voice-text interaction data, obtain rich high-level voice interaction feature information, and improve the accuracy of the model voice transcription text.

[0115] Optionally, the processor parsing the audio and video interaction data to obtain the voice and text interaction data includes: the processor separating the audio and video interaction data to obtain the voice interaction data and the video interaction data corresponding to the audio and video interaction data. The processor parses the voice interaction data corresponding to the audio and video interaction data to obtain the voice and text interaction data corresponding to the audio and video interaction data.

[0116] Among them, the voice interaction data corresponding to the audio and video interaction data can be parsed in a manner of obtaining the voice-text interaction data by parsing the voice interaction data to obtain the voice-text interaction data corresponding to the audio and video interaction data.

[0117] Exemplarily, the processor can separate the audio and video interaction data by writing a script or tool processing. Specifically, Python (a computer programming language) is used to write an audio and video interaction separation script, or a third-party audio and video separation tool is used to perform a separation operation on the audio and video interaction data containing both video interaction data and audio interaction data, and obtain voice interaction data and video interaction data corresponding to the audio and video interaction data.

[0118] Optionally, the processor parsing the image interaction data to obtain the image text interaction data includes: the processor effectively processes the image interaction data to obtain the effective image interaction data. The processor transcribes the effective image interaction data into the image text interaction data.

[0119] Exemplarily, the processor effectively processes the image interaction data including: the processor determines the image frame data of the image interaction data, and the processor pre-screens the image frame data, removes the blurred image frame data, and uses the clear image frame data as the effective image interaction data.

[0120] Exemplarily, the processor converts the valid image interaction data into image-text interaction data including: constructing a deep fusion model based on the Bert model (Bidirectional Encoder Representation from Transformers, a pre-trained language representation model) and knowledge distillation (a method of model compression). The processor inputs the valid image interaction data into the deep fusion model based on Bert and knowledge distillation, and uses the data output by the deep fusion model based on Bert and knowledge distillation as the image-text interaction data.

[0121] Specifically, constructing a deep fusion model based on the Bert model and knowledge distillation includes: using knowledge distillation to migrate the knowledge tasks and network model parameters of the Bert teacher model to the Bert student model. The number of network layers and network parameters of the migrated Bert student model are fragmented to retain useful and effective network models, and a deep fusion model based on the Bert model and knowledge distillation that is compressed to a variable network size is obtained.

[0122] Exemplarily, the processor converts the valid image interaction data into image-text interaction data, including: constructing a fusion model based on knowledge distillation and diffusion model. The processor inputs the valid image interaction data into the fusion model based on knowledge distillation and diffusion model, and uses the data output by the fusion model based on knowledge distillation and diffusion model as the image-text interaction data.

[0123] Specifically, constructing a fusion model based on knowledge distillation and diffusion model includes: constructing and training at least one conditional diffusion model as a teacher model, the teacher model includes at least one encoder network and at least one decoder network. Initializing at least one time- and space-continuous student model, and using the parameters of the teacher model as initial parameters to obtain the same parameters of the student model and the teacher model. Performing knowledge distillation training on the student model, after completing a round of knowledge distillation, using the student model as the teacher model for a new training round, and repeating the knowledge distillation training; obtaining the trained student model and using it as a fusion model based on knowledge distillation and diffusion model.

[0124] In this embodiment, by transcribing effective image interaction data into image-text interaction data in the above manner, it is possible to mine the rich semantic feature information of the sentence sequence and improve the accuracy of transcription.

[0125] Optionally, the processor parsing the video interaction data to obtain the image-text interaction data includes: the processor converting the video interaction data into image interaction data. The processor parsing the image interaction data to obtain the image-text interaction data.

[0126] Optionally, the processor parsing the audio and video interaction data to obtain the image and text interaction data includes: the processor separating the audio and video interaction data to obtain the voice interaction data and the video interaction data corresponding to the audio and video interaction data; the processor converting the video interaction data into image interaction data; and the processor parsing the image interaction data to obtain the image and text interaction data.

[0127] Wherein, when the audio and video interaction data are separated, the image interaction data corresponding to the audio and video interaction data are separated into frame formats and stored.

[0128] Combination Figure 3 As shown, the embodiment of the present disclosure also provides another method for recommending dish information. The recommendation method includes:

[0129] S301: The processor obtains state information and action information of multimodal interaction data.

[0130] S302: The processor obtains an attention feature vector representing the association between state information and action information.

[0131] S303: The processor obtains a dish interaction fusion feature vector based on user feedback information of historical dish information.

[0132] S304, the processor determines the optimized dish information corresponding to the multimodal interaction data according to the attention feature vector and the dish interaction fusion feature vector.

[0133] S305: The processor outputs the optimized dish information corresponding to the multimodal interaction data in a preset output format.

[0134] In this embodiment, by outputting the optimized dish information corresponding to the multimodal interaction data in a preset output form, the output form is diversified, so that users can choose a satisfactory dish information receiving form according to their own needs or preferences, thereby improving the user experience.

[0135] Optionally, the preset output form includes a preset generation form and a preset access form.

[0136] Optionally, the processor outputs the optimized dish information corresponding to the multimodal interaction data in a preset output form, including: the processor generates the optimized dish information corresponding to the multimodal interaction data in a preset generation form, and accesses the optimized dish information corresponding to the multimodal interaction data in a preset access form.

[0137] Exemplarily, the preset generation form represents the data generation type of the optimized dish information. The preset generation form includes one or more of text form, image form, voice form, video form and audio and video form. For example: for the generation in text form, the text result is directly output to the text presentation, and the corresponding text output of the optimized dish information is presented to the user; for the generation in image form, the CLIP model method is used to generate the image result to the image presentation, and the corresponding image output of the optimized dish information is presented to the user; for the generation in voice form, text-to-speech conversion such as waveform splicing and end-to-end neural network model is used to generate voice results to voice broadcast, and the output form of the optimized dish information according to voice playback is presented to the user; similarly, the home appliance can also output the optimized dish information in the form of video or audio and video. In this way, the output method of the optimized dish information is more diverse and personalized, especially in the form of image, video or audio and video, which can fully express the quality of the dish information and bring a colorful visual experience to the user.

[0138] For example, the preset contact form indicates the optimization of the display mode of the dish information. The preset contact form includes a variety of forms of built-in or external, such as outbound calls, SMS contact, email notification, large screen display, voice broadcast, text output, smart speakers, pop-up UI, app, PAD, web and other result contact modes, which meet the needs of optimizing the display mode of dish information and digital display.

[0139] Combination Figure 4 As shown, the embodiment of the present disclosure provides a device 200 for recommending dish information, including a first acquisition module 21, a second acquisition module 22, a third acquisition module 23 and a determination module 24. The first acquisition module 21 is configured to obtain state information and action information of multimodal interaction data; wherein the state information is used to represent the description of the current environmental state, and the action information is used to represent the actions or decisions that can be taken in the corresponding state. The second acquisition module 22 is configured to obtain an attention feature vector that characterizes the association between the state information and the action information. The third acquisition module 23 is configured to obtain a dish interaction fusion feature vector based on user feedback information of historical dish information. The determination module 24 is configured to determine the optimized dish information corresponding to the multimodal interaction data based on the attention feature vector and the dish interaction fusion feature vector.

[0140] The device for recommending dish information provided by the embodiment of the present disclosure is useful for improving the accuracy and efficiency of recommending dish information of household appliances.

[0141] Optionally, the device 200 for recommending dish information further includes an output module. The output module is configured to output the optimized dish information corresponding to the multimodal interaction data in a preset output form.

[0142] Combination Figure 5 As shown, the embodiment of the present disclosure provides another device 300 for recommending dish information, including a processor 100 and a memory 101. Optionally, the device may also include a communication interface 102 and a bus 103. The processor 100, the communication interface 102, and the memory 101 may communicate with each other through the bus 103. The communication interface 102 may be used for information transmission. The processor 100 may call the logic instructions in the memory 101 to execute the method for recommending dish information of the above embodiment.

[0143] In addition, the logic instructions in the memory 101 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0144] The memory 101 is a computer-readable storage medium that can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 100 executes the function application and data processing by running the program instructions / modules stored in the memory 101, that is, the method for recommending dish information in the above embodiment is implemented.

[0145] The memory 101 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 101 may include a high-speed random access memory and may also include a non-volatile memory.

[0146] Combination Figure 6 As shown, an embodiment of the present disclosure provides a household appliance 1, comprising: a household appliance body, and the above-mentioned recommendation device 200 (300) for dish information. The recommendation device 200 (300) for dish information is installed on the household appliance body. The installation relationship described here is not limited to placement inside the product, but also includes installation connections with other components of the product, including but not limited to physical connections, electrical connections or signal transmission connections, etc. Those skilled in the art can understand that the recommendation device 200 (300) for dish information can be adapted to a feasible product body, thereby realizing other feasible embodiments.

[0147] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned method for recommending dish information.

[0148] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0149] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0150] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0151] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0152] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0153] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for recommending dish information, characterized in that: include: Obtaining state information and action information of multimodal interaction data; wherein the state information is used to represent the description of the current environment state, and the action information is used to represent the actions or decisions that can be taken in the corresponding state; Obtaining an attention feature vector representing the correlation between state information and action information; Based on the user feedback information of historical dish information, a dish interaction fusion feature vector is obtained; The optimized dish information corresponding to the multimodal interaction data is determined according to the attention feature vector and the dish interaction fusion feature vector.

2. The recommendation method according to claim 1, characterized in that: Obtaining the attention feature vector that represents the association between state information and action information, including: Constructing a first association vector having a first association relationship between state information; constructing a second association vector having a second association relationship between the action information; constructing a third association vector having a third association relationship between the state information and the action information; An attention feature vector is determined based on the first association vector, the second association vector and the third association vector.

3. The recommendation method according to claim 1, characterized in that: The optimized dish information corresponding to the multimodal interaction data is determined according to the attention feature vector and the dish interaction fusion feature vector, including: When the attention feature vector and the dish interaction fusion feature vector feature matching is successful, a plurality of dish information corresponding to the attention feature vector and the dish interaction fusion feature vector is generated; Sort the multiple dish information to obtain the sorting results of the multiple dish information; The dish information at the preset sorting position in the sorting result is used as the optimized dish information corresponding to the multimodal interaction data.

4. The recommendation method according to claim 1, characterized in that: Based on the user feedback information of historical dish information, the dish interaction fusion feature vector is obtained, including: Vectorize the user feedback information to obtain the user feedback embedding vector; Perform end-to-end encoding and decoding on the user feedback embedding vector to obtain the encoded and decoded user feedback embedding vector; Perform multi-dimensional reward analysis on the encoded and decoded user feedback embedding vector to obtain a user dish feedback information feature vector with a dish reward value; The user dish feedback information feature vectors are aggregated into dish interaction fusion feature vectors.

5. The recommendation method according to any one of claims 1 to 4, characterized in that: Obtain state information and action information of multimodal interaction data, including: Obtaining total text interaction data of multimodal interaction data; Extract state features and action features from total text interaction data; Vectorizing the state feature and the action feature respectively to obtain a state text feature vector corresponding to the state feature and an action text feature vector corresponding to the action feature; The state text feature vector and the action text feature vector are binarized respectively to obtain state information corresponding to the state text feature vector and action information corresponding to the action text feature vector.

6. The recommendation method according to claim 5, characterized in that: The total text interaction data of multimodal interaction data is obtained as follows: Parsing the multimodal interaction data to obtain text summary interaction data; wherein the text summary interaction data includes one or more of the text interaction data, voice-text interaction data, and image-text interaction data; The text summary interaction data is processed into a unified format to obtain the total text interaction data.

7. The recommendation method according to claim 6, characterized in that: The multimodal interaction data includes one or more of text interaction data, image interaction data, voice interaction data, video interaction data, and audio and video interaction data; Parse multimodal interaction data and obtain text summary interaction data, including: Parsing text interaction data to obtain the text interaction data; Parsing voice interaction data and / or audio and video interaction data to obtain voice-text interaction data; Parse one or more of the image interaction data, the video interaction data, and the audio and video interaction data to obtain the image text interaction data.

8. A device for recommending dish information, characterized in that: include: A first acquisition module is configured to obtain state information and action information of the multimodal interaction data; wherein the state information is used to represent the description of the current environment state, and the action information is used to represent the actions or decisions that can be taken in the corresponding state; A second acquisition module is configured to obtain an attention feature vector representing the association relationship between the state information and the action information; A third acquisition module is configured to obtain a dish interaction fusion feature vector based on user feedback information of historical dish information; The determination module is configured to determine the optimized dish information corresponding to the multimodal interaction data based on the attention feature vector and the dish interaction fusion feature vector.

9. A device for recommending dish information, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the method for recommending dish information as described in any one of claims 1 to 7 when running the program instructions.

10. A household appliance, characterized in that: include: Home appliances; and, The device for recommending dish information as described in claim 8 or 9 is installed on the main body of the home appliance.

Images