Man-machine interaction system based on multi-modal large model

Abstract

The invention discloses a man-machine interaction system based on a multi-modal large model, and relates to the technical field of man-machine interaction, and the man-machine interaction system comprises an authorization module, a man-machine interaction analysis unit, a user confirmation processing unit and a local database. According to the method, intelligent man-machine interaction is carried out, retrieval is carried out according to the actual body condition of a user when the user carries out fuzzy questioning, and all keywords of voice input information are intelligently detected when the user carries out accurate questioning, so that a retrieval result is intelligently returned, the user experience is improved, the remaining food material amount and the remaining seasoning amount in a user home are intelligently detected, and the user experience is improved. And whether food materials and seasonings of the recommended recipes are complete at home is judged, and intelligent analysis is performed on the take-out purchasing platform under the condition that the materials of the recommended recipes are incomplete, so that a user can make a selection in the recommended recipes conveniently, and the selection difficulty degree of the user is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a human-computer interaction system based on a multimodal large model. Background Art

[0002] At present, the research on multimodal large models has made significant progress and has been widely used in visual question answering, speech recognition, text generation, sentiment analysis and other fields. However, human-computer interaction in the traditional cooking field is generally performed by users searching, and the application returns search results through artificial intelligence. When users further put forward requirements or make vague requests, it is often difficult to identify and invalid search results are easily returned, thereby reducing the user experience. Therefore, it is very necessary to study human-computer interaction systems based on multimodal large models in the cooking field.

[0003] Prior art, such as the invention patent application with announcement number: CN117784937A, discloses an intelligent human-computer interaction system and method based on multimodal features, wherein the system includes: firstly, collecting pictures / videos of the user's environment and converting them into text descriptions through a picture-to-text conversion module or a video-to-text module, and at the same time collecting and translating the user's voice commands through a voice recognition module. Subsequently, both the environment description text and the text commands are submitted to the natural language understanding module for further interpretation. By applying this intelligent human-computer interaction system and method, users can be provided with services with higher intelligence levels and more accurate performance.

[0004] Prior art, such as the invention patent application with announcement number: CN117762257A, discloses a human-computer interaction system and method based on a multimodal large language model, wherein the system includes: an image acquisition unit for acquiring images and preprocessing, an image processing unit for extracting features from the acquired images, and a display unit for displaying the extracted features in the form of text on a display screen; a sound acquisition unit for acquiring sounds and preprocessing, a speech processing unit for receiving and processing sounds to obtain corresponding output answers, and a speech output unit for outputting sounds to achieve real-time communication with users. This invention solves the problem that the intelligent system cannot perform multimodal intelligent human-computer interaction such as speech and vision.

[0005] It can be seen from the above scheme that the current human-computer interaction system lacks fuzzy and precise recommendations for recommended recipes based on the user's own physical information and voice input information. Users sometimes cannot fully understand their own cooking needs, so fuzzy questions often appear when asking questions. When asking fuzzy questions, there will be no restrictions on prices, dish names and tastes. At this time, artificial intelligence cannot understand and does not search according to the user's actual physical condition, resulting in the returned results being too broad, rather than a special search for the user's actual situation, thereby reducing the user's experience. At the same time, there is a lack of attention to the missing parameters of each recommended recipe of the intelligent calculation user and the purchase amount of these materials if they want to buy them. Users sometimes cannot clearly know the remaining amount of ingredients and seasonings at home. When obtaining each recommended recipe, they often cannot start cooking in time. It is necessary to determine whether the ingredients and seasonings of each recommended recipe are complete at home. If not, they need to purchase on various takeaway purchasing platforms, compare prices from three stores, which is time-consuming and labor-intensive, making it difficult for users to make choices among the recommended recipes, thereby increasing the difficulty of user selection. Summary of the invention

[0006] The purpose of the present invention is to provide a human-computer interaction system based on a multimodal large model, which solves the problems existing in the background technology.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solution: The present invention provides a human-computer interaction system based on a multimodal large model, including: an authorization module for obtaining various human-computer interaction related permissions of the user.

[0008] The human-computer interaction analysis unit is used to obtain the user's voice input information when the user makes cooking-related inquiries to the cooking application, obtain the user's blood data, determine the user's physical condition, and analyze the user's recommended recipes.

[0009] The human-computer interaction analysis unit includes: a user information acquisition module, a scene multimodal analysis and judgment module, a recipe fuzzy recommendation module and a recipe precise recommendation module.

[0010] The user information acquisition module is used to obtain the user's voice input information when the user makes cooking-related inquiries, analyze the correlation between the user's voice input information and cooking, and determine whether it is necessary to re-acquire the user's voice input information. If necessary, the user is reminded that the voice input information cannot be recognized and the user is reminded to re-enter the voice input information. If not necessary, the scene multimodal analysis and judgment module is executed.

[0011] The scene multimodal analysis and judgment module is used to obtain the characteristic values ​​of the user's blood data and each hazardous disease, and judge whether the user's voice input information contains sub-restriction information. If not, the recipe fuzzy recommendation module is executed; if contained, the sub-restriction information of the user's voice input information is obtained, and the recipe precise recommendation module is executed.

[0012] The recipe fuzzy recommendation module is used to analyze the user's recommended recipes.

[0013] The recipe accurate recommendation module is used to evaluate the user's recommended recipes based on the sub-restriction information of the user's voice input information.

[0014] The user confirmation processing unit is used to analyze the missing parameters and missing quantities of the user's recommended recipes in combination with the food information and auxiliary material information of the user's environment, and obtain the user's confirmed recipes by querying the user, and based on the missing parameters of the user's recommended recipes, extract the missing parameters of the user's confirmed recipes and perform material supplement operations.

[0015] The user confirmation processing unit includes: a missing parameter analysis module and a user confirmation purchase module.

[0016] The missing parameter analysis module is used to obtain the remaining characteristic quantities of each stored ingredient in the user's refrigerator, the storage time, and the remaining quantity of each seasoning in the kitchen, analyze the missing ingredients and their missing characteristic quantities and the missing seasonings and their missing quantities used by the user to cook each recommended recipe, and summarize them as the missing parameters and missing quantities of each recommended recipe of the user, calculate the takeaway purchase amount for purchasing the missing parameters of each recommended recipe of the user, and sort the recommended recipes of the user in ascending order according to the takeaway purchase amount, so as to obtain the sorted recommended recipes of the user.

[0017] The user confirmation purchase module is used to send the sorted recommended recipes of the user and the takeaway purchase amount of the missing parameters to the user's cooking application, and query the user to determine whether the user has selected a recipe. If the user has selected a recipe, the recipe selected by the user is marked as a confirmed recipe, and the user is asked whether it is necessary to automatically purchase the missing ingredients and missing seasonings for cooking the confirmed recipe. If the user agrees, the missing ingredients and missing seasonings for cooking each recommended recipe of the user are purchased through the takeaway platform.

[0018] The beneficial effects of the present invention are: (1) The authorization module of the present invention facilitates subsequent analysis by obtaining relevant permissions of the user.

[0019] (2) The human-computer interaction analysis unit of the present invention performs intelligent human-computer interaction based on the fields of the user's voice input information and the characteristic values ​​of each blood data. When the user asks a vague question, the unit performs a search based on the user's actual physical condition. When the user asks a precise question, the unit intelligently detects the keywords in the voice input information, thereby intelligently returning the search results and improving the user experience.

[0020] (3) The user confirmation processing unit of the present invention uses the amount of ingredients and seasonings in the user's home as an objective reference condition for human-computer interaction, thereby facilitating artificial intelligence to intelligently detect the remaining amount of ingredients and seasonings in the user's home based on these objective reference conditions, and to determine whether the ingredients and seasonings of each recommended recipe are complete at home. In the event that the materials of each recommended recipe are incomplete, an intelligent analysis is performed on the takeaway purchasing platform, thereby intelligently obtaining the takeaway purchase amount of each recommended recipe, thereby facilitating the user to make a choice among the recommended recipes, thereby reducing the difficulty of the user's selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a schematic diagram of the system module of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Reference Figure 1 As shown, the present invention provides a human-computer interaction system based on a multimodal large model, including: an authorization module, a human-computer interaction analysis unit, a user confirmation processing unit and a local database.

[0025] It should be noted that the authorization module is connected to the human-computer interaction analysis unit, the human-computer interaction analysis unit is connected to the user confirmation processing unit, and the local database is connected to the human-computer interaction analysis unit and the user confirmation processing unit.

[0026] It should also be noted that the local database is used to store association thresholds, various hazardous symptoms of users, various cooking-related keywords, various keywords for precise dish names, various keywords for various flavors, the demand for each ingredient and each condiment for each recipe in the cooking application, the allowed consumption of each condiment corresponding to the characteristic value interval of each blood data, the allowed consumption of each condiment and each prohibited ingredient corresponding to each hazardous disease, the price range, dish name and flavor of each recipe in the cooking application, the demand for each ingredient and each condiment for each applicable recipe in the cooking application, the demand characteristic quantity of each ingredient and each condiment for each recommended recipe of the user, and the allowed storage time of each stored ingredient.

[0027] The authorization module is used to obtain the user's various human-computer interaction related permissions.

[0028] It should be noted that the various human-computer interaction related permissions include: the user's refrigerator and kitchen camera call permission, the user's voice recognition permission, the user's blood data acquisition permission and other human-computer interaction related permissions.

[0029] The authorization module of the present invention obtains various relevant permissions of the user to facilitate subsequent analysis.

[0030] The human-computer interaction analysis unit is used to obtain the user's voice input information and the user's blood data when the user makes cooking-related inquiries to the cooking application, determine the user's physical condition, and analyze the user's recommended recipes.

[0031] In a specific embodiment, the method for obtaining the user's voice input information is as follows: obtaining the user's voice input information from a cooking application.

[0032] It should be noted that the blood data include: heart rate, blood oxygen saturation, blood viscosity and other data.

[0033] It should be noted that the recommended recipes of users are the recommended recipes of users analyzed by the recipe fuzzy recommendation module or the recommended recipes of users by the recipe precise recommendation module, both of which are the results of discussions based on different situations, corresponding to the results of fuzzy inquiry and precise inquiry of users respectively.

[0034] In a specific embodiment of the present invention, the human-computer interaction analysis unit includes: a user information acquisition module, a scene multimodal analysis and judgment module, a recipe fuzzy recommendation module and a recipe precise recommendation module.

[0035] The user information acquisition module is used to obtain the user's voice input information when the user makes cooking-related inquiries, analyze the correlation between the user's voice input information and cooking, and determine whether it is necessary to re-acquire the user's voice input information. If necessary, the user is reminded that the voice input information cannot be recognized and the user is reminded to re-enter the voice input information. If not necessary, the scene multimodal analysis and judgment module is executed.

[0036] In a specific embodiment, the determination of whether it is necessary to re-acquire the user's voice input information is performed by obtaining a correlation threshold from a local database, and comparing the correlation between the user's voice input information and cooking-related information with the correlation threshold; if the correlation between the user's voice input information and cooking-related information is greater than the correlation threshold, it is determined that it is not necessary to re-acquire the user's voice input information; otherwise, it is determined that it is necessary to re-acquire the user's voice input information.

[0037] The scene multimodal analysis and judgment module is used to obtain the characteristic values ​​of the user's blood data and each hazardous disease, and judge whether the user's voice input information contains sub-restriction information. If not, the recipe fuzzy recommendation module is executed; if contained, the sub-restriction information of the user's voice input information is obtained, and the recipe precise recommendation module is executed.

[0038] It should be noted that the sub-restriction information includes: price range, precise dish name and flavor. The user's voice input information may contain one or more sub-restriction information at the same time. For example, the user's voice input information is "I want to make a spicy shredded pork with green peppers within a budget of less than 10 yuan", where the precise dish name is shredded pork with green peppers, the price range is 0 yuan to 10 yuan, and the flavor is spicy.

[0039] In a specific embodiment, the characteristic values ​​of each blood data and each hazardous disease of the user are obtained by a specific method: the characteristic values ​​of each blood data of the user are obtained from an intelligent blood data monitoring function carried by a cooking application, the user pre-inputs his own hazardous diseases, and the user's hazardous diseases are obtained through a local database.

[0040] The recipe fuzzy recommendation module is used to analyze the user's recommended recipes.

[0041] The recipe accurate recommendation module is used to evaluate the user's recommended recipes based on the sub-restriction information of the user's voice input information.

[0042] In a specific embodiment of the present invention, the analysis of the association between the user's voice input information and cooking-related information is specifically performed by obtaining various keywords related to cooking from a local database to form a keyword set A related to cooking.

[0043] Convert the user's voice input information into text, thereby obtaining the user's voice-to-text information, extracting the keywords of the user's voice-to-text information, constructing a keyword set B for obtaining the user's voice-to-text information, and calculating the correlation between the user's voice input information and cooking .

[0044] It should be noted that the cooking-related keywords include keywords such as cooking, cooking, and cuisine.

[0045] In a specific embodiment of the present invention, the specific method for determining whether the user's voice input information contains sub-restriction information is as follows: obtaining keywords for each precise dish name and keywords for each flavor from a local database.

[0046] Based on the user's voice input information, the field of the user's voice input information is obtained, and the field of the user's voice input information is compared with the keywords of each precise dish name. If the field of the user's voice input information contains a keyword of a precise dish name, it is determined that the user's voice input information contains sub-restriction information. Otherwise, it is determined that the user's voice input information does not contain sub-restriction information.

[0047] The fields of the user's voice input information are compared with the keywords of each flavor. If the field of the user's voice input information contains a keyword of a certain flavor, it is determined that the user's voice input information contains sub-restriction information. Otherwise, it is determined that the user's voice input information does not contain sub-restriction information.

[0048] The fields of the user's voice input information are put into the price range judgment model to determine whether the user's voice input information contains price range sub-restriction information.

[0049] In a specific embodiment of the present invention, the specific analysis method for analyzing the user's recommended recipes is as follows: based on the characteristic values ​​of the user's blood data and the harmful symptoms, the user's prohibited ingredients and the upper limit consumption of each condiment are analyzed.

[0050] The required quantity of each ingredient and the required quantity of each seasoning of each recipe in the cooking application are obtained from the local database. If an ingredient of a recipe in the cooking application is inconsistent with all prohibited ingredients of the user, the required quantity of each seasoning of the recipe in the cooking application is compared with the appropriate consumption amount of each seasoning of the user. If the required quantity of a seasoning of the recipe in the cooking application is less than or equal to the upper limit consumption amount of the seasoning of the user, the recipe is marked as a recommended recipe of the user, thereby screening the recommended recipes of the user.

[0051] In a specific embodiment of the present invention, the specific analysis method for analyzing the user's prohibited ingredients and appropriate consumption amounts of each condiment is as follows: obtaining the allowed consumption amount of each condiment corresponding to the characteristic value interval of each blood data from a local database, and mapping the allowed consumption amount of each condiment in the user's blood data based on the characteristic value of each blood data of the user.

[0052] The allowed consumption amount of each condiment and each prohibited food material corresponding to each hazardous disease are obtained from the local database, and the allowed consumption amount of each condiment and each prohibited food material of the user in terms of the hazardous disease are mapped according to the hazardous disease of the user.

[0053] The user's allowed consumption amount of each condiment in terms of blood data is compared with the allowed consumption amount of each condiment in terms of hazardous diseases. If the user's allowed consumption amount of a condiment in terms of blood data is greater than the allowed consumption amount of the condiment in terms of hazardous diseases, the allowed consumption amount of the condiment in the user's blood data is used as the appropriate consumption amount of the condiment for the user; otherwise, the allowed consumption amount of the condiment in terms of the hazardous diseases is used as the appropriate consumption amount of the condiment for the user, thereby obtaining the appropriate consumption amount of each condiment for the user.

[0054] In a specific embodiment of the present invention, the specific evaluation method for evaluating each recommended recipe of the user is: obtaining the price range, dish name and taste of each recipe in the cooking application from the local database.

[0055] If the sub-restriction information of the user's voice input information only includes a price range, the price range in the user's voice input information is compared with the price range of each recipe in the cooking application. If the price range of a recipe in the cooking application includes the price range of the user's voice input information, the recipe is marked as an applicable recipe.

[0056] If the sub-restriction information of the user's voice input information only includes the precise dish name, the precise dish name in the user's voice input information is compared with the dish names of each recipe in the cooking application. If the dish name of a recipe in the cooking application is consistent with the precise dish name in the user's voice input information, the recipe is marked as an applicable recipe.

[0057] If the sub-restriction information of the user's voice input information only includes taste, the taste in the user's voice input information is compared with the taste of each recipe in the cooking application. If the taste of a recipe in the cooking application is consistent with the taste of the user's voice input information, the recipe is marked as an applicable recipe.

[0058] If there are two or more sub-restriction information in the user's voice input information, and if the user's voice input information contains a price range and an exact dish name, the price range and dish name of each recipe in the cooking application will be compared with the price range and exact dish name of the user's voice input information, and based on the corresponding comparison method, the user's applicable recipes will be screened.

[0059] Summarizes all applicable recipes of users.

[0060] The required quantity of each ingredient and the required quantity of each seasoning of each applicable recipe in the cooking application are obtained from the local database, and the method of analyzing each recommended recipe of the user according to the recipe fuzzy recommendation module is used to evaluate each recommended recipe of the user in the same way.

[0061] The human-computer interaction analysis unit of the present invention performs intelligent human-computer interaction based on the fields of the user's voice input information and the characteristic values ​​of each blood data. When the user asks vague questions, it searches according to the user's actual physical condition. When the user asks precise questions, it intelligently detects the keywords of the voice input information, thereby intelligently returning the search results and improving the user experience.

[0062] The user confirmation processing unit is used to analyze the missing parameters and missing quantities of the user's recommended recipes in combination with the food information and auxiliary material information of the user's environment, and obtain the user's confirmed recipes by querying the user, and based on the missing parameters of the user's recommended recipes, extract the missing parameters of the user's confirmed recipes and perform material supplement operations.

[0063] In a specific embodiment of the present invention, the user confirmation processing unit includes: a missing parameter analysis module and a user confirmation purchase module.

[0064] The missing parameter analysis module is used to obtain the remaining characteristic quantities of each stored ingredient in the user's refrigerator, the storage time, and the remaining quantity of each seasoning in the kitchen, analyze the missing ingredients and their missing characteristic quantities and the missing seasonings and their missing quantities used by the user to cook each recommended recipe, and summarize them as the missing parameters and missing quantities of each recommended recipe of the user, calculate the takeaway purchase amount for purchasing the missing parameters of each recommended recipe of the user, and sort the recommended recipes of the user in ascending order according to the takeaway purchase amount, so as to obtain the sorted recommended recipes of the user.

[0065] In a specific embodiment, the remaining characteristic quantity and storage time of each stored food ingredient in the user's refrigerator and the remaining quantity of each condiment in the kitchen are obtained by performing image analysis of the stored food ingredients through a camera installed inside the refrigerator to obtain the remaining characteristic quantity and storage time of each stored food ingredient in the user's refrigerator, and obtaining the remaining quantity of each condiment in the user's kitchen through a camera installed at the condiment placement in the kitchen.

[0066] It should be noted that the above-mentioned seasonings include salt, vinegar, soy sauce and other seasonings.

[0067] The user confirmation purchase module is used to send the sorted recommended recipes of the user and the takeaway purchase amount of the missing parameters to the user's cooking application, and query the user to determine whether the user has selected a recipe. If the user has selected a recipe, the recipe selected by the user is marked as a confirmed recipe, and the user is asked whether it is necessary to automatically purchase the missing ingredients and missing seasonings for cooking the confirmed recipe. If the user agrees, the missing ingredients and missing seasonings for cooking each recommended recipe of the user are purchased through the takeaway platform.

[0068] In a specific embodiment of the present invention, the missing ingredients and their missing characteristic quantities and the missing condiments and their missing quantities that the user uses to cook each recommended recipe are analyzed, and the specific analysis method is: obtaining the required characteristic quantities of each ingredient and the required characteristic quantities of each condiment for each recommended recipe of the user, and the allowed storage time of each stored ingredient from a local database.

[0069] If an ingredient in a user's recommended recipe is inconsistent with all the stored ingredients in the refrigerator, the ingredient is marked as a missing ingredient, and the required feature quantity of the ingredient is used as the missing feature quantity of the missing ingredient.

[0070] If an ingredient in a recommended recipe of the user is consistent with a stored ingredient in the refrigerator, the demand characteristic quantity of the ingredient in the recommended recipe of the user is compared with the remaining characteristic quantity of the stored ingredient in the refrigerator. If the demand characteristic quantity of the ingredient in the recommended recipe of the user is greater than the remaining characteristic quantity of the stored ingredient in the refrigerator, the ingredient is marked as a missing ingredient, and the difference between the demand characteristic quantity of the ingredient in the recommended recipe of the user and the remaining characteristic quantity of the stored ingredient in the refrigerator is used as the missing characteristic quantity of the missing ingredient. Otherwise, the storage time of the stored ingredient in the refrigerator is compared with the allowed storage time. If the storage time of the stored ingredient in the refrigerator is greater than the allowed storage time, the ingredient is marked as a missing ingredient, and the demand characteristic quantity of the ingredient is used as the missing characteristic quantity of the missing ingredient.

[0071] The missing ingredients and their missing feature quantities that the user needs to cook each recommended recipe are summarized.

[0072] Similarly, the missing seasonings and the missing amounts used by the user to cook each recommended recipe are analyzed.

[0073] In a specific embodiment of the present invention, the takeaway purchase amount for purchasing missing parameters of each recommended recipe of the user is calculated by the following specific calculation method: obtaining the takeaway purchase amount corresponding to each missing ingredient in each missing feature value interval from each takeaway delivery platform around the user; mapping the takeaway purchase amount of each takeaway delivery platform for purchasing the missing ingredients when the user makes each recommended recipe based on the missing ingredients and their missing feature values ​​when the user makes each recommended recipe; extracting the minimum takeaway purchase amount for purchasing the missing ingredients when the user makes each recommended recipe, and using it as the takeaway purchase amount for purchasing the missing ingredients for each recommended recipe of the user.

[0074] Similarly, the amount of takeout purchases of missing seasonings for each recommended recipe of the user can be obtained through analysis.

[0075] The takeaway purchase amount for the missing ingredients and the takeaway purchase amount for the missing seasonings of each recommended recipe of the user are added together to calculate the takeaway purchase amount for the missing parameters of each recommended recipe of the user.

[0076] The user confirmation processing unit of the present invention uses the amount of ingredients and seasonings in the user's home as an objective reference condition for human-computer interaction, thereby facilitating artificial intelligence to intelligently detect the remaining amount of ingredients and seasonings in the user's home based on these objective reference conditions, and determine whether the ingredients and seasonings of each recommended recipe are complete at home. In the event that the materials of each recommended recipe are incomplete, intelligent analysis is performed on the takeaway purchasing platform to intelligently obtain the takeaway purchasing amount of each recommended recipe, thereby facilitating users to make choices among the recommended recipes, thereby reducing the difficulty of selection for users.

[0077] The above contents are merely examples and explanations of the concept of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, they shall all fall within the protection scope of the present invention.

Claims

1. A human-computer interaction system based on a multimodal large model, characterized in that: include: Authorization module, used to obtain the user's various human-computer interaction related permissions; A human-computer interaction analysis unit, used to obtain the user's voice input information when the user makes cooking-related inquiries to the cooking application, obtain the user's blood data, determine the user's physical condition, and analyze the user's recommended recipes; The user confirmation processing unit is used to analyze the missing parameters and missing quantities of the user's recommended recipes in combination with the food information and auxiliary material information of the user's environment, and obtain the user's confirmed recipes by querying the user, and based on the missing parameters of the user's recommended recipes, extract the missing parameters of the user's confirmed recipes and perform material supplement operations.

2. According to claim 1, a human-computer interaction system based on a multimodal large model is characterized in that: The human-computer interaction analysis unit includes: a user information acquisition module, a scene multimodal analysis and judgment module, a recipe fuzzy recommendation module and a recipe precise recommendation module; The user information acquisition module is used to acquire the user's voice input information when the user makes a cooking-related inquiry, analyze the correlation between the user's voice input information and cooking, and determine whether it is necessary to re-acquire the user's voice input information. If necessary, remind the user that the voice input information cannot be recognized and remind the user to re-enter the voice input information. If not, execute the scene multimodal analysis and judgment module; The scenario multimodal analysis and judgment module is used to obtain the characteristic values ​​of each blood data and each hazardous disease of the user, and judge whether the user's voice input information contains sub-restriction information. If not, the recipe fuzzy recommendation module is executed; if it is contained, the sub-restriction information of the user's voice input information is obtained, and the recipe precise recommendation module is executed; The recipe fuzzy recommendation module is used to analyze each recommended recipe of the user; The recipe accurate recommendation module is used to evaluate the user's recommended recipes based on the sub-restriction information of the user's voice input information.

3. The human-computer interaction system based on a multimodal large model according to claim 2, characterized in that: The specific analysis method of analyzing the correlation between the user's voice input information and cooking is as follows: Acquire various keywords related to cooking from a local database to form a keyword set A related to cooking; Convert the user's voice input information into text, thereby obtaining the user's voice-to-text information, extracting the keywords of the user's voice-to-text information, constructing a keyword set B for obtaining the user's voice-to-text information, and calculating the correlation between the user's voice input information and cooking .

4. The human-computer interaction system based on a multimodal large model according to claim 2, characterized in that: The specific method of judging whether the user's voice input information contains sub-restriction information is as follows: Obtain each keyword of each precise dish name and each keyword of each flavor from the local database; According to the user's voice input information, the field of the user's voice input information is obtained, and the field of the user's voice input information is compared with each keyword of each precise dish name. If the field of the user's voice input information contains a certain keyword of a certain precise dish name, it is determined that the user's voice input information contains sub-restriction information; otherwise, it is determined that the user's voice input information does not contain the sub-restriction information; Compare the fields of the user's voice input information with the keywords of each flavor. If the fields of the user's voice input information contain a keyword of a certain flavor, it is determined that the user's voice input information contains sub-restriction information. Otherwise, it is determined that the user's voice input information does not contain sub-restriction information. The fields of the user's voice input information are put into the price range judgment model to determine whether the user's voice input information contains price range sub-restriction information.

5. The human-computer interaction system based on a multimodal large model according to claim 2, characterized in that: The specific analysis method of analyzing the recommended recipes of the user is as follows: Analyze the user's prohibited food ingredients and upper limit consumption of each condiment according to the characteristic values ​​of each blood data of the user and each hazardous disease; The required quantity of each ingredient and the required quantity of each seasoning of each recipe in the cooking application are obtained from the local database. If an ingredient of a recipe in the cooking application is inconsistent with all prohibited ingredients of the user, the required quantity of each seasoning of the recipe in the cooking application is compared with the appropriate consumption amount of each seasoning of the user. If the required quantity of a seasoning of the recipe in the cooking application is less than or equal to the upper limit consumption amount of the seasoning of the user, the recipe is marked as a recommended recipe of the user, thereby screening the recommended recipes of the user.

6. The human-computer interaction system based on a multimodal large model according to claim 5, characterized in that: The specific analysis method of analyzing the user's prohibited food ingredients and the appropriate consumption amount of each seasoning is as follows: Obtaining the allowable consumption amount of each condiment corresponding to the characteristic value interval of each blood data from the local database, and mapping the allowable consumption amount of each condiment in the blood data of the user according to the characteristic value of each blood data of the user; Obtaining the allowable consumption amount of each condiment and each prohibited food material corresponding to each harmful disease from the local database, and mapping the allowable consumption amount of each condiment and each prohibited food material in terms of the harmful disease of the user according to each harmful disease of the user; The user's allowed consumption amount of each condiment in terms of blood data is compared with the allowed consumption amount of each condiment in terms of hazardous diseases. If the user's allowed consumption amount of a condiment in terms of blood data is greater than the allowed consumption amount of the condiment in terms of hazardous diseases, the allowed consumption amount of the condiment in the user's blood data is used as the appropriate consumption amount of the condiment for the user; otherwise, the allowed consumption amount of the condiment in terms of the hazardous diseases is used as the appropriate consumption amount of the condiment for the user, thereby obtaining the appropriate consumption amount of each condiment for the user.

7. The human-computer interaction system based on a multimodal large model according to claim 2, characterized in that: The specific evaluation method for evaluating each recommended recipe of the user is as follows: Obtain the price range, dish name and taste of each recipe in the cooking application from the local database; If the sub-restriction information of the user's voice input information only includes a price range, the price range in the user's voice input information is compared with the price range of each recipe in the cooking application, and if the price range of a recipe in the cooking application includes the price range of the user's voice input information, the recipe is marked as an applicable recipe; If the sub-restriction information of the user's voice input information only includes the precise dish name, the precise dish name in the user's voice input information is compared with the dish name of each recipe in the cooking application, and if the dish name of a recipe in the cooking application is consistent with the precise dish name in the user's voice input information, the recipe is marked as an applicable recipe; If the sub-restriction information of the user's voice input information only includes taste, the taste in the user's voice input information is compared with the taste of each recipe in the cooking application, and if the taste of a recipe in the cooking application is consistent with the taste of the user's voice input information, the recipe is marked as an applicable recipe; If there are two or more sub-restriction information in the user's voice input information, and if the user's voice input information includes a price range and an accurate dish name, the price range and dish name of each recipe in the cooking application are compared with the price range and accurate dish name of the user's voice input information, and the user's applicable recipes are screened according to the corresponding comparison method; Summarize the user's applicable recipes; The required quantity of each ingredient and the required quantity of each seasoning of each applicable recipe in the cooking application are obtained from the local database, and the method of analyzing each recommended recipe of the user according to the recipe fuzzy recommendation module is used to evaluate each recommended recipe of the user in the same way.

8. The human-computer interaction system based on a multimodal large model according to claim 1, characterized in that: The user confirmation processing unit includes: a missing parameter analysis module and a user confirmation purchase module; The missing parameter analysis module is used to obtain the remaining characteristic quantities and storage time of each stored food in the user's refrigerator and the remaining quantity of each seasoning in the kitchen, analyze the missing food ingredients and their missing characteristic quantities and the missing seasonings and their missing quantities used by the user to cook each recommended recipe, and summarize them into the missing parameters and their missing quantities of each recommended recipe of the user, calculate the takeaway purchase amount of each recommended recipe of the user to purchase the missing parameters, and sort the recommended recipes of the user in ascending order according to the takeaway purchase amount, so as to obtain the sorted recommended recipes of the user; The user confirmation purchase module is used to send the sorted recommended recipes of the user and the takeaway purchase amount of the missing parameters to the user's cooking application, and query the user to determine whether the user has selected a recipe. If the user has selected a recipe, the recipe selected by the user is marked as a confirmed recipe, and the user is asked whether it is necessary to automatically purchase the missing ingredients and missing seasonings for cooking the confirmed recipe. If the user agrees, the missing ingredients and missing seasonings for cooking each recommended recipe of the user are purchased through the takeaway platform.

9. The human-computer interaction system based on a multimodal large model according to claim 8, characterized in that: The specific analysis method of analyzing the missing ingredients and their missing feature quantities and the missing seasonings and their missing quantities used by the user to cook each recommended recipe is as follows: Acquire the required characteristic quantity of each ingredient and each seasoning of each recommended recipe of the user and the allowed storage time of each stored ingredient from the local database; If an ingredient in a user's recommended recipe is inconsistent with all the stored ingredients in the refrigerator, the ingredient is marked as a missing ingredient, and the required feature value of the ingredient is used as the missing feature value of the missing ingredient; If an ingredient of a certain recommended recipe of the user is consistent with a certain stored ingredient in the refrigerator, the demand feature quantity of the ingredient of the recommended recipe of the user is compared with the remaining feature quantity of the stored ingredient in the refrigerator. If the demand feature quantity of the ingredient of the recommended recipe of the user is greater than the remaining feature quantity of the stored ingredient in the refrigerator, the ingredient is marked as a missing ingredient, and the difference between the demand feature quantity of the ingredient of the recommended recipe of the user and the remaining feature quantity of the stored ingredient in the refrigerator is used as the missing feature quantity of the missing ingredient. Otherwise, the storage time of the stored ingredient in the refrigerator is compared with the allowed storage time. If the storage time of the stored ingredient in the refrigerator is greater than the allowed storage time, the ingredient is marked as a missing ingredient, and the demand feature quantity of the ingredient is used as the missing feature quantity of the missing ingredient. Summarize the missing ingredients and their missing features used by the user to cook each recommended recipe; Similarly, the missing seasonings and the missing amounts used by the user to cook each recommended recipe are analyzed.

10. The human-computer interaction system based on a multimodal large model according to claim 8, characterized in that: The specific calculation method for calculating the takeaway purchase amount of the missing parameters of each recommended recipe of the user is as follows: Obtain the takeaway purchase amount corresponding to each missing ingredient in each missing feature value interval from each takeaway delivery platform around the user, and map the takeaway purchase amount of each takeaway delivery platform used to purchase the missing ingredients when the user makes each recommended recipe based on the missing ingredients and their missing feature values, and extract the minimum takeaway purchase amount used to purchase the missing ingredients when the user makes each recommended recipe, and use it as the takeaway purchase amount for purchasing the missing ingredients for each recommended recipe of the user; Similarly, the amount of takeout purchases of missing seasonings for each recommended recipe of the user is obtained through analysis; The takeaway purchase amount for the missing ingredients and the takeaway purchase amount for the missing seasonings of each recommended recipe of the user are added together to calculate the takeaway purchase amount for the missing parameters of each recommended recipe of the user.

Images