Cooking method based on cooking ai large model and ai virtual human system, and intelligent cooking device thereof

Abstract

The present invention relates to a cooking method based on a cooking AI large model and an AI virtual human system, and an intelligent cooking device thereof. The present invention is characterized by being applied to an intelligent cooking device, the intelligent cooking device being provided with an information collection module, a storage module, a processor, a display screen, and a wireless communication module. The information collection module is used to receive cooking demand information and virtual human demand information, and push the information to a cooking AI large model and an AI virtual human system, respectively. The AI virtual human system is a cloud platform AI virtual human system or a built-in AI virtual human system. The cooking AI large model performs analysis and generates a new cooking method. The AI virtual human system performs analysis and generates a specific AI virtual human. During cooking, the display screen or a mobile terminal of a user outputs the specific AI virtual human. The present invention achieves the application of an AI large model and an AI virtual human in the field of cooking. Moreover, a user or a specific virtual human controls the intelligent cooking device to cook food that meets user demands, and the user can enjoy delicious food cooked by using artificial intelligence technology, and can also enjoy a comfortable cooking process. The built-in AI virtual human system generates a specific AI virtual human within the intelligent cooking device, which is not subject to traffic restrictions and has low costs.

Description

A cooking method and intelligent cooking device based on a cooking AI large model and an AI virtual human system Technical Field

[0001] The present invention relates to a cooking device and method, and in particular to a cooking method and an intelligent cooking device based on a cooking AI large model and an AI virtual human system. Background Art

[0002] With the development of science and technology and the accelerated pace of life, smart cooking devices have emerged, which can realize automated cooking.

[0003] The human-computer intelligent interaction system is one of the important applications of intelligent technology. It can automatically give feedback on questions asked by users, which can not only relieve users' boredom, but also assist users in their work. However, traditional intelligent interaction systems generally use text or voice to feedback the answers to questions after users ask questions. During the interaction process, the machine equipment cannot make facial expressions or movements while answering the questions. The interaction process is relatively "stiff" and cannot simulate the interaction process between people, which makes the user experience average.

[0004] Regarding the combination of AI big models and smart cooking devices, the latest existing technologies include patent application number CN 116843510 A, published on October 3, 2023, entitled "Smart Cooking Machine Cloud Platform Data Management System and Method Based on AI Big Model," and application number 202310784823.7;

[0005] This invention application discloses a cloud platform data management system and method for intelligent cooking machines based on an AI big model, which relates to the technical field of data management systems. The management method includes the following steps: various parameter data of the cooking process are collected through the collection end, and the parameter data are transmitted to the cloud platform through the network to ensure real-time and stability. The processing end analyzes the stored parameter data through the AI ​​big model to analyze whether there is any abnormality in the intelligent cooking machine during the cooking process. When the intelligent cooking machine is analyzed to have an abnormality, it is judged that there is a deviation in the taste of the dish based on the analysis result, and the management system sends a warning signal. At this time, other intelligent cooking machines are replaced for cooking. The invention can judge whether there is a deviation in the taste of the dish cooked this time based on the operating status of the intelligent cooking machine, so as to be able to issue a timely warning to ensure the quality and taste of the dish.

[0006] The role of applying AI big models to smart cooking machines in the existing technology is only for the processing end to analyze the stored parameter data through the AI ​​big model to analyze whether there is any abnormality in the smart cooking machine during the cooking process. When the analysis shows that there is an abnormality in the smart cooking machine, it is judged based on the analysis results that there is a deviation in the taste of the dish, and it is necessary to replace other smart cooking machines to ensure the quality and taste of the dish. In fact, replacing other smart cooking machines for cooking may not necessarily ensure the quality and taste of the dish, because the other replaced smart cooking machines may also have abnormalities. In this way, users need to constantly replace smart cooking machines to ensure the quality and taste of the dish, and no new cooking methods are generated, and no interaction with users.

[0007] Summary of the Invention

[0008] Based on the deficiencies of the existing technology, the present invention provides a cooking method based on a cooking AI big model and an AI virtual human system and an intelligent cooking device thereof. The information collection module of the intelligent cooking device receives cooking demand information and virtual human demand information input by the user in voice, video, text, picture or 3D model format, and pushes this cooking demand information and this virtual human demand information to the cooking AI big model and the AI ​​virtual human system respectively, thereby generating a new cooking method and generating a specific AI virtual human. The user can interact with the specific AI virtual human. The specific AI virtual human can be a character image of a virtual relative, friend, celebrity or chef, etc. The specific AI virtual human can be presented in multiple multimedia formats such as video and sound. The user or the specific virtual human controls the intelligent cooking device to cook food that meets the user's needs. The user can not only enjoy delicious food cooked by artificial intelligence technology, but also enjoy a comfortable cooking process. Since the external AI virtual human system is generated on a cloud platform server, it has sufficient computing power to generate a more realistic specific AI virtual human.

[0009] In order to solve the above-mentioned problems in the prior art, the present invention provides a cooking method based on a cooking AI big model and an AI virtual human system, which is characterized in that it is applied to an intelligent cooking device, and the intelligent cooking device is provided with an information collection module, a storage module, a processor, a display screen and a wireless communication module, and the information collection module, the storage module, the display screen, and the wireless communication module are electrically connected to the processor, and the information collection module is used to receive cooking demand information and virtual human demand information input by the user in voice, video, text, picture or 3D model format, and push this cooking demand information and this virtual human demand information to the cooking AI big model and the AI ​​virtual human system respectively, and the AI ​​virtual human system is an AI virtual human system of a cloud platform or a built-in AI virtual human system, and the cooking AI big model analyzes the cooking demand information to generate a new cooking method, and the new cooking method is sent to the user's mobile phone through the wireless communication module. The terminal and the processor, the new cooking method includes cooking parameters and cooking operation instructions, and the cooking AI big model converts the cooking parameters and the cooking operation instructions into text information and sends it to the AI ​​virtual human system, the AI ​​virtual human system analyzes the virtual human demand information and the text information to generate a specific AI virtual human, and the AI ​​virtual human system sends the specific AI virtual human to the user mobile terminal and the display screen of the intelligent cooking device through the cloud platform or the wireless communication module. The cooking AI big model generates a new cooking method in two steps: constructing a cooking AI big model and calling the cooking AI big model. The AI ​​virtual human system generates a specific AI virtual human in two steps: constructing a specific AI virtual human and calling a specific AI virtual human. When the user confirms the new cooking method through the user mobile terminal or the display screen to cook, the display screen or the user mobile terminal outputs a specific AI virtual human.

[0010] As an improvement to the cooking method based on the cooking AI big model and the AI ​​virtual human system of the present invention, the construction of the cooking AI big model includes six steps: collecting cooking data, preprocessing cooking data, selecting an AI big model that can be applied to cooking, training the cooking AI big model, verifying and testing the cooking AI big model, and deploying and maintaining the cooking AI big model on a cloud platform. The cooking requirement information input by the user in the format of voice, video, text, picture or 3D model is used to generate a new cooking method by calling the cooking AI big model. The calling of the cooking AI big model includes assembling query statements, the AI ​​big model performing reasoning operations, and the AI ​​big model returning results. The new cooking method is sent to the user's mobile terminal and the processor through the cloud platform after the reasoning operation of the cooking AI big model is completed.

[0011] As an improvement to the cooking method based on the cooking AI big model and the AI ​​virtual human system of the present invention, the construction of the cooking AI big model includes six steps: collecting cooking data, preprocessing cooking data, selecting an AI big model that can be applied to cooking, training the cooking AI big model, verifying and testing the cooking AI big model, and deploying and maintaining the cooking AI big model in the storage module. The cooking requirement information input by the user in the format of voice, video, text, picture or 3D model is used to generate a new cooking method by calling the cooking AI big model. The calling of the cooking AI big model includes assembling a query statement, the AI ​​big model performing reasoning operations, and the AI ​​big model returning results. The new cooking method is sent to the user's mobile terminal and the processor through the wireless communication module after the reasoning operation of the cooking AI big model is completed.

[0012] As an improvement of the cooking method based on the cooking AI big model and AI virtual human system of the present invention, the construction of a specific AI virtual human includes seven steps: character generation, voice generation, lip synchronization, action generation, synthetic display, test verification and deployment maintenance. The virtual human demand information input by the user in voice, video, text, picture or 3D model format is used to generate a specific AI virtual human through the AI ​​virtual human system. The calling of the specific AI virtual human includes generating text instruction content for the specific AI virtual human. The corresponding text instruction content can be generated by the cooking AI big model, and the text instruction content can be transmitted to the specific AI virtual human through API call, and the video is generated by the specific AI virtual human.

[0013] As an improvement to the cooking method based on the cooking AI big model and the AI ​​virtual human system of the present invention, the construction of the cooking AI big model includes:

[0014] Step 1: Collect cooking data: Collect all information about cooking methods of ingredients presented through voice, video, text, pictures or 3D models;

[0015] Step 2: Preprocessing cooking data: All collected information about cooking methods is processed to ensure its integrity and usability. This includes converting information in different formats into text and editing the text according to a specific format to facilitate subsequent training of the AI ​​model.

[0016] Step 3: Select AI models applicable to cooking: Select domestic and international third-party AI models, and measure them using accuracy, response speed, and diversity indicators;

[0017] Step 4: Train the cooking AI model: After organizing the cooking dataset in step 2, fine-tune the cooking dataset using a third-party AI model. After training, a cooking AI model containing all relevant data about cooking methods is generated.

[0018] Step 5: Verify and test the cooking AI big model: Perform a task-specific performance test on the cooking AI big model generated in step 4. If the performance fails, repeat steps 1, 2, 3, and 4 and retrain until the performance passes. Generate a cooking AI big model and store it on the cloud platform or in the storage module of the smart cooking device.

[0019] Step 6. Deploy and maintain the cooking AI big model: Deploy the newly generated AI big model to the cloud platform or the storage module of the smart cooking device, and carry out continuous maintenance and updates, and update the data regularly to ensure the timeliness and accuracy of the data.

[0020] As an improvement to the cooking method based on the cooking AI big model and AI virtual human system of the present invention, the third step selects an AI big model applicable to cooking, and the selected AI big model is the Baichuan2-13B AI big model. The parameters of the AI ​​big model are as follows: hidden layer dimension: 5,120, number of layers: 40, number of attention heads: 40, vocabulary size: 64,000, total number of parameters: 13,264,901,120, training data (tokens): 1.4 trillion, position encoding: ALiBi, maximum length: 4,096;

[0021] As an improvement to the cooking method based on the cooking AI big model and AI virtual human system of the present invention, the training process of the fourth step of training the cooking AI big model is as follows: first, download the model weights of baichuan13b from huggingface, then download the belle dataset train_0.5M_CN to the local and put it in the dataset folder under the project directory, and finally run the sft_lora.py script. Then, Baichuan LLM is quantized using qlora's nf4 and double quantization methods. Finally, lora is used to fine-tune the instructions.

[0022] As an improvement to the cooking method based on the cooking AI big model and the AI ​​virtual human system of the present invention, the construction of a specific AI virtual human applied to the AI ​​virtual human system of the cloud platform includes:

[0023] The first step, character generation, involves creating a virtual human model from multiple user-uploaded headshots. The character generation submodule is responsible for generating a specific virtual human image based on user-input photos. This specific virtual human generation technology utilizes AIGC technology.

[0024] The second step is voice generation: This involves generating a simulated voice model from multiple user-uploaded voice segments. The voice generation submodule is responsible for generating the voice output of a virtual digital human based on input text or voice, using both speech synthesis and voice cloning technologies.

[0025] Third, lip generation: Generate a corresponding lip model based on the generated speech. The lip generation module is responsible for synchronizing the generated speech with the lips of the specific virtual person, making the specific virtual person's conversation appear more realistic and natural.

[0026] Step 4: Action Generation: Generate a simulated action model from multiple action videos uploaded by users. The action generation submodule is responsible for generating the action output of the virtual digital human based on the input video, including two sub-steps: driving and rendering.

[0027] Step 5. Synthetic display: The synthetic display module is responsible for synthesizing the voice and action output of the virtual digital human into video and displaying it on different devices, including 2D and 3D display devices.

[0028] Step 6: Verification test: The specific AI virtual human video generated in step 5 will be tested and evaluated for the effectiveness of a specific task. If the evaluation fails, the first, second, third, fourth, and fifth steps will be repeated and retrained until the effect evaluation passes.

[0029] Step 7: Deployment and maintenance: Deploy the newly generated specific AI virtual person to the operating system of the cloud platform and perform continuous maintenance and updates.

[0030] As an improvement to the cooking method based on the cooking AI large model and the AI ​​virtual human system of the present invention, the construction of a specific AI virtual human applied to the built-in AI virtual human system includes:

[0031] The first step, character generation, involves creating a virtual human model from multiple user-uploaded profile photos. The character generation submodule is responsible for generating a specific virtual human image based on user-entered photos. This specific virtual human generation technology utilizes face-swapping technology.

[0032] Step 2: Speech Generation: Generate a simulated speech model based on the input text. The speech generation submodule is responsible for generating the speech output of the virtual digital human based on the input text.

[0033] Step 3: Lip Generation: Generate a corresponding lip model based on the generated speech. The lip generation module is responsible for synchronizing the generated speech with the lips of the specific virtual person, making the specific virtual person's conversation appear more realistic and natural.

[0034] Step 4: Action Generation: Generate a simulated action model using the built-in action template selected by the user. The action generation submodule is responsible for generating simulated actions based on the built-in action template selected by the user, including two sub-steps: conversion and rendering.

[0035] Step 5. Synthetic display: The synthetic display module is responsible for synthesizing the voice and action output of the virtual digital human into video and displaying it on different devices, including 2D and 3D display devices.

[0036] Step 6: Verification test: The specific AI virtual human video generated in step 5 will be tested and evaluated for the effectiveness of a specific task. If the evaluation fails, the first, second, third, fourth, and fifth steps will be repeated and retrained until the effect evaluation passes.

[0037] Step 7: Deployment and maintenance: Deploy the newly generated specific AI virtual human to the operating system of the smart cooking appliance and perform continuous maintenance and updates.

[0038] As an improvement to the cooking method based on the cooking AI large model and AI virtual human system of the present invention, the calling of a specific AI virtual human specifically includes the following steps:

[0039] The first step is to generate text instructions for a specific AI virtual person. This can be generated by a large cooking AI model or by a system program based on business logic.

[0040] The second step is to pass the text command content to the AI ​​virtual human system through API calls, and the AI ​​virtual human system will generate the video. This includes the following processes:

[0041] 1) Text-to-speech: The text content is converted into the voice of a specific AI virtual human through the voice generation module of the AI ​​virtual human system, and the voice uploaded by the user can be cloned.

[0042] 2) Lip generation: The speech content generated in the first step is synchronized with the lips of a specific AI virtual person through an algorithm, which can achieve synchronization of both Chinese and English.

[0043] 3) Action generation: Select the action template uploaded by the user, output the corresponding action posture of the specific AI virtual human according to the user's action template, and combine it with the lip synchronization in the second step.

[0044] 4) Video synthesis: Output the content generated in the previous three steps into a standard video format and return it.

[0045] The present invention provides an intelligent cooking device, characterized in that it is provided with an information collection module, a storage module, a processor and a wireless communication module. The information collection module is used to collect all information about cooking methods of ingredients and virtual human demand information input by the user through voice, video, text, picture or 3D model format. The processor is used to execute the new cooking method generated by the cooking AI large model in claim 1, 2 or 3 and prompt the user or a specific AI virtual human to perform cooking operation instructions.

[0046] As an improvement to the intelligent cooking device of the present invention, the intelligent cooking device of the present invention is also provided with a built-in AI virtual human system, which is located in the storage module, electrically connected to the processor, and connected to the user mobile terminal via the wireless communication module.

[0047] As an improvement to the intelligent cooking device of the present invention, it further includes an AI virtual human system located on a cloud platform, and the AI ​​virtual human system is connected to the user's mobile terminal via the wireless communication module.

[0048] As an improvement of the intelligent cooking device of the present invention, the intelligent cooking device is provided with intelligent cooking device stations for stir-frying, sautéing, stir-frying, deep-frying, cooking, frying, sticking, roasting, braising, stewing, steaming, blanching, boiling, stewing, sautéing, mixing, marinating, roasting, braising, freezing, pulling out silk, honey-juicing, smoking, rolling, sliding or baking.

[0049] As an improvement to the intelligent cooking device of the present invention, the intelligent cooking device stations for stir-frying, sautéing, deep-frying, cooking, frying, sticking, roasting, braising, stewing, steaming, blanching, boiling, stewing, sautéing, mixing, marinating, roasting, braising, freezing, candied food, honey-glazed food, smoking, rolling, sliding or baking are provided with corresponding operation detection feedback systems, and the operation detection feedback system is used to detect whether the cooking operations performed by the user meet the requirements of the cooking method of the cooking AI large model.

[0050] As an improvement of the intelligent cooking device of the present invention, the intelligent cooking device is provided with a human-computer interaction module, which is used for information interaction between the intelligent cooking device and the user, including the user operating the user mobile terminal to confirm cooking parameters and start cooking operation instructions.

[0051] The present invention provides a cooking method and intelligent cooking device based on a cooking AI large model and an AI virtual human system. The intelligent cooking device has the following beneficial effects: the information collection module of the intelligent cooking device receives cooking requirement information and virtual human requirement information input by a user in voice, video, text, image, or 3D model format, and pushes the cooking requirement information and virtual human requirement information to the cooking AI large model and the AI ​​virtual human system, respectively, thereby generating a new cooking method and a specific AI virtual human. The user can interact with the specific AI virtual human, which can be a character image of a virtual relative, friend, celebrity, or chef, and can be presented in multiple multimedia formats such as video and sound. The user or the specific virtual human controls the intelligent cooking device to cook food that meets the user's needs. The user can not only enjoy delicious food cooked using artificial intelligence technology, but also enjoy a comfortable cooking process. Since the external AI virtual human system is generated on a cloud platform server and has sufficient computing power, it can generate a more realistic specific AI virtual human. The specific AI virtual human generated by the internal AI virtual human system is generated within the intelligent cooking device, is not subject to traffic restrictions, and is low in cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] FIG1 is a flowchart of constructing a specific AI virtual human according to a preferred embodiment of the cooking method and intelligent cooking device based on the cooking AI big model and AI virtual human system of the present invention.

[0053] FIG2 is a flowchart of calling a specific AI virtual human according to a preferred embodiment of the cooking method and intelligent cooking device based on the cooking AI big model and AI virtual human system of the present invention.

[0054] FIG3 is a workflow diagram of a preferred embodiment of a cooking method and an intelligent cooking device thereof based on a cooking AI big model and an AI virtual human system according to the present invention.

[0055] FIG4 is a workflow diagram of another embodiment 1 of the cooking method and intelligent cooking device based on the cooking AI big model and AI virtual human system of the present invention.

[0056] FIG5 is a workflow diagram of another embodiment 2 of the cooking method and intelligent cooking device based on the cooking AI big model and AI virtual human system of the present invention.

[0057] FIG6 is a workflow diagram of another third embodiment of the cooking method and intelligent cooking device based on the cooking AI big model and AI virtual human system of the present invention.

[0058] FIG7 is a workflow diagram of another fourth embodiment of the cooking method and intelligent cooking device based on the cooking AI big model and AI virtual human system of the present invention.

[0059] FIG8 is a workflow diagram of another fifth embodiment of the cooking method and intelligent cooking device based on the cooking AI big model and AI virtual human system of the present invention.

[0060] FIG9 is a flowchart of constructing a specific AI virtual human according to another sixth embodiment of the cooking method and intelligent cooking device thereof based on the cooking AI big model and AI virtual human system of the present invention.

[0061] FIG10 is a flowchart of another sixth embodiment of the cooking method and intelligent cooking device based on the cooking AI big model and AI virtual human system of the present invention. DETAILED DESCRIPTION

[0062] The present invention is further described below in conjunction with Figures 1-10 and specific implementation methods and other embodiments. It should be noted that, under the premise of no conflict, the various technical features described below can be arbitrarily combined to form new embodiments.

[0063] In a preferred embodiment, referring to Figures 1-3, the present invention provides a cooking method based on a cooking AI big model and an AI virtual human system, characterized in that it is applied to an intelligent cooking device, and the intelligent cooking device is provided with an information collection module 13, a storage module 12, a processor 11, a display screen 8 and a wireless communication module 9. The information collection module 13, the storage module 12, the display screen 8, the wireless communication module 9 are electrically connected to the processor 11, and the information collection module 13 is used to receive cooking demand information and virtual human demand information input by the user 5 in voice, video, text, picture or 3D model format, and push this cooking demand information and this virtual human demand information to the cooking AI big model and the AI ​​virtual human system respectively. In this embodiment, the AI ​​virtual human system and the cooking AI big module are both located on the cloud platform. The cooking AI big model analyzes the cooking demand information to generate a new cooking method, and the new cooking method is sent to the user mobile terminal and the processor via the wireless communication module. It includes cooking parameters and cooking operation instructions. At the same time, the cooking AI big model converts the cooking parameters and the cooking operation instructions into text information and sends it to the AI ​​virtual human system. The AI ​​virtual human system analyzes the virtual human demand information and the text information to generate a specific AI virtual human. The AI ​​virtual human system sends the specific AI virtual human to the user mobile terminal and the display screen of the intelligent cooking device through the cloud platform. The cooking AI big model generates a new cooking method, which includes two steps: building a cooking AI big model and calling the cooking AI big model. The AI ​​virtual human system generates a specific AI virtual human, which includes two steps: building a specific AI virtual human and calling a specific AI virtual human. When the user confirms the new cooking method through the user mobile terminal or the display screen to cook, the display screen or the user mobile terminal outputs a specific AI virtual human; wherein, the specific AI virtual human can be a character image of a virtual relative, friend, celebrity or chef, etc., and the specific AI virtual human can be presented in various multimedia formats such as video and sound.

[0064] The virtual human requirement information is the character feature information such as images, actions and voices of relatives, friends, celebrities or chefs required to build a specific virtual human.

[0065] In a preferred embodiment, the cooking AI big model analyzes the cooking demand information to generate a new cooking method, and the new cooking method is sent to the user mobile terminal 7 and the processor 11 via the wireless communication module 9. The new cooking method includes cooking parameters and cooking operation instructions;

[0066] With reference to FIG. 1 , in a preferred embodiment, the process of constructing a specific AI virtual human 1 in the AI ​​virtual human system of the cloud platform 6 based on the cooking AI large model and the cooking method of the AI ​​virtual human system of the present invention is as follows:

[0067] 101. Character Generation: Generates a virtual human model from multiple headshots uploaded by User 5. The character generation submodule is responsible for generating a specific virtual human image based on the photos input by User 5. Specifically, User 5 uploads multiple headshots to the cloud platform server, which then uses AIGC technology to generate the user's desired virtual human model. The entire process takes approximately 30 minutes. Specific virtual human generation technologies use AIGC, such as stable diffusion and Lora.

[0068] 102. Speech Generation: Generates a simulated speech model from multiple voice segments uploaded by User 5. The speech generation submodule is responsible for generating the speech output of a virtual digital human based on input text or speech. It includes two technologies: speech synthesis and speech cloning. Speech synthesis technology converts text into speech through software algorithms, using end-to-end deep learning-based models such as Tacotron and WaveNet. Speech cloning technology transfers the speech characteristics of a real person to a virtual digital human through software algorithms, using deep learning-based vocoder models such as VoiceClone and Real-Time-Voice-Cloning.

[0069] 103. Lip Generation: Generates a corresponding lip model based on the generated speech. The lip generation module is responsible for synchronizing the generated speech with the lips of the specific virtual person, making the conversation with the specific virtual person appear more realistic and natural. Lip generation technology uses deep learning-based lip synchronization models such as SyncNet.

[0070] 104. Action Generation: Generates a simulated action model from multiple action videos uploaded by User 5. The action generation submodule is responsible for generating the action output of the virtual digital human based on the input video, and includes two sub-steps: driving and rendering. The driving step involves converting the uploaded video into the expression and posture parameters of the virtual digital human through software algorithms, using deep learning-based sequence-to-sequence models such as LipGAN and Speech-Driven Animation. The rendering step involves converting the expression and posture parameters of the virtual digital human into image or video output through software algorithms, typically using deep learning-based image-to-image or video-to-video models such as the First Order Motion Model and Neural Talking Head.

[0071] 105. Composite Display: The composite display module is responsible for synthesizing the virtual human's voice and motion output into video and displaying it on various devices, including 2D and 3D displays. 2D display devices are those that output two-dimensional images, such as mobile phones, televisions, projectors, and LED displays. 3D display devices are those that output three-dimensional images, such as autostereoscopic displays, AR, and VR.

[0072] 106. Verification test: The specific AI virtual human video generated in step 105 will be used to perform a specific task effect detection and evaluation. If the evaluation effect fails, steps 101, 102, 103, 104, and 105 will be repeated and retrained until the effect evaluation passes.

[0073] 107. Deployment and maintenance: Deploy the newly generated specific AI virtual human to the operating system of the cloud platform 6 and perform continuous maintenance and updates.

[0074] In another sixth embodiment, referring to Figures 9-10, the present invention provides a cooking method based on a cooking AI big model and an AI virtual human system, characterized in that it is applied to an intelligent cooking device, wherein the intelligent cooking device is provided with an information collection module 13, a storage module 12, a processor 11, a display screen 8 and a wireless communication module 9, wherein the information collection module 13, the storage module 12, the display screen 8, the wireless communication module 9 are electrically connected to the processor 11, and the information collection module 13 is used to receive cooking demand information and virtual human demand information input by the user 5 in the form of voice, video, text, picture or 3D model, and push the cooking demand information and the virtual human demand information to the cooking AI big model and the storage module 11 respectively. 2's built-in AI virtual human system; the built-in AI virtual human system analyzes and generates a specific AI virtual human, the cooking AI big model analyzes the cooking demand information to generate a new cooking method, the new cooking method is sent to the user's mobile terminal and the processor through the wireless communication module, the new cooking method includes cooking parameters and cooking operation instructions, and at the same time, the cooking AI big model converts the cooking parameters and the cooking operation instructions into text information and sends it to the AI ​​virtual human system, the AI ​​virtual human system analyzes the virtual human demand information and the text information to generate a specific AI virtual human, and the AI ​​virtual human system sends the specific AI virtual human to the user's mobile terminal and the display screen of the intelligent cooking device through the wireless communication module. The cooking AI big model generates a new cooking method in two steps: building a cooking AI big model and calling the cooking AI big model. The AI ​​virtual human system generates a specific AI virtual human in two steps: building a specific AI virtual human and calling a specific AI virtual human. When the user confirms the new cooking method through the user mobile terminal or the display screen to cook, the display screen or the user mobile terminal outputs the specific AI virtual human; wherein, the specific AI virtual human can be a character image of a virtual relative, friend, celebrity or chef, etc., and the specific AI virtual human can be presented in a variety of multimedia formats such as video and sound; wherein, the virtual human demand information is the character feature information such as images, actions and voices of relatives, friends, celebrities or chefs required to build a specific virtual human. In this embodiment, the process of building a specific AI virtual human 1 applied to the built-in AI virtual human system is as follows:

[0075] 101. Character Generation: This module generates a virtual human model from multiple user-uploaded profile photos. The character generation submodule is responsible for generating a specific virtual human image based on the user's input photos. The user selects a pre-installed virtual human. Using face-swapping technology, the selected virtual human's face is replaced with the user's uploaded profile photo. The entire process takes approximately one minute. Virtual human generation typically utilizes face-swapping technologies such as DeepFace.

[0076] 102. Speech Generation: Generates a simulated speech model from text input. The speech generation submodule is responsible for generating the speech output of a virtual digital human based on the input text. Speech synthesis technology converts text into speech through software algorithms, using end-to-end deep learning-based models such as Tacotron and WaveNet.

[0077] 103. Lip Generation: Generates a corresponding lip model based on the generated speech. The lip generation module is responsible for synchronizing the generated speech with the lips of the specific virtual person, making the conversation with the specific virtual person appear more realistic and natural. Lip generation technology uses deep learning-based lip synchronization models such as SyncNet.

[0078] 104. Action Generation: Generates a simulated action model using the built-in action template selected by the user. The action generation submodule is responsible for generating simulated actions based on the built-in action template selected by the user, and includes two sub-steps: conversion and rendering. The specific operation of the conversion step allows the user to select a specific virtual human action template that has been built into the system, and converts the selected specific virtual human action template into the expression and posture parameters of the virtual digital human through an algorithm, using a sequence-to-sequence model based on deep learning, such as LipGAN and Speech-Driven Animation. The rendering step refers to converting the expression and posture parameters of the virtual digital human into image or video output through a software algorithm, using an image-to-image or video-to-video model based on deep learning, such as the First Order Motion Model and Neural Talking Head.

[0079] 105. Composite Display: The composite display module is responsible for synthesizing the virtual human's voice and motion output into video and displaying it on various devices, including 2D and 3D displays. 2D display devices are those that output two-dimensional images, such as mobile phones, televisions, projectors, and LED displays. 3D display devices are those that output three-dimensional images, such as autostereoscopic displays, AR, and VR.

[0080] 106. Verification test: The specific AI virtual human video generated in step 105 will be used to perform a specific task effect detection and evaluation. If the evaluation effect fails, steps 101, 102, 103, 104, and 105 will be repeated and retrained until the effect evaluation passes.

[0081] 107. Deployment and maintenance: Deploy the newly generated specific AI virtual human to the operating system of the smart cooking appliance and perform continuous maintenance and updates.

[0082] With reference to FIG. 2 , in a preferred embodiment, the process of calling a specific AI virtual human 2 in the cooking method based on the cooking AI large model and the AI ​​virtual human system of the present invention is as follows:

[0083] 201. The text instruction content for a specific AI virtual person can be generated by a large cooking AI model or by a system program according to business logic.

[0084] 202. The text instruction content is transmitted to the AI ​​virtual human system of the cloud platform through an API call. In another sixth embodiment, the text instruction content is transmitted to the built-in AI virtual human system through an API call, and the AI ​​virtual human system generates the video. This includes the following steps:

[0085] 202-1. Text-to-speech: The text content is converted into the voice of a specific AI virtual human through the voice generation module of the AI ​​virtual human system, and the voice uploaded by user 5 can be cloned.

[0086] 202-2. Lip Generation: The speech content generated in step 202-1 is synchronized with the lips of a specific AI virtual person through an algorithm, and both Chinese and English can be synchronized.

[0087] 202-3. Action generation: Select the action template uploaded by the user, output the corresponding action posture of the specific AI virtual human according to the user's action template, and combine it with the lip synchronization of step 202-2.

[0088] 202-4. Video synthesis: Output the content generated in the previous three steps into a standard video format and return it.

[0089] In a preferred embodiment, the process of constructing the cooking AI big model 3 of the cooking method based on the cooking AI big model and the AI ​​virtual human system of the present invention is as follows:

[0090] Step 1: Collect cooking data: Collect all information about cooking methods of ingredients presented through voice, video, text, pictures or 3D models;

[0091] Step 2: Preprocessing cooking data: All collected information about cooking methods is processed to ensure its integrity and usability. This includes converting information in different formats into text and editing the text according to a specific format to facilitate subsequent training of the AI ​​model.

[0092] Step 3: Select AI models applicable to cooking: Select domestic and international third-party AI models, and measure them using accuracy, response speed, and diversity indicators;

[0093] Step 4: Train the cooking AI model: After organizing the cooking dataset in step 2, fine-tune the cooking dataset using a third-party AI model. After training, a cooking AI model containing all relevant data about cooking methods is generated.

[0094] Step 5: Verify and test the cooking AI model: Perform a task-specific performance test on the cooking AI model generated in step 4. If the performance fails, repeat steps 1, 2, 3, and 4, retraining the model until the performance passes. The cooking AI model is generated and stored on the cloud platform 6.

[0095] Step 6: Deploy and maintain the cooking AI model: Deploy the newly generated AI model to the cloud platform 6 and continuously maintain and update it. Update the data regularly to ensure its timeliness and accuracy.

[0096] In this embodiment, the selection of the process of constructing the cooking AI big model 3 based on the cooking AI big model and the cooking method of the AI ​​virtual human system of the present invention is specifically the AI ​​big model of Baichuan2-13B applied to cooking. The parameters of the AI ​​big model are as follows: hidden layer dimension: 5,120, number of layers: 40, number of attention heads: 40, vocabulary size: 64,000, total parameter quantity: 13,264,901,120, training data (tokens): 1.4 trillion, position encoding: ALiBi, maximum length: 4,096;

[0097] In this embodiment, the training process of the cooking AI big model 3 of the cooking method based on the cooking AI big model and the AI ​​virtual human system of the present invention is as follows: first, download the model weights of baichuan13b from huggingface, then download the belle dataset train_0.5M_CN to the local and put it in the dataset folder under the project directory, and finally run the sft_lora.py script, then quantize Baichuan LLM using qlora's nf4 and double quantization methods, and finally, use lora to fine-tune the instructions.

[0098] In a preferred embodiment, when user 5 uses the APP of the user mobile terminal 7 or the information collection module 13 of the smart cooking device to send out the virtual human demand information, for example, the user sends out his mother's photos, videos, voice and other information, the AI ​​virtual human system of the cloud platform will analyze the information of the user's mother and generate a specific AI virtual human with the image of the user's mother, and send this specific AI virtual human to the user mobile terminal 7 and the display screen 8 of the smart cooking device. In this way, the specific AI virtual human with the image of the user's mother will communicate and interact with user 5 through the user mobile terminal 7 and the display screen 8 of the smart cooking device, giving the user a warm feeling of communicating with his mother. For example, when user 5 needs to cook "scrambled eggs with tomatoes", user 5 can say to display screen 8: "Mom, I want to make scrambled eggs with tomatoes." The specific AI virtual person in the image of the user's mother on display screen 8 will make a corresponding smile and nod in response: "Okay, son, I will make scrambled eggs with tomatoes for you." The specific AI virtual person will then transmit this cooking requirement information to the cooking AI big model of the cloud platform 6. The cooking AI big model will analyze the cooking requirement information and generate a new cooking method. At the same time, the cooking AI big model will convert the cooking parameters and the cooking operation instructions into text information and send it to the AI ​​virtual person system. The AI ​​virtual person system will analyze the text information to generate the text instruction content of the specific AI virtual person and then send it to the specific AI virtual person. The specific AI virtual person will generate a video based on the text instruction content. Then the specific AI virtual person in the image of the user's mother on display screen 8 will anthropomorphically guide user 5 to press The cooking parameters and the cooking operation instructions are used for cooking. During the cooking process, the cooking AI large model will send the new cooking method from the cloud platform 6 to the wireless communication module 9 of the intelligent cooking device. The wireless communication module 9 transmits the new cooking method to the processor 11, the cooking parameters of the new cooking method or the cooking operation instructions that require the cooperation of the user 5. The cooking operation instructions include the operation steps and cooking work steps on the workstation of the intelligent cooking device. The processor 11 first determines the workstation of the intelligent cooking device according to the new cooking method, and then sets the relevant parameters for the workstation of the intelligent cooking device, and then controls the workstation of the intelligent cooking device to cook. During this cooking process, when some steps require the cooperation of the user 5, the processor 11 will guide the user 5 to complete them through the specific AI virtual person on the user mobile terminal 7 or the display screen 8. When cooking is completed, the processor 11 will control the workstation of the intelligent cooking device to perform a self-cleaning operation.

[0099] The above is just an example of the cooking process for one dish. In practice, the system can be further intelligently and personalized based on the needs of the user 5. The AI ​​cooking model can customize a variety of cooking methods based on the user 5's needs, and the user 5 can also input photos, videos, and voice information of multiple characters to customize the appearance, figure, expression, posture, voice, and other settings of a specific AI avatar. When using the intelligent cooking device, the user 5 interacts with the specific AI avatar to perform relevant cooking operations, giving the user 5 the experience of interacting with a real person and cooking together, making the cooking process more enjoyable and humane.

[0100] In this embodiment, the cooking method based on the cooking AI big model and the AI ​​virtual human system of the present invention, calling the cooking AI big model 4 specifically includes the following steps:

[0101] The first step is to assemble the query statement;

[0102] The second step is for the AI ​​big model to perform inference operations: The query statement is passed to the AI ​​big model, which then performs inference operations. This involves the following four steps:

[0103] 1. Understanding the input: Distributed semantic parsing first receives a text sequence and converts it into word vectors. This process is based on the distributed semantics assumption that the meaning of a word is determined by its context.

[0104] 2. Parameter association: context-focus chaining, inputting these word vectors into the Transformer Encoder to generate context representation;

[0105] 3. Generate answers: Generative probabilistic modeling. The model initializes the decoder part of the Transformer and inputs the encoder output and the current output sequence into the decoder. The decoder generates a probability distribution for the next word and selects the word with the highest probability or other set probability distribution as the output. This word will be added to the output sequence.

[0106] 4. Choose the most appropriate answer: Dynamic word string evolution, repeat the above steps, adding new words to the output sequence each time, until a complete output sequence is generated;

[0107] Step 3: The AI ​​big model returns the result: After the cooking AI big model completes the inference operation, it returns the information related to the new cooking method and the cooking operation instructions of the new cooking method, including providing the user with content in the form of text, pictures, audio, video or 3D models;

[0108] In a preferred embodiment, referring to FIG3 , the present invention provides an intelligent cooking device for executing a cooking method based on a cooking AI big model and an AI virtual human system, characterized in that an information collection module 13, a storage module 12, a display screen 8, a processor 11 and a wireless communication module 9 are provided. In this embodiment, the AI ​​virtual human system and the cooking AI big model are both located on a cloud platform, and the AI ​​virtual human system is connected to the user's mobile terminal via the cloud platform. The information collection module 13 is used to collect all information on the cooking method of ingredients and virtual human demand information input by the user 5 in the form of voice, video, text, picture or 3D model, and the processor 11 is communicated with the cloud platform 6 and the user's mobile terminal 7. The processor 11 is connected via the wireless communication module 9, and is used to execute the new cooking method generated by the cooking AI big model in claim 1, 2, or 3 and prompt the user 5 or a specific AI virtual person to perform cooking operation instructions. The information collection module 13 is used to receive the cooking requirement information and virtual person requirement information input by the user 5 in voice, video, text, picture or 3D model format, and push this cooking requirement information and this virtual person requirement information to the cooking AI big model and the AI ​​virtual person system of the cloud platform respectively. The cooking AI big model analyzes the cooking requirement information to generate a new cooking method, and the new cooking method is sent to the user mobile terminal 7 and the processor 11 via the wireless communication module 9. The new cooking method includes cooking parameters and cooking operation instructions. At the same time, the cooking AI big model converts the cooking parameters and the cooking operation instructions into text information and sends it to the AI ​​virtual human system. The AI ​​virtual human system analyzes the virtual human demand information and the text information to generate a specific AI virtual human. The AI ​​virtual human system sends the specific AI virtual human to the user mobile terminal 7 and the display screen 8 of the intelligent cooking device through the cloud platform 6. When the user 5 confirms the new cooking method through the user mobile terminal 7 or the display screen 8 to cook, the display screen 8 or the user mobile terminal 7 outputs the specific AI virtual human, or the information collection module 13 communicates with the user through the wireless communication module 9. The user mobile terminal 7 is connected to receive cooking demand information, the wireless communication module 9 is connected to the processor 11, the cloud platform 6 and the user mobile terminal 7, and is used for information transmission between the three. The cooking AI big model is deployed on the cloud platform 6, and the transmission of cooking AI big model information is completed through the cloud platform 6. The processor 11 can set the working parameters of the smart cooking device stations of frying 18, stewing 16, boiling 19, deep-frying 17, steaming 20 and stir-frying 15 of the smart cooking device, and control each smart cooking device station to execute relevant instructions, such as standby, start, stop, cleaning and other work instructions. When the cooking demand information of the user 5 is transmitted to the cooking AI big model of the cloud platform 6 via the wireless communication module 9,After the cooking AI model analyzes and generates a new cooking method, the wireless communication module 9 sends it to the user mobile terminal 7. After confirmation by the user 5, the cooking AI model sends the new cooking method to the processor 11. This cooking method includes specific cooking data information. The processor 11 will use this cooking data information to set parameters for the relevant intelligent cooking device stations (frying 18, stewing 16, boiling 19, deep-frying 17, steaming 20, and stir-frying 15) and control them to execute relevant instructions.

[0109] In another sixth embodiment, referring to FIG10, the present invention provides an intelligent cooking device for executing a cooking method based on a cooking AI big model and a built-in AI virtual human system, characterized in that it is provided with an information collection module 13, a processor 11, a built-in AI virtual human system, a display screen 8, a storage module 12 and a wireless communication module 9. In this embodiment, the cooking AI big model is provided on a cloud platform, the built-in AI virtual human system is located in the storage module 12, the information collection module 13, the wireless communication module 9, the storage module 12, the built-in AI virtual human system, the display screen 8 and the processor 11 The built-in AI virtual human system is electrically connected to the user's mobile terminal through the wireless communication module. The information collection module 13 is used to collect all information about the cooking method of ingredients and the virtual human demand information input by the user 5 in voice, video, text, picture or 3D model format. The processor 11 is connected to the cloud platform 6 and the user's mobile terminal 7 through the wireless communication module 9. The processor 11 is used to execute the new cooking method generated by the cooking AI large model in claim 1, 2 or 3 and prompt the user 5 or a specific AI virtual human to perform cooking operation instructions. The information collection module 13 is used to receive the cooking instructions from the user 5. The user 5 inputs the cooking demand information and the virtual human demand information in the form of voice, video, text, picture or 3D model, and pushes the cooking demand information and the virtual human demand information to the AI ​​big model and the built-in AI virtual human system of the cooking cloud platform 6 respectively. The cooking AI big model analyzes the cooking demand information and generates a new cooking method. The new cooking method is sent to the user mobile terminal 7 and the processor 11 through the wireless communication module 9. The new cooking method includes cooking parameters and cooking operation instructions. At the same time, the cooking AI big model converts the cooking parameters and the cooking operation instructions into text information and sends it to the built-in AI virtual human system. I virtual human system, the built-in AI virtual human system analyzes the virtual human demand information and the text information to generate a specific AI virtual human, the built-in AI virtual human system sends the specific AI virtual human to the user mobile terminal 7 and the display screen 8 of the intelligent cooking device via the wireless communication module. When the user 5 confirms the new cooking method through the user mobile terminal 7 or the display screen 8 and starts cooking, the display screen 8 or the user mobile terminal 7 outputs the specific AI virtual human, or the information collection module 13 connects to the user mobile terminal 7 via the wireless communication module 9 to receive the cooking demand information. The wireless communication module 9 connects the processor 11 with the cloud platform 6 and the user mobile terminal 7 for information transmission between the three. The cooking AI large model is deployed on the cloud platform 6, and the transmission of cooking AI large model information is completed through the cloud platform 6. The processor 11 can set the working parameters of the intelligent cooking device's frying 18, boiling 19, steaming 20, stir-frying 15, stewing 16, and deep-frying 17 stations.Each intelligent cooking device station is controlled to execute relevant instructions, such as standby, start, stop, and cleaning instructions. When the user 5's cooking demand information is transmitted to the cooking AI model of the cloud platform 6 via the wireless communication module 9, the cooking AI model analyzes and generates a new cooking method, which is then sent by the wireless communication module 9 to the user mobile terminal 7. After confirmation by the user 5, the cooking AI model will send this new cooking method to the processor 11. This cooking method includes specific cooking data information. The processor 11 will set parameters for the relevant intelligent cooking device stations (frying 18, boiling 19, steaming 20, stir-frying 15, stewing 16, and deep-frying 17) based on this cooking data information and control them to execute relevant instructions.

[0110] In this embodiment, the intelligent cooking device stations of frying 18, stewing 16, boiling 19, frying 17, steaming 20 and stir-frying 15 of the intelligent cooking device of the present invention are provided with a common operation detection feedback system 14, and the operation detection feedback system 14 includes detection devices such as cameras, infrared detection, radar detection, magnetic detection, and weight detection, which are used to detect whether the cooking operation performed by the user 5 in the intelligent cooking device meets the requirements of the cooking method based on the cooking AI large model and the AI ​​virtual human system. For example, in the intelligent cooking device station of frying 18, the intelligent cooking device requires the user 5 to put the ingredients to be fried on the station, but the user 5 does not put them or When it is misplaced, the operation detection feedback system 14 detects it through the camera and feeds back this information to the processor 11 of the intelligent cooking device. The processor 11 controls the intelligent cooking device to pause the next operation and sends information to the user 5 for correction. After the user 5 corrects it, the operation detection feedback system 14 detects it through the camera and feeds back this information to the processor 11. The processor 11 controls the intelligent cooking device to perform the next operation. The above is just an example. In addition, infrared detection, radar detection, magnetic detection, and weight detection can be used alone or in combination for detection.

[0111] In another embodiment 1, referring to FIG. 4 , the intelligent cooking device stations of stir-frying 15 , stewing 16 , frying 17 , and pan-frying 18 of the intelligent cooking device of the present invention are provided with a common operation detection feedback system 14 , and the rest are the same as the preferred embodiment of the present invention;

[0112] In another embodiment 2, referring to FIG. 5 , the intelligent cooking device stations of the stir-frying 15 , stewing 16 , and frying 17 of the intelligent cooking device of the present invention are provided with a common operation detection feedback system 14 , and the rest are the same as the preferred embodiment of the present invention;

[0113] In another third embodiment, referring to FIG. 6 , the stir-frying 15 and stewing 16 intelligent cooking device stations of the intelligent cooking device of the present invention are provided with a common operation detection feedback system 14 , and the rest are the same as the preferred embodiment of the present invention;

[0114] In another fourth embodiment, referring to FIG. 7 , the intelligent cooking device station of the stir-fry 15 of the intelligent cooking device of the present invention is provided with an operation detection feedback system 14 , and the rest is the same as the preferred embodiment of the present invention;

[0115] In another fifth embodiment, referring to FIG. 8 , the cooking AI model of the intelligent cooking device of the present invention is built into the storage module 12 of the intelligent cooking device, and the rest is the same as the preferred embodiment of the present invention;

[0116] In other embodiments, the smart cooking device is one or a combination of a cooker, an air fryer, an induction cooker, a microwave oven, an oven, a steam oven, an electric rice cooker, an electric pressure cooker, an electric stew pot, an electric stew pot, a slow cooker, an integrated stove or a frying pan, and one or more smart cooking devices include one or more cooking functions of stir-frying, air frying, baking, frying, braising, stewing, steaming, boiling or baking to cook ingredients.

[0117] In other embodiments, the smart cooking device prompts the user 5 to place different ingredients on different workstations on the smart cooking device for cooking. For example, the smart cooking device workstation 15 for stir-frying is different from the smart cooking device workstations 20 for steaming, 17 for frying, and 18 for pan-frying in the smart cooking device. The user 5 needs to be prompted to cook at the corresponding smart cooking device workstation; the cooking work steps include the cooking steps that need to be performed when the smart cooking device prompts the user 5 to cook different ingredients; the cooking parameters include the cooking time, cooking temperature, and cooking power of the ingredients; the cooking operation instructions that require the cooperation of the user 5 include confirming the cooking parameters on the mobile terminal or confirming that the operation of the smart cooking device workstation and cooking work steps has been completed.

[0118] In other embodiments, the intelligent cooking device is provided with a projection device, and the AI ​​virtual human system analyzes and generates a specific AI virtual human, including but not limited to various multimedia formats such as video and sound, and projects the specific AI virtual human outside the intelligent cooking device body through the projection device, such as on a desktop or wall.

[0119] In other embodiments, the smart cooking device is an integrated body of multiple cooking functions including stir-frying, air frying, baking, frying, braising, stewing, steaming, boiling or baking. For example, the smart cooking device is a cooking utensil that integrates functions such as air frying, baking, frying, and steaming, or a cooking utensil that integrates functions such as stir-frying, air frying, braising, stewing, and boiling, etc.

[0120] In other embodiments, the intelligent cooking device is provided with multiple cooking functions including one or more cooking functions of stir-frying, sautéing, deep-frying, cooking, frying, sticking, roasting, braising, stewing, steaming, blanching, boiling, stewing, sautéing, mixing, marinating, roasting, braising, freezing, candied food, honey sauce, smoking, rolling, sliding or baking. For example, the intelligent cooking device is a cooking utensil that integrates functions such as honey sauce, roasting, frying and steaming, or a cooking utensil that integrates functions such as stir-frying, air frying, braising, stewing and boiling, etc.

[0121] In this embodiment, the intelligent cooking device of the present invention is provided with a human-computer interaction module 10, which includes a touch screen, a sound input and output unit, an image input unit and a camera device. The human-computer interaction module 10 is electrically connected to the processor 11. The user inputs virtual human demand information in the human-computer interaction module 10 through voice, video, text, picture or 3D model format. The processor 11 sends the virtual human demand information to the AI ​​virtual human system of the cloud platform 6 via the wireless communication module 9. The AI ​​virtual human system analyzes the virtual human demand information to generate a specific AI virtual human, and sends the specific AI virtual human to the human-computer interaction module 10 of the intelligent cooking device through the wireless communication module 9. The user 5 can communicate with the specific AI virtual human about cooking demand information through the human-computer interaction module 10 in the form of voice, video, text, picture or 3D model. The specific AI virtual human sends this cooking demand information to the processor 11 through the human-computer interaction module 10, and the processor 11 sends this cooking demand information to the storage module 12. The cooking AI big model will analyze the cooking demand information and generate a new cooking method, and then send the new cooking method to the processor 11. At the same time, the cooking AI big model will convert the cooking parameters and the cooking operation instructions into text information and send it to the specific AI virtual person of the human-computer interaction module 10 through the processor 11. The specific AI virtual person will analyze the text information and synthesize the video. Then, the specific AI virtual person will interact with the user 5 through the synthesized video and voice through the human-computer interaction module 10. When the user 5 confirms the cooking operation instructions to the specific AI virtual person through gestures or voice, the processor 11 will first determine the work station of the intelligent cooking device according to the new cooking method, and then set the relevant parameters for the work station of the intelligent cooking device, and then control the work station of the intelligent cooking device to cook. During this cooking process, when some steps require the cooperation of the user 5, the processor 11 will communicate with the user 5 through the specific AI virtual person of the human-computer interaction module 10 and guide the user 5 to perform relevant operations through the specific AI virtual person.

[0122] In this embodiment, the touch screen and display screen 8 of the human-computer interaction module 10 of the intelligent cooking device of the present invention are integrated, and the user 5 and the specific AI virtual person can exchange information through the human-computer interaction module 10 or the user mobile terminal 7.

[0123] The cooking method based on the cooking AI large model and AI virtual human system and the intelligent cooking device thereof are used for cooking in the following specific implementation steps:

[0124] The cooking data preprocessing process for constructing the AI ​​cooking model 3 in the cooking method based on the AI ​​cooking model and AI virtual human system of the present invention is specifically implemented as follows: All collected information is processed to ensure its integrity and usability. This includes converting information in different formats into text and editing the text information according to a specific format to facilitate subsequent training of the AI ​​model.

[0125] The data format is as follows:

[0126] instruction: Task instruction, cannot be empty.

[0127] input: Task input, can be empty. If not empty, the project will process the training data internally.

[0128] Instruction and input are spliced ​​together as the input of the task

[0129] output: task output, cannot be empty

[0130] The data example is as follows:

[0131] {

[0132] "instruction":"How to cook braised pork pre-cooked dishes",

[0133] "input":"",

[0134] 1. Place the pork belly skin-side down in the pan and fry over low heat until both sides are golden brown and a slightly crispy crust forms on the surface. 2. Leave a small amount of oil in the pan, add the ginger and garlic cloves and stir-fry until fragrant.

[0135] 3. Add water, enough to cover the meat. 4. Reduce heat to low, cover and simmer for about 60-90 minutes, or until the meat is tender and the broth is thick.

[0136] }

[0137] The third step of the process of calling the cooking AI big model 4 of the cooking method based on the cooking AI big model and the AI ​​virtual human system of the present invention is specifically implemented as follows: after the cooking AI big model completes the inference operation, it returns the relevant information of the new cooking method and the cooking operation instructions of the new cooking method, including the content provided to the user 5 in text, picture, audio, video or 3D model format, and returns the content in Jason format as follows:

[0138] {

[0139] “recipeName”:recipeName / / cooking method name,

[0140] "recipeText":text / / cooking method information,

[0141] "recipePic":pic_url / / Url of the picture of cooking method information,

[0142] "recipeAudio": audio_url / / audio URL of cooking method information,

[0143] "recipeVideo": video_url / / Video URL of cooking method information,

[0144] "recipe3D":3D_url / / 3D model URL of cooking method information,

[0145] “recipeCommand”:recipeCommand / / cooking method cooking operation instructions

[0146] }

[0147] The cooking method based on the cooking AI large model and AI virtual human system and the intelligent cooking device thereof for cooking braised pork are specifically described as follows:

[0148] 1. The request for cooking braised pork is sent to the cooking AI model;

[0149] 2. Call the cooking AI big model through the Prompt query statement, and the cooking AI big model performs inference operations.

[0150] 3. After the inference operation, the cooking AI model returns the following results:

[0151] {

[0152] "recipeName":"Braised Pork Cooking Method" / / Cooking method name,

[0153] "recipeText":"1. Place the pork belly skin-side down in the pan and fry over low heat until both sides are golden brown and a slightly crispy crust forms on the surface of the meat. 2. Leave a little oil in the pan and add the ginger and garlic cloves and stir-fry until fragrant. 3. Add water to just cover the pork. 4. Reduce heat to low, cover the pot and simmer for about 60-90 minutes, or until the pork is tender and the gravy is thick." / / Recipe information,

[0154] "recipePic": "None" / / None,

[0155] “recipeAudio”:

[0156] "https: / / www.ixigua.com / 7289152022341747234?logTag=43fb2a078d33289b226b" / / Audio URL for cooking method information,

[0157] “recipeVideo”:

[0158] "https: / / www.ixigua.com / 7289152022341747234?logTag=43fb2a078d33289b226b" / / Video URL for cooking method information,

[0159] "recipe3D":3D_url / / 3D model URL of cooking method information,

[0160] "recipeCommand": "Fry for 30 seconds, adjust the temperature to 100 degrees, stir-fry for 20 seconds, steam for 2 minutes, and then finish the operation" / / Cooking method and cooking operation instructions

[0161] }

[0162] The present invention provides a cooking method and intelligent cooking device based on a large cooking AI model and an AI virtual human system. The intelligent cooking device has the following beneficial effects: the information collection module of the intelligent cooking device receives cooking requirement information and virtual human requirement information input by a user in voice, video, text, image, or 3D model format, and pushes the cooking requirement information and virtual human requirement information to the large cooking AI model and the AI ​​virtual human system, respectively, thereby generating a new cooking method and a specific AI virtual human. The user can interact with the specific AI virtual human, which can be a virtual image of a relative, friend, celebrity, or chef, and can be presented in multiple multimedia formats such as video and sound. The user or the specific virtual human controls the intelligent cooking device to cook food that meets the user's needs. The user can not only enjoy delicious food cooked using artificial intelligence technology, but also enjoy a comfortable cooking process. Since the external AI virtual human system is generated on a cloud platform server with sufficient computing power, it can generate a more realistic specific AI virtual human. The specific AI virtual human generated by the internal AI virtual human system is generated within the intelligent cooking device, is not subject to traffic restrictions, and is low-cost.

[0163] The present invention has been described in detail above. The above description is only a preferred embodiment of the present invention and should not limit the scope of implementation of the present invention. That is, all equivalent changes and modifications made within the scope of this application should still fall within the scope of the present invention.

Claims

1. A cooking method based on a cooking AI large model and an AI virtual human system, characterized in that, Applied to an intelligent cooking device, the intelligent cooking device is provided with an information collection module, a storage module, a processor, a display screen and a wireless communication module. The information collection module, the storage module, the display screen and the wireless communication module are electrically connected to the processor. The information collection module is used to receive cooking requirement information and virtual human requirement information input by the user in the formats of voice, video, text, picture or 3D model, and respectively push the cooking requirement information and the virtual human requirement information to a cooking AI large model and an AI virtual human system. The AI virtual human system is an AI virtual human system of a cloud platform or a built-in AI virtual human system. The cooking AI large model analyzes the cooking requirement information to generate a new cooking method. The new cooking method is sent to the user mobile terminal and the processor through the wireless communication module. The new cooking method includes cooking parameters and cooking operation instructions. At the same time, the cooking AI large model converts the cooking parameters and the cooking operation instructions into text information and sends it to the AI virtual human system. The AI virtual human system analyzes the virtual human requirement information and the text information to generate a specific AI virtual human. The AI virtual human system sends the specific AI virtual human to the user mobile terminal and the display screen of the intelligent cooking device through the cloud platform or through the wireless communication module. The cooking AI large model generating a new cooking method includes two steps: constructing the cooking AI large model and invoking the cooking AI large model. The AI virtual human system generating a specific AI virtual human includes two steps: constructing the specific AI virtual human and invoking the specific AI virtual human. When the user confirms to cook according to the new cooking method through the user mobile terminal or the display screen, the display screen or the user mobile terminal outputs the specific AI virtual human.

2. The cooking method based on the cooking AI large model and the AI virtual human system according to claim 1, characterized in that: The constructing of the cooking AI large model includes six steps: collecting cooking data, preprocessing the cooking data, selecting an AI large model applicable to cooking, training the cooking AI large model, validating and testing the cooking AI large model, and deploying and maintaining the cooking AI large model on the cloud platform. The cooking requirement information input by the user in the formats of voice, video, text, picture or 3D model generates a new cooking method by invoking the cooking AI large model. The invoking of the cooking AI large model includes assembling a query statement, the AI large model performing inference operations and the AI large model returning results. After the new cooking method is completed by the inference operation of the cooking AI large model, it is sent to the user mobile terminal and the processor through the cloud platform.

3. The cooking method based on the cooking AI large model and the AI virtual human system according to claim 1, characterized in that: The construction of the cooking AI large model includes six steps: collecting cooking data, preprocessing cooking data, selecting an AI large model applicable to cooking, training the cooking AI large model, validating and testing the cooking AI large model, and deploying and maintaining the cooking AI large model in the storage module. The cooking requirement information input by the user in the form of voice, video, text, picture or 3D model generates a new cooking method by calling the cooking AI large model. The calling of the cooking AI large model includes assembling a query statement, the AI large model performing inference operations, and the AI large model returning results. The new cooking method is sent to the user's mobile terminal and the processor through the wireless communication module after being completed by the inference operation of the cooking AI large model.

4. The cooking method based on the cooking AI large model and the AI virtual human system according to claim 1, wherein: The construction of a specific AI virtual human includes seven steps: character generation, voice generation, lip synchronization, action generation, synthesis display, test verification, and deployment and maintenance. The virtual human requirement information input by the user in the form of voice, video, text, picture or 3D model generates a specific AI virtual human through the AI virtual human system. The calling of the specific AI virtual human includes generating the text instruction content of the specific AI virtual human, which can generate the corresponding text instruction content through the cooking AI large model. Through API call, the text instruction content is passed to the specific AI virtual human, and the specific AI virtual human is used to generate a video.

5. The cooking method based on the cooking AI large model and the AI virtual human system according to claim 2 or 3, characterized in that: The construction of the cooking AI large model includes: The first step, collecting cooking data: collecting all information on cooking methods of ingredients presented in the form of voice, video, text, picture or 3D model; The second step, preprocessing cooking data: processing all the collected information on cooking methods of ingredients to ensure the integrity and availability of the information, including uniformly converting information in different formats into text and editing the text information in a certain format to facilitate the subsequent training of the AI large model; The third step, selecting an AI large model applicable to cooking: selecting third-party AI large models at home and abroad, measured by indicators such as accuracy, response speed, and diversity; The fourth step, training the cooking AI large model: organizing the cooking data set through the second step, and then fine-tuning and training the cooking data set with a third-party AI large model. After the training is completed, a cooking AI large model with all relevant data on cooking methods is generated; The fifth step, validating and testing the cooking AI large model: detecting and evaluating the effect of the cooking AI large model generated in the fourth step for specific tasks. If the evaluation effect fails, continue to repeat the steps of the first step, the second step, the third step, and the fourth step, and retrain until the effect evaluation passes. Generate the cooking AI large model and store it in the storage module of the cloud platform or the intelligent cooking device; The sixth step, deploying and maintaining the cooking AI large model: deploying the newly generated AI large model into the storage module of the cloud platform or the intelligent cooking device, and performing continuous maintenance and update, regularly updating the data to ensure the timeliness and accuracy of the data.

6. The cooking method based on the cooking AI large model and the AI virtual human system according to claim 5, characterized in that: In the third step, select an AI large model applicable to cooking. The selected AI large model is Baichuan2-13B, and the parameters of this AI large model are as follows: Hidden layer dimension: 5,120, Number of layers: 40, Number of attention heads: 40, Vocabulary size: 64,000, Total number of parameters: 13,264,901,120, Training data (tokens): 1.4 trillion, Position encoding: ALiBi, Maximum length: 4,096.

7. The cooking method based on the cooking AI large model and the AI virtual human system according to claim 6, characterized in that: The training process of the cooking AI large model in the fourth step is as follows: First, download the model weights of baichuan13b from huggingface. Then, download the belle dataset train_0.5M_CN to the local and place it in the dataset folder under the project directory. Finally, run the sft_lora.py script. Next, quantize the Baichuan LLM using the nf4 and double quantization methods of qlora. Finally, perform instruction fine-tuning using lora.

8. The cooking method based on the cooking AI large model and the AI virtual human system according to claim 4, wherein: The construction of a specific AI virtual human for the AI virtual human system applied to the cloud platform includes: The first step, character generation: Generate a model of a virtual digital human through multiple avatar photos uploaded by the user. The character generation sub-module is responsible for generating the image of a specific virtual human based on the photos input by the user. The specific virtual human generation technology uses AIGC technology; The second step, voice generation: Generate a simulated voice model through multiple voice segments uploaded by the user. The voice generation sub-module is responsible for generating the voice output of the virtual digital human based on the input text or voice, including two technologies: text-to-speech and voice cloning; The third step, lip generation: Generate a corresponding lip model based on the generated voice. The lip generation module is responsible for synchronizing the generated voice with the lips of the specific virtual human to make the chat of the specific virtual human appear more real and natural; The fourth step, action generation: Generate a simulated action model through multiple action videos uploaded by the user. The action generation sub-module is responsible for generating the action output of the virtual digital human based on the input video, including two sub-steps: driving and rendering; The fifth step, synthesis and display: The synthesis and display module is responsible for synthesizing The voice and action outputs of the virtual digital human into a video and displaying it on different devices, including 2D and 3D display devices; The sixth step, verification and testing: Perform effect detection and evaluation of specific tasks on the specific AI virtual human video generated in the fifth step. If the evaluation effect fails, continue to repeat the steps of the first, second, third, fourth, and fifth steps and retrain until the effect evaluation passes; The seventh step, deployment and maintenance: Deploy the newly generated specific AI virtual human to the operating system of the cloud platform and perform continuous maintenance and updates.

9. The cooking method based on the cooking AI large model and the AI virtual human system according to claim 4, characterized in that: The construction of a specific AI virtual human for the built-in AI virtual human system includes: The first step, character generation: Generate a model of a virtual digital human through multiple avatar photos uploaded by the user. The character generation sub-module is responsible for generating the image of a specific virtual human based on the photos input by the user. The specific virtual human generation technology uses face-swapping technology; Step 2, Speech Generation: Generate a simulated speech model based on the input text. The speech generation sub-module is responsible for generating the voice output of the virtual digital human according to the input text; Step 3, Lip Generation: Generate a corresponding lip model based on the generated speech. The lip generation module is responsible for synchronizing the generated speech with the lips of a specific virtual human, so that the chat of the specific virtual human appears more real and natural; Step 4, Action Generation: Generate a simulated action model through the built-in action templates selected by the user. The action generation sub-module is responsible for generating simulated actions according to the built-in action templates selected by the user, including two sub-steps of transformation and rendering; Step 5, Synthesis and Display: The synthesis and display module is responsible for synthesizing the voice and action outputs of the virtual digital human into a video and displaying it on different devices, including 2D and 3D display devices; Step 6, Verification and Testing: Perform an effect detection and evaluation of specific tasks on the specific AI virtual human video generated in the fifth step. If the evaluation effect fails, repeat the steps of the first, second, third, fourth, and fifth steps and retrain until the effect evaluation passes; Step 7, Deployment and Maintenance: Deploy the newly generated specific AI virtual human to the operating system of the intelligent cooking appliance and perform continuous maintenance and updates.

10. The cooking method based on the cooking AI large model and the AI virtual human system according to claim 4, wherein: The invocation of the specific AI virtual human specifically includes the following steps: Step 1, Generate the text instruction content of the specific AI virtual human, which can be generated by the cooking AI large model Or the system program generates the corresponding text instruction content according to the business logic; Step 2, Through API call, transmit the text instruction content to the AI virtual human system, and the AI virtual human system performs video generation, which includes the following processes: 1) Text-to-Speech: Convert the text content into the voice of the specific AI virtual human through the speech generation module of the AI virtual human system, and the voice uploaded by the user can be cloned; 2) Lip Generation: Synchronize the lips of the specific AI virtual human through an algorithm for the voice content generated in the first step, and both Chinese and English can be synchronized; 3) Action Generation: Select the action template uploaded by the user, and output the corresponding action postures of the specific AI virtual human according to the user's action template, and combine the lip synchronization in the second step; 4) Video Synthesis: Output the content generated in the previous three steps into a standard video format and return it.

11. An intelligent cooking device, characterized in that, An information collection module, a storage module, a processor, a display screen, and a wireless communication module are provided. The information collection module is used to collect all information about the cooking methods of ingredients and virtual human requirements information input by the user in the form of voice, video, text, pictures, or 3D models. The processor is used to execute the new cooking method generated by the cooking AI large model in claim 1 or 2 or 3 and prompt the user or the specific AI virtual human for cooking operation instructions.

12. The intelligent cooking device according to claim 11, characterized in that: An internal AI virtual human system is also provided. The internal AI virtual human system is located in the storage module. The internal AI virtual human system is electrically connected to the processor, and the internal AI virtual human system is connected to the user's mobile terminal through the wireless communication module.

13. The intelligent cooking device according to claim 11, wherein: It further includes an AI virtual human system provided on the cloud platform, and the AI virtual human system is connected to the user mobile terminal through the cloud platform.

14. The intelligent cooking device according to claim 11, wherein: There is a work station of an intelligent cooking device equipped with stir-frying, quick-frying, stir-frying with sauce, deep-frying, cooking in sauce, frying, pan-frying, roasting, braising, stewing, steaming, quick-boiling, boiling, stewing, cooking with wine and scallions, mixing, pickling, roasting, marinating, freezing, making candied dishes, making honeyed dishes, smoking, rolling, sliding or baking.

15. The intelligent cooking device according to claim 14, wherein: The work station of the intelligent cooking device for stir-frying, quick-frying, stir-frying with sauce, deep-frying, cooking in sauce, frying, pan-frying, roasting, braising, stewing, steaming, quick-boiling, boiling, stewing, cooking with wine and scallions, mixing, pickling, roasting, marinating, freezing, making candied dishes, making honeyed dishes, smoking, rolling, sliding or baking is provided with a corresponding operation detection and feedback system, and the operation detection and feedback system is used to detect whether the cooking operation performed by the user meets the requirements of the cooking method of the cooking AI large model.

16. The intelligent cooking device according to claim 11, wherein: There is a human-computer interaction module, and the human-computer interaction module is used for information interaction between the intelligent cooking device and the user, including the user operating the user mobile terminal to confirm cooking parameters and start a cooking operation instruction.

Images