Interaction method, device and system with autonomous learning and generation capabilities

Abstract

The invention discloses an interaction method, device and system with autonomous learning and generation capabilities, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring multi-modal data; the multi-modal data at least comprises internal data associated with the refrigeration equipment and external data interacted with the refrigeration equipment; performing fine tuning on the pre-training language model based on the multi-modal data to obtain a pre-training language model after fine tuning; and training the pre-training language model after fine tuning by adopting a reinforcement learning algorithm based on human feedback to obtain a large-scale language model so as to facilitate interaction by using the large-scale language model. According to the embodiment of the invention, by combining the multi-modal data, autonomous learning can be realized, and the self decision process can be continuously optimized, so as to better meet user requirements and adapt to complex and variable use situations, improve the processing capability of the model on tasks such as refrigeration equipment food material management, enable the model to be more accurate in output, and improve the user experience. And the user interaction experience is obviously improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to an interactive method, device and system with autonomous learning and generation capabilities. Background Technology

[0002] With the rapid development of artificial intelligence technology, its applications in daily life are becoming increasingly widespread, especially in the fields of refrigerator food management, nutrition and health, and dietary information services. AI technology, through its functions such as intelligently identifying ingredients, analyzing nutritional components, and providing personalized dietary suggestions, greatly improves users' efficiency in managing their diet and their quality of life.

[0003] In related technologies, AI models and systems often employ pre-training, which involves updating and maintaining them via remote servers at fixed intervals. However, this approach may not be able to adapt in a timely manner to rapid changes in food information, such as seasonal changes in ingredients and evolving user preferences, resulting in inaccurate information output by the AI ​​models. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes an interactive method, apparatus, and system with autonomous learning and generation capabilities to improve the accuracy of the information output by the model and enhance the user's interactive experience.

[0005] Firstly, this application provides an interactive method with autonomous learning and generation capabilities, including:

[0006] Acquire multimodal data; the multimodal data includes at least internal data associated with the refrigeration equipment and external data interacting with the refrigeration equipment;

[0007] The pre-trained language model is fine-tuned based on the multimodal data to obtain the fine-tuned pre-trained language model.

[0008] A fine-tuned pre-trained language model is trained using a reinforcement learning algorithm based on human feedback to obtain a large language model, which is then used for interaction.

[0009] According to the interactive method with autonomous learning and generation capabilities of this application, multimodal data is acquired; the multimodal data includes at least internal data associated with the refrigeration equipment and external data interacting with the refrigeration equipment; a pre-trained language model is fine-tuned based on the multimodal data to obtain a fine-tuned pre-trained language model; a reinforcement learning algorithm based on human feedback is used to train the fine-tuned pre-trained language model to obtain a large language model, which is then used for interaction. This embodiment of the application, by combining multimodal data processing to fine-tune the pre-trained language model, enables the model to more comprehensively learn and understand the usage environment and user behavior of the refrigeration equipment. By employing a reinforcement learning algorithm combined with human feedback, the model can achieve autonomous learning, continuously optimizing its decision-making process to better meet user needs and adapt to complex and changing usage scenarios. This improves the model's ability to handle tasks such as food management with the refrigeration equipment, making the model's output more accurate and significantly enhancing the user interaction experience.

[0010] According to one embodiment of this application, the step of fine-tuning the pre-trained language model based on the multimodal data to obtain the fine-tuned pre-trained language model includes:

[0011] The multimodal data is converted into text data using a model adapted to the type of the multimodal data;

[0012] The prompting process designs prompts for each text element in the text data and generates embedding vectors based on the text data to obtain a training dataset.

[0013] The pre-trained language model is fine-tuned based on the training dataset to obtain the fine-tuned pre-trained language model.

[0014] In this embodiment, multimodal data is converted into text data by using a model adapted to different data types, thereby solving the problem of data heterogeneity. Then, through prompting engineering, precise prompts are designed for the text data, which can guide the model to pay more attention to the key information in the data, enhance the model's understanding and processing of text content, and the generated embedding vectors fuse the text data and convert it into a format that the model can understand, thus forming a high-quality training dataset.

[0015] According to one embodiment of this application, converting the multimodal data into text data using a model adapted to the type of the multimodal data includes:

[0016] When the type of multimodal data is speech data, the speech data is converted into text data by an end-to-end deep neural network model based on a Gaussian neural network model or an encoding / decoding structure.

[0017] In this embodiment, the Gaussian neural network model, with its powerful pattern recognition capabilities, can effectively extract features from speech signals and convert them into text information; while the end-to-end deep neural network model with an encoder-decoder architecture realizes the mapping from input speech to output text, a structure particularly suitable for processing sequence-to-sequence tasks. The application of these two models not only improves the accuracy of speech recognition but also accelerates the conversion speed, enabling the system to respond to user voice input in real time.

[0018] According to one embodiment of this application, converting the multimodal data into text data using a model adapted to the type of the multimodal data includes:

[0019] When the type of multimodal data is video data, the learning task is to convert the video data into text data. A temporally and spatially continuous student model is introduced through a preset teacher model so that the parameters of the student model are the same as the parameters of the teacher model.

[0020] The student model is trained using transfer learning and knowledge distillation techniques to learn the knowledge of the teacher model, resulting in a trained student model.

[0021] The multimodal data is converted into text data based on the trained student model.

[0022] This embodiment innovatively combines transfer learning, knowledge distillation, and a student-teacher model mechanism to achieve efficient conversion from video data to text data. First, a temporally and spatially continuous student model is introduced through a pre-defined teacher model, ensuring its parameters are identical to the pre-defined teacher model. This allows the student model to learn from the same starting point as the teacher model. Then, utilizing transfer learning and knowledge distillation techniques, the student model learns and absorbs the knowledge and experience of the teacher model. This process not only accelerates the student model's learning speed but also improves performance in processing video-to-text conversion tasks.

[0023] According to one embodiment of this application, the step of fine-tuning the pre-trained language model based on the multimodal data to obtain the fine-tuned pre-trained language model includes:

[0024] The encoder is optimized by performing compression pruning on the embedding layer and the activation layer with multi-head attention mechanism of the encoder, respectively.

[0025] The multimodal data is encoded based on the optimized encoder to obtain encoded data;

[0026] The pre-trained language model is fine-tuned based on the encoded data to obtain the fine-tuned pre-trained language model.

[0027] In this embodiment, by optimizing the encoder, particularly by compressing and pruning the embedding layer and the activation layer with multi-head attention, the complexity and computational burden of the model are effectively reduced. Compressing and pruning the embedding layer removes unimportant parameters and reduces the dimensionality of the model, while pruning the activation layer helps to simplify the network structure. Together, they improve the model's running efficiency and generalization ability.

[0028] According to one embodiment of this application, the optimization of the encoder further includes:

[0029] A feedforward network block is constructed using multiple feedforward network layers, residual network layers, and normalized network layers to facilitate encoding operations on the multimodal data based on the feedforward network block, thereby obtaining encoded data.

[0030] In this embodiment, the feedforward network block, constructed using multiple feedforward network layers, residual network layers, and normalization network layers, not only enhances the model's ability to perform nonlinear transformations of data, but also solves the gradient vanishing problem in deep network training through residual connections. This enables deep networks to learn complex data representations more effectively and reduces the resource consumption of normalization computation. Furthermore, it allows encoding operations to capture key information in the data, generating high-quality encoded data and providing rich and accurate input for pre-trained language models.

[0031] According to one embodiment of this application, the method further includes:

[0032] The optimized decoder includes: constructing a decoding module based on an attention mechanism, convolutional layers, activation layers, and normalization layers, so that the decoding module can decode the encoded data to obtain speech data and / or video data; compressing or quantizing the encoded data based on semantic compression / quantization, and decoding it using an autoregressive decoder to obtain text data.

[0033] In this embodiment, the decoder structure and decoding strategy are optimized to achieve multimodal decoding of encoded data, achieving high accuracy and efficiency regardless of whether it is converted into speech, video, or text data. First, a decoding module based on attention mechanisms, convolutional layers, activation layers, and normalization layers is used. The attention mechanism enhances the model's focus on key information, the combination of convolutional and activation layers improves the extraction of spatial features, and the normalization layer further stabilizes the decoding process. This enables accurate decoding of encoded data into speech and / or video data, meeting the needs of multimodal data conversion. Second, semantic compression / quantization techniques are used to compress or quantize the encoded data, effectively reducing data storage and transmission requirements while maintaining the core semantic information of the data. Based on this, an autoregressive decoder is used for decoding, allowing the model to gradually generate text data, improving the coherence and readability of the generated text.

[0034] In some embodiments, the method further includes:

[0035] When the multimodal data includes image data, the image data is segmented into multiple patches, and the multiple patches are unified into a two-dimensional vector space;

[0036] The two-dimensional vector space is mapped to a one-dimensional vector space of the same length, and a position embedding vector is generated to provide the position information of each patch in the image data;

[0037] The feature vector of the one-dimensional vector space is extracted by combining the position embedding vector;

[0038] A multilayer perceptron is used to identify the food categories in the image data based on the feature vectors.

[0039] In this embodiment, the image data is divided into multiple two-dimensional patches to reduce the dimensionality of the data. These two-dimensional vectors are mapped to a one-dimensional vector space and a position embedding vector is generated. This allows the model to capture the position information of each patch in the original image, providing important spatial context for understanding the image content. It integrates local visual information and positional information to form a comprehensive understanding of the image content. Then, a multilayer perceptron is used to analyze the feature vectors, enabling the model to accurately identify the food categories in the image.

[0040] Secondly, this application provides an interactive device with autonomous learning and generation capabilities, comprising:

[0041] An acquisition module is used to acquire multimodal data; the multimodal data includes at least internal data associated with the refrigeration equipment and external data interacting with the refrigeration equipment;

[0042] The fine-tuning module is used to fine-tune the pre-trained language model based on the multimodal data to obtain the fine-tuned pre-trained language model.

[0043] The interaction module is used to train the fine-tuned pre-trained language model using a reinforcement learning algorithm based on human feedback to obtain a large language model, so as to facilitate interaction using the large language model.

[0044] According to the interactive device with autonomous learning and generation capabilities of this application, multimodal data is acquired; the multimodal data includes at least internal data associated with the refrigeration equipment and external data interacting with the refrigeration equipment; a pre-trained language model is fine-tuned based on the multimodal data to obtain a fine-tuned pre-trained language model; a reinforcement learning algorithm based on human feedback is used to train the fine-tuned pre-trained language model to obtain a large language model, which is then used for interaction. This embodiment of the application, by combining multimodal data processing to fine-tune the pre-trained language model, enables the model to more comprehensively learn and understand the usage environment and user behavior of the refrigeration equipment. By employing a reinforcement learning algorithm combined with human feedback, the model can achieve autonomous learning, continuously optimizing its decision-making process to better meet user needs and adapt to complex and changing usage scenarios. This improves the model's ability to handle tasks such as food management with the refrigeration equipment, making the model's output more accurate and significantly enhancing the user interaction experience.

[0045] Thirdly, this application provides an interactive system with autonomous learning and generation capabilities, including:

[0046] Refrigeration equipment;

[0047] The server includes an interactive device with autonomous learning and generation capabilities as described in the second aspect.

[0048] Fourthly, this application provides a cooling device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the interactive method with autonomous learning and generation capabilities as described in the first aspect above.

[0049] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the interactive method with autonomous learning and generation capabilities as described in the first aspect above.

[0050] In a sixth aspect, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the interactive method with autonomous learning and generation capabilities as described in the first aspect above.

[0051] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the interactive method with autonomous learning and generation capabilities as described in the first aspect above.

[0052] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0053] According to the interactive method with autonomous learning and generation capabilities of this application, multimodal data is acquired; the multimodal data includes at least internal data associated with the refrigeration equipment and external data interacting with the refrigeration equipment; a pre-trained language model is fine-tuned based on the multimodal data to obtain a fine-tuned pre-trained language model; a reinforcement learning algorithm based on human feedback is used to train the fine-tuned pre-trained language model to obtain a large language model, which is then used for interaction. This embodiment of the application, by combining multimodal data processing to fine-tune the pre-trained language model, enables the model to more comprehensively learn and understand the usage environment and user behavior of the refrigeration equipment. By employing a reinforcement learning algorithm combined with human feedback, the model can achieve autonomous learning, continuously optimizing its decision-making process to better meet user needs and adapt to complex and changing usage scenarios. This improves the model's ability to handle tasks such as food management with the refrigeration equipment, making the model's output more accurate and significantly enhancing the user interaction experience.

[0054] Furthermore, in some embodiments, multimodal data is converted into text data by using models adapted to different data types, thereby solving the problem of data heterogeneity. Then, by using cue engineering to design precise cue information for the text data, the model can be guided to pay more attention to the key information in the data, enhancing the model's understanding and processing of the text content. The generated embedding vectors then fuse the text data and convert it into a format that the model can understand, thus forming a high-quality training dataset.

[0055] Furthermore, in some embodiments, the Gaussian neural network model, with its powerful pattern recognition capabilities, can effectively extract features from speech signals and convert them into text information; while the end-to-end deep neural network model with an encoder-decoder architecture achieves the mapping from input speech to output text, a structure particularly suitable for processing sequence-to-sequence tasks. The application of these two models not only improves the accuracy of speech recognition but also accelerates the conversion speed, enabling the system to respond to user voice input in real time.

[0056] Furthermore, in some embodiments, transfer learning, knowledge distillation, and a student-teacher model mechanism are innovatively combined to achieve efficient conversion from video data to text data. First, a temporally and spatially continuous student model is introduced through a pre-defined teacher model, ensuring its parameters are identical to the pre-defined teacher model. This allows the student model to learn from the same starting point as the teacher model. Then, utilizing transfer learning and knowledge distillation techniques, the student model learns and absorbs the knowledge and experience of the teacher model. This process not only accelerates the student model's learning speed but also improves performance in processing video-to-text conversion tasks.

[0057] Furthermore, in some embodiments, by optimizing the encoder, particularly by compressing and pruning the embedding layer and the activation layer with multi-head attention, the complexity and computational burden of the model are effectively reduced. Compressing and pruning the embedding layer can remove unimportant parameters and reduce the dimensionality of the model, while pruning the activation layer helps to simplify the network structure. Together, they improve the model's running efficiency and generalization ability.

[0058] Furthermore, in some embodiments, a feedforward network block constructed using multiple feedforward network layers, residual network layers, and normalization network layers not only enhances the model's ability to perform nonlinear transformations of data, but also solves the gradient vanishing problem in deep network training through residual connections. This enables deep networks to learn complex data representations more effectively and reduces the resource consumption of normalization computation. It also allows encoding operations to capture key information in the data, generate high-quality encoded data, and provide rich and accurate input for pre-trained language models.

[0059] Furthermore, in some embodiments, the decoder structure and decoding strategy are optimized to achieve multimodal decoding of encoded data, achieving high accuracy and efficiency whether converted to speech, video, or text data. First, a decoding module built based on attention mechanisms, convolutional layers, activation layers, and normalization layers is used. The attention mechanism enhances the model's focus on key information, the combination of convolutional and activation layers improves the ability to extract spatial features from the data, and the normalization layer further stabilizes the decoding process. This enables accurate decoding of encoded data into speech and / or video data, meeting the needs of multimodal data conversion. Second, semantic compression / quantization techniques are used to compress or quantize the encoded data, effectively reducing data storage and transmission requirements while maintaining the core semantic information of the data. Based on this, an autoregressive decoder is used for decoding, allowing the model to gradually generate text data, improving the coherence and readability of the generated text.

[0060] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0061] The above and / or additional aspects and advantages of this application will become apparent and readily understood in conjunction with the following description of the embodiments in conjunction with the accompanying drawings, wherein:

[0062] Figure 1 This is a schematic diagram illustrating a scenario example provided in an embodiment of this application;

[0063] Figure 2 This is an example diagram of an interactive method with autonomous learning and generation capabilities according to an embodiment of this application;

[0064] Figure 3 This is a flowchart illustrating an interactive method with autonomous learning and generation capabilities provided in an embodiment of this application.

[0065] Figure 4 This is a schematic diagram of the structure of an interactive device with autonomous learning and generation capabilities provided in an embodiment of this application;

[0066] Figure 5 This is an example architecture diagram of an interactive system with autonomous learning and generation capabilities provided in the embodiments of this application.

[0067] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0069] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0070] In this embodiment, the refrigeration equipment can be any device with a refrigeration function. For example, the refrigeration equipment can be a household refrigerator, a refrigerated display case in a supermarket or catering industry, a freezer / icebox, a refrigerated truck, a wine cabinet, a medical refrigeration device, an industrial refrigeration system, etc. Of course, the refrigeration equipment can also be other devices with a refrigeration function, and this embodiment does not limit this.

[0071] The interactive methods, apparatus and systems with autonomous learning and generation capabilities provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0072] First, the business process of the interactive method, device, and system with autonomous learning and generation capabilities provided in the embodiments of this application is introduced through a scenario, such as... Figure 1 As shown.

[0073] In this scenario example, a refrigerator is used as an example of a refrigeration device. This refrigerator can be a smart refrigerator equipped with various sensors, such as those for home use, industrial applications, vehicle use, built-in installations, or hotel use. Figure 2 As shown. The refrigerator may have a communication module, enabling it to communicate with servers, clients, etc., via wireless networks or Bluetooth; the refrigerator may also have a microphone array, capable of collecting user voice and issuing voice prompts to users; the refrigerator may also be equipped with a screen, camera, and other devices, enabling it to interact with users through voice, video, and text by setting up various components.

[0074] In this scenario example, various sensors within the refrigerator and its interactions with the outside world can collect data on the refrigerator itself, as well as data related to diet, ingredients, health, and nutrition. This data can be in the form of voice, video, or text. After collection, this data can be sent to the autonomous learning and production module. This module can be deployed on a server, where the server uses this data to perform autonomous learning encoding and decoding to train the model, enabling it to learn and produce independently. The server receives data associated with the refrigerator and interacts with the user. For example, it can relearn from accumulated data such as the user's food preferences and interests, and then recommend, search for, and retrieve valuable food information for the user based on its memory capabilities. Based on information about the refrigerator's external environment, the model can autonomously learn and issue warnings and push valuable information to the user, allowing them to better understand and manage the refrigerator's operating status. Based on uploaded short video menu content, the model can generate and infer production plans for the uploaded video based on existing information such as the refrigerator's data and ingredients. Based on historical and real-time collected data, the model can infer and autonomously generate diet and nutrition reports, food expiration reminders, periodically updated food purchase lists, optimal food menu combinations, and food storage information.

[0075] In this embodiment, the server can send model-generated information to the corresponding refrigerator, so that the refrigerator can display this information on its screen. Alternatively, the server can push this information to an app associated with the refrigerator or send it to the user via SMS.

[0076] The following describes an interactive method with autonomous learning and generation capabilities provided in the embodiments of this application. The interactive method with autonomous learning and generation capabilities can be applied to a terminal, and can be executed by hardware or software in the terminal.

[0077] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0078] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0079] The interactive method with autonomous learning and generation capabilities provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the interactive method with autonomous learning and generation capabilities. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The following uses an electronic device as the execution subject to illustrate the interactive method with autonomous learning and generation capabilities provided in this application embodiment.

[0080] like Figure 3 As shown, the interactive method with autonomous learning and generation capabilities includes steps 310, 320 and 330.

[0081] Step 310: Obtain multimodal data; the multimodal data includes at least internal data associated with the refrigeration equipment and external data interacting with the refrigeration equipment.

[0082] In this embodiment, multimodal data is a collection of information from various data types, which may originate from different data sources and contain different data types. For example, the data types may include text data, image data, video data, audio data, etc. The data source may be various sensors in the cooling device, such as image sensors, microphone arrays, etc., or data from external servers, clients, etc.

[0083] like Figure 2 As shown, taking a refrigerator as an example, the refrigerator 200 can have several layers, including a smart refrigerator with constant temperature for freezing and refrigeration. It features human-computer interaction interfaces such as gestures, voice, text, audio and video, image acquisition, microphone arrays, radar signal acquisition (e.g., images), tools such as OCR devices, documents, VR, and AR. Presentation formats include APP / PC / web / mini-programs, and large screens. Supported protocols include TCP / IP, Wi-Fi, and Bluetooth. Supported networks include 5G, Wi-Fi, wired network, and 6G. Applications include: home, hotel, industrial, indoor and outdoor, military, and aerospace. Embedding methods include: flush embedding, automotive embedding, and integration. It can acquire data collected by these components of the refrigerator 200, such as images of food inside the refrigerator, interactions with the user via voice and gestures, and communication data with a remote APP or server. Figure 2As shown in section 101, internal data associated with the refrigerator 200 and external data from interactions with the refrigeration equipment can be acquired. For example, data can be acquired through human-computer interaction, multi-turn dialogue, intelligent agents, multi-agent systems, personal intelligent assistants, data transmission, data generation tools, and data uploads. This data can be acquired through various channels such as 5G / 6G, WiFi, apps, microphones, mobile phones, and Bluetooth. As shown in section 102, this data can include text data, image data, video data, and audio data. Internal data can include refrigerator-specific data, the storage location of food and items, etc., while external data can include health and nutrition data, and data from user interactions such as dietary preferences, user needs, and user-uploaded videos.

[0084] In some embodiments, after acquiring the multimodal data, preprocessing can be performed on the multimodal data. For example, cleaning and format conversion operations can be performed on the multimodal data. Figure 2 As shown in 103, the acquired multimodal data can be cleaned and format-converted, such as parsing and storing information data formats. Audio and video data can be separated into voice and video using tools or scripts to obtain valid voice and video data. Text data preprocessing can include stop word removal, deduplication, and other related tasks. Image data can undergo image enhancement, cropping, and other preprocessing. Through these preprocessing steps for multimodal data, valid data of the corresponding data types can be obtained.

[0085] In some embodiments, the acquired multimodal data can be current data or historical data. For example, it can include data within a preset historical time period, such as data from 5 days ago, 3 days ago, 8 hours ago, 1 hour ago, etc. Figure 2 As shown in 105, the historical data obtained further ensures the comprehensiveness and richness of the generated high-quality dataset.

[0086] Step 320: Fine-tune the pre-trained language model based on multimodal data to obtain the fine-tuned pre-trained language model.

[0087] A pre-trained model is a model that is first trained on a batch of text data, and then further trained or used on this pre-trained model. A pre-trained language model (PLM) is a type of model in natural language processing (NLP) that is pre-trained on a large amount of text data to learn a general representation of language.

[0088] In the embodiments of this application, the pre-trained language model can be a transformer-based model, such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), etc., or a large language model (LLM) with semantic compression such as decoder-ONLY, encoder-ONLY, and encoder-decoder. This includes, but is not limited to, LLM models with graph neural networks, attention mechanisms, transformer models and their variants or improvements, distillation network LLM, latent / diffusion model LLM, U-net network and other deep network model LLMs, as well as fused deep network model LLMs such as transformer-based BiLSTM+MMCNN-RNN, BiLSTM+MMCNN+Attention, MMCNN+BiLSTM+Attention fusion model LLM, GRU+CNN, GRU+CNN+Attention model, etc.

[0089] In this embodiment, fine-tuning involves using a pre-trained model to custom-train certain tasks, making the pre-trained language model "better understand" the task. For example, multimodal data can be used as training samples to train the pre-trained language model, and the trained pre-trained language model can be used to further improve its abstract understanding of specific multimodal data, thereby enhancing the user's interaction experience with the cooling equipment.

[0090] Step 330: Train the fine-tuned pre-trained language model using a reinforcement learning algorithm based on human feedback to obtain a large language model, so as to facilitate interaction using the large language model.

[0091] Reinforcement Learning with Human Feedback (RLHF) is a reinforcement learning method that combines traditional reinforcement learning techniques with human feedback to improve the learning effectiveness and performance of intelligent systems. In RLHF, intelligent systems obtain feedback through interaction with humans and adjust their behavior accordingly. Human feedback can take various forms, including positive rewards, negative punishments, and guiding suggestions. By analyzing this feedback, intelligent systems can gradually improve their decision-making abilities and behavioral performance. Through interactive learning with humans, intelligent systems can adapt more quickly to changing environments and demands, and better meet human needs and expectations.

[0092] In this embodiment, after fine-tuning the pre-trained language model, it can be deployed to a practical application, enabling the model to interact with the user. During this interaction, the model generates corresponding output information based on the user's queries or requests, and collects user feedback on the model's output. This feedback can be explicit, such as ratings, comments, likes / dislikes, or implicit, such as the user's continuous interaction time, click-through rate, and repeat visit rate.

[0093] Before training the fine-tuned pre-trained language model using a reinforcement learning algorithm based on human feedback, it's necessary to select a reinforcement learning algorithm. Specifically, a reward function can be defined based on collected user feedback. This reward function evaluates the quality of the model's output and provides guidance for reinforcement learning. A suitable reinforcement learning algorithm is selected based on its efficiency, applicability, and task suitability; examples include PPO (Proximal Policy Optimization), DPO (Distributive Proximal Policy Optimization), and DQN (Deep Q-Network). The reinforcement learning algorithm is then used to optimize the model's policy, enabling it to learn how to generate more user-friendly outputs under the guidance of the given reward function.

[0094] During training, reinforcement learning algorithms are used to train the model, enabling it to learn how to select the optimal action under different states. After each interaction with the user, the model is iteratively updated based on the feedback from the reward function. Through continuous interaction, the model continuously learns and optimizes its strategy to adapt to changes in user behavior and preferences. Of course, during training, the hyperparameters of the reinforcement learning algorithm, such as the learning rate, discount factor, and exploration rate, can be adjusted based on the model's performance. After training is complete, the reinforcement learning-trained and optimized model can be redeployed to the application to provide a higher quality user interaction experience.

[0095] According to the interactive method with autonomous learning and generation capabilities of this application, multimodal data is acquired; the multimodal data includes at least internal data associated with the refrigeration equipment and external data interacting with the refrigeration equipment; a pre-trained language model is fine-tuned based on the multimodal data to obtain a fine-tuned pre-trained language model; a reinforcement learning algorithm based on human feedback is used to train the fine-tuned pre-trained language model to obtain a large-scale language model, which is then used for interaction. This embodiment of the application, by combining multimodal data processing to fine-tune the pre-trained language model, enables the model to more comprehensively learn and understand the usage environment and user behavior of the refrigeration equipment. By employing a reinforcement learning algorithm combined with human feedback, the model can achieve autonomous learning, continuously optimizing its decision-making process to better meet user needs and adapt to complex and changing usage scenarios. This improves the model's ability to handle tasks such as food management with the refrigeration equipment, making the model's output more accurate and significantly enhancing the user interaction experience.

[0096] In some embodiments, the pre-trained language model is fine-tuned based on multimodal data to obtain a fine-tuned pre-trained language model, including:

[0097] Multimodal data is converted into text data using a model adapted to the type of multimodal data;

[0098] The prompting process designs prompts for each piece of text information in the text data and generates embedding vectors based on the text data to obtain the training dataset.

[0099] The pre-trained language model is fine-tuned based on the training dataset to obtain the fine-tuned pre-trained language model.

[0100] In this embodiment, since multimodal data contains different types of data, before fine-tuning the pre-trained language model, the multimodal data can be converted into data of the same data type and fused to obtain a high-quality training dataset, thereby improving the efficiency of model fine-tuning. Figure 2 As shown in 104, different models can be used to convert multimodal data into text data for different types of data.

[0101] For example, for image data, image recognition techniques, such as Convolutional Neural Networks (CNNs), can be used to identify objects and scenes in the image and generate descriptive text; for video data, CNNs or Recurrent Neural Networks (RNNs) can be used to process the video frame sequence and convert it into descriptive text; for audio data, speech recognition techniques, such as deep learning models, can be used to convert speech signals into text.

[0102] In this embodiment, prompt engineering is the process of designing and selecting text prompts that can effectively guide a language model to produce the expected output. These prompts are text snippets input into the model to guide its generation or classification tasks. Prompt messages are the actual text provided to the language model, serving as a starting point for the generation task or guidance for the classification task. Prompt messages can be questions, instructions, examples, or any form of text, with the aim of stimulating the model to respond in a specific way.

[0103] In this embodiment, prompting engineering can be used to set appropriate prompts for the text data based on the task type. These prompts are then combined with the text data, and embedding techniques such as Word2Vec, GloVe, or BERT are used to convert the text data and prompts into numerical vectors. This achieves the fusion of multimodal data, enabling these vectors to capture the semantic information of the text, thus forming a training dataset. The pre-trained language model is then fine-tuned based on the training dataset to obtain the fine-tuned pre-trained language model.

[0104] In this embodiment, multimodal data is converted into text data by using a model adapted to different data types, thereby solving the problem of data heterogeneity. Then, through prompting engineering, precise prompts are designed for the text data, which can guide the model to pay more attention to the key information in the data, enhance the model's understanding and processing of text content, and the generated embedding vectors fuse the text data and convert it into a format that the model can understand, thus forming a high-quality training dataset.

[0105] In some embodiments, multimodal data is converted into text data using a model adapted to the type of multimodal data, including:

[0106] When the type of multimodal data is speech data, an end-to-end deep neural network model based on a Gaussian neural network model or an encoding / decoding structure can convert the speech data into text data.

[0107] In this embodiment, language data can be converted into text data by constructing a language recognition model. For example, a speech recognition model can be constructed using methods such as Gaussian Mixture Model (GMM) and Hidden Markov Model (HMM); alternatively, a deep learning-based model, such as an end-to-end deep neural network model with encoding and decoding structures like Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), or Transformer, can be selected to construct the speech recognition model, or the Kaldi open-source speech recognition architecture can be chosen.

[0108] In this embodiment, considering the spatiotemporal characteristics and contextual information relationship features of the voice data of the refrigeration equipment, a transformer block speech recognition model architecture based on convolution enhancement can be adopted, which is beneficial for obtaining the contextual information of the voice data; considering that the local feature information of the voice data is very important, convolution enhancement can be used to obtain efficient local speech semantic feature information.

[0109] Taking the construction of a speech recognition model based on an end-to-end deep neural network with an encoder-decoder structure as an example, the model's encoder and decoder parts are designed first. The encoder extracts speech features, and the decoder generates text sequences. Then, end-to-end training is performed, using a large amount of labeled speech and corresponding text data to train the model to directly map from speech to text. By inputting the language data into the trained model, the model's output text data can be obtained.

[0110] In this embodiment, the Gaussian neural network model, with its powerful pattern recognition capabilities, can effectively extract features from speech signals and convert them into text information; while the end-to-end deep neural network model with an encoder-decoder architecture realizes the mapping from input speech to output text, a structure particularly suitable for processing sequence-to-sequence tasks. The application of these two models not only improves the accuracy of speech recognition but also accelerates the conversion speed, enabling the system to respond to user voice input in real time.

[0111] In some embodiments, multimodal data is converted into text data using a model adapted to the type of multimodal data, including:

[0112] When the type of multimodal data is video data, the learning task is to convert the video data into text data. A temporally and spatially continuous student model is introduced through a pre-set teacher model so that the parameters of the student model are the same as those of the teacher model.

[0113] The student model is trained using transfer learning and knowledge distillation techniques to learn the knowledge of the teacher model, resulting in a trained student model.

[0114] Multimodal data is converted into text data based on the trained student model.

[0115] In this embodiment, audio and images in video data can be separated. Considering that the sentences for image feature recognition of human speech are complex, including factors such as varying sentence lengths, different word structures, and contextual relationships between sentences, an image recognition method based on a transfer and distillation-diffusion deep fusion network model with spatiotemporal and long-distance dependent features can be adopted. The key steps of the distillation-diffusion deep fusion model are as follows: the output results of the teacher model in the transfer and distillation-diffusion deep fusion model are improved and fused to mine rich semantic feature information of the sentence sequence. Here, a deep neural network fusion model structure based on a diffusion network model can also be adopted.

[0116] Specifically, a learning task can be defined: how to extract useful information from image data separated from video data and convert it into text data. Based on this learning task, a temporally and spatially continuous student model is introduced through a pre-defined teacher model, ensuring that the student model's parameters are identical to those of the teacher model. The teacher model can be a pre-trained or selected model; for example, it could be a model pre-trained on a large dataset, such as data from the internet, books, or articles. A student model can be created, maintaining temporal and spatial continuity in its architecture to handle the time-series characteristics of the video data. The student model's parameters are initialized to be identical to those of the teacher model, allowing the student model to learn from a similar starting point.

[0117] After obtaining the student model, transfer learning techniques can be used to transfer the knowledge learned by the teacher model in the video-to-text conversion task to the student model. Through knowledge distillation, the student model can learn to imitate the output of the teacher model, using the teacher model's output as the training target. Specifically, during training, the student model's training process can be converted into a series of discrete time steps, each time step representing a training cycle or iteration. Based on these discrete time steps, N iterations are performed until the teacher model is distilled to half the number of steps required for the student model (N / 2). At this point, the student model has learned half of the teacher model's knowledge, or its performance has reached half the level of the teacher model. Then, based on the trained student model, multimodal data is converted into text data.

[0118] This embodiment innovatively combines transfer learning, knowledge distillation, and a student-teacher model mechanism to achieve efficient conversion from video data to text data. First, a temporally and spatially continuous student model is introduced through a pre-defined teacher model, ensuring its parameters are identical to the pre-defined teacher model. This allows the student model to learn from the same starting point as the teacher model. Then, utilizing transfer learning and knowledge distillation techniques, the student model learns and absorbs the knowledge and experience of the teacher model. This process not only accelerates the student model's learning speed but also improves performance in processing video-to-text conversion tasks.

[0119] In some embodiments, the pre-trained language model is fine-tuned based on multimodal data to obtain a fine-tuned pre-trained language model, including:

[0120] The encoder is optimized, including compressing and pruning the embedding layer and the activation layer with multi-head attention mechanism, respectively.

[0121] The multimodal data is encoded based on the optimized encoder to obtain encoded data;

[0122] The pre-trained language model is fine-tuned based on the encoded data to obtain the fine-tuned pre-trained language model.

[0123] In this embodiment, the pre-trained language model can be an encoder-decoder architecture. During the training process, the encoder and decoder are trained end-to-end as a whole. Gradient descent or its variants (such as Adam, RMSprop) can be used during training to update the parameters of the encoder and decoder to reduce the value of the loss function.

[0124] The encoder is responsible for reading the input data and converting it into a fixed-size internal representation, typically a vector or a set of vectors that capture key information and context from the input data. During training, the encoder's parameters are optimized using a large amount of input data to accurately capture and represent the features of the input data. The decoder is responsible for converting the internal representation generated by the encoder back into the format of the output data, such as text, image, audio, or video. The decoder typically generates the output step-by-step, taking previous outputs into account at each step. Similar to the encoder, the decoder's parameters are also optimized through training to accurately reconstruct or generate output data from the internal representation.

[0125] In this embodiment, the encoder can be optimized to improve the model's runtime efficiency and generalization ability. For example... Figure 2As shown in Figure 106, compression pruning can be applied to the embedding layer and the activation layer with multi-head attention mechanism in the encoder. The embedding layer typically transforms input data (such as words, characters, or other tokens) into dense vectors that capture the semantic and syntactic features of the input data. Compression pruning reduces the number of parameters in the embedding layer, thereby reducing model complexity and computational cost while preserving important feature representations as much as possible. Multi-head self-attention (MHSA) is a mechanism that allows the model to focus on multiple positions in the input sequence simultaneously, helping to capture complex dependencies in sequence data. The activation layer introduces non-linearity, enabling the model to learn and simulate more complex function mappings. By analyzing the output of multi-head attention, attention heads or connections that contribute less to the final task can be identified and pruned, thereby reducing model size and computational burden while preserving key model functions and performance as much as possible.

[0126] Specifically, the compression pruning process for the embedding layer can include: dividing the embedding layer into several sub-blocks to achieve finer-grained coding units, which is beneficial for extracting more detailed semantic features later; and performing RMSNorm (Root Mean Square Normalization) normalization calculation on the input and output data of the embedding layer. RMSNorm is a normalization method that adjusts the weights of each neuron by calculating the mean and root mean square of its output. A threshold for RMSNorm is pre-set to determine which neurons are important. For neurons with RMSNorm values ​​less than the threshold, pruning is performed, i.e., removing the weights corresponding to these neurons from the model and deleting rows or columns of the transpose matrix. In other words, during the pruning of the embedding layer, the transpose matrix retains useful parameters while removing network parameters that have little impact on model pre-training and fine-tuning. Since there is no residual network in the embedding layer, the number of network layers is reduced, improving computational efficiency.

[0127] The process of compressing and pruning activation layers with multi-head attention mechanisms can include: using multi-head attention mechanisms to process information in parallel, allowing simultaneous attention to multiple positions in the sequence; adding residual connections to the multi-head attention mechanism to aid in training deep networks; using skip connections to directly add inputs to the outputs of subsequent layers to alleviate the gradient vanishing problem; applying RMSnorm for normalization after the multi-head attention mechanism (i.e., post-Norm); and analyzing the activation layer output, for example, identifying unimportant neurons through RMSnorm values ​​and removing these unimportant neurons, such as GeLU or ReLU layers. This achieves compression and pruning of activation layers with multi-head attention mechanisms without affecting model performance and pre-training quality.

[0128] In this embodiment, by optimizing the encoder, particularly by compressing and pruning the embedding layer and the activation layer with multi-head attention, the complexity and computational burden of the model are effectively reduced. Compressing and pruning the embedding layer removes unimportant parameters and reduces the dimensionality of the model, while pruning the activation layer helps to simplify the network structure. Together, they improve the model's running efficiency and generalization ability.

[0129] In some embodiments, optimizing the encoder further includes:

[0130] Multiple feedforward network layers, residual network layers, and normalized network layers are used to construct feedforward network blocks, so as to perform encoding operations on multimodal data based on the feedforward network blocks to obtain encoded data.

[0131] In this embodiment, such as Figure 2 As shown in 106, a feed-forward network block (FFN) can also be constructed. This can be achieved by using N FFN network layers, residual network layers, and RMSnorm to form an FFN Block. By using the FFN Block, the normalization operation of the residual network layer and RMSnorm can be completed on the basis of the feed-forward FFN layer operation, and finally the encoded data is output.

[0132] In this embodiment, the feedforward network block, constructed using multiple feedforward network layers, residual network layers, and normalization network layers, not only enhances the model's ability to perform nonlinear transformations of data, but also solves the gradient vanishing problem in deep network training through residual connections. This enables deep networks to learn complex data representations more effectively and reduces the resource consumption of normalization computation. Furthermore, it allows encoding operations to capture key information in the data, generating high-quality encoded data and providing rich and accurate input for pre-trained language models.

[0133] In some embodiments, the method further includes:

[0134] The decoder is optimized by: constructing a decoding module based on attention mechanisms, convolutional layers, activation layers, and normalization layers, so that the decoding module can decode the encoded data to obtain speech data and / or video data; compressing or quantizing the encoded data based on semantic compression / quantization, and decoding it using an autoregressive decoder to obtain text data.

[0135] In this embodiment, by optimizing the decoder, it can be enabled to decode data into text data, video data, audio data, etc., as needed. Figure 2 As shown in 107, information and content can be automatically acquired and generated through knowledge distillation, such as teacher-student models and quantification techniques: text data, video data, audio data, etc.

[0136] For speech and video output data, a decoding module with a full attention mechanism is constructed based on attention mechanisms, convolutional layers, activation layers, and normalization layers. In the decoding module, a convolutional layer (conV) is used after the attention mechanism to extract local features in the feature space. After the convolutional layer, activation functions such as ReLU (Rectified Linear Unit) and GeLU (Gaussian Error Linear Unit) are applied to introduce non-linearity and increase the expressive power of the decoding. Then, a normalization layer, such as LayerNorm or BatchNorm, processes the output after convolution and activation to stabilize the training process and improve the model's generalization ability.

[0137] For text output data, the encoded text features can be semantically compressed / quantized by a network model at the word segmentation level, and then an autoregressive variable decoder can be used to obtain text format data output.

[0138] In this embodiment, the decoder structure and decoding strategy are optimized to achieve multimodal decoding of encoded data, achieving high accuracy and efficiency regardless of whether it is converted into speech, video, or text data. First, a decoding module based on attention mechanisms, convolutional layers, activation layers, and normalization layers is used. The attention mechanism enhances the model's focus on key information, the combination of convolutional and activation layers improves the extraction of spatial features, and the normalization layer further stabilizes the decoding process. This enables accurate decoding of encoded data into speech and / or video data, meeting the needs of multimodal data conversion. Second, semantic compression / quantization techniques are used to compress or quantize the encoded data, effectively reducing data storage and transmission requirements while maintaining the core semantic information of the data. Based on this, an autoregressive decoder is used for decoding, allowing the model to gradually generate text data, improving the coherence and readability of the generated text.

[0139] In some embodiments, the method further includes:

[0140] When multimodal data includes image data, the image data is segmented into multiple patches, and these patches are then unified into a two-dimensional vector space.

[0141] The two-dimensional vector space is mapped to a one-dimensional vector space of the same length, and a position embedding vector is generated to provide the position information of each patch in the image data;

[0142] Feature vectors in a one-dimensional vector space are extracted by combining position embedding vectors;

[0143] A multilayer perceptron is used to identify the food categories in image data based on feature vectors.

[0144] In this embodiment, information and content generation can be automatically achieved through techniques such as teacher-student models or quantification: images. These images can include categories such as ingredients, food, beverages, soy sauce, and vinegar. Figure 2 As shown in 107, the categories of food ingredients in an image can be identified by constructing a large visual model based on the Vi-transformer structural model.

[0145] Specifically, the input image data can first be segmented into multiple non-overlapping patches. These patches can be segmented into fixed sizes, such as 8x8 pixels or 16x16 pixels. Each patch can be represented as a two-dimensional tensor, which can be flattened into a two-dimensional vector. All patches are then unified into a two-dimensional vector space, that is, the two-dimensional vectors of each patch are concatenated into a large two-dimensional tensor, represented as a matrix.

[0146] For a two-dimensional vector space, methods such as one-dimensional convolution and fully connected layers can be used to map the two-dimensional vector space to a one-dimensional vector space of the same length, thus generating patch embeddings. For learnable classification prediction, location embedding vectors can also be generated to provide the positional information of each patch in the image data. Location embeddings can be fixed parameter vectors or learned according to specific needs. By generating location embedding vectors, multi-layered sequence information such as position, part-of-speech tagging, and word order can be obtained, providing more information for the classification embedding vectors.

[0147] Next, the one-dimensional vector of each patch can be combined with the corresponding position embedding vector through simple vector concatenation or weighted summation. Then, a Transformer encoder block is used to encode the vector. After passing through the Transformer encoder block, an encoded and feature-extracted vector representation is obtained, i.e., the feature vector. This feature vector is then passed to a Multilayer Perceptron (MLP) for classification. The MLP can be trained using labeled image data. After training, the MLP can further process and transform the input vector through multiple fully connected layers and activation functions, ultimately outputting the category result of ingredients, food, or fruits and vegetables.

[0148] In this embodiment, the image data is divided into multiple two-dimensional patches to reduce the dimensionality of the data. These two-dimensional vectors are mapped to a one-dimensional vector space and a position embedding vector is generated. This allows the model to capture the position information of each patch in the original image, providing important spatial context for understanding the image content. It integrates local visual information and positional information to form a comprehensive understanding of the image content. Then, a multilayer perceptron is used to analyze the feature vectors, enabling the model to accurately identify the food categories in the image.

[0149] In some embodiments, for a trained model, such as Figure 2 As shown in 110, the model can be deployed and applied, such as providing cloud services, including voice, image, text and video content. It realizes a visual, user-friendly and quantifiable multi-task joint application service interface, which includes model evaluation service, model encapsulation, model plugin, model service activation, calling interface, interface results, matching downstream tasks, etc.

[0150] The interactive method with autonomous learning and generation capabilities provided in this application can be executed by an interactive device with autonomous learning and generation capabilities. This application uses an interactive device with autonomous learning and generation capabilities executing the interactive method as an example to illustrate the interactive device with autonomous learning and generation capabilities provided in this application.

[0151] This application also provides an interactive device with autonomous learning and generation capabilities.

[0152] like Figure 4 As shown, the interactive device with autonomous learning and generation capabilities includes:

[0153] The acquisition module 410 is used to acquire multimodal data; the multimodal data includes at least internal data associated with the refrigeration equipment and external data interacting with the refrigeration equipment;

[0154] The fine-tuning module 420 is used to fine-tune the pre-trained language model based on multimodal data to obtain the fine-tuned pre-trained language model.

[0155] The interaction module 430 is used to train the fine-tuned pre-trained language model using a reinforcement learning algorithm based on human feedback to obtain a large language model, so as to facilitate interaction using the large language model.

[0156] According to the interactive device with autonomous learning and generation capabilities of this application, multimodal data is acquired; the multimodal data includes at least internal data associated with the refrigeration equipment and external data interacting with the refrigeration equipment; a pre-trained language model is fine-tuned based on the multimodal data to obtain a fine-tuned pre-trained language model; a reinforcement learning algorithm based on human feedback is used to train the fine-tuned pre-trained language model to obtain a large-scale language model, which is then used for interaction. This embodiment of the application, by combining multimodal data processing to fine-tune the pre-trained language model, enables the model to more comprehensively learn and understand the usage environment and user behavior of the refrigeration equipment. By employing a reinforcement learning algorithm combined with human feedback, the model can achieve autonomous learning, continuously optimizing its decision-making process to better meet user needs and adapt to complex and changing usage scenarios. This improves the model's ability to handle tasks such as food management with the refrigeration equipment, making the model's output more accurate and significantly enhancing the user interaction experience.

[0157] In some embodiments, the fine-tuning module 420 is further configured to:

[0158] Multimodal data is converted into text data using a model adapted to the type of multimodal data;

[0159] The prompting process designs prompts for each piece of text information in the text data and generates embedding vectors based on the text data to obtain the training dataset.

[0160] The pre-trained language model is fine-tuned based on the training dataset to obtain the fine-tuned pre-trained language model.

[0161] In some embodiments, the fine-tuning module 420 is further configured to:

[0162] When the type of multimodal data is speech data, an end-to-end deep neural network model based on a Gaussian neural network model or an encoding / decoding structure can convert the speech data into text data.

[0163] In some embodiments, the fine-tuning module 420 is further configured to:

[0164] When the type of multimodal data is video data, the learning task is to convert the video data into text data. A temporally and spatially continuous student model is introduced through a pre-set teacher model so that the parameters of the student model are the same as those of the teacher model.

[0165] The student model is trained using transfer learning and knowledge distillation techniques to learn the knowledge of the teacher model, resulting in a trained student model.

[0166] Multimodal data is converted into text data based on the trained student model.

[0167] In some embodiments, the fine-tuning module 420 is further configured to:

[0168] The encoder is optimized, including compressing and pruning the embedding layer and the activation layer with multi-head attention mechanism, respectively.

[0169] The multimodal data is encoded based on the optimized encoder to obtain encoded data;

[0170] The pre-trained language model is fine-tuned based on the encoded data to obtain the fine-tuned pre-trained language model.

[0171] In some embodiments, the fine-tuning module 420 is further configured to:

[0172] Multiple feedforward network layers, residual network layers, and normalized network layers are used to construct feedforward network blocks, so as to perform encoding operations on multimodal data based on the feedforward network blocks to obtain encoded data.

[0173] In some embodiments, the fine-tuning module 420 is further configured to:

[0174] The decoder is optimized by: constructing a decoding module based on attention mechanisms, convolutional layers, activation layers, and normalization layers, so that the decoding module can decode the encoded data to obtain speech data and / or video data; compressing or quantizing the encoded data based on semantic compression / quantization, and decoding it using an autoregressive decoder to obtain text data.

[0175] In some embodiments, the interactive device with autonomous learning and generation capabilities further includes:

[0176] The food ingredient classification module is used to segment image data into multiple patches when multimodal data includes image data, and unify multiple patches into a two-dimensional vector space; map the two-dimensional vector space to a one-dimensional vector space of the same length, and generate position embedding vectors to provide the position information of each patch in the image data; extract feature vectors from the one-dimensional vector space by combining the position embedding vectors; and use a multilayer perceptron to identify the food ingredient categories in the image data based on the feature vectors.

[0177] This application embodiment also provides an interactive system with autonomous learning and generation capabilities, including: a refrigeration device; and a server, including the aforementioned interactive device with autonomous learning and generation capabilities. Taking a refrigerator as an example, the refrigeration device... Figure 5As shown, in this system, the server and cooling equipment can be connected via the internet, mobile network, Wi-Fi, or data cable. In this system, the server can be a single server or a server cluster. With a single server, the training process of a large language model can be completed on that server, and after training, the large language model can also be deployed on that server. Conversely, with a server cluster, model training and deployment can occur on different servers. For example, model training can be done on one server, while model deployment can be done on another, reducing server load through a distributed approach.

[0178] The interactive device with autonomous learning and generation capabilities in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific implementation.

[0179] The interactive device with autonomous learning and generation capabilities in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit it.

[0180] In some embodiments, this application also provides a cooling device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described interactive method embodiments with autonomous learning and generation capabilities, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0181] In some embodiments, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored on the memory 602 and executable on the processor 601. When the program is executed by the processor 601, it implements the various processes of the above-described interactive method embodiment with autonomous learning and generation capabilities, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0182] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0183] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described interactive method embodiments with autonomous learning and generation capabilities, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0184] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0185] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described interactive method with autonomous learning and generation capabilities.

[0186] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0187] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described interactive method embodiments with autonomous learning and generation capabilities, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0188] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0189] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0191] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0192] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0193] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. An interactive method with autonomous learning and generation capability, characterized in that, The method comprises: acquiring multi-modal data; the multi-modal data at least comprises internal data associated with a refrigeration device and external data interacting with the refrigeration device; fine-tuning a pre-trained language model based on the multi-modal data to obtain a fine-tuned pre-trained language model; training the fine-tuned pre-trained language model by using a reinforcement learning algorithm based on human feedback to obtain a large language model, so as to facilitate interaction by using the large language model.

2. The method of claim 1, wherein, The fine-tuning of the pre-trained language model based on the multi-modal data to obtain the fine-tuned pre-trained language model comprises: converting the multi-modal data into text data by using a model suitable for the type of the multi-modal data; designing prompt information for each text information in the text data by prompting engineering, and generating an embedding vector based on the text data to obtain a training data set; fine-tuning the pre-trained language model based on the training data set to obtain the fine-tuned pre-trained language model.

3. The method of claim 2, wherein, The conversion of the multi-modal data into text data by using a model suitable for the type of the multi-modal data comprises: in the case that the type of the multi-modal data is speech data, converting the language data into text data based on a Gaussian neural network model or an end-to-end deep neural network model with a coding and decoding structure.

4. The method of claim 2, wherein, The conversion of the multi-modal data into text data by using a model suitable for the type of the multi-modal data comprises: in the case that the type of the multi-modal data is video data, taking the conversion of the video data into text data as a learning task, introducing a time and space continuous student model through a preset teacher model, so that the parameters of the student model are the same as those of the teacher model; training the student model by using a transfer learning and knowledge distillation technology to learn the knowledge of the teacher model to obtain a trained student model; converting the multi-modal data into text data based on the trained student model.

5. The method of claim 1, wherein, The fine-tuning of the pre-trained language model based on the multi-modal data to obtain the fine-tuned pre-trained language model comprises: optimizing the encoder, including: respectively compressing and pruning the embedding layer and the activation layer with multi-head attention mechanism of the encoder; encoding the multi-modal data based on the optimized encoder to obtain encoded data; fine-tuning the pre-trained language model based on the encoded data to obtain the fine-tuned pre-trained language model.

6. The method of claim 5, wherein, The optimization of the encoder further comprises: constructing a feedforward network block by using multiple feedforward network layers, residual network layers and normalization network layers, so as to perform encoding operation on the multi-modal data based on the feedforward network block to obtain encoded data.

7. The method of claim 1, wherein, The method further comprises: optimizing the decoder, including: constructing a decoding module based on attention mechanism, convolution layer, activation layer and normalization layer, so that the decoding module decodes the encoded data to obtain speech data and / or video data; compressing or quantizing the encoded data based on semantic compression / quantization, and decoding by using an autoregressive decoder to obtain text data.

8. The method of claim 1, wherein, The method further comprises: In the case of the multi-modal data including image data, the image data is split into a plurality of patches, and the plurality of patches are unified into a two-dimensional vector space; The two-dimensional vector space is mapped to a one-dimensional vector space of the same length, and a position embedding vector is generated to provide position information of each patch in the image data; A feature vector of the one-dimensional vector space is extracted in combination with the position embedding vector; A multilayer perceptron is used to identify a food material category in the image data according to the feature vector.

9. An interactive device with autonomous learning and generation capabilities, characterized by, Comprising: An acquisition module configured to acquire multi-modal data; The multi-modal data at least includes internal data associated with a refrigeration device and external data interacting with the refrigeration device; A fine-tuning module configured to fine-tune a pre-trained language model based on the multi-modal data to obtain a fine-tuned pre-trained language model; An interaction module configured to train the fine-tuned pre-trained language model using a human feedback-based reinforcement learning algorithm to obtain a large language model, so as to facilitate interaction using the large language model.

10. An interactive system with autonomous learning and generation capabilities, characterized by, Comprising: A refrigeration device; A server comprising the apparatus of claim 9.

11. A refrigeration appliance, a memory, a processor, and a computer program stored on the memory and loadable into the processor, characterized in that The processor implements the method of any one of claims 1-8 when executing the program.

12. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the method of any one of claims 1-8 when executed by the processor. The computer program implements the method of any one of claims 1-8 when executed by the processor.

Images